Software developed at Tufts has been successful at identifying COVID-19 pneumonia in more than 99 percent of the X-ray images it processes
For patients with COVID-19, terrifying shortness of breath can set in virtually overnight. In many cases, it’s caused by an aggressive pneumonia infection in the lungs, which fills them with thick fluid and robs the body of life-giving oxygen.
Detecting these severe cases early on is essential for treating them successfully. At the moment, however, the only way to tell whether a patient’s pneumonia is caused by the coronavirus is by examining X-ray and CT scans of the chest—and as cases rack up worldwide, radiologists are being deluged with images, creating a backlog that may delay critical decisions about care.
One solution, said Karen Panetta, may involve taking some of that workload away from humans. Panetta, a professor of electrical and computer engineering at the School of Engineering, is working to create artificial intelligence (AI) that can spot cases of COVID-19 pneumonia and flag them for review.
Using X-rays and CT scans from an international COVID-19 database, her lab is training AI software to comb through thousands of images, matching those that share similar traits. By comparing X-rays of pneumonia caused by bacterial infections, chronic smoking, and COVID-19, she says, the AI can gradually learn to identify features unique to each one, be it a particular shape, area of contrast, or other trait. Once the software finds potential matches, it uses statistical analysis to sort COVID cases from non-COVID ones.
“Think of it this way: If you’re watching football, and you’ve seen the same play run fifty times, you know what each player will do if their team runs it again,” said Panetta, who is also dean of graduate education at the School of Engineering. “But if a new guy comes in and you haven’t seen him before, how are you going to predict what he’ll do? You’ll look back on the last fifty times you saw the same play and take an educated guess based on your experience. Our software is basically doing the same thing with these images.”
Panetta’s COVID-19 work builds on research that her lab has already been doing to detect cancerous tumors. In breast cancer, she notes, her AI software looks at the nuclei of individual cells in a biopsy sample, and searches for distinct patterns that match known cases.
Cancer-free samples tend to have orderly nuclei contained in an oval structure, but if the cancer progresses, those patterns tend to break down. Using AI and machine learning, it’s possible to train the AI to spot new cancer cases autonomously based on those traits.
“We had already developed all these tools for image processing, machine learning, and AI methods for cancer, so COVID-19 was just a more timely application of the same technology,” she said. “We’re just tuning the software for a different use case.”
The results are already promising. So far, her lab’s software has been successful at identifying COVID-19 pneumonia in more than 99 percent of the images it processes.
Getting to that point hasn’t been so simple, however. The machine learning tools she uses to train the software are only as good as the data they’re fed—and while humans can easily ignore slight imperfections in an image, those same glitches can trip up even the best machine.
In some of the images in the COVID-19 database, she noted, large black rectangles appear where patient’s personal information has been blocked for anonymity. In others, technicians have underexposed the X-ray, making the entire image slightly cloudy, or have superimposed X-ray and CT scans, creating a confusing hybrid image.
“X-ray and CT scans aren’t always in pristine condition. They require a lot of enhancement and pre-processing to clean up those imperfections so they’re on equal footing,” she says. The AI also has to be smart enough not to misdiagnose an image because it sees anomaly.
“Everyone thinks AI is this magical black box, but it’s not Zoltar,” Panetta said, referring to the all-knowing fortune-telling machine from the Tom Hanks movie, Big. “You have to constantly tweak it to improve it.”
Another complication, she added, is that while AI can identify images that look like other cases of COVID pneumonia, it can’t tell exactly why those images meet the criteria from a medical point of view. To fill in those gaps, Panetta is looking to team up with experienced radiologists at Tufts, and wants to add medical annotation and context to each image.
Even if that improved AI software isn’t available to clinicians during the current pandemic—which it very well may not be, since FDA approval can take years—Panetta hopes it could still be used in the future to educate medical personnel. If another outbreak happens down the road, she reasons, hospitals will need all the training they can get.
“Right now, even doctors on the front lines have probably only seen a few hundred cases of COVID-19 pneumonia, but there are hundreds of thousands of cases happening worldwide,” she said. “If we can aggregate all that data into one place with images, symptoms, and patient info, it may be possible to use AI to study the disease more effectively,” she said.
That could help identify the patterns the cases all share. “For doctors who have never seen a patient with COVID-19,” said Panetta, “it could generate a portfolio that tells them what to look out for.”
David Levin is a Boston-based freelance science writer.