Pneumonia AI: combining Computer Vision & Augmented Reality
By: Iman Zadeh, Sr. Principal Applied Scientist, Oracle Health and AI
Thanks go to the following co-authors for their contributions to this article: Paul Parkinson, Architect and Developer Advocate for Microservices with Oracle DB; Olaitan Olaleye, Principal Applied Scientist, Oracle Health and AI; and Simo Lin, Sr. Software Developer, Oracle Health and AI.
COVID-19 has taught us many lessons, including the importance of accessibility and agility in healthcare. For example, let’s consider Pneumonia1, which many of us have had to deal with during the COVID-19 era. Chest X-ray imaging is widely regarded as the gold standard for detecting Pneumonia2. The pandemic showed us that we need a greater volume of human resources to serve the large population of people suffering from Pneumonia at any given time. Also, there tends to be a notable imbalance in the ratio of patients to available specialists, resulting in longer waiting times3.
Imagine a situation where patients are allowed to review their own X-rays, whether in the form of a physical print off,
on a screen or with VR while waiting for their specialists. Now imagine there was a technology that a less-practiced specialist or a nurse could use to detect Pneumonia with a high confidence level by looking at the X-ray images through an head-mounted display (HMD), or head-mounted display. If it existed, this technology would provide impactful benefits for the whole healthcare system, like being used to separate high-risk patients who need care more urgently.
Artificial intelligence (AI) and machine learning (ML) are playing a greater role in many areas of our lives, effecting a range of industries from transportation to education and healthcare4. Many cloud service providers now offer ML model training services5 to train large-scale and complex models without the need for an extensive background in computer science and computational infrastructure. This means that building AI/ML models and integrating them into products has never been easier.
In the last 15 years, there has been considerable research and development in Virtual Reality (VR), Augmented Reality (AR), and Extended Reality (XR)6, transforming these fields from the stuff of sci-fi to real, usable tech. In addition, many studies have underlined VR/AR/XR’s benefits in different medical applications and that AI/ML could play a crucial role in them7.
Merging AI/ML with XR
The following block diagram illustrates the high-level concept of operation for the above-mentioned use case and technology:
When a user looks through their HMD, the object detector separates the X-ray images from other objects. It then sends the detected object to the Pneumonia Classifier for classification. The object detection and classification outputs are augmented on the HMD display.
Developing computer vision models
We used the custom model object detection and image classification components of the OCI Vision Service8 provided by Oracle Cloud Infrastructure (OCI)9 to build a proof of concept for this technology. Using OCI offers fast and efficient computing power to train large-scale models and maintains the high availability and scalability of the developed technology.
We trained the object detection and image classification models on open-access datasets using the custom model training capability provided by OCI Vision Service. To learn more about the features and capabilities provided by the Vision Service, click here8. We achieved 99.03%% and 94.15% of accuracy for our x-ray detection and classification models, respectively. After the model training stage, we just need to deploy the models and connect the outputs to the HMD.
XR implementation — the details
The HMD used in this case is a Hololens 210. The solution can be implemented in a variety of ways and to different degrees using other HMDs or even hand-held devices. The application running on the Hololens (written using Unity and C# language) takes pictures with its built-in camera using the wearer’s field of vision as a reference and does so at regular intervals. This provides a hands-free experience that is, at the very least, more convenient. Still, in many situations like in the operating room, it’s necessary for users’ hands to interact with the real (or XR) world, thus making it a more optimal solution. This also means the system can pick up contextual information that the practitioner may not be aware of, or have access to, while quickly gathering and processing such information without the practitioner having to explicitly instruct it to, thus saving time as well. This leads to the optimization that mixed reality provides, an exciting byproduct of immersion.
The Hololens then sends these pictures to OCI Object Storage11 via secure REST calls, where they can be conveniently accessed by the OCI Vision AI Service8 directly, and can also be stored in a database. There are several different approaches and architectures that can be used from this point to conduct the logic and calls to the Oracle Vision AI APIs for processing the images sent by the XR device. For example, as far as language used, initial versions were written using the OCI CLI, Java, and Python, with a final Java GraalVM-native image version being written. Also, initial versions would monitor the Object Store for image upload activity. However, as the Java GraalM native image starts almost instantaneously and the service conducts a particular short-lived routine, it is a good candidate for a serverless function. Optionally, the OCI Notification Service can listen to Object Store changes and call serverless functions as a result.
The Java service receives a notification of image upload to object storage and conducts a series of actions as follows:
- Conducts an OCI Vision AI service API call backed by an x-ray object detection model and provides it the object storage location of the image sent by the Hololens.
- Receives a reply from the object detection model with the percentage chance of the x-ray being in the image and the bounding coordinates of it.
- Crops the original image using the bounding coordinates.
- Conducts another OCI Vision AI service API call backed by an x-ray classification model and provides it the cropped image of the x-ray.
- Receives a reply from the image classification model with percentage change of x-ray containing signs of abnormalities/pneumonia.
- The Hololens application receives this reply and notifies the wearer with an audible notification — this is configurable and can also be visual. In the case of this application, the information includes the picture of the cropped x-ray with its discovered details listed and stored in a virtual menu located on the wrist and viewable by the wearer only. This approach prevents the interruption of the wearer though it is also possible to overlay the results on the real-life x-ray from which they were derived.
ML and AI development becomes accessible and the possibilities endless
Before the availability of ML capabilities on cloud services, the development of ML models was limited to people with extensive backgrounds in ML/AI. Things have changed and now customers or curious individuals can train and deploy models for industry-specific use cases without having ML expertise by using their own data.
In this blog and by using the custom model training capability of the OCI Vision, we hope to have demonstrated one of the many applications in which AI and XR technologies could merge and be used in health and life sciences. Since our study is a showcase, we just used non-specific pre-built image classification and object detection models as a baseline for custom model training. The results could be better if we used a base model which was initially trained on similar images (such as x-ray from the chest or other body segments, MRI scans, and such). At Oracle Health and AI, we strive to provide the best services. So, stay tuned for more news and research on the general use of AI and VR/AR/XR, specifically in health and life sciences!
References
1. Pneumonia — Symptoms and causes — Mayo Clinic. https://www.mayoclinic.org/diseases-conditions/pneumonia/symptoms-causes/syc-20354204.
2. Inui, S. et al. The role of chest imaging in the diagnosis, management, and monitoring of coronavirus disease 2019 (COVID-19). Insights Imaging 12, 155 (2021).
3. Bhatt, A., Ganatra, A. & Kotecha, K. COVID-19 pulmonary consolidations detection in chest X-ray using progressive resizing and transfer learning techniques. Heliyon 7, e07211 (2021).
4. The state of AI in 2022 — and a half decade in review | McKinsey. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2022-and-a-half-decade-in-review.
5. AI Services and Tools for Your Business. https://www.oracle.com/artificial-intelligence/ai-services/.
6. AR, VR, MR, and XR — what they mean and how they’ll transform lives. https://www.qualcomm.com/news/onq/2022/09/ar--vr--mr--and-xr---what-they-mean-and-how-they-ll-transform-li.
7. Bansal, G., Rajgopal, K., Chamola, V., Xiong, Z. & Niyato, D. Healthcare in Metaverse: A Survey on Current Metaverse Applications in Healthcare. IEEE Access 10, 119914–119946 (2022).
8. AI Computer Vision Capabilities. https://www.oracle.com/artificial-intelligence/vision/.
9. Discover the Next Generation Cloud Platform. https://www.oracle.com/cloud/.
10. HoloLens 2 — Pricing and Options | Microsoft HoloLens. https://www.microsoft.com/en-us/hololens/buy.
11. Overview of Object Storage. https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/objectstorageoverview.htm.
~
About the Author:
Iman completed a Ph.D. in bioengineering, followed by two postdoctoral fellowships in neuroscience at UC Davis and UCLA. He worked at UCLA as a project scientist in the first multi-site project funded by the National Institute of Mental Health (NIMH) to discover biomarkers for autism using advanced machine-learning approaches. Then, he joined Hughes Research Laboratories (HRL) in 2016 as a staff research scientist to design revolutionary brain-machine interfaces to increase users’ efficiencies while interacting with computers. Iman joined Oracle in 2020, is a Sr Principal Applied Scientist at Oracle Health and AI and works on state-of-the-art machine and deep learning projects.
