Google LLC and the U.S. Department of Defense are reportedly working on an artificial intelligence-enabled microscope project designed to help pathologists streamline their workflow without interrupting their established processes.

Called the Augmented Reality Microscope, the microscope brings real-time machine learning into an optical microscope for applications in cancer diagnostics and other areas that rely on bright-field microscopes for visual specimen inspection. According to the GitHub page for the project, the ARM offers novel optics for parallax-free digital overlay projection in an optical microscope, real-time machine learning inference and state-of-the-art convolutional neural networks for accurate cancer detection and classification.

A convolutional neural network, known as a CNN, is a type of deep learning model that is well-suited for image processing tasks. CNNs work by extracting features from images using convolutional filters, which are small matrices of weights. The features are then used to classify the image or detect objects within it.

The idea behind the ARM is detailed in a scientific paper published in 2019. The paper describes a process to overlay AI-based information onto the current view of the sample in real time, enabling seamless integration of AI into routine workflows.

The involvement of the DOD in working on the ARM project with Google comes from a report by CNBC, which describes how doctors are using early versions of the ARM in lab tests to diagnose conditions like cancer. In one example, doctors at a hospital in Seattle used an ARM to determine the severity of prostate cancer in a patient’s case.

As of now, there are only 13 ARMs in existence. The early models are described as “a lot like a microscope that could be found in a high school biology classroom,” but the microscope differs in that it is also connected to a computer that houses the AI models.

Like a traditional microscope, a user places a glass slide in the ARM and then the AI can outline where cancer cells are located. The AI can also indicate how bad the cancer is and generates a heat map that shows the boundary of the cancer in pixelated form.

The ARM, at least for now, is not meant to replace existing digital pathology systems but is being pitched as a way to help health organizations bypass the need to use such systems. One key feature is the ability to take screen grabs of slides, which is less expensive to store using an ARM.

Currently, an ARM will cost around $90,000 to $100,000, but that will likely change with time as the technology matures. The technology is still being developed, with Google now offering four algorithms for the ARM, which can identify breast cancer, cervical cancer, prostate cancer and mitosis.

Image: Ideogram