Siddhartha Dalal got his introduction to probabilistic analysis in the wake of the 1986 Space Shuttle Challenger disaster.

Dalal’s research on behalf of the National Academy of Sciences found that NASA’s estimates of a 0.5 percent risk of the o-ring gasket failure that caused the explosion was dramatically off-target. At the 31-degree Fahrenheit air temperature on the morning of the launch, the risk was more than 16 percent. In other words, the Challenger lifted off with a one-in-six chance of exploding.

“There was no evidence of failure because 24 flights had happened without incident,” he told the MIT Chief Data Officer and Data Quality Symposium on Thursday, “but there had been partial failures that could have formed a better statistical base.” In fact, that data would have shown a clear correlation between temperature and o-ring failure.

As chief data scientist and senior vice president at American International Group Inc., Dalal is now applying technologies like computer vision, natural language processing and sensors to probabilistic risk analysis. Big data, he said, is transforming the discipline of insurance underwriting.

“The insurance industry all about understanding risk,” he said. “The last four years have been amazing in the advances we’re making.”

The discipline of risk analysis that underlies insurance underwriting has historically relied on humans asking the right questions, but machine learning excels at letting the data determine what questions to ask. This yields correlations humans would never see.

For example, Dalal showed how text mining could have revealed the health risks in materials such as asbestos, methyl tertiary butyl ether, and Bisphenol A years before the public alarm was raised. A frequency analysis of 20 million academic articles conducted by Rand Corp. showed a clear body of scientific evidence existed, but the dispersion of articles was too great to be obvious to humans.

“There are more than 5,000 monthly medical journals appearing every month alone. The PubMed database has 20 million articles. How can we read this many?” he asked.

Machines can, and that’s why Dalal sees so much promise in algorithms that find patterns humans can’t. For example, computers are much better at recognizing pattern in images than humans. They can achieve 97 percent accuracy in facial recognition, compared with less than 85 percent for humans.

Those capabilities can applied to analyzing photos of a construction site for look for safety risks or identifying weaknesses in bridge supports, for example. In the same way that a more complete correlation of temperature to o-ring failure might have prevented the Challenger disaster, machine learning works best when “we’re not looking for the obvious risks. We’re looking at areas where there is no clear evidence of risk at all.”

Machine learning can also harmonize seemingly dissonant trends, such as increasing life expectancy and growth in life-threatening diseases like diabetes and Alzheimer’s. Underwriters can use these models to find correlations that enable insurers to price policies with pinpoint precision and find new entirely new correlations.

Does that mean that human underwriters will become obsolete? Not at all, in Dalal’s view.

“It’s not about AI [artificial intelligence], it’s about IA – intelligence amplification,” he said. “Machines perform well, but with a high rate of false positives. Humans perform less well, but with a high rate of false negatives.” The combination will be revolutionary.

Photo: Paul Gillin