Too many of the AI tools we are being sold have been built around the data we happen to have rather than the problems that actually need solving. We need to invert the pipeline and let practitioners build the solutions they need rather than let engineers decide what should be built.

A surprising number of artificial intelligence (AI) solutions in the radiology space focus on ‘segmentation’—the process of tracing the outline of an organ or tumour on a scan. AI has always excelled at image recognition, and with statistical metrics like Dice to measure spatial overlap and similarity between image segmentations, it has become highly accurate at this task. Which is why I was surprised to learn, a few weeks ago, that notwithstanding the proliferation of these tools, radiologists have very little use for segmentation in their daily practice.

We tend to develop AI solutions around the data we have, rather than the things that are hard to do. This is like the drunk who lost his keys in the park but is searching for them under the streetlight simply because that is where the light is better.

What might be a far better use of our AI resources is deploying them for triage. If we can get AI to review the hundreds of sections of an MRI scan and identify the few that really matter for the diagnosis, this will be of significant benefit to busy radiologists. The trouble is that triage is a much harder problem to solve, and the risk that AI might fail to identify cases that ought to have been highlighted for referral is one that few AI developers are willing to take. Which is why so few triage solutions are being built.

In India, the stakes are higher than almost anywhere else. With roughly one radiologist for every 100,000 people, each has to process more scans in a day than is humanly possible. What we need are not tools that draw perfect outlines, but ones that identify the scans that deserve attention.

Repeating Patterns

This pattern repeats across different areas of medical practice. Even though AI can distinguish between malignant melanoma and harmless moles, there is no point in deploying AI solutions for this in India, where melanomas are relatively rare. What’s worse, since these models struggle to identify melanomas in people with darker pigmentation, they will fail those who would have benefited the most from earlier detection.

In other instances, even though the task may be right, the setting is wrong. While Epic’s sepsis-prediction model performed acceptably in testing, when it was deployed in American hospitals, it missed close to two-thirds of sepsis cases and generated so many false alarms that these warnings became part of the background noise in busy hospitals. As much as it may have been built to solve real-world problems, its deployment failed to account for the environments within which it would be used.

This is not a problem unique to medicine. Legal-AI firms have focused on developing contract-drafting solutions because it is easier to train models on the hundreds of thousands of contracts available in a law firm’s archives than to get them to understand how to exercise the judgement and experience that lawyers are actually paid for. AI solutions like this are the result of searching for keys under streetlights. They tend to be built by technologists who define problems based on what data is available instead of what problem needs solving.

What We Need

One of the most exciting medical AI solutions over the past two years is arguably the Ambient Scribe—a software that listens to a consultation and drafts a clinical note. This is AI being used not to implement clinical judgement or provide medical expertise, but to solve the one thing doctors complain about most—the hours they spend documenting their patient care. According to the Permanente Medical Group, this solution is used 2.5 million times in the first year and has returned close to 16,000 hours of physician time.

Another example of a useful AI solution is the chest X-ray tool developed by Indian healthcare startup Qure.ai, which helps screen for tuberculosis. Now deployed across scores of Indian health facilities, it resulted in a 15% increase in the number of cases detected in at least one evaluation. This solution is useful for the same reason that segmentation is not: it does the job the practitioner needs done.

In domain after domain, experience has shown that rather than building solutions from data that happens to be available, it is far more effective to focus on the problem that needs solving. India’s Saagu Baagu programme was successful precisely because it was designed around what Indian chilli farmers asked for. The Mozhigal educational AI solution, which I discussed at length in a previous article in this column, has been successful because it was a solution designed for slow learners rather than an AI overlay on existing education workflows.

Building What’s Needed

We need to invert the way we build. Rather than asking engineers what their models can do for a given sector, we need to get the practitioners in that domain to tell model builders what really needs to get done and how. This shift in approach should be reflected in our procurement processes as well, as a tender that asks for an “AI-powered diagnostic solution” is letting the technology define the problem. Instead, practitioners should specify the outcomes they need and then leave it to AI specialists to figure out how to implement them.

In most instances, the reason AI fails is that it is answering questions that don’t need answering. This, in turn, is typically because it was built by those who know what the technology can do but have no idea about what actually needs to be done.

The light may be better under the streetlight. But the keys will still be lying in the park.