AI’s real value in drug discovery may be choosing the right experiment

The promise surrounding AI in drug discovery has a habit of receding the closer you get to it. You deploy a tool but it doesn’t quite work as hoped today, yet the reassurance is always that it will work tomorrow. Then tomorrow arrives and the horizon has shifted again.

 This is known as the rabbit problem. Anyone who lived through other hype cycles around new technology inclusing FAIR data will recognise the shape of it immediately.

 Dr Nick Lynch has lived through several such cycles. He spent 13 years at AstraZeneca, was a Co-founder at the Pistoia Alliance and worked on the FAIRPlus project that built out the FAIR Cookbook to help organisations get to grips with making data findable, accessible, interoperable and reusable (FAIR). Since 2014 he has run Curlew Research, a consultancy focused on data and informatics strategy in the life sciences. His starting position is worth stating plainly, because it frames everything that follows:

“FAIR is very much a journey… we can get ahead of ourselves in maybe over-promising what can be done, but also underplaying the effort, especially in the change management, and cost and time that it takes. And I think all of those can probably be applied to AI too.”

That is the warning. The rest of the conversation is about where AI genuinely earns its place, which turns out to be a more specific, more disciplined claim than most of the marketing suggests.

The data problem hasn’t changed; the data type has

It is tempting to treat AI as a new answer to an old problem. Lynch is careful to separate the two. The challenge of ever-increasing data volume, rising density and dimensionality of experiments, and the drug discovery community all ‘fishing in the same global set of targets’ is decades old. Cheap genomics, cloud storage and data lakes were supposed to solve it years ago.

What is actually new in the last few years is narrower: the ability to work at scale with unstructured data, text, literature, images, the contents of PubMed, etc. Lynch is at pains to not let the large language model (LLM) moment erase what came before it:

“Let’s not forget that ML/AI has been with us in drug discovery for decades… from a computational perspective, it’s been more of an evolution and the more traditional ML methods still have a large role to play.”

The honest version of the AI story, then, is not a revolution that replaces statistics and machine learning. It is an additional layer that sits on top of them its value depends entirely on what you point it at.

Enriching the ground truths, not inventing them

This is the heart of the argument where Lynch resists easy language. The instinct is to say that LLMs can verify our scientific knowledge, but he pushes back on that word:

“Verifying, I think, is a strong word… it could be a combination of enriching and validating.”

This distinction matters. Organisations have spent decades and considerable human effort building curated knowledge – for example, knowledge graphs, pathway data and target information – and all with known provenance and assessable maturity. The mistake would be to hope an LLM could conjure that ground truth from nothing or remove the hard work of making data FAIR through some act of magic. Lynch is blunt that this is “too much of a jump in many cases.”

The defensible role is the opposite. Where humans cannot process the sheer volume of available data, AI can be set to work enriching, linking and stress-testing existing, well-validated ground truths. AI can run validation autonomously to some degree, challenging an assumption and testing it against the literature at a scale no human team could manage. Curation that was once entirely human-led becomes, according to Lynch, “human-in-the-loop AI curated” as volumes grow.

However, this is harbouring a hidden risk that deserves acknowledging – and our conversation unearths it. Models, especially those trained on public data, have only ever seen what people chose to publish. ‘Dark data’ – the experiments that failed or were never written up – remain invisible. Enriching a ground truth with AI does not fix that blind spot; if anything it can launder it, lending confidence to conclusions drawn from a systematically incomplete picture. The technique is powerful precisely where the underlying ground truth is sound and unreliable where it is thin.

The actual headline: directional value

If there is one phrase from this exchange worth pinning to the wall, it is this: when asked what AI is genuinely good for in early discovery, Lynch reframes the question around which experiments to run next.

“Ultimately we do need to do experiments… what we are being helped with more now is exactly what experiments to run to enrich our decision process.”

This is a deflationary, and far more credible, pitch than ‘AI will discover your drug.’ Its value is in giving direction, using AI to identify experiments that achieve two things simultaneously: progress the project and enrich the data space for later. Not just for the current question, but for questions not yet asked.

Ultimately we do need to do experiments… what we are being helped with more now is exactly what experiments to run to enrich our decision process.

“It’s using the AI to help us with exactly what experiments will add both directional value – as in, it really progresses the project – but also data that might actually enrich the data space for not just now, but later.”

It is the opposite of the old combinatorial-chemistry reflex of building enormous, undiverse libraries on the assumption that volume equals value. Done well, AI lets a team be laser-focused on the highest-information experiments, while staying imaginative enough to occasionally cast wider, budget permitting, to fill genuine data gaps. Companies that have committed to high-volume experimentation, eg, Recursion, have done so precisely to build out those gaps in the data.

There is an obvious caveat, which Lynch raises himself without prompting: for chemistry, a recommendation is worthless if the molecule cannot be made. Synthetic feasibility must be embedded in the agents and the recommendations, or you simply generate a list of attractive molecules nobody can synthesise.

From ‘make / don’t make’ to navigating the space

The more forward-looking thread is a shift in how decisions get framed. Early machine learning tended towards binary, discriminative calls: make or don’t make, progress or don’t progress. Lynch’s view is that this was always a simplification of the real problem.

Drug discovery is multi-parameter optimisation (MPO), where the boundaries are not clean and changing one property degrades another. Human intuition tops out at three or four dimensions; the data has many more. The case for AI here is genuinely about reach, navigating a high-dimensional space we are not equipped to hold in our heads – and doing so without some of our reductive biases.

“Our human brains, we maybe can only cope at most three, maybe four, but with the dimensionality of data, we’ve got a better chance,” says Lynch.

There is a quiet tension worth flagging. We are, as Lynch notes, “big on reduction”; we like to collapse complexity into something we can visualise in three dimensions, to make it simple so we can act on it. AI’s promise is partly that it need not behave like that. However, a recommendation a team cannot interpret is a recommendation it will struggle to trust or act on. The dimensionality that makes AI valuable is the same dimensionality that makes its outputs hard to scrutinise. That is not a reason to avoid it; it is a reason to be honest that ‘navigating the space’ is a capability still being earned, not a solved problem.

Where this leaves you

The useful version of AI in early discovery, on this reading, is unglamorous and specific: it is not a machine that knows the answer, but a toolset that tells you which experiment to run next –quietly building a richer dataset while doing so. The spend it justifies is spend on direction and spend on data; not spend on the promise.

The discipline it demands is the old one. Define the questions you actually want answered first. Be honest about the gaps in your ground truth. And treat every reassurance that the tool will deliver tomorrow with the same scepticism you would apply to any other rabbit you have been chasing for a decade.