Scientific discovery is one of the most sophisticated human activities. First, scientists must understand existing knowledge and identify an important gap.
Next, they must formulate a research question and design and conduct an experiment in search of an answer.
Next, they must analyze and interpret the results of the experiment, which may raise another research question.
Can a process of this complexity be automated? Last week, Sakana AI Labs announced the creation of an “AI scientist” – an artificial intelligence system that they claim can make scientific discoveries in the field of machine learning in a fully automated manner.
Using generative big language models (LLM) like those behind ChatGPT and other AI chatbots, the system can generate ideas, select a promising idea, code new algorithms, sketch the results and write a letter which summarizes the experiment and its findings, complete with references.
Sakana claims the AI tool can undertake the full life cycle of a scientific experiment at a cost of just US$15 per paper – less than the cost of a scientist’s lunch.
Those are some big claims. Do they stack? And even if they do, would an army of AI scientists churning out research papers at inhuman speeds really be good news for science?
How a computer can ‘do science’
A lot of science is done openly, and almost all scientific knowledge is written down somewhere (or we wouldn’t have a way to “know” it). Millions of scientific papers are freely available online in repositories such as arXiv and PubMed.
LLMs trained with this data grasp the language of science and its patterns. So it’s perhaps not at all surprising that a generative LLM can produce what looks like a good scientific paper – he’s swallowed plenty of examples he can copy.
What is less clear is whether an AI system can produce one interesting scientific paper. Most importantly, good science requires innovation.
But is it interesting?
Scientists don’t want to be told things they already know. Rather, they want to learn new things, especially new things that are significantly different from what they already know. This requires judgment about the purpose and value of a contribution.
The Sakana system attempts to handle interest in two ways. First, it “flags” new paper ideas for similarity to existing research (indexed in the Semantic Researcher repository). Anything too similar is discarded.
Second, Sakana’s system introduces a “peer review” step – using another LLM to judge the quality and novelty of the paper created. Here again, there are many examples of peer review online at sites such as openreview.net that can guide you on how to critique a paper. LLMs have swallowed these too.
AI can be a poor judge of AI output
Reactions are mixed to the production of Sakana AI. Some have described it as producing “endless scientific gradients”.
Even the system’s own review of its results judges the papers weak at best. This is likely to improve as technology evolves, but the question of whether automated scientific papers are worthwhile remains.
The ability of LLMs to judge research quality is also an open question. My work (soon to be published in Research Synthesis Methods) shows that LLMs are not good at judging risk of bias in medical research studies, although this too may improve over time.
Sakana’s system automates discoveries in computational research, which is much easier than in other types of science that require physical experiments. Sakana’s experiments are done with code, which is also structured text that LLMs can be trained to generate.
AI tools to support scientists, not replace them
AI researchers have been developing systems to support science for decades. Given the large volumes of published research, even finding relevant publications for a specific scientific question can be challenging.
Specialized search tools use AI to help scientists find and synthesize existing work. This includes the aforementioned Semantic Researcher, but also newer systems such as Elicit, Research Rabbit, scite and Consensus.
Text mining tools like PubTator dig deeper into papers to identify key focus points, such as specific genetic mutations and diseases, and their established relationships. This is particularly useful for curating and organizing scientific information.
Machine learning has also been used to support the synthesis and analysis of medical evidence, in tools such as Robot Reviewer. Summaries that compare and contrast claims in papers from Scholarcy help conduct literature reviews.
All of these tools are intended to help scientists do their work more effectively, not to replace them.
AI research may exacerbate existing problems
While Sakana AI states that it does not see the role of human scientists diminishing, the company’s vision of “a fully AI-driven scientific ecosystem” would have huge implications for science.
One concern is that, if AI-generated documents flood the scientific literature, future AI systems may be trained to produce AI and undergo model collapse. This means that they may become increasingly ineffective at innovating.
However, the implications for science go far beyond the impacts on scientific AI systems themselves.
There are already bad actors in science, including “paper mills” churning out bogus papers. This problem will only get worse when a scientific paper can be produced for US$15 and a vague initial demand.
The need to check for errors in a mountain of automatically generated research can quickly overwhelm the capacity of actual scientists. The peer review system is probably already broken, and throwing more questionable quality research into the system won’t fix it.
Science is fundamentally based on faith. Scientists emphasize the integrity of the scientific process so that we can be sure that our understanding of the world (and now, the world’s machines) is valid and improving.
A scientific ecosystem where AI systems are key players raises fundamental questions about the meaning and value of this process and what level of trust we should have in AI scientists. Is this the kind of scientific ecosystem we want?
Karin Verspoor, Dean, School of Computing Technologies, RMIT University, RMIT University
This article is republished from The Conversation under a Creative Commons license. Read the original article.