Research
Better at biology. Still getting experiments wrong.
We built the BenchBench-Cloning benchmark with Benchling to test whether models can turn a scientist's objective into a complete cloning plan.
We built the BenchBench-Cloning benchmark with Benchling to test whether models can turn a scientist's objective into a complete cloning plan. We found a lot of the work can be right, but the experiment still breaks.
We evaluated eight AI systems on 65 molecular cloning tasks. The strongest, GPT-6 Astra, scored 30.6% overall. Pass@10 did help, with GPT-6 Astra solving 36 of 50 tasks at least once given ten tries. However, 14 tasks were never solved by it.
We checked protocols in a cloning simulator and designs with task-specific checks, including LLM-as-a-judge on some tasks. The 65 tasks came from published studies and experiments, with expert reformulation to reduce the chance that agents had seen them before. An important limitation is that we only tested experimental planning and design, not actual wet-lab outcomes, and we used one-shot evaluations rather than multi-turn conversations.
One recurring failure was an incomplete experimental design. In one task, two related constructs needed the same unwanted sequence feature removed. Astra and Sol consistently fixed the first, but neither fixed the second. In another, 38 of 39 submitted panels correctly included four expected constructs, but every panel missed a fifth control needed to distinguish two explanations for the result. What the models did focus on was largely correct, but they often ignored necessary aspects of the solution.
Other plans broke because the steps did not work together. Models picked primers that targeted the right sequence but could not work together under the proposed conditions, overlooking things like methylation constraints and vector preparation that affect the reaction. Many steps were individually reasonable, but invalid in combination.
There were also gaps in the build plans. Some answers had correct final sequences but no complete route to build them. Missing oligo tables, unresolved source references, and fragments with no reaction to produce them left the scientist to fill in the gaps.
These are the kinds of failures we designed DNAForge around.
DNAForge provides one workspace for humans and agents to see the whole picture, from lab notebook and inventory to DNA design and sequencing, building a unified brain for genome-scale design, assembly, and analysis.
The BenchBench-Cloning paper is co-authored by Aditya Sivakumar, Peter Ciaccia, Hassaan Qadir, Ian Hsu and Ashu Singhal, with authors from Benchling and DNAForge. This blog post is by Hassaan Qadir.
Sources: Figure 2 and original paper, CC BY 4.0