GPT-6-Astra, running in OpenAI’s Codex coding environment, solved 49 of 50 problems in a test of theoretical-physics research tasks. The study’s authors report that the run cost just under $116 in total, or about $2.40 per problem. The test compared commercially available AI agents and gave them no human hints.
With web search enabled, Anthropic’s Opus 5.5 in Claude Code scored 92%. Google’s Gemini 3.8 Flash scored 68%, while the open-weight models DeepSeek-V4.1-Flash and GLM 5.3 scored 54% and 42%. Without web search, Astra and Opus each scored 92%.
A test of calculations, not discoveries
The 50 problems cover 11 subfields, from fluid flows and biological processes to particle physics and cosmology. They are mostly routine research calculations: applying known methods to a modified question, rather than solving major open problems. The authors say tasks like these can serve as early projects for PhD students.
One example required adapting equations from an older paper about black holes and writing a custom program to calculate the result. The researchers say the problems were designed so an answer could not simply be copied from the paper. The test ran each agent twice: with web search and without internet access.
The score has limits
The researchers found that some apparent AI mistakes were actually small errors in the experts’ reference answers. After correcting those answers, Astra’s responses in those cases were correct. The test’s one problem that Astra did not solve stumped every model; the authors also acknowledge that its expected answer may have been unfairly difficult to infer.
A high score on these calculations does not show that an AI can choose worthwhile research questions or make a physics breakthrough. The authors say current agents still struggle to judge which problems are interesting and to switch to a more tractable question when needed. The problem set and test setup are available on Hugging Face.
Why it matters
For physics students and supervisors, the results suggest that some calculation-heavy tasks once used as starter research projects may be easier to automate; the authors say such tasks may no longer be suitable as first projects unless the aim is practice. The test does not show whether AI can choose valuable questions or produce new physics, so its effect on research beyond these calculations remains unknown.
