Oossa

OpenAI researchers rely on coding agents, despite their limits

OpenAI says its researchers’ use of coding agents has climbed sharply. The tools handle routine work well, but still need close human oversight.

By Published by Oossa: 3 min read

Mohammad Rahmani · Unsplash

OpenAI says its researchers are using coding agents more heavily, even as the systems struggle with complex engineering and produce code that needs careful review. In October, the median researcher’s use of internal tools such as Codex and Dots was equivalent to $2,400 in tokens per day at regular API list prices, up 62 percent from September. For the most active 10 percent of users, the equivalent reached $23,000 per day.

Those figures estimate what the usage would cost at public list prices; they are not cash payments or researchers’ actual salaries. OpenAI says the agents ran for the equivalent of 5.7 eight-hour workdays per researcher per calendar day, and about eight days in pre- and post-training teams. More than two-thirds of tracked researchers ran at least four agents at once during peak use. The company’s comparison of agent costs with salary costs uses September figures and excludes equity, which OpenAI says makes up 70 to 80 percent of its employees’ total compensation. Agent use was 2 percent of the company’s overall research-compute use.

Agents do better on routine work

OpenAI reports that the share of autonomous agents completing two- to four-hour tasks without human intervention rose from 23 percent in March to 46 percent in October. On four- to eight-hour tasks, the reported success rate went from 13 to 35 percent. The results vary sharply by task: agents completed training-run monitoring 79 percent of the time and answered technical questions 72 percent of the time, but succeeded only 29 percent of the time when building evaluation or reinforcement-learning environments, and 25 percent when writing code for inference and serving systems.

The company also describes strong employee appetite for the tools, while cautioning that its surveys are limited. In an internal, non-representative survey of 101 employees, the median respondent said they would rather lose 40 percent of their team’s staff or 30 percent of its compute capacity than go three months without current coding agents. In a separate quick poll of 28 employees, 27 preferred the latest internal model to a mid-career human engineer. Those answers show enthusiasm among surveyed staff, not a general measure of agent performance.

Human judgment still matters

Interviews with ten OpenAI researchers point to a persistent weakness: agents lack what interviewees called “taste” for promising research ideas and for good code or data. They can make incorrect assumptions, take unnecessary steps, and produce code changes or data that researchers cannot use. The researchers said close oversight is needed to prevent wasted computing time. Teams working on post-training and safety said the key ideas for GPT-6 Astra and GPT-6.2 Astra came from people; some pretraining researchers said Codex helped them reach important insights while working on GPT-6.2 Astra.

OpenAI presents the figures as indicators of research acceleration and possible recursive self-improvement, where AI helps build later AI systems. But the company warns that current results may not scale in a straight line. The Decoder also notes that METR estimated AI use had nearly doubled development pace at Anthropic during an 11-week examination; that is a separate organization’s estimate, not a direct comparison with OpenAI. OpenAI had said in early September that it met its own goal of an automated AI research intern, while chief scientist Jakub Pachocki warned about control and oversight gaps.

Why it matters

For people following how AI research is done, OpenAI’s figures show that coding agents can take on some routine work and that its researchers value access to them. But the reported failures on complex engineering, along with researchers’ accounts of unusable outputs, mean the company’s data does not show that agents can replace research judgment or work independently across the process.

Was this article useful?
Share

Read next

Oossa · Newsletter

The week in AI, explained

Every Monday: the stories worth knowing, in plain language. Free, no spam.