# Study Finds Gaps in LLM Code Verification

> An arXiv paper separates verifier acceptance from proof-to-spec binding and real-world correctness, finding large gaps across three code benchmarks.

Oossa · 2026-11-05 · https://oossa.com/en/study-finds-gaps-in-llm-code-verification

A paper posted to arXiv on November 5 examines what “verified” means for code generated by large language models. It separates three checks: whether a verifier accepts code, whether its proof is bound to a specification, and whether that specification reflects the original requirement.

In one test, the authors report 99.1% verifier acceptance on HumanEval, but only 1.9% of samples had a certificate tied to a spec-defined predicate or function. In a separate Rust pipeline designed to ensure that binding, 11.48% of structurally verified MBPP-Rust samples and 14.82% on a VeriContest slice still failed hidden tests.

## The facts

- The paper reports 99.1% verifier acceptance and 1.9% spec-bound certificates on HumanEval.
- On the VeriContest LC slice, the authors report a 7.79% end-to-end rate (535 of 6,870); comparisons with other models were not matched for task budget, repair or test harness.

## Why it matters

For teams using proof-checking to assess generated code, the results show that verifier acceptance alone may not establish either a meaningful link to requirements or correctness on hidden tests.

## Sources & references

1. [Verified but Vacuous: Three Failure Surfaces of LLM-Driven Verified Code Generation](https://arxiv.org/abs/2611.05590) – arXiv, 2026-11-05

Last updated: 2026-11-05