Separating signal from noise in coding evaluations

OpenAI released an analysis that uncovers problems in SWE‑Bench Pro, a widely used benchmark for assessing AI models on programming tasks. The report highlights concerns about the benchmark’s reliability and accuracy, questioning the validity of current evaluations of such models.

Summary written by Kernelia from the original article by OpenAI News. The story and its rights belong to its author.