SWE Bench Pro is also gamed and shouldn't be trusted. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

saberience 26 days ago | parent | context | favorite | on: GPT-5.6

SWE Bench Pro is also gamed and shouldn't be trusted.

mnicky 26 days ago | [–]

Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.

Sabinus 26 days ago | [–]

Why that benchmark in particular?

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact