Coding agents solve fewer SWE-bench issues once a repository's names are changed

SWE-bench Verified is the score AI coding tools quote to show they can fix real bugs. A team from Shanghai Jiao Tong University and two other universities changed how the code inside it looks, kept its behaviour identical, and ran four models again. Every model solved fewer issues. Their paper, Schrödinger's Code Repository, went up on arXiv on August 21st.

Slide text

Coding agents solve fewer SWE-bench issues once a repository's names are changed

Silin Chen, Yufei Yang, Xiaodong Gu and colleagues · Shanghai Jiao Tong University, Xi'an Jiaotong University, East China Normal University

Chen et al. · Schrödinger's Code Repository · arXiv 2609.27891 · 21 Aug 2026

The benchmark is 500 real issues from popular open-source Python projects. An agent gets the issue and the code, makes a fix, and the project's own tests decide whether it worked. The catch is that these projects are public and famous. Django is one of them, and their code, issues and fixes can end up in a model's training data.

Before changing anything, the authors checked what the models already remembered. Reviewers gave each model the issue's ID and a few details from it, with no code, and compared its answers to the real fix. For every model, about two thirds of the issues showed it knew details it hadn't been shown. For about one in five, it recalled the patch or the test itself.

So the authors built a way to take that familiarity away.

Their tool, SchrodingerRepo, builds a fresh copy of each repository whenever an agent starts. It rewords the issue, renames the project's own classes, functions and files, reorders the functions inside a file, and rewrites the code around the bug. Each version is checked against the project's tests, so the bug and the correct fix stay exactly the same.

Here's their own example. Django's QuerySet class becomes LedgerSuite. The swap is consistent, so query turns into ledger everywhere it appears, while Python itself and outside libraries keep their real names. Every command the agent runs is translated back, so it still works on the real project underneath.

Then they ran four models on both versions.

Here's the share of issues each model solved, on the original code and on the renamed code. GPT-5.4 mini fell from about 47 percent to about 36. DeepSeek, the strongest here, fell from about 73 to 67. Across the four models the drop ran from 6 to 14 percentage points, and the authors report it as statistically significant.

Tested one change at a time, the renaming did most of the damage. Remapped names alone cost each model six to seven points. Rewording the issue barely mattered. Reordering and rewriting the code cost a few points at most. The authors read this as agents leaning on names they have seen before.

The cost went up too. On the renamed code, each agent read between two and a half and three and a half times as many tokens per issue. More than 80 percent of the extra steps went to finding its way around: listing folders, searching and reading files, and running small probes. Editing and testing took about the same effort.

One Django bug shows the difference. On the original code, the agent searched for Django's familiar method name and found the faulty line almost at once, in 37 steps. On the renamed code it listed folders, searched the source and the tests, and read configuration files first: 217 steps. It submitted the same fix both times.

Could renaming just make the bugs harder? The authors checked on 110 newer issues from SWE-rebench, created after GPT-5.4 mini was released, so it's unlikely to have seen them. Its score stayed at about 17 percent either way. Only the cost went up, by about a fifth. So the renamed repositories are just as fixable, and the extra cost comes from unfamiliar code.

Here's what the paper does and doesn't show.

Here are the limits. Every project was Python. Every run used one simple open-source agent, mini swe agent, with command-line tools only, and the authors say agents that navigate with IDE or language-server tools need more work to test. Three of the four models are small or fast versions, and the check on newer issues used one model and 110 issues. It's a preprint.

For anyone choosing a coding agent, a SWE-bench Verified score partly measures how well a model knows famous projects. The paper didn't test private code, but a private codebase sits outside the models' training data. On unfamiliar code these agents explored far more and read several times the tokens. The authors argue for testing on code a model can't have memorized, and newer issues or a trial on your own repository do that.

One limit to keep in mind: this is four models, one agent and Python only. Within that, taking away the familiar names cost every model points and multiplied how much it read. The paper is on arXiv, and the code and data are on GitHub. That's how coding agents fare on SWE-bench once a repository's names are changed.