Why many AGENTS.md and CLAUDE.md files cost more tokens without improving results

Researchers at ETH Zurich benchmarked context files across Claude Code, Codex, and Qwen Code.

Their results are a useful warning, not a universal rule: in that benchmark, the average AGENTS.md or CLAUDE.md increased cost without improving task success. But the useful conclusion is not “context files are bad.” It is that their value depends on what they contain.

1. What the data says

The benchmark reported the following results (see the full paper):

• LLM-generated context files: about 3% lower task success than no file
• Developer-written files: about 4% higher task success
• Both added roughly 20% inference cost and extra steps per task
• On repositories with little documentation, LLM-generated files helped (+2.7%)

Notice the split. In this study, auto-generated files hurt while developer-written files helped. The variable was what the files contained.

The agents were not ignoring those files. When a context file mentioned a tool, usage rose by as much as 160×. The instructions were followed; the problem was that some instructions added noise rather than useful context.

2. Do not let a generated summary become permanent context

It is tempting to run /init, accept a tidy project summary, and leave it there. But a summary can duplicate what the agent can already inspect, become stale, and then send the agent in the wrong direction. An empty file can be ignored; an inaccurate one can be followed.

Use one test for every line:

Is this already in the code?

If yes, consider cutting it. The study found that context files helped when they carried information the agent could not infer from the repository. That helps explain why they performed better on documentation-poor projects and worse when they duplicated available context.

3. What belongs in the file

Five categories are worth considering. Keep the first three at a high level: they are context, not a repository summary. Once a file becomes detailed and stale, it risks recreating the problem the benchmark measured.

A. What this project is

Its purpose, where it runs, its scale, and how critical it is.

Take “an app that identifies plants from a picture.” The same description can imply very different expectations:

• For hobby growers, a wrong answer may be low risk.
• For identifying edible plants in the field, a wrong answer can be dangerous and needs much stricter testing.

Same codebase. Completely different expectations around edge cases, error handling, and testing.

Knowing the stakes helps an agent calibrate the level of care the work requires.

B. The north star

Where the project is actually headed. This one is underrated.

When the agent knows the end goal, it can choose an implementation that fits the direction of travel instead of one that only closes today’s ticket.

A model often cannot infer direction from a snapshot of the current code. That’s exactly the kind of information the data says helps.

C. A very high-level view of the structure

What technology the repo uses and the rough architectural shape or dependency flow.

Not a file tree. Just enough that the agent knows what world it’s operating in before it starts reading.

This gives a new session a starting point without making it rediscover basic structure.

Keep this high-level. Detailed descriptions can load irrelevant information and become outdated, forcing the agent to rediscover the current state anyway.

D. General behavior and approach

Broad rules that apply everywhere and aren’t tied to any one task.

• Prefer switch statements over long if/else chains.
• Follow conventions the code does not make obvious.
• Never use British spelling.

These are ambient. They apply to every session, so they belong in the always-loaded file.

Put only rules that apply across sessions here. That keeps prompts shorter and output more consistent.

E. Pitfalls you have actually seen

Not hypothetical ones. The mistakes you watch the agent repeat across multiple sessions.

When you see the same error more than once, consider adding it to AGENTS.md or CLAUDE.md. Curate this from observation; do not generate it up front.

Review this list when you change the model, system prompt, or harness. Agent behaviour can change enough that old rules stop helping.

4. Put task-specific pitfalls in skills

Not every pitfall belongs in AGENTS.md.

If a mistake belongs to one kind of task and you know when it happens, put it in a skill rather than the global file. AGENTS.md and CLAUDE.md load every session, so narrow rules become noise for work that does not need them.

Two real examples.

Reviewing pull-request comments. An agent can read only the top-level comments and miss inline diff comments. That is not a global behaviour problem; it belongs in a skill for addressing pull-request feedback.

Raising pull requests with a particular harness. If a harness repeatedly produces malformed Markdown in pull-request descriptions, capture that as a task-specific rule in a skill.

The distinction is simple:

• Generic and always-on rules go in AGENTS.md or CLAUDE.md.
• Specific, task-triggered rules go in a skill.

Get this wrong and the global file can turn back into the bloated context the benchmark cautions against.

5. The mental model

AGENTS.md and CLAUDE.md are not project documentation. They hold context the code cannot provide and durable guardrails learned through real work. Keep them short, curate them by hand, and put the specifics in skills.

The study did not show that context files are useless. It showed that outcomes varied with their contents. A short, hand-maintained file of non-inferable context is the strongest pattern it reports.

6. A practical check

Before adding a line, ask:

• Is this information unavailable in the repository?
• Does it apply to most sessions?
• Is it still true for this harness and model?

If the answer is no, remove it, move it to a skill, or keep it in ordinary project documentation instead.