gstack now runs its eval cases on Codex every Monday, beside Claude Opus 5.5

1 hour ago

gstack 1.91.147.0 to 1.91.149.0: gstack's paid eval cases now run on Codex, and the Monday census runs Claude Opus 5.5 and Codex gpt-6.1-sol side by side. On its first pilot Codex passed 48 of 52 gate cases at about a sixth of the cost; Claude-only cases are skipped, never failed. Release notes 1.91.149.0: https://github.com/garrytan/gstack/blob/main/CHANGELOG.md#1911490---2026-10-19 Release notes 1.91.148.0: https://github.com/garrytan/gstack/blob/main/CHANGELOG.md#1911480---2026-10-19 Release notes 1.91.147.0: https://github.com/garrytan/gstack/blob/main/CHANGELOG.md#1911470---2026-10-19 Commits: https://github.com/garrytan/gstack/commits/main

Ask

Ask about this presentation

Answers are generated from this presentation.

Chapters

  1. 0:00Codex
  2. 0:14Pilot
  3. 0:24Claude-only
  4. 0:37Monthly
  5. 0:50Try it
Show transcript

Codex

gstack logo

gstack now runs its eval cases on Codex every Monday, beside Claude

claude-opus
Claude Opus 5.5
codex-sol
Codex · gpt-6.1-sol
CodexPilotClaude-onlyMonthlyTry it
garrytan/gstack · CHANGELOG.md · 1.91.149.0

gstack's paid eval cases can now run on Codex, each one through a full Codex runtime. Every Monday the census runs two arms side by side: Claude Opus 5.5, and Codex on gpt-6.1-sol.

Pilot

On its first run, Codex passed 48 of 52 gate cases at a sixth of the cost

48 / 52
gate cases passed on gpt-6.1-sol, no retries
$4.63
for the 52 cases, against $28.93 on Claude Fable
CodexPilotClaude-onlyMonthlyTry it
garrytan/gstack · CHANGELOG.md · 1.91.149.0

On its first pilot, Codex cleared 48 of the 52 gate cases with no retries. The whole run cost about a sixth of the same cases on Claude Fable.

Claude-only

Claude-only cases skip on Codex and never count as failed

52 adapter-only gate cases
run on Codex
26 rework cases
stay Claude-only
thresholds and Claude verdicts
unchanged
CodexPilotClaude-onlyMonthlyTry it
garrytan/gstack · CHANGELOG.md · 1.91.149.0

Cases that need Claude are skipped on Codex with a recorded reason, never marked failed. The 26 rework cases stay Claude-only, and no threshold or Claude verdict changed.

Monthly

Once a month the census adds Claude Fable and gpt-6-astra

Monthly arms: claude-fable and codex-astra, on the 1st
Real cost: Codex runs are priced from their token usage
Host on record: plans, reports and the ledger name a non-Claude host
CodexPilotClaude-onlyMonthlyTry it
garrytan/gstack · CHANGELOG.md · 1.91.149.0

Once a month, the census adds Claude Fable and Codex on gpt-6-astra. Codex runs now report a real dollar cost, and reports name the host whenever it is not Claude.

Try it

Running one case on Codex yourself: the release notes, “If you…”

CHANGELOG.mdgstack 1.91.149.0 release notes
Which agent do you run gstack in?
CodexPilotClaude-onlyMonthlyTry it
garrytan/gstack · CHANGELOG.md · 1.91.149.0

To run a single case on Codex yourself, follow the If you section of gstack's release notes. Which agent do you run gstack in?