ECC's verification-loop skill stops reporting a failed build as a pass

1 month ago

ECC's verification-loop skill piped the build into tail without pipefail, so a failed build exited 0 and could pass the pre-PR check. Pull request #3455 sets pipefail in the build, lint and test steps and swaps head for sed. The Claude Code plugin and Pi copies are fixed; the .agents (Codex, Copilot CLI) and .kiro copies are not, and the fix is not yet in a tagged release (latest v2.2.3, Oct 1). https://github.com/affaan-m/ECC/pull/3455 https://github.com/affaan-m/ECC/blob/main/skills/verification-loop/SKILL.md

Ask about this presentation

Chapters

  1. 0:00ECC's verification-loop skill stops reporting a failed build as a pass
  2. 0:16verification-loop ends in a READY or NOT READY verdict
  3. 0:31A bash pipeline returns the exit code of its last command
  4. 0:46pipefail was on only in the type check, after the build had run
  5. 0:57pipefail now opens the build, lint and test steps
  6. 1:11Type check and lint output now goes through sed, so passing runs still pass
  7. 1:26The Claude Code plugin and the Pi copy are fixed. Other copies are not
  8. 1:42ECC's verification-loop skill stops reporting a failed build as a pass
Show transcript
  1. The ECC plugin's verification-loop skill now reports a failed build as failed. Its build step piped the build into tail, so a broken build came back as a pass. If your agent ran it before a pull request, that pass may have been wrong.

    Slide text
    github.com/affaan-m/ECC merged Oct 11, 2026
    Build: PASS
    build failed · exit 1
    github.com/affaan-m/ECC/pull/3455 · skills/verification-loop/SKILL.md
  2. verification-loop is ECC's check before a pull request. It runs the build, a type check, lint, tests and a secrets search, then writes a report that says ready or not ready. The build runs first.

    Slide text
    github.com/affaan-m/ECC verification-loop · PR #3455
    VERIFICATION REPORT
    Build: [PASS/FAIL]
    Types: [PASS/FAIL] (X errors)
    Lint: [PASS/FAIL] (X warnings)
    Tests: [PASS/FAIL] (X/Y passed, Z% coverage)
    Security: [PASS/FAIL] (X issues)
    Overall: [READY/NOT READY] for PR
    skills/verification-loop/SKILL.md · Output Format
  3. In bash, a pipeline exits with the status of its last command. The build step sent npm run build into tail, so the pipeline reported tail's result, and a failed build exited zero, the code for success.

    Slide text
    github.com/affaan-m/ECC verification-loop · PR #3455
    # Phase 1, before the fix
    npm run build 2>&1 | tail -20
    $ bash -c 'false | tail -1; echo $?'
    0
    github.com/affaan-m/ECC/pull/3455 · pull request description
  4. pipefail is the bash setting that fails a pipeline when any command in it fails. The skill switched it on only in the type check, the second phase, after the build had already run.

    Slide text
    github.com/affaan-m/ECC verification-loop · PR #3455
    1 Build no pipefail
    2 Types set -o pipefail
    3 Lint no pipefail
    4 Tests no pipefail
    github.com/affaan-m/ECC/pull/3455 · commit 9c21aaa
  5. The fix turns pipefail on at the top of the build, lint and test steps, the three that pipe a check into head or tail. The same failed command now exits one, and the report says the build failed.

    Slide text
    github.com/affaan-m/ECC verification-loop · PR #3455
    + set -o pipefail # report the build's exit status, not tail's
    npm run build 2>&1 | tail -20
    $ bash -c 'set -o pipefail; false | tail -1; echo $?'
    1
    github.com/affaan-m/ECC/pull/3455 · commit 9c21aaa
  6. With pipefail on, head stops reading after thirty lines, and a clean linter still writing output can fail on a broken pipe. So the merged change swaps head for sed, which shows thirty lines but reads everything.

    Slide text
    github.com/affaan-m/ECC verification-loop · PR #3455
    npm run lint 2>&1 | head -30 npm run lint 2>&1 | sed -n '1,30p'
    github.com/affaan-m/ECC/pull/3455 · merge commit a003286
  7. The fix covers the skills folder the Claude Code plugin installs, and the copy for Pi. The copy under dot agents, which Codex and Copilot CLI load, still has the old build line, and so do the dot kiro copy and six translations.

    Slide text
    github.com/affaan-m/ECC verification-loop · PR #3455
    skills/verification-loop fixed
    pi/core/skills/verification-loop fixed
    .agents/skills · Codex, Copilot CLI old build line
    .kiro/skills old build line
    six translated copies old build line
    github.com/affaan-m/ECC/pull/3455 · .agents/ and .kiro/ SKILL.md on main · README
  8. The fix is on main. ECC's latest tagged release, from October 1, came out before it. ECC's verification-loop skill stops reporting a failed build as a pass. It's pull request 3455 in the ECC repository.

    Slide text
    github.com/affaan-m/ECC verification-loop · PR #3455
    Build: PASS build failed · exit 1

    Merged to main. The latest tagged release, from Oct 1, predates the fix.

    github.com/affaan-m/ECC/pull/3455
    github.com/affaan-m/ECC/releases

What happened in AI today, in your inbox

One short email each evening. Read it and you’re caught up.

Built with ❤️ in New York