Agent Skills' code review skill now drops findings your code already rules out

2 hours ago

Agent Skills' code-review-and-quality skill now checks each finding against the callers, guards and tests that reach the changed code before reporting it, labels bugs already in the base apart from new regressions, and accepts zero findings as a valid result. Merged November 1 in pull request #736; on main, after release 0.6.15. https://github.com/addyosmani/agent-skills/pull/736 https://github.com/addyosmani/agent-skills/blob/main/skills/code-review-and-quality/SKILL.md

Ask about this presentation

Chapters

  1. 0:00Agent Skills' code review skill now drops findings your code already rules out
  2. 0:16Three sentences added to Step 4
  3. 0:29The owner guard disproves the bypass claim
  4. 0:43A new regression and an old defect, labelled apart
  5. 0:57A clean change gets zero findings
  6. 1:07On main, unmeasured on a model
  7. 1:24A review that can say zero
Show transcript
  1. In Addy Osmani's Agent Skills, the code review and quality skill now checks each finding against the callers, guards and tests that reach the changed code, and drops it when they rule it out. So a review stops flagging bugs your code already prevents.

    Slide text
    github.com/addyosmani/agent-skills code-review-and-quality

    Agent Skills’ code review skill now drops findings your code already rules out

    any actor can withdraw checked against the caller · dropped
    agent-skills · PR #736 · merged Nov 1, 2026
  2. In Step 4, before any finding is reported, the reviewer now checks the callers, guards and tests. Bugs already in the base stay apart from new regressions. And zero findings is a valid answer.

    Slide text
    github.com/addyosmani/agent-skills SKILL.md · Step 4: Categorize Findings

    Three sentences now come before any finding is reported

    ### Step 4: Categorize Findings
    
    Before reporting a finding, check it against the callers, guards and tests that reach the changed code, and drop it if they rule it out.
    Label defects that already exist in the base separately from regressions this change introduces.
    Zero findings is a valid result; do not invent one.
    agent-skills · skills/code-review-and-quality/SKILL.md · PR #736
  3. One new test hands the reviewer a claim: the lookup function has no owner check, so anyone can withdraw. But its only caller refuses anyone who isn't the owner. So the claim is rejected.

    Slide text
    github.com/addyosmani/agent-skills eval case 4 · claim C

    The skill’s own test: a bypass claim the caller disproves

    C: lookup has no owner check, so any actor can withdraw.
    
    def withdraw(actor, owner, amount, accounts):
        if actor != owner:
            raise PermissionError("owner required")
        balance = lookup(owner, accounts)
    C rejected: the only caller checks the owner first
    agent-skills · evals/fixtures/review-evidence/after.py · PR #736
  4. In the same test, less than or equal becomes less than, so withdrawing your whole balance now fails. That bug is new. The fee was already wrong before the change, so it is labelled pre-existing.

    Slide text
    github.com/addyosmani/agent-skills eval case 2 · before.py vs after.py

    One bug is new, and one was already there

    new regression 0 < amount <= balance → 0 < amount < balance
    withdrawing the full balance now raises invalid amount
    already in the base fee() returns 0 in both snapshots
    the contract says the basic fee is one unit
    agent-skills · evals/fixtures/review-evidence · PR #736
  5. A third test rewrites the check without changing what it allows. The expected review reports zero new defects and invents no blocker.

    Slide text
    github.com/addyosmani/agent-skills eval case 3 · before.py vs clean.py

    A change that keeps the behaviour gets zero findings

        return 0 < amount <= balance
        return amount > 0 and amount <= balance
    0 new defects no invented blocker
    agent-skills · evals/cases/code-review-and-quality.json · PR #736
  6. It merged November first, the day after release 0.6.15, so it is on main, not yet in a tagged release, and the review command gets it too. No graded model run was made, so the gain is unmeasured.

    Slide text
    github.com/addyosmani/agent-skills scope

    On main since November 1, with no measured gain yet

    where main branch, merged after release 0.6.15
    skill code-review-and-quality, which the review command invokes
    measured no graded behavioral run, no effectiveness claim
    agent-skills · PR #736 · release 0.6.15
  7. False alarms are what agent reviews cost you most. I would take this rule. A review that checks the caller and can report nothing is one you can believe when it does flag something.

    Slide text
    github.com/addyosmani/agent-skills code-review-and-quality

    A review that can say zero is easier to trust

    If agents review your pull requests, take this rule.

    github.com/addyosmani/agent-skills/pull/736
    agent-skills · PR #736 · merged Nov 1, 2026

What happened in AI today, in your inbox

One short email each evening. Read it and you’re caught up.

Built with ❤️ in New York