You can tell AI review has become noise when the pull request opens and everyone sighs. Forty comments. Three of them might matter. Nobody wants to be the person who sorts them. The author replies “done” to the easy ones and ignores the rest. The human reviewer skims, says looks good, and the risky part ships with a green conversation.
The model is not haunted. The seat is wrong. Noise is what you get when a first-pass review has no scope, no severity, and no owner. It is the most common way an AI code pipeline fails.
What noise looks like on a real PR
A 380-line change adds a webhook handler and a retry table. The risk is duplicate events and a missing idempotency key. The model leaves 36 comments. Twelve are import order and naming. Eight are “consider adding logging.” Six are restating the function in prose. Two are about a file the author did not change, because the tool walked the import graph. One comment mentions retries. It is buried.
The author is not lazy for ignoring that thread. The thread trained them. After two weeks of this, hiding the bot is the rational move. You did not get more review. You got a mute button.
Why models spray
A model is rewarded for being helpful. Helpful, in training, often means “say more.” A human reviewer is rewarded for being brief, because they have four other PRs and a meeting. Those incentive systems do not match.
The tool also rarely knows your house rules. It will fight the linter. It will fight the formatter. It will suggest a pattern you abandoned last quarter. Without a tight scope, it fills the vacuum with generic advice.
Wide context makes this worse. “Here is the repository” feels powerful. It is how you get comments on generated clients and vendored code. Feed the diff. If the model needs a file to understand a call, give it that file. Do not give it a tour.
A noise budget
Decide, in writing, how many automated comments a PR may receive before the bot is considered broken. A useful starting budget for a small team: five comments on a typical change, ten on a large one, and zero on a docs-only PR.
If a run exceeds the budget, that is a product bug. Do not ask authors to “just skim.” Skimming is the habit you are trying to prevent. Tighten the job until the budget holds.
Severity helps the budget. Require every comment to be one of: blocker, should-fix, or nit. Then hide nits by default, or stop emitting them. If the vendor cannot tag severity, treat every comment as a should-fix and keep the count low. A flat list of thirty equal items is how the one blocker disappears.
Give the comments an owner
Noise is not only volume. It is volume with no name attached. If “the team” owns AI comments, they will sit. Put one line in the pull request template: the author resolves or dismisses automated comments before requesting review. The reviewer is allowed to ignore unresolved nits. They are not allowed to ignore an unresolved blocker.
Dismissal should be cheap. A click. A short reason if you want data. If dismissing takes a thread and a justification paragraph, people will leave junk open and train the next reviewer to do the same.
Ownership also means someone tunes the tool. That is not the same person as the author of today’s PR. Pick a rotating owner for the review seat — two weeks is enough — and let them turn rules off. A pipeline with no tuner becomes a museum of old prompts.
When to narrow, pause, or stop
Narrow when comments are real but off-target. Turn off style. Exclude generated paths. Limit the model to the languages you actually ship. Ask it to speak only about the files in the diff. This is the usual fix. It is also the one teams skip because “we will prompt-engineer it later.”
Pause when the team has muted the bot but you still think the seat is right. Take a week off. Merge without the comments. Then turn it back on with a smaller job. A pause is cheaper than a quiet war in the review UI.
Stop when the keep rate stays low after you narrowed. Or when the tool is so slow that comments arrive after merge. Or when nobody will own it. Stopping is not a failure of AI. It is a failure of that seat, on that team, with that tool. You can try again on one PR later.
If you never had a pipeline — only a paste habit — stopping the bot will not hurt delivery. Read what is an AI code pipeline before you install a louder one.
Do not fix noise with a second model
A common panic: the comments are bad, so add another agent to summarize the first agent. Now you have two sources and a summary that smooths away the one useful blocker. Summaries are for logs. Review comments should already be short.
Another panic: move the same spray into CI and make it required. That turns noise into a gate. Authors will spend their afternoon satisfying a model instead of shipping. If you want a CI seat, make it “the pass ran,” not “the pass approved you.” AI in CI for small teams is the longer version.
A cleanup pass for this week
Open the last five pull requests that got automated review. For each comment, mark keep or discard. You are not grading the model for sport. You are measuring the seat.
If you kept fewer than a quarter, cut the job in half: half the files, half the comment types, half the severity levels you emit. Run the next PR. Measure again.
If you kept most of them, the problem may not be noise. The problem may be that nobody merges. That is a habit. A quiet bot will not create it. A loud bot will not either.
When the comments are few and pointed, put a first pass on the next open PR and let a human finish. That is the loop described in add AI to your review pipeline. Noise was the detour. The pipeline is still the point.