Make Dependabot triage cheaper and more decisive by williammartin · Pull Request #14079 · cli/cli

williammartin · 2026-08-05T14:19:51Z

N/A. No related issue.

Description

Dependabot PR Triage runs hourly, posts one advisory comment per open Dependabot PR saying whether it looks safe to merge and how confident it is, and never merges or approves anything. Two things were wrong with it.

Most runs cost money and produced nothing. Usually nothing has changed, but the model still started and spent 8 LLM calls establishing that, mostly re-reading every triage comment it had ever posted. That work is mechanical, and its cost grew as the comments accumulated.

The comments were vague, and occasionally wrong. Nearly everything came out Medium, because High required reading upstream "end to end" while another rule said to stop early and cap at Medium. Both could not hold at once. The skill also never told the agent to read the PR's diff or the source tree, so it reasoned from the title: it called a direct go.mod requirement indirect, and cited a workflow file that does not exist here.

What changes:

A shell gate runs before the model. It lists open Dependabot PRs, drops any whose CI is still running or whose head SHA matches the one recorded in the last triage comment, and writes the rest to a work list. If nothing is left it emits a noop, exiting before the engine starts at zero AI Credits. The agent is handed that list and told not to re-derive it, so scope is deterministic rather than something the model searches for.
The skill requires evidence before deciding: the PR's diff, direct or indirect per the manifest, which of the dependency's packages this repository imports, and the upstream release. High now means the evidence the conclusion rests on was observed, not that all upstream history was read.
A coherence check for a gh-aw failure we hit repeatedly: a bump that rewrites uses: lines but leaves the generated manifest pinned to the old SHA, which silently reverts on the next recompile.

How did you test this change?

The gate is a shell script, so I extracted it from the compiled .lock.yml and ran it against live cli/cli data. All seven open Dependabot PRs are already assessed at their current head, so it should find no work, and does:

PRs with terminal CI: 7
PR #14068: already assessed at 60d0bf988110ab59e792914bd952564549974b10, skipping
... (all 7)
Work list: 0 PR(s)
{"type":"noop","message":"No Dependabot PRs need triage: ..."}

Also covered: a negative control with the marker made unfindable (every PR correctly comes back as needing assessment), the three pr_number dispatch paths, and fixtures for the CI branch that cannot be triggered on demand.

Key points

None

Notes for reviewers

Please look hardest at integrity-proxy: false in .github/workflows/shared/dependabot-triage-security.md. It is the only security-relevant call here.

Configuring an integrity policy makes the compiler wrap custom pre-agent steps in a DIFC proxy, so their gh calls are filtered like the agent's. That proxy applies min-integrity but not trusted-users, which resolve later at runtime. The gate finds its marker by reading back its own cli-triage[bot] comments, and the app posts with author_association: NONE, which is exactly what min-integrity: approved filters out and what trusted-users normally exempts. Behind the proxy the marker is never found, every PR looks unassessed, and the workflow re-comments on all of them hourly, which is the behaviour this PR exists to stop.

Turning it off does not widen the injection surface. The gate hands nothing it reads to the model, only PR numbers, head SHAs and CI states, and it matches the marker solely within comments already narrowed to the app's own login. That login check, not integrity filtering, is what stops a third party forging a marker. The agent is untouched and still runs under the full policy via the MCP gateway.

Two smaller calls, both explained in comments at the line: noop no longer reports to the shared "no-op runs" issue, since it is now the ordinary idle outcome of an hourly workflow and would add roughly 24 comments a day there forever; and the gate warns rather than paginates when the PR listing hits its 100 cap.

Authorship and follow-up

Who wrote this:

A human wrote it.
An agent wrote it under close human direction.
An agent wrote it independently, and no human has guided the implementation beyond the initial prompt.

Who answers review comments:

@williammartin will read and reply directly. Name the account.
An agent will draft replies and @username will read them before they are posted.
Nobody has explicitly committed to replying.

The triage workflow spent an LLM turn deciding it had nothing to do. A no-op scheduled run cost 50-75 AI Credits because the agent walked the dedup protocol itself: fetching open PRs, reading every prior triage comment, and comparing head SHAs. That work is entirely deterministic, and the most expensive single call was the agent re-ingesting its own past comments, so the cost grew every time the workflow commented. Move that comparison into a shell step that runs after checkout but before the engine starts. It writes a work list to /tmp/gh-aw/dependabot-worklist.json, and when the list is empty it emits a `noop` safe output, which makes the harness exit before any inference is billed. This also hardens scope. The agent no longer decides which PRs are in range, so it cannot be talked into assessing a PR outside the work list by content in a PR it is reading. `issues: read` is needed because PR conversation comments are served by the issues API, and the timeout moves to 30 minutes because the runs that do have work now do strictly more evidence gathering per PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 190d246d-0b1f-4ce6-9aa6-dee10d3d4cf8

The triager almost never returned High confidence, and when its prose disagreed with a human reviewer it was usually because it had guessed at something it could have read. It named workflow files that do not exist in this repository, and it called a direct `go.mod` requirement indirect. Both mistakes share a cause: the skill never told the agent to look at the PR's own diff or at the checked-out source tree. It had access to both the whole time. So replace inference with five required evidence items - the diff, the dependency's position in the manifest, the repository's actual import surface, CI state, and upstream release evidence - and make High confidence conditional on having gathered them. The old definition of High was unreachable by construction. It asked for the upstream change to be read "end to end" while a separate instruction capped confidence at Medium rather than reading indefinitely, so any non-trivial bump fell through to Medium no matter how clear it was. Redefine High as decision-relevant completeness: a four-release bump that touches nothing this repository imports is High once you have verified that, because reading the rest could not change the answer. Also drop the dedup protocol, which the pre-flight step now performs deterministically, and add an in-repo coherence check for bumps that edit generated files without updating the version each file records - the gh-aw lock files being the case that prompted it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 190d246d-0b1f-4ce6-9aa6-dee10d3d4cf8

Copilot

Pull request overview

This pull request reduces the recurring cost of the scheduled “Dependabot PR Triage” agentic workflow by moving purely mechanical scoping/dedup/CI-gating work into a deterministic pre-flight shell step, and tightens the triager skill so recommendations and confidence are based on explicitly gathered evidence (diff, manifest position, usage trace, CI, upstream release), with an added coherence check for generated gh-aw outputs.

Changes:

Added a pre-flight gate step to compute /tmp/gh-aw/dependabot-worklist.json and emit a noop safe output when there’s nothing new to assess.
Updated the dependabot-triager skill to treat the work list as authoritative scope and to require specific evidence before deciding recommendation/confidence.
Regenerated the compiled workflow lockfile to reflect the new step, permissions, and timeout.

Show a summary per file

File	Description
.github/workflows/dependabot-triage.md	Adds deterministic work-list computation and updates the agent prompt to consume it.
.github/workflows/dependabot-triage.lock.yml	Regenerated compiled workflow reflecting the new gate step and updated permissions/timeout.
.github/skills/dependabot-triager/SKILL.md	Reworks scope/dedup/CI assumptions and defines required evidence + revised confidence rubric.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 3/3 changed files
Comments generated: 2
Review effort level: Lite

The evidence rule told the agent that a dependency in the first `require` block is direct. That is a `go mod tidy` formatting convention, not the semantics. What actually marks a requirement indirect is the trailing `// indirect` comment on its own line, and Go's parser reads it that way regardless of block: put a commented and an uncommented require in the same block and `go mod edit -json` still reports Indirect true and false respectively. The two agree in this repository today, so nothing was misclassified. But the rule would break on a reorganised or hand-edited file, and misreporting a direct dependency as indirect is precisely the error the required-evidence section exists to prevent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 190d246d-0b1f-4ce6-9aa6-dee10d3d4cf8

Adding a custom pre-agent `steps:` block made the compiler wrap it in a DIFC proxy, because a guard policy is configured. That proxy applies `min-integrity` but not `trusted-users`, which are resolved at runtime after it starts. The pre-flight finds its dedup marker by reading back its own `cli-triage[bot]` comments, and those are precisely what `min-integrity: approved` filters out - the app posts with author_association NONE, which is why `trusted-users` exists here at all. So the marker was never found, every open Dependabot PR looked unassessed on every run, and the workflow would have re-triaged and re-commented on all of them hourly: the exact failure this design was written to prevent, moved from the agent to a place with no model to notice it. Turning the proxy off does not widen the injection surface. The pre-flight hands nothing it reads to the model - it extracts PR numbers, head SHAs and CI states - and it matches the marker only inside comments already narrowed to the app's own login. That login check, not integrity, is what stops a third party forging a marker. The agent still runs under the full policy via the MCP gateway. Verified against cli/cli: all seven open Dependabot PRs are correctly recognised as already assessed at their current head, and a run with the login filter pointed at a non-existent bot correctly reports them as needing assessment. Three smaller corrections ride along, all fallout from the same review: - Silence no-op issue reporting. gh-aw posts a comment to a shared "no-op runs" issue on every noop, and noop is now the routine idle outcome of an hourly reconciler, so that would have been roughly 24 comments a day forever. The run log already records why a run did nothing. - Drop CI state from the required-evidence count. The pre-flight now guarantees terminal CI, so the agent never gathers it and it could never be the missing item that caps confidence. Counting it made the gate for High confidence four items dressed up as five. - Log which PRs the terminal-CI gate excluded. A check that never reports would otherwise keep a PR out of triage permanently and silently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 18bc01f9-9498-4bd8-9fd0-70308491b695

Copilot

Review details

Files reviewed: 4/4 changed files
Comments generated: 2
Review effort level: Lite

`statusCheckRollup` contexts are CheckRun and StatusContext objects, which sit behind the `checks` and `statuses` scopes. The workflow token had neither, so the field would have come back unreadable at runtime even though it reads fine with a developer token, which is what I tested with. The dangerous part was not the missing permission but how the gate reacted to it. `[.statusCheckRollup[]? | select(pending)] | length == 0` cannot tell "this PR has no checks" from "I could not read this PR's checks", so an unreadable rollup counted as terminal CI and the PR would have been assessed while its CI was still running. Silently wrong beats loudly broken only from the outside. So the classification now treats a null rollup as pending and names it in the skip log, and the permissions are granted. The gate fails safe if either is ever dropped again. Also warn when the listing hits the 100-PR cap. gh truncates silently and the ordering is stable, so PRs past the cap would never be reached on a later run either. Paginating for a case that far outside anything this repository has seen, and well above the 20-comment safe-output cap, is not worth the extra requests, but the condition should not be invisible. Reported by Copilot review on #14079. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 18bc01f9-9498-4bd8-9fd0-70308491b695

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.97.0` → `v2.98.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.98.0`](https://github.com/cli/cli/releases/tag/v2.98.0): GitHub CLI 2.98.0 [Compare Source](cli/cli@v2.97.0...v2.98.0) #### Security A security vulnerability has been identified, and fixed, that binds the local forwarded port to all available network interfaces by default. Users of `gh codespace ports forward` are advised to update `gh` to version `v2.98.0` as soon as possible. For more information see: <GHSA-vfhh-p7hm-pxfh> #### Support worktrees in `pr checkout` Users can now checkout a pull request into a git worktree by using the new `--worktree PATH` flag in `gh pr checkout`: ```shell gh pr checkout 12 --worktree ../wt-feature ``` #### Add semantic search to `search issues` The `gh search issues` command now supports semantic search for issues. Users can select the search type by passing the `--search-type` flag: ```shell gh search issues --search-type semantic ... gh search issues --search-type hybrid ... ``` For more information about semantic search see: ["Improved Search for github issues is now generally available"](https://github.blog/changelog/2026-04-02-improved-search-for-github-issues-is-now-generally-available/). #### What's Changed ##### ✨ Features - Add --worktree flag to gh pr checkout by [@tidy-dev](https://github.com/tidy-dev) in [#13946](cli/cli#13946) - Set GH\_EXTENSION=1 when gh invokes an extension by [@williammartin](https://github.com/williammartin) in [#14072](cli/cli#14072) - Add --search-type flag for semantic and hybrid issue search by [@michaeljacholke](https://github.com/michaeljacholke) in [#14006](cli/cli#14006) ##### 🐛 Fixes - Fix `RESTWithNext` error type, repairing `gh status` and attestation retries by [@williammartin](https://github.com/williammartin) in [#13988](cli/cli#13988) - Trim spaces when parsing X-Oauth-Scopes in `gh release create` by [@williammartin](https://github.com/williammartin) in [#14065](cli/cli#14065) - Fix project item-add output for non-TTY by [@zwick](https://github.com/zwick) in [#14056](cli/cli#14056) ##### 📚 Docs & Chores - Slim down dependabot triage comments by [@williammartin](https://github.com/williammartin) in [#14019](cli/cli#14019) - Require explicit MR review ownership by [@williammartin](https://github.com/williammartin) in [#14028](cli/cli#14028) - Collapse spam triage into the agentic issue-triage workflow by [@williammartin](https://github.com/williammartin) in [#14027](cli/cli#14027) - Run Dependabot triage every hour by [@sergiou87](https://github.com/sergiou87) in [#14030](cli/cli#14030) - Route deploy key requests through api.Client by [@williammartin](https://github.com/williammartin) in [#13989](cli/cli#13989) - Route ssh key requests through api.Client by [@williammartin](https://github.com/williammartin) in [#13994](cli/cli#13994) - Route gpg key requests through api.Client by [@williammartin](https://github.com/williammartin) in [#13997](cli/cli#13997) - Route autolink requests through api.Client by [@williammartin](https://github.com/williammartin) in [#14013](cli/cli#14013) - Route extension requests through api.Client by [@williammartin](https://github.com/williammartin) in [#14059](cli/cli#14059) - Route release creation through api.Client by [@williammartin](https://github.com/williammartin) in [#14062](cli/cli#14062) - Tell agents to use the MR template in AGENTS.md by [@williammartin](https://github.com/williammartin) in [#14074](cli/cli#14074) - Make Dependabot triage cheaper and more decisive by [@williammartin](https://github.com/williammartin) in [#14079](cli/cli#14079) - Route release deletions through api.Client by [@williammartin](https://github.com/williammartin) in [#14077](cli/cli#14077) - Give Dependabot triage a real reachability check by [@williammartin](https://github.com/williammartin) in [#14087](cli/cli#14087) - Restore automatic spam issue closure by [@williammartin](https://github.com/williammartin) in [#14088](cli/cli#14088) - Add a scheduled tech debt burndown skill by [@williammartin](https://github.com/williammartin) in [#14095](cli/cli#14095) - Use reflect.Pointer instead of deprecated reflect.Ptr by [@williammartin](https://github.com/williammartin) in [#14098](cli/cli#14098) - Clarify what belongs in the MR template's testing section by [@williammartin](https://github.com/williammartin) in [#14103](cli/cli#14103) - Rename cli-code-reviewer skill to code-review by [@BagToad](https://github.com/BagToad) in [#14116](cli/cli#14116) - Add aw-actions group to dependabot configuration by [@babakks](https://github.com/babakks) in [#14123](cli/cli#14123) - Isolate tests from local machine's auth and git configuration by [@BagToad](https://github.com/BagToad) in [#14128](cli/cli#14128) - Don't ask for feature detection cleanup comments when not needed by [@babakks](https://github.com/babakks) in [#14139](cli/cli#14139) - Accept pre-release tags in deployment validation by [@BagToad](https://github.com/BagToad) in [#14193](cli/cli#14193) - ci: add temporary step to verify Linux repo signing keys by [@babakks](https://github.com/babakks) in [#14202](cli/cli#14202) - Revert "ci: add temporary step to verify Linux repo signing keys" by [@babakks](https://github.com/babakks) in [#14203](cli/cli#14203) - Fix issue triage action compatibility \[skip changelog] by [@tidy-dev](https://github.com/tidy-dev) in [#14207](cli/cli#14207) #####

Dependencies - chore(deps): bump github.com/sigstore/sigstore-go from 1.2.2 to 1.3.0 by [@dependabot](https://github.com/dependabot)\[bot] in [#14047](cli/cli#14047) - chore(deps): bump the codeql-actions group across 1 directory with 3 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#14049](cli/cli#14049) - chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.0 by [@dependabot](https://github.com/dependabot)\[bot] in [#14048](cli/cli#14048) - chore(deps): bump github.com/google/go-containerregistry from 0.21.7 to 0.21.8 by [@dependabot](https://github.com/dependabot)\[bot] in [#14066](cli/cli#14066) - chore(deps): bump actions/attest from 4.2.1 to 4.2.2 by [@dependabot](https://github.com/dependabot)\[bot] in [#14100](cli/cli#14100) - chore(deps): bump azure/login from 3.0.0 to 3.0.1 by [@dependabot](https://github.com/dependabot)\[bot] in [#14101](cli/cli#14101) - chore(deps): bump the codeql-actions group across 1 directory with 3 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#14091](cli/cli#14091) - chore(deps): bump github/gh-aw-actions/setup-cli from 0.83.4 to 0.85.4 by [@dependabot](https://github.com/dependabot)\[bot] in [#14068](cli/cli#14068) - chore(deps): bump github.com/google/go-containerregistry from 0.21.8 to 0.21.9 by [@dependabot](https://github.com/dependabot)\[bot] in [#14119](cli/cli#14119) - chore(deps): bump github.com/klauspost/compress from 1.19.1 to 1.19.2 by [@dependabot](https://github.com/dependabot)\[bot] in [#14120](cli/cli#14120) - chore(deps): bump the aw-actions group with 2 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#14147](cli/cli#14147) - chore: sign APT repository with both keys by [@babakks](https://github.com/babakks) in [#13271](cli/cli#13271) - chore(deps): bump github.com/yuin/goldmark from 1.8.4 to 1.8.5 by [@dependabot](https://github.com/dependabot)\[bot] in [#14029](cli/cli#14029) - chore(deps): bump actions/attest from 4.2.0 to 4.2.1 by [@dependabot](https://github.com/dependabot)\[bot] in [#14050](cli/cli#14050) - Bump golangci-lint in CI to v2.12.2 by [@williammartin](https://github.com/williammartin) in [#14102](cli/cli#14102) - chore(deps): bump the aw-actions group with 2 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#14124](cli/cli#14124) - chore(deps): bump google.golang.org/protobuf from 1.36.11 to 1.36.12 by [@dependabot](https://github.com/dependabot)\[bot] in [#14140](cli/cli#14140) - Upgrade gh-aw workflows to v0.85.4 by [@tidy-dev](https://github.com/tidy-dev) in [#14141](cli/cli#14141) - Bump Go to 1.26.6 by [@github-actions](https://github.com/github-actions)\[bot] in [#14143](cli/cli#14143) - chore: bump go to 1.26.7 by [@babakks](https://github.com/babakks) in [#14205](cli/cli#14205) - chore(deps): bump github.com/stretchr/testify from 1.11.1 to 1.12.1 by [@dependabot](https://github.com/dependabot)\[bot] in [#14204](cli/cli#14204) - chore(deps): bump the codeql-actions group across 1 directory with 3 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#14169](cli/cli#14169) - chore(deps): bump golang.org/x/crypto from 0.54.0 to 0.55.0 by [@dependabot](https://github.com/dependabot)\[bot] in [#14164](cli/cli#14164) - chore(deps): bump charm.land/lipgloss/v2 from 2.0.5 to 2.0.6 by [@dependabot](https://github.com/dependabot)\[bot] in [#14166](cli/cli#14166) - Bump gh-aw-actions to v0.87.1 and recompile agentic workflows by [@BagToad](https://github.com/BagToad) in [#14210](cli/cli#14210) #### New Contributors - [@sergiou87](https://github.com/sergiou87) made their first contribution in [#14030](cli/cli#14030) - [@michaeljacholke](https://github.com/michaeljacholke) made their first contribution in [#14006](cli/cli#14006) **Full Changelog**: <cli/cli@v2.97.0...v2.98.0> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).

williammartin and others added 2 commits August 5, 2026 16:05

Copilot AI lite review requested due to automatic review settings August 5, 2026 14:19

Copilot started reviewing on behalf of williammartin August 5, 2026 14:20 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

Comment thread .github/workflows/dependabot-triage.md

Comment thread .github/skills/dependabot-triager/SKILL.md Outdated

williammartin and others added 2 commits August 5, 2026 16:30

williammartin requested a lite review from Copilot August 5, 2026 14:58

Copilot started reviewing on behalf of williammartin August 5, 2026 14:59 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

Comment thread .github/workflows/dependabot-triage.md

Comment thread .github/workflows/dependabot-triage.lock.yml

williammartin marked this pull request as ready for review August 5, 2026 15:23

williammartin requested a review from a team as a code owner August 5, 2026 15:23

williammartin requested a review from sergiou87 August 5, 2026 15:23

williammartin merged commit ae253a8 into trunk Aug 5, 2026
20 checks passed

williammartin deleted the williammartin-congenial-bassoon branch August 5, 2026 15:23

williammartin mentioned this pull request Aug 6, 2026

Give Dependabot triage a real reachability check #14087

Merged

6 tasks

BrewTestBot mentioned this pull request Aug 20, 2026

gh 2.98.0 Homebrew/homebrew-core#299922

Merged

scottames-github-bot Bot mentioned this pull request Aug 26, 2026

chore(deps): update ⬆️ mise-packages scottames/dots#1029

Merged

1 task

Navigation Menu

Uh oh!

Make Dependabot triage cheaper and more decisive - #14079

Make Dependabot triage cheaper and more decisive#14079
williammartin merged 5 commits into
trunkfrom
williammartin-congenial-bassoon

williammartin commented Aug 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

williammartin commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How did you test this change?

Key points

Notes for reviewers

Authorship and follow-up

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

williammartin commented Aug 5, 2026 •

edited

Loading