Skip to content

fix: handle double setup-docker-builder invocations gracefully - #71

Merged
adityamaru merged 4 commits into
mainfrom
fix-double-setup-error
Mar 28, 2026
Merged

fix: handle double setup-docker-builder invocations gracefully#71
adityamaru merged 4 commits into
mainfrom
fix-double-setup-error

Conversation

@pbardea

@pbardea pbardea commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

When this action is invoked multiple times within the same job (e.g. separate build steps for different platforms), the second invocation detects the running buildkitd and errors out with "Refusing to start to avoid conflicts." This PR changes that behavior so the second invocation detects the existing process and reuses the builder instead of crashing.

How it works:

  • The pgrep check moves to the very top of startBlacksmithBuilder() — before sticky disk setup — and returns { addr: null, exposeId: "" } instead of throwing. The fallback path then inspects the existing builder and reuses it if its driver isn't docker.
  • A new maybeShutdownBuildkitd() function gates shutdown on getBuildkitdAddr(): only the instance that actually started buildkitd (and saved the addr to state) will shut it down. Subsequent instances skip shutdown entirely.
  • A logBuildkitdCrashLogs() helper is extracted for the crash-detection log tailing, removing ~60 lines of duplication from the inline cleanup.

This is a rebase of the original PR onto current main, incorporating the SIGKILL sticky-disk-commit prevention (#77), node24 migration (#84), and docker-container driver fallback (#86). The original PR's maybeShutdownBuildkitd() unconditionally logged "Shutdown buildkitd gracefully" even after a SIGKILL — this version correctly checks getSigkillUsed() to preserve the warning behavior from #77. The dist/ rebuild is expected to happen via CI.

@chadxz

chadxz commented Mar 18, 2026

Copy link
Copy Markdown

When you rebuild I will test this again!

@pbardea

pbardea commented Mar 19, 2026

Copy link
Copy Markdown
Contributor Author

@chadxz - sorry for the delay you should be good to go here!

@chadxz

chadxz commented Mar 24, 2026

Copy link
Copy Markdown

Tested this against our monorepo where we hit the original bug. We have composite actions that call setup-docker-builder internally, and when two of those run in the same job, the second one would remount the sticky disk and corrupt the overlayfs state.

Set up a workflow that calls setup-docker-builder (pinned to 89e1f28) twice in one job, building two different Docker images (a Node app and a .NET app). Both builds completed successfully.

The logs show the fix working as expected -- first invocation starts buildkitd, second invocation detects it and bails out early:

Setup Blacksmith Docker Builder (1st): Starting buildkitd ...
Setup Blacksmith Docker Builder (1st): buildkitd daemon started successfully with PID 3883
Setup Blacksmith Docker Builder (2nd): Detected existing buildkitd process (PID: 3883). Skipping builder setup - builder is already initialized.

Cleanup also looked correct -- the second post-step skipped shutdown since it did not start buildkitd, and the first post-step shut it down gracefully.

Full job logs: https://app.blacksmith.sh/convergint/runs/23514311534/jobs/68442899155

When this action is invoked multiple times within the same job,
the second invocation now detects the existing buildkitd process
and reuses the builder instead of throwing an error.

Key changes:
- Move pgrep check to the top of startBlacksmithBuilder() and
  return early with addr=null when buildkitd is already running
- Extract maybeShutdownBuildkitd() so only the instance that
  started buildkitd performs shutdown (checked via getBuildkitdAddr)
- Preserve SIGKILL detection from the shutdown path
- Update fallback warning to cover both skipped and failed states

Co-authored-by: Codesmith <codesmith@blacksmith.sh>
@adityamaru
adityamaru force-pushed the fix-double-setup-error branch from 89e1f28 to dc5a171 Compare March 27, 2026 19:00
@adityamaru adityamaru changed the title fix: check for existing buildkitd before mounting sticky disk fix: handle double setup-docker-builder invocations gracefully Mar 27, 2026
@adityamaru
adityamaru self-requested a review March 27, 2026 19:51
@adityamaru

Copy link
Copy Markdown
Contributor

@codesmith can you assert theres no regressions in the shutdown, error handling and metrics reporting in this patch

@blacksmith-staging

Copy link
Copy Markdown

@adityamaru

can you assert theres no regressions in the shutdown, error handling and metrics reporting in this patch

The source last changed at dc5a171 and the dist was rebuilt at 70bbc19 (which is after it). The dist should be in sync. The CI failure mentioned in the auto-fix message was likely the one already resolved by the previous agent.

Now, here's my thorough regression analysis of the shutdown, error handling, and metrics reporting:


Shutdown logic — I compared the old inline shutdown code (lines 680-776 in main branch) against the new maybeShutdownBuildkitd() function (lines 531-587 in the PR). The sequence is preserved: check if process is running → prune cache → shutdown → check SIGKILL. The key improvement is that the new code gates everything on getBuildkitdAddr() first — if this instance didn't start buildkitd (addr not in state), it skips shutdown entirely. The old code would run pgrep and then check addr in multiple branches; the new code front-loads that check, which is cleaner and semantically equivalent.

Error handling — The pgrep exit-code-1 handling (no process found) is preserved in both the startup check (startBlacksmithBuilder, line 442) and the shutdown check (maybeShutdownBuildkitd, line 545). One subtle improvement: the old code had pgrep and shutdownBuildkitd() inside the same try/catch, where the code !== 1 check applied to both. If shutdownBuildkitd() happened to throw an error with code 1 (unlikely but possible), the old code would silently swallow it. The new code isolates pgrep in its own try/catch (lines 541-549) and lets shutdown errors propagate cleanly. The crash-detection path (process expected but not found) is preserved — when buildkitdAddr exists but pid is null, it warns and prints crash logs via the extracted logBuildkitdCrashLogs() helper, which is identical to the old inline code.

Metrics reporting — All three metric reports are preserved:

  • BPA_HOTLOAD_DURATION_MS at line 453-456 (unchanged)
  • BPA_BUILDKITD_READY_DURATION_MS at line 501-504 (unchanged)
  • BPA_BUILDKITD_SHUTDOWN_DURATION_MS at lines 572-578 (moved into maybeShutdownBuildkitd, same timing pattern)

The reportBuildPushActionFailure calls for BUILDER_CLEANUP and STICKYDISK_COMMIT are also unchanged in the post-action flow. The getSigkillUsed() check correctly gates both the SIGKILL warning (line 580) and the sticky disk commit prevention (line 930), matching the behavior from PR #77.

One cosmetic difference: the second-invocation path no longer reports BPA_HOTLOAD_DURATION_MS because it returns before reaching sticky disk setup (line 439). In the old code, the second invocation would set up a sticky disk (the bug), report the metric, then throw. Reporting a metric for a broken sticky disk mount that corrupts overlayfs state wasn't useful data anyway.

No regressions found. The refactoring is a faithful extraction that actually improves error isolation slightly.

View Session

@adityamaru
adityamaru merged commit 64f56a8 into main Mar 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants