Skip to content

feat(cli): apply object-lock retention in repository sync-to - #5508

Open
fastlorenzo wants to merge 1 commit into
kopia:masterfrom
fastlorenzo:feature/sync-to-object-locks
Open

feat(cli): apply object-lock retention in repository sync-to#5508
fastlorenzo wants to merge 1 commit into
kopia:masterfrom
fastlorenzo:feature/sync-to-object-locks

Conversation

@fastlorenzo

Copy link
Copy Markdown

Fixes #3759

Problem

As described in #3759, it is currently impossible to maintain an immutable copy of a repository whose primary storage does not support object locking (filesystem, SFTP, ...) by syncing it to object-lock capable storage:

  • repository sync-to writes every blob with empty PutOptions, so nothing on the destination ever receives an object lock.
  • Blobs already in sync are never re-uploaded, so even manually-applied locks would silently expire — and maintenance --extend-object-locks only runs against the connected (local) repository, whose backend rejects retention (blob-retention: unsupported put-blob option).

Solution

Add destination-only retention support to sync-to, reusing the existing locking machinery:

kopia repository sync-to s3 --bucket my-bucket ... --retention-mode COMPLIANCE --retention-period 30d
  • New --retention-mode / --retention-period flags (same names/values as repository create, validated by the existing format.BlobStorageConfiguration.Validate()). When set, the destination storage is wrapped with the same locking wrapper used by repo/open.go, so blobs matching the locking prefixes (packs, indexes, kopia.repository, kopia.blobcfg) are uploaded with retention. The source repository is not modified; no repository-format changes, no new persisted state.
  • Each run also extends object locks on locking-prefixed blobs that are already in sync, so immutability does not lapse between syncs. This costs one retention-update API request per locked blob per run; --no-extend-object-locks skips it for frequent syncs (with a less frequent lock-refreshing run recommended, always with a ≥24h safety margin before lock expiry).
  • Syncing with retention flags to a destination that does not support object locking fails fast with an actionable error instead of silently degrading.
  • wrapLockingStorage is moved from repo/open.go to repo/locking_storage.go and exported (WrapLockingStorage, plus IsLockingStorageBlobID) so the CLI and the open path share one implementation of the prefix rules.
  • The blobtesting fake's ExtendBlobRetention now matches S3 PutObjectRetention semantics (places retention on versions that previously had none and updates the mode), which also covers upgrading an existing unlocked mirror to a locked one.

Testing

  • Unit tests for IsLockingStorageBlobID/WrapLockingStorage (repo/locking_storage_test.go).
  • CLI test (cli/command_repository_sync_retention_test.go) using the in-memory versioned storage and a movable fake clock: verifies retention on locking-prefixed blobs only (including the initial kopia.repository write), lock extension moving expiry forward on re-sync, --no-extend-object-locks and --dry-run not mutating locks, flag validation failures, the unsupported-destination error (both on fresh sync and extension-only runs), and that the synchronized copy is a fully working repository.
  • e2e: sync-to filesystem with retention flags fails with the actionable message.
  • Manually verified against minio with an object-locked bucket: locks applied (X-Amz-Object-Lock-Mode/Retain-Until-Date) on locking blobs only, re-sync advances the retain-until date, deletion of a locked version is rejected as WORM-protected.
  • make lint and existing test suites pass.

Docs

Updated site/content/docs/Advanced/Synchronization/_index.md and site/content/docs/Advanced/Ransomware Protection/_index.md: workflow, extension cost and scheduling guidance, recommended minimal credentials (no governance bypass), and --delete implications for immutable replicas.

🤖 Generated with Claude Code

Allow maintaining an immutable replica of a repository whose primary
storage does not support object locking (e.g. filesystem or SFTP) by
synchronizing it to object-lock capable storage (e.g. S3):

  kopia repository sync-to s3 ... --retention-mode COMPLIANCE --retention-period 30d

- add --retention-mode/--retention-period flags to sync-to and wrap the
  destination storage with the existing locking wrapper so blobs matching
  the locking prefixes are written with retention options
- extend object locks on destination blobs that are already in sync
  (and therefore never re-uploaded) on every run, so protection does not
  lapse between syncs; opt out with --no-extend-object-locks
- fail fast with an actionable error when the destination storage does
  not support object locking
- export IsLockingStorageBlobID and WrapLockingStorage from the repo
  package (moved from repo/open.go) for reuse
- make the blobtesting fake's ExtendBlobRetention match S3
  PutObjectRetention semantics (sets retention even on versions that
  previously had none, and updates the mode)
- document the workflow in the Synchronization and Ransomware
  Protection docs

Fixes kopia#3759

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fastlorenzo
fastlorenzo requested a review from a team July 19, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sync-to command and object locks

1 participant