feat(cli): apply object-lock retention in repository sync-to - #5508
Open
fastlorenzo wants to merge 1 commit into
Open
feat(cli): apply object-lock retention in repository sync-to#5508fastlorenzo wants to merge 1 commit into
fastlorenzo wants to merge 1 commit into
Conversation
Allow maintaining an immutable replica of a repository whose primary storage does not support object locking (e.g. filesystem or SFTP) by synchronizing it to object-lock capable storage (e.g. S3): kopia repository sync-to s3 ... --retention-mode COMPLIANCE --retention-period 30d - add --retention-mode/--retention-period flags to sync-to and wrap the destination storage with the existing locking wrapper so blobs matching the locking prefixes are written with retention options - extend object locks on destination blobs that are already in sync (and therefore never re-uploaded) on every run, so protection does not lapse between syncs; opt out with --no-extend-object-locks - fail fast with an actionable error when the destination storage does not support object locking - export IsLockingStorageBlobID and WrapLockingStorage from the repo package (moved from repo/open.go) for reuse - make the blobtesting fake's ExtendBlobRetention match S3 PutObjectRetention semantics (sets retention even on versions that previously had none, and updates the mode) - document the workflow in the Synchronization and Ransomware Protection docs Fixes kopia#3759 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3759
Problem
As described in #3759, it is currently impossible to maintain an immutable copy of a repository whose primary storage does not support object locking (filesystem, SFTP, ...) by syncing it to object-lock capable storage:
repository sync-towrites every blob with emptyPutOptions, so nothing on the destination ever receives an object lock.maintenance --extend-object-locksonly runs against the connected (local) repository, whose backend rejects retention (blob-retention: unsupported put-blob option).Solution
Add destination-only retention support to
sync-to, reusing the existing locking machinery:--retention-mode/--retention-periodflags (same names/values asrepository create, validated by the existingformat.BlobStorageConfiguration.Validate()). When set, the destination storage is wrapped with the same locking wrapper used byrepo/open.go, so blobs matching the locking prefixes (packs, indexes,kopia.repository,kopia.blobcfg) are uploaded with retention. The source repository is not modified; no repository-format changes, no new persisted state.--no-extend-object-locksskips it for frequent syncs (with a less frequent lock-refreshing run recommended, always with a ≥24h safety margin before lock expiry).wrapLockingStorageis moved fromrepo/open.gotorepo/locking_storage.goand exported (WrapLockingStorage, plusIsLockingStorageBlobID) so the CLI and the open path share one implementation of the prefix rules.blobtestingfake'sExtendBlobRetentionnow matches S3PutObjectRetentionsemantics (places retention on versions that previously had none and updates the mode), which also covers upgrading an existing unlocked mirror to a locked one.Testing
IsLockingStorageBlobID/WrapLockingStorage(repo/locking_storage_test.go).cli/command_repository_sync_retention_test.go) using the in-memory versioned storage and a movable fake clock: verifies retention on locking-prefixed blobs only (including the initialkopia.repositorywrite), lock extension moving expiry forward on re-sync,--no-extend-object-locksand--dry-runnot mutating locks, flag validation failures, the unsupported-destination error (both on fresh sync and extension-only runs), and that the synchronized copy is a fully working repository.sync-to filesystemwith retention flags fails with the actionable message.X-Amz-Object-Lock-Mode/Retain-Until-Date) on locking blobs only, re-sync advances the retain-until date, deletion of a locked version is rejected as WORM-protected.make lintand existing test suites pass.Docs
Updated
site/content/docs/Advanced/Synchronization/_index.mdandsite/content/docs/Advanced/Ransomware Protection/_index.md: workflow, extension cost and scheduling guidance, recommended minimal credentials (no governance bypass), and--deleteimplications for immutable replicas.🤖 Generated with Claude Code