Skip to content

0015: Automatic import after a cluster is rebuilt

Date: 2026-10-09 · Status: accepted

Context

Tenax has no database. Its state is: - The store (S3): manifests, data, tags. This is the source of truth. - Git: the BackupStore, policies and targets. - Kubernetes: Backup objects, which mirror the manifests.

After a cluster is lost, the backups are safe in S3. The new cluster, however, starts with no Backup objects, and until now someone had to remember to run tenax catalog import before restores and retention showed them. Recovery should not depend on remembering a command.

Two traps were found while automating it: 1. The target controller keeps only the newest keepBackupObjects Backup objects per target (default 20). Importing every manifest would make import and pruning undo each other every hour. 2. The controller copies an object's pin into the backup's tags.json. An object created without the stored pin would unpin the backup, and retention could then delete it. The manual import had this bug.

Decision

  • The controller imports by itself (internal/recovery), per store, from the store reconciler:
  • on start;
  • when the set of targets changes;
  • every controller.autoImport.interval (default 1 h).

It is on by default: controller.autoImport.enabled. - For each existing target, it creates the missing Backup objects for the newest keepBackupObjects manifests, the same number pruning keeps, oldest first. - Each imported object starts with the pin from tags.json, so nothing is unpinned. - It is read-only on the store. The result goes to BackupStore.status.lastImportTime and lastImportCreated, an Imported event, and tenax_imported_backups_total. - tenax catalog import uses the same code.

Consequences

  • A rebuilt cluster with the same store, identities and targets shows its backups within minutes, without a manual step. The chaos test TestBackupObjectsLostAreImported deletes a target's Backup objects, restarts the controller and requires them back, pins included.
  • Deleting a Backup object no longer hides a backup: it comes back on the next import. Backups are removed by retention, never by deleting objects (that was already the rule, since S3 is the truth).
  • Backups older than keepBackupObjects get no object. They stay in the store, under retention and in the console's catalog, which reads S3 directly.
  • Each run lists every target's manifests and reads the tags of the backups it imports. That is a few S3 requests per target per hour.