commit 73b137f4c87dd0fb71ee3bb273210fe6ad7b66f9
parent 18067796a026d91496d602821f1c3b2bde2a2ca3
Author: Iván Ávalos <avalos@disroot.org>
Date: Mon, 10 Aug 2026 00:12:54 +0200
dd92: document runBackupCycle, getBackupDiagnostics and backup-status notifications
Diffstat:
1 file changed, 236 insertions(+), 42 deletions(-)
diff --git a/design-documents/092-incremental-backup-sync.rst b/design-documents/092-incremental-backup-sync.rst
@@ -882,14 +882,42 @@ later additions of the same denomination can be safely discarded.
Add a coin
~~~~~~~~~~
-A coin is generated by the wallet but not yet signed by the exchange.
+A coin comes into the wallet (withdrawn or refreshed) and is signed by the
+exchange.
+
+The wallet database stores per-coin key material, so the increment carries
+the coin **as it stands** -- key, blinding key, signature and status --
+rather than deriving it from a seed as earlier designs did. The wallet
+records an ``add-coin`` when the coin is created and a ``spend-coin`` when
+it is spent; the full collection pass emits the ``add-coin`` form for any
+coin the backup has never seen, whatever state it is in. The ``spend-coin``
+section is applied after the ``add-coin`` section, so a coin that was spent
+before a cycle ran restores in its spent state.
+
+Restoring the coin also recomputes the wallet's *coin availability* rows
+(the counts the balance reads) from the restored coins, so a restored
+wallet shows the same balance as the wallet that made the backup. The
+withdrawal groups, the reserves store and the refresh groups are still not
+backed up: restored coins are spendable and count towards the balance, but
+cannot be recouped, and pending withdrawals are not restored.
.. ts:def:: AddCoinInc
interface AddCoinInc {
type: "add-coin";
coinSource: CoinSource;
- denominationId: string;
+ sourceTransactionId?: string;
+ coinPub: string;
+ coinPriv: string;
+ denomPubHash: string;
+ denomSig: UnblindedDenominationSignature;
+ exchangeBaseUrl: string;
+ exchangeMasterPub: string;
+ blindingKey: string;
+ coinEvHash: string;
+ status: CoinStatus;
+ visible?: number;
+ maxAge: number;
ageCommitmentProof?: AgeCommitmentProof;
}
@@ -905,39 +933,19 @@ A coin is generated by the wallet but not yet signed by the exchange.
type: "withdrawal";
withdrawalGroupId: string;
coinNumber: number;
+ reservePub: string;
}
.. TODO: RefreshCoinSource (backup refresh groups?)
-* **Primary key:** ``[coinSource]``
-* **Deletion groups:** ``[coins, denominations, withdrawals]``
-
-Merge strategy
-++++++++++++++
-
-No merge is required, new coins are unique.
-
-Sign a coin
-~~~~~~~~~~~
-
-A coin is signed by the exchange.
-
-.. ts:def:: SignCoinInc
-
- interface SignCoinInc {
- type: "sign-coin";
- coinSource: CoinSource;
- denomSig: UnblindedDenominationSignature;
- }
-
-* **Primary key:** ``[coinSource]``
-* **Deletion groups:** ``[coins, withdrawals]``
+* **Primary key:** ``[coinPub]``
+* **Deletion groups:** ``[coins]``
Merge strategy
++++++++++++++
-No merge is required, only one signature for a given coin can be issued by the
-exchange, further attempts to sign it will fail.
+Last write wins: a coin is unique and its parameters never change, so the
+latest copy wins.
Spend a coin
~~~~~~~~~~~~
@@ -949,16 +957,23 @@ A signed coin is spent by the user.
interface SpendCoinInc {
type: "spend-coin";
coinSource: CoinSource;
+ sourceTransactionId?: string;
+ coinPub: string;
+ coinPriv: string;
+ denomPubHash: string;
+ denomSig: UnblindedDenominationSignature;
+ exchangeBaseUrl: string;
+ exchangeMasterPub: string;
+ blindingKey: string;
+ coinEvHash: string;
+ status: CoinStatus;
+ visible?: number;
+ maxAge: number;
+ ageCommitmentProof?: AgeCommitmentProof;
}
-* **Primary key:** ``[coinSource]``
-* **Deletion groups:** ``[coins, withdrawals]``
-
-Merge strategy
-++++++++++++++
-
-No merge is required, each coin can only be spent once, further attempts at
-spending the coin will fail.
+* **Primary key:** ``[coinPub]``
+* **Deletion groups:** ``[coins]``
Add a token
~~~~~~~~~~~
@@ -1902,6 +1917,30 @@ cycle. Independently, a periodic task runs a cycle at a fixed interval,
covering increments whose trigger never fired, e.g. because the wallet was
offline or the operation has no critical point.
+Full collection pass
+~~~~~~~~~~~~~~~~~~~~
+
+Eager recording covers every transaction family, but a record can still
+exist that no transaction ever reported: one that predates the backup, or
+one of a kind whose creation path bypasses the record handle. A periodic
+*full collection pass* is the safety net: it walks every record kind the
+backup manages (the ``backupSources`` of ``sources.ts``) and turns the
+records that have never been backed up into "start" increments.
+
+The pass is expensive -- it reads every denomination, exchange, bank
+account and transaction the wallet holds -- so it does not run on every
+cycle. It runs when a watermark, ``lastFullCollection`` in the wallet's
+backup configuration record, is older than 24 hours (or absent, i.e. never
+run). A cycle that woke from a critical point therefore stays cheap while
+still backing up whatever the transactions themselves reported.
+
+A forced cycle (see ``runBackupCycle`` in the wallet-core API below)
+bypasses the watermark and runs the pass regardless. This is the tool for
+developer diagnostics: everything the pass would collect is reported by
+``getBackupDiagnostics`` before the cycle runs, so the two requests
+together show exactly what is waiting to be backed up and what a forced
+cycle would add.
+
Restore process
---------------
@@ -2068,6 +2107,96 @@ takes over the providers.
Theirs = "theirs";
}
+``runBackupCycle`` runs a backup cycle now, instead of waiting for the
+periodic task. This is the dedicated "back up now" request; earlier
+implementations triggered a cycle by re-adding the active provider. The
+cycle runs synchronously and is serialized against any other cycle, and
+the response reports what it did.
+
+.. ts:def:: RunBackupCycleRequest
+
+ interface RunBackupCycleRequest {
+ // Run the full-collection pass even when its periodic watermark
+ // (24h since the last pass) has not elapsed. Harmless -- the pass
+ // only reads the wallet database -- and user interfaces are
+ // expected to only expose it in developer mode.
+ force?: boolean;
+ }
+
+.. ts:def:: RunBackupCycleResponse
+
+ interface RunBackupCycleResponse {
+ stats: BackupCycleStats;
+ }
+
+The same statistics are persisted by the wallet after every cycle,
+whatever triggered it, and are reported by ``getBackupDiagnostics`` as
+the "last cycle" outcome. The ``outcome`` field says how the cycle
+ended: ``"ok"`` (including idle cycles with nothing to push),
+``"payment-required"`` (the account is unpaid) or ``"error"``.
+
+.. ts:def:: BackupCycleStats
+
+ interface BackupCycleStats {
+ timestamp: TalerPreciseTimestamp;
+ // How the cycle ended: "ok", "payment-required" or "error".
+ outcome: "ok" | "payment-required" | "error";
+
+ // What the cycle pushed to the provider.
+ pushed: {
+ // Whether the full-collection pass ran in this cycle.
+ fullCollectionRan: boolean;
+ // Nonce of the block uploaded, if there was anything to upload.
+ blockNonce?: string;
+ incrementCount: number;
+ // Number of increments per increment type, keyed by the
+ // increment type's wire string (e.g. "payment-start").
+ incrementsByType: { [type: string]: number };
+ blobRefCount: number;
+ };
+
+ // What the cycle's pull applied from the provider.
+ pulled: {
+ blocksApplied: number;
+ blocksSkipped: number;
+ incrementCount: number;
+ incrementsByType: { [type: string]: number };
+ blobRestoreCount: number;
+ };
+ }
+
+``getBackupDiagnostics`` reports aggregated statistics about what the
+backup holds: what a cycle would back up right now, and what the last
+cycle restored. It is intended for developer tooling; user interfaces
+are expected to only expose it in developer mode, but the request itself
+is harmless and available on every platform.
+
+.. ts:def:: BackupDiagnostics
+
+ interface BackupDiagnostics {
+ // The increments waiting in the eager pending buffer: what a normal
+ // (unforced) cycle would push right now.
+ pending: IncrementStatSummary;
+
+ // The records the backup has never seen, which only the periodic
+ // full-collection pass picks up: what a forced cycle would add.
+ fullCollectionCandidates: IncrementStatSummary;
+
+ // Outcome of the last backup cycle, when at least one has run.
+ lastCycle?: BackupCycleStats;
+ }
+
+.. ts:def:: IncrementStatSummary
+
+ interface IncrementStatSummary {
+ incrementCount: number;
+ // Number of increments per increment type, keyed by the increment
+ // type's wire string.
+ incrementsByType: { [type: string]: number };
+ // Number of distinct blob references the increments carry.
+ blobRefCount: number;
+ }
+
Account keys are not part of any of these payloads: they are derived from the
wallet root key and the provider's base URL, so each provider sees an
unlinkable account public key and only the root key has to be preserved.
@@ -2077,6 +2206,58 @@ unlinkable account public key and only the root key has to be preserved.
account_priv = KDF(32, wallet_root_priv,
"taler-sync-account-key-salt", provider_base_url)
+Backup notifications
+--------------------
+
+The wallet pushes a ``backup-status`` notification to its clients
+(``NotificationType.BackupStatus``) as a backup cycle runs, through the
+regular wallet notification listener. Clients should use it instead of
+polling ``getBackupInfo`` to track a cycle: it reports the phase the cycle
+is in and, on the terminal phases, the outcome and the relevant counters.
+
+.. ts:def:: BackupStatusNotification
+
+ interface BackupStatusNotification {
+ type: "backup-status";
+ providerBaseUrl: string;
+ // "started", "pulling" or "pushing" for the progress phases; the
+ // cycle ends in exactly one of "completed", "error" and
+ // "payment-required".
+ phase: "started" | "pulling" | "pushing" |
+ "completed" | "error" | "payment-required";
+ // Number of increments packed into the block being pushed
+ // (at "pushing").
+ pendingIncrementCount?: number;
+ // Number of blocks the pull applied (at "completed").
+ pulledBlocks?: number;
+ // Nonce of the block pushed (at "completed").
+ pushedBlockNonce?: string;
+ // Reason of the failure (at "error").
+ error?: TalerErrorDetail;
+ // taler://pay/... URI of the prepared account payment (at
+ // "payment-required"); absent when the provider answered a bare
+ // 402 without a pay URI.
+ talerUri?: string;
+ timestamp: TalerPreciseTimestamp;
+ }
+
+The wallet emits ``started`` when a cycle begins, ``pulling`` before the
+linked list is fetched, ``pushing`` with the increment count before the
+packed block (and its blobs) is uploaded, and a terminal phase when the
+cycle ends:
+
+* ``completed`` -- the cycle ran without error and without requiring
+ payment (``pulledBlocks`` / ``pushedBlockNonce`` carry the counters);
+* ``payment-required`` -- the account is unpaid; a payment transaction
+ may already have been prepared, and the UI should take the user to it;
+* ``error`` -- the cycle failed (with ``error`` as the reason); the
+ wallet retries on its own schedule, so the notification is only for
+ the user interface.
+
+An earlier ``backup-error`` notification type (``BackupOperationError``)
+was part of a legacy backup proof of concept and has been removed in
+favor of the ``error`` phase of ``backup-status``.
+
Definition of done
==================
@@ -2096,17 +2277,30 @@ Definition of done
blobs (uploaded ahead of the referencing blocks with their reference-count
adjustments, and fetched and stored back into the contract-terms store on
the pull side); a transaction whose terms are not available is shown in a
- reduced form instead of failing the transaction listing. The withdrawal
- and coin families are blocked on the still undefined "reserves" increment
- type (proposed above), and the token family on a schema redesign: the
- increments in this document model seed-derived tokens, while the wallet
- database now stores per-token key material, so restoring from the
- documented increments could not produce spendable tokens).
+ reduced form instead of failing the transaction listing. The coin family
+ (``add-coin`` / ``spend-coin``, carrying the per-coin key material the
+ wallet database stores) is implemented, and restoring a coin recomputes
+ the coin-availability rows so that the balance matches across wallets.
+ The withdrawal families and the reserves store are still blocked on the
+ "reserves" increment type (proposed above), so restored coins cannot be
+ recouped and pending withdrawals are not restored; the token family is
+ blocked on a schema redesign: the increments in this document model
+ seed-derived tokens, while the wallet database now stores per-token key
+ material, so restoring from the documented increments could not produce
+ spendable tokens). The ``runBackupCycle`` and ``getBackupDiagnostics``
+ request handlers, the per-cycle statistics (persisted for the diagnostics
+ view), the forced full-collection pass and the ``backup-status``
+ notifications are implemented. The native (sqlite) backend persists the
+ ``originBlocks`` of every backup-managed record; earlier native databases
+ are migrated on open).
* [x] Design sync API (+ auth).
* [ ] Server-side implementation (partial: block GET/POST/PUT/DELETE, object
store GET/POST with reference counting, /config and payments done;
reconciliation mechanism still missing).
-* [ ] UI/UX for backup and sync.
+* [ ] UI/UX for backup and sync (done in the Android wallet: "back up now"
+ uses ``runBackupCycle``, with a force-full-backup control and a backup
+ diagnostics card exposed in developer mode, and a progress display driven
+ by the ``backup-status`` notifications).
Alternatives
============
