commit 703a71813e390f533993e4e414471c63b0958d97
parent 36d2d3d52dd240d0982bb54ba027784defbe8350
Author: Iván Ávalos <avalos@disroot.org>
Date: Tue, 18 Aug 2026 00:17:21 +0200
dd92: formatting and cleanup
Diffstat:
1 file changed, 529 insertions(+), 560 deletions(-)
diff --git a/design-documents/092-incremental-backup-sync.rst b/design-documents/092-incremental-backup-sync.rst
@@ -68,22 +68,20 @@ Threat model
The design protects the confidentiality of the wallet's backup contents
against any party that does not hold the wallet's backup encryption key,
-including the backup service itself. Blocks and blobs are end-to-end
-encrypted with keys derived from secrets that only the user's wallets know,
-so neither a passive network observer nor the operator of the backup service
-can learn anything about the contents of a backup from the data they can
-access.
+including the backup service itself. Blocks and blobs are end-to-end encrypted
+with keys derived from secrets that only the user's wallets know, so neither a
+passive network observer nor the operator of the backup service can learn
+anything about the contents of a backup from the data they can access.
Within this model, the backup service is trusted to honor deletion requests
and to not retain deleted blocks nor previous versions of updated blocks. The
-protocol does **not** defend against a service that fails to do so: while
-such a service still cannot decrypt the retained data, it can defeat the
-plausible deniability requirement by preserving evidence that certain
-information once existed in the backup, and countering this would be
-impractical for an incremental, multi-device protocol. Users must therefore
-trust the sync server operator in such cases, as well as to refrain from
-misusing the metadata that the protocol necessarily exposes to it (see
-:ref:`limitations`).
+protocol does **not** defend against a service that fails to do so: while such
+a service still cannot decrypt the retained data, it can defeat the plausible
+deniability requirement by preserving evidence that certain information once
+existed in the backup, and countering this would be impractical for an
+incremental, multi-device protocol. Users must therefore trust the sync server
+operator in such cases, as well as to refrain from misusing the metadata that
+the protocol necessarily exposes to it (see :ref:`limitations`).
Proposed solution
=================
@@ -213,27 +211,19 @@ add to the synchronization group.
Block store API
~~~~~~~~~~~~~~~
-The account key is the base32-encoded Crockford representation of an
-EdDSA public key that identifies the backup account. All upload
-requests must be signed by the corresponding private key; the signature
-is transmitted in the request body.
+The account key is the base32-encoded Crockford representation of an EdDSA
+public key that identifies the backup account. All upload requests must be
+signed by the corresponding private key; the signature is transmitted in the
+request body.
Binary values in URLs, headers and JSON bodies (nonces, UIDs, hashes,
-signatures and the encrypted payloads themselves) are all base32-encoded
-in Crockford representation, as is usual for Taler.
+signatures and the encrypted payloads themselves) are all base32-encoded in
+Crockford representation, as is usual for Taler.
-Signatures use EdDSA with the account private key. Each signature
-payload follows the common Taler signing structure with a
-``purpose`` field (see :ref:`Signatures` in the API common conventions
-for the general format). The specific payloads are:
-
-.. warning::
-
- The three purpose numbers below are **not in the GANA registry yet**, so
- both the server and the wallet spell them out locally. Until they are
- registered, the wire format depends on whether a given build happened to
- have them defined, and a mismatch shows up only as a signature that fails
- to verify.
+Signatures use EdDSA with the account private key. Each signature payload
+follows the common Taler signing structure with a ``purpose`` field (see
+:ref:`Signatures` in the API common conventions for the general format). The
+specific payloads are:
.. sourcecode:: c
@@ -275,17 +265,17 @@ for the general format). The specific payloads are:
struct GNUNET_HashCode hash;
};
-Absent optional nonces (``prev_nonce`` / ``next_nonce``) are
-treated as all-zeros in the signed data.
+Absent optional nonces (``prev_nonce`` / ``next_nonce``) are treated as
+all-zeros in the signed data.
-The ``refs_hash`` field covers the ``object_refs`` of the request, so
-that the reference-count adjustments cannot be altered in transit. It
-is the SHA-512 hash over a canonical *binary* encoding of the
-references — not over their JSON representation.
+The ``refs_hash`` field covers the ``object_refs`` of the request, so that the
+reference-count adjustments cannot be altered in transit. It is the SHA-512
+hash over a canonical *binary* encoding of the references — not over their
+JSON representation.
-Each reference is laid out as the 64 raw UID bytes followed by the
-adjustment as a signed 16-bit integer in network byte order, and the
-resulting 66-byte records are concatenated in ascending order of UID:
+Each reference is laid out as the 64 raw UID bytes followed by the adjustment
+as a signed 16-bit integer in network byte order, and the resulting 66-byte
+records are concatenated in ascending order of UID:
.. code-block:: text
@@ -295,29 +285,28 @@ resulting 66-byte records are concatenated in ascending order of UID:
| adjustment (2 byte, int16) |
+----------------------------+
-Sorting by UID is required because ``object_refs`` travels as a JSON
-object, whose member order is not preserved. A request without any
-references hashes the empty byte string.
-
-A UID may appear at most once, since the wire format keys the references
-by UID and could not otherwise transmit them faithfully.
-
-The server stores the ``upload_sig`` with the block, together with the
-rest of the signed context (``old_hash`` and ``refs_hash``), and returns
-them in the block list. A wallet therefore verifies every block's
-stored signature against the account key before applying it; a block
-whose signature does not verify must not be applied.
-
-Operations that rewrite the links of an existing block (an append
-relinks the previous tail, a delete relinks both of its neighbours)
-require that block's *new* signature to be uploaded along with the
-operation. This is an ordinary ``TALER_SIGNATURE_SYNC_BLOCK_UPLOAD``
-signature over the relinked block's new nonces, carried in the
-``relink_prev`` / ``relink_next`` fields of the request. The server
-verifies it against the current state of the relinked block and stores
-it in the block's row; relinking never changes the block's data, so the
-signature's ``old_hash`` and ``new_hash`` are both the block's stored
-hash.
+Sorting by UID is required because ``object_refs`` travels as a JSON object,
+whose member order is not preserved. A request without any references hashes
+the empty byte string.
+
+A UID may appear at most once, since the wire format keys the references by
+UID and could not otherwise transmit them faithfully.
+
+The server stores the ``upload_sig`` with the block, together with the rest of
+the signed context (``old_hash`` and ``refs_hash``), and returns them in the
+block list. A wallet therefore verifies every block's stored signature
+against the account key before applying it; a block whose signature does not
+verify must not be applied.
+
+Operations that rewrite the links of an existing block (an append relinks the
+previous tail, a delete relinks both of its neighbours) require that block's
+*new* signature to be uploaded along with the operation. This is an ordinary
+``TALER_SIGNATURE_SYNC_BLOCK_UPLOAD`` signature over the relinked block's new
+nonces, carried in the ``relink_prev`` / ``relink_next`` fields of the
+request. The server verifies it against the current state of the relinked
+block and stores it in the block's row; relinking never changes the block's
+data, so the signature's ``old_hash`` and ``new_hash`` are both the block's
+stored hash.
.. http:get:: /config
@@ -340,22 +329,22 @@ hash.
version: string;
}
- ``storage_limit_in_megabytes`` is the per-upload limit enforced for
- both blocks and objects; exceeding it yields ``413``. ``version``
- follows the Taler ``current:revision:age`` convention.
+ ``storage_limit_in_megabytes`` is the per-upload limit enforced for both
+ blocks and objects; exceeding it yields ``413``. ``version`` follows the
+ Taler ``current:revision:age`` convention.
.. http:get:: /backups/${ACCOUNT_KEY}
Report the state of the account: when it expires, and how much of the
- storage allowance its backup uses. Requires no signature, like the
- other read endpoints -- the account public key is the capability, and
- the stored data is client-encrypted.
+ storage allowance its backup uses. Requires no signature, like the other
+ read endpoints -- the account public key is the capability, and the stored
+ data is client-encrypted.
- This is the only endpoint that answers for an expired account rather
- than demanding payment: when the account expires is precisely what
- the caller is asking, so a ``402`` here would be useless. Wallets
- use it to tell the user how long the backup is paid for without
- waiting for the next write to fail.
+ This is the only endpoint that answers for an expired account rather than
+ demanding payment: when the account expires is precisely what the caller is
+ asking, so a ``402`` here would be useless. Wallets use it to tell the
+ user how long the backup is paid for without waiting for the next write to
+ fail.
**Response**
@@ -422,24 +411,24 @@ hash.
refs_hash: HashCodeString;
}
- ``data`` is the encrypted block payload as it was uploaded, and
- hashes to ``block_hash``. ``prev_nonce`` and ``next_nonce`` are
- absent for the first and last block of the linked list respectively.
- ``upload_sig`` is the signature stored with the block, and
- ``old_hash`` / ``refs_hash`` the remainder of the signed context; the
- wallet verifies the signature before applying the block.
+ ``data`` is the encrypted block payload as it was uploaded, and hashes to
+ ``block_hash``. ``prev_nonce`` and ``next_nonce`` are absent for the first
+ and last block of the linked list respectively. ``upload_sig`` is the
+ signature stored with the block, and ``old_hash`` / ``refs_hash`` the
+ remainder of the signed context; the wallet verifies the signature before
+ applying the block.
.. http:post:: /backups/${ACCOUNT_KEY}/blocks/${NONCE}
- Upload a new block and append it at the end of the account's linked
- list. If a block with the same nonce already exists, the content
- hash is compared: if it matches, a ``304 Not modified`` is returned;
- if it differs, the client should use ``PUT`` instead.
+ Upload a new block and append it at the end of the account's linked list.
+ If a block with the same nonce already exists, the content hash is
+ compared: if it matches, a ``304 Not modified`` is returned; if it differs,
+ the client should use ``PUT`` instead.
- The request must include an ``If-None-Match`` header containing the
- quoted base32-encoded SHA-512 hash of the encrypted block data. This
- hash is used by the server to detect duplicates, and the server
- rejects the upload if the ``data`` in the body does not hash to it.
+ The request must include an ``If-None-Match`` header containing the quoted
+ base32-encoded SHA-512 hash of the encrypted block data. This hash is used
+ by the server to detect duplicates, and the server rejects the upload if
+ the ``data`` in the body does not hash to it.
**Request**
@@ -531,16 +520,15 @@ hash.
.. http:put:: /backups/${ACCOUNT_KEY}/blocks/${NONCE}
- Replace an existing block's content in-place. Semantics are identical
- to ``POST`` on the same endpoint, with one addition: the
- ``If-Match`` header must contain the quoted base32-encoded SHA-512
- hash of the old block data that is being replaced. The server rejects
- the request with ``409 Conflict`` if the old hash, ``prev_nonce`` or
- ``next_nonce`` do not match the stored block.
+ Replace an existing block's content in-place. Semantics are identical to
+ ``POST`` on the same endpoint, with one addition: the ``If-Match`` header
+ must contain the quoted base32-encoded SHA-512 hash of the old block data
+ that is being replaced. The server rejects the request with ``409
+ Conflict`` if the old hash, ``prev_nonce`` or ``next_nonce`` do not match
+ the stored block.
- The ``upload_sig`` must also cover the old data hash (from
- ``If-Match``) in addition to the new data hash (from
- ``If-None-Match``).
+ The ``upload_sig`` must also cover the old data hash (from ``If-Match``) in
+ addition to the new data hash (from ``If-None-Match``).
.. note::
@@ -556,10 +544,10 @@ hash.
.. http:delete:: /backups/${ACCOUNT_KEY}/blocks/${NONCE}
- Delete an existing block from the linked list. The request must
- include an ``If-Match`` header containing the quoted base32-encoded
- SHA-512 hash of the block data to delete, which the server uses to
- detect concurrent modifications.
+ Delete an existing block from the linked list. The request must include an
+ ``If-Match`` header containing the quoted base32-encoded SHA-512 hash of
+ the block data to delete, which the server uses to detect concurrent
+ modifications.
**Request**
@@ -640,9 +628,9 @@ block store. Any blobs with a reference count of zero will be deleted from the
server after a preconfigured expiration period.
Uploads are keyed by UID and are idempotent: re-uploading a UID that the
-account already holds is accepted and changes nothing, so a wallet that
-is unsure whether a blob is already present can simply upload it again.
-The stored contents of an existing UID are never replaced.
+account already holds is accepted and changes nothing, so a wallet that is
+unsure whether a blob is already present can simply upload it again. The
+stored contents of an existing UID are never replaced.
Blob format
~~~~~~~~~~~
@@ -660,13 +648,13 @@ key and the hash of the unencrypted file:
Every blob therefore has its own key. The 64-byte ``uid``, which is the
SHA-512 hash of that key, is what indexes the object in the store and is the
only one of the two the sync server ever learns; the key itself is stored
-*inside the blocks* that reference the blob, where it doubles as the
-reference to the object that has to be fetched.
+*inside the blocks* that reference the blob, where it doubles as the reference
+to the object that has to be fetched.
-The key is thus all a wallet needs to both locate and decrypt a blob, which
-is the only thing a block carries. The `secretbox`_ nonce is consequently
-derived from the key as well, as the first 24 bytes of ``H(key)``. Nonce
-reuse cannot occur, because distinct plaintexts derive distinct keys.
+The key is thus all a wallet needs to both locate and decrypt a blob, which is
+the only thing a block carries. The `secretbox`_ nonce is consequently derived
+from the key as well, as the first 24 bytes of ``H(key)``. Nonce reuse cannot
+occur, because distinct plaintexts derive distinct keys.
Because the key is derived from the plaintext, blobs are content-addressed:
identical contents yield the same key, UID and ciphertext, so an unchanged
@@ -687,8 +675,8 @@ blob is only ever uploaded once.
Object store API
~~~~~~~~~~~~~~~~
-Objects are scoped to the account: a UID is only ever visible to the
-account that uploaded it.
+Objects are scoped to the account: a UID is only ever visible to the account
+that uploaded it.
.. http:get:: /backups/${ACCOUNT_KEY}/objects/${UID}
@@ -715,13 +703,13 @@ account that uploaded it.
.. http:post:: /backups/${ACCOUNT_KEY}/objects/${UID}
- Upload an encrypted blob and store it in the hash-indexed object
- store. The ``$UID`` is the object's unique identifier.
+ Upload an encrypted blob and store it in the hash-indexed object store.
+ The ``$UID`` is the object's unique identifier.
The object is stored with a reference count of zero; it only becomes
- referenced once a block naming it in ``object_refs`` is uploaded.
- Until then it is subject to expiry, so blobs should be uploaded
- shortly before the block that references them.
+ referenced once a block naming it in ``object_refs`` is uploaded. Until
+ then it is subject to expiry, so blobs should be uploaded shortly before
+ the block that references them.
**Request**
@@ -788,23 +776,22 @@ carefuly devised for every top-level operation type in the block, so that
wallets can deterministically agree on a consistent global state.
One rule cuts across all of the transaction families: **a transaction only
-ever moves towards its end.** The wallets of a group work on the same
-transactions at the same time, so an increment that would take a record
-back to a state it has already moved past is describing an older view of
-it, and only its origin block is recorded. The terminal states are ranked
-rather than simply frozen, so that two wallets which reached *different*
-ones both settle on the same one:
+ever moves towards its end.** The wallets of a group work on the same
+transactions at the same time, so an increment that would take a record back
+to a state it has already moved past is describing an older view of it, and
+only its origin block is recorded. The terminal states are ranked rather than
+simply frozen, so that two wallets which reached *different* ones both settle
+on the same one:
.. code-block:: text
done > failed > aborted > expired > (not terminal)
-Preferring ``done`` is deterministic, which is what convergence needs, and
-it is also the truthful answer: a transaction that finished actually moved
-the money. Without the rule, a wallet that completed a withdrawal would
-pull in the abort another device had issued against the copy it restored,
-and end up showing an abandoned transaction while holding the coins it
-produced.
+Preferring ``done`` is deterministic, which is what convergence needs, and it
+is also the truthful answer: a transaction that finished actually moved the
+money. Without the rule, a wallet that completed a withdrawal would pull in
+the abort another device had issued against the copy it restored, and end up
+showing an abandoned transaction while holding the coins it produced.
Add or update an exchange
~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -932,53 +919,50 @@ Add a coin
A coin comes into the wallet (withdrawn or refreshed) and is signed by the
exchange.
-The wallet database stores per-coin key material, so the increment carries
-the coin **as it stands** -- key, blinding key, signature and status --
-rather than deriving it from a seed as earlier designs did. The wallet
-records an ``add-coin`` when the coin is created and a ``spend-coin`` when
-it is spent; the full collection pass emits the ``add-coin`` form for any
-coin the backup has never seen, whatever state it is in. The ``spend-coin``
-section is applied after the ``add-coin`` section, so a coin that was spent
-before a cycle ran restores in its spent state.
-
-Restoring the coin also recomputes the wallet's *coin availability* rows
-(the counts the balance reads) from the restored coins, so a restored
-wallet shows the same balance as the wallet that made the backup. The
-counts are always derived and never carried, which is what makes the
-restore idempotent; only a coin that is spendable (status ``fresh``)
-counts, matching what the wallet's own bookkeeping does with a suspended
-one.
-
-For the two balances to agree, *every* change to whether a coin counts has
-to reach the other wallets, not only spending: a coin melted into a
-refresh, recouped from a revoked denomination, written off with its
-denomination, or suspended by the user is reported with a ``spend-coin``
-increment carrying its new status. The section is the coin's terminal
-update, whatever brought it about. A change that is not reported is the
-one way the two devices can end up disagreeing about how much money the
-user has, since a coin that is already backed up is never offered again by
-the full collection pass. The
-reserves (and with them the ability to recoup a restored coin) are backed
+The wallet database stores per-coin key material, so the increment carries the
+coin **as it stands** -- key, blinding key, signature and status -- rather
+than deriving it from a seed as earlier designs did. The wallet records an
+``add-coin`` when the coin is created and a ``spend-coin`` when it is spent;
+the full collection pass emits the ``add-coin`` form for any coin the backup
+has never seen, whatever state it is in. The ``spend-coin`` section is
+applied after the ``add-coin`` section, so a coin that was spent before a
+cycle ran restores in its spent state.
+
+Restoring the coin also recomputes the wallet's *coin availability* rows (the
+counts the balance reads) from the restored coins, so a restored wallet shows
+the same balance as the wallet that made the backup. The counts are always
+derived and never carried, which is what makes the restore idempotent; only a
+coin that is spendable (status ``fresh``) counts, matching what the wallet's
+own bookkeeping does with a suspended one.
+
+For the two balances to agree, *every* change to whether a coin counts has to
+reach the other wallets, not only spending: a coin melted into a refresh,
+recouped from a revoked denomination, written off with its denomination, or
+suspended by the user is reported with a ``spend-coin`` increment carrying its
+new status. The section is the coin's terminal update, whatever brought it
+about. A change that is not reported is the one way the two devices can end
+up disagreeing about how much money the user has, since a coin that is already
+backed up is never offered again by the full collection pass.
+
+The reserves (and with them the ability to recoup a restored coin) are backed
up by the ``add-reserve`` family, and the withdrawal family
(``withdrawal-start`` / ``withdrawal-abort`` / ``withdrawal-done`` /
-``withdrawal-fail``, referencing the reserve by
-``[exchangeBaseUrl, reservePub]``, and carrying the ``wgInfo`` with the
-``taler://withdraw`` URI that identifies the bank's operation) restores
-the withdrawal transactions themselves and lets a restored wallet
-continue a pending one -- the bank's operation is keyed by that URI, and
-the reserve key pair and the coin seed are in the backup too; only an
-expired bank operation cannot be resumed. A refreshed coin's melt is
-backed up by the refresh family below, so a restored coin can be
-recouped-refreshed as well as recouped (see the recoup discussion under
-"Add a reserve").
+``withdrawal-fail``, referencing the reserve by ``[exchangeBaseUrl,
+reservePub]``, and carrying the ``wgInfo`` with the ``taler://withdraw`` URI
+that identifies the bank's operation) restores the withdrawal transactions
+themselves and lets a restored wallet continue a pending one -- the bank's
+operation is keyed by that URI, and the reserve key pair and the coin seed are
+in the backup too; only an expired bank operation cannot be resumed. A
+refreshed coin's melt is backed up by the refresh family below, so a restored
+coin can be recouped-refreshed as well as recouped (see the recoup discussion
+under "Add a reserve").
``exchangeWithdrawValues`` carries the blinding values the exchange
-contributed to the withdraw, which a recoup has to replay. For an RSA
-coin they are the constant ``{"cipher": "RSA"}``; for a Clause-Schnorr
-coin they are the R-values, which nothing can re-derive, so they have to
-travel in the increment. The field is optional because it was added
-after the increment was first released: a coin from a wallet that
-predates it is treated as RSA.
+contributed to the withdraw, which a recoup has to replay. For an RSA coin
+they are the constant ``{"cipher": "RSA"}``; for a Clause-Schnorr coin they
+are the R-values, which nothing can re-derive, so they have to travel in the
+increment. The field is optional because it was added after the increment was
+first released: a coin from a wallet that predates it is treated as RSA.
.. ts:def:: AddCoinInc
@@ -1069,11 +1053,11 @@ A token is generated by the wallet but not yet signed by the merchant (the
wallet database calls this a *slate*).
Like coins, tokens were originally designed as seed-derived: the increment
-carried ``[secretSeed, choiceIndex, outputIndex]`` and the wallet
-re-derived the key pair from it. The wallet database stores per-token key
-material instead, so the increments carry the token as it stands, and the
-token's *use* public key is the primary key of the family. The three
-increments share one body, ``TokenIncBase``:
+carried ``[secretSeed, choiceIndex, outputIndex]`` and the wallet re-derived
+the key pair from it. The wallet database stores per-token key material
+instead, so the increments carry the token as it stands, and the token's *use*
+public key is the primary key of the family. The three increments share one
+body, ``TokenIncBase``:
.. ts:def:: TokenIncBase
@@ -1127,9 +1111,9 @@ No merge is required, new tokens are unique.
Sign a token
~~~~~~~~~~~~
-A token is signed by the merchant. Applying this increment also removes
-the slate the token was issued from, the same way the wallet's own
-issuance flow does.
+A token is signed by the merchant. Applying this increment also removes the
+slate the token was issued from, the same way the wallet's own issuance flow
+does.
.. ts:def:: SignTokenInc
@@ -1177,15 +1161,14 @@ Start a withdrawal
User initiates a withdrawal.
-The increment references the reserve by ``[exchangeBaseUrl, reservePub]``
-(see the "Add a reserve" section): the restored wallet takes the reserve's
-key pair from the reserve record. It also carries the ``wgInfo`` -- for a
-bank-integrated withdrawal, the ``taler://withdraw`` URI that identifies
-the bank's withdrawal operation. That URI, the reserve key pair and the
-coin seed (all in the backup) are everything a restored wallet needs to
-continue a withdrawal that was still pending on the other device; the only
-thing that cannot be resumed is a bank operation the bank has already
-expired or deleted.
+The increment references the reserve by ``[exchangeBaseUrl, reservePub]`` (see
+the "Add a reserve" section): the restored wallet takes the reserve's key pair
+from the reserve record. It also carries the ``wgInfo`` -- for a
+bank-integrated withdrawal, the ``taler://withdraw`` URI that identifies the
+bank's withdrawal operation. That URI, the reserve key pair and the coin seed
+(all in the backup) are everything a restored wallet needs to continue a
+withdrawal that was still pending on the other device; the only thing that
+cannot be resumed is a bank operation the bank has already expired or deleted.
.. ts:def:: WithdrawalStartInc
@@ -1280,9 +1263,9 @@ Store all ``failReason`` in the database.
Set the reserve seed
~~~~~~~~~~~~~~~~~~~~
-The wallet derives every reserve key pair from a single wallet-level seed
-(32 random bytes), so that the backup carries no per-reserve key material:
-the private key of reserve ``i`` is re-derived as
+The wallet derives every reserve key pair from a single wallet-level seed (32
+random bytes), so that the backup carries no per-reserve key material: the
+private key of reserve ``i`` is re-derived as
.. code-block:: text
@@ -1290,11 +1273,11 @@ the private key of reserve ``i`` is re-derived as
and the public key from the private one (``eddsa_get_public``). The seed
itself is wallet state and travels in the backup like the wallet root key;
-this increment is what the backup carries it as. It is created lazily at
-the first reserve created after this feature ships, so wallets that predate
-it do not grow a seed until they create their next reserve. Reserves
-created before the seed existed keep their random key pairs and are backed
-up with the ``reservePriv`` fallback of ``add-reserve`` below.
+this increment is what the backup carries it as. It is created lazily at the
+first reserve created after this feature ships, so wallets that predate it do
+not grow a seed until they create their next reserve. Reserves created before
+the seed existed keep their random key pairs and are backed up with the
+``reservePriv`` fallback of ``add-reserve`` below.
.. ts:def:: SetReserveSeedInc
@@ -1321,9 +1304,9 @@ Add a reserve
A reserve is created by the wallet for every withdrawal and for the merge
capability of P2P payments, and its key pair lives in the wallet's
``reserves`` object store (see the ``WalletReserve`` record in ``db.ts``).
-The increment carries the record's identity -- the exchange and the
-reserve's derivation index -- and, for the reserves that predate the seed,
-the private key.
+The increment carries the record's identity -- the exchange and the reserve's
+derivation index -- and, for the reserves that predate the seed, the private
+key.
.. ts:def:: AddReserveInc
@@ -1346,73 +1329,71 @@ Last write wins: the identity of a reserve never changes, and a re-recorded
increment (e.g. by the full collection pass) carries the same index and the
same key material.
-The public key of the reserve is *not* carried: it is derived from the
-private key on restore (``eddsa_get_public``), whether the private key was
-re-derived from the seed or restored from ``reservePriv``. The restored
-record therefore has the same ``reservePub`` as the wallet that created the
-reserve, which is what the other increments reference it by (see below).
-The ``WalletReserve`` record gains ``exchangeBaseUrl``, ``reserveIndex``
-and the ``reserveSeedDerived`` marker (which decides whether the full
-collection pass emits the index-only form or the index-plus-private-key
-form); the exchange base URL is required by the increment and was missing
-from the record (see the ``FIXME: Should reference exchange.`` comment in
-``db.ts`` and the redundant ``exchangeBaseUrl`` of
-``WithdrawalGroupRecord``).
+The public key of the reserve is *not* carried: it is derived from the private
+key on restore (``eddsa_get_public``), whether the private key was re-derived
+from the seed or restored from ``reservePriv``. The restored record therefore
+has the same ``reservePub`` as the wallet that created the reserve, which is
+what the other increments reference it by (see below). The ``WalletReserve``
+record gains ``exchangeBaseUrl``, ``reserveIndex`` and the
+``reserveSeedDerived`` marker (which decides whether the full collection pass
+emits the index-only form or the index-plus-private-key form); the exchange
+base URL is required by the increment and was missing from the record (see the
+``FIXME: Should reference exchange.`` comment in ``db.ts`` and the redundant
+``exchangeBaseUrl`` of ``WithdrawalGroupRecord``).
The remaining fields of ``WalletReserve`` (``status``, the KYC thresholds,
-``kycAccessToken``, ``amlReview``) are all derivable by querying the
-exchange and are deliberately not backed up, so that a restored wallet
-re-derives them instead of trusting stale state.
+``kycAccessToken``, ``amlReview``) are all derivable by querying the exchange
+and are deliberately not backed up, so that a restored wallet re-derives them
+instead of trusting stale state.
Recoup
++++++
-The reserve increment is what keeps recoup working on a restored wallet.
-The recoup request itself is signed by the *coin*: the coin record
-(``add-coin``) carries the coin private key, the blinding key and the
-denomination signature the request needs, and the request names the reserve
-only by its public key, which the coin source carries. After the exchange
-confirms the recoup, the wallet queries the reserve's balance and withdraws
-it back into coins; that re-withdrawal needs the reserve *private* key,
-which is exactly what ``add-reserve`` restores. The recoup of a refreshed
-coin (``recoup-refresh``) likewise needs only the coin records -- the
-refreshed coin plus the old coin the refresh source names -- so no
-refresh-group data is involved.
+The reserve increment is what keeps recoup working on a restored wallet. The
+recoup request itself is signed by the *coin*: the coin record (``add-coin``)
+carries the coin private key, the blinding key and the denomination signature
+the request needs, and the request names the reserve only by its public key,
+which the coin source carries. After the exchange confirms the recoup, the
+wallet queries the reserve's balance and withdraws it back into coins; that
+re-withdrawal needs the reserve *private* key, which is exactly what
+``add-reserve`` restores. The recoup of a refreshed coin (``recoup-refresh``)
+likewise needs only the coin records -- the refreshed coin plus the old coin
+the refresh source names -- so no refresh-group data is involved.
The upcoming batch recoup protocol (``vRECOUP``, see ``api-exchange.rst``)
-adds, per coin, the Clause-Schnorr blinding data
-(``cs_session_nonce`` and the ``cs_r_pubs`` of the exchange's
-``/blinding-prepare``) for post-quantum denominations. The wallet does not
-store that data anywhere yet; when it does, the ``add-coin`` increment must
-carry it (as optional fields). That is a coin-family extension; the
-reserve side of a post-quantum recoup stays as described above.
+adds, per coin, the Clause-Schnorr blinding data (``cs_session_nonce`` and the
+``cs_r_pubs`` of the exchange's ``/blinding-prepare``) for post-quantum
+denominations. The wallet does not store that data anywhere yet; when it
+does, the ``add-coin`` increment must carry it (as optional fields). That is
+a coin-family extension; the reserve side of a post-quantum recoup stays as
+described above.
Why the schema matters to the other increment types
++++++++++++++++++++++++++++++++++++++++++++++++++++
-The ``reserves`` store is referenced, directly or through its row id, by
-the withdrawal groups (``reservePub``/``reservePriv``), the coin sources
-(``WithdrawCoinSource.reservePub``, used for recouping), the exchange
-entries (``currentMergeReserveRowId``) and the peer-pull-credit records
+The ``reserves`` store is referenced, directly or through its row id, by the
+withdrawal groups (``reservePub``/``reservePriv``), the coin sources
+(``WithdrawCoinSource.reservePub``, used for recouping), the exchange entries
+(``currentMergeReserveRowId``) and the peer-pull-credit records
(``mergeReserveRowId``):
* ``withdrawal-start`` is the most obvious case: the wallet's
- ``WithdrawalGroupRecord`` embeds the reserve key pair and the exchange
- base URL. With ``add-reserve``, a ``withdrawal-start`` increment can
- reference the reserve by ``[exchangeBaseUrl, reservePub]`` instead of
- carrying the key pair, avoiding duplication.
-* ``add-coin`` / ``spend-coin`` reference the reserve through the
- withdrawal coin source's ``reservePub``; the restored reserve record is
- what makes the restored coin recoupable (see above).
-* The exchange entries and the peer-pull-credit records reference the
- merge reserve by a *row id* into the ``reserves`` store, which is not
- portable across wallets. The ``add-exchange`` increment does not carry
- the ``currentMergeReserveRowId`` pointer, so a restored exchange entry
- starts without one; the merge reserve remains findable by its public
- key, and re-linking the pointer on restore is a follow-up.
-
-Every increment family in this document is implemented; see the
-"Definition of done" section for what remains.
+ ``WithdrawalGroupRecord`` embeds the reserve key pair and the exchange base
+ URL. With ``add-reserve``, a ``withdrawal-start`` increment can reference
+ the reserve by ``[exchangeBaseUrl, reservePub]`` instead of carrying the key
+ pair, avoiding duplication.
+* ``add-coin`` / ``spend-coin`` reference the reserve through the withdrawal
+ coin source's ``reservePub``; the restored reserve record is what makes the
+ restored coin recoupable (see above).
+* The exchange entries and the peer-pull-credit records reference the merge
+ reserve by a *row id* into the ``reserves`` store, which is not portable
+ across wallets. The ``add-exchange`` increment does not carry the
+ ``currentMergeReserveRowId`` pointer, so a restored exchange entry starts
+ without one; the merge reserve remains findable by its public key, and
+ re-linking the pointer on restore is a follow-up.
+
+Every increment family in this document is implemented; see the "Definition of
+done" section for what remains.
Start a deposit
~~~~~~~~~~~~~~~
@@ -2011,11 +1992,11 @@ The wallet melts the remainder of one or more coins into fresh ones -- as
change after a payment, or to renew a coin whose denomination is about to
expire.
-The group carries the plan; how far it has got lives in the per-coin
-sessions below. A restored group is what lets a wallet that melted a coin
-and then lost the device still collect the change: the exchange holds the
-first melt commitment, and a wallet that re-melted with a fresh seed could
-not reveal against it.
+The group carries the plan; how far it has got lives in the per-coin sessions
+below. A restored group is what lets a wallet that melted a coin and then
+lost the device still collect the change: the exchange holds the first melt
+commitment, and a wallet that re-melted with a fresh seed could not reveal
+against it.
.. ts:def:: RefreshStartInc
@@ -2044,8 +2025,8 @@ Refresh session
The melt of one coin of a refresh group.
-Everything the reveal step needs -- the fresh coins' key material included
--- is derived from ``sessionPublicSeed`` together with the old coin and the
+Everything the reveal step needs -- the fresh coins' key material included --
+is derived from ``sessionPublicSeed`` together with the old coin and the
chosen denominations, all of which travel here, so this is the part of a
refresh that has to be backed up.
@@ -2105,21 +2086,20 @@ The refresh could not be completed.
Derived operations: refunds, recoups and denomination losses
------------------------------------------------------------
-The three families below differ from every other one in this document:
-the wallet does not start them, it *learns* about them. A refund is the
+The three families below differ from every other one in this document: the
+wallet does not start them, it *learns* about them. A refund is the
merchant's answer to a refund query, a recoup is forced by an exchange
revoking a denomination, and a denomination loss is what the wallet has to
write off when a denomination expires or is withdrawn from circulation.
Any wallet holding the coins can ask the same question and get the same
-answer, which is what decides how they are backed up: **only a finished
-one travels, and it restores as finished.** Backing up a pending one
-would hand the second device work on an operation it cannot see the whole
-of -- it would go and query a merchant about a refund that is already
-settled on the first device -- and would leave the user looking at an
-operation that is long over elsewhere but "pending" here. A pending one
-is simply not collected, and keeps no origin block, so a later pass offers
-it up once it has finished.
+answer, which is what decides how they are backed up: **only a finished one
+travels, and it restores as finished.** Backing up a pending one would hand
+the second device work on an operation it cannot see the whole of -- it would
+go and query a merchant about a refund that is already settled on the first
+device -- and would leave the user looking at an operation that is long over
+elsewhere but "pending" here. A pending one is simply not collected, and
+keeps no origin block, so a later pass offers it up once it has finished.
Refund
~~~~~~
@@ -2129,8 +2109,8 @@ A refund the merchant granted, as it finally stood.
The refund *items* (one per coin) are deliberately not carried: nothing
outside the refund query itself reads them, the transaction is rendered
entirely from the group, and their identity is the merchant's
-(``coin_pub``/``rtransaction_id``), so a wallet that does query gets the
-same ones back.
+(``coin_pub``/``rtransaction_id``), so a wallet that does query gets the same
+ones back.
.. ts:def:: RefundInc
@@ -2158,8 +2138,8 @@ same ones back.
Merge strategy
++++++++++++++
-Last write wins: the increment describes one finished operation, and there
-is nothing to reconcile field by field.
+Last write wins: the increment describes one finished operation, and there is
+nothing to reconcile field by field.
Recoup
~~~~~~
@@ -2168,9 +2148,9 @@ Coins reclaimed from an exchange that revoked their denomination.
What the recoup *did* to the coins reaches the other wallets as coin
increments; this is what makes the operation itself appear. Its per-coin
-progress is not carried -- it describes a run the other wallet did not
-make -- and a restored recoup is marked finished for every coin, so that
-the second device does not go and re-submit somebody else's recoup.
+progress is not carried -- it describes a run the other wallet did not make --
+and a restored recoup is marked finished for every coin, so that the second
+device does not go and re-submit somebody else's recoup.
.. ts:def:: RecoupInc
@@ -2198,16 +2178,16 @@ Denomination loss
A denomination the wallet had to write off, with the coins it cost.
-Unlike the two above this one is not merely history: until the other
-wallets learn of it they keep the affected coins in their balance, and the
-two devices disagree about how much money the user has. The coins
-themselves carry the same news -- their status becomes ``denom-loss`` --
-and this is what makes the transaction appear.
+Unlike the two above this one is not merely history: until the other wallets
+learn of it they keep the affected coins in their balance, and the two devices
+disagree about how much money the user has. The coins themselves carry the
+same news -- their status becomes ``denom-loss`` -- and this is what makes the
+transaction appear.
-``denomLossEventId`` is **derived from the loss** rather than drawn at
-random. Both wallets notice the same expiry on their own, each updating
-the exchange and seeing the same denominations go; with random identifiers
-the user would end up with the same loss listed twice.
+``denomLossEventId`` is **derived from the loss** rather than drawn at random.
+Both wallets notice the same expiry on their own, each updating the exchange
+and seeing the same denominations go; with random identifiers the user would
+end up with the same loss listed twice.
.. code-block:: text
@@ -2293,33 +2273,32 @@ Backup process
Collecting increments
~~~~~~~~~~~~~~~~~~~~~
-Recording runs inside the very transaction that performs the withdrawal,
-the payment or the deposit, which is what makes wallet state and backup
-state commit together -- and also means that anything the recording throws
-takes that operation down with it. It must therefore be impossible for
-the backup to fail an operation: the eager recording is an *optimisation*,
-not the guarantee. A record whose increment never made it keeps its
-``originBlocks`` unset, which is exactly what the full collection pass
-looks for, so a failure costs a delay and nothing else. Recording, waking
-the cycle and queueing a deletion all log and swallow; the critical-point
-hold fails open.
+Recording runs inside the very transaction that performs the withdrawal, the
+payment or the deposit, which is what makes wallet state and backup state
+commit together -- and also means that anything the recording throws takes
+that operation down with it. It must therefore be impossible for the backup
+to fail an operation: the eager recording is an *optimisation*, not the
+guarantee. A record whose increment never made it keeps its ``originBlocks``
+unset, which is exactly what the full collection pass looks for, so a failure
+costs a delay and nothing else. Recording, waking the cycle and queueing a
+deletion all log and swallow; the critical-point hold fails open.
The same applies to key material the wallet *derives* for an operation. A
reserve key pair comes from the reserve seed, so a seed the wallet cannot
-decode would otherwise block every withdrawal, permanently, since the seed
-is stored. An unusable seed instead falls back to a random reserve key
-pair, which the backup carries as ``reservePriv`` the way it does for
-reserves that predate the seed, and the seed itself is left untouched --
-reserves already derived from it are named by their index, so replacing it
-would make them underivable elsewhere.
-
-Stored key material is checked before it is decoded, because the two
-Crockford base32 decoders a wallet may run on do not agree: the JavaScript
-one ignores trailing padding bits that are not zero, while the native
-(qtart) one rejects the string outright. A value decoded unchecked
-therefore works in a browser extension and throws on a phone. Re-encoding
-the decoded bytes and comparing settles it on either runtime, and is what
-the restore path uses to refuse a malformed seed rather than store one.
+decode would otherwise block every withdrawal, permanently, since the seed is
+stored. An unusable seed instead falls back to a random reserve key pair,
+which the backup carries as ``reservePriv`` the way it does for reserves that
+predate the seed, and the seed itself is left untouched -- reserves already
+derived from it are named by their index, so replacing it would make them
+underivable elsewhere.
+
+Stored key material is checked before it is decoded, because the two Crockford
+base32 decoders a wallet may run on do not agree: the JavaScript one ignores
+trailing padding bits that are not zero, while the native (qtart) one rejects
+the string outright. A value decoded unchecked therefore works in a browser
+extension and throws on a phone. Re-encoding the decoded bytes and comparing
+settles it on either runtime, and is what the restore path uses to refuse a
+malformed seed rather than store one.
Wallet transactions record what they changed by appending increments to a
pending buffer, held in the wallet's backup configuration record. The
@@ -2334,34 +2313,34 @@ increments with, so recording is a no-op rather than an error.
The backup cycle
~~~~~~~~~~~~~~~~
-One cycle takes whatever increments have accumulated, packs them into a
-block, and appends that block to the account's linked list:
+One cycle takes whatever increments have accumulated, packs them into a block,
+and appends that block to the account's linked list:
1. In a single database transaction, move the pending increments out of the
- buffer and into an *in-flight block*, storing its nonce, hash, contents
- and the nonce of the block it is to be appended after.
+ buffer and into an *in-flight block*, storing its nonce, hash, contents and
+ the nonce of the block it is to be appended after.
2. Upload any blobs the block references, then the block itself.
3. Once the provider has acknowledged the block, discard the in-flight block
and advance the pointer to the last acknowledged block.
The hand-over in step 1 is what makes the cycle resilient: the increments are
-never absent from both the buffer and a block. A wallet that dies at any
-point either finds increments still pending, or finds an in-flight block and
-retries it — under its **original nonce**, which the server answers with
-``304 Not modified`` if the upload did in fact land. Increments are thus
-neither lost nor backed up twice, and a cycle that has packed a block always
-retries it before packing new increments, so the linked list stays ordered.
+never absent from both the buffer and a block. A wallet that dies at any point
+either finds increments still pending, or finds an in-flight block and retries
+it — under its **original nonce**, which the server answers with ``304 Not
+modified`` if the upload did in fact land. Increments are thus neither lost
+nor backed up twice, and a cycle that has packed a block always retries it
+before packing new increments, so the linked list stays ordered.
A cycle also pulls the account's linked list before packing new increments,
applying any blocks it has not seen before (see "Restore process" below), so
that new blocks are appended after the current end of the list.
-An account that has not been paid for yet answers every request with
-``402 Payment required``, and only the upload endpoints carry the ``Taler:``
-header with a ``taler://pay/...`` URI. A cycle that is answered this way
-while pulling therefore pushes whatever it has pending, so the payment is
-settled — automatically when the annual fee is zero — and subsequent writes
-are accepted.
+An account that has not been paid for yet answers every request with ``402
+Payment required``, and only the upload endpoints carry the ``Taler:`` header
+with a ``taler://pay/...`` URI. A cycle that is answered this way while
+pulling therefore pushes whatever it has pending, so the payment is settled —
+automatically when the annual fee is zero — and subsequent writes are
+accepted.
Backup schedule
---------------
@@ -2369,113 +2348,109 @@ Backup schedule
A backup runs at *critical points* of wallet operations, and on a schedule
otherwise.
-A critical point is one past which losing the device loses money or user
-data that cannot be reconstructed. The canonical example is a withdrawal:
-coin secrets are derived from the withdrawal group's seed, so a backup is
-triggered once every planchet has been generated and persisted but **before**
-the exchange is asked to sign them. Past that point the exchange considers
-the coins withdrawn while a wallet restored from an older backup could no
-longer reconstruct them.
+A critical point is one past which losing the device loses money or user data
+that cannot be reconstructed. The canonical example is a withdrawal: coin
+secrets are derived from the withdrawal group's seed, so a backup is triggered
+once every planchet has been generated and persisted but **before** the
+exchange is asked to sign them. Past that point the exchange considers the
+coins withdrawn while a wallet restored from an older backup could no longer
+reconstruct them.
A cycle is triggered after the recording transaction commits; if the wallet
-stops before it runs, the increments simply stay pending until the next
-cycle. Independently, a periodic task runs a cycle every hour, covering
-increments whose trigger never fired, e.g. because the wallet was offline
-or the operation has no critical point. A cycle that could not reach the
-provider is retried after five minutes, and one that is waiting for the
-account payment to be prepared after thirty seconds -- the payment is what
-unlocks every upload, so it is worth retrying as soon as the provider's
-merchant backend recovers.
-
-Waking the cycle is not always enough. Past a critical point the wallet
-has already revealed key material to somebody else -- the exchange has
-signed the planchets, the purse exists and can be paid into -- and the
-cycle runs concurrently, so the operation would go ahead regardless.
-Those points therefore *hold*: the task returns to the scheduler and is
-retried, and only proceeds once the pending buffer has reached the
-provider. The hold is skipped when the account is unpaid, since no cycle
-can drain the buffer until the user pays and freezing every such
-transaction would be the worse failure.
+stops before it runs, the increments simply stay pending until the next cycle.
+Independently, a periodic task runs a cycle every hour, covering increments
+whose trigger never fired, e.g. because the wallet was offline or the
+operation has no critical point. A cycle that could not reach the provider is
+retried after five minutes, and one that is waiting for the account payment to
+be prepared after thirty seconds -- the payment is what unlocks every upload,
+so it is worth retrying as soon as the provider's merchant backend recovers.
+
+Waking the cycle is not always enough. Past a critical point the wallet has
+already revealed key material to somebody else -- the exchange has signed the
+planchets, the purse exists and can be paid into -- and the cycle runs
+concurrently, so the operation would go ahead regardless. Those points
+therefore *hold*: the task returns to the scheduler and is retried, and only
+proceeds once the pending buffer has reached the provider. The hold is
+skipped when the account is unpaid, since no cycle can drain the buffer until
+the user pays and freezing every such transaction would be the worse failure.
Each request for a cycle names how much is at stake, and the most urgent
reason asked for since the last cycle that reached the provider is what
decides how hard a *failing* cycle retries:
-* ``irrecoverable-secret`` -- key material a lost device would turn into
- lost money. Retried after fifteen seconds: the transaction that
- produced it is held until the buffer drains, so a longer wait is also
- how long that transaction sits still.
-* ``transaction-milestone`` -- a state the user would notice losing, but
- one that can be reconstructed.
-* ``account-payment`` -- the sync account's own payment moved; nothing of
- the user's is at stake.
+* ``irrecoverable-secret`` -- key material a lost device would turn into lost
+ money. Retried after fifteen seconds: the transaction that produced it is
+ held until the buffer drains, so a longer wait is also how long that
+ transaction sits still.
+* ``transaction-milestone`` -- a state the user would notice losing, but one
+ that can be reconstructed.
+* ``account-payment`` -- the sync account's own payment moved; nothing of the
+ user's is at stake.
-The last two fall back to the ordinary five-minute retry. The urgency is
-not persisted: after a restart the pending increments are still there and
-the critical points ask again on their next retry, so it re-establishes
-itself rather than having to be reconstructed.
+The last two fall back to the ordinary five-minute retry. The urgency is not
+persisted: after a restart the pending increments are still there and the
+critical points ask again on their next retry, so it re-establishes itself
+rather than having to be reconstructed.
Full collection pass
~~~~~~~~~~~~~~~~~~~~
-Eager recording covers every transaction family, but a record can still
-exist that no transaction ever reported: one that predates the backup, or
-one of a kind whose creation path bypasses the record handle. A periodic
-*full collection pass* is the safety net: it walks every record kind the
-backup manages (the ``backupSources`` of ``sources.ts``) and turns the
-records that have never been backed up into "start" increments.
-
-The pass is expensive -- it reads every denomination, exchange, bank
-account and transaction the wallet holds -- so it does not run on every
-cycle. It runs when a watermark, ``lastFullCollection`` in the wallet's
-backup configuration record, is older than 24 hours (or absent, i.e. never
-run). A cycle that woke from a critical point therefore stays cheap while
-still backing up whatever the transactions themselves reported.
-
-A forced cycle (see ``runBackupCycle`` in the wallet-core API below)
-bypasses the watermark and runs the pass regardless. This is the tool for
-developer diagnostics: everything the pass would collect is reported by
-``getBackupDiagnostics`` before the cycle runs, so the two requests
-together show exactly what is waiting to be backed up and what a forced
-cycle would add.
+Eager recording covers every transaction family, but a record can still exist
+that no transaction ever reported: one that predates the backup, or one of a
+kind whose creation path bypasses the record handle. A periodic *full
+collection pass* is the safety net: it walks every record kind the backup
+manages (the ``backupSources`` of ``sources.ts``) and turns the records that
+have never been backed up into "start" increments.
+
+The pass is expensive -- it reads every denomination, exchange, bank account
+and transaction the wallet holds -- so it does not run on every cycle. It
+runs when a watermark, ``lastFullCollection`` in the wallet's backup
+configuration record, is older than 24 hours (or absent, i.e. never run). A
+cycle that woke from a critical point therefore stays cheap while still
+backing up whatever the transactions themselves reported.
+
+A forced cycle (see ``runBackupCycle`` in the wallet-core API below) bypasses
+the watermark and runs the pass regardless. This is the tool for developer
+diagnostics: everything the pass would collect is reported by
+``getBackupDiagnostics`` before the cycle runs, so the two requests together
+show exactly what is waiting to be backed up and what a forced cycle would
+add.
Restore process
---------------
-Restoring a wallet on a (fresh) device is the pull half of the backup
-cycle, driven by a recovery document from ``getBackupRecovery``:
+Restoring a wallet on a (fresh) device is the pull half of the backup cycle,
+driven by a recovery document from ``getBackupRecovery``:
-1. ``loadBackupRecovery`` installs the recovery's root key and providers,
- and drops the wallet's own block pointers, so the device starts from
- nothing.
+1. ``loadBackupRecovery`` installs the recovery's root key and providers, and
+ drops the wallet's own block pointers, so the device starts from nothing.
2. Once the user activates a recovered provider (``addBackupProvider`` with
- ``activate``), the backup cycle downloads the account's linked
- list, decodes each block it has not seen before, CRDT-applies its
- increments to the local database -- recording the block's nonce in the
- ``originBlocks`` of every record it touched -- and stores the blocks
- locally.
+ ``activate``), the backup cycle downloads the account's linked list,
+ decodes each block it has not seen before, CRDT-applies its increments to
+ the local database -- recording the block's nonce in the ``originBlocks``
+ of every record it touched -- and stores the blocks locally.
Because the same root key derives the same per-provider account keys, a
recovering wallet sees exactly the blocks any other wallet in the group
-uploaded and applies them with the same merge rules, so all devices
-converge on the same state.
+uploaded and applies them with the same merge rules, so all devices converge
+on the same state.
Restore schedule
----------------
-Restoring happens on demand: it starts when a recovery document is loaded
-and the recovered provider is activated. Afterwards the restored wallet is
-kept up to date by the same periodic backup task as every other wallet --
-the pull half runs on every cycle, so changes made by other devices are
-picked up at the cycle interval.
+Restoring happens on demand: it starts when a recovery document is loaded and
+the recovered provider is activated. Afterwards the restored wallet is kept
+up to date by the same periodic backup task as every other wallet -- the pull
+half runs on every cycle, so changes made by other devices are picked up at
+the cycle interval.
Wallet-core API
---------------
Backup providers and the wallet's backup key are managed through the
wallet-core API. All requests below are available on every platform. The
-request handlers described here are implemented; the collection and
-scheduling mechanisms described above drive them.
+request handlers described here are implemented; the collection and scheduling
+mechanisms described above drive them.
.. ts:def:: AddBackupProviderRequest
@@ -2489,30 +2464,29 @@ scheduling mechanisms described above drive them.
activate?: boolean;
}
-The cycle never *waits* for the account payment. Downloading the
-provider's proposal and paying it are the purchase's own task, so the
-cycle only ever looks at where that purchase has got to -- confirming it
-when it is waiting for a decision, and otherwise leaving it alone -- and
-comes back when the purchase transitions, or on its retry interval. Every
-step is therefore idempotent and survives a wallet that stops in the
-middle.
-
-``addBackupProvider`` registers a sync server: it stores a provider record
-and -- when ``activate`` is set -- makes it the active sync target and
-wakes the backup cycle. The request itself does not talk to the provider
-and returns as soon as the record is written; an unreachable provider, or
-one that is not a sync server, therefore shows up as a failing (and
-retrying) cycle rather than as an error from this request.
-
-The first cycle is what learns the provider's terms (it fetches
-``/config`` and reports the result with the ``terms-fetched`` phase of the
-``backup-status`` notification) and what settles the account payment: a
-sync account only exists once it has been paid for, and the server rejects
-every upload (even at a zero annual fee) until then. A zero-fee account is
-paid automatically; any other account produces a payment transaction that
-the user confirms from the wallet, and the ``payment-required`` phase of
-the notification carries its ``taler://pay/...`` URI. Clients follow all
-of this through the notifications, not through this request's response:
+The cycle never *waits* for the account payment. Downloading the provider's
+proposal and paying it are the purchase's own task, so the cycle only ever
+looks at where that purchase has got to -- confirming it when it is waiting
+for a decision, and otherwise leaving it alone -- and comes back when the
+purchase transitions, or on its retry interval. Every step is therefore
+idempotent and survives a wallet that stops in the middle.
+
+``addBackupProvider`` registers a sync server: it stores a provider record and
+-- when ``activate`` is set -- makes it the active sync target and wakes the
+backup cycle. The request itself does not talk to the provider and returns as
+soon as the record is written; an unreachable provider, or one that is not a
+sync server, therefore shows up as a failing (and retrying) cycle rather than
+as an error from this request.
+
+The first cycle is what learns the provider's terms (it fetches ``/config``
+and reports the result with the ``terms-fetched`` phase of the
+``backup-status`` notification) and what settles the account payment: a sync
+account only exists once it has been paid for, and the server rejects every
+upload (even at a zero annual fee) until then. A zero-fee account is paid
+automatically; any other account produces a payment transaction that the user
+confirms from the wallet, and the ``payment-required`` phase of the
+notification carries its ``taler://pay/...`` URI. Clients follow all of this
+through the notifications, not through this request's response:
.. ts:def:: AddBackupProviderResponse
@@ -2596,8 +2570,8 @@ gotten, based on the payment transaction the wallet opened for it:
newTerms: BackupProviderTerms };
``getBackupRecovery`` returns the secret needed to restore the wallet on
-another device, along with the providers to fetch the blocks from. It is
-what the user backs up out of band, and what a restoring wallet is fed.
+another device, along with the providers to fetch the blocks from. It is what
+the user backs up out of band, and what a restoring wallet is fed.
.. ts:def:: BackupRecovery
@@ -2614,8 +2588,8 @@ what the user backs up out of band, and what a restoring wallet is fed.
paperKey?: string;
}
-The paper key is line-oriented, so that a line is the unit to copy, parse
-and transpose:
+The paper key is line-oriented, so that a line is the unit to copy, parse and
+transpose:
.. code-block:: text
@@ -2625,31 +2599,31 @@ and transpose:
PROVIDER: https://sync.example.com/
URI: taler://restore/... (the machine-readable form, LSD0006 5.7)
-The ``URI`` line is the canonical machine form: a device restoring from a
-scan or a file needs nothing but that line. The ``KEY`` / ``PROVIDER``
-lines are the human form, and the checksum catches a transcription error
-before it silently restores a different -- empty -- sync group.
-
-``loadBackupRecovery`` feeds such a recovery document into a wallet, which
-is how a second (or replacing) device joins the sync group. The wallet
-adopts the recovery's root key -- the key every per-provider account key is
-derived from, so adopting it *is* what joining the group means -- and adds
-the recovery's providers. There is no "keep my own key" variant: a wallet
-that kept its own key would derive different account keys and so would not
-be in the group at all.
-
-Adopting another root key also detaches the wallet from the group it was
-in: the blocks it stored are encrypted under a key it no longer has, and
-the ``originBlocks`` lists that reference them are meaningless. Both are
-cleared. That deliberately leaves the wallet's own records looking "never
-backed up", which is what they are with respect to the group being joined:
-the full collection pass then offers them up, instead of the pull's
-"deleted iff absent from all origin blocks" sweep removing them for not
-appearing in the new group's linked list.
-
-The providers are registered but not activated; the client activates one
-with ``addBackupProvider`` (``activate: true``), and that is what starts
-the cycle which pulls the backup.
+The ``URI`` line is the canonical machine form: a device restoring from a scan
+or a file needs nothing but that line. The ``KEY`` / ``PROVIDER`` lines are
+the human form, and the checksum catches a transcription error before it
+silently restores a different -- empty -- sync group.
+
+``loadBackupRecovery`` feeds such a recovery document into a wallet, which is
+how a second (or replacing) device joins the sync group. The wallet adopts
+the recovery's root key -- the key every per-provider account key is derived
+from, so adopting it *is* what joining the group means -- and adds the
+recovery's providers. There is no "keep my own key" variant: a wallet that
+kept its own key would derive different account keys and so would not be in
+the group at all.
+
+Adopting another root key also detaches the wallet from the group it was in:
+the blocks it stored are encrypted under a key it no longer has, and the
+``originBlocks`` lists that reference them are meaningless. Both are cleared.
+That deliberately leaves the wallet's own records looking "never backed up",
+which is what they are with respect to the group being joined: the full
+collection pass then offers them up, instead of the pull's "deleted iff absent
+from all origin blocks" sweep removing them for not appearing in the new
+group's linked list.
+
+The providers are registered but not activated; the client activates one with
+``addBackupProvider`` (``activate: true``), and that is what starts the cycle
+which pulls the backup.
.. ts:def:: RecoveryLoadRequest
@@ -2661,12 +2635,11 @@ the cycle which pulls the backup.
periodic task. This is the dedicated "back up now" request; earlier
implementations triggered a cycle by re-adding the active provider.
-The request only *wakes* the cycle and returns an empty object
-immediately: the cycle runs asynchronously (and is serialized against any
-other cycle), reports its progress and outcome through the
-``backup-status`` notifications, and persists its statistics for
-``getBackupDiagnostics``. Clients track the cycle through those, not
-through this request's response.
+The request only *wakes* the cycle and returns an empty object immediately:
+the cycle runs asynchronously (and is serialized against any other cycle),
+reports its progress and outcome through the ``backup-status`` notifications,
+and persists its statistics for ``getBackupDiagnostics``. Clients track the
+cycle through those, not through this request's response.
.. ts:def:: RunBackupCycleRequest
@@ -2678,11 +2651,11 @@ through this request's response.
force?: boolean;
}
-The statistics are persisted by the wallet after every cycle,
-whatever triggered it, and are reported by ``getBackupDiagnostics`` as
-the "last cycle" outcome. The ``outcome`` field says how the cycle
-ended: ``"ok"`` (including idle cycles with nothing to push),
-``"payment-required"`` (the account is unpaid) or ``"error"``.
+The statistics are persisted by the wallet after every cycle, whatever
+triggered it, and are reported by ``getBackupDiagnostics`` as the "last cycle"
+outcome. The ``outcome`` field says how the cycle ended: ``"ok"`` (including
+idle cycles with nothing to push), ``"payment-required"`` (the account is
+unpaid) or ``"error"``.
.. ts:def:: BackupCycleStats
@@ -2717,11 +2690,11 @@ ended: ``"ok"`` (including idle cycles with nothing to push),
};
}
-``getBackupDiagnostics`` reports aggregated statistics about what the
-backup holds: what a cycle would back up right now, and what the last
-cycle restored. It is intended for developer tooling; user interfaces
-are expected to only expose it in developer mode, but the request itself
-is harmless and available on every platform.
+``getBackupDiagnostics`` reports aggregated statistics about what the backup
+holds: what a cycle would back up right now, and what the last cycle restored.
+It is intended for developer tooling; user interfaces are expected to only
+expose it in developer mode, but the request itself is harmless and available
+on every platform.
.. ts:def:: BackupDiagnostics
@@ -2764,8 +2737,8 @@ Backup notifications
The wallet pushes a ``backup-status`` notification to its clients
(``NotificationType.BackupStatus``) as a backup cycle runs, through the
regular wallet notification listener. Clients should use it instead of
-polling ``getBackupInfo`` to track a cycle: it reports the phase the cycle
-is in and, on the terminal phases, the outcome and the relevant counters.
+polling ``getBackupInfo`` to track a cycle: it reports the phase the cycle is
+in and, on the terminal phases, the outcome and the relevant counters.
.. ts:def:: BackupStatusNotification
@@ -2794,49 +2767,48 @@ is in and, on the terminal phases, the outcome and the relevant counters.
}
The wallet emits ``started`` when a cycle begins, ``pulling`` before the
-linked list is fetched, ``pushing`` with the increment count before the
-packed block (and its blobs) is uploaded, ``terms-fetched`` when it has
-read the provider's ``/config`` (which is where a newly added provider's
-terms come from, so a client showing them refreshes on it), and a terminal
-phase when the cycle ends:
-
-* ``completed`` -- the cycle ran without error and without requiring
- payment (``pulledBlocks`` / ``pushedBlockNonce`` carry the counters);
-* ``payment-required`` -- the account is unpaid; a payment transaction
- may already have been prepared, and the UI should take the user to it;
-* ``error`` -- the cycle failed (with ``error`` as the reason); the
- wallet retries on its own schedule, so the notification is only for
- the user interface. The reason is also persisted, and reported by
- ``getBackupInfo`` as the active provider's ``lastError``, so a client
- that was not listening at the time still sees it.
+linked list is fetched, ``pushing`` with the increment count before the packed
+block (and its blobs) is uploaded, ``terms-fetched`` when it has read the
+provider's ``/config`` (which is where a newly added provider's terms come
+from, so a client showing them refreshes on it), and a terminal phase when the
+cycle ends:
+
+* ``completed`` -- the cycle ran without error and without requiring payment
+ (``pulledBlocks`` / ``pushedBlockNonce`` carry the counters);
+* ``payment-required`` -- the account is unpaid; a payment transaction may
+ already have been prepared, and the UI should take the user to it;
+* ``error`` -- the cycle failed (with ``error`` as the reason); the wallet
+ retries on its own schedule, so the notification is only for the user
+ interface. The reason is also persisted, and reported by ``getBackupInfo``
+ as the active provider's ``lastError``, so a client that was not listening
+ at the time still sees it.
A cycle whose pull applied anything additionally emits a ``balance-change``
-notification. The apply path writes coins and transactions straight into
-the database, so none of the transaction state machines report them; the
-``backup-status`` notification says a cycle finished, not that the
-wallet's contents changed, and a client that refreshed on it alone would
-show a restoring wallet as empty until something else happened.
+notification. The apply path writes coins and transactions straight into the
+database, so none of the transaction state machines report them; the
+``backup-status`` notification says a cycle finished, not that the wallet's
+contents changed, and a client that refreshed on it alone would show a
+restoring wallet as empty until something else happened.
-An earlier ``backup-error`` notification type (``BackupOperationError``)
-was part of a legacy backup proof of concept and has been removed in
-favor of the ``error`` phase of ``backup-status``.
+An earlier ``backup-error`` notification type (``BackupOperationError``) was
+part of a legacy backup proof of concept and has been removed in favor of the
+``error`` phase of ``backup-status``.
.. _limitations:
Limitations
===========
-While the design minimizes the metadata that the backup service is exposed
-to, some leakage is inherent to the protocol and cannot be avoided in a
-practical way. The service necessarily learns how many blocks and blobs an
-account holds, how much data is uploaded and downloaded, and when these
-operations take place. Kilobyte padding ensures that the size of an
-individual block or blob reveals little about the contents it carries, but it
-cannot conceal the overall volume of activity, the number of operations
-performed, nor their distribution in time. In particular, the number of
-blocks in an account grows with every performed operation, so the block count
-itself is a lower bound on the amount of activity that cannot be disguised by
-padding.
+While the design minimizes the metadata that the backup service is exposed to,
+some leakage is inherent to the protocol and cannot be avoided in a practical
+way. The service necessarily learns how many blocks and blobs an account
+holds, how much data is uploaded and downloaded, and when these operations
+take place. Kilobyte padding ensures that the size of an individual block or
+blob reveals little about the contents it carries, but it cannot conceal the
+overall volume of activity, the number of operations performed, nor their
+distribution in time. In particular, the number of blocks in an account grows
+with every performed operation, so the block count itself is a lower bound on
+the amount of activity that cannot be disguised by padding.
Timing patterns are particularly hard to hide. Backups run at critical points
of wallet operations and on a periodic schedule, and some of these critical
@@ -2858,68 +2830,65 @@ Definition of done
* [ ] Design incremental sync.
* [x] Design backup/restore schedules.
* [x] Design wallet-core API.
-* [x] Wallet-core implementation. The machinery -- block and blob
- encoding, CRDT merge, the sync protocol client and its signatures,
- increment collection, the scheduled backup cycle with its
- pull/merge/apply half, the API request handlers, the account payment
- flow, and item deletion (retro-redaction of the ``originBlocks`` plus
- the pull-side "deleted iff absent from all origin blocks" sweep) -- is
- done, and so is **every increment family in this document**: the
- exchange, global-trust, bank-account, donau and denomination entities;
- the reserve family (``set-reserve-seed`` / ``add-reserve``, with the
- seed-derived key pairs and the ``reservePriv`` fallback for reserves
- that predate the seed), which is what makes a restored coin recoupable;
- the withdrawal, deposit, merchant-payment, peer-push-credit,
- peer-push-debit, peer-pull-debit and peer-pull-credit transaction
- families; the refresh family, whose per-coin session seed lets a
- restored wallet finish a melt instead of losing the change; and the coin
- and token families, which carry the per-record key material the wallet
- database stores (the seed-derived modelling of earlier drafts is gone
- from both).
+* [x] Wallet-core implementation. The machinery -- block and blob encoding,
+ CRDT merge, the sync protocol client and its signatures, increment
+ collection, the scheduled backup cycle with its pull/merge/apply half, the
+ API request handlers, the account payment flow, and item deletion
+ (retro-redaction of the ``originBlocks`` plus the pull-side "deleted iff
+ absent from all origin blocks" sweep) -- is done, and so is **every
+ increment family in this document**: the exchange, global-trust,
+ bank-account, donau and denomination entities; the reserve family
+ (``set-reserve-seed`` / ``add-reserve``, with the seed-derived key pairs and
+ the ``reservePriv`` fallback for reserves that predate the seed), which is
+ what makes a restored coin recoupable; the withdrawal, deposit,
+ merchant-payment, peer-push-credit, peer-push-debit, peer-pull-debit and
+ peer-pull-credit transaction families; the refresh family, whose per-coin
+ session seed lets a restored wallet finish a melt instead of losing the
+ change; and the coin and token families, which carry the per-record key
+ material the wallet database stores (the seed-derived modelling of earlier
+ drafts is gone from both).
Contract terms travel as blobs -- uploaded ahead of the blocks that
- reference them, with their reference counts adjusted, and fetched and
- stored back into the contract-terms store on the pull side; a
- transaction whose terms are not available is shown in a reduced form
- instead of failing the transaction listing. Restoring a coin recomputes
- the coin-availability rows, so a restored wallet shows the same balance
- as the wallet that made the backup, and a restored wallet can continue a
- pending withdrawal (only an expired bank operation cannot be resumed).
- ``runBackupCycle`` and ``getBackupDiagnostics``, the per-cycle
- statistics, the forced full-collection pass and the ``backup-status``
- notifications are all in place, on both database backends: the native
- (sqlite) schema stores the backup providers and blocks and the
- ``originBlocks`` of every backup-managed record, and a wallet migrating
- from the IndexedDB backend carries all three across.
-
- The three *derived* families -- refund, recoup and denomination loss --
- are implemented as finished facts, and every change to whether a coin
- counts towards the balance (spend, refresh, recoup, denomination loss,
- suspend) is reported as a coin increment, so two wallets converge on the
- same balance rather than only on the same coins. A transaction can no
- longer be taken back out of a terminal state by an increment describing
- an older view of it.
+ reference them, with their reference counts adjusted, and fetched and stored
+ back into the contract-terms store on the pull side; a transaction whose
+ terms are not available is shown in a reduced form instead of failing the
+ transaction listing. Restoring a coin recomputes the coin-availability
+ rows, so a restored wallet shows the same balance as the wallet that made
+ the backup, and a restored wallet can continue a pending withdrawal (only an
+ expired bank operation cannot be resumed). ``runBackupCycle`` and
+ ``getBackupDiagnostics``, the per-cycle statistics, the forced
+ full-collection pass and the ``backup-status`` notifications are all in
+ place, on both database backends: the native (sqlite) schema stores the
+ backup providers and blocks and the ``originBlocks`` of every backup-managed
+ record, and a wallet migrating from the IndexedDB backend carries all three
+ across.
+
+ The three *derived* families -- refund, recoup and denomination loss -- are
+ implemented as finished facts, and every change to whether a coin counts
+ towards the balance (spend, refresh, recoup, denomination loss, suspend) is
+ reported as a coin increment, so two wallets converge on the same balance
+ rather than only on the same coins. A transaction can no longer be taken
+ back out of a terminal state by an increment describing an older view of it.
Known gaps, none of which loses money: refund *items* are not carried
- (nothing outside the refund query reads them, and the merchant hands
- back the same ones); the exchange entries and peer-pull-credit records
- do not restore their ``currentMergeReserveRowId`` pointer, since it is a
- row id local to one database; recoup transactions are backed up and
- restored but the wallet does not yet render them as transactions; and a
- wallet cannot join a sync group written by a *newer* wallet -- it
- refuses the blocks rather than re-uploading a truncated view of them.
+ (nothing outside the refund query reads them, and the merchant hands back
+ the same ones); the exchange entries and peer-pull-credit records do not
+ restore their ``currentMergeReserveRowId`` pointer, since it is a row id
+ local to one database; recoup transactions are backed up and restored but
+ the wallet does not yet render them as transactions; and a wallet cannot
+ join a sync group written by a *newer* wallet -- it refuses the blocks
+ rather than re-uploading a truncated view of them.
* [x] Design sync API (+ auth).
* [ ] Server-side implementation (partial: block GET/POST/PUT/DELETE, object
store GET/POST with reference counting, /config and payments done;
reconciliation mechanism still missing).
-* [x] UI/UX for backup and sync, in the Android wallet: adding and removing
- a provider, the account payment prompt, the recovery as a QR code and as
- a paper key (written down or saved to a file) with its import
- counterpart, "back up now" through ``runBackupCycle`` with a
- force-full-backup control and a diagnostics card in developer mode, and
- a progress display driven by the ``backup-status`` notifications. The
- web extension shows the cycle in its wallet-activity view, but has no
- provider management user interface yet.
+* [x] UI/UX for backup and sync, in the Android wallet: adding and removing a
+ provider, the account payment prompt, the recovery as a QR code and as a
+ paper key (written down or saved to a file) with its import counterpart,
+ "back up now" through ``runBackupCycle`` with a force-full-backup control
+ and a diagnostics card in developer mode, and a progress display driven by
+ the ``backup-status`` notifications. The web extension shows the cycle in
+ its wallet-activity view, but has no provider management user interface yet.
Alternatives
============