api-sync.rst (18717B)
1 .. 2 This file is part of GNU TALER. 3 Copyright (C) 2018-2021 Taler Systems SA 4 5 TALER is free software; you can redistribute it and/or modify it under the 6 terms of the GNU Affero General Public License as published by the Free Software 7 Foundation; either version 2.1, or (at your option) any later version. 8 9 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 10 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 11 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 12 13 You should have received a copy of the GNU Affero General Public License along with 14 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 15 16 @author Christian Grothoff 17 18 .. _sync-api: 19 20 ====================================== 21 Backup and Synchronization RESTful API 22 ====================================== 23 24 The backup and synchronization service uses an EdDSA key 25 to identify the "account" of the user. The key is Crockford 26 Base32-encoded in the URI to access the data and used to sign requests 27 as well as to encrypt the contents (see below). These signatures are 28 provided in detached form as HTTP headers. 29 30 Once the user activates backup or synchronization, the client should 31 display the key as a QR code as well as in text format together 32 with the synchronization service's URL and ask the user to print this 33 key material and keep it safe. 34 35 The actual format of the backup is not relevant for the 36 backup and synchronization service, as the service must only ever see 37 a padded and encrypted version of the data. 38 39 However, there are a few general rules that will apply to 40 any version of the backup. Still, except for the 41 32-byte minimum upload size, the synchronization service 42 itself cannot not enforce these rules. 43 44 * First, the database should be compressed (i.e. gzip), then 45 padded to a power of 2 in kilobytes or a multiple of 46 megabytes, then encrypted and finally protected with 47 an HDKF. 48 * The encryption should use an SHA-512 nonce which 49 is prefixed to the actual database, and combined with 50 the master key to create the encryption symmetric secret. 51 With every revision of the backup (but only real 52 revisions or merge operations), a fresh nonce must be 53 used to ensure that the symmetric secret differs every 54 time. HKDFs are used to derive symmetric key material 55 for authenticated encryption (encrypt-then-mac or a 56 modern AEAD-cipher like Keccak). Given that AES is more 57 easily available and will likey increase the code of 58 the wallet less, AES plus a SHA-512 HMAC should suffice 59 for now. 60 * The client must enable merging databases in a way that is 61 associative and commutative. For most activities, this implies 62 merging lists, applying expirations, dropping duplicates and 63 sorting the result. For deletions (operations by which the user 64 removed records prior to their scheduled expiration), it means 65 keeping a summarizing log of all deletion operations and applying 66 the deletions after each merge. A summarizing log of a deletion 67 operation would combine two deletion operations of the form 68 "delete all transactions smaller than amount X before time T" and 69 "delete all transactions smaller than amount Y before time T" 70 into "delete all transactions smaller than amount max(X,Y) before 71 time T". Similar summarizations should be applied to all 72 deletion operations supported by the client. Deletion operations 73 themselves are associated with an expiration time reflecting the 74 expiration of the longest lasting record that they explicitly 75 deleted. 76 Purchases do not have an expiration time, thus they create 77 a challenge if an indivdiual purchase is deleted. Thus, when 78 an individual purchase is deleted, the client is to keep track 79 of the deletion with a deletion record. The deletion record 80 still includes the purchase amount and purchase date. Thus, 81 when purchases are deleted "in bulk" in a way that would have 82 covered the individual deletion, such deletion records may 83 still be subsumed by a more general deletion clause. In addition 84 to the date and amount, the deletion record should only contain 85 a salted hash of the original purchase record's primary key, 86 so as to minimize information leakage. 87 * The database should contain a "last modified" timestamp to ensure 88 we do not go backwards in time if the synchronization service is 89 malicious. Merging two databases means taking the max of the 90 "last modified" timestamps, not setting it to the current time. 91 The client should reject a "fast forward" database update if the 92 result would imply going back in time. If the client receives a 93 database with a timestamp into the future, it must still 94 increment it by the smallest possible amount when uploading an 95 update. 96 * In general, the merge operation should be implemented in such a way 97 that it deals gracefully with adversarial devices from rogue 98 devices connected to the same account. 99 100 It is assumed that the synchronization service is only ever accessed 101 over TLS, and that the synchronization service is trusted to not build 102 user's location profiles by linking client IP addresses and client 103 keys. 104 105 .. contents:: Table of Contents 106 :local: 107 108 109 --------------- 110 Version History 111 --------------- 112 113 The current protocol version is **v2**. 114 115 * No components use Sync at this point, so there are no dependencies. 116 117 **Version history:** 118 119 * ``v2``: add the ``implementation`` field to ``/config`` 120 121 **Upcoming versions:** 122 123 * ``vBLOBS``: changes for blob backups 124 * ``vBACKUP``: changes for incremental backups 125 126 **Ideas for future version:** 127 128 * ``vXXX``: marker for features not yet targeted for release 129 130 .. include:: tos.rst 131 132 ----------------------- 133 Receiving Configuration 134 ----------------------- 135 136 .. http:get:: /config 137 138 Obtain the key configuration settings of the storage service. 139 140 **Response:** 141 142 Returns a `SyncTermsOfServiceResponse`. 143 144 .. ts:def:: SyncTermsOfServiceResponse 145 146 interface SyncTermsOfServiceResponse { 147 // Name of the service 148 name: "sync"; 149 150 // Maximum backup size supported. 151 storage_limit_in_megabytes: Integer; 152 153 // Fee for an account, per year. 154 annual_fee: Amount; 155 156 // Maximum liability of the provider in case of data loss. 157 liability_limit: Amount; 158 159 // libtool-style representation of the Sync protocol version, see 160 // https://www.gnu.org/software/libtool/manual/html_node/Versioning.html#Versioning 161 // The format is "current:revision:age". 162 version: string; 163 164 // URN of the implementation (needed to interpret 'revision' in version). 165 // @since **v2**, may become mandatory in the future. 166 implementation?: string; 167 168 } 169 170 .. _sync: 171 172 ---------------------- 173 Recovering Backup Data 174 ---------------------- 175 176 .. http:get:: /backups/${ACCOUNT-KEY} 177 178 Download latest version of the backup. 179 The returned headers must include "Etags" based on 180 the hash of the (encrypted) database. The server must 181 check the client's caching headers and only return the 182 full database if it has changed since the last request 183 of the client. 184 185 This method is generally only performed once per device 186 when the private key and URL of a synchronization service are 187 first given to the client on the respective device. Once a 188 client has made a backup, it should always use the POST method. 189 190 A signature is not required, as (1) the account-key should 191 be reasonably private and thus unauthorized users should not 192 know how to produce the correct request, and (2) the 193 information returned is encrypted to the private key anyway 194 and thus virtually useless even to an attacker who somehow 195 managed to obtain the public key. 196 197 **Response** 198 199 :http:statuscode:`200 OK`: 200 The body contains the current version of the backup 201 as known to the server. 202 203 :http:statuscode:`204 No content`: 204 This is a fresh account, no previous backup data exists at 205 the server. 206 207 :http:statuscode:`304 Not modified`: 208 The version available at the server is identical to that 209 specified in the ``If-None-Match`` header. 210 211 :http:statuscode:`404 Not found`: 212 The backup service is unaware of a matching account. 213 214 :http:statuscode:`410 Gone`: 215 The backup service has closed operations. The body will 216 contain the latest version still available at the server. 217 The body may be empty if no version is available. 218 The user should be urged to find another provider. 219 220 :http:statuscode:`429 Too many requests`: 221 This account has exceeded thresholds for the number of 222 requests. The client should try again later, and may want 223 to decrease its synchronization frequency. 224 225 .. note:: 226 227 "200 OK" responses include an HTTP header 228 "Sync-Signature" with the signature of the 229 client from the original upload, and an 230 "Sync-Previous" with the version that was 231 being updated (unless this is the first revision). 232 "Sync-Previous" is only given to enable 233 signature validation. 234 235 236 --------------------- 237 Uploading Backup Data 238 --------------------- 239 240 .. http:post:: /backups/${ACCOUNT-KEY} 241 242 Upload a new version of the account's database, or download the 243 latest version. The request SHOULD include the ``Expect: 100 Continue`` 244 header. The client then SHOULD wait for ``100 Continue`` before proceeding 245 with the upload, regardless of the size of the upload. 246 247 **Request** 248 249 The request must include a ``If-Match`` header indicating the latest 250 version of the account's database known to the client. If the server 251 knows a more recent version, it will respond with a ``409 conflict`` 252 and return the server's version in the response. The client must 253 then merge the two versions before retrying the upload. Note that 254 a ``409 Conflict`` response will typically be given before the upload, 255 (instead of ``100 continue``), but may also be given after the upload, 256 for example due to concurrent activities from other accounts on the 257 same account! 258 259 The request MUST also include an "Sync-Signature" signing 260 the ``If-Match`` SHA-512 value and the SHA-512 hash of the body with 261 the account private key. 262 263 Finally, the SHA-512 hash of the body MUST also be given in an 264 ``If-None-Match`` header of the request (so that the signature can be verified 265 before the upload is allowed to proceed). 266 267 The uploaded body must have at least 32 bytes of payload (see 268 suggested upload format beginning with an ephemeral key). 269 270 :query paying: 271 Optional argument providing an order identifier. 272 The client is promising that it is already paying on a 273 related order. This will cause the 274 server to delay processing until the respective payment 275 has arrived (if the operation requires a payment). Useful 276 if the server previously returned a ``402 Payment required`` 277 and the client wants to proceed as soon as the payment 278 went through. 279 :query pay: 280 Optional argument, any non-empty value will do, 281 suggested is ``y`` for ``yes``. 282 The client insists on making a payment for the respective 283 account, even if this is not yet required. The server 284 will respond with a ``402 Payment required``, but only 285 if the rest of the request is well-formed (account 286 signature must match). Clients that do not actually 287 intend to make a new upload but that only want to pay 288 may attempt to upload the latest backup again, as this 289 option will be checked before the ``304 Not modified`` 290 case. 291 :query fresh: 292 Optional argument, any non-empty value will do, 293 suggested is ``y`` for ``yes``. 294 The client insists on a fresh order to be generated, say 295 because the one returned before was claimed (but not paid) 296 by another wallet. 297 298 299 **Response** 300 301 :http:statuscode:`204 No content`: 302 The transfer was successful, and the server has registered 303 the new version. 304 305 :http:statuscode:`304 Not modified`: 306 The server is already aware of this version of the client. 307 Returned before ``100 continue`` to avoid upload. 308 FIXME: Might be better to use ``412 Precondition Failed`` here 309 in the future! 310 311 :http:statuscode:`400 Bad request`: 312 Most likely, the uploaded body is too short (less than 32 bytes). 313 314 :http:statuscode:`402 Payment required`: 315 The synchronization service requires payment before the 316 account can continue to be used. The fulfillment URL 317 should be the ``/$ACCOUNT-KEY`` URL, but can be safely ignored 318 by the client. The contract should be shown to the user 319 in the canonical dialog, possibly in a fresh tab. 320 321 :http:statuscode:`403 Forbidden`: 322 The signature is invalid or missing (or body does not match). 323 324 :http:statuscode:`409 Conflict`: 325 The server has a more recent version than what is given 326 in ``If-Match``. The more recent version is returned. The 327 client should merge the two versions and retry using the 328 given response's "E-Tag" in the next attempt in ``If-Match``. 329 330 :http:statuscode:`410 Gone`: 331 The backup service has closed operations. The body will 332 contain the latest version still available at the server. 333 The body may be empty if no version is available. 334 The user should be urged to find another provider. 335 336 :http:statuscode:`411 Length required`: 337 The client must specify the ``Content-length`` header before 338 attempting upload. While technically optional by the 339 HTTP specification, the synchronization service may require 340 the client to provide the length upfront. 341 342 :http:statuscode:`413 Request entity too large`: 343 The requested upload exceeds the quota for the type of 344 account. The client should suggest to the user to 345 migrate to another backup and synchronization service 346 (like with ``410 Gone``). 347 348 :http:statuscode:`429 Too many requests`: 349 This account has exceeded daily thresholds for the number of 350 requests. The client should try again later, and may want 351 to decrease its synchronization frequency. 352 353 .. note:: 354 355 Responses with a body include an HTTP header 356 "Sync-Signature" with the signature of the 357 client from the original upload, and an 358 "If-Match" with the version that is 359 being updated (unless this is the first revision). 360 361 362 363 --------------------------- 364 Special constraints for Tor 365 --------------------------- 366 367 We might introduce the notion of a "constraint" into the client's 368 database that states that the database is a "Tor wallet". Then, 369 synchronizing a "Tor-wallet" with a non-Tor wallet should trigger a 370 stern warning and require user confirmation (as otherwise 371 cross-browser synchronization may weaken the security of Tor browser 372 users). 373 374 375 ------------------------------------------------ 376 Discovery of backup and synchronization services 377 ------------------------------------------------ 378 379 The client should keep a list of "default" synchronization services 380 per currency (by the currency the synchronization service accepts 381 for payment). If a synchronization service is entirely free, it 382 should be kept in a special list that is always available. 383 384 Extending (or shortening) the list of synchronization services should 385 be possible using the same mechanism that is used to add/remove 386 auditors or exchanges. 387 388 The client should urge the user to make use of a synchronization 389 service upon first withdrawal, suggesting one that is free or 390 accepts payment in the respective currency. If none is available, 391 the client should warn the user about the lack of available 392 backups and synchronization and suggest to the user to find a 393 reasonable service. Once a synchronization service is selected, 394 the client should urge the user to print the respective key 395 material. 396 397 When the client starts the first time on a new device, it should 398 ask the user if he wants to synchronize with an existing client, 399 and if so, ask the user to enter the respective key and the 400 (base) URL of the synchronization service. 401 402 403 ------------------------- 404 Synchronization frequency 405 ------------------------- 406 407 Generally, the client should attempt to synchronize at a randomized 408 time interval between 30 and 300 seconds of being started, unless it 409 already synchronized less than two hours ago already. Afterwards, 410 the client should synchronize every two hours, or after purchases 411 exceed 5 percent of the last bulk amount that the user withdrew. 412 In all cases the exact time of synchronization should be randomized 413 between 30 and 300 seconds of the specified event, both to minimize 414 obvious correlations and to spread the load. 415 416 If the two hour frequency would exceed half of the rate budget offered 417 by the synchronization provider, it should be reduced to remain below 418 that threshold. 419 420 421 ------------------------------- 422 Synchronization user experience 423 ------------------------------- 424 425 The menu should include three entries for synchronization: 426 427 * "synchronize" to manually trigger synchronization, 428 insensitive if no synchronization provider is available 429 * "export backup configuration" to re-display (and possibly 430 print) the synchronization and backup parameters (URL and 431 private key), insensitive if no synchronization 432 provider is available, and 433 * "import backup configuration" to: 434 435 * import another devices' synchronization options 436 (by specifying URL and private key, or possibly 437 scanning a QR code), or 438 * select a synchronization provider from the list, 439 including manual specification of a URL; here 440 confirmation should only be possible if the provider 441 is free or can be paid for; in this case, the 442 client should trigger the payment interaction when 443 the user presses the "select" button. 444 * a special button to "disable synchronization and backup" 445 446 One usability issue here is that we are asking users to deal with a 447 private key. It is likely better to map private keys to trustwords 448 (PEP-style). Also, when putting private keys into a QR code, there is 449 the danger of the QR code being scanned and interpreted as a "public" 450 URL. Thus, the QR code should use the schema 451 ``taler://sync/$SYNC-DOMAIN/$SYNC-PATH#private-key`` where 452 ``$SYNC-DOMAIN`` is the domainname of the synchronization service and 453 ``$SYNC-PATH`` the (usually empty) path. By putting the private key after 454 ``#``, we may succeed in disclosing the value even to eager Web-ish 455 interpreters of URLs. Note that the actual synchronization service 456 must use the HTTPS protocol, which means we can leave out this prefix. 457 458 459 --------------------------- 460 Web Security Considerations 461 --------------------------- 462 463 To ensure that the Taler Web extension (and others) can access the 464 service despite Web "security", all service endpoints must set the 465 header:: 466 467 Access-Control-Allow-Origin: *