taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

076-paywall-proxy.rst (21329B)


      1 DD 76: Paivana - Fighting AI Bots with GNU Taler
      2 ################################################
      3 
      4 :Design status: Accepted
      5 :Implementation status: Implemented
      6 :DD shepherd: TBD
      7 :Historical contributors: Florian Dold, Christian Grothoff
      8 :First published: 2025-11-26
      9 :Last substantive change: 2026-11-19
     10 :Implementation evidence: ``merchant`` (2026-01-20; 2026-04-25; 2026-04-27; 2026-08-04); ``paivana`` (2026-04-19)
     11 :Normative references: ``taler-paivana-manual.rst``, ``frags/paivana-httpd-manual.rst``, and ``core/api-merchant.rst``
     12 
     13 Summary
     14 =======
     15 
     16 This design document describes the architecture of an AI Web firewall using GNU
     17 Taler, as well as new features that are required for the implementation.
     18 
     19 Motivation
     20 ==========
     21 
     22 AI bots are causing enormous amounts of traffic by scraping sites like git
     23 forges. They neither respect robots.txt nor 5xx HTTP responses. Solutions like
     24 Anubis and IP-based blocking do not work anymore at this point.
     25 
     26 Requirements
     27 ============
     28 
     29 * Must withstand high traffic from bots, requests before a payment happened
     30   must be *very* cheap, both in terms of response generation and database
     31   interaction. This includes good support for caching.
     32 * Should work not just for our paivana-httpd but also for Turnstile-style
     33   paywalls that need to work with purely static paywall pages without
     34   PHP sessions.
     35 
     36 
     37 Proposed Solution
     38 =================
     39 
     40 Architecture
     41 ------------
     42 
     43 * paivana-httpd is a reverse proxy that sits between ingress HTTP(S) traffic
     44   and the protected upstream service.
     45 * paivana-httpd is configured with a particular merchant backend.
     46 * A payment template must be set up in the merchant backend (called ``{template_id}``
     47   from here on).
     48 
     49 Steps:
     50 
     51 * Browser visits ``{website}``
     52   (for example, ``https://git.taler.net``) where
     53   ``{domain}`` is the domain name of ``{website}``.
     54 * paivana-httpd working as a reverse-proxy for
     55   ``{website}``. Whenever called for a non-whitelisted
     56   URL, it checks for a the presence of a Paivana cookie valid for
     57   this client IP address and ``{website}`` at this time.
     58   The *Paivana Cookie* is computed as:
     59 
     60   ``cookie_expiration_str || '-' || crock32(HKDF(salt=cookie_expiration_enc, ikm=paivana_server_secret, info=website || '\0' || client_ip))``.
     61 
     62   The cookie may expire at or after the ``min_expiration`` in the
     63   ``paivana_id``, up to the paid contract's ``max_pickup_time``.
     64   ``cookie_expiration_str`` encodes that instant as decimal ASCII seconds
     65   since the Epoch. The HKDF salt ``cookie_expiration_enc`` encodes the
     66   same instant as an unsigned 64-bit big-endian count of microseconds.
     67   ``HKDF`` is GNUnet's HKDF (``GNUNET_CRYPTO_hkdf_gnunet()``, which
     68   extracts with HMAC-SHA-512 and expands with HMAC-SHA-256), and the
     69   output is 512 bits.
     70   Using a keyed PRF instead of a plain hash over the concatenation
     71   ensures that the cookie cannot be forged without the server secret
     72   and that the inputs are unambiguously separated: ``website`` is
     73   terminated by a zero byte before ``client_ip`` is appended, so
     74   different ``(website, client_ip)`` pairs can never yield the same
     75   ``info`` string.
     76   ``crock32`` is GNUnet's Crockford-inspired base32 encoding.
     77 
     78   The cookie is computed and verified exclusively by paivana-httpd;
     79   the browser only stores and returns it and thus never has to
     80   reconstruct this value.
     81 
     82   * If such a cookie is set and valid, the request is
     83     reverse-proxied to upstream. *Stop.*
     84   * Otherwise, an HTTP 303 See Other to
     85     ``/.well-known/paivana/templates/$ID#$WEBSITE``
     86     is returned. Here, ``$ID`` is the template ID and
     87     ``$WEBSITE`` is base64url-encoding of the full URL of
     88     the website currently being visited. This way,
     89     the template page can be fully static and cached, and the
     90     JavaScript logic on that page can learn which website
     91     to pay for (and after payment redirect the browser there).
     92 
     93 * When the browser requests ``/.well-known/paivana/templates/$ID``
     94    a static **cachable** paywall page is returned,
     95    including a machine-readable ``Paivana`` HTTP header with
     96    the ``taler://pay-template/`` URL minus the client-computed
     97    ``{paivana_id}`` and fullfillment URL (see below).
     98 
     99 * The paywall page chooses the minimum acceptable access expiry and generates
    100   a fresh random 16-byte ``nonce`` for the article URL ``website``.
    101 
    102   ``paivana_id := min_expiration_str || '-' || b64url(SHA256(nonce || website || '\0' || min_expiration_enc))``.
    103 
    104   ``min_expiration_str`` is the minimum expiry in decimal ASCII seconds since
    105   the Epoch. ``min_expiration_enc`` encodes the same instant as an unsigned
    106   64-bit big-endian count of microseconds (seconds multiplied by 1000000).
    107   The hash covers the binary nonce, the UTF-8 URL with its terminating zero
    108   byte, and the eight timestamp bytes, in that order.
    109 
    110   The paywall chooses how much remaining access to request. For example, a
    111   page opened at 12:50 can ask for one hour by setting the minimum to 13:50.
    112   A new purchase must be able to cover that minimum. The template's pickup
    113   duration describes the access a new order buys. A paywall can ask for a
    114   shorter period so a reader can reuse part of an earlier purchase.
    115 
    116   Here ``b64url`` is the RFC 7515 base64 URL encoder without
    117   padding, used to keep the result short (same reason for the use of
    118   SHA-256).
    119   The same computation could also easily be done by a non-JS client
    120   that processes the ``Paivana`` HTTP header (or a GNU Taler wallet
    121   running as a Web extension).
    122 
    123 * Based on this paivana ID, a
    124   ``taler://pay-template/{merchant_backend}/{template_id}?session_id={paivana_id}&fulfillment_url={website}``
    125   URI is generated and rendered as a QR code and link, prompting
    126   the user to pay for access to the ``{website}`` using GNU Taler.
    127   The page can retire its QR code and link after a local timeout and let the
    128   reader request a fresh offer. This timeout is local to the page. A payment
    129   already confirmed can still be redeemed while its contract grants access.
    130 
    131 * The JavaScript in the paywall page running in the browser
    132   (or the non-JS client) long-polls
    133   on a new ``https://{merchant_backend}/sessions/{paivana_id}``
    134   endpoint that returns when an order with the given session ID has been paid
    135   for (regardless of the order ID, which is not known to the browser).
    136 * A wallet now needs to instantiate the pay template, passing the
    137   ``session_id`` and the ``fulfillment_url`` as an additional inputs
    138   to the order creation (the session ID here will work just like
    139   existing use of ``session_ids`` in session-bound payments).
    140   Similarly, the ``{website}`` works as the fulfillment URL as usual.
    141 * The wallet replays a qualifying paid order under the new session ID, or
    142   asks the reader to confirm a new purchase.
    143 * When the long-poller returns and the payment has succeeded, the
    144   browser (still rendering the paywall page) also learns the order ID.
    145 * The JavaScript of the paywall page (or the non-JS client
    146   processing the ``Paivana`` HTTP header) then POSTs the order ID,
    147   ``nonce``, ``expiration``
    148   and ``website`` to ``{domain}/.well-known/paivana``.
    149   In this JSON request, the ``nonce`` is ``crock32``-encoded and
    150   the JSON member ``expiration`` carries the minimum used in the ID as a
    151   GNU Taler timestamp (``{"t_s": ...}``, in seconds). The proxy uses those
    152   values to reconstruct the hash inputs.
    153 
    154   Note that by the time this POST is made, the client already has the
    155   merchant backend's word that the order was paid: that is precisely
    156   what its long poll on ``/sessions/{paivana_id}`` returned, and it is
    157   where the order ID being posted came from.  The step below is
    158   therefore a *confirmation* of something the client has been told, and
    159   not an open-ended wait for a payment that may still be in progress.
    160 
    161 * paivana-httpd re-computes the paivana ID from ``nonce``, ``website``
    162   and ``expiration``, and asks the merchant backend, over its own
    163   authenticated connection, whether the posted order ID was paid under
    164   exactly that session ID.  Recomputing rather than accepting the ID is
    165   what binds the answer to this request: a client cannot post an order
    166   it paid for one article and be let into another, because a different
    167   ``website`` yields a different paivana ID and the order is then not
    168   found under it.
    169 
    170   The reply is accepted only if all of the following hold:
    171 
    172   * the order status is *paid*, and the order has neither been refunded
    173     nor has a refund pending — otherwise a client could take its money
    174     back and keep the cookie;
    175   * the contract's ``fulfillment_url``, if it has one, equals the
    176     posted ``website``; if it has none, the ``website`` must lie under
    177     paivana-httpd's own configured base URL, so that the client cannot
    178     choose which site it is admitted to;
    179   * the contract's ``max_pickup_time`` is strictly in the future and is at
    180     least the requested minimum. An absent or ``never`` deadline grants
    181     unlimited access.
    182 
    183   The wallet applies the same deadline checks when selecting a paid order
    184   for this merchant and article. An order that still grants access satisfies
    185   a minimum in the past. For example, at 12:50 an order valid until 13:00
    186   covers a minimum of 13:00, even in a browser with empty storage.
    187   Scanning the fresh QR associates the paid order with the browser's new
    188   session. A minimum of 13:15 requires a different qualifying order or
    189   confirmation of a new purchase. The wallet cannot revise the minimum in
    190   a scanned QR. A wallet constructing its own session can choose the minimum
    191   after inspecting the reader's purchases.
    192 
    193   The proxy issues the cookie with ``Max-Age`` computed from the actual
    194   cookie expiry and redirects to the article. For example, an order valid
    195   until 13:00 can yield a cookie expiring at 13:00 when the page requested
    196   a minimum of 12:55.
    197 
    198   This query is made as a **long poll with a short, fixed bound** (5
    199   seconds in the current implementation).  Both halves matter:
    200 
    201   * *Long poll*, because the client's confirmation and the backend's own
    202     view of the order can be a moment apart, and because paivana-httpd
    203     and the client may be talking to different backend processes.  An
    204     honest client that is merely early is waited for rather than turned
    205     away, which is the difference between a working paywall and one that
    206     intermittently refuses people who have paid.
    207   * *Short and bounded*, because this endpoint is unauthenticated and
    208     reachable before any payment has been shown to exist.  The wait is
    209     the interval for which an attacker can pin a connection by posting a
    210     random order ID, so it is a cost that is deliberately kept small.
    211     The same bound is applied client-side, so a merchant backend that
    212     stops answering cannot pin connections either.
    213 
    214   Where the order genuinely was not paid, the client is told so (HTTP
    215   409 Conflict) after that bound has elapsed; where the backend did not
    216   answer at all, it gets 504 Gateway Timeout, and where the backend
    217   answered something unusable, 502 Bad Gateway.  Distinguishing these
    218   matters operationally: only the first is the client's fault.
    219 
    220 * The browser reloads the page with the correct
    221   Paivana cookie (see first step).
    222 
    223 
    224 Problems:
    225 ---------
    226 
    227 * A smart attacker might still create a lot of orders via the pay-template.
    228 
    229   * Solution A: Don't care, unlikely to happen in the first place.
    230   * Solution B: Rate-limit template instantiation on a per-IP basis.
    231 
    232 Accepted risks:
    233 ---------------
    234 
    235 Four properties of this design were examined and deliberately kept as they
    236 are.  Each is recorded here together with the assumption that makes it
    237 acceptable, because a deployment that does not satisfy the assumption does
    238 not get the property.
    239 
    240 Only one price per URL
    241 ~~~~~~~~~~~~~~~~~~~~~~
    242 
    243 paivana-httpd quotes a price by taking the first configured template whose
    244 anchored ``website_regex`` matches the requested URL; a template configured
    245 without a regex matches every URL.  At redemption, the checks listed above
    246 are all that can be made: the contract carries a fulfillment URL and a
    247 ``max_pickup_time``, and it does not carry the identity of the template it
    248 was instantiated from.  paivana-httpd therefore cannot tell an order created
    249 under one template from an order created under another, and in particular
    250 cannot check that the amount paid is the amount its own template search
    251 would have quoted for the URL being unlocked.
    252 
    253 Where an instance carries more than one paivana template — or one template
    254 without a ``website_regex``, which matches everything — this is exploitable
    255 in the obvious way.  A client that wants an expensive URL instantiates the
    256 cheap template with that URL as its fulfillment URL, pays the cheap price,
    257 and posts the result for redemption; both orders name the same fulfillment
    258 URL, which is all the redemption check inspects.  The exposure is wider than
    259 the configured regular expressions suggest, because the merchant backend
    260 matches ``website_regex`` unanchored where paivana-httpd anchors it: the set
    261 of URLs the backend will sell a template for is a superset of the set
    262 paivana-httpd paywalls with it.
    263 
    264 The mitigation is a property of the configuration rather than of the code.
    265 A merchant instance used by paivana-httpd carries exactly one paivana
    266 template, so that every URL it paywalls has exactly one price and there is
    267 nothing to substitute; differentiated pricing across a site is then a matter
    268 of separate instances, each with its own template and its own paivana-httpd.
    269 **The risk is accepted on the assumption that a deployment presents a single
    270 price for every URL it paywalls.**  A deployment that puts two paivana
    271 templates on one instance is selling its expensive articles at the cheaper
    272 price.
    273 
    274 Closing the gap properly requires the merchant backend to record the
    275 instantiating ``template_id`` in the contract terms and to report it with
    276 the order status; paivana-httpd could then re-run its own template search
    277 for the posted ``website`` and require the two to agree.  That is a
    278 merchant-side change in a separate upstream, and this document does not
    279 assume it.
    280 
    281 Payment buys access, not a seat
    282 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    283 
    284 The redemption POST is idempotent and unmetered, and nothing records that an
    285 order or a paivana ID has already been redeemed.  The cookie it returns is
    286 bound to the address of whoever posted the redemption, not to the address
    287 that paid.  Anyone holding the four posted fields — order ID, ``nonce``,
    288 ``expiration`` and ``website`` — can therefore obtain their own cookie, for
    289 their own address, while the merchant confirms payment under that session
    290 and the contract still grants access. Replaying the order under a fresh
    291 session can supersede the old session at the merchant. Cookies already
    292 issued remain valid through their own expiry.
    293 
    294 This is intended behaviour and not a defect.  What is sold is access to one
    295 resource until one moment, and the buyer may pass that on, in the same way
    296 and for the same reasons that the buyer of a newspaper may hand it to the
    297 next reader.  The purchase stays bounded by what was bought: sharing extends
    298 a payment to more readers, never to more URLs and never past the paid contract’s deadline,
    299 so a client that wants the whole site still pays for the whole site.  That
    300 bound is what lets the design's actual goal — making bulk automated
    301 retrieval expensive — survive the sharing.  **The risk is accepted on the
    302 assumption that deployments price access per resource and per unit of time,
    303 and that none of them requires per-seat licensing**, which this design
    304 cannot provide and must not be configured as though it could.
    305 
    306 One consequence has to be stated plainly, because the construction of the
    307 cookie invites the opposite reading: binding the cookie to the client
    308 address is a cookie-theft mitigation and nothing else.  It ensures that a
    309 cookie which leaks — from a log, a shared machine, a proxy — is useless to
    310 whoever picks it up.  It provides no anti-sharing property whatsoever, since
    311 the redemption that mints cookies is open to every address.
    312 
    313 The redemption endpoint is thus unmetered by intent.  Metering it would not
    314 restore any property this design claims; the rate-limiting question raised
    315 above for template instantiation is a question about load, it applies to
    316 this endpoint in the same form, and it is open in the same way.
    317 
    318 Entropy of the server secret
    319 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    320 
    321 ``paivana_server_secret`` is derived from the configured secret by a single
    322 unsalted SHA-512.  There is no stretching and no salt, so the cost of
    323 guessing that secret offline from one observed cookie is one hash per
    324 candidate: the strength of every cookie the deployment will ever issue is
    325 the entropy of the configured string, and nothing more.
    326 
    327 The requirement that follows is placed on the operator.  The configured
    328 secret must carry at least 128 bits drawn from a cryptographic random
    329 source, and must never be a passphrase, a hostname, a token reused from
    330 elsewhere, or the placeholder that ships in the sample configuration — which
    331 is an example rather than a secret, and leaves a deployment that keeps it
    332 with no secret at all.  Where no secret is configured, paivana-httpd uses a
    333 fresh random value per process, which is safe but invalidates every
    334 outstanding cookie whenever the service restarts.
    335 
    336 Absorbing the requirement into the construction was considered and rejected.
    337 A memory-hard KDF exists to make human-chosen, low-entropy secrets expensive
    338 to guess; it buys a fixed factor, no fixed factor rescues a guessable
    339 phrase, and against 128 genuine bits it buys nothing that is needed.
    340 ``paivana_server_secret`` is a machine-generated configuration value that
    341 nobody has to remember or type, so the situation a KDF defends against is
    342 one the deployment can simply not be in.  **The risk is accepted on the
    343 assumption that the secret is produced by a random generator and never
    344 chosen by a person**; where a person chooses it, the cookies are forgeable
    345 and the paywall is decorative.  How to generate such a secret is operator
    346 guidance and belongs with the manual rather than here.
    347 
    348 Truncation is unreportable to an HTTP/1.0 client
    349 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    350 
    351 paivana-httpd relays bodies as they arrive rather than assembling them
    352 first, which is what lets it serve content larger than memory and lets the
    353 client start receiving before the upstream has finished.  The cost is that
    354 the upstream's status line and headers reach the client long before the
    355 body is complete, so an upstream that fails mid-body cannot be reported as
    356 ``502`` — that status has already been spent.
    357 
    358 What is left is to break the framing, which for almost every client is
    359 enough: a declared ``Content-Length`` is left unmet, or a chunked response
    360 is closed without its terminating chunk, and RFC 9112 section 8.1.2
    361 requires a recipient to treat either as a failed message.  The exception is
    362 an HTTP/1.0 client receiving a response whose length the upstream never
    363 declared.  Such a client cannot be sent chunks, so the close of the
    364 connection *is* the end-of-body marker, and a truncated body is
    365 byte-for-byte indistinguishable from a complete one.
    366 
    367 The alternative would be to buffer each response until it is known to be
    368 complete, which is exactly the property being given up, and which bounds
    369 every response by memory to buy correct reporting for one obsolete client
    370 version. **The risk is accepted on the assumption that clients speak
    371 HTTP/1.1**, which every browser and every HTTP library in current use has
    372 done since well before this design; where a genuine HTTP/1.0 client is
    373 expected, the upstream should be configured to declare a
    374 ``Content-Length``, which restores detection for it too.
    375 
    376 Implementation
    377 --------------
    378 
    379 * [x] Merchant backend can look up order IDs under a Paivana session ID.
    380 * [x] Merchant backend can instantiate Paivana templates with ``paivana_id``
    381   and the target website.
    382 * [x] Paivana component implemented.
    383 * [x] Wallet/Web utility support implemented.
    384 
    385 
    386 Test Plan
    387 =========
    388 
    389 * Deploy it for git.taler.net
    390 
    391 Definition of Done
    392 ==================
    393 
    394 * [x] Merchant, Paivana, and wallet-side protocol support implemented.
    395 * [x] Protocol and operator documentation published.
    396 * [ ] Production deployment and end-to-end QC recorded.
    397 
    398 Alternatives
    399 ============
    400 
    401 * Do not re-use the session ID mechanism but introduce some new concept.
    402   This has the drawback of us needing additional tables and indicies,
    403   and also the existing use of the session ID is very parallel to this one.
    404 * Instead of doing a 303 See Other, cache control could have been achieved by
    405   specifying a "Vary: Cookie" HTTP header. We may combine these and use
    406   that to additionally enable caching of the 303 See Other. The 303 solution
    407   has the advantage that there is only one page to cache per template, and
    408   the disadvantage of an additional redirect. Note that this is purely
    409   a frontend design choice, wallets and merchant backends work nicely with
    410   either approach.
    411 
    412 Drawbacks
    413 =========
    414 
    415 * This exposes an order ID to anyone who knows the session ID. This is
    416   clearly not an issue in this context, and for the existing uses of
    417   the session ID it also seems clear that knowledge of the session ID
    418   requires an attacker to have access that would easily also already
    419   give them any order ID, so this seems harmless.
    420 
    421 
    422 Discussion / Q&A
    423 ================