taler-docs

076-paywall-proxy.rst (20372B)

---

 DD 76: Paivana - Fighting AI Bots with GNU Taler
 ################################################
 
 Summary
 =======
 
 This design document describes the architecture of an AI Web firewall using GNU
 Taler, as well as new features that are required for the implementation.
 
 Motivation
 ==========
 
 AI bots are causing enormous amounts of traffic by scraping sites like git
 forges. They neither respect robots.txt nor 5xx HTTP responses. Solutions like
 Anubis and IP-based blocking do not work anymore at this point.
 
 Requirements
 ============
 
 * Must withstand high traffic from bots, requests before a payment happened
   must be *very* cheap, both in terms of response generation and database
   interaction. This includes good support for caching.
 * Should work not just for our paivana-httpd but also for Turnstile-style
   paywalls that need to work with purely static paywall pages without
   PHP sessions.
 
 
 Proposed Solution
 =================
 
 Architecture
 ------------
 
 * paivana-httpd is a reverse proxy that sits between ingress HTTP(S) traffic
   and the protected upstream service.
 * paivana-httpd is configured with a particular merchant backend.
 * A payment template must be set up in the merchant backend (called ``{template_id}``
   from here on).
 
 Steps:
 
 * Browser visits ``{website}``
   (for example, ``https://git.taler.net``) where
   ``{domain}`` is the domain name of ``{website}``.
 * paivana-httpd working as a reverse-proxy for
   ``{website}``. Whenever called for a non-whitelisted
   URL, it checks for a the presence of a Paivana cookie valid for
   this client IP address and ``{website}`` at this time.
   The *Paivana Cookie* is computed as:
 
   ``expiration || '-' || crock32(HKDF(salt=expiration, ikm=paivana_server_secret, info=website || '\0' || client_ip))``.
 
   where ``expiration`` in the prefix is the expiration time for the
   cookie (and thus the access to the article) in seconds
   (to keep it short) while in the salt it is the binary GNUnet
   absolute time (microseconds) in network byte order.
   Note that this value is the *end of the access being sold*, chosen by
   the client and capped by the contract; it is not a statement about
   when anything happened, and in particular it is not the client's idea
   of the current time.  (It was called ``cur_time`` in earlier drafts of
   this document, which invited exactly that misreading.)
   ``HKDF`` is GNUnet's HKDF (``GNUNET_CRYPTO_hkdf_gnunet()``, which
   extracts with HMAC-SHA-512 and expands with HMAC-SHA-256), and the
   output is 512 bits.
   Using a keyed PRF instead of a plain hash over the concatenation
   ensures that the cookie cannot be forged without the server secret
   and that the inputs are unambiguously separated: ``website`` is
   terminated by a zero byte before ``client_ip`` is appended, so
   different ``(website, client_ip)`` pairs can never yield the same
   ``info`` string.
   ``crock32`` is GNUnet's Crockford-inspired base32 encoding.
 
   The cookie is computed and verified exclusively by paivana-httpd;
   the browser only stores and returns it and thus never has to
   reconstruct this value.
 
   * If such a cookie is set and valid, the request is
     reverse-proxied to upstream. *Stop.*
   * Otherwise, an HTTP 303 See Other to
     ``/.well-known/paivana/templates/$ID#$WEBSITE``
     is returned. Here, ``$ID`` is the template ID and
     ``$WEBSITE`` is base64url-encoding of the full URL of
     the website currently being visited. This way,
     the template page can be fully static and cached, and the
     JavaScript logic on that page can learn which website
     to pay for (and after payment redirect the browser there).
 
 * When the browser requests ``/.well-known/paivana/templates/$ID``
    a static **cachable** paywall page is returned,
    including a machine-readable ``Paivana`` HTTP header with
    the ``taler://pay-template/`` URL minus the client-computed
    ``{paivana_id}`` and fullfillment URL (see below).
 
 * The browser (rendering the paywall page) generates a random
   *paivana ID* via JS using the end of the access it intends to buy
   (``expiration``) in seconds since the Epoch and the current URL
   (``{website}``) plus some freshly generated entropy (``{nonce}``):
 
   ``paivana_id := expiration || '-' || b64url(SHA256(nonce || website || '\0' || expiration))``.
 
   The exact byte string that is hashed is the concatenation of:
 
   * the 16-byte (128-bit) binary ``nonce``;
   * the UTF-8 encoding of ``website``, including its terminating
     zero byte (which separates it unambiguously from the timestamp);
   * ``expiration`` as an 8-byte **big-endian (network byte order)
     number of microseconds** since the Epoch, that is, the value of
     the seconds-based ``expiration`` multiplied by 1000000.
 
   Note that ``expiration`` thus appears twice in two different
   encodings: the ``paivana_id`` *prefix* is the timestamp in
   **seconds** (as decimal ASCII, to keep the identifier short),
   while the hashed value is the same instant in **microseconds**
   in network byte order.
 
   The client is free to pick this value — it is asking for access until
   a particular moment, and it is the contract's ``max_pickup_time`` that
   decides whether it may have it.  Since the same value goes into the
   session ID the order is created under, it cannot be revised after the
   fact.
 
   Here ``b64url`` is the RFC 7515 base64 URL encoder without
   padding, used to keep the result short (same reason for the use of
   SHA-256).
   The same computation could also easily be done by a non-JS client
   that processes the ``Paivana`` HTTP header (or a GNU Taler wallet
   running as a Web extension).
 
 * Based on this paivana ID, a
   ``taler://pay-template/{merchant_backend}/{template_id}?session_id={paivana_id}&fulfillment_url={website}``
   URI is generated and rendered as a QR code and link, prompting
   the user to pay for access to the ``{website}`` using GNU Taler.
 
 * The JavaScript in the paywall page running in the browser
   (or the non-JS client) long-polls
   on a new ``https://{merchant_backend}/sessions/{paivana_id}``
   endpoint that returns when an order with the given session ID has been paid
   for (regardless of the order ID, which is not known to the browser).
 * A wallet now needs to instantiate the pay template, passing the
   ``session_id`` and the ``fulfillment_url`` as an additional inputs
   to the order creation (the session ID here will work just like
   existing use of ``session_ids`` in session-bound payments).
   Similarly, the ``{website}`` works as the fulfillment URL as usual.
 * The wallet then must pay for the resulting order
   by talking to the Merchant backend.
 * When the long-poller returns and the payment has succeeded, the
   browser (still rendering the paywall page) also learns the order ID.
 * The JavaScript of the paywall page (or the non-JS client
   processing the ``Paivana`` HTTP header) then POSTs the order ID,
   ``nonce``, ``expiration``
   and ``website`` to ``{domain}/.well-known/pavivana``.
   In this JSON request, the ``nonce`` is ``crock32``-encoded and
   ``expiration`` is a normal GNU Taler timestamp object
   (``{"t_s": ...}``, in seconds); the server re-derives the binary
   inputs given above from these values.
 
   Note that by the time this POST is made, the client already has the
   merchant backend's word that the order was paid: that is precisely
   what its long poll on ``/sessions/{paivana_id}`` returned, and it is
   where the order ID being posted came from.  The step below is
   therefore a *confirmation* of something the client has been told, and
   not an open-ended wait for a payment that may still be in progress.
 
 * paivana-httpd re-computes the paivana ID from ``nonce``, ``website``
   and ``expiration``, and asks the merchant backend, over its own
   authenticated connection, whether the posted order ID was paid under
   exactly that session ID.  Recomputing rather than accepting the ID is
   what binds the answer to this request: a client cannot post an order
   it paid for one article and be let into another, because a different
   ``website`` yields a different paivana ID and the order is then not
   found under it.
 
   The reply is accepted only if all of the following hold:
 
   * the order status is *paid*, and the order has neither been refunded
     nor has a refund pending — otherwise a client could take its money
     back and keep the cookie;
   * the contract's ``fulfillment_url``, if it has one, equals the
     posted ``website``; if it has none, the ``website`` must lie under
     paivana-httpd's own configured base URL, so that the client cannot
     choose which site it is admitted to;
   * ``expiration`` is not later than the contract's ``max_pickup_time``,
     which is what stops a client from buying five minutes of access and
     minting itself a cookie valid for a year.
 
   If so, paivana-httpd issues the Paivana cookie described above, with
   ``Max-Age`` derived from ``expiration``, and redirects to the
   ``{website}``.
 
   This query is made as a **long poll with a short, fixed bound** (5
   seconds in the current implementation).  Both halves matter:
 
   * *Long poll*, because the client's confirmation and the backend's own
     view of the order can be a moment apart, and because paivana-httpd
     and the client may be talking to different backend processes.  An
     honest client that is merely early is waited for rather than turned
     away, which is the difference between a working paywall and one that
     intermittently refuses people who have paid.
   * *Short and bounded*, because this endpoint is unauthenticated and
     reachable before any payment has been shown to exist.  The wait is
     the interval for which an attacker can pin a connection by posting a
     random order ID, so it is a cost that is deliberately kept small.
     The same bound is applied client-side, so a merchant backend that
     stops answering cannot pin connections either.
 
   Where the order genuinely was not paid, the client is told so (HTTP
   409 Conflict) after that bound has elapsed; where the backend did not
   answer at all, it gets 504 Gateway Timeout, and where the backend
   answered something unusable, 502 Bad Gateway.  Distinguishing these
   matters operationally: only the first is the client's fault.
 
 * The browser reloads the page with the correct
   Paivana cookie (see first step).
 
 
 Problems:
 ---------
 
 * A smart attacker might still create a lot of orders via the pay-template.
 
   * Solution A: Don't care, unlikely to happen in the first place.
   * Solution B: Rate-limit template instantiation on a per-IP basis.
 
 Accepted risks:
 ---------------
 
 Four properties of this design were examined and deliberately kept as they
 are.  Each is recorded here together with the assumption that makes it
 acceptable, because a deployment that does not satisfy the assumption does
 not get the property.
 
 Only one price per URL
 ~~~~~~~~~~~~~~~~~~~~~~
 
 paivana-httpd quotes a price by taking the first configured template whose
 anchored ``website_regex`` matches the requested URL; a template configured
 without a regex matches every URL.  At redemption, the checks listed above
 are all that can be made: the contract carries a fulfillment URL and a
 ``max_pickup_time``, and it does not carry the identity of the template it
 was instantiated from.  paivana-httpd therefore cannot tell an order created
 under one template from an order created under another, and in particular
 cannot check that the amount paid is the amount its own template search
 would have quoted for the URL being unlocked.
 
 Where an instance carries more than one paivana template — or one template
 without a ``website_regex``, which matches everything — this is exploitable
 in the obvious way.  A client that wants an expensive URL instantiates the
 cheap template with that URL as its fulfillment URL, pays the cheap price,
 and posts the result for redemption; both orders name the same fulfillment
 URL, which is all the redemption check inspects.  The exposure is wider than
 the configured regular expressions suggest, because the merchant backend
 matches ``website_regex`` unanchored where paivana-httpd anchors it: the set
 of URLs the backend will sell a template for is a superset of the set
 paivana-httpd paywalls with it.
 
 The mitigation is a property of the configuration rather than of the code.
 A merchant instance used by paivana-httpd carries exactly one paivana
 template, so that every URL it paywalls has exactly one price and there is
 nothing to substitute; differentiated pricing across a site is then a matter
 of separate instances, each with its own template and its own paivana-httpd.
 **The risk is accepted on the assumption that a deployment presents a single
 price for every URL it paywalls.**  A deployment that puts two paivana
 templates on one instance is selling its expensive articles at the cheaper
 price.
 
 Closing the gap properly requires the merchant backend to record the
 instantiating ``template_id`` in the contract terms and to report it with
 the order status; paivana-httpd could then re-run its own template search
 for the posted ``website`` and require the two to agree.  That is a
 merchant-side change in a separate upstream, and this document does not
 assume it.
 
 Payment buys access, not a seat
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The redemption POST is idempotent and unmetered, and nothing records that an
 order or a paivana ID has already been redeemed.  The cookie it returns is
 bound to the address of whoever posted the redemption, not to the address
 that paid.  Anyone holding the four posted fields — order ID, ``nonce``,
 ``expiration`` and ``website`` — can therefore obtain their own cookie, for
 their own address, at any point until ``expiration``, and a buyer who
 publishes those four fields has given the article to everyone who reads
 them.
 
 This is intended behaviour and not a defect.  What is sold is access to one
 resource until one moment, and the buyer may pass that on, in the same way
 and for the same reasons that the buyer of a newspaper may hand it to the
 next reader.  The purchase stays bounded by what was bought: sharing extends
 a payment to more readers, never to more URLs and never past ``expiration``,
 so a client that wants the whole site still pays for the whole site.  That
 bound is what lets the design's actual goal — making bulk automated
 retrieval expensive — survive the sharing.  **The risk is accepted on the
 assumption that deployments price access per resource and per unit of time,
 and that none of them requires per-seat licensing**, which this design
 cannot provide and must not be configured as though it could.
 
 One consequence has to be stated plainly, because the construction of the
 cookie invites the opposite reading: binding the cookie to the client
 address is a cookie-theft mitigation and nothing else.  It ensures that a
 cookie which leaks — from a log, a shared machine, a proxy — is useless to
 whoever picks it up.  It provides no anti-sharing property whatsoever, since
 the redemption that mints cookies is open to every address.
 
 The redemption endpoint is thus unmetered by intent.  Metering it would not
 restore any property this design claims; the rate-limiting question raised
 above for template instantiation is a question about load, it applies to
 this endpoint in the same form, and it is open in the same way.
 
 Entropy of the server secret
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 ``paivana_server_secret`` is derived from the configured secret by a single
 unsalted SHA-512.  There is no stretching and no salt, so the cost of
 guessing that secret offline from one observed cookie is one hash per
 candidate: the strength of every cookie the deployment will ever issue is
 the entropy of the configured string, and nothing more.
 
 The requirement that follows is placed on the operator.  The configured
 secret must carry at least 128 bits drawn from a cryptographic random
 source, and must never be a passphrase, a hostname, a token reused from
 elsewhere, or the placeholder that ships in the sample configuration — which
 is an example rather than a secret, and leaves a deployment that keeps it
 with no secret at all.  Where no secret is configured, paivana-httpd uses a
 fresh random value per process, which is safe but invalidates every
 outstanding cookie whenever the service restarts.
 
 Absorbing the requirement into the construction was considered and rejected.
 A memory-hard KDF exists to make human-chosen, low-entropy secrets expensive
 to guess; it buys a fixed factor, no fixed factor rescues a guessable
 phrase, and against 128 genuine bits it buys nothing that is needed.
 ``paivana_server_secret`` is a machine-generated configuration value that
 nobody has to remember or type, so the situation a KDF defends against is
 one the deployment can simply not be in.  **The risk is accepted on the
 assumption that the secret is produced by a random generator and never
 chosen by a person**; where a person chooses it, the cookies are forgeable
 and the paywall is decorative.  How to generate such a secret is operator
 guidance and belongs with the manual rather than here.
 
 Truncation is unreportable to an HTTP/1.0 client
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 paivana-httpd relays bodies as they arrive rather than assembling them
 first, which is what lets it serve content larger than memory and lets the
 client start receiving before the upstream has finished.  The cost is that
 the upstream's status line and headers reach the client long before the
 body is complete, so an upstream that fails mid-body cannot be reported as
 ``502`` — that status has already been spent.
 
 What is left is to break the framing, which for almost every client is
 enough: a declared ``Content-Length`` is left unmet, or a chunked response
 is closed without its terminating chunk, and RFC 9112 section 8.1.2
 requires a recipient to treat either as a failed message.  The exception is
 an HTTP/1.0 client receiving a response whose length the upstream never
 declared.  Such a client cannot be sent chunks, so the close of the
 connection *is* the end-of-body marker, and a truncated body is
 byte-for-byte indistinguishable from a complete one.
 
 The alternative would be to buffer each response until it is known to be
 complete, which is exactly the property being given up, and which bounds
 every response by memory to buy correct reporting for one obsolete client
 version. **The risk is accepted on the assumption that clients speak
 HTTP/1.1**, which every browser and every HTTP library in current use has
 done since well before this design; where a genuine HTTP/1.0 client is
 expected, the upstream should be configured to declare a
 ``Content-Length``, which restores detection for it too.
 
 Implementation:
 ---------------
 
 * Merchant backend needs way to lookup order IDs under a ``session_id``
   (DONE: e027e729..b476f8ae)
 * Merchant backend needs way to instantiate templates with
   a given ``session_id`` and ``fulfillment_url``. This also
   requires extending the allowed responses for templates in general.
 * Paivana component needs to be implemented
 * Wallet-core needs support for a ``session_id`` and
   ``fulfillment_url`` in pay templates.
 
 
 Test Plan
 =========
 
 * Deploy it for git.taler.net
 
 Definition of Done
 ==================
 
 N/A
 
 Alternatives
 ============
 
 * Do not re-use the session ID mechanism but introduce some new concept.
   This has the drawback of us needing additional tables and indicies,
   and also the existing use of the session ID is very parallel to this one.
 * Instead of doing a 303 See Other, cache control could have been achieved by
   specifying a "Vary: Cookie" HTTP header. We may combine these and use
   that to additionally enable caching of the 303 See Other. The 303 solution
   has the advantage that there is only one page to cache per template, and
   the disadvantage of an additional redirect. Note that this is purely
   a frontend design choice, wallets and merchant backends work nicely with
   either approach.
 
 Drawbacks
 =========
 
 * This exposes an order ID to anyone who knows the session ID. This is
   clearly not an issue in this context, and for the existing uses of
   the session ID it also seems clear that knowledge of the session ID
   requires an attacker to have access that would easily also already
   give them any order ID, so this seems harmless.
 
 
 Discussion / Q&A
 ================