076-paywall-proxy.rst (6970B)
1 DD 76: Paivana - Fighting AI Bots with GNU Taler 2 ################################################ 3 4 Summary 5 ======= 6 7 This design document describes the architecture of an AI Web firewall using GNU 8 Taler, as well as new features that are required for the implementation. 9 10 Motivation 11 ========== 12 13 AI bots are causing enormous amounts of traffic by scraping sites like git 14 forges. They neither respect robots.txt nor 5xx HTTP responses. Solutions like 15 Anubis and IP-based blocking do not work anymore at this point. 16 17 Requirements 18 ============ 19 20 * Must withstand high traffic from bots, requests before a payment happened 21 must be *very* cheap, both in terms of response generation and database 22 interaction. This includes good support for caching. 23 * Should work not just for our paivana-httpd but also for Turnstile-style 24 paywalls that need to work with purely static paywall pages without 25 PHP sessions. 26 27 28 Proposed Solution 29 ================= 30 31 Architecture 32 ------------ 33 34 * paivana-httpd is a reverse proxy that sits between ingress HTTP(S) traffic 35 and the protected upstream service. 36 * paivana-httpd is configured with a particular merchant backend. 37 * A payment template must be set up in the merchant backend (called ``{template_id}`` 38 from here on). 39 40 Steps: 41 42 * Browser visits ``{website}`` 43 (for example, ``https://git.taler.net``) where 44 ``{domain}`` is the domain name of ``{website}``. 45 * paivana-httpd working as a reverse-proxy for 46 ``{website}``. Whenever called for a non-whitelisted 47 URL, it checks for a the presence of a Paivana cookie valid for 48 this client IP address and ``{website}`` at this time. 49 The *Paivana Cookie* is computed as: 50 51 ``cur_time || '-' || crock32(SHA512(website || client_ip || paivana_server_secret || cur_time))``. 52 53 where ``cur_time`` in the prefix is the current time in seconds 54 (to keep it short) while in the hash it is usually binary GNUnet 55 timestamp in network byte order. 56 ``crock32`` is GNUnet's Crockford-inspired base32 encoding. 57 58 * If such a cookie is set and valid, the request is 59 reverse-proxied to upstream. *Stop.* 60 * Otherwise, an HTTP 302 Redirect to 61 ``/.well-known/paivana/templates/$ID#SITE`` 62 is returned. Here, ``$ID`` is the template ID and 63 ``$SITE`` is the website currently being visited. This way, 64 the template page can be fully static and cached, and the 65 JavaScript logic on that page can learn which website 66 to pay for (and after payment go back there). 67 68 * When the browser requests ``/.well-known/paivana/templates/$ID`` 69 a static **cachable** paywall page is returned, 70 including a machine-readable ``Paivana`` HTTP header with 71 the ``taler://pay-template/`` URL minus the client-computed 72 ``{paivana_id}`` and fullfillment URL (see below). 73 74 * The browser (rendering the paywall page) generates a random 75 *paivana ID* via JS using the current time (``cur_time``) in seconds 76 since the Epoch and the current URL (``{website}``) plus some 77 freshly generated entropy (``{nonce}``): 78 79 ``paivana_id := cur_time || '-' || b64url(SHA256(nonce || website || cur_time))``. 80 81 Here ``b64url`` is the RFC 7515 base64 URL encoder, used to keep 82 the result short (same reason for the use of SHA-256). 83 The same computation could also easily be done by a non-JS client 84 that processes the ``Paivana`` HTTP header (or a GNU Taler wallet 85 running as a Web extension). 86 87 * Based on this paivana ID, a 88 ``taler://pay-template/{merchant_backend}/{template_id}?session_id={paivana_id}&fulfillment_url={website}`` 89 URI is generated and rendered as a QR code and link, prompting 90 the user to pay for access to the ``{website}`` using GNU Taler. 91 92 * The JavaScript in the paywall page running in the browser 93 (or the non-JS client) long-polls 94 on a new ``https://{merchant_backend}/sessions/{paivana_id}`` 95 endpoint that returns when an order with the given session ID has been paid 96 for (regardless of the order ID, which is not known to the browser). 97 * A wallet now needs to instantiate the pay template, passing the 98 ``session_id`` and the ``fulfillment_url`` as an additional inputs 99 to the order creation (the session ID here will work just like 100 existing use of ``session_ids`` in session-bound payments). 101 Similarly, the ``{website}`` works as the fulfillment URL as usual. 102 * The wallet then must pay for the resulting order 103 by talking to the Merchant backend. 104 * When the long-poller returns and the payment has succeeded, the 105 browser (still rendering the paywall page) also learns the order ID. 106 * The JavaScript of the paywall page (or the non-JS client 107 processing the ``Paivana`` HTTP header) then POSTs the order ID, 108 ``nonce``, ``cur_time`` 109 and ``website`` to ``{domain}/.well-known/pavivana``. 110 * paivana-httpd computes the paivana ID and checks if the given 111 order ID was indeed paid recently for the computed paivana ID. 112 If so, it generates an HTTP response which the Paivana cookie 113 and redirects to the fulfillment URL (which is the original {website}). 114 * The browser reloads the page with the correct 115 Paivana cookie (see first step). 116 117 118 Problems: 119 --------- 120 121 * A smart attacker might still create a lot of orders via the pay-template. 122 123 * Solution A: Don't care, unlikely to happen in the first place. 124 * Solution B: Rate-limit template instantiation on a per-IP basis. 125 126 Implementation: 127 --------------- 128 129 * Merchant backend needs way to lookup order IDs under a ``session_id`` 130 (DONE: e027e729..b476f8ae) 131 * Merchant backend needs way to instantiate templates with 132 a given ``session_id`` and ``fulfillment_url``. This also 133 requires extending the allowed responses for templates in general. 134 * Paivana component needs to be implemented 135 * Wallet-core needs support for a ``session_id`` and 136 ``fulfillment_url`` in pay templates. 137 138 139 Test Plan 140 ========= 141 142 * Deploy it for git.taler.net 143 144 Definition of Done 145 ================== 146 147 N/A 148 149 Alternatives 150 ============ 151 152 * Do not re-use the session ID mechanism but introduce some new concept. 153 This has the drawback of us needing additional tables and indicies, 154 and also the existing use of the session ID is very parallel to this one. 155 * Instead of doing a 302 Redirect, cache control could have been achieved by 156 specifying a "Vary: Cookie" HTTP header. We may combine these and use 157 that to additionally enable caching of the 302 Redirect. The 302 solution 158 has the advantage that there is only one page to cache per template, and 159 the disadvantage of an additional redirect. Note that this is purely 160 a frontend design choice, wallets and merchant backends work nicely with 161 either approach. 162 163 Drawbacks 164 ========= 165 166 * This exposes an order ID to anyone who knows the session ID. This is 167 clearly not an issue in this context, and for the existing uses of 168 the session ID it also seems clear that knowledge of the session ID 169 requires an attacker to have access that would easily also already 170 give them any order ID, so this seems harmless. 171 172 173 Discussion / Q&A 174 ================