076-paywall-proxy.rst (5830B)
1 DD 76: Paivana - Fighting AI Bots with GNU Taler 2 ################################################ 3 4 Summary 5 ======= 6 7 This design document describes the architecture of an AI Web firewall using GNU 8 Taler, as well as new features that are required for the implementation. 9 10 Motivation 11 ========== 12 13 AI bots are causing enormous amounts of traffic by scraping sites like git 14 forges. They neither respect robots.txt nor 5xx HTTP responses. Solutions like 15 Anubis and IP-based blocking do not work anymore at this point. 16 17 Requirements 18 ============ 19 20 * Must withstand high traffic from bots, requests before a payment happened 21 must be *very* cheap, both in terms of response generation and database 22 interaction. 23 * Should work not just for our paivana-httpd but also for Turnstile-style 24 paywalls that need to work with purely static paywall pages without 25 PHP sessions. 26 27 28 Proposed Solution 29 ================= 30 31 Architecture 32 ------------ 33 34 * paivana-httpd is a reverse proxy that sits between ingress HTTP(S) traffic 35 and the protected upstream service. 36 * paivana-httpd is configured with a particular merchant backend. 37 * A payment template must be set up in the merchant backend (called ``{template_id}`` 38 from here on). 39 40 Steps: 41 42 * Browser visits ``{website}`` 43 (for example, ``https://git.taler.net``) where 44 ``{domain}`` is the domain name of ``{website}``. 45 * paivana-httpd working as a reverse-proxy for 46 ``{website}``. Whenever called for a non-whitelisted 47 URL, it checks for a the presence of a Paivana cookie valid for 48 this client IP address and ``{website}`` at this time. 49 The *Paivana Cookie* is computed as: 50 51 ``cur_time || '-' || H(website || client_ip || paivana_server_secret || cur_time)``. 52 53 where ``cur_time`` in the prefix is the current time in seconds 54 (to keep it short) while in the hash it is usually binary GNUnet 55 timestamp in network byte order. 56 57 * If such a cookie is set and valid, the request is 58 reverse-proxied to upstream. *Stop.* 59 * Otherwise, a static non-cachable paywall page is returned, 60 including a machine-readable ``Paivana`` HTTP header with 61 the ``taler://pay-template/`` URL minus the client-computed 62 ``{paivana_id}`` and fullfillment URL (see below). 63 *Continue.* 64 65 * The browser (rendering the paywall page) generates a random 66 *paivana ID* via JS using the current time (``cur_time``) in seconds 67 since the Epoch and the current URL (``{website}``) plus some 68 freshly generated entropy (``{nonce}``): 69 70 ``paivana_id := cur_time || '-' || H(nonce || website || cur_time)``. 71 72 The same computation could also easily be done by a non-JS client 73 that processes the ``Paivana`` HTTP header (or a GNU Taler wallet 74 running as a Web extension). 75 76 * Based on this paivana ID, a 77 ``taler://pay-template/{merchant_backend}/{template_id}?session_id={paivana_id}&fulfillment_url={website}`` 78 URI is generated and rendered as a QR code and link, prompting 79 the user to pay for access to the ``{website}`` using GNU Taler. 80 81 * The JavaScript in the paywall page running in the browser 82 (or the non-JS client) long-polls 83 on a new ``https://{merchant_backend}/sessions/{paivana_id}`` 84 endpoint that returns when an order with the given session ID has been paid 85 for (regardless of the order ID, which is not known to the browser). 86 * A wallet now needs to instantiate the pay template, passing the 87 ``session_id`` and the ``website`` as an additional inputs 88 to the order creation (the session ID here will work just like 89 existing use of ``session_ids`` in session-bound payments). 90 Similarly, the ``website`` works as the fulfillment URL as usual. 91 * The wallet then must pay for the resulting order 92 by talking to the Merchant backend. 93 * When the long-poller returns and the payment has succeeded, the 94 browser (still rendering the paywall page) also learns the order ID. 95 * The JavaScript of the paywall page (or the non-JS client 96 processing the ``Paivana`` HTTP header) then POSTs the order ID, 97 ``nonce``, ``cur_time`` 98 and ``website`` to ``{domain}/.well-known/pavivana``. 99 * paivana-httpd computes the paivana ID and checks if the given 100 order ID was indeed paid recently for the computed paivana ID. 101 If so, it generates an HTTP response which the Paivana cookie 102 and redirects to the fulfillment URL (which is the original {website}). 103 * The browser reloads the page with the correct 104 Paivana cookie (see first step). 105 106 107 Problems: 108 --------- 109 110 * A smart attacker might still create a lot of orders via the pay-template. 111 112 * Solution A: Don't care, unlikely to happen in the first place. 113 * Solution B: Rate-limit template instantiation on a per-IP basis. 114 115 Implementation: 116 --------------- 117 118 * Merchant backend needs way to lookup order IDs under a ``session_id`` 119 (DONE: e027e729..b476f8ae) 120 * Merchant backend needs way to instantiate templates with 121 a given ``session_id`` and ``fulfillment_url``. This also 122 requires extending the allowed responses for templates in general. 123 * Paivana component needs to be implemented 124 * Wallet-core needs support for a ``session_id`` and 125 ``fulfillment_url`` in pay templates. 126 127 128 Test Plan 129 ========= 130 131 * Deploy it for git.taler.net 132 133 Definition of Done 134 ================== 135 136 N/A 137 138 Alternatives 139 ============ 140 141 * Do not re-use the session ID mechanism but introduce some new concept. 142 This has the drawback of us needing additional tables and indicies, 143 and also the existing use of the session ID is very parallel to this one. 144 145 Drawbacks 146 ========= 147 148 * This exposes an order ID to anyone who knows the session ID. This is 149 clearly not an issue in this context, and for the existing uses of 150 the session ID it also seems clear that knowledge of the session ID 151 requires an attacker to have access that would easily also already 152 give them any order ID, so this seems harmless. 153 154 155 Discussion / Q&A 156 ================