README (53525B)
1 paivana tests 2 ============= 3 4 This directory contains five test programs: 5 6 reverse_proxy an integration suite for the reverse-proxy side of 7 paivana-httpd, driven by test_reverse_proxy.sh 8 paywall an integration suite for the paywall itself, driven 9 by test_paywall.sh against a real GNU Taler system 10 client_address a unit test for the client address the access 11 cookie is keyed on (test_client_address.c) 12 cookie_header a unit test for the `Set-Cookie` line paivana emits 13 for that cookie (test_cookie_header.c) 14 cookie_access a unit test for the access decision the cookie 15 value encodes, and for the `paivana_id` the order 16 is created under (test_cookie_access.c) 17 18 The reverse-proxy suite runs paivana-httpd with `-n` (paywall 19 disabled) so no merchant backend is required: it only verifies that the 20 proxy correctly forwards HTTP requests and responses. The paywall 21 suite is the other half -- everything `-n` switches off -- and needs an 22 exchange, a merchant backend and a bank, so it skips where those are 23 not installed. Everything below describes the reverse-proxy suite 24 except the sections at the end. 25 26 What gets built 27 --------------- 28 29 The test suite uses four diverse upstream HTTP server implementations 30 so that paivana is not exercised only against libmicrohttpd peers: 31 32 upstream_mhd C / libmicrohttpd (built always) 33 upstream_go Go (net/http) (built if `go` is found) 34 upstream_rs Rust (std::net) (built if `rustc` is found) 35 upstream_py Python (stdlib) (pure interpreter; needs python3 36 at `make check` time) 37 38 Two further, special-purpose upstreams are also built: 39 40 early_response_upstream 41 C / raw sockets, single-connection. Sends a 413 42 response immediately after reading the request 43 headers, BEFORE consuming the request body — used 44 to exercise paivana's handling of an early upstream 45 response that lands while the client upload is 46 still in flight. Writes the body byte count it 47 observed to a receipt file, as a diagnostic. 48 49 With `--no-drain` it additionally stops reading 50 once it has answered, and parks the connection with 51 the rest of the request queued in a deliberately 52 tiny receive buffer. That is what makes paivana's 53 outbound socket back up: measured on loopback, the 54 upstream holds ~12 KiB unread while ~240 KiB of the 55 request sits undeliverable in paivana's send buffer. 56 A proxy that waited for that write to finish before 57 acting on the response it already holds would 58 deadlock; without `--no-drain` the condition never 59 arises, because the upstream keeps reading. 60 61 stream_upstream 62 C / raw sockets, one process per connection. Serves 63 bodies far larger than memory, at a rate the driver 64 chooses, and in framings a conforming server library 65 will not emit: a declared `Content-Length` that is 66 not delivered, a chunked response with no terminating 67 chunk, a connection that answers and then goes silent 68 for ever. Also reads and verifies a request body, 69 optionally answering before it has finished. 70 71 Bodies are a deterministic function of their own byte 72 offset rather than stored data, so a 200 MiB case 73 costs no disk on either side. Deliberately not a 74 constant byte: a repeated character would pass a 75 comparison that duplicated or dropped a whole aligned 76 block, which is exactly the mistake a ring buffer 77 with wrong wrap arithmetic makes -- and exactly the 78 mistake this caught during development. 79 80 The four canned upstreams all implement the same endpoints (see 81 "Endpoints" below). Two test clients talk to paivana directly: 82 `pipeline_client`, which uses BSD sockets to pipeline requests, and 83 `stream_client`, which verifies a body against the same generated 84 pattern as it arrives and reports what it saw about the *framing* -- 85 whether the response was chunked, what `Content-Length` reached the 86 client, how long the first byte took relative to the last. Verifying 87 incrementally is the point: a body written to a file and compared 88 afterwards says nothing about whether paivana streamed it or assembled 89 it first. 90 91 They all bind 127.0.0.1 and nothing else. They are not hardened in 92 any way -- POST /echo reflects whatever body it is given and GET 93 /large/10485760 hands out 10 MiB per request -- and they have no 94 business being reachable from the network for the duration of `make 95 check'. They also all reject an argument that is not a port in 96 1..65535 rather than defaulting: a silent fallback binds a port the 97 driver is not waiting for, and the failure then surfaces five seconds 98 later as "did not start on port NNNNN", naming the wrong thing. 99 100 Layout of the driver 101 -------------------- 102 103 `test_reverse_proxy.sh` is the single test program automake runs. 104 For each available upstream (mhd / go / py / rs) it: 105 106 1. starts the upstream on its port (see "Ports used"), 107 2. starts paivana-httpd -n pointed at that upstream, 108 3. runs a battery of HTTP tests with curl, wget, and the raw-socket 109 pipelining client, 110 4. stops paivana and moves on to the next upstream. 111 112 Cross-cutting error-path tests (405, 413, 502) are also covered, and 113 the final case restarts paivana pointed at a dead port to exercise 114 upstream-failure handling. 115 116 What each test covers 117 --------------------- 118 119 Per-upstream battery (`run_battery`): 120 121 GET /hello happy-path GET, body proxied unchanged 122 GET /status/201 2xx response status is forwarded intact 123 GET /status/404 4xx response status is forwarded intact 124 GET /status/500 5xx response status is forwarded intact 125 HEAD /hello HEAD method: the status is forwarded 126 HEAD /large/131072 RFC 9110 section 9.3.2's one normative 127 requirement on HEAD -- "MUST NOT send 128 content in the response" -- on a path 129 that yields 128 KiB under GET, so there 130 is something to leak. Read off the 131 socket rather than through curl, which 132 discards a body a HEAD response has no 133 business carrying and would therefore 134 report the bug as a pass. 135 GET /large/131072 128 KiB response body arrives byte for 136 byte, compared against the 'A'..'Z' 137 cycle the upstreams generate rather 138 than merely counted. (Bodies are 139 buffered whole, not streamed: a length 140 that matches says nothing about a 141 buffer reassembled in the wrong order.) 142 POST /echo request body is forwarded unchanged; 143 body round-trip 144 POST /echo (128 KiB) the same in the request direction and 145 at a size that spans several reads: 146 128 KiB of random bytes posted and 147 compared with what comes back. POST 148 /upload below checks only the count the 149 upstream reports, so without this 150 nothing here would notice a request 151 body that arrived complete but corrupt. 152 POST /upload (64 KiB) large random POST upload; upstream 153 reports the byte count it saw 154 PUT /put PUT method + body forwarding 155 PATCH /patch PATCH method + body forwarding 156 (paivana sets CUSTOMREQUEST) 157 DELETE /item/1 DELETE method, 204 No Content 158 OPTIONS /hello OPTIONS method, Allow header survives 159 the round-trip 160 GET /echo-headers paivana adds the reverse-proxy headers 161 X-Forwarded-For, X-Forwarded-Proto, Via 162 Host: rewritten the Host the upstream sees is the 163 authority of DESTINATION_BASE_URL, not 164 the one the client dialed, and is 165 host[:port] and nothing else (RFC 9110 166 §7.2 — no userinfo, no path, no query). 167 Note this only covers the destination 168 URLs paivana will actually accept: 169 TALER_is_web_url() rejects userinfo and 170 IPv6-literal DESTINATION_BASE_URLs at 171 startup, so those cannot be reached 172 from the driver. 173 custom X-Test header arbitrary client request headers are 174 forwarded unchanged 175 X-Upstream response header upstream response headers survive the 176 round-trip back to the client, and the 177 value names the upstream this battery 178 was pointed at -- a restart that 179 silently kept the previous destination 180 would satisfy a presence check 181 182 Forwarding-header tests (run once): 183 184 no -f a client's own X-Forwarded-For / 185 -Proto / -Host must not reach the 186 upstream: paivana is the outermost 187 proxy and replaces them with what it 188 can see for itself. In particular the 189 scheme must come from the transport, 190 not from a header the client wrote -- 191 TALER_mhd_is_https() believes 192 X-Forwarded-Proto, so paivana asks MHD 193 about the TLS session instead. 194 -f, chain extension with -f paivana is behind a trusted 195 proxy: the inbound chain is preserved 196 and paivana's own peer appended to the 197 right, rather than the chain being 198 thrown away. Also covers a repeated 199 X-Forwarded-For arriving as two field 200 lines (RFC 9110 §5.3: one combined 201 header must reach the upstream). 202 -f, trusted -Proto / -Host the values a trusted proxy sent are 203 passed through unchanged. 204 unix socket the deployment the Debian packaging 205 ships. A Unix peer has no address, so 206 with -f the inbound chain is forwarded 207 unadorned (nothing is appended, and no 208 placeholder is invented -- the hop is 209 recorded in Via), and without -f no 210 X-Forwarded-For is emitted at all. 211 This is the only case that reaches the 212 address-less code paths. 213 RFC 7239 Forwarded the standardized header is handled like 214 X-Forwarded-For: extended under -f, 215 replaced without it. Since paivana 216 prefers it when both are present, its 217 for/proto/host must also reach the 218 X-Forwarded-* headers, or an origin that 219 speaks only those would be told the 220 proxy was the client. A chain 221 containing a hop X-Forwarded-For cannot 222 express (§6.3 "unknown") yields no 223 synthesized chain rather than one with a 224 hop silently missing. 225 unix socket, Forwarded unlike X-Forwarded-For, RFC 7239 can 226 name an address-less hop, so paivana's 227 own element reads for=unknown rather 228 than being omitted. 229 TRUSTED_PROXIES startup a policy that parses to nothing usable 230 (missing trailing ';', a /0 network, an 231 address of the wrong family, junk) must 232 abort startup rather than silently 233 trusting nobody; usable ones must start. 234 WHITELIST startup an expression regcomp(3) cannot compile 235 must abort startup rather than leave 236 paivana matching against an 237 uninitialised regex_t; usable ones must 238 start. Two of the refused cases are 239 the anchoring: "a)|(b" and "(a$|^b" do 240 not balance on their own, so wrapping 241 them in "^(%s)$" yields an alternation 242 that has climbed out of the group and a 243 whitelist matching far more than it 244 says. paivana compiles the value bare 245 first for that reason, which is what 246 these two reach. 247 248 The matching itself is still out of 249 reach here: the regexec sits behind the 250 paywall that `-n` switches off, and 251 without `-n` paivana needs a merchant 252 backend to serve it templates before it 253 will start at all. So the anchoring at 254 *match* time -- a WHITELIST of "/free/" 255 waiving payment for every URL merely 256 containing it -- has no end-to-end 257 regression test. 258 259 Cross-cutting tests (run once): 260 261 POST /.well-known/paivana with `-n` the payment endpoint answers 262 501 rather than falling through to the 263 proxy -- the one paywall-side branch 264 `-n` does not shield. Both ways to get 265 it wrong are silent: forwarding the POST 266 would hand the origin payment data it 267 has no business seeing, and claiming the 268 path for every method would shadow 269 whatever the origin serves there, so the 270 GET of the same path is checked to still 271 be forwarded. 272 273 TRACE method unsupported HTTP verb yields 405 Method 274 Not Allowed (paivana rejects it, the 275 upstream is never contacted) 276 2 MiB POST upload request bodies above the 1 MiB 277 MAX_REQUEST_SIZE are rejected with 278 413 Content Too Large 279 curl keep-alive x3 three GETs over one keep-alive TCP 280 connection all succeed 281 wget /hello third-party client interop 282 HTTP/1.1 pipelining (x4) four requests sent back-to-back on a 283 single TCP connection *before* reading 284 any response; responses must come back 285 in the same order and with the correct 286 status codes (200, 201, 200, 404). 287 This specifically tests that paivana's 288 per-request state machine and MHD's 289 keep-alive handling cooperate correctly. 290 upstream down with paivana pointed at a closed port, 291 clients receive 502 Bad Gateway with 292 the built-in "Bad Gateway" HTML body 293 early upstream response against early_response_upstream, a 294 768 KiB POST that the upstream answers 295 with a 413 BEFORE reading the body. 296 Paivana must forward that 413 to the 297 client rather than turning it into a 298 502 — the early-response path, i.e. 299 #UP_DRAINING. 300 Note that the upstream is NOT expected 301 to see the whole body: RFC 9110 §9.3 302 lets a client stop sending once it has 303 a final response, and libcurl does 304 (plain curl against this upstream 305 sends ~128 KiB of the 768 KiB and 306 stops). The receipt is a diagnostic; 307 the test only requires that it appear, 308 i.e. that the exchange finished 309 upstream-side. 310 early response, no drain the same, with --no-drain: having 311 answered, the upstream never reads 312 again, so paivana's outbound socket 313 stays full with a request it can no 314 longer finish sending. It must still 315 answer its own client, promptly, with 316 the upstream's 413. Bounded by 317 timeout(1) rather than curl --max-time 318 because the failure mode is a hang: 319 paivana's stall watchdog would 320 eventually turn it into a truncated 321 response, which must not be allowed to 322 look like a slow pass. The drain-mode 323 case above cannot catch this — the 324 upstream there keeps reading, so the 325 socket never stays full. 326 327 The streaming tests (`test_streaming`) 328 -------------------------------------- 329 330 Everything above would pass equally well against the fully-buffered 331 proxy this replaced: every body in it fits in one buffer. These 332 cases are about what is new — that a body is no longer bounded by 333 memory, and that it starts reaching the client before the origin has 334 finished sending it. 335 336 200 MiB, Content-Length five times the 40 MiB ceiling that used 337 to make this a 502 outright. Body 338 verified byte for byte, and the 339 origin's own Content-Length must reach 340 the client rather than one recomputed 341 from an assembled buffer. 342 200 MiB, chunked the same body without a declared 343 length; must stay chunked to the 344 client instead of being silently 345 converted. 346 chunked to an HTTP/1.0 an HTTP/1.0 client cannot be sent 347 client chunks, so the close of the connection 348 has to be the framing. 349 Range -> 206 Content-Range and the partial body pass 350 through the streamed path. 351 HEAD on a large resource MHD does not run the content reader for 352 a HEAD but does emit the size the 353 response was created with, so the 354 length the equivalent GET would have 355 had now reaches the client (RFC 9110 356 §9.3.2). Buffering could only ever 357 report 0 here. 358 204 / 304 no body either way; the 304 still 359 carries the length of the body it does 360 not send (RFC 9110 §8.6). 361 200 MiB upload, both the request body is streamed too, so 362 framings the origin sees it byte-exact and sees 363 the client's own framing reproduced -- 364 a declared length stays declared, 365 chunked stays chunked. 366 small POST the overwhelmingly common case, which 367 now takes the same path as the large 368 one. 369 chunked upstream stops the status is long gone by the time the 370 mid-stream origin gives up, so the only remaining 371 way to say "incomplete" is to close 372 without the terminating chunk. curl 18 373 is the client noticing. 374 upstream goes quiet MHD will not time this out (a suspended 375 connection is off its timeout lists) 376 and CURLOPT_TIMEOUT is deliberately 377 unset, so paivana's own stall watchdog 378 is the only thing that can end it. 379 upstream never answers distinct from an upstream that is not 380 there, which is a 502: this is a 504, 381 and the time-to-headers clock is what 382 tells them apart. 383 100 abandoned downloads the ownership handshake between MHD's 384 completion notifier and the content 385 reader's free callback runs on every 386 request now, so a mistake in it is a 387 use-after-free or a leak on all 388 traffic. RSS across a hundred 389 abandoned transfers is the cheap 390 detector; ASan is the thorough one. 391 abandoned upload a Content-Length was declared upstream 392 that can no longer be delivered; the 393 origin has to be told the request is 394 broken rather than left waiting. 395 early 413 during a only reachable because the request body 396 200 MiB upload is streamed: with it buffered first the 397 origin could not have answered before 398 seeing all of it. 399 trailers, 1xx what patch 0034 established, re-checked 400 on the streamed path: neither may be 401 merged into a response that has already 402 been queued. 403 404 The congestion tests (`test_congestion`) 405 ---------------------------------------- 406 407 `test_streaming` shows that a large body gets through intact. It does 408 not show that it got through *without being held in memory*, and every 409 case in it would pass against a version that quietly buffered the lot 410 -- so on their own they leave the central claim of the change untested. 411 These are the cases that test it. 412 413 Three things are measured that the client cannot see for itself: 414 415 paivana's VmRSS while a body many times the buffer size is in 416 flight. This is the bound, stated directly. 417 418 How long the *origin* took to write its body, which `stream_upstream` 419 reports per connection on stderr ("served target=... bytes=N ms=M"). 420 A proxy that buffers takes everything at line rate however slowly its 421 own client reads; one that relays can only take what the client has 422 made room for. From the client end the two are indistinguishable, 423 which is why the origin has to report its own timing. 424 425 paivana's CPU time across an interval when nothing is moving. 426 Busy-waiting is the classic failure of a suspend/resume design and is 427 otherwise invisible: the transfer still completes, correctly, with a 428 core pinned for its duration. 429 430 Rate limits (`--read-rate`, `--upload-rate` on the client, `rate=` on 431 the upstream) are what make any of this reproducible. On loopback with 432 both ends going flat out, the kernel socket buffers absorb everything 433 and no ring ever fills. 434 435 The RSS bounds are skipped under `--enable-sanitizers`. ASan's 436 quarantine -- the thing that lets it catch a use-after-free -- holds 437 freed chunks rather than reusing them, so RSS there tracks total bytes 438 moved instead of bytes held: the 64 MiB case grows ~58 MB instrumented 439 against ~0.5 MB not, for identical code. The transfers still run and 440 LSan still watches them; the pacing and CPU assertions are unaffected 441 and are checked in both builds. 442 443 These sizes are deliberately *not* divided by PAIVANA_TEST_SCALE. Each 444 case is rate-limited, so its duration is set by the rate and not by the 445 size, and the sanitised build is no slower for them. Scaling them 446 would also break the pacing assertions: the kernel socket buffers hold 447 a fixed couple of megabytes however small the body is, so at a 448 twentieth of the size the origin legitimately finishes well ahead of 449 the client and "was it throttled" stops having a stable answer. 450 451 64 MiB through a slow peak RSS over baseline must stay within a 452 client few megabytes. Measured: +552 kB across 453 20 samples, against a hard 502 for this 454 size before the change. Checked against a 455 build with the ring cap removed, which 456 grows 12544k -> 78312k for the same body: 457 the bound does detect buffering. 458 upstream pacing the same transfer from the other end: the 459 origin's own elapsed time must track the 460 client's rather than finishing in a 461 fiftieth of it. Measured: 3597 ms to 462 write 64 MiB to a client that read for 463 4000 ms, where buffering would have taken 464 under 100 ms on loopback. 465 upload pacing the same assertion in the request 466 direction, against /sink. 467 32 concurrent throttled the per-request cost is what multiplies, 468 downloads so this is where a bound that holds for 469 one request and not for thirty-two shows. 470 Mixed rates, so the fast ones finish while 471 the slow ones are still going. Measured: 472 10.6 MB of growth for 32 x 8 MiB in 473 flight, about 339 kB each. Runs with 474 PER_IP_CONNECTION_LIMIT lifted, which is 475 otherwise exactly 32 and would have the 476 case measure connection limiting instead. 477 idle transfer an origin dribbling 200 B/s leaves paivana 478 with nothing to do for ~5 s. CPU must 479 stay near zero (measured: 1 jiffy, i.e. 480 10 ms, over 5107 ms), and the first byte 481 must still arrive at once -- measured at 482 1 ms -- rather than after the last. 483 1 KiB receive buffer makes libcurl drain paivana's socket in 484 tiny units, so MHD's content reader is 485 called hundreds of times where the default 486 buffer needs a handful -- each one a 487 chance for the ring to empty and the 488 connection to suspend and resume. Chunked, 489 so MHD's chunk framing is re-entered every 490 time. 491 slow at both ends neither side able to keep up with the 492 other on one request. Both rings spend 493 the transfer alternately full and empty 494 and the two halves of the state machine 495 have to interleave without deadlocking or 496 dropping a byte. 497 498 The base64url cross-check (`test_base64url.sh`) 499 ----------------------------------------------- 500 501 The paivana ID is `<expiration>-<base64url(sha256(...))>`. The daemon 502 builds it with `GNUNET_STRINGS_base64url_encode()`; the browser rebuilds 503 it in `paywall.js` to recognise the payment it has just made. Nothing 504 in either program forces the two encoders to agree, and if they do not, 505 the ID never matches, the payment appears not to go through, and 506 neither side logs anything wrong. 507 508 They have already disagreed twice. Once on the decode side: the daemon 509 emits the RFC 4648 section 5 (URL-safe) alphabet, and the browser fed it 510 to `atob()`, which only knows section 4 and throws on `-` or `_` -- at 511 module scope, so the whole script died and the paywall could not be 512 paid. Once on the encode side: the browser used 513 `Uint8Array.prototype.toBase64`, which is a 2024-25 addition (Firefox 514 133, Safari 18.2, Chrome 140) and is simply not a function on anything 515 older. Two bugs of the same shape in one file is what this test is for. 516 517 `base64url_vectors` prints 369 vectors as the *daemon* produces them -- 518 every length from 0 to 96, so both amounts of padding and none are 519 crossed; every single byte value, because `-` and `_` are only 520 reachable from particular high bit patterns and are exactly the two 521 characters the section 4 alphabet spells differently; and sixteen 522 32-byte blocks, that being the size which actually occurs. 523 `test_base64url.sh` lifts `base64url()` out of `paywall.js` by matching 524 braces -- rather than keeping a copy here, which would be a second 525 implementation to hold in step, and holding implementations in step by 526 hand is the thing that failed -- and compares. It refuses to pass on 527 fewer than 300 vectors, so a generator that broke would fail rather 528 than trivially agree. Skips (77) without node. 529 530 Checked by breaking it: with the alphabet translation removed from 531 `paywall.js`, it reports `paywall.js gave "WH2ix+wRNls", the daemon 532 gives "WH2ix-wRNls"`. 533 534 The client_address unit test 535 ---------------------------- 536 537 `test_client_address.c` covers PAIVANA_HTTPD_resolve_forwarding(), 538 the single walk over the forwarding chain that decides both the client 539 address PAIVANA_HTTPD_get_client_address() hands to the cookie MAC and 540 the scheme and authority PAIVANA_HTTPD_get_base_url() rebuilds the 541 website string from. That function is deliberately pure — it takes 542 the socket peer, the ordered field lines of each forwarding header and 543 the trust configuration, and nothing else — so the whole policy is 544 reachable without an MHD connection; the MHD half is a thin adapter. 545 546 The access cookie is an HMAC over (expiration, website, client 547 address). A host therefore has to produce the *same bytes* however 548 paivana learns its address, or the cookie it was issued silently stops 549 verifying and the visitor is asked to pay again. The test asserts: 550 551 - an X-Forwarded-For value and the socket address of the same host 552 yield identical bytes (including ::ffff:a.b.c.d from a dual-stack 553 listener versus a.b.c.d from a proxy), 554 - alternative spellings of one address are one identity 555 ("::1" / "0:0:0:0:0:0:0:1", upper/lower case hex), 556 - a value that is not a bare IP address is refused rather than 557 turned into an identity of its own (port suffixes, brackets, RFC 558 7239 "unknown"/"_hidden", hostnames, zone ids, junk), 559 - a cookie issued for one host is not accepted for another. 560 561 A table-driven group then covers the walk itself. `-f` means "we are 562 behind a trusted reverse proxy", so the socket peer is trusted 563 implicitly and TRUSTED_PROXIES / TRUSTED_PROXIES6 name the *additional* 564 hops further out; the walk steps leftwards over a node only while that 565 node is trusted and stops at the first one that is not. The rows 566 cover: 567 568 - without `-f`, the socket peer wins even with every forwarding 569 header present, 570 - a single proxy and a single element, in both spellings, 571 - two and three proxies with only some of them listed, and an 572 untrusted node in the middle, which stops the walk where it should, 573 - a chain of nothing but trusted hops, where the leftmost is all 574 there is, 575 - IPv4, bracketed IPv6, IPv6 with a port, RFC 7239 §6.3 "unknown" 576 and an obfuscated identifier, 577 - repeated field lines of one header and a field line that is itself 578 a list (RFC 9110 §5.3), 579 - quoted strings with escapes, and an unterminated one, which used 580 to be read past the end of the header, 581 - malformed, empty and whitespace-only headers, all of which fall 582 back to the socket peer rather than losing it, 583 - `Forwarded` winning where both headers are present, 584 - a chain of 2500 elements, which is refused outright rather than 585 walked: every element used to be located by rescanning the header 586 from byte 0. 587 588 A second table covers what the base URL is built from: `proto=` and 589 `host=` taken from the same element the address came from, the 590 X-Forwarded-Proto/-Host/-Port fallbacks, an X-Forwarded-Host that 591 already carries a port together with an X-Forwarded-Port (which must 592 not yield "example.com:8443:8443"), ports re-rendered rather than 593 echoed, and hosts and schemes that are refused because they are not 594 one. 595 596 A further group covers the rendering back out — a `for=` identifier, 597 an X-Forwarded-For chain, and RFC 7239 §4 values, where a parameter 598 that would otherwise splice a second forwarded-element into a header 599 we build is either quoted or reported as absent. 600 601 A separate group pins the behaviour of GNUnet's 602 GNUNET_STRINGS_parse_ipv{4,6}_policy() that load_trusted_proxies() 603 compensates for: the mandatory trailing ';', the v4/v6 disagreement 604 about spaces, the two ways those parsers return "nothing usable" 605 without returning NULL (a /0 network, which is indistinguishable from 606 the list terminator, and an address of the wrong family), and the one 607 way they return "usable, but not what was written" — a final entry 608 without its ';', or anything after the last ';', is dropped and the 609 prefix reported as success, which is why the loader counts separators. 610 If upstream ever fixes these, this group is what says so. 611 612 The startup validation built on top of that is in the integration 613 suite instead, since it is about whether the daemon comes up. 614 615 It links paivana-httpd_helper.c and paivana-httpd_cookie.c directly 616 and supplies the daemon globals itself, so it needs no MHD connection 617 and no merchant backend. The integration suite cannot cover any of 618 this: with `-n` the cookie path is never reached, so the client 619 address is never computed. 620 621 The cookie unit tests 622 --------------------- 623 624 The same applies to the two cookie tests, and for the same reason: 625 `-n` sets do_forward before the request is looked at, so nothing in 626 the integration suite ever mints or checks a cookie. Both link only 627 paivana-httpd_cookie.c. 628 629 `test_cookie_header.c` is about the header paivana emits, i.e. about 630 whether the credential the client just paid for ever comes back: the 631 `Path` re-encoding (the browser matches against the encoded request 632 path, while the URL paivana holds has been decoded by MHD), the RFC 633 6265 §4.1.1 grammar the attribute has to satisfy, attribute injection 634 through a path containing ';', `Secure`, and the `Max-Age` floor that 635 keeps a sub-second access from being deleted on arrival. 636 637 `test_cookie_access.c` is about the decision made when it does come 638 back. The cookie is a bearer token we hand to the party most 639 interested in widening it, so each of the three things it is minted 640 for -- expiration, website, client address -- is checked to be inside 641 the MAC and re-checked on presentation, including the obvious attempt: 642 reading the expiration off the value and writing a later one. Each of 643 the ways check_cookie() can reject a value has its own case, so that a 644 malformed value ends in a refusal rather than in a read past the end 645 of a string the client chose. A separate group covers the values that 646 are not malformed at all but merely respelled -- a leading '+', a 647 leading space, a leading zero, junk between the seconds and the '-' -- 648 each of which decodes to the same seconds and the same hash as a 649 cookie we really issued, and so is a second live spelling of one 650 credential unless the parser refuses it. `-g` is covered here and 651 nowhere else. 652 653 The `paivana_id` is pinned against a golden vector computed 654 independently from the definition the paywall page implements 655 (src/frontend/paywall.js, makePaivanaId()). Neither side ever sends 656 it; both derive it from their own copy of (nonce, website, expiration) 657 and expect the other to have got the same string, so the two 658 implementations agreeing IS the protocol, and a change on either side 659 that this vector does not survive means every order is created under 660 an id the other side will not look for. 661 662 663 The benchmark (`benchmark.sh`) 664 ------------------------------ 665 666 Not a test -- it asserts nothing about correctness and its result 667 depends on the machine. It answers two questions: how much 668 throughput does putting paivana in front of an origin cost, and how 669 fast is paivana on its own when the paywall turns a client away? 670 671 meson test --benchmark proxy_overhead -C build # the first 672 meson test --benchmark paywall_page -C build # the second 673 674 Both are registered with meson's `benchmark()` rather than `test()`, 675 which is what keeps them out of `make check`: `meson test` does not 676 run benchmarks. They are two entries rather than one so that they 677 skip independently -- the paywall arm needs things the proxy arm does 678 not, and should not be able to take it down with it. Run the script 679 directly for the knobs -- `-c` clients, `-s` page size, `-d` seconds, 680 `-m direct|proxy|paywall|both|all`. It exits 77 when rustc was not 681 available to build upstream_rs, or when the build is sanitized (those 682 timings measure ASan), and 1 if any request failed, since a run with 683 failures has not measured throughput. 684 685 N curl workers fetch a fixed-size page for a fixed time. In `direct` 686 they fetch it straight from upstream_rs, in `proxy` through paivana 687 in front of that same upstream_rs, and in `paywall` they fetch 688 paivana's own 402 page with no upstream in the path at all. For each 689 arm it reports requests, requests/s, MB/s (10^6), the page size and 690 the server processes' CPU time; then the proxy/direct ratio, and the 691 paywall/proxy one. Nothing touches disk: the upstream's page comes 692 from a buffer it fills once at startup -- it used to regenerate it a 693 byte at a time per request, which was real work charged to the arm 694 that has no proxy in it -- the paywall page comes out of paivana's 695 response cache, and the clients discard every body. 696 697 The expected size is measured from a warm-up request rather than 698 assumed, because in paywall mode nothing here knows it up front: it 699 is whatever `paywall.en.must` renders to (50300 bytes as of writing). 700 That measured size is then what every later response is checked 701 against, so a short body counts as a failure rather than as 702 throughput that was not achieved. 703 704 Three properties of the setup shape the number, and all three make 705 the proxy look better than it is, so the reported ratio is a floor: 706 707 - Every request gets a fresh TCP connection in *both* arms. That 708 is forced, not chosen: upstream_rs answers one request per 709 connection and closes, so the direct arm cannot keep-alive at 710 all, while paivana's client side happily would -- MHD strips the 711 upstream's hop-by-hop `Connection: close` and decides the client 712 connection's fate itself. Measured: without the clients sending 713 `Connection: close`, curl's second request through paivana 714 reports num_connects=0 and the same request direct reports 1. 715 So the clients send it, and both arms are charged one TCP setup 716 per request. 717 718 - curl's own process startup (12.5 ms on the machine this was 719 written on -- it links openssl, nghttp2, brotli, zstd, ldap) is 720 amortised by handing each curl invocation a batch of URLs. The 721 batch size converges at run time rather than being computed from 722 the page size, because how long a batch takes depends on the 723 per-worker request rate, which is what is being measured: a size 724 picked up front overshot a 3 s run by 7% at `-c 32`. At a fixed 725 batch of 8 the startup was 44% of the run and the reported rate 726 came out at half the truth. 727 728 Both of the above are per-request constants added to *both* 729 arms, so they pull the ratio toward 1. 730 731 - paivana is single-threaded by construction -- one GNUnet 732 scheduler driving MHD and libcurl -- and upstream_rs spawns a 733 thread per connection, so on a multi-core box the direct arm may 734 use every core and the proxy arm may not. That is a real 735 property of paivana rather than an artefact of the harness, but 736 it does make the ratio a function of the core count, which is 737 why CPU seconds and `nproc` are printed with it. 738 739 A fourth applies to paywall mode only: the paywall page is not the 740 `-s` size, so that arm is comparable to the others in requests/s and 741 not in MB/s. The script prints both page sizes next to the ratio and 742 does not offer a MB/s one. 743 744 For orientation, one run on a 24-core machine at the defaults (8 745 clients, 64 KiB page, 10 s): 746 747 direct 34932 req/s 2289 MB/s upstream_rs 3.07 cores 748 proxy 6816 req/s 447 MB/s paivana 0.97 cores 749 paywall 33993 req/s 1710 MB/s paivana 0.90 cores 750 751 so 0.20x of direct through the proxy, and 4.99x the forwarded rate 752 for the paywall page (50300 bytes). paivana is at 0.97 cores 753 forwarding: it is saturating its single thread, which is the bound 754 that matters. Sweeping the paywall arm shows the same bound from the 755 other side -- 26852 req/s at `-c 4` and 0.81 cores, 33504 at `-c 8`, 756 36049 at `-c 16` and 0.98 cores, 35892 at `-c 32` -- i.e. it stops 757 scaling exactly where the thread runs out, at about 36k req/s. 758 759 Do not quote those figures. They are stable to a couple of percent 760 when the machine is quiet, but an *earlier* set on the same 24-core 761 box read 13375 req/s direct and 4707 through paivana, i.e. 0.35x 762 rather than 0.20x, because the direct arm was then getting only 1.5 763 cores instead of 3.1. The clients are a bash loop forking curl and 764 they compete with the servers for the machine, so under contention 765 the fastest arm loses the most and the ratio flatters the proxy. The 766 run's own CPU numbers are what tell you which regime you were in. 767 768 Why the paywall arm needs a merchant backend, and why a stub is 769 honest here. paivana does not open its listen socket until it has 770 fetched a template from a merchant backend 771 (PAIVANA_HTTPD_load_templates -> templates_ready -> 772 PAIVANA_HTTPD_serve_requests), so paywall mode cannot measure 773 anything without one. It starts `merchant_stub`, which answers the 774 two GETs of that startup exchange -- shaped as 775 merchant_api_get-private-templates{,-TEMPLATE_ID}.c parse them, with 776 the contract test_paywall.sh POSTs to a real backend -- and nothing 777 else. That is the whole of what a real backend would do here: the 778 template is fetched once, the page is rendered locally from it, and 779 the rendered MHD_Response is cached per (language, encoding) in 780 load_paywall(), so from the first measured request onwards a live 781 merchant is exactly as idle as the stub. This benchmark never buys 782 anything, so no code path that can tell the two apart is reached. 783 The stub does check the bearer token, since paivana building that 784 header out of MERCHANT_ACCESS_TOKEN is the one part of the exchange 785 that could silently regress; a 401 makes paivana refuse to start 786 rather than start against a configuration nobody would deploy. To 787 check the claim instead of taking it, set PAIVANA_BENCH_MERCHANT_URL 788 (and PAIVANA_BENCH_MERCHANT_TOKEN) at a live backend carrying a 789 `paivana` template -- named `premium`, or whatever 790 PAIVANA_BENCH_TEMPLATE_ID says. 791 792 Like test_paywall.sh, paywall mode stages `paywall.en.must` into a 793 scratch prefix and points PAIVANA_PREFIX at it, rather than requiring 794 `make install` for a page that lives in the build tree. 795 796 The likeliest cause of failures here is not paivana: one connection 797 per request against a fixed server port pins the 4-tuple for the 798 TIME_WAIT duration, and a few runs back to back can fill the local 799 ephemeral range (28k ports by default against ~13k connections per 800 run). The script says so when it sees transfers that never 801 connected. 802 803 804 Environment variables 805 --------------------- 806 807 The driver script honors: 808 809 PAIVANA_HTTPD path to paivana-httpd (default: the in-tree build) 810 SRCDIR directory containing the upstream sources and the 811 conf template (default: dirname of the script) 812 BUILDDIR directory containing upstream_mhd, pipeline_client, 813 upstream_go, upstream_rs (default: $PWD) 814 KEEP_TMP=1 do not delete the scratch dir on exit 815 PAIVANA_PORT_BASE 816 first port of the block the suite binds 817 (default 18400); see below 818 819 benchmark.sh honors the first four of those, plus: 820 821 PAIVANA_BENCH_PORT_BASE 822 first port of its own block (default 18600) 823 PAIVANA_BENCH_MERCHANT_URL 824 a live merchant backend for paywall mode, instead 825 of starting merchant_stub 826 PAIVANA_BENCH_MERCHANT_TOKEN 827 bearer token for it 828 PAIVANA_BENCH_TEMPLATE_ID 829 template to ask that backend for (default 830 `premium`, which is what merchant_stub serves) 831 832 Ports used 833 ---------- 834 835 Every port is an offset off PAIVANA_PORT_BASE, which defaults to 18400 836 -- the 184xx / 185xx range, chosen to avoid collisions with real 837 services. The suite checks all ten before it starts anything and 838 exits 77 (meson reads that as SKIP) if one of them is taken, naming 839 it. 840 841 That check is not a formality. Readiness used to be "does something 842 accept on this port", which is a different question from "did our 843 child come up": a paivana that lost the bind to a squatter -- most 844 often a stale one of its own from an earlier run -- read as started, 845 and the suite then ran its checks against the wrong process. With a 846 stale paivana of a different vintage they even pass. The startup 847 validation cases are the worst affected, since those decide "refused" 848 from exactly that probe. 849 850 base + 1 (18401) upstream_mhd 851 base + 2 (18402) upstream_go 852 base + 3 (18403) upstream_py 853 base + 4 (18404) upstream_rs 854 base + 5 (18405) early_response_upstream 855 base + 6 (18406) early_response_upstream --no-drain 856 base + 7 (18407) truncating upstream (short-body test) 857 base + 8 (18408) stream_upstream (streaming tests) 858 base + 99 (18499) dead port (for "upstream down" test) 859 base + 100 (18500) paivana-httpd 860 861 Move the base to run the suite in two checkouts at once, or beside a 862 paivana you are debugging: 863 864 PAIVANA_PORT_BASE=18700 meson test -C build reverse_proxy 865 866 benchmark.sh binds its own three ports off a separate base, so it can 867 run beside the suite: PAIVANA_BENCH_PORT_BASE, default 18600, giving 868 18601 for upstream_rs, 18602 for paivana and 18603 for merchant_stub. 869 It checks the ones the chosen `-m` actually needs, and skips the same 870 way. The `paywall_page` benchmark entry passes `-b 18610` so that 871 the two entries cannot collide if anyone runs the benchmarks in 872 parallel, which is not meson's default but is one flag away. 873 874 Endpoints (implemented by every upstream) 875 ----------------------------------------- 876 877 GET /hello text "Hello from <name>\n" 878 GET /status/NNN respond with status NNN and a trivial 879 text body "status NNN\n" 880 GET /large/N N bytes of 'A'..'Z' repeating 881 GET /slow/N sleep N ms, then "slept\n" 882 GET /echo-headers text listing of received request 883 headers, "Key: Value\n" per line 884 POST /echo body is echoed verbatim 885 POST /upload "Received N bytes\n" 886 PUT /put "PUT received N\n" 887 PATCH /patch "PATCH received N\n" 888 DELETE /item* 204 No Content 889 OPTIONS * 204 + Allow: GET, POST, PUT, ... 890 891 Every response also carries an `X-Upstream:` header whose value 892 identifies which server handled it (mhd, go, py, rs); the client 893 test cases use it to confirm that responses are coming back from 894 the expected backend. 895 896 897 The paywall suite 898 ----------------- 899 900 `test_paywall.sh` covers what `-n` hides. It puts a real GNU Taler 901 system behind paivana-httpd -- a fakebank, an exchange and a merchant 902 backend, started with `taler-unified-setup.sh` exactly as the merchant 903 and anastasis suites start theirs -- creates a Paivana template on the 904 merchant instance, buys access with `taler-wallet-cli`, and checks what 905 the daemon does with the result. 31 checks, about 25 seconds. 906 907 It skips (exit 77) rather than failing when the environment cannot 908 support it: no `taler-unified-setup.sh`, `taler-wallet-cli`, 909 `taler-merchant-httpd`, `jq`, `python3` or PostgreSQL, no built paywall 910 template, or one of its ports already in use. A skip names what was 911 missing. 912 913 Ports. paivana's own two move with `PAIVANA_PORT_BASE` (+110 and 914 +111), but the Taler system's are fixed at 9966 (merchant), 8081 915 (exchange) and 8082 (bank) -- the same ones the merchant suite uses, so 916 the two cannot run at once and this suite skips when they are busy. It 917 also wants a PostgreSQL database named `paivanacheck`, which it creates 918 if it can; `talercheck` is deliberately not reused, since the merchant's 919 own tests would then be clobbering these tables and vice versa. 920 921 The paywall template is staged into a throwaway prefix and reached 922 through `PAIVANA_PREFIX`, so a build tree is enough and `make install` 923 is not required. 924 925 Why the client half is written out by hand. The paywall page computes 926 a payment identifier from (nonce, website, expiration) and the daemon 927 computes the same identifier independently; neither ever sends it to 928 the other, so the two agreeing IS the protocol. `paivana_id.py` 929 re-derives it -- and the Crockford base32 encoding of the nonce -- from 930 the definition `src/frontend/paywall.js` implements, which is what makes 931 this a test of both ends rather than of one end twice. It agrees with 932 the golden vector in `test_cookie_access.c`, which was computed the same 933 way; if you change the derivation, three places have to move together. 934 935 What it covers, in order: 936 937 * the unpaid path: 302 to the paywall, the template named in the 938 Location and the website base64url-encoded in the fragment, the 939 402 page itself, its `Paivana:` pay-template URI and its CSP; 940 * the whitelist, and specifically that a WHITELIST expression is 941 anchored at both ends -- `/echo-headers` waives that path and not 942 `/x/echo-headers` or `/echo-headers/x`. The regexec that decides 943 this sits behind the paywall, so no other test in the tree can 944 reach it; 945 * the redemption endpoint's refusals: a body missing its fields, a 946 nonce of the wrong length, an order the merchant never saw; 947 * a real payment, redeemed for a real access cookie, and that cookie 948 opening the URL it was minted for and no other; 949 * that rewriting the expiration in the cookie value invalidates it 950 (the expiration is the KDF salt) and that a malformed cookie is 951 refused rather than mis-parsed; 952 * an order bought for a DIFFERENT fulfillment URL under the session 953 we then claim. The merchant sells it, the session lookup succeeds, 954 and the only thing between that and a cookie for a page nobody paid 955 for is paivana comparing the contract's fulfillment URL against the 956 website claimed. This is the one case that reaches that comparison: 957 naming another website in the redemption changes the payment 958 identifier, so every simpler attempt is refused earlier, by the 959 session lookup; 960 * that redemption is repeatable from anywhere, which is deliberate 961 (design document 076, "Payment buys access, not a seat"). The 962 check is here so that a change of mind about it surfaces as a test 963 failure rather than as a silent change of policy. 964 965 The checks were verified not to be vacuous by breaking the code under 966 them, one property at a time: dropping the `^(...)$` wrapping around 967 WHITELIST turns the two anchoring cases red (`/x/echo-headers` reaches 968 the origin); dropping the website from the cookie's keyed hash lets the 969 paid cookie open `/item` as well; and skipping the fulfillment-URL 970 comparison lets an order bought for `/elsewhere` mint a cookie for 971 `/item`.