README (14077B)
1 GNU libmicrohttpd -- fuzzing testsuite (src/testzzuf) 2 ===================================================== 3 4 The testcases in this directory require zzuf, and most of them also 5 require socat. 6 7 zzuf randomly mangles the TCP byte stream that reaches MHD. The goal is 8 to expose problems in MHD's error handling by introducing random syntax 9 errors into otherwise valid HTTP traffic. 10 11 There are two modes of operation: 12 13 * direct mode (zzuf_test_runner.sh) 14 zzuf runs the test binary itself and intercepts the data that MHD 15 reads from its accepted sockets. Only possible when MHD uses 16 accept() (not accept4()) and when no sanitizer is active, because 17 zzuf and the sanitizers both intercept the standard library. 18 19 * socat mode (zzuf_socat_test_runner.sh) 20 zzuf runs a socat relay that listens on 127.0.0.121:10121 and 21 forwards the mangled stream to MHD on 127.0.0.1:4010. The test 22 binary (and therefore MHD) runs outside of zzuf, so this mode works 23 with sanitizers and with accept4(). configure selects this mode 24 automatically when it is required (FORCE_USE_ZZUF_SOCAT). 25 26 Because the byte stream is corrupted at random, the testcases do NOT 27 check that whatever the client receives is what was expected. A testcase 28 "fails" when MHD crashes, aborts, hangs, reports an internal 29 inconsistency through its callbacks, or when a sanitizer fires. 30 31 32 What is being tested 33 -------------------- 34 35 Every test starts a series of MHD daemons -- one per polling mode that 36 the current build supports (internal select(), internal select() with 37 thread-per-connection, internal poll(), internal poll() with 38 thread-per-connection, internal epoll, external polling, external polling 39 without thread safety) -- and runs a client against each of them. 40 41 Clients come in two flavours: 42 43 * libcurl clients, for everything libcurl can express; 44 * a small raw socket client (mhd_zzuf_common.c), for everything libcurl 45 cannot express: chunk extensions, request pipelining, header-less 46 requests and hand-written "Authorization:" headers. 47 48 Both flavours use 127.0.0.101 as their source address, while MHD listens 49 on 127.0.0.1 (and socat on 127.0.0.121). zzuf is told to fuzz only the 50 traffic of the server side address, so the data the *clients* read is 51 left alone. 52 53 54 The daemon option matrix 55 ------------------------ 56 57 Historically all fuzzing tests here used essentially one set of daemon 58 options. Several MHD code paths are only reachable with other options, 59 most importantly the read-buffer "shift back" in get_req_headers(), which 60 is only executed when the connection memory pool is smaller than 61 MHD_BUF_INC_SIZE (1500 bytes). 62 63 mhd_zzuf_common.c therefore defines a matrix of option profiles: 64 65 name mem_limit conn_limit timeout discipline 66 ---------------- --------- ---------- ------- ------------------ 67 plain - - 2 0 68 pool-128 128 - 2 0 69 pool-256-strict 256 - 2 1 70 pool-512-lax 512 - 2 -1 71 pool-1024-conn 1024 4 2 2 72 pool-1400-hard 1400 2 1 3 73 pool-768-loose 768 - 1 -3 74 legacy-strict 192 - 2 1 (STRICT_FOR_CLIENT) 75 legacy-lax 320 - 2 -1 (STRICT_FOR_CLIENT) 76 77 "discipline" is MHD_OPTION_CLIENT_DISCIPLINE_LVL, except for the two 78 "legacy" profiles which use the older MHD_OPTION_STRICT_FOR_CLIENT so 79 that the option translation code is covered as well. 80 81 Every daemon that a test starts uses the next profile of the matrix, so a 82 single run of one test program sweeps the whole matrix. 83 84 MHD_OPTION_SERVER_INSANITY is deliberately *not* used: in this version of 85 MHD the only value of "enum MHD_DisableSanityCheck" is MHD_DSC_SANE (0), 86 so the option cannot actually disable any check. 87 88 89 The test programs 90 ----------------- 91 92 All programs decide what to do from their own name (argv[0]). A "10" 93 in the name selects HTTP/1.0, a "_close" suffix makes MHD close the 94 connection after every reply, and a "_hostile" marker enables the option 95 matrix described above. 96 97 test_get, test_get_chunked, test_post, test_post_form, test_put, 98 test_put_chunked, test_put_large, test_get_long_uri, 99 test_get_long_header (+ "_close" and "10" variants) 100 The original tests: well-formed libcurl traffic, single set of 101 daemon options. 102 103 test_get_hostile, test_get_chunked_hostile, test_post_hostile, 104 test_put_chunked_hostile, test_put_large_hostile 105 The very same client/server exchanges, but sweeping the daemon 106 option matrix. 107 108 test_get_args (+ "_close", "10") 109 GET requests with hostile query strings: arguments without '=', 110 empty names, empty values, trailing '&', percent-encoded and 111 invalid percent-encoded names, duplicated names and a very long 112 tail argument. All daemons of this test use a connection memory 113 pool smaller than MHD_BUF_INC_SIZE. 114 In addition to the libcurl requests, the raw client sends 115 *header-less* HTTP/1.0 requests: libcurl always sends at least a 116 "Host:" header, so with libcurl alone the last parsed element of a 117 request header is always a header and never a query argument. 118 119 test_get_chunked_ext, test_put_chunked_ext (+ "_close") 120 Chunked request bodies whose chunk-size lines carry chunk 121 extensions: "name=value", flags without a value, quoted-string 122 values containing ';', several extensions on one line, extensions 123 on the terminating "0" chunk, extensions after bad whitespace, an 124 extension line longer than the whole memory pool, and trailers 125 (footers) after the last chunk. The body is always split across 126 two send() calls so that MHD sees partial chunk-size lines. 127 Every third iteration appends a pipelined follow-up request, so a 128 chunk-extension line that is not consumed correctly shows up as a 129 request boundary de-synchronisation. 130 test_get_chunked_ext additionally makes MHD answer with a chunked 131 reply that carries a footer. 132 These tests also run a small deterministic *self-check*, see below. 133 134 test_pipelined (+ "_close") 135 Two to five complete requests written with a single send() call, 136 mixing GET/HEAD/POST/PUT, bodies with Content-Length, chunked 137 bodies, and a leading empty line. Every other iteration the batch 138 is split in the middle of a request. 139 140 test_digest_auth (+ "_close") [only if ENABLE_DAUTH] 141 libcurl performs a complete Digest handshake, so zzuf's bit flips 142 land inside real "WWW-Authenticate:" and "Authorization:" headers. 143 The raw client additionally sends hand-crafted "Authorization: 144 Digest" headers with an unknown "algorithm" token, an empty 145 "algorithm", a "-sess" algorithm, an over-long "response" (quoted 146 and unquoted), an over-long hex userhash, an over-long username, 147 a missing "response", garbage in "nc", an unterminated quoted 148 string and a bare "Digest" scheme without parameters. 149 The variants that have to survive MHD's nonce check are built from 150 a nonce that the raw client first obtains from MHD's own 401 reply. 151 152 test_basic_auth (+ "_close") [only if ENABLE_BAUTH] 153 libcurl performs a Basic handshake; the raw client adds truncated 154 Base64, characters outside the Base64 alphabet, payloads without 155 the ':' separator, payloads decoding to binary zeros, padding in 156 the middle, an empty value, a 4 KiB payload, whitespace inside the 157 payload and an unknown authentication scheme. 158 159 Automake conditionals used by this directory: 160 FORCE_USE_ZZUF_SOCAT, USE_COVERAGE, VHEAVY_TESTS (already used before) 161 ENABLE_DAUTH, ENABLE_BAUTH (new here) 162 All of them are defined in configure.ac. The directory as a whole is 163 only built when RUN_ZZUF_TESTS is true (see src/Makefile.am), which in 164 turn requires --enable-heavy-tests and a usable libcurl. 165 166 167 The deterministic self-check 168 ---------------------------- 169 170 Bit flipping can show that a parser crashes, but it cannot show that a 171 parser produces the *wrong result*. test_get_chunked_ext and 172 test_put_chunked_ext therefore also verify, once per run, that a chunked 173 body with chunk extensions is decoded byte-exactly and answered with 174 "200". 175 176 That verification obviously must not be fuzzed. In socat mode zzuf only 177 fuzzes the socat relay, so MHD's own port is a clean channel: the 178 self-check connects to MHD directly, bypassing socat. In direct mode 179 every byte that MHD accepts is fuzzed and no clean channel exists, so the 180 self-check is silently skipped there. Since socat mode is mandatory 181 whenever MHD is built with sanitizers, the sanitizer workflow below 182 always includes the self-check. 183 184 185 Running under the sanitizers 186 ---------------------------- 187 188 ./configure --enable-heavy-tests --enable-asserts \ 189 --enable-sanitizers=address,undefined 190 make 191 make -C src/testzzuf check 192 193 Enabling sanitizers automatically forces the socat mode (see 194 configure.ac), because zzuf and the sanitizers cannot both intercept the 195 standard library calls. 196 197 The Makefile appends 198 199 ASAN_OPTIONS=abort_on_error=1:detect_leaks=0 200 UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 201 202 to whatever configure has already put into ASAN_OPTIONS / UBSAN_OPTIONS. 203 Aborting on the first error is essential: otherwise the input that 204 produced the error is lost. Leak detection is off because MHD 205 legitimately abandons half-processed requests when the input is broken, 206 and because the test process may be killed by zzuf. 207 208 The values can be overridden per run: 209 210 make ZZUF_ASAN_OPTIONS=abort_on_error=1:detect_leaks=1 check 211 212 213 Seeds, replaying and sweeping 214 ----------------------------- 215 216 A single seed (the default is 0): 217 218 make ZZUF_SEED=1234 check 219 220 A range of seeds; zzuf starts one child per seed and stops at the first 221 crash: 222 223 make ZZUF_SEED_START=0 ZZUF_SEED_STOP=500 check 224 225 ZZUF_SEED_START takes precedence over ZZUF_SEED. A large range takes a 226 correspondingly long time, so for CI it is usually better to sweep a 227 moderate range per job and to give different jobs different ranges. 228 229 Both runner scripts print a prominent, easy to grep block when a test 230 fails: 231 232 ############################################################### 233 ## FUZZING TEST FAILED 234 ## test : ./test_get_args 235 ## exit code : 134 236 ## seed(s) used: 0 237 ## FAILING SEED(S): 17 238 ## Replay a single failing seed with: 239 ## make ZZUF_SEED=17 TESTS=test_get_args check 240 ############################################################### 241 242 In direct mode the failing seeds are extracted from zzuf's own per-child 243 messages ("zzuf[s=17,r=...]: signal 6 (SIGABRT)"). In socat mode the 244 test program runs outside of zzuf, so the runner reports the exit code 245 (or the signal) of the *server side* together with the seed the run was 246 started with. 247 248 Other useful knobs: 249 250 make ZZUF_FLAGS='--ratio=0.01:0.1' check extra zzuf flags 251 make SOCAT_FLAGS='-v' check extra socat flags 252 make TESTS='test_get_args test_pipelined' check run a subset 253 ./configure --enable-heavy-tests=full 200 instead of 10 client 254 iterations per daemon 255 256 257 What zzuf-style fuzzing fundamentally cannot find 258 ------------------------------------------------- 259 260 This is important, because three of the four remotely triggerable bugs 261 that were fixed on 2026-07-27 had been sitting in code that this 262 directory nominally "covered". 263 264 zzuf flips bits in an otherwise valid, libcurl-generated byte stream. 265 That gives a *uniformly random* perturbation of a *structurally valid* 266 input. Consequently: 267 268 * It cannot invent structure that the generator never produces. 269 libcurl never emits chunk extensions, never pipelines, never sends a 270 request without a "Host:" header, and never sends an "Authorization:" 271 header unless the test asks it to. No amount of bit flipping turns a 272 libcurl request into a request with a chunk extension: the fuzzer 273 would have to insert bytes, not change them, and it would have to 274 keep the surrounding framing intact. 275 276 * It essentially never produces a *semantically hostile but 277 syntactically valid* token. "algorithm=BOGUS-ALGO" or a 278 "response=" with 128 hex digits are perfectly well-formed HTTP; they 279 are simply values a well-behaved client never sends. Random bit 280 flips overwhelmingly produce inputs that MHD rejects in the first few 281 hundred bytes of parsing, so the deep parsers are never reached with 282 values that are valid enough to get there. Reaching the over-long 283 "response" check even requires a *server generated* nonce, which a 284 random mutation can never produce. 285 286 * It only explores the option space that the tests configure. The 287 read-buffer shift-back arithmetic simply does not run unless the 288 connection memory pool is below 1500 bytes. 289 290 * It only detects crashes and hangs. A parser that silently accepts a 291 malformed chunk-extension line, or that mis-frames a request, looks 292 exactly like a healthy parser to this suite unless the resulting 293 memory access happens to be invalid. 294 295 The additions described above buy back a good deal of this: the raw 296 client provides the structures libcurl cannot express, the option matrix 297 provides the configurations, and the deterministic self-check provides an 298 oracle for "wrong result" instead of only "crash". But they do it by 299 hand-writing interesting inputs, which does not generalise. 300 301 The complementary approach is a structure-aware harness, which lives in 302 src/fuzz/. There the HTTP request is *generated* from the fuzzer's 303 random bytes (so the fuzzer can produce any header, any chunk extension, 304 any authentication parameter, in any combination) and the fuzzer is 305 coverage-guided, so it can learn its way into the deep parsers instead of 306 stumbling into them. Use src/testzzuf for "does MHD survive a hostile 307 network", and src/fuzz for "does MHD parse hostile HTTP correctly". 308 309