BUILD-INTEGRATION.md (9256B)
1 # Build integration for `src/fuzz/` 2 3 > **Status: applied.** The three changes described below are already 4 > present in this tree — `configure.ac` carries the `--enable-fuzzing` 5 > option and the `ENABLE_FUZZING` conditional, `AC_CONFIG_FILES` lists 6 > `src/fuzz/Makefile`, and `src/Makefile.am` adds `fuzz` to `SUBDIRS` 7 > under that conditional. Only §1.3 (the optional configuration-summary 8 > line) is *not* applied. The sections below are kept as the record of 9 > what was changed and as the recipe for porting `src/fuzz/` to another 10 > branch. 11 > 12 > For running these same harnesses on OSS-Fuzz — a different build path 13 > that does **not** go through `src/fuzz/Makefile.am` — see 14 > `../../contrib/oss-fuzz/` and §6 at the end of this file. 15 16 `src/fuzz/Makefile.am` is complete. Three files outside `src/fuzz/` 17 have to be touched. The snippets below are the *exact* text that was 18 added; nothing else changes. 19 20 The harnesses are guarded by a new automake conditional 21 `ENABLE_FUZZING` (`--enable-fuzzing`, default **no**), because they are 22 only meaningful in a build with sanitizers and assertions, and because 23 they need a static archive of the library (they call functions that are 24 hidden in the shared object). 25 26 --- 27 28 ## 1. `configure.ac` 29 30 ### 1.1 The `--enable-fuzzing` option and the conditional 31 32 Add this next to the other feature checks — a good spot is right after 33 the `AM_CONDITIONAL([RUN_ZZUF_TESTS], ...)` / 34 `AM_CONDITIONAL([FORCE_USE_ZZUF_SOCAT], ...)` pair (around line 5833 in 35 the v1.0.7 tree): 36 37 ```m4 38 # Fuzzing harnesses (src/fuzz) 39 AC_MSG_CHECKING([[whether to build the fuzzing harnesses]]) 40 AC_ARG_ENABLE([[fuzzing]], 41 [AS_HELP_STRING([[--enable-fuzzing]], 42 [build the in-process fuzzing harnesses in src/fuzz and run them ] 43 [as part of "make check"; requires a static build of the library ] 44 [and is only really useful together with --enable-asserts and ] 45 [--enable-sanitizers=address,undefined [no]])], 46 [], [enable_fuzzing="no"]) 47 AS_VAR_IF([enable_fuzzing], ["yes"], 48 [ 49 AS_VAR_IF([enable_static], ["no"], 50 [AC_MSG_RESULT([[no]]) 51 AC_MSG_ERROR([[--enable-fuzzing requires --enable-static]])], 52 [AC_MSG_RESULT([[yes]])]) 53 ], 54 [enable_fuzzing="no" 55 AC_MSG_RESULT([[no]])]) 56 AM_CONDITIONAL([ENABLE_FUZZING], [[test "x$enable_fuzzing" = "xyes"]]) 57 ``` 58 59 ### 1.2 Register the new `Makefile` 60 61 In the final `AC_CONFIG_FILES([...])` list (around line 5931 of the 62 v1.0.7 tree) add `src/fuzz/Makefile`, i.e. change 63 64 ``` 65 src/microhttpd/Makefile 66 ... 67 src/testzzuf/Makefile]) 68 ``` 69 70 to 71 72 ``` 73 src/microhttpd/Makefile 74 ... 75 src/testzzuf/Makefile 76 src/fuzz/Makefile]) 77 ``` 78 79 ### 1.3 (optional, NOT applied) mention it in the configuration summary 80 81 If you want the harness state to show up in the final summary, add a 82 line next to the other `AC_MSG_NOTICE` entries: 83 84 ```m4 85 AC_MSG_NOTICE([[ Fuzzing harnesses: ${enable_fuzzing}]]) 86 ``` 87 88 --- 89 90 ## 2. `src/Makefile.am` 91 92 Add the conditional `SUBDIRS` entry. The current file reads 93 94 ```make 95 SUBDIRS = include microhttpd . 96 97 if RUN_LIBCURL_TESTS 98 SUBDIRS += testcurl 99 if RUN_ZZUF_TESTS 100 SUBDIRS += testzzuf 101 endif 102 endif 103 ``` 104 105 Add immediately after that block: 106 107 ```make 108 if ENABLE_FUZZING 109 SUBDIRS += fuzz 110 endif 111 ``` 112 113 Automake derives `DIST_SUBDIRS` from all branches of `SUBDIRS` 114 automatically, so `make dist` keeps working and ships `src/fuzz/` 115 regardless of the conditional. 116 117 --- 118 119 ## 3. `Makefile.am` (top level) — nothing to do 120 121 `src/fuzz/` is reached through `src/Makefile.am`; the top-level file 122 needs no change. 123 124 --- 125 126 ## 4. Regenerate and build 127 128 ```sh 129 autoreconf -fi # or ./bootstrap 130 ./configure --enable-fuzzing \ 131 --enable-static \ 132 --enable-asserts \ 133 --enable-sanitizers=address,undefined \ 134 --disable-doc --disable-examples 135 make 136 make -C src/fuzz check 137 ``` 138 139 A full fuzzing session: 140 141 ```sh 142 make -C src/fuzz check \ 143 MHD_FUZZ_ITERATIONS=5000000 \ 144 MHD_FUZZ_SEED=$RANDOM 145 ``` 146 147 Replay the checked-in corpus (what CI should do after a fix): 148 149 ```sh 150 make -C src/fuzz check-corpus 151 ``` 152 153 --- 154 155 ## 5. Notes / caveats 156 157 * **`--enable-static` is mandatory.** `fuzz_str` and 158 `fuzz_auth_header` call `MHD_hex_to_bin()`, `MHD_str_quote()`, 159 `MHD_pool_create()`, `MHD_get_rq_dauth_params_()` etc., which are 160 built with `$(HIDDEN_VISIBILITY_CFLAGS)` and are therefore not 161 exported from `libmicrohttpd.so`. The `-static` in the per-program 162 `_LDFLAGS` makes libtool pick `.libs/libmicrohttpd.a`. 163 164 * `fuzz_auth_header` is only built when `HAVE_ANYAUTH` is true (Basic or 165 Digest authentication enabled). Inside the harness the individual 166 code paths are additionally guarded by `DAUTH_SUPPORT` / 167 `BAUTH_SUPPORT`, and `fuzz_str` guards `MHD_str_unquote()`, 168 `MHD_str_quote()` and `MHD_base64_to_bin_n()` the same way, so a 169 `--disable-dauth --disable-bauth` build still compiles. 170 171 * `fuzz_postprocessor` needs `--enable-postprocessor` (the default); if 172 you build with `--disable-postprocessor`, additionally guard it with 173 the existing `HAVE_POSTPROCESSOR` conditional: 174 175 ```make 176 if HAVE_POSTPROCESSOR 177 check_PROGRAMS += fuzz_postprocessor 178 endif 179 ``` 180 181 (it is currently listed unconditionally, matching the default build). 182 183 * The harnesses include internal headers (`internal.h`, 184 `memorypool.h`, `gen_auth.h`, `mhd_str.h`), hence the 185 `-I$(top_srcdir)/src/microhttpd` in `AM_CPPFLAGS`. `MHD_config.h` is 186 found through automake's default `-I$(top_builddir)`. 187 188 * `make check` in `src/fuzz` runs the full discipline / memory-limit 189 range (`MHD_FUZZ_MIN_DISCIPLINE=-3`, `MHD_FUZZ_MIN_MEM_LIMIT=0`) with 190 50000 iterations per harness, roughly 3 seconds in total under 191 ASAN+UBSAN. The findings K1-K8 of `README` section 6 are all fixed on 192 master, so the full range is clean; their reproducers stay in 193 `corpus/known-findings/`, and `make check-corpus` replays that 194 directory alongside the generated corpus. Both only mean anything on a 195 tree configured with `--enable-asserts`: most of those reproducers trip 196 an `mhd_assert()`, which compiles away without it. 197 198 --- 199 200 ## 6. The OSS-Fuzz build path (does not use this Makefile.am) 201 202 `contrib/oss-fuzz/` builds the *same four harness sources* for 203 libFuzzer/AFL++/honggfuzz without going through `src/fuzz/Makefile.am` 204 at all. It configures the library out of tree, then compiles each 205 harness by hand with `-DFUZZ_NO_MAIN` and links it against 206 `$LIB_FUZZING_ENGINE`: 207 208 ```sh 209 $CC $CFLAGS -DFUZZ_NO_MAIN \ 210 -I$BUILD -I$SRCDIR -I$SRCDIR/src/include \ 211 -I$SRCDIR/src/microhttpd -I$SRCDIR/src/fuzz \ 212 -c $SRCDIR/src/fuzz/fuzz_request.c -o $WORK/fuzz_request.o 213 $CXX $CXXFLAGS $WORK/fuzz_request.o -o $OUT/fuzz_request \ 214 $LIB_FUZZING_ENGINE $BUILD/src/microhttpd/.libs/libmicrohttpd.a -lpthread 215 ``` 216 217 Two consequences for anyone editing `src/fuzz/`: 218 219 * **`LLVMFuzzerTestOneInput()` must stay unconditional.** Only 220 `main()` may be `#ifndef FUZZ_NO_MAIN`; a harness whose fuzz target is 221 itself conditional silently produces an empty OSS-Fuzz binary. 222 * **Anything the fuzz target needs must not live inside the 223 `#ifndef FUZZ_NO_MAIN` block of `fuzz_common.h`.** The PRNG, the crash 224 bookkeeping, `fuzz_report_finding()` and `fuzz_ignore_sigpipe()` are 225 outside it; the generator loop, the mutator, the corpus walker and 226 `main()` are inside. 227 228 This is easy to get wrong and the failure is silent, so it is worth 229 spelling out how it already bit us once. `signal (SIGPIPE, SIG_IGN)` 230 used to sit in `fuzz_install_handlers()`, i.e. inside the block. Under 231 the built-in driver everything looked perfect; under `-DFUZZ_NO_MAIN` 232 the call vanished, and since `fuzz_request` writes into a socketpair 233 whose peer MHD closes on any input that terminates the connection 234 early, the process took a SIGPIPE and died after a few dozen 235 executions. libFuzzer does not intercept SIGPIPE (there is no 236 `-handle_sigpipe`; see `-help=1`), so there was no stack trace, no 237 artifact and no crash report -- the target just stopped, which reads 238 exactly like a clean run that found nothing. It is now 239 `fuzz_ignore_sigpipe()`, called from both the driver and 240 `LLVMFuzzerTestOneInput()`. 241 242 The lesson generalises: a bug in this split cannot be caught by 243 `make -C src/fuzz check`, because that path always defines `main()`. 244 After touching `fuzz_common.h`, build the OSS-Fuzz way as well and 245 confirm the target still runs -- `contrib/oss-fuzz/build.sh` works 246 standalone on any machine with clang and `libclang-rt-dev`: 247 248 ```sh 249 git clone --shared . /tmp/mhd-fuzz-src # build.sh needs a tree 250 # with no in-tree config.status 251 WORK=/tmp/mhd-fuzz-work OUT=/tmp/mhd-fuzz-out \ 252 MHD_SRC=/tmp/mhd-fuzz-src /tmp/mhd-fuzz-src/contrib/oss-fuzz/build.sh 253 /tmp/mhd-fuzz-out/fuzz_request /tmp/mhd-fuzz-out/../corpus -max_total_time=60 254 echo "exit=$?" # anything but 0 here is a bug in the harness, 255 # not a finding 256 ``` 257 258 `contrib/oss-fuzz/` also relies on two things this directory provides: 259 `make -C src/fuzz refresh-corpus` (to regenerate `corpus/`) and the 260 `corpus/known-findings/` reproducers, which it packages into 261 `fuzz_request_seed_corpus.zip` so that they become permanent 262 regressions. It is deliberately **not** part of `contrib/ci/jobs/`.