README (6412B)
1 GNU libextractor -- OSS-Fuzz integration 2 ======================================== 3 4 This directory is the OSS-Fuzz project definition for libextractor. The 5 harness *sources* live in `src/fuzz/`; read `src/fuzz/README` first, it 6 explains what each target does and why the harnesses find things that 7 running `extract` over mutated files does not. 8 9 build.sh compiles the harnesses against the engine 10 Dockerfile the OSS-Fuzz builder image 11 project.yaml project metadata 12 make_seed_corpus.sh builds the per-target seed corpora 13 dicts/ libFuzzer dictionaries 14 *.options per-target libFuzzer options 15 16 17 ------------------------------------------------------------------- 18 1. What gets built 19 ------------------------------------------------------------------- 20 21 25 targets by default: four core targets (`fuzz_datasource`, `fuzz_ipc`, 22 `fuzz_convert`, `fuzz_unzip`) and one per in-tree parser 23 (`fuzz_applefile` ... `fuzz_zip`). 24 25 Two things are deliberately left out. 26 27 `fuzz_extract` needs `lt_dlopen()` to load the plugin modules, and this 28 build configures `--disable-shared` because the per-plugin targets link 29 the plugin sources directly. A target that loads uninstrumented shared 30 objects gives the engine no feedback from the code it is supposed to be 31 fuzzing, so shipping it here would be worse than not shipping it. It is 32 built and run in tree by `make -C src/fuzz check`. 33 34 The plugins that wrap a third-party parser -- gif, jpeg, tiff, flac, 35 ogg, archive, mime -- are not built either. giflib, libjpeg, libtiff, 36 FLAC, libvorbis, libarchive and libmagic each have their own OSS-Fuzz 37 project; linking an uninstrumented copy here would attribute their bugs 38 to libextractor and would make the MemorySanitizer configuration 39 impossible. For a *local* campaign that wants the glue code covered -- 40 the part that turns what those libraries return into a metadata 41 callback, which is ours -- set `LE_FUZZ_EXTRA_PLUGINS=1` and build.sh 42 adds them. 43 44 The ole2 plugin is a third case. It is excluded for the same reason -- 45 libgsf and glib are not instrumented here -- but unlike the wrappers 46 above it parses a good deal of the header itself before libgsf is ever 47 called, so leaving it out leaves *our* code unfuzzed. A signed-overflow 48 defect sat in exactly that code (issue 13) until a campaign covered the 49 target. Set `LE_FUZZ_GSF=1` to keep libgsf in the configure line and 50 build `fuzz_ole2`; it needs pkg-config to find `libgsf-1`, and warns and 51 skips the target if it cannot. Any local campaign should use it. 52 53 54 ------------------------------------------------------------------- 55 2. Running it locally 56 ------------------------------------------------------------------- 57 58 build.sh works standalone on any machine with clang and 59 `libclang-rt-dev`. It never writes into the source tree, but it does 60 need a tree with no in-tree `config.status`: 61 62 rsync -a --exclude=.git --exclude='.libs' --exclude='*.o' \ 63 /path/to/libextractor/ /tmp/le-src/ 64 65 WORK=/tmp/le-work OUT=/tmp/le-out LE_SRC=/tmp/le-src \ 66 /tmp/le-src/contrib/oss-fuzz/build.sh 67 68 unzip -q /tmp/le-out/fuzz_unzip_seed_corpus.zip -d /tmp/corpus 69 /tmp/le-out/fuzz_unzip /tmp/corpus \ 70 -dict=/tmp/le-out/fuzz_unzip.dict -max_total_time=600 71 72 On a Debian toolchain clang may not find libstdc++ on its own; add 73 74 CXX="clang++ --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/15" 75 76 Variables build.sh honours, all with defensive defaults so that the 77 script is runnable by hand: `SRC`, `LE_SRC`, `WORK`, `OUT`, `CC`, `CXX`, 78 `CFLAGS`, `CXXFLAGS`, `LIB_FUZZING_ENGINE`, `SANITIZER`, 79 `FUZZING_ENGINE`, `ARCHITECTURE`, plus the three local-only extras 80 `LE_FUZZ_EXTRA_PLUGINS`, `LE_FUZZ_GSF` and `LE_UBSAN_HALT`. 81 82 FUZZING_ENGINE=afl ./build.sh # AFL++ targets 83 FUZZING_ENGINE=none ./build.sh # the harnesses' own driver, 84 # no engine needed at all 85 SANITIZER=undefined ./build.sh # UBSan-only campaign 86 87 Via OSS-Fuzz's own helper: 88 89 python infra/helper.py build_image libextractor 90 python infra/helper.py build_fuzzers --sanitizer address libextractor 91 python infra/helper.py check_build libextractor 92 python infra/helper.py run_fuzzer libextractor fuzz_unzip 93 94 95 ------------------------------------------------------------------- 96 3. UndefinedBehaviorSanitizer is left recovering by default 97 ------------------------------------------------------------------- 98 99 The local `SANITIZER=address` default folds UBSan into the ASan build 100 but does *not* pass `-fno-sanitize-recover=undefined`. 101 102 That is a deliberate departure from what OSS-Fuzz does, and the reason is 103 worth knowing before changing it: libextractor's parsers contain a 104 handful of signed-shift and misaligned-load sites that fire on almost 105 every input (`pack.c`, `riff_extractor.c`, `elf_extractor.c`, 106 `real_extractor.c`). With UBSan halting, every run dies within seconds 107 on one of those and ASan never gets to the memory-safety bugs, which are 108 the ones that matter. UBSan reports each source location once per 109 process either way, so nothing is lost from the logs. 110 111 Set `LE_UBSAN_HALT=1` for the OSS-Fuzz behaviour, or use 112 `SANITIZER=undefined` for a campaign whose whole point is UB. On 113 OSS-Fuzz proper the two sanitizers are separate builds and this does not 114 arise. 115 116 117 ------------------------------------------------------------------- 118 4. Seed corpora 119 ------------------------------------------------------------------- 120 121 `make_seed_corpus.sh` assembles three sources per target: 122 123 1. the checked-in `src/fuzz/corpus/<target>/`, which 124 `make -C src/fuzz refresh-corpus` regenerates from each harness' 125 built-in seeds; 126 2. the real files under `src/plugins/testdata/`, mapped to targets by 127 filename prefix and prefixed with the harness' configuration bytes 128 (all zero, which selects exactly what production does). This is the 129 part that matters: a fuzzer that has to *invent* a valid ZIP central 130 directory before it reaches the interesting code will not get there; 131 3. `src/fuzz/corpus/known-findings/<target>-*`, the reproducers of the 132 entries in `issues.txt`, so that each stays a permanent regression 133 in the OSS-Fuzz corpus too. 134 135 The testdata files are deliberately not duplicated into 136 `src/fuzz/corpus/`: they are already in the tree, and the prefix is 137 cheap to add at build time.