libextractor

README.forensics (23488B)

---

 heif -- heif_test.heic, heif_test.avif
   Generated by contrib/gen_heif_testdata.sh (python3 only, deterministic).
   Hand-constructed ISO base media file format containers: the box tree,
   the item structures and the property associations are exactly what
   ISO/IEC 14496-12, ISO/IEC 23008-12 and the AV1 Image File Format
   specify, but the 'mdat' payload is a placeholder rather than a real
   HEVC or AV1 bitstream, so the files identify but do not decode.  That
   is on purpose: the plugin is a container parser and the values it must
   report are then known exactly.
 
   heif_test.heic (916 bytes): major brand 'heic'.  Three items -- the
   primary image (id 1), a thumbnail (id 2, tied to the primary by an
   'iref' of type 'thmb') and an 'Exif' metadata item (id 3).  The 'ipco'
   property container holds the thumbnail's 16x12 'ispe' *first* and the
   primary item's 96x64 'ispe' second, so a parser that reports the first
   'ispe' it finds gets the wrong answer and only a correct
   pitm -> ipma -> ipco walk reports 96x64.  Also carries pixi (10 bit),
   irot (90 degrees), imir, colr nclx with transfer 16 (PQ, i.e. HDR) and
   an hvcC shell.
 
   heif_test.avif (631 bytes): major brand 'mif1' with 'avif' only among
   the compatible brands, which is what several AVIF writers emit; both
   the MIME type and the codec therefore have to come from that list.
   One image item, 128x72, no 'pixi', so the bit depth has to be taken
   from the 'av1C' configuration record.
 
   Licence: CC0.  No third party content.
 
 webp -- webp_test.webp
   Generated by contrib/gen_webp_testdata.sh (needs ffmpeg with libwebp,
   Debian package "ffmpeg", plus python3; deterministic).
 
   webp_test.webp (752 bytes): an extended (VP8X) animated WebP carrying
   every optional chunk the container defines -- ICCP, ANIM, three ANMF
   frames, EXIF and XMP -- with the alpha, animation, ICC, EXIF and XMP
   feature flags set.  Canvas 40x24.  The three frames hold a genuine
   lossless VP8L bitstream encoded by libwebp from a solid colour
   generated by lavfi (nothing is read from disk); libwebp reads the
   result back as a valid 3 frame RGBA animation.  Frame durations are
   100, 150 and 250 ms, summing to 500 ms.  The ICC (136 bytes), EXIF and
   XMP payloads are placeholders: the plugin only reports that they are
   present and parsing them is the exiv2 plugin's job.
 
   Licence: CC0.  No third party content.
 
 sqlite -- sqlite_test.db
   SQLite 3 database, 5120 bytes.  Generated by contrib/gen_sqlite_testdata.sh
   with the sqlite3 CLI: 1 KiB pages, UTF-8, incremental auto-vacuum, WAL
   journal mode, application_id 1279613012 (= "LEXT"), user_version 4242, two
   tables and an index, with rows deleted and an index dropped so that a
   freelist page survives.  The generator then rewrites the SQLite version at
   header offset 96 to 3045001 ("SQLite 3.45.1") and makes version-valid-for
   agree with the change counter, so the file does not depend on the sqlite3
   that produced it.  test_sqlite asserts on the page size, journal mode,
   change counter, page count, free space, schema cookie, user_version,
   vacuum mode, encoding, application id (numeric and as its four-character
   tag), writer version and logical size.  Licence: CC0 / public domain.
 
 tar -- tar_test.tar
   POSIX ustar archive, 10240 bytes.  Generated by contrib/gen_tar_testdata.sh
   using Python's tarfile with every ownership field set explicitly:
   uname "forensic", gname "analysts", uid 1000, gid 100, plus one member
   owned by root/root.  Contains a directory, three regular files, a symlink,
   a hard link and one member whose 107-character path has to be split across
   the ustar prefix field.  Modification times are hard-coded (1700000000 and
   1710506096); nothing is read from the machine that runs the generator.
   test_tar asserts on the format, member count, summed member size, newest
   mtime, both owners and groups, both uid/gid pairs, the five distinct
   permission masks, all seven member names and both link targets.
   Licence: CC0 / public domain.
 
 iso9660 -- iso9660_test.iso
   ISO 9660 (ECMA-119) image, 43008 bytes = 21 sectors.  Generated by
   contrib/gen_iso9660_testdata.sh, which builds the volume descriptor set
   by hand with Python struct rather than calling genisoimage or xorriso: a
   mastering program stamps its own name and the current time into the
   descriptors, which would be neither reproducible nor assertable.  Layout is
   16 zero sectors of system area, then the primary volume descriptor, an
   El Torito boot record, a Joliet supplementary descriptor (escape "%/E"),
   the set terminator and one root directory extent whose "." record carries
   SUSP "SP" and Rock Ridge "RR" entries.  This is over the 32 KB guideline
   and cannot be smaller: ECMA-119 puts the first descriptor at byte 32768.
   test_iso9660 asserts on the system/volume/publisher/preparer/application
   identifiers, the copyright, abstract and bibliographic file identifiers,
   the block size, volume size, volume set size, all four dec-datetime fields
   and the Joliet, Rock Ridge and El Torito detections.
   Licence: CC0 / public domain.
 
 
 geotiff -- geotiff_test.tif
   Generated by contrib/gen_geotiff_testdata.sh (python3 only,
   deterministic; no gdal-bin needed).  Hand-built with struct rather
   than produced by gdal_translate: GDAL's own output is several KB and
   its numbers are whatever the projection happens to give, whereas a
   hand-built file can be 1 KB and can use round numbers for the
   tiepoint and the pixel scale so that the bounding box the plugin
   derives from them is an exact constant to assert on.
 
   geotiff_test.tif (1099 bytes): a classic little-endian TIFF 6.0
   baseline image, 16x16, 8 bit greyscale, uncompressed, with the
   GeoTIFF 1.1 tags for EPSG:32633 (WGS 84 / UTM zone 33N).
   ModelPixelScaleTag is 10 m, ModelTiepointTag maps raster (0,0) to
   model (500000, 5400000), so the extent is exactly
   500000,5399840,500160,5400000 -- a value that can only come out right
   if the tiepoint, the scale and the image dimensions were all read and
   combined.  GeoKeyDirectoryTag carries eight keys covering all three
   TIFFTagLocation cases the specification allows (inline SHORT and
   '|'-terminated substrings of GeoAsciiParamsTag), and GDAL's two
   private tags (GDAL_METADATA with three <Item> elements, GDAL_NODATA)
   are present so the provenance scan is exercised too.  Byte order,
   BigTIFF and the ModelTransformationTag path are covered by variants
   built during development but not committed, since asserting on them
   would only duplicate this file's expectations.
 
   Licence: CC0.  No third party content.
 
 diskimage (QCOW2 / VMDK / VHD / VHDX)
 -------------------------------
 
   Generated by contrib/gen_diskimage_testdata.sh (needs qemu-utils and
   python3).  qemu-img is the producer for all three; python3 only
   overwrites the fields qemu-img randomises per run, so re-running the
   script gives byte-identical files.
 
   diskimage_test.qcow2 (1792 bytes): a QCOW2 version 3 differencing
   image, 1 MiB virtual, 512 byte clusters, backed by
   "diskimage_base.qcow2" (which the script creates in a temporary
   directory and throws away -- the plugin never opens it, only reports
   its name).  Created with lazy_refcounts=on and compression_type=zstd
   so that the compatible-feature and incompatible-feature paths and the
   version 3 compression type byte are all exercised.  Untouched
   qemu-img output; the QCOW2 header has no per-run fields.
 
   diskimage_test.vmdk (16384 bytes): a monolithicSparse VMDK, 1 MiB
   virtual, 64 KiB grains, backed by "diskimage_base.vmdk".  Covers both
   the binary sparse extent header and the text descriptor embedded in
   it at offset 512.  The CID and parentCID qemu-img picks at random are
   rewritten to 1f2e3d4c and 5a6b7c8d and the descriptor slot is padded
   back to its original length, so no offset in the header changes.
   TRUNCATED from the 64 KiB qemu-img writes to 16 KiB: everything past
   the descriptor is zeroed grain tables.  The file is therefore no
   longer a mountable VMDK, but the header (80 bytes) and the descriptor
   (offset 512, 10 KiB) are intact and are all the plugin reads.
 
   diskimage_test.vhdx (4096 bytes): the file identifier block of a
   VHDX, with the UTF-16LE creator string overwritten with a fixed value
   so the test does not depend on the local qemu version.  TRUNCATED,
   and unavoidably so: [MS-VHDX] fixes the header at 64 KiB and the
   region table at 192 KiB, and requires the metadata region to start at
   a 1 MiB boundary, so the smallest conformant VHDX is over 2 MiB.  The
   test can only assert the signature-level items; the plugin's region
   table, metadata table and parent locator walk was checked by hand
   against full qemu-img output and against a hand-built conformant
   image with a parent locator and a non-zero LogGuid.
 
   There is no diskimage_test.vhd: the plugin reads the Microsoft VHD
   ("conectix") footer as well, but src/plugins/Makefile.am has no
   EXTRA_DIST entry for one.  Run the generator with GEN_FULL=1 to get
   a deterministic sample plus the untruncated VMDK and VHDX.
 
   Licence: CC0.  The images contain no data at all, only headers.
 
 pecoff -- pecoff_test.exe, pecoff_test.dll
   Generated by contrib/gen_pecoff_testdata.sh.  The .exe needs the
   Debian package "gcc-mingw-w64-x86-64"; the .dll needs only python3.
   Both are deterministic: the linker is told not to insert a timestamp
   and the script writes a fixed one afterwards.
 
   pecoff_test.exe (4608 bytes): a real x86-64 PE32+ image built by
   mingw-w64 from a three line program, linked without the C runtime
   (-nostdlib) so that its whole import table is KERNEL32.dll!ExitProcess
   and the import hash is therefore a value the test can spell out --
   MD5("kernel32.exitprocess"), which is what pefile computes for the
   same file.  Carries a VERSIONINFO resource written by windres, which
   is what exercises the resource directory walk and the nested,
   32-bit-aligned, UTF-16 StringFileInfo parser.  TimeDateStamp is set to
   1700000000 (2023-11-14T22:13:20Z) after linking.
 
   pecoff_test.dll (3584 bytes): assembled from the PE specification with
   python3, because mingw cannot produce the two artifacts that matter
   most here.  A Rich header (Microsoft's linker writes those and nothing
   else does) masked with the key 0x1A2B3C4D and holding four
   (product id, build, use count) triples; a CodeView RSDS record naming
   C:\Users\builder\source\repos\pecoff_test\x64\Release\pecoff_test.pdb
   with a fixed GUID and age; an export directory with three named
   exports; a certificate table; and a .pack section filled with SHA-256
   output so that its entropy lands where a packed section's would.  The
   certificate table is not a signature and verifies as nothing -- it is
   the shortest DER that gives the commonName scan something to find, and
   the name it carries is "GNU libextractor test signer".  Verified
   against pefile: machine, subsystem, Rich values, CodeView record and
   export names all read back as intended.
 
   Licence: CC0.  No third party content.
 
 lnk -- lnk_test.lnk
   Generated by contrib/gen_lnk_testdata.sh (python3 only, deterministic).
 
   lnk_test.lnk (706 bytes): a Windows shell link assembled from
   [MS-SHLLINK] rather than produced by Windows, so that every value the
   test asserts is a constant the generator owns.  It carries a
   LinkTargetIDList (two shell items) that the parser has to step over
   without looking at, a LinkInfo whose VolumeID has drive type "fixed",
   serial 1A2B-3C4D and the unicode volume label EVIDENCE, and whose
   target path is split across LocalBasePathUnicode
   ("C:\Users\analyst\Desktop\") and CommonPathSuffixUnicode
   ("evidence.txt") so that the join is tested; UTF-16 string data for
   all five optional strings; and a TrackerDataBlock naming the machine
   "lab-ws-07" whose droid file identifier 5f3e2d1c-a4b6-11d2-8ae7-
   001b44113ab7 is a version 1 UUID carrying the MAC address
   00:1b:44:11:3a:b7.  The three FILETIMEs in the header are the
   target's, and all differ: 2021-03-04T05:06:07Z (creation),
   2021-06-15T14:30:00Z (write) and 2022-07-08T09:10:11Z (access).
 
   Cross-checked with liblnk's lnkinfo (20240423), which reads back the
   same timestamps, serial, label, paths, strings and tracker GUIDs.
 
   Licence: CC0.  No third party content.  The machine name, MAC address
   and user name are invented.
 
 plist -- plist_binary.plist, plist_xml.plist
   Generated by contrib/gen_plist_testdata.sh (python3 only, using the
   standard library's plistlib; deterministic).  Both files hold the same
   dictionary written once in each representation, so the two solution
   sets in test_plist.c differ only where the formats genuinely do.
 
   The dictionary mixes the keys of an application bundle Info.plist
   (CFBundleIdentifier, CFBundleShortVersionString, DTPlatformName,
   DTXcode, BuildMachineOSBuild, ...) with those of an iOS backup
   Info.plist (DeviceName, ProductType, SerialNumber, LastBackupDate),
   which is the forensic case the plugin exists for, plus one integer
   under an unrecognised key so the "key: value" fallback is exercised.
 
   plist_binary.plist (613 bytes): bplist00, 32 objects, two byte offsets
   and one byte object references.  NSHumanReadableCopyright is
   "Copyright (C) 2026 Grüße" with a copyright sign and two umlauts, so
   plistlib has to store it as a UTF-16BE string object (marker 0x6n) --
   the string decoder most likely to be got wrong, and the one the ASCII
   path (0x5n) cannot fake.  LastBackupDate is a date object (0x33), an
   eight byte big-endian IEEE double counting seconds from 2001-01-01,
   and only comes back as 2024-03-14T15:09:26Z if both the epoch shift
   and the byte order were handled.
 
   plist_xml.plist (1126 bytes): the same dictionary as a DTD-bound XML
   document, which is what the bounded <key>/value scan reads.
 
   The parser was also checked against the XML property lists shipped by
   Debian's cmake and qt6 packages, and against a binary plist produced
   by libplist's plistutil (an implementation independent of plistlib,
   and one that interleaves the key and value object indices rather than
   writing them in two runs).  Those files are not committed.
 
   Licence: CC0.  Every identifier, serial number and device name in the
   file is invented.
 
 id3 -- id3_test.mp3
   Generated by contrib/gen_id3_testdata.sh.  Debian packages needed:
   python3, python3-mutagen and lame.  Deterministic.
 
   id3_test.mp3 (7546 bytes): one second of a 440 Hz sine at 44100 Hz,
   mono, written as a WAV by python and encoded by LAME 3.100 at -V 5
   --vbr-new.  LAME is the encoder rather than ffmpeg on purpose: ffmpeg
   writes its own Xing/LAME tag stamped "Lavf lame", which does not carry
   the lowpass, ReplayGain, preset or VBR method fields that make the
   LAME tag the fingerprint it is.  Note that the encoder version string
   ("LAME3.100") and the settings summary derived from that tag are
   asserted on literally, so regenerating with a different LAME release
   means updating those two expectations.
 
   The tag is ID3v2.4 and exercises all four text encodings in one file:
   TIT2 is UTF-8, TPE1 is UTF-16 with a byte order mark, TALB is
   ISO-8859-1, and all three carry characters outside ASCII, so a plugin
   that passed the bytes through would produce invalid UTF-8 rather than
   quietly wrong text.  TCON is stored as "(52)" and has to come back
   resolved to "Electronic".  The USLT and APIC frames are both longer
   than 127 bytes, which is where the synchsafe frame size of ID3v2.4
   first differs from the plain big-endian integer of 2.3 -- everything
   after them only parses if that was handled correctly.  The provenance
   frames are TENC, TSSE, TOWN, TFLT, TMED, a UFID with a MusicBrainz
   style owner, a PRIV owned by com.apple.iTunes, two URL frames and a
   TXXX with the description "apID".  The attached picture is a 208 byte
   PNG built by the generator with zlib.
 
   A hand-written ID3v1 tag follows the audio.  Every one of its fields
   is deliberately different from its ID3v2 counterpart, so only the two
   it alone supplies -- the year 1999 and the comment -- may reach the
   caller; anything else showing up means the gap filling is broken.
 
   The parser was also checked during development against files written
   by three other producers, none of them committed: mutagen at ID3v2.3
   (plain frame sizes, UTF-16 text), ffmpeg's own ID3v2.3 writer and its
   "Lavf"/"Lavc" Xing tag, a hand-built ID3v2.2 tag with three character
   frame identifiers, a hand-built unsynchronised ID3v2.3 tag, an
   ID3v1-only file, a stereo file (which puts the Xing tag at offset 36
   rather than 21) and a CBR file (an "Info" tag rather than "Xing").
 
   Licence: CC0.  The audio is a generated sine tone; the licensee
   address, purchase account, MusicBrainz identifier and every name in
   the tag are invented.
 
 apk -- apk_test.apk, apk_test.jar
   Generated by contrib/gen_apk_testdata.sh (python3 only, deterministic).
 
   apk_test.apk (4723 bytes): a hand-built Android package.  The
   AndroidManifest.xml is real binary XML written out from the chunk
   layout in AOSP's ResourceTypes.h -- a UTF-16LE string pool (which is
   what aapt emits for the manifest), a resource map with the attribute
   resource ids read off a real aapt2 build, and START/END_ELEMENT
   chunks.  It declares package org.gnu.libextractor.apktest,
   versionCode 1719, versionName 1.19.0, compileSdkVersion 33,
   minSdkVersion 21, targetSdkVersion 33 and five uses-permission
   elements.  The <application> element mixes literal and reference
   valued attributes on purpose: android:label is a literal string and
   must be reported, android:icon and android:networkSecurityConfig are
   resource references and a bare @0x7f... must not be reported as a
   title.  The archive also holds three classes*.dex stubs, three
   lib/<abi>/ stubs (arm64-v8a, armeabi-v7a, x86_64), a META-INF JAR
   manifest (so that the plugin has to prefer the Android manifest), a
   v1 signature under the alias ANDROIDD, and an APK Signing Block with
   v2 and v3 identifiers spliced in between the last entry and the
   central directory.
 
   apk_test.jar (3108 bytes): a JAR whose META-INF/MANIFEST.MF carries
   the JDK identification (Created-By, Build-Jdk, Build-Jdk-Spec), the
   Implementation-*/Specification-* triples, the OSGi Bundle-* headers
   and a Class-Path.  Implementation-Title is 98 bytes long and is
   therefore folded across two lines, so reading it back proves the
   continuation lines are unfolded.
 
   Both archives are signed by the same PKCS#7 block, a throwaway
   two-element chain ("libextractor test CA" issuing "libextractor test
   signer", serial 0x1719) generated once with openssl and embedded in
   the generator as base64; the private keys were discarded, so the
   signature protects nothing.  The certificate authority comes first
   inside the certificates set, which is what makes the test prove that
   the signer is resolved through the SignerInfo's issuerAndSerialNumber
   rather than simply taken from the first certificate.
 
   Licence: CC0.  No third party content; every name, package
   identifier and certificate subject is invented.
 
 ebook -- ebook_test.epub, ebook_test.mobi
   Generated by contrib/gen_ebook_testdata.sh (python3 only,
   deterministic).  Calibre is deliberately not used.
 
   ebook_test.epub (2645 bytes): an OCF container with the uncompressed
   `mimetype' member first as the specification requires, a
   META-INF/container.xml pointing at OEBPS/content.opf, and an EPUB 3
   package document with rich Dublin Core: title, creator, publisher,
   language, date, two subjects, a description indented across three
   lines (so that reading it back proves the white space collapsing), a
   rights field containing an &amp; entity, one dc:identifier with
   opf:scheme="ISBN" and one that is a urn:uuid, a dc:contributor with
   opf:role="bkp" naming the tool, <meta name="calibre:timestamp">,
   <meta name="generator"> and <meta property="dcterms:modified">.  The
   manifest holds seven <item> entries and the spine three <itemref>
   entries, so the entry count also proves the two are told apart.  A
   META-INF/encryption.xml (font obfuscation) is present.
 
   ebook_test.mobi (946 bytes): a hand-built Palm database, type/creator
   BOOKMOBI, three records.  The PDB creation and modification times are
   written in the 1904 epoch (2026-01-01 and 2026-02-01).  Record 0
   holds a PalmDOC header with encryption type 1, a 232 byte MOBI header
   declaring UTF-8 text and the EXTH flag, and an EXTH header with
   seventeen records covering author, publisher, description, ISBN,
   subject, published date, contributor (the tool), rights, ASIN,
   creator software triple, watermark, cdetype, last update time,
   updated title and language, followed by the full name.
 
   Licence: CC0.  No third party content; the book, its author, its
   ISBN and its watermark are invented.
 
 gpx / kml / mbox
 ----------------
 
   Generated by contrib/gen_gpx_testdata.sh, contrib/gen_kml_testdata.sh
   and contrib/gen_mbox_testdata.sh.  All three are plain text written
   out literally by a POSIX shell heredoc -- no tools, no timestamps
   taken from the clock, byte-for-byte reproducible.
 
   gpx_test.gpx (2146 bytes): a GPX 1.1 track with creator="Garmin eTrex
   30" (the field that names the recording device, and the one the
   plugin also reports as a device model), a full <metadata> block with
   name, desc carrying an &amp; entity, author name and a split
   <email id= domain=>, copyright with a <license>, two <link href=>,
   a <time> of 2024-03-15T07:12:00Z, keywords, and an explicit <bounds>.
   Two <wpt>, a <rte> of two <rtept> and one <trkseg> of five <trkpt>
   with elevations 408.2-430.7 m and times from 07:12:33 to 07:24:53,
   which is where the 0:12:20 duration and the 1215 m haversine sum in
   the test come from.  The coordinates are a walk through central
   Zurich, invented, not anybody's real track.
 
   kml_test.kml (2027 bytes): an OGC KML 2.2 Document over the same
   coordinates, so that the two plugins can be compared -- note that the
   tuples are lon,lat,alt, the reverse of the GPX attribute order, which
   is what the bounding box assertion actually tests.  It has a
   <description> whose HTML sits in a CDATA section, an atom:author and
   atom:link, a <TimeStamp>, a <LookAt> (the viewer's position, which
   the plugin reports as a comment rather than as the data's position),
   three Placemarks (two Points and a LineString) and a <NetworkLink>
   whose <href> points at a remote host.
 
   mbox_test.mbox (3504 bytes): three messages in mbox (mboxo) form.
   The first carries a three-hop Received chain with two IPv4 literals
   and one [IPv6:...] literal, an X-Originating-IP, a DKIM-Signature
   with d=example.com, an Authentication-Results, X-Mailer, an RFC 2047
   From and Subject (base64/UTF-8 and quoted-printable/ISO-8859-1 in one
   field), In-Reply-To and References, and a multipart/mixed body with
   two attachments -- one with a quoted filename and one that only
   exists as RFC 2231 filename*0/filename*1 continuation segments.
   Date: Tue, 12 Mar 2024 09:41:07 +0100 is Unix time 1710232867.  The
   third message's body holds a `>From ' line, the mboxo escaping, so
   the message count staying at three proves the separator scan is not
   fooled by it.
 
   Licence: CC0.  All names, addresses and hosts are from the RFC 2606 /
   RFC 5737 / RFC 3849 reserved ranges; the coordinates and the mail are
   invented.