gen_fulltext_pdf_dvi_testdata.sh (5515B)
1 #!/bin/sh 2 # This file is part of libextractor. 3 # Copyright (C) 2026 Christian Grothoff 4 # 5 # Regenerate src/plugins/testdata/pdf_fulltext.pdf and 6 # src/plugins/testdata/dvi_fulltext.dvi, the test documents for the 7 # "fulltext" option of the pdf and dvi plugins. 8 # 9 # Requires: pdflatex and latex (TeX Live) with the fancyhdr, geometry 10 # and lineno packages and the EC fonts' TFM files. SOURCE_DATE_EPOCH 11 # makes the output reproducible for a given TeX Live version. 12 # 13 # Both documents come from the same LaTeX source (on A5 paper, so that 14 # there are few words per page): 15 # - a running header ("Notes on Paged Text" / "Working Draft") and a 16 # page number in the footer of every page, which must not end up in 17 # the full text; 18 # - indented paragraphs, a one-line paragraph, a paragraph that runs 19 # across a page break, several words hyphenated at line ends, the 20 # fi/ff/ffl ligatures, TeX quotes and dashes and accented letters; 21 # - PDF only: a two-column page with line numbers in the outer 22 # margins (lineno), which the extractor must drop, reading the left 23 # column before the right one; 24 # - DVI only: a paragraph set in the T1 (Cork) encoding. 25 # 26 # The text is invented and is dedicated to the public domain (CC0). 27 set -e 28 29 srcdir=$(cd "$(dirname "$0")/.." && pwd) 30 out="$srcdir/src/plugins/testdata" 31 tmp=$(mktemp -d) 32 trap 'rm -rf "$tmp"' EXIT 33 34 SOURCE_DATE_EPOCH=1700000000 35 FORCE_SOURCE_DATE=1 36 export SOURCE_DATE_EPOCH FORCE_SOURCE_DATE 37 38 cat > "$tmp/ft.tex" <<'TEX' 39 \documentclass[12pt]{article} 40 \usepackage[a5paper,margin=20mm,headheight=14pt]{geometry} 41 \ifdefined\dvimode\else\usepackage[switch]{lineno}\fi 42 \usepackage{fancyhdr} 43 \pagestyle{fancy} 44 \fancyhf{} 45 \fancyhead[L]{Notes on Paged Text} 46 \fancyhead[R]{Working Draft} 47 \fancyfoot[C]{\thepage} 48 \renewcommand{\headrulewidth}{0pt} 49 \hyphenation{re-con-struc-tion} 50 \emergencystretch=2em 51 \begin{document} 52 \section*{Introduction} 53 Paged documents scatter their words across many small boxes, and a 54 careful extractor has to put them back in reading order. This first 55 paragraph is deliberately long so that it spans several lines of the 56 page and so that the justification of the lines becomes visible to 57 anybody who looks at the result with a critical eye. A few more 58 sentences follow here only to fill up the first page, so that a later 59 paragraph is forced to continue on the next page. The quick brown dog 60 sleeps under the old oak tree while the children play in the garden 61 and the bees buzz around the flowers. 62 63 The second paragraph is indented like every other paragraph that 64 follows another one, and the extractor should notice that indentation 65 and start a new paragraph here. Its reconstruction depends on the 66 geometry alone, since the file itself does not record where one 67 paragraph ends and the next one begins. 68 69 Short paragraph number three. 70 71 The fourth paragraph mentions a few typographic details: the office 72 has fluffy waffles, an efficient staff and a ``quoted phrase'' with 73 pages 10--20 and a pause---like this one. The caf\'e serves na\"ive 74 guests. 75 76 The fifth paragraph runs across the page break, right past the 77 running header and the page number, neither of which must appear in 78 the extracted text. We continue writing some more text here so that 79 this paragraph covers enough lines to overflow from the first page 80 onto the second page, where it simply goes on as if nothing had 81 happened, and the extractor must join both halves into one single 82 paragraph without inserting any break at all. To make sure that this 83 really happens, the paragraph goes on for a while longer with a few 84 more sentences of plain filler text that nobody will ever read, apart 85 from the test suite, which checks every single word of it. 86 87 The sixth paragraph is the last one of the single column part and it 88 ends the page with a sentence about zebras jumping over lazy 89 foxes. 90 91 \ifdefined\dvimode 92 \newpage 93 {\fontencoding{T1}\selectfont 94 The seventh paragraph is set in the T1 encoding: Stra\ss e, 95 \"Uberma\ss, \'etude and \c{c}a va.\par} 96 \else 97 \setlength{\columnsep}{18pt} 98 \twocolumn 99 \small\sloppy 100 \begin{linenumbers} 101 The seventh paragraph opens the two column part of the document. The 102 left column is read completely before the right column, and the line 103 numbers printed in the margins next to each column must be dropped by 104 the extractor because they are not part of the text at all. 105 106 The eighth paragraph continues in the left column with another few 107 lines of harmless filler text about apples, pears and plums that grow 108 in an orchard near the river. 109 110 The ninth paragraph keeps going with filler about the weather, which 111 was mild and pleasant in the morning but turned rather cold later on. 112 The ninth paragraph keeps going with filler about the weather, which 113 was mild and pleasant in the morning but turned rather cold later on. 114 115 \newpage 116 The tenth paragraph is the first one of the right column and it talks 117 about lighthouses that guide ships safely through the night along 118 rocky coasts. 119 120 The eleventh paragraph concludes the document with a final remark. 121 \end{linenumbers} 122 \fi 123 \end{document} 124 TEX 125 126 cd "$tmp" 127 # twice each, so that lineno's references settle 128 pdflatex -interaction=batchmode '\pdftrailerid{}\input{ft.tex}' >/dev/null 129 pdflatex -interaction=batchmode '\pdftrailerid{}\input{ft.tex}' >/dev/null 130 latex -interaction=batchmode -jobname=ftdvi '\def\dvimode{}\input{ft.tex}' \ 131 >/dev/null 132 latex -interaction=batchmode -jobname=ftdvi '\def\dvimode{}\input{ft.tex}' \ 133 >/dev/null 134 cp ft.pdf "$out/pdf_fulltext.pdf" 135 cp ftdvi.dvi "$out/dvi_fulltext.dvi"