path: root/doc/lj.txt

        Reading file meta-data with extract and libextractor


                     by Christian Grothoff


INTRODUCTION

Modern file formats have provisions to annotate the contents of the
file with descriptive information.  This development is driven by the
need to find a better way to organize data than merely using
filenames.  The problem with such meta-data is that the way it is
stored is not standardized across different file formats.  This makes
it difficult for format-agnostic tools such as file managers or
file-sharing applications to make use of the information.  It also
results in a plethora of format-specific tools that are used to
extract the meta-data, such as AVInfo [7] id3edit [8], jpeginfo [9]
or Vocoditor [10].

In this article, the libextractor library and the extract tool are
introduced.  The goal of the libextractor project [1] is to provide a
uniform interface for obtaining meta-data from different file formats.
libextractor is currently used by evidence [3], the file-manager for
the forthcomming version of Enlightenment [13], and GNUnet [4], an
anonymous, censorship-resistant peer-to-peer file-sharing system.  The
extract tool is a command-line interface to the library.  libextractor
is licensed under the GNU Public License [14].

libextractor shares some similarities with the popular file [11] tool
which uses the first bytes in a file to guess the mime-type.
libextractor differs from file that it tries to obtain much more
information than just the mime-type.  Depending on the file format,
libextractor can obtain additional information.  Examples of this
extra information include the name of the software used to create the
file, the author, descriptions, album titles, image dimensions or the
duration of the movie.

libextractor achieves this by using specific parser code for many
popular formats.  The list currently includes mp3, ogg, real-media,
mpeg, riff (avi), gif, jpeg, png, tiff, html, pdf, ps and zip as well
as generic methods such as mime-type detection.  Many other formats
exist [5], but among the more popular formats only a few proprietary
formats are not supported.  Integrating support for new formats is
easy since libextractor uses plugins to gather data.  libextractor
plugins are shared libraries that typically provide code to parse one
particular format.  At the end of the article we will show how to
integrate support for new formats into the library.  libextractor
gathers the meta-data obtained from the various plugins and provides
clients with a list of pairs consisting of a classification and a
character sequence.  The classification is used to organize the
meta-data into categories like title, creator, subject, description
and so on [6].


[ INSTALLING LIBEXTRACTOR AND USING EXTRACT ]

The simplest way to install libextractor is to use one of the binary
packages which are available online for many distributions.  Note
that under Debian, the extract tool is in a separate package extract
[15] and headers required to compile other applications against
libextractor are in libextractor0-devel [16].  If you want to compile
libextractor from source you will need an unusual amount of memory:
256 MB system memory is roughly the minimum, since gcc will take about
200 MB to compile one of the plugins.  Otherwise, compiling by hand
follows the usual sequence:

$ wget http://ovmj.org/libextractor/download/libextractor-0.3.1.tar.gz
$ tar xvfz libextractor-0.3.1.tar.gz
$ cd libextractor-0.3.1
$ ./configure --prefix=/usr/local
$ make
# make install

After installing libextractor, the extract tool can be used to obtain
meta-data from documents.  By default, the extract tool uses the
canonical set of plugins, which consists of all file-format-specific
plugins supported by the current version of libextractor together with
the mime-type detection plugin.  For example, extract returns something
like the following for LinuxJournal's webpage:

$ wget -q http://www.linuxjournal.com/
$ extract index.html
description - The Monthly Magazine of the Linux Community
keywords - linux, linux journal, magazine
author - Linux Journal  - The Premier Magazine of the Linux Community
title - Linux Journal  - The Premier Magazine of the Linux Community

If you are a user of bibtex [12] the option -b is likely to come in
handy to automatically create bibtex entries from documents that have
been properly equipped with meta-data:

$ wget -q http://www.copyright.gov/legislation/dmca.pdf
$ extract -b ~/dmca.pdf
% BiBTeX file
@misc{ unite2001the_d,
    title = "The Digital Millennium Copyright Act of 1998",
    author = "United States Copyright Office - jmf",
    note = "digital millennium copyright act circumvention technological protection management information online service provider liability limitation computer maintenance competitiion repair ephemeral recording webcasting distance education study vessel hull",
    year = "2001",
    month = "10",
    key = "Copyright Office Summary of the DMCA",
    pages = "18"
}

Another interesting option is "-B LANG".  This option loads one of the
language specific (but format-agnostic) plugins.  These plugins
attempt to find plaintext in a document by matching strings in the
document against a dictionary.  If the need for 200 MB of memory to
compile libextractor seems mysterious, the answer lies in these
plugins.  In order to be able to perform a fast dictionary search, a
bloomfilter [17] is created that allows fast probabilistic matching;
gcc finds the resulting datastructure a bit hard to swallow.  The
option -B is useful for formats that are undocumented or currently
unsupported.  Note that the printable plugins typically print the
entire text of the document in order.  A typical use is:

$ wget -q http://www.bayern.de/HDBG/polges.doc
$ extract -B de polges.doc | head -n 4 
unknown - FEE Politische Geschichte Bayerns Herausgegeben vom Haus der Geschichte als Heft der zur Geschichte und Kultur Redaktion Manfred Bearbeitung Otto Copyright Haus der Geschichte M�nchen Gestaltung f�rs Internet Rudolf Inhalt im.
unknown - und das Deutsche Reich.
unknown - und seine.
unknown - Henker im Zeitalter von Reformation und Gegenreformation.

This is a rather precise description of the text for a German
speaker.  The supported languages at the moment are Danish (da),
German (de), English (en), Spanish (es), Italian (it) and Norvegian
(no).  Supporting other languages is merely a question of adding
(free) dictionaries in an appropriate character set.  Further options
are described in the extract manpage (man 1 extract).


[ USING LIBEXTRACTOR IN YOUR PROJECTS ]

The shortest program using libextractor looks roughly like this
(compilation requires passing the option -Lextractor to gcc):

#include <extractor.h>                 
int main(int argc, char * argv[]) {
  EXTRACTOR_ExtractorList * plugins;
  EXTRACTOR_KeywordList   * md_list;
  plugins = EXTRACTOR_loadDefaultLibraries(); 
  md_list = EXTRACTOR_getKeywords(plugins, argv[1]);
  EXTRACTOR_printKeywords(stdout, md_list); 
  EXTRACTOR_freeKeywords(md_list); 
  EXTRACTOR_removeAll(plugins); /* unload plugins */
}

The EXTRACTOR_KeywordList is a simple linked list containing
a keyword and a keyword type.  For details and additional functions
for loading plugins and manipulating the keyword list, see the
libextractor manpage (man 3 libextractor).  Java programmers
should note that a Java class that uses JNI to communicate with
libextractor is also available.


[ WRITING PLUGINS ]

The most complicated thing when writing a new plugin for libextractor
is the writing of the actual parser for a specific format.
Nevertheless, the basic pattern is always the same.  The plugin
library must be called libextractor_XXX.so where XXX denotes the file
format of the plugin.  The library must export a method
libextractor_XXX_extract with the following signature:

struct EXTRACTOR_Keywords * 
libextractor_XXX_extract
   (char * filename,
    char * data,
    size_t size,
    struct EXTRACTOR_Keywords * prev);

The argument filename specifies the name of the file that is being
processed.  data is a pointer to the (typically mmapped) contents of
the file, and size is the filesize.  Most plugins to not make use of
the filename and just directly parse data directly, staring by
verifying that the header of the data matches the specific format.
prev is the list of keywords that have been extracted so far by other
plugins for the file.  The function is expected to return an updated
list of keywords.  If the format does not match the expectations of
the plugin, prev is returned.  Most plugins use a function like
addKeyword to extend the list:

static void addKeyword
   (struct EXTRACTOR_Keywords ** list,
    char * keyword,
    EXTRACTOR_KeywordType type) 
{
  EXTRACTOR_KeywordList * next;
  next = malloc(sizeof(EXTRACTOR_KeywordList));
  next->next = *list;
  next->keyword = keyword;
  next->keywordType = type;
  *list = next;
}

A typical use of addKeyword is to add the mime-type once the file format
has been established.  For example, the JPEG-extractor checks the first
bytes of the JPEG header and then either aborts or claims the file to be
a JPEG:

if ( (data[0] != 0xFF) || (data[1] != 0xD8) )
  return prev; /* not a JPEG */
addKeyword(&prev,
           strdup("image/jpeg"),
           EXTRACTOR_MIMETYPE);
/* ... more parsing code here ... */
return prev;

Note that the strdup here is important since the string will be
deallocated later, typically in EXTRACTOR_freeKeywords().  A list of
supported keyword classifications (EXTRACTOR_XXXX) can be found in the
extractor.h header file [18].



[ CONCLUSION ]

libextractor is a simple extensible C library for obtaining meta-data
from documents.  Its plugin architecture and broad support for formats
set it apart from format-specific tools.  The design is limited by the
fact that libextractor cannot be used to update meta-data, which more
specialized tools typically support.


[ REFERENCES ]

[1] http://ovmj.org/libextractor/  
[2] http://getid3.sf.net/ 
[3] http://evidence.sf.net/
[4] http://ovmj.org/GNUnet/
[5] http://www.wotsit.org/
[6] http://dublincore.org/documents/dcmi-terms/
[7] http://freshmeat.net/projects/aviinfo/
[8] http://freshmeat.net/projects/id3edit/
[9] http://freshmeat.net/projects/jpeginfo/
[10] http://freshmeat.net/projects/vocoditor/
[11] http://freshmeat.net/projects/file/
[12] http://dmoz.org/Computers/Software/Typesetting/TeX/BibTeX/
[13] http://enlightenment.org/
[14] http://www.gnu.org/licenses/gpl.html
[15] http://packages.debian.org/extract
[16] http://packages.debian.org/libextractor0-devel
[17] http://ovmj.org/GNUnet/download/bloomfilter.ps
[18] http://ovmj.org/libextractor/doxygen/html/extractor_8h-source.html