File System Archive Example (Data Only)

This example archives file data into an Accumulo table. Files with duplicate data are only stored once. The example has the following classes:

  • CharacterHistogram - A MapReduce that computes a histogram of byte frequency for each file and stores the histogram alongside the file data. An example use of the ChunkInputFormat.
  • ChunkCombiner - An Iterator that dedupes file data and sets their visibilities to a combined visibility based on current references to the file data.
  • ChunkInputFormat - An Accumulo InputFormat that provides keys containing file info (List<Entry<Key,Value») and values with an InputStream over the file (ChunkInputStream).
  • ChunkInputStream - An input stream over file data stored in Accumulo.
  • FileDataIngest - Takes a list of files and archives them into Accumulo keyed on hashes of the files.
  • FileDataQuery - Retrieves file data based on the hash of the file. (Used by the dirlist.Viewer.)
  • KeyUtil - A utility for creating and parsing null-byte separated strings into/from Text objects.
  • VisibilityCombiner - A utility for merging visibilities into the form (VIS1) (VIS2) …

This example is coupled with the dirlist example. See README.dirlist for instructions.

If you haven’t already run the README.dirlist example, ingest a file with FileDataIngest.

$ ./bin/accumulo org.apache.accumulo.examples.simple.filedata.FileDataIngest instance zookeepers username password dataTable exampleVis 1000 $ACCUMULO_HOME/README

Open the accumulo shell and look at the data. The row is the MD5 hash of the file, which you can verify by running a command such as ‘md5sum’ on the file.

> scan -t dataTable

Run the CharacterHistogram MapReduce to add some information about the file.

$ bin/tool.sh lib/examples-simple*[^cs].jar org.apache.accumulo.examples.simple.filedata.CharacterHistogram instance zookeepers username password dataTable exampleVis exampleVis

Scan again to see the histogram stored in the ‘info’ column family.

> scan -t dataTable