Apache Accumulo 1.9.2

19 Jul 2018

Apache Accumulo 1.9.2 contains fixes for critical write-ahead log bugs. Users of any previous version of 1.8 or 1.9 are encouraged to upgrade immediately to avoid those issues.

  • User Manual - In-depth developer and administrator documentation
  • Javadocs - Accumulo 1.9 API
  • Examples - Code with corresponding readme files that give step by step instructions for running example code

Notable Changes

Fixes for Critical WAL Bugs (affects versions 1.8.0-1.9.1)

Multiple bugs were fixed in 1.9.2 which affects the behavior of the write-ahead log mechanism. These vary in significance, ranging from moderate to critical.

  • #537 - (Critical) Since 1.8.0, a bug existed which could cause some write-ahead logs to be removed (garbage collected) before Accumulo was finished with them. These removed logs could have contained important state tracking information. Without the state contained in these logs, some data in the remaining logs could have been replayed into a table when not needed. This could have reintroduced deleted data, or introduced duplicate data (which can interfere with combiners).
  • #538 - (Moderate) A bug was introduced in 1.9.1 which resulted in some false positive IllegalStateExceptions to occur, preventing log recovery.
  • #539 - (Moderate) Since 1.8.0, a race condition existed which could cause a log file which contains data to be recovered to not be recorded, thus making it invisible to recovery, if a tserver died within a very small window. #559 fixes this issue and may also fix a 1.9.1 deadlock caused by the fix for #441.

Even if you primarily use bulk ingest, Accumulo’s own metadata tables can be affected by these bugs, causing unexpected behavior after an otherwise routine and recoverable server failure. As such, these bugs should be a concern to all users.

Fixes for concurrency bugs gathering table information (affects 1.8.0-1.9.1)

Bugs were found with the master.status.threadpool.size property. If this property were set to a value other than 1, it could cause 100% CPU, hanging, or ConcurrentModificationExceptions.

These bugs were fixed in #546.

Caching of file lengths

RFiles stores metadata at the end of file. When opening a rfile Accumulo seeks to the end and reads metadata. To do this seek the file length is needed. Before opening a file its length is requested from the namenode. This can add more load to a busy namenode. To alleviate this, a small cache of file lengths was added in #467.

Monitor time unit bug

A bug was found in the monitor which displayed time durations (for example, those pertaining to bulk imports) in incorrect time units.

This bug was fixed in #553.

Other Changes

  • GitHub - List of issues tracked on GitHub corresponding to this release
  • 1.9.1 release notes - Release notes showing changes in the previous release

Upgrading

View the Upgrading Accumulo documentation for guidance.

Testing

  • All ITs passed with Hadoop 3.0.0 (hadoop.profile=3)
  • All ITs passed with Hadoop 2.6.4 (hadoop.profile=2)
  • Ran 3 continuous ingesters successfully for 24 hours on a 10 node cluster with agitation and pausing. Verification for all 3 tests was successful.
  • Ran continuous ingest for 24 hours and verified without agitation on a 10 node cluster.
  • Tested Apache Fluo build and ITs passed against this version.
  • Ran a single-node cluster with Uno and created a table, ingested data, flushed, compacted, scanned, and deleted the table.

View all releases in the archive