Apache Accumulo 1.6.1

25 Sep 2014

Apache Accumulo 1.6.1 is a maintenance release on the 1.6 version branch. This release contains changes from over 175 issues, comprised of bug-fixes, performance improvements and better test cases.

Below are resources for this release:

As this is a maintenance release, Apache Accumulo 1.6.1 has no client API incompatibilities over Apache Accumulo 1.6.0. Users of 1.6.0 are strongly encouraged to update as soon as possible to benefit from the improvements.

New users are encouraged to use this release over 1.6.0 or any other older releases. For information about improvements since Accumulo 1.5, see the 1.6.0 release notes.

Performance Improvements

Apache Accumulo 1.6.1 includes a number of performance-related fixes over previous versions. Many of these improvements were also included in the recently released Apache Accumulo 1.5.2.

Write-Ahead Log sync performance

The Write-Ahead Log (WAL) files are used to ensure durability of updates made to Accumulo. A sync is called on the file in HDFS to make sure that the changes to the WAL are persisted to disk, which allows Accumulo to recover in the case of failure. ACCUMULO-2766 fixed an issue where an operation against a WAL would unnecessarily wait for multiple syncs, slowing down the ingest on the system.

Minor-Compactions not aggressive enough

On a system with ample memory provided to Accumulo, long hold-times were observed which blocks the ingest of new updates. Trying to free more server-side memory by running minor compactions more frequently increased the overall throughput on the node. These changes were made in ACCUMULO-2905.

HeapIterator optimization

Iterators, a notable feature of Accumulo, are provided to users as a server-side programming construct, but are also used internally for numerous server operations. One of these system iterator is the HeapIterator which implements a PriorityQueue of other Iterators. One way this iterator is used is to merge multiple files in HDFS to present a single, sorted stream of Key-Value pairs. ACCUMULO-2827 introduces a performance optimization to the HeapIterator which can improve the speed of the HeapIterator in common cases.

Write-Ahead log sync implementation

In Hadoop-2, two implementations of sync are provided: hflush and hsync. Both of these methods provide a way to request that the datanodes write the data to the underlying medium and not just hold it in memory (the fsync syscall). While both of these methods inform the Datanodes to sync the relevant block(s), hflush does not wait for acknowledgement from the Datanodes that the sync finished, where hsync does. To provide the most reliable system “out of the box”, Accumulo defaults to hsync so that your data is as secure as possible in a variety of situations (notably, unexpected power outages).

The downside is that performance tends to suffer because waiting for a sync to disk is a very expensive operation. ACCUMULO-2842 introduces a new system property, tserver.wal.sync.method, that lets users to change the HDFS sync implementation from hsync to hflush. Using hflush instead of hsync may result in about a 30% increase in ingest performance.

For users upgrading from Hadoop-1 or Hadoop-0.20 releases, hflush is the equivalent of how sync was implemented in these older versions of Hadoop and should give comparable performance.

Other improvements

Use of Hadoop CredentialProviders

Apache Hadoop 2.6.0 introduced a new API aimed at providing ways to separate sensitive values from being stored in plaintext as a part of HADOOP-10607. Accumulo has had two sensitive configuration properties stored in accumulo-site.xml for every standard installation: instance.secret and trace.token.property.password. If either of these properties are compromised, it could lead to unwanted access of Accumulo. ACCUMULO-2464 modifies Accumulo so that it can stored any sensitive configuration properties in a Hadoop CredentialProvider. With sensitive values removed from accumulo-site.xml, it can be shared without concern and security can be focused solely on the CredentialProvider.

Notable Bug Fixes

Add configurable maximum frame size to Apache Thrift proxy

The Thrift proxy server was subject to memory exhaustion, typically due to bad input, where the server would attempt to allocate a very large buffer and die in the process. ACCUMULO-2658 introduces a configuration parameter, like ACCUMULO-2360, to prevent this error.

Offline tables can prevent tablet balancing

Before 1.6.1, when a table with many tablets was created, ingested into, and taken offline, tablet balancing may have stopped. This would happen if there were tablet migrations for the table, because the migrations couldn’t occur. The balancer will not run when there are outstanding migrations; therefore, a system could become unbalanced. ACCUMULO-2694 introduces a fix to ensure that offline tables do not block balancing and improves the server-side logging.

MiniAccumuloCluster process management

MiniAccumuloCluster had a few issues which could cause deadlock or a method that never returns. Most of these are related to management of the Accumulo processes (ACCUMULO-2764, ACCUMULO-2985, and ACCUMULO-3055).

IteratorSettings not correctly serialized in RangeInputSplit

The Writable interface methods on the RangeInputSplit class accidentally omitted calls to serialize the IteratorSettings configured for the Job. ACCUMULO-2962 fixes the serialization and adds some additional tests.

Constraint violation causes hung scans

A failed bulk import transaction had the ability to create an infinitely retrying loop due to a constraint violation. This directly prevents scans from completing, but will also hang compactions. ACCUMULO-3096 fixes the issue so that the constraint no longer hangs the entire system.

Unable to upgrade cleanly from 1.5

When upgrading a table from 1.5.1 to 1.6.0, a user experienced an error where the table never came online. ACCUMULO-2974 fixes an issue from the change of file references stored as absolute paths instead of relative paths in the Accumulo metadata table.

Guava dependency changed

ACCUMULO-3100 lowered the dependency on Guava from 15.0 to 14.0.1. This dependency now matches what Hadoop is depending on for the 2.x.y version line. Depending on a newer version of Guava introduces many issues stemming from deprecated classes in use by Hadoop which have been removed. While installations of Accumulo will likely work as expected with newer versions of Guava on the classpath (because the Hadoop processes will have their own classpath), use of MiniDfsClusters with the new Guava version will result in errors.

Users can attempt to use a newer version of Guava on the Accumulo server classpath; however, the success is dependent on Hadoop client libraries not using (missing) Guava methods internally.

Scanners eat InterruptedException

Scanners previously consumed InterruptedExceptions and did not exit after. In multi-threaded environments, this is very problematic as there is no means to stop the Scanner from reading data. ACCUMULO-3030 fixes the Scanner so that interrupts are observed and the Scanner exits as expected.

Documentation

The following documentation updates were made:

Testing

Each unit and functional test only runs on a single node, while the RandomWalk and Continuous Ingest tests run on any number of nodes. Agitation refers to randomly restarting Accumulo processes and Hadoop Datanode processes, and, in HDFS High-Availability instances, forcing NameNode failover.

OS Hadoop Nodes ZooKeeper HDFS HA Tests
Gentoo Apache 2.6.0-SNAPSHOT 2 Apache 3.4.5 No Unit and Functional Tests, ContinuousIngest w/ verification (2B entries)
CentOS 6 Apache 2.3.0 20 Apache 3.4.5 No 24/hr RandomWalk, ContinuousIngest w/ verification w/ and w/o agitation (17B entries), 24hr Randomwalk test

View all releases in the archive