Apache Accumulo 1.5.2 Release Notes

Apache Accumulo 1.5.2 is a maintenance release on the 1.5 version branch. This release contains changes from over 100 issues, comprised of bug fixes (client side and server side), new test cases, and updated Hadoop support contributed by over 30 different contributors and committers. As this is a maintenance release, Apache Accumulo 1.5.2 has no client API incompatibilities over Apache Accumulo 1.5.0 and 1.5.1 and requires no manual upgrade process. Users of 1.5.0 or 1.5.1 are strongly encouraged to update as soon as possible to benefit from the improvements.

Users who are new to Accumulo are encouraged to use a 1.6 release as opposed to the 1.5 line as development has already shifted towards the 1.6 line. For those who cannot or do not want to upgrade to 1.6, 1.5.2 is still an excellent choice over earlier versions in the 1.5 line.

Performance Improvements

Apache Accumulo 1.5.2 includes a number of performance-related fixes over previous versions.

Write-Ahead Log sync performance

The Write-Ahead Log (WAL) files are used to ensure durability of updates made to Accumulo. A sync is called on the file in HDFS to make sure that the changes to the WAL are persisted to disk, which allows Accumulo to recover in the case of failure. ACCUMULO-2766 fixed an issue where an operation against a WAL would unnecessarily wait for multiple syncs, slowing down the ingest on the system.

Minor-Compactions not aggressive enough

On a system with ample memory provided to Accumulo, long hold-times were observed which blocks the ingest of new updates. Trying to free more server-side memory by running minor compactions more frequently increased the overall throughput on the node. These changes were made in ACCUMULO-2905.

HeapIterator optimization

Iterators, a notable feature of Accumulo, are provided to users as a server-side programming construct, but are also used internally for numerous server operations. One of these system iterator is the HeapIterator which implements a PriorityQueue of other Iterators. One way this iterator is used is to merge multiple files in HDFS to present a single, sorted stream of Key-Value pairs. ACCUMULO-2827 introduces a performance optimization to the HeapIterator which can improve the speed of the HeapIterator in common cases.

Write-Ahead log sync implementation

In Hadoop-2, two implementations of sync are provided: hflush and hsync. Both of these methods provide a way to request that the datanodes write the data to the underlying medium and not just hold it in memory (the fsync syscall). While both of these methods inform the Datanodes to sync the relevant block(s), hflush does not wait for acknowledgement from the Datanodes that the sync finished, where hsync does. To provide the most reliable system “out of the box”, Accumulo defaults to hsync so that your data is as secure as possible in a variety of situations (notably, unexpected power outages).

The downside is that performance tends to suffer because waiting for a sync to disk is a very expensive operation. ACCUMULO-2842 introduces a new system property, tserver.wal.sync.method, that lets users to change the HDFS sync implementation from hsync to hflush. Using hflush instead of hsync may result in about a 30% increase in ingest performance.

For users upgrading from Hadoop-1 or Hadoop-0.20 releases, hflush is the equivalent of how sync was implemented in these older versions of Hadoop and should give comparable performance.

Server-side mutation queue size

When users desire writes to be as durable as possible, using hsync, the ingest performance of the system can be improved by increasing the tserver.mutation.queue.max property. The cost of this change is that it will cause TabletServers to use additional memory per writer. In 1.5.1, the value of this parameter defaulted to a conservative 256K, which resulted in sub-par ingest performance.

1.5.2 and ACCUMULO-3018 increases this buffer to 1M which has a noticeable positive impact on ingest performance with a minimal increase in TabletServer memory usage.

Notable Bug Fixes

Fixes MapReduce package name change

1.5.1 inadvertently included a change to RangeInputSplit which created an incompatibility with 1.5.0. The original class has been restored to ensure that users accessing the RangeInputSplit class do not have to alter their client code. See ACCUMULO-2586 for more information

Add configurable maximum frame size to Apache Thrift proxy

The Thrift proxy server was subject to memory exhaustion, typically due to bad input, where the server would attempt to allocate a very large buffer and die in the process. ACCUMULO-2658 introduces a configuration parameter, like ACCUMULO-2360, to prevent this error.

Offline tables can prevent tablet balancing

Before 1.5.2, when a table with many tablets was created, ingested into, and taken offline, tablet balancing may have stoppped. This would happen if there were tablet migrations for the table, because the migrations couldn’t occur. The balancer will not run when there are outstanding migrations; therefore, a system could become unbalanced. ACCUMULO-2694 introduces a fix to ensure that offline tables do not block balancing and improves the server-side logging.

MiniAccumuloCluster process management

MiniAccumuloCluster had a few issues which could cause deadlock or a method that never returns. Most of these are related to management of the Accumulo processes (ACCUMULO-2764, ACCUMULO-2985, and ACCUMULO-3055).

IteratorSettings not correctly serialized in RangeInputSplit

The Writable interface methods on the RangeInputSplit class accidentally omitted calls to serialize the IteratorSettings configured for the Job. ACCUMULO-2962 fixes the serialization and adds some additional tests.

Constraint violation causes hung scans

A failed bulk import transaction had the ability to create an infinitely retrying loop due to a constraint violation. This directly prevents scans from completing, but will also hang compactions. ACCUMULO-3096 fixes the issue so that the constraint no longer hangs the entire system.

Documentation

The following documentation updates were made:

Testing

Each unit and functional test only runs on a single node, while the RandomWalk and Continuous Ingest tests run on any number of nodes. Agitation refers to randomly restarting Accumulo processes and Hadoop Datanode processes, and, in HDFS High-Availability instances, forcing NameNode failover.

OS Hadoop Nodes ZooKeeper HDFS High-Availability Tests
Gentoo Apache 2.6.0-SNAPSHOT 1 Apache 3.4.5 No Unit and Functional Tests, ContinuousIngest w/ verification (1B entries)
CentOS 6 Apache 2.3.0 20 Apache 3.4.5 No 24/hr RandomWalk, 24/hr ContinuousIngest w/ verification w/ and w/o agitation (30B and 23B entries)