Accumulo 2.x Documentation  >>  Administration  >>  Monitoring & Metrics

Monitoring & Metrics

Monitoring

Accumulo Monitor

The Accumulo Monitor provides a web UI with information on the health and status of Accumulo.

The monitor can be viewed at:

  • http://localhost:9995 - if Accumulo is running locally (for development)
  • http://<MONITOR_HOST>:9995/ - if Accumulo is running on a cluster

The Overview page (shown below) contains some summary information about the Accumulo instance and graphs displaying various metrics over time. These include ingest and scan performance and other useful measurements.

monitor overview

The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table). Metrics typically include number of entries (key/value pairs), ingest and query rates. The number of running scans, major and minor compactions are in the form number_running (number_queued). Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.

monitor master

The Server Activity page graphically displays tablet server status, with each server represented as a circle or square. Different metrics may be assigned to the nodes’ color and speed of oscillation. The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization). Similarly, the Overall Max metric picks the metric with the maximum normalized value.

The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle. The Traces page displays data for recent traces performed (see the following section for information on tracing). The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes. Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.

The Accumulo monitor does a best-effort to not display any sensitive information to users; however, the monitor is intended to be a tool used with care. It is not a production-grade webservice. It is a good idea to whitelist access to the monitor via an authentication proxy or firewall. It is strongly recommended that the Monitor is not exposed to any publicly-accessible networks.

Things highlighted in yellow may be in need of attention. If anything is highlighted in red on the monitor page, it is something that definitely needs attention.

SSL

SSL may be enabled for the monitor page by setting the following properties in the accumulo.properties file:

If the Accumulo conf directory has been configured (in particular the accumulo-env.sh file must be set up), the accumulo-util gen-monitor-cert command can be used to create the keystore and truststore files with random passwords. The command will print out the properties that need to be added to the accumulo.properties file. The stores can also be generated manually with the Java keytool command, whose usage can be seen in the accumulo-util script.

If desired, the SSL ciphers allowed for connections can be controlled via the following properties in accumulo.properties:

If SSL is enabled, the monitor URL can only be accessed via https. This also allows you to access the Accumulo shell through the monitor page. The left navigation bar will have a new link to Shell. An Accumulo user name and password must be entered for access to the shell.

Metrics

Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.

Legacy Metrics

Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can be enabled by setting general.legacy.metrics to true in accumulo.properties and placing the accumulo-metrics.xml configuration file on the classpath (which is typically done by placing the file in the conf/ directory). A template for accumulo-metrics.xml can be found in conf/templates of the Accumulo tarball.

Hadoop Metrics2

Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia. All metric sources are exposed via JMX when using Metrics2.

Metrics2 is configured by examining the classpath for a file that matches hadoop-metrics2*.properties. The Accumulo tarball contains an example hadoop-metrics2-accumulo.properties file in conf/templates which can be copied to conf/ to place on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to remove ambiguity. The contents of hadoop-metrics2-accumulo.properties can be added to a central hadoop-metrics2.properties in $HADOOP_CONF_DIR.

As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files from growing without bound.

Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check the Accumulo processes log files when debugging missing metrics output.

Below are additional resources for configuring Metrics2:

Find documentation for all releases in the archive