Accumulo 2.x Documentation >> Administration >> Monitoring & Metrics
Monitoring & Metrics
The Accumulo Monitor provides a web UI with information on the health and status of Accumulo.
The monitor can be viewed at:
- http://localhost:9995 - if Accumulo is running locally (for development)
http://<MONITOR_HOST>:9995/- if Accumulo is running on a cluster
The Overview page (shown below) contains some summary information about the Accumulo instance and graphs displaying various metrics over time. These include ingest and scan performance and other useful measurements.
The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table). Metrics typically include number of entries (key/value pairs), ingest and query rates. The number of running scans, major and minor compactions are in the form number_running (number_queued). Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.
The Server Activity page graphically displays tablet server status, with each server represented as a circle or square. Different metrics may be assigned to the nodes’ color and speed of oscillation. The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization). Similarly, the Overall Max metric picks the metric with the maximum normalized value.
The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle. The Traces page displays data for recent traces performed (see the following section for information on tracing). The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes. Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.
The Accumulo monitor does a best-effort to not display any sensitive information to users; however, the monitor is intended to be a tool used with care. It is not a production-grade webservice. It is a good idea to whitelist access to the monitor via an authentication proxy or firewall. It is strongly recommended that the Monitor is not exposed to any publicly-accessible networks.
Things highlighted in yellow may be in need of attention. If anything is highlighted in red on the monitor page, it is something that definitely needs attention.
SSL may be enabled for the monitor page by setting the following properties in the
If the Accumulo conf directory has been configured (in particular the
accumulo-env.sh file must be set up), the
accumulo-util gen-monitor-cert command can be used to create the keystore and truststore files with random passwords. The command
will print out the properties that need to be added to the
accumulo.properties file. The stores can also be generated manually with the
keytool command, whose usage can be seen in the
If desired, the SSL ciphers allowed for connections can be controlled via the following properties in
If SSL is enabled, the monitor URL can only be accessed via https. This also allows you to access the Accumulo shell through the monitor page. The left navigation bar will have a new link to Shell. An Accumulo user name and password must be entered for access to the shell.
Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.
Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can
be enabled by setting general.legacy.metrics to
accumulo.properties and placing the
configuration file on the classpath (which is typically done by placing the file in the
conf/ directory). A template for
accumulo-metrics.xml can be found in
conf/templates of the Accumulo tarball.
Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia. All metric sources are exposed via JMX when using Metrics2.
Metrics2 is configured by examining the classpath for a file that matches
hadoop-metrics2*.properties. The Accumulo tarball
contains an example
hadoop-metrics2-accumulo.properties file in
conf/templates which can be copied to
conf/ to place
on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite
and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple
Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to
remove ambiguity. The contents of
hadoop-metrics2-accumulo.properties can be added to a central
As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files from growing without bound.
Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check the Accumulo processes log files when debugging missing metrics output.
Below are additional resources for configuring Metrics2: