Package org.apache.accumulo.core.spi.compaction

Class RatioBasedCompactionPlanner

java.lang.Object
org.apache.accumulo.core.spi.compaction.RatioBasedCompactionPlanner
All Implemented Interfaces:
CompactionPlanner
public class RatioBasedCompactionPlanner extends Object implements CompactionPlanner
Finds the largest continuous set of small files that meet the compaction ratio and do not prevent future compactions.

The following configuration options are supported. Replace <service> with the name of the compaction service you are configuring.

  • Note that the CompactionCoordinator and at least one running Compactor must be assigned to the "large" compactor group.
  • compaction.service.<service>.planner.opts.maxOpen This determines the maximum number of files that will be included in a single compaction.
  • compaction.service.<service>.planner.opts.groups This is a json array of compactor group objects which have the following fields:
    Default Compaction Planner Group options
    Field Name Description
    group name of the compactor resource group (required)
    maxSize threshold sum of the input files (required for all but one of the configs)

    This 'groups' object provides information that is used for mapping a compaction job to a compactor group. The maxSize field determines the maximum size of compaction that will run in a group. The maxSize field can have a suffix of K,M,G for kilobytes, megabytes, or gigabytes and represents the sum of the input files for a given compaction. One group can have no max size and it will run everything that is too large for the other groups. If all groups have a max size, then system compactions will only run for compactions smaller than the largest max size. User and selector compactions will always run, even if there is no group for their size. These compactions will run on the group with the largest max size. The following example value for this property will create three separate compactor groups. "small" will run compactions of files whose file size sum is less than 100M, "medium" will run compactions of files whose file size sum is less than 500M, and "large" will run all other compactions on Compactors configured to pull jobs from the large group. 
     
 [
  {"group":"small", "maxSize":"100M"},
  {"group":"medium", "maxSize":"500M"},
  {"group": "large"}
 ]

Starting with Accumulo 2.1.3, this plugin will use the table config option "table.file.max". When the following four conditions are met, then this plugin will try to find a lower compaction ratio that will result in a compaction:

  1. When a tablet has no compactions running
  2. Its number of files exceeds table.file.max
  3. System compactions are not finding anything to compact
  4. No files are selected for user compaction
For example, given a tablet with 20 files, and table.file.max is 15 and no compactions are planned. If the compaction ratio is set to 3, then this plugin will find the largest compaction ratio less than 3 that results in a compaction. The lowest compaction ratio that will be considered in this search defaults to 1.1. Starting in 2.1.4, the lower bound for the search can be set using tserver.compaction.major.service.<service>.opts.lowestRatio
Since:
4.0.0
See Also:

  • Constructor Details

    • RatioBasedCompactionPlanner

      public RatioBasedCompactionPlanner()

  • Method Details

    • init

      public void init(CompactionPlanner.InitParameters params)
      Specified by:
      init in interface CompactionPlanner

    • makePlan

      public CompactionPlan makePlan(CompactionPlanner.PlanningParameters params)
      Description copied from interface: CompactionPlanner

      Plan what work a compaction service should do. The kind of compaction returned by CompactionPlanner.PlanningParameters.getKind() determines what must be done with the files returned by CompactionPlanner.PlanningParameters.getCandidates(). The following are the expectations for the candidates for each kind.

      • CompactionKind.SYSTEM The planner is not required to do anything with the candidates and can choose to compact zero or more of them. The candidates may represent a subset of all the files in the case where a user compaction is in progress or other compactions are running.
      • CompactionKind.USER and CompactionKind.SELECTED. The planner is required to eventually compact all candidates. Its ok to return a compaction plan that compacts a subset. When the planner compacts a subset, it will eventually be called again later. When it is called later the candidates will contain the files it did not compact and the results of any previous compactions it scheduled. The planner must eventually compact all of the files in the candidate set down to a single file. The compaction service will keep calling the planner until it does.

      For a user and selector compaction assume the same thing happens, it will result in a slightly different outcome.

      1. The candidate set passed to makePlan contains the files [F1,F2,F3,F4] and kind is USER
      2. The planner returns a job to compact files [F1,F2] on executor E1
      3. The compaction runs compacting [F1,F2] into file [F5]

      For the case above, eventually the planner will called again with a candidate set of [F3,F4,F5] and it must eventually compact those three files to one.

      When a planner returns a compactions plan, task will be queued on a compactor group. Previously queued task that do not match the latest plan are removed. The planner is called periodically, whenever a new file is added, and whenever a compaction finishes.

      Use CompactionPlanner.PlanningParameters.createPlanBuilder() to build the plan this function returns.

      Specified by:
      makePlan in interface CompactionPlanner
      See Also: