Package org.apache.accumulo.core.iterators.user

Class TransformingIterator

java.lang.Object
org.apache.accumulo.core.iterators.WrappingIterator
org.apache.accumulo.core.iterators.user.TransformingIterator
All Implemented Interfaces:
OptionDescriber, SortedKeyValueIterator<Key,Value>, YieldingKeyValueIterator<Key,Value>
public abstract class TransformingIterator extends WrappingIterator implements OptionDescriber
The TransformingIterator allows portions of a key (except for the row) to be transformed. This iterator handles the details that come with modifying keys (i.e., that the sort order could change). In order to do so, however, the iterator must put all keys sharing the same prefix in memory. Prefix is defined as the parts of the key that are not modified by this iterator. That is, if the iterator modifies column qualifier and timestamp, then the prefix is row and column family. In that case, the iterator must load all column qualifiers for each row/column family pair into memory. Given this constraint, care must be taken by users of this iterator to ensure it is not run in such a way that will overrun memory in a tablet server.

If the implementing iterator is transforming column families, then it must also override untransformColumnFamilies(Collection) to handle the case when column families are fetched at scan time. The fetched column families will/must be in the transformed space, and the untransformed column families need to be passed to this iterator's source. If it is not possible to write a reverse transformation (e.g., the column family transformation depends on the row value or something like that), then the iterator must not fetch specific column families (or only fetch column families that are known to not transform at all).

If the implementing iterator is transforming column visibilities, then users must be careful NOT to fetch column qualifiers from the scanner.

If the implementing iterator is transforming column visibilities, then the user should be sure to supply authorizations via the AUTH_OPT iterator option (note that this is only necessary for scan scope iterators). The supplied authorizations should be in the transformed space, but the authorizations supplied to the scanner should be in the untransformed space. That is, if the iterator transforms A to 1, B to 2, C to 3, etc, then the auths supplied when the scanner is constructed should be A,B,C,... and the auths supplied to the iterator should be 1,2,3,... The reason for this is that the scanner performs security filtering before this iterator is called, so the authorizations need to be in the original untransformed space. Since the iterator can transform visibilities, it is possible that it could produce visibilities that the user cannot see, so the transformed keys must be tested to ensure the user is allowed to view them. Note that this test is not necessary when the iterator is not used in the scan scope since no security filtering is performed during major and minor compactions. It should also be noted that this iterator implements the security filtering rather than relying on a follow-on iterator to do it so that we ensure the test is performed.

  • Field Details

    • AUTH_OPT

      public static final String AUTH_OPT
      See Also:

    • MAX_BUFFER_SIZE_OPT

      public static final String MAX_BUFFER_SIZE_OPT
      See Also:

    • keys

      protected ArrayList<org.apache.accumulo.core.util.Pair<Key,Value>> keys

    • keyPos

      protected int keyPos

    • scanning

      protected boolean scanning

    • seekRange

      protected Range seekRange

    • seekColumnFamilies

      protected Collection<ByteSequence> seekColumnFamilies

    • seekColumnFamiliesInclusive

      protected boolean seekColumnFamiliesInclusive

  • Constructor Details

    • TransformingIterator

      public TransformingIterator()

  • Method Details

    • init

      public void init(SortedKeyValueIterator<Key,Value> source, Map<String,String> options, IteratorEnvironment env) throws IOException
      Description copied from interface: SortedKeyValueIterator
      Initializes the iterator. Data should not be read from the source in this method.
      Specified by:
      init in interface SortedKeyValueIterator<Key,Value>
      Overrides:
      init in class WrappingIterator
      Parameters:
      source - SortedKeyValueIterator source to read data from.
      options - Map map of string option names to option values.
      env - IteratorEnvironment environment in which iterator is being run, provided by Accumulo itself and is expected to be non-null.
      Throws:
      IOException - unused.

    • describeOptions

      public OptionDescriber.IteratorOptions describeOptions()
      Description copied from interface: OptionDescriber
      Gets an iterator options object that contains information needed to configure this iterator. This object will be used by the accumulo shell to prompt the user to input the appropriate information.
      Specified by:
      describeOptions in interface OptionDescriber
      Returns:
      an iterator options object

    • validateOptions

      public boolean validateOptions(Map<String,String> options)
      Description copied from interface: OptionDescriber
      Check to see if an options map contains all options required by an iterator and that the option values are in the expected formats.
      Specified by:
      validateOptions in interface OptionDescriber
      Parameters:
      options - a map of option names to option values
      Returns:
      true if options are valid, false otherwise (IllegalArgumentException preferred)

    • deepCopy

      public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment env)
      Description copied from interface: SortedKeyValueIterator
      Creates a deep copy of this iterator as though seek had not yet been called. init should be called on an iterator before deepCopy is called. init should not need to be called on the copy that is returned by deepCopy; that is, when necessary init should be called in the deepCopy method on the iterator it returns. The behavior is unspecified if init is called after deepCopy either on the original or the copy. A proper implementation would call deepCopy on the source.
      Specified by:
      deepCopy in interface SortedKeyValueIterator<Key,Value>
      Overrides:
      deepCopy in class WrappingIterator
      Parameters:
      env - IteratorEnvironment environment in which iterator is being run, provided by Accumulo itself and is expected to be non-null.
      Returns:
      SortedKeyValueIterator a copy of this iterator (with the same source and settings).

    • hasTop

      public boolean hasTop()
      Description copied from interface: SortedKeyValueIterator
      Returns true if the iterator has more elements. Note that if this iterator has yielded (@see YieldingKeyValueIterator.enableYielding(YieldCallback)), this this method must return false.
      Specified by:
      hasTop in interface SortedKeyValueIterator<Key,Value>
      Overrides:
      hasTop in class WrappingIterator
      Returns:
      true if the iterator has more elements.

    • getTopKey

      public Key getTopKey()
      Description copied from interface: SortedKeyValueIterator
      Returns top key. Can be called 0 or more times without affecting behavior of next() or hasTop(). Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.

      For performance reasons, iterators reserve the right to reuse objects returned by getTopKey when SortedKeyValueIterator.next() is called, changing the data that the object references. Iterators that need to save an object returned by getTopKey ought to copy the object's data into a new object in order to avoid aliasing bugs.

      Specified by:
      getTopKey in interface SortedKeyValueIterator<Key,Value>
      Overrides:
      getTopKey in class WrappingIterator
      Returns:
      K

    • getTopValue

      public Value getTopValue()
      Description copied from interface: SortedKeyValueIterator
      Returns top value. Can be called 0 or more times without affecting behavior of next() or hasTop().

      For performance reasons, iterators reserve the right to reuse objects returned by getTopValue when SortedKeyValueIterator.next() is called, changing the underlying data that the object references. Iterators that need to save an object returned by getTopValue ought to copy the object's data into a new object in order to avoid aliasing bugs.

      Specified by:
      getTopValue in interface SortedKeyValueIterator<Key,Value>
      Overrides:
      getTopValue in class WrappingIterator
      Returns:
      V

    • next

      public void next() throws IOException
      Description copied from interface: SortedKeyValueIterator
      Advances to the next K,V pair. Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.
      Specified by:
      next in interface SortedKeyValueIterator<Key,Value>
      Overrides:
      next in class WrappingIterator
      Throws:
      IOException - if an I/O error occurs.

    • seek

      public void seek(Range range, Collection<ByteSequence> columnFamilies, boolean inclusive) throws IOException
      Description copied from interface: SortedKeyValueIterator
      Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns. An iterator does not have to stop at the end of the range. The whole range is provided so that iterators can make optimizations. Seek may be called multiple times with different parameters after SortedKeyValueIterator.init(org.apache.accumulo.core.iterators.SortedKeyValueIterator<K, V>, java.util.Map<java.lang.String, java.lang.String>, org.apache.accumulo.core.iterators.IteratorEnvironment) is called. Iterators that examine groups of adjacent key/value pairs (e.g. rows) to determine their top key and value should be sure that they properly handle a seek to a key in the middle of such a group (e.g. the middle of a row). Even if the client always seeks to a range containing an entire group (a,c), the tablet server could send back a batch of entries corresponding to (a,b], then reseek the iterator to range (b,c) when the scan is continued. columnFamilies is used, at the lowest level, to determine which data blocks inside of an RFile need to be opened for this iterator. This set of data blocks is also the set of locality groups defined for the given table. If no columnFamilies are provided, the data blocks for all locality groups inside of the correct RFile will be opened and seeked in an attempt to find the correct start key, regardless of the startKey in the range. In an Accumulo instance in which multiple locality groups exist for a table, it is important to ensure that columnFamilies is properly set to the minimum required column families to ensure that data from separate locality groups is not inadvertently read.
      Specified by:
      seek in interface SortedKeyValueIterator<Key,Value>
      Overrides:
      seek in class WrappingIterator
      Parameters:
      range - Range of keys to iterate over.
      columnFamilies - Collection of column families to include or exclude.
      inclusive - boolean that indicates whether to include (true) or exclude (false) column families.
      Throws:
      IOException - if an I/O error occurs.

    • transformKeys

      protected void transformKeys() throws IOException
      Reads all keys matching the first key's prefix from the source iterator, transforms them, and sorts the resulting keys. Transformed keys that fall outside of our seek range or can't be seen by the user are excluded.
      Throws:
      IOException

    • includeTransformedKey

      protected boolean includeTransformedKey(Key transformedKey)
      Determines whether or not to include transformedKey in the output. It is possible that transformation could have produced a key that falls outside of the seek range, a key with a visibility the user can't see, a key with a visibility that doesn't parse, or a key with a column family that wasn't fetched. We only do some checks (outside the range, user can see) if we're scanning. The range check is not done for major/minor compaction since seek ranges won't be in our transformed key space and we will never change the row so we can't produce keys that would fall outside the tablet anyway.
      Parameters:
      transformedKey - the key to check
      Returns:
      true if the key should be included and false if not

    • canSee

      protected boolean canSee(Key key)
      Indicates whether or not the user is able to see key. If the user has not supplied authorizations, or the iterator is not in the scan scope, then this method simply returns true. Otherwise, key's column visibility is tested against the user-supplied authorizations, and the test result is returned. For performance, the test results are cached so that the same visibility is not tested multiple times.
      Parameters:
      key - the key to test
      Returns:
      true if the key is visible or iterator is not scanning, and false if not

    • canSeeColumnFamily

      protected boolean canSeeColumnFamily(Key key)
      Indicates whether or not key can be seen, according to the fetched column families for this iterator.
      Parameters:
      key - the key whose column family is to be tested
      Returns:
      true if key's column family is one of those fetched in the set passed to our seek(Range, Collection, boolean) method

    • computeReseekRange

      protected Range computeReseekRange(Range range)
      Possibly expand range to include everything for the key prefix we are working with. That is, if our prefix is ROW_COLFAM, then we need to expand the range so we're sure to include all entries having the same row and column family as the start/end of the range.
      Parameters:
      range - the range to expand
      Returns:
      the modified range

    • isSetAfterPart

      protected boolean isSetAfterPart(Key key, PartialKey part)
      Indicates whether or not any part of key excluding part is set. For example, if part is ROW_COLFAM_COLQUAL, then this method determines whether or not the column visibility, timestamp, or delete flag is set on key.
      Parameters:
      key - the key to check
      part - the part of the key that doesn't need to be checked (everything after does)
      Returns:
      true if anything after part is set on key, and false if not

    • copyPartialKey

      protected Key copyPartialKey(Key key, PartialKey part)
      Creates a copy of key, copying only the parts of the key specified in part. For example, if part is ROW_COLFAM_COLQUAL, then this method would copy the row, column family, and column qualifier from key into a new key.
      Parameters:
      key - the key to copy
      part - the parts of key to copy
      Returns:
      the new key containing part of key

    • replaceColumnFamily

      protected Key replaceColumnFamily(Key originalKey, org.apache.hadoop.io.Text newColFam)
      Make a new key with all parts (including delete flag) coming from originalKey but use newColFam as the column family.

    • replaceColumnQualifier

      protected Key replaceColumnQualifier(Key originalKey, org.apache.hadoop.io.Text newColQual)
      Make a new key with all parts (including delete flag) coming from originalKey but use newColQual as the column qualifier.

    • replaceColumnVisibility

      protected Key replaceColumnVisibility(Key originalKey, org.apache.hadoop.io.Text newColVis)
      Make a new key with all parts (including delete flag) coming from originalKey but use newColVis as the column visibility.

    • replaceKeyParts

      protected Key replaceKeyParts(Key originalKey, org.apache.hadoop.io.Text newColFam, org.apache.hadoop.io.Text newColQual, org.apache.hadoop.io.Text newColVis)
      Make a new key with a column family, column qualifier, and column visibility. Copy the rest of the parts of the key (including delete flag) from originalKey.

    • replaceKeyParts

      protected Key replaceKeyParts(Key originalKey, org.apache.hadoop.io.Text newColQual, org.apache.hadoop.io.Text newColVis)
      Make a new key with a column qualifier, and column visibility. Copy the rest of the parts of the key (including delete flag) from originalKey.

    • untransformColumnFamilies

      protected Collection<ByteSequence> untransformColumnFamilies(Collection<ByteSequence> columnFamilies)
      Reverses the transformation applied to column families that are fetched at seek time. If this iterator is transforming column families, then this method should be overridden to reverse the transformation on the supplied collection of column families. This is necessary since the fetch/seek will be performed in the transformed space, but when passing the column family set on to the source, the column families need to be in the untransformed space.
      Parameters:
      columnFamilies - the column families that have been fetched at seek time
      Returns:
      the untransformed column families that would transform info columnFamilies

    • getKeyPrefix

      protected abstract PartialKey getKeyPrefix()
      Indicates the prefix of keys that will be transformed by this iterator. In other words, this is the part of the key that will not be transformed by this iterator. For example, if this method returns ROW_COLFAM, then transformKeys() may be changing the column qualifier, column visibility, or timestamp, but it won't be changing the row or column family.
      Returns:
      the part of the key this iterator is not transforming

    • transformRange

      protected abstract void transformRange(SortedKeyValueIterator<Key,Value> input, TransformingIterator.KVBuffer output) throws IOException
      Transforms input. This method must not change the row part of the key, and must only change the parts of the key after the return value of getKeyPrefix(). Implementors must also remember to copy the delete flag from originalKey onto the new key. Or, implementors should use one of the helper methods to produce the new key. See any of the replaceKeyParts methods.
      Parameters:
      input - An iterator over a group of keys with the same prefix. This iterator provides an efficient view, bounded by the prefix, of the underlying iterator and can not be seeked.
      output - An output buffer that holds transformed key values. All key values added to the buffer must have the same prefix as the input keys.
      Throws:
      IOException
      See Also:

    • setAuthorizations

      public static void setAuthorizations(IteratorSetting config, Authorizations auths)
      Configure authorizations used for post transformation filtering.

    • setMaxBufferSize

      public static void setMaxBufferSize(IteratorSetting config, long maxBufferSize)
      Configure the maximum amount of memory that can be used for transformation. If this memory is exceeded an exception will be thrown.
      Parameters:
      maxBufferSize - size in bytes