public class ShingleFilter
extends org.apache.lucene.analysis.TokenFilter
A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.
For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".
This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_"). It does not handle a position increment of 0.
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_SHINGLE_SIZE
default maximum shingle size is 2.
|
static char[] |
FILLER_TOKEN
filler token for when positionIncrement is more than 1
|
static java.lang.String |
TOKEN_SEPARATOR
The string to use when joining adjacent tokens to form a shingle
|
Constructor and Description |
---|
ShingleFilter(org.apache.lucene.analysis.TokenStream input)
Construct a ShingleFilter with default shingle size.
|
ShingleFilter(org.apache.lucene.analysis.TokenStream input,
int maxShingleSize)
Constructs a ShingleFilter with the specified single size from the
TokenStream input |
ShingleFilter(org.apache.lucene.analysis.TokenStream input,
java.lang.String tokenType)
Construct a ShingleFilter with the specified token type for shingle tokens.
|
Modifier and Type | Method and Description |
---|---|
boolean |
incrementToken() |
org.apache.lucene.analysis.Token |
next()
Deprecated.
Will be removed in Lucene 3.0. This method is final, as it should
not be overridden. Delegates to the backwards compatibility layer.
|
org.apache.lucene.analysis.Token |
next(org.apache.lucene.analysis.Token reusableToken)
Deprecated.
Will be removed in Lucene 3.0. This method is final, as it should
not be overridden. Delegates to the backwards compatibility layer.
|
void |
reset() |
void |
setMaxShingleSize(int maxShingleSize)
Set the max shingle size (default: 2)
|
void |
setOutputUnigrams(boolean outputUnigrams)
Shall the output stream contain the input tokens (unigrams) as well as
shingles? (default: true.)
|
void |
setTokenType(java.lang.String tokenType)
Set the type of the shingle tokens produced by this filter.
|
getOnlyUseNewAPI, setOnlyUseNewAPI
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
public static final char[] FILLER_TOKEN
public static final int DEFAULT_MAX_SHINGLE_SIZE
public static final java.lang.String TOKEN_SEPARATOR
public ShingleFilter(org.apache.lucene.analysis.TokenStream input, int maxShingleSize)
TokenStream
input
input
- input streammaxShingleSize
- maximum shingle size produced by the filter.public ShingleFilter(org.apache.lucene.analysis.TokenStream input)
input
- input streampublic ShingleFilter(org.apache.lucene.analysis.TokenStream input, java.lang.String tokenType)
input
- input streamtokenType
- token type for shingle tokenspublic void setTokenType(java.lang.String tokenType)
tokenType
- token tokenTypepublic void setOutputUnigrams(boolean outputUnigrams)
outputUnigrams
- Whether or not the output stream shall contain
the input tokens (unigrams)public void setMaxShingleSize(int maxShingleSize)
maxShingleSize
- max size of output shinglespublic final boolean incrementToken() throws java.io.IOException
incrementToken
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public final org.apache.lucene.analysis.Token next(org.apache.lucene.analysis.Token reusableToken) throws java.io.IOException
next
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public final org.apache.lucene.analysis.Token next() throws java.io.IOException
next
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public void reset() throws java.io.IOException
reset
in class org.apache.lucene.analysis.TokenFilter
java.io.IOException
Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.