public class EdgeNGramTokenizer
extends org.apache.lucene.analysis.Tokenizer
This Tokenizer
create n-grams from the beginning edge or ending edge of a input token.
MaxGram can't be larger than 1024 because of limitation.
Modifier and Type | Class and Description |
---|---|
static class |
EdgeNGramTokenizer.Side
Specifies which side of the input the n-gram should be generated from
|
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_GRAM_SIZE |
static int |
DEFAULT_MIN_GRAM_SIZE |
static EdgeNGramTokenizer.Side |
DEFAULT_SIDE |
Constructor and Description |
---|
EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
java.io.Reader input,
EdgeNGramTokenizer.Side side,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
|
EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
java.io.Reader input,
java.lang.String sideLabel,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
|
EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source,
java.io.Reader input,
EdgeNGramTokenizer.Side side,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
|
EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source,
java.io.Reader input,
java.lang.String sideLabel,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
|
EdgeNGramTokenizer(java.io.Reader input,
EdgeNGramTokenizer.Side side,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
|
EdgeNGramTokenizer(java.io.Reader input,
java.lang.String sideLabel,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
|
Modifier and Type | Method and Description |
---|---|
void |
end() |
boolean |
incrementToken()
Returns the next token in the stream, or null at EOS.
|
org.apache.lucene.analysis.Token |
next()
Deprecated.
Will be removed in Lucene 3.0. This method is final, as it should
not be overridden. Delegates to the backwards compatibility layer.
|
org.apache.lucene.analysis.Token |
next(org.apache.lucene.analysis.Token reusableToken)
Deprecated.
Will be removed in Lucene 3.0. This method is final, as it should
not be overridden. Delegates to the backwards compatibility layer.
|
void |
reset() |
void |
reset(java.io.Reader input) |
getOnlyUseNewAPI, setOnlyUseNewAPI
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
public static final EdgeNGramTokenizer.Side DEFAULT_SIDE
public static final int DEFAULT_MAX_GRAM_SIZE
public static final int DEFAULT_MIN_GRAM_SIZE
public EdgeNGramTokenizer(java.io.Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
input
- Reader
holding the input to be tokenizedside
- the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source, java.io.Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
source
- AttributeSource
to useinput
- Reader
holding the input to be tokenizedside
- the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, java.io.Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
factory
- AttributeSource.AttributeFactory
to useinput
- Reader
holding the input to be tokenizedside
- the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic EdgeNGramTokenizer(java.io.Reader input, java.lang.String sideLabel, int minGram, int maxGram)
input
- Reader
holding the input to be tokenizedsideLabel
- the name of the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source, java.io.Reader input, java.lang.String sideLabel, int minGram, int maxGram)
source
- AttributeSource
to useinput
- Reader
holding the input to be tokenizedsideLabel
- the name of the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, java.io.Reader input, java.lang.String sideLabel, int minGram, int maxGram)
factory
- AttributeSource.AttributeFactory
to useinput
- Reader
holding the input to be tokenizedsideLabel
- the name of the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic final boolean incrementToken() throws java.io.IOException
incrementToken
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public final void end()
end
in class org.apache.lucene.analysis.TokenStream
public final org.apache.lucene.analysis.Token next(org.apache.lucene.analysis.Token reusableToken) throws java.io.IOException
next
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public final org.apache.lucene.analysis.Token next() throws java.io.IOException
next
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public void reset(java.io.Reader input) throws java.io.IOException
reset
in class org.apache.lucene.analysis.Tokenizer
java.io.IOException
public void reset() throws java.io.IOException
reset
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.