public class DutchAnalyzer extends Analyzer
Analyzer
for Dutch language.
Supports an external list of stopwords (words that will not be indexed at all), an external list of exclusions (word that will not be stemmed, but indexed) and an external list of word-stem pairs that overrule the algorithm (dictionary stemming). A default set of stopwords is used unless an alternative list is specified, but the exclusion list is empty by default.
NOTE: This class uses the same Version
dependent settings as StandardAnalyzer
.
Modifier and Type | Field and Description |
---|---|
static java.lang.String[] |
DUTCH_STOP_WORDS
List of typical Dutch stopwords.
|
overridesTokenStreamMethod
Constructor and Description |
---|
DutchAnalyzer()
Deprecated.
Use
DutchAnalyzer(Version) instead |
DutchAnalyzer(java.io.File stopwords)
Deprecated.
Use
DutchAnalyzer(Version, File) instead |
DutchAnalyzer(java.util.HashSet stopwords)
Deprecated.
Use
DutchAnalyzer(Version, HashSet) instead |
DutchAnalyzer(java.lang.String[] stopwords)
Deprecated.
Use
DutchAnalyzer(Version, String[]) instead |
DutchAnalyzer(Version matchVersion)
Builds an analyzer with the default stop words (
DUTCH_STOP_WORDS )
and a few default entries for the stem exclusion table. |
DutchAnalyzer(Version matchVersion,
java.io.File stopwords)
Builds an analyzer with the given stop words.
|
DutchAnalyzer(Version matchVersion,
java.util.HashSet stopwords)
Builds an analyzer with the given stop words.
|
DutchAnalyzer(Version matchVersion,
java.lang.String[] stopwords)
Builds an analyzer with the given stop words.
|
Modifier and Type | Method and Description |
---|---|
TokenStream |
reusableTokenStream(java.lang.String fieldName,
java.io.Reader reader)
Returns a (possibly reused)
TokenStream which tokenizes all the
text in the provided Reader . |
void |
setStemDictionary(java.io.File stemdictFile)
Reads a stemdictionary file , that overrules the stemming algorithm
This is a textfile that contains per line
word\tstem, i.e: two tab seperated words
|
void |
setStemExclusionTable(java.io.File exclusionlist)
Builds an exclusionlist from the words contained in the given file.
|
void |
setStemExclusionTable(java.util.HashSet exclusionlist)
Builds an exclusionlist from a Hashtable.
|
void |
setStemExclusionTable(java.lang.String[] exclusionlist)
Builds an exclusionlist from an array of Strings.
|
TokenStream |
tokenStream(java.lang.String fieldName,
java.io.Reader reader)
Creates a
TokenStream which tokenizes all the text in the
provided Reader . |
close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setOverridesTokenStreamMethod, setPreviousTokenStream
public static final java.lang.String[] DUTCH_STOP_WORDS
public DutchAnalyzer()
DutchAnalyzer(Version)
insteadDUTCH_STOP_WORDS
)
and a few default entries for the stem exclusion table.public DutchAnalyzer(Version matchVersion)
DUTCH_STOP_WORDS
)
and a few default entries for the stem exclusion table.public DutchAnalyzer(java.lang.String[] stopwords)
DutchAnalyzer(Version, String[])
insteadstopwords
- public DutchAnalyzer(Version matchVersion, java.lang.String[] stopwords)
matchVersion
- stopwords
- public DutchAnalyzer(java.util.HashSet stopwords)
DutchAnalyzer(Version, HashSet)
insteadstopwords
- public DutchAnalyzer(Version matchVersion, java.util.HashSet stopwords)
stopwords
- public DutchAnalyzer(java.io.File stopwords)
DutchAnalyzer(Version, File)
insteadstopwords
- public DutchAnalyzer(Version matchVersion, java.io.File stopwords)
stopwords
- public void setStemExclusionTable(java.lang.String[] exclusionlist)
exclusionlist
- public void setStemExclusionTable(java.util.HashSet exclusionlist)
public void setStemExclusionTable(java.io.File exclusionlist)
public void setStemDictionary(java.io.File stemdictFile)
public TokenStream tokenStream(java.lang.String fieldName, java.io.Reader reader)
TokenStream
which tokenizes all the text in the
provided Reader
.tokenStream
in class Analyzer
TokenStream
built from a StandardTokenizer
filtered with StandardFilter
, StopFilter
,
and DutchStemFilter
public TokenStream reusableTokenStream(java.lang.String fieldName, java.io.Reader reader) throws java.io.IOException
TokenStream
which tokenizes all the
text in the provided Reader
.reusableTokenStream
in class Analyzer
TokenStream
built from a StandardTokenizer
filtered with StandardFilter
, StopFilter
,
and DutchStemFilter
java.io.IOException
Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.