opennlp.tools.util.featuregen

Class TokenPatternFeatureGenerator

java.lang.Object

opennlp.tools.util.featuregen.TokenPatternFeatureGenerator

All Implemented Interfaces:

AdaptiveFeatureGenerator

public class TokenPatternFeatureGenerator
extends java.lang.Object
implements AdaptiveFeatureGenerator

Partitions tokens into sub-tokens based on character classes and generates class features for each of the sub-tokens and combinations of those sub-tokens.

Constructor Summary

Constructors

Constructor and Description
TokenPatternFeatureGenerator() Initializes a new instance.
TokenPatternFeatureGenerator(Tokenizer supportTokenizer) Initializes a new instance.

Method Summary

Modifier and Type	Method and Description
void	createFeatures(java.util.List<java.lang.String> feats, java.lang.String[] toks, int index, java.lang.String[] preds) Adds the appropriate features for the token at the specified index with the specified array of previous outcomes to the specified list of features.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface opennlp.tools.util.featuregen.AdaptiveFeatureGenerator

clearAdaptiveData, updateAdaptiveData

Constructor Detail

TokenPatternFeatureGenerator

public TokenPatternFeatureGenerator()

Initializes a new instance. For tokinization the SimpleTokenizer is used.

TokenPatternFeatureGenerator

public TokenPatternFeatureGenerator(Tokenizer supportTokenizer)

Initializes a new instance.

Parameters:

supportTokenizer -

Method Detail

createFeatures

public void createFeatures(java.util.List<java.lang.String> feats,
                           java.lang.String[] toks,
                           int index,
                           java.lang.String[] preds)

Description copied from interface: AdaptiveFeatureGenerator

Adds the appropriate features for the token at the specified index with the specified array of previous outcomes to the specified list of features.

Specified by:

createFeatures in interface AdaptiveFeatureGenerator

Parameters:

feats - The list of features to be added to.

toks - The tokens of the sentence or other text unit being processed.

index - The index of the token which is currently being processed.

preds - The outcomes for the tokens prior to the specified index.