opennlp.tools.tokenize

Class DefaultTokenContextGenerator

java.lang.Object

opennlp.tools.tokenize.DefaultTokenContextGenerator

All Implemented Interfaces:

TokenContextGenerator

public class DefaultTokenContextGenerator
extends java.lang.Object
implements TokenContextGenerator

Generate events for maxent decisions for tokenization.

Constructor Summary

Constructors

Constructor and Description
DefaultTokenContextGenerator() Creates a default context generator for tokenizer.
DefaultTokenContextGenerator(java.util.Set<java.lang.String> inducedAbbreviations) Creates a default context generator for tokenizer.

Method Summary

Modifier and Type	Method and Description
java.lang.String[]	getContext(java.lang.String sentence, int index) Returns an array of features for the specified sentence string at the specified index.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultTokenContextGenerator

public DefaultTokenContextGenerator()

Creates a default context generator for tokenizer.

DefaultTokenContextGenerator

public DefaultTokenContextGenerator(java.util.Set<java.lang.String> inducedAbbreviations)

Creates a default context generator for tokenizer.

Parameters:

inducedAbbreviations - the induced abbreviations

Method Detail

getContext

public java.lang.String[] getContext(java.lang.String sentence,
                                     int index)

Description copied from interface: TokenContextGenerator

Returns an array of features for the specified sentence string at the specified index.

Specified by:

getContext in interface TokenContextGenerator

Parameters:

sentence - The string for a sentence.

index - The index to consider splitting as a token.

Returns:

an array of features for the specified sentence string at the specified index.