opennlp.tools.tokenize

Class DictionaryDetokenizer

java.lang.Object

opennlp.tools.tokenize.DictionaryDetokenizer

All Implemented Interfaces:

Detokenizer

public class DictionaryDetokenizer
extends java.lang.Object
implements Detokenizer

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a DetokenizationDictionary object.

See Also:

Detokenizer, DetokenizationDictionary

Nested Class Summary

Nested classes/interfaces inherited from interface opennlp.tools.tokenize.Detokenizer

Detokenizer.DetokenizationOperation

Constructor Summary

Constructors

Constructor and Description
DictionaryDetokenizer(DetokenizationDictionary dict)

Method Summary

Modifier and Type	Method and Description
Detokenizer.DetokenizationOperation[]	detokenize(java.lang.String[] tokens) Detokenize the input tokens.
java.lang.String	detokenize(java.lang.String[] tokens, java.lang.String splitMarker) Detokenize the input tokens into a String.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DictionaryDetokenizer

public DictionaryDetokenizer(DetokenizationDictionary dict)

Method Detail

detokenize

public Detokenizer.DetokenizationOperation[] detokenize(java.lang.String[] tokens)

Description copied from interface: Detokenizer

Detokenize the input tokens.

Specified by:

detokenize in interface Detokenizer

Parameters:

tokens - the tokens to detokenize.

Returns:

the merge operations to detokenize the input tokens.

detokenize

public java.lang.String detokenize(java.lang.String[] tokens,
                                   java.lang.String splitMarker)

Description copied from interface: Detokenizer

Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.

Specified by:

detokenize in interface Detokenizer

Parameters:

tokens - the token which should be concatenated

splitMarker - the split marker or null

Returns:

the concatenated tokens