edu.illinois.cs.cogcomp.nlp.tokenizer

Class Tokenizer.Tokenization

java.lang.Object

edu.illinois.cs.cogcomp.nlp.tokenizer.Tokenizer.Tokenization

Enclosing interface:

Tokenizer

public static class Tokenizer.Tokenization
extends Object

Constructor Summary

Constructors

Constructor and Description
Tokenization(String[] tokens, IntPair[] characterOffsets, int[] sentenceEndTokenIndexes)

Method Summary

Modifier and Type	Method and Description
IntPair[]	getCharacterOffsets()
int[]	getSentenceEndTokenIndexes()
List<Pair<String[],IntPair[]>>	getTokenizedSentences() get a list of pairs, each pair corresponding to a sentence's tokens and their *absolute* character offsets.
String[]	getTokens()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenization

public Tokenization(String[] tokens,
                    IntPair[] characterOffsets,
                    int[] sentenceEndTokenIndexes)

Method Detail

getTokenizedSentences

public List<Pair<String[],IntPair[]>> getTokenizedSentences()

get a list of pairs, each pair corresponding to a sentence's tokens and their *absolute* character offsets.

getTokens

public String[] getTokens()

getSentenceEndTokenIndexes

public int[] getSentenceEndTokenIndexes()

getCharacterOffsets

public IntPair[] getCharacterOffsets()