tokenization

There's noneedfor atokenizationphase,andlookaheadassertions canbeused foralimited degreeofcontext-sensitivity.沒有必要再進行符號化解析，或者用於有限度上下文敏感的前向斷言。

ShallowEnglishlanguageparsingcustomizedfornoveltydetectionincludessentence boundarydetection,tokenization, part-of-speechtagging,andmorphologicalanalysis.針對新信息檢測的英文淺層語言分析主要包括斷句、詞彙切分、詞性標註以及詞形還原等自然語言處理過程。

Tokenizationmattersa lotforrandomtextcollections; infairnesstoNLTK, itsbundledcorporahave beenpackagedforeasyandaccuratetokenizationwithWSTokenizer.斷詞方法對隨機文本集合來説至關重要；公平地講，NLTK捆綁的全集已經通過WSTokenizer打包為易用且準確的斷詞工具。 ^[2]