| Pretrained Punkt Models -- Jan Strunk (New version trained after issues 313 and 514 had been corrected) |
|
|
| Most models were prepared using the test corpora from Kiss and Strunk (2006). Additional models have |
| been contributed by various people using NLTK for sentence boundary detection. |
|
|
| For information about how to use these models, please confer the tokenization HOWTO: |
| http://nltk.googlecode.com/svn/trunk/doc/howto/tokenize.html |
| and chapter 3.8 of the NLTK book: |
| http://nltk.googlecode.com/svn/trunk/doc/book/ch03.html#sec-segmentation |
|
|
| There are pretrained tokenizers for the following languages: |
|
|
| File Language Source Contents Size of training corpus(in tokens) Model contributed by |
| ======================================================================================================================================================================= |
| czech.pickle Czech Multilingual Corpus 1 (ECI) Lidove Noviny ~345,000 Jan Strunk / Tibor Kiss |
| Literarni Noviny |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| danish.pickle Danish Avisdata CD-Rom Ver. 1.1. 1995 Berlingske Tidende ~550,000 Jan Strunk / Tibor Kiss |
| (Berlingske Avisdata, Copenhagen) Weekend Avisen |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| dutch.pickle Dutch Multilingual Corpus 1 (ECI) De Limburger ~340,000 Jan Strunk / Tibor Kiss |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| english.pickle English Penn Treebank (LDC) Wall Street Journal ~469,000 Jan Strunk / Tibor Kiss |
| (American) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| estonian.pickle Estonian University of Tartu, Estonia Eesti Ekspress ~359,000 Jan Strunk / Tibor Kiss |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| finnish.pickle Finnish Finnish Parole Corpus, Finnish Books and major national ~364,000 Jan Strunk / Tibor Kiss |
| Text Bank (Suomen Kielen newspapers |
| Tekstipankki) |
| Finnish Center for IT Science |
| (CSC) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| french.pickle French Multilingual Corpus 1 (ECI) Le Monde ~370,000 Jan Strunk / Tibor Kiss |
| (European) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| german.pickle German Neue Zürcher Zeitung AG Neue Zürcher Zeitung ~847,000 Jan Strunk / Tibor Kiss |
| (Switzerland) CD-ROM |
| (Uses "ss" |
| instead of "ß") |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| greek.pickle Greek Efstathios Stamatatos To Vima (TO BHMA) ~227,000 Jan Strunk / Tibor Kiss |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| italian.pickle Italian Multilingual Corpus 1 (ECI) La Stampa, Il Mattino ~312,000 Jan Strunk / Tibor Kiss |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| norwegian.pickle Norwegian Centre for Humanities Bergens Tidende ~479,000 Jan Strunk / Tibor Kiss |
| (Bokmål and Information Technologies, |
| Nynorsk) Bergen |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| polish.pickle Polish Polish National Corpus Literature, newspapers, etc. ~1,000,000 Krzysztof Langner |
| (http://www.nkjp.pl/) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| portuguese.pickle Portuguese CETENFolha Corpus Folha de São Paulo ~321,000 Jan Strunk / Tibor Kiss |
| (Brazilian) (Linguateca) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| slovene.pickle Slovene TRACTOR Delo ~354,000 Jan Strunk / Tibor Kiss |
| Slovene Academy for Arts |
| and Sciences |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| spanish.pickle Spanish Multilingual Corpus 1 (ECI) Sur ~353,000 Jan Strunk / Tibor Kiss |
| (European) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swedish.pickle Swedish Multilingual Corpus 1 (ECI) Dagens Nyheter ~339,000 Jan Strunk / Tibor Kiss |
| (and some other texts) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| turkish.pickle Turkish METU Turkish Corpus Milliyet ~333,000 Jan Strunk / Tibor Kiss |
| (Türkçe Derlem Projesi) |
| University of Ankara |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| The corpora contained about 400,000 tokens on average and mostly consisted of newspaper text converted to |
| Unicode using the codecs module. |
|
|
| Kiss, Tibor and Strunk, Jan (2006): Unsupervised Multilingual Sentence Boundary Detection. |
| Computational Linguistics 32: 485-525. |
|
|
| ---- Training Code ---- |
|
|
| # import punkt |
| import nltk.tokenize.punkt |
|
|
| # Make a new Tokenizer |
| tokenizer = nltk.tokenize.punkt.PunktSentenceTokenizer() |
|
|
| # Read in training corpus (one example: Slovene) |
| import codecs |
| text = codecs.open("slovene.plain","Ur","iso-8859-2").read() |
|
|
| # Train tokenizer |
| tokenizer.train(text) |
|
|
| # Dump pickled tokenizer |
| import pickle |
| out = open("slovene.pickle","wb") |
| pickle.dump(tokenizer, out) |
| out.close() |
|
|
| --------- |
| |
