Question: Why NLTK TweetTokenizer?

[freecraver]

Thanks for your work on this nice project.

I intend to create a library for text simplification, and potentially would like to integrate your package.
The selection of a tokenizer has an impact on the obtained readability scores and I was wondering how you approached this issue.

Was there any specific reason for choosing the Tweet-Tokenizer over e.g. the default/recommended Nltk-Tokenizer which better depicts the Penn Treebank's definition of word-boundaries?

py-readability-metrics/readability/text/analyzer.py

Line 128 in 3ffb97f

tokenizer = TweetTokenizer()

[cdimascio]

@freecraver I'm open to changing the tokenizer. Would you be interested in investigating the effort to switch over?

[freecraver]

Sure - please check #27 for my suggested changes.