distinct

README.md

@@ -53,7 +53,7 @@ Downloading builder script: 100%|███████████████
 - **mode** *(string): 'Expectation-Adjusted-Distinct' or 'Distinct' for diversity calculation. If 'Expectation-Adjusted-Distinct', the scores for both modes will be returned. The default value is 'Expectation-Adjusted-Distinct'*
 - **vocab_size** *(int): For calculating 'Expectation-Adjusted-Distinct', either vocab_size or  dataForVocabCal should not be None. Default value is None*
 - **dataForVocabCal** *(list of string): dataForVocabCal for calculating the vocab_size for 'Expectation-Adjusted-Distinct'. Typically, it should be a list of sentences consisting the task dataset. For calculating 'Expectation-Adjusted-Distinct', either vocab_size or dataForVocabCal should not be None. Default value is None*
-- **tokenizer** *(string or tokenizer class): tokenizer for splitting sentences into words. Default value is Tokenizer13a(). NLTK tokenizer is available.*
+- **tokenizer** *(string or tokenizer class): tokenizer for splitting sentences into words. Default value is Tokenizer13a(). Note Tokenizer13a doesn't exclude punctuation marks. NLTK tokenizer is available.*
 
 ### Output Values