Module Import alias ImportFrom alias TryExcept body ImportFrom alias ImportFrom alias handlers ExceptHandler type NameLoad body Print Str Assign NameStore Call NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Dict Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore ListComp Call AttributeLoad NameLoad attr comprehension NameStore NameLoad For NameStore Call AttributeLoad NameLoad attr body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Index NameLoad Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad keyword AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str If NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpAdd Call AttributeLoad Str attr SubscriptLoad NameLoad Index Num Str Return ListComp Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword Str comprehension NameStore Call NameLoad SubscriptLoad NameLoad Index Num decorator_list
N/A N/A sys spacy.en English N/A N/A nltk wordpunct_tokenize nltk.corpus stopwords N/A N/A N/A ImportError N/A N/A [!] You need to install nltk (http://nltk.org/index.html) N/A nlp N/A English _calculate_languages_ratios N/A N/A text N/A N/A N/A 
    Calculate probability of given text to be written in several languages and
    return a dictionary that looks like {'french': 2, 'spanish': 4, 'english': 0}
    
    @param text: Text whose language want to be detected
    @type text: str
    
    @return: Dictionary with languages and unique stopwords seen in analyzed text
    @rtype: dict
     N/A languages_ratios N/A N/A 
    nltk.wordpunct_tokenize() splits all punctuations into separate tokens
    
    >>> wordpunct_tokenize("That's thirty minutes away. I'll be there in ten.")
    ['That', "'", 's', 'thirty', 'minutes', 'away', '.', 'I', "'", 'll', 'be', 'there', 'in', 'ten', '.']
     N/A tokens N/A wordpunct_tokenize text N/A words N/A N/A N/A word lower N/A word tokens N/A language N/A N/A stopwords fileids N/A N/A stopwords_set N/A set N/A N/A stopwords words language N/A words_set N/A set words N/A common_elements N/A N/A words_set intersection stopwords_set N/A N/A languages_ratios N/A language N/A len common_elements N/A languages_ratios N/A detect_language N/A N/A text N/A N/A N/A 
    Calculate probability of given text to be written in several languages and
    return the highest scored.
    
    It uses a stopwords based approach, counting how many unique stopwords
    are seen in analyzed text.
    
    @param text: Text whose language want to be detected
    @type text: str
    
    @return: Most scored language guessed
    @rtype: str
     N/A ratios N/A _calculate_languages_ratios text N/A most_rated_language N/A max ratios key N/A ratios get N/A most_rated_language N/A tokenize_document N/A N/A docpair verbose N/A False N/A N/A 
    Gets tokens from a text in English
     N/A verbose N/A N/A N/A N/A N/A sys stdout write N/A N/A N/A working on doc {} format N/A docpair N/A 0 N/A N/A N/A N/A N/A x lower_ encode ascii errors ignore N/A x N/A nlp N/A docpair N/A 1 N/A