Module Expr Str ImportFrom alias ImportFrom alias Import alias ImportFrom alias alias FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Num Num body Expr Str Assign NameStore Num For NameStore NameLoad body Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad Num Call AttributeLoad NameLoad attr BinOpSub Num BinOpDiv Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore BinOpDiv Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore BinOpMult BinOpMult NameLoad BinOpPow NameLoad NameLoad BinOpPow NameLoad NameLoad If CompareGt NameLoad NameLoad body Assign NameStore NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Num Num body Expr Str Assign NameStore Num For TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad body AugAssignAdd NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad Return BinOpDiv NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str For TupleStore NameStore NameStore Call NameLoad Call NameLoad NameLoad Call NameLoad NameLoad body If CompareEq NameLoad NameLoad body Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Assign NameStore ListLoad Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad Assign NameStore ListLoad For NameStore Call NameLoad Num BinOpAdd Call NameLoad NameLoad Num body For NameStore Call NameLoad NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad For NameStore Call NameLoad NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body If CompareNotIn NameLoad NameLoad body Continue orelse If CompareEqEq Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Num body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad orelse Assign NameStore Call NameLoad NameLoad BinOpAdd BinOpSub NameLoad NameLoad Num For NameStore Call NameLoad Num NameLoad body If CompareLt BinOpAdd NameLoad NameLoad NameLoad body Assign NameStore Call NameLoad Call NameLoad NameLoad NameLoad BinOpAdd BinOpAdd NameLoad NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareEqEq NameLoad NameLoad Num body Assign NameStore Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Break If CompareLtE NameLoad NameLoad body Assign NameStore Call NameLoad Call NameLoad NameLoad BinOpSub NameLoad NameLoad BinOpAdd NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareEqEq NameLoad NameLoad Num body Assign NameStore Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpSub BinOpAdd NameLoad Call NameLoad NameLoad Num Break Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Assign TupleStore NameStore NameStore TupleLoad NameLoad Call NameLoad NameLoad NameLoad Assign NameStore ListLoad NameLoad While CompareIsNot NameLoad NameLoad body Assign TupleStore NameStore NameStore TupleLoad NameLoad Call NameLoad NameLoad NameLoad If BoolOpAnd CompareIsNot NameLoad NameLoad CompareEq BinOpAdd NameLoad Num NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad orelse If CompareGt Call NameLoad NameLoad Num body Expr Yield Call NameLoad NameLoad Assign NameStore ListLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad GeneratorExp Call NameLoad Call NameLoad NameLoad Num comprehension NameStore NameLoad Assign NameStore Call NameLoad NameLoad Num Assign NameStore BinOpSub BinOpDiv BinOpMult Num NameLoad NameLoad Num If NameLoad body Return BinOpDiv BinOpAdd NameLoad Num Num orelse Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad GeneratorExp BinOpPow BinOpSub NameLoad NameLoad Num comprehension TupleStore NameStore NameStore Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore BinOpSub Num BinOpDiv NameLoad Call NameLoad BinOpAdd NameLoad Num Num If NameLoad body Return BinOpDiv BinOpAdd NameLoad Num Num orelse Return NameLoad decorator_list
N/A N/A  RIBES score implementation  __future__ division itertools islice N/A math nltk.util ngrams choose sentence_ribes N/A N/A references hypothesis alpha beta N/A 0.25 0.1 N/A N/A 
    The RIBES (Rank-based Intuitive Bilingual Evaluation Score) from 
    Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh and 
    Hajime Tsukada. 2010. "Automatic Evaluation of Translation Quality for 
    Distant Language Pairs". In Proceedings of EMNLP. 
    http://www.aclweb.org/anthology/D/D10/D10-1092.pdf 
    
    The generic RIBES scores used in shared task, e.g. Workshop for 
    Asian Translation (WAT) uses the following RIBES calculations:
    
        RIBES = kendall_tau * (alpha**p1) * (beta**bp)
    
    Please note that this re-implementation differs from the official
    RIBES implementation and though it emulates the results as describe
    in the original paper, there are further optimization implemented 
    in the official RIBES script.
    
    Users are encouraged to use the official RIBES script instead of this 
    implementation when evaluating your machine translation system. Refer
    to http://www.kecl.ntt.co.jp/icl/lirg/ribes/ for the official script.
    
    :param references: a list of reference sentences
    :type reference: list(list(str))
    :param hypothesis: a hypothesis sentence
    :type hypothesis: list(str)
    :param alpha: hyperparameter used as a prior for the unigram precision.
    :type alpha: float
    :param beta: hyperparameter used as a prior for the brevity penalty.
    :type beta: float
    :return: The best ribes score from one of the references.
    :rtype: float
     N/A best_ribes -1.0 N/A reference references N/A N/A worder N/A word_rank_alignment reference hypothesis N/A nkt N/A kendall_tau worder N/A bp N/A min 1.0 N/A N/A math exp N/A 1.0 N/A N/A len reference N/A len hypothesis N/A p1 N/A N/A len worder N/A len hypothesis N/A _ribes N/A N/A nkt N/A p1 alpha N/A bp beta N/A N/A _ribes best_ribes N/A N/A best_ribes _ribes N/A best_ribes N/A corpus_ribes N/A N/A list_of_references hypotheses alpha beta N/A 0.25 0.1 N/A N/A 
    This function "calculates RIBES for a system output (hypothesis) with 
    multiple references, and returns "best" score among multi-references and 
    individual scores. The scores are corpus-wise, i.e., averaged by the number 
    of sentences." (c.f. RIBES version 1.03.1 code).
    
    Different from BLEU's micro-average precision, RIBES calculates the 
    macro-average precision by averaging the best RIBES score for each pair of 
    hypothesis and its corresponding references 

    >>> hyp1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'which',
    ...         'ensures', 'that', 'the', 'military', 'always',
    ...         'obeys', 'the', 'commands', 'of', 'the', 'party']
    >>> ref1a = ['It', 'is', 'a', 'guide', 'to', 'action', 'that',
    ...          'ensures', 'that', 'the', 'military', 'will', 'forever',
    ...          'heed', 'Party', 'commands']
    >>> ref1b = ['It', 'is', 'the', 'guiding', 'principle', 'which',
    ...          'guarantees', 'the', 'military', 'forces', 'always',
    ...          'being', 'under', 'the', 'command', 'of', 'the', 'Party']
    >>> ref1c = ['It', 'is', 'the', 'practical', 'guide', 'for', 'the',
    ...          'army', 'always', 'to', 'heed', 'the', 'directions',
    ...          'of', 'the', 'party']
    
    >>> hyp2 = ['he', 'read', 'the', 'book', 'because', 'he', 'was', 
    ...         'interested', 'in', 'world', 'history']
    >>> ref2a = ['he', 'was', 'interested', 'in', 'world', 'history', 
    ...          'because', 'he', 'read', 'the', 'book']
    
    >>> list_of_references = [[ref1a, ref1b, ref1c], [ref2a]]
    >>> hypotheses = [hyp1, hyp2]
    >>> round(corpus_ribes(list_of_references, hypotheses),4)
    0.3597
    
    :param references: a corpus of lists of reference sentences, w.r.t. hypotheses
    :type references: list(list(list(str)))
    :param hypotheses: a list of hypothesis sentences
    :type hypotheses: list(list(str))
    :param alpha: hyperparameter used as a prior for the unigram precision.
    :type alpha: float
    :param beta: hyperparameter used as a prior for the brevity penalty.
    :type beta: float
    :return: The best ribes score from one of the references.
    :rtype: float
     N/A corpus_best_ribes 0.0 N/A N/A references hypothesis N/A zip list_of_references hypotheses N/A N/A corpus_best_ribes N/A sentence_ribes references hypothesis alpha beta N/A N/A corpus_best_ribes N/A len hypotheses N/A position_of_ngram N/A N/A ngram sentence N/A N/A N/A 
    This function returns the position of the first instance of the ngram 
    appearing in a sentence.
    
    Note that one could also use string as follows but the code is a little
    convoluted with type casting back and forth:
        
        char_pos = ' '.join(sent)[:' '.join(sent).index(' '.join(ngram))]
        word_pos = char_pos.count(' ')
        
    Another way to conceive this is:
    
        return next(i for i, ng in enumerate(ngrams(sentence, len(ngram))) 
                    if ng == ngram)
                    
    :param ngram: The ngram that needs to be searched
    :type ngram: tuple
    :param sentence: The list of tokens to search from.
    :type sentence: list(str)
     N/A N/A i sublist N/A enumerate N/A ngrams sentence N/A len ngram N/A N/A N/A ngram sublist N/A N/A i N/A word_rank_alignment N/A N/A reference hypothesis character_based N/A False N/A N/A     
    This is the word rank alignment algorithm described in the paper to produce
    the *worder* list, i.e. a list of word indices of the hypothesis word orders 
    w.r.t. the list of reference words.
    
    Below is (H0, R0) example from the Isozaki et al. 2010 paper, 
    note the examples are indexed from 1 but the results here are indexed from 0:
    
        >>> ref = str('he was interested in world history because he '
        ... 'read the book').split()
        >>> hyp = str('he read the book because he was interested in world '
        ... 'history').split()
        >>> word_rank_alignment(ref, hyp)
        [7, 8, 9, 10, 6, 0, 1, 2, 3, 4, 5]
        
    The (H1, R1) example from the paper, note the 0th index:
    
        >>> ref = 'John hit Bob yesterday'.split()
        >>> hyp = 'Bob hit John yesterday'.split()
        >>> word_rank_alignment(ref, hyp)
        [2, 1, 0, 3]

    Here is the (H2, R2) example from the paper, note the 0th index here too:
    
        >>> ref = 'the boy read the book'.split()
        >>> hyp = 'the book was read by the boy'.split()
        >>> word_rank_alignment(ref, hyp)
        [3, 4, 2, 0, 1]
        
    :param reference: a reference sentence
    :type reference: list(str)
    :param hypothesis: a hypothesis sentence
    :type hypothesis: list(str)
     N/A worder N/A N/A hyp_len N/A len hypothesis N/A ref_ngrams N/A N/A hyp_ngrams N/A N/A n N/A range 1 N/A N/A len reference 1 N/A N/A ng N/A ngrams reference n N/A N/A N/A N/A ref_ngrams append ng N/A ng N/A ngrams hypothesis n N/A N/A N/A N/A hyp_ngrams append ng N/A N/A i h_word N/A enumerate hypothesis N/A N/A N/A h_word reference N/A N/A N/A N/A N/A N/A N/A hypothesis count h_word N/A N/A reference count h_word 1 N/A N/A N/A N/A worder append N/A N/A reference index h_word N/A N/A max_window_size N/A max i N/A N/A hyp_len i 1 N/A window N/A range 1 max_window_size N/A N/A N/A N/A i window hyp_len N/A N/A right_context_ngram N/A tuple N/A islice hypothesis i N/A N/A i window 1 N/A num_times_in_ref N/A N/A ref_ngrams count right_context_ngram N/A num_times_in_hyp N/A N/A hyp_ngrams count right_context_ngram N/A N/A num_times_in_ref num_times_in_hyp 1 N/A N/A pos N/A position_of_ngram right_context_ngram reference N/A N/A N/A worder append pos N/A N/A N/A window i N/A N/A left_context_ngram N/A tuple N/A islice hypothesis N/A i window N/A i 1 N/A num_times_in_ref N/A N/A ref_ngrams count left_context_ngram N/A num_times_in_hyp N/A N/A hyp_ngrams count left_context_ngram N/A N/A num_times_in_ref num_times_in_hyp 1 N/A N/A pos N/A position_of_ngram left_context_ngram reference N/A N/A N/A worder append N/A N/A pos N/A len left_context_ngram 1 N/A N/A worder N/A find_increasing_sequences N/A N/A worder N/A N/A N/A 
    Given the *worder* list, this function groups monotonic +1 sequences. 
    
        >>> worder = [7, 8, 9, 10, 6, 0, 1, 2, 3, 4, 5]
        >>> list(find_increasing_sequences(worder))
        [(7, 8, 9, 10), (0, 1, 2, 3, 4, 5)]
    
    :param worder: The worder list output from word_rank_alignment
    :param type: list(int)
     N/A items N/A iter worder N/A N/A a b N/A None N/A next items None N/A result N/A b N/A N/A b None N/A N/A N/A a b N/A b N/A next items None N/A N/A N/A b None N/A N/A a 1 b N/A N/A N/A N/A result append b N/A N/A N/A N/A len result 1 N/A N/A N/A N/A tuple result N/A result N/A b N/A kendall_tau N/A N/A worder normalize N/A True N/A N/A 
    Calculates the Kendall's Tau correlation coefficient given the *worder*
    list of word alignments from word_rank_alignment(), using the formula:
    
        tau = 2 * num_increasing_pairs / num_possible pairs -1
    
    Note that the no. of increasing pairs can be discontinuous in the *worder*
    list and each each increasing sequence can be tabulated as choose(len(seq), 2) 
    no. of increasing pairs, e.g.
    
        >>> worder = [7, 8, 9, 10, 6, 0, 1, 2, 3, 4, 5]
        >>> number_possible_pairs = choose(len(worder), 2)
        >>> round(kendall_tau(worder, normalize=False),3)
        -0.236
        >>> round(kendall_tau(worder),3)
        0.382
    
    :param worder: The worder list output from word_rank_alignment
    :type worder: list(int)
    :param normalize: Flag to indicate normalization
    :type normalize: boolean
    :return: The Kendall's Tau correlation coefficient.
    :rtype: float
     N/A worder_len N/A len worder N/A increasing_sequences N/A find_increasing_sequences worder N/A num_increasing_pairs N/A sum N/A N/A choose N/A len seq 2 N/A seq increasing_sequences N/A num_possible_pairs N/A choose worder_len 2 N/A tau N/A N/A N/A 2 num_increasing_pairs num_possible_pairs 1 N/A normalize N/A N/A N/A N/A tau 1 2 N/A N/A tau N/A spearman_rho N/A N/A worder normalize N/A True N/A N/A 
    Calculates the Spearman's Rho correlation coefficient given the *worder* 
    list of word alignment from word_rank_alignment(), using the formula:
    
        rho = 1 - sum(d**2) / choose(len(worder)+1, 3)  
        
    Given that d is the sum of difference between the *worder* list of indices
    and the original word indices from the reference sentence.
    
    Using the (H0,R0) and (H5, R5) example from the paper
    
        >>> worder =  [7, 8, 9, 10, 6, 0, 1, 2, 3, 4, 5]
        >>> round(spearman_rho(worder, normalize=False), 3)
        -0.591
        >>> round(spearman_rho(worder), 3)
        0.205
    
    :param worder: The worder list output from word_rank_alignment
    :param type: list(int)
     N/A worder_len N/A len worder N/A sum_d_square N/A sum N/A N/A N/A wi i 2 N/A N/A wi i N/A zip worder N/A range worder_len N/A rho N/A 1 N/A sum_d_square N/A choose N/A worder_len 1 3 N/A normalize N/A N/A N/A N/A rho 1 2 N/A N/A rho N/A