Module Import alias Import alias Import alias Import alias Import alias identifier ImportFrom alias ImportFrom alias Import alias Import alias Import alias FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults Num body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad keyword SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call NameLoad NameLoad body If CompareEq NameLoad BinOpSub NameLoad Num body Break For NameStore Call NameLoad BinOpAdd NameLoad Num NameLoad body Assign NameStore Call NameLoad BinOpAdd Call AttributeLoad NameLoad attr TupleLoad NameLoad SubscriptLoad NameLoad Index NameLoad Num Call AttributeLoad NameLoad attr TupleLoad SubscriptLoad NameLoad Index NameLoad NameLoad Num TryExcept body Assign NameStore BinOpDiv NameLoad BinOpSub BinOpAdd SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index NameLoad NameLoad handlers ExceptHandler body Assign NameStore Num If CompareGtE NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad SubscriptLoad NameLoad Index NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad While NameLoad body If CompareEq Call NameLoad NameLoad Num body Break Assign NameStore SubscriptLoad NameLoad Index Num Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Return Call NameLoad SubscriptLoad NameLoad Slice NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return BinOpMult BinOpDiv Call NameLoad Call NameLoad NameLoad Call NameLoad NameLoad BinOpDiv Num BinOpAdd Num Call AttributeLoad NameLoad attr UnaryOpUSub BinOpDiv Call NameLoad Call NameLoad NameLoad Num decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Assign NameStore Call NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore NameLoad Assign NameStore Call NameLoad NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad For NameStore NameLoad body Assign SubscriptStore NameLoad Index NameLoad BinOpDiv Call NameLoad NameLoad NameLoad AugAssignSub NameStore Num If CompareIs NameLoad NameLoad body Return NameLoad Assign NameStore NameLoad Assign NameStore Call NameLoad NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad For NameStore NameLoad body If CompareIn NameLoad NameLoad body AugAssignMult SubscriptStore NameLoad Index NameLoad BinOpDiv Call NameLoad NameLoad NameLoad AugAssignSub NameStore Num Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad body Assign TupleStore NameStore NameStore TupleLoad Call NameLoad Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad If NameLoad body Assign NameStore Call NameLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad ListLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad AttributeLoad NameLoad attr Num Assign NameStore Call NameLoad NameLoad For NameStore SubscriptLoad NameLoad Index NameLoad body Assign SubscriptStore NameLoad Index NameLoad BinOpAdd Call AttributeLoad NameLoad attr NameLoad Num SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index NameLoad For NameStore SubscriptLoad NameLoad Index NameLoad body Assign SubscriptStore NameLoad Index NameLoad BinOpAdd Call AttributeLoad NameLoad attr NameLoad Num SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index NameLoad For NameStore NameLoad body If CompareNotIn NameLoad NameLoad body Assign SubscriptStore NameLoad Index NameLoad ListLoad Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr SubscriptLoad NameLoad Index NameLoad If NameLoad body If CompareNotIn NameLoad NameLoad body Assign SubscriptStore NameLoad Index NameLoad ListLoad Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr BinOpDiv Call NameLoad SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr Index NameLoad SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr For NameStore NameLoad body Assign SubscriptStore NameLoad Index NameLoad Call NameLoad SubscriptLoad NameLoad Index NameLoad If NameLoad body For NameStore NameLoad body Assign SubscriptStore NameLoad Index NameLoad Call NameLoad SubscriptLoad NameLoad Index NameLoad Assign NameStore Call NameLoad NameLoad NameLoad orelse Assign NameStore Call NameLoad NameLoad Return TupleLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr If CompareEq Call NameLoad NameLoad Num body If CompareIn TupleLoad SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index Num NameLoad body Assign NameStore SubscriptLoad NameLoad Index TupleLoad SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index Num Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd SubscriptLoad NameLoad Index Num Str SubscriptLoad NameLoad Index Num orelse Expr Call NameLoad BinOpMod Str NameLoad orelse If CompareIn NameLoad NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr NameLoad orelse Expr Call NameLoad BinOpMod Str NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad For NameStore NameLoad body Assign NameStore Num Assign NameStore NameLoad For NameStore SubscriptLoad NameLoad Index NameLoad body If CompareLt NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index NameLoad body Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index NameLoad Assign NameStore NameLoad If CompareIs NameLoad NameLoad body Print Str NameLoad Print NameLoad Expr Call NameLoad Num Assign SubscriptStore NameLoad Index NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Assign NameStore Call NameLoad Assign NameStore Call NameLoad NameLoad Str Assign NameStore Call NameLoad BinOpAdd SubscriptLoad NameLoad Slice Num BinOpSub BinOpSub Call NameLoad NameLoad Call NameLoad SubscriptLoad Call AttributeLoad NameLoad attr Str Index BinOpSub Call NameLoad Call AttributeLoad NameLoad attr Str Num Num BinOpMod Str NameLoad Str Print Str For TupleStore NameStore NameStore Call NameLoad NameLoad body Print NameLoad SubscriptLoad NameLoad Index Num Print NameLoad BinOpMod Str TupleLoad SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index NameLoad Print Str decorator_list If CompareEq NameLoad Str body Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str keyword Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign TupleStore NameStore NameStore TupleLoad Call NameLoad Call NameLoad Assign NameStore Call NameLoad Assign NameStore Dict Str Str Str AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr TryExcept body Assign NameStore SubscriptLoad NameLoad Index AttributeLoad NameLoad attr handlers ExceptHandler body Expr Call NameLoad Str Print Str Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str For NameStore NameLoad body If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Continue Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str Print NameLoad For NameStore NameLoad body Print NameLoad Assign NameStore Call NameLoad NameLoad Str Expr Call AttributeLoad NameLoad attr NameLoad Assign TupleStore SubscriptStore NameLoad Index NameLoad SubscriptStore NameLoad Index NameLoad SubscriptStore NameLoad Index NameLoad SubscriptStore NameLoad Index NameLoad SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Print Str Assign TupleStore NameStore NameStore NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Assign NameStore Call NameLoad NameLoad NameLoad NameLoad AttributeLoad NameLoad attr orelse Assign NameStore Call NameLoad Assign NameStore SubscriptLoad ListComp NameLoad comprehension NameStore Call NameLoad NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad BoolOpOr UnaryOpNot AttributeLoad NameLoad attr BoolOpAnd AttributeLoad NameLoad attr CompareIn NameLoad NameLoad Slice AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore ListComp TupleLoad SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad comprehension TupleStore NameStore NameStore Call NameLoad NameLoad Expr Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad NameLoad AttributeLoad NameLoad attr
N/A N/A nltk N/A math N/A os N/A glob N/A networkx nx optparse OptionParser nltk.stem.porter PorterStemmer N/A read N/A config N/A lexicon extract_max_weight_indep_terms_greedy N/A N/A terms_weights term_occ freq size min_occ N/A 0.7 N/A N/A G N/A N/A nx Graph N/A l N/A len terms_weights N/A term_weights N/A dict terms_weights N/A t term_weights N/A N/A N/A N/A G add_node t score N/A term_weights N/A t N/A terms N/A N/A term_weights keys N/A N/A i t N/A enumerate terms N/A N/A N/A i N/A l 1 N/A N/A N/A j N/A range N/A i 1 l N/A N/A occ N/A float N/A N/A N/A term_occ get N/A t N/A terms N/A j 0 N/A N/A term_occ get N/A N/A terms N/A j t 0 N/A N/A N/A edge_weight N/A occ N/A N/A N/A freq N/A t N/A freq N/A N/A terms N/A j occ N/A N/A N/A N/A edge_weight 0 N/A N/A edge_weight min_occ N/A N/A N/A N/A G add_edge t N/A terms N/A j N/A indep N/A list N/A sorted_terms N/A sorted term_weights key N/A term_weights get reverse True N/A True N/A N/A N/A N/A len sorted_terms 0 N/A N/A N/A node N/A sorted_terms N/A 0 N/A N/A N/A indep append node N/A ng N/A N/A G neighbors node N/A N/A N/A sorted_terms remove node N/A n ng N/A N/A N/A N/A sorted_terms remove n N/A N/A N/A G remove_node n N/A N/A N/A G remove_node node N/A N/A set N/A indep N/A size N/A get_aggregated_score N/A N/A args N/A N/A N/A N/A N/A N/A float N/A sum args N/A len args N/A 1.0 N/A 1 N/A N/A math exp N/A N/A N/A float N/A len args 2 N/A discriminative_coverage N/A N/A term_weights hit_ratio N/A None N/A N/A weights N/A dict N/A i N/A len term_weights N/A l i N/A sorted_terms N/A sorted term_weights key N/A term_weights get reverse True N/A t sorted_terms N/A N/A N/A weights N/A t N/A N/A float i l N/A i 1 N/A N/A hit_ratio None N/A N/A weights N/A i l N/A sorted_terms N/A sorted hit_ratio key N/A hit_ratio get reverse True N/A t sorted_terms N/A N/A N/A t term_weights N/A N/A N/A weights N/A t N/A N/A float i l N/A i 1 N/A weights N/A get_raw_lexicon N/A N/A collections tweets_terms word_set tweets_cls word_occ fd mean_function discriminative_function use_hit_ratio N/A False N/A N/A N/A occs fd_all N/A N/A dict N/A N/A nltk FreqDist N/A term_weights N/A dict N/A use_hit_ratio N/A N/A hit_ratios N/A dict N/A d collections N/A N/A lex N/A N/A lexicon Lexicon N/A tweets_terms N/A d N/A word_set N/A d N/A tweets_cls N/A d N/A N/A config positive_class_label N/A config negative_class_label N/A fd N/A d N/A config positive_class_label 20 N/A w N/A discriminative_function lex N/A occ N/A word_occ N/A d N/A N/A N/A occs N/A occ N/A N/A N/A occs get occ 0 N/A N/A word_occ N/A d N/A occ N/A fr N/A fd N/A d N/A N/A N/A fd_all N/A fr N/A N/A N/A fd_all get fr 0 N/A N/A fd N/A d N/A fr N/A term w N/A N/A N/A term term_weights N/A N/A N/A term_weights N/A term N/A N/A N/A N/A N/A term_weights N/A term append N/A w N/A term N/A use_hit_ratio N/A N/A N/A term hit_ratios N/A N/A N/A hit_ratios N/A term N/A N/A N/A N/A N/A hit_ratios N/A term append N/A N/A float N/A N/A N/A lex terms_frequency_per_class N/A N/A lex main_class N/A term N/A N/A lex class_occ N/A N/A lex main_class N/A term term_weights N/A N/A N/A term_weights N/A term N/A mean_function N/A term_weights N/A term N/A use_hit_ratio N/A N/A term term_weights N/A N/A N/A hit_ratios N/A term N/A mean_function N/A hit_ratios N/A term N/A term_weights N/A discriminative_coverage term_weights hit_ratios N/A N/A term_weights N/A discriminative_coverage term_weights N/A N/A term_weights occs fd_all N/A reverse_stemmed_terms_set N/A N/A stemmed_terms reverse_stemming reverse_bigrams_stemming N/A N/A N/A terms N/A N/A w stemmed_terms N/A N/A ws N/A N/A w split N/A N/A N/A len ws 2 N/A N/A N/A N/A N/A ws N/A 0 N/A ws N/A 1 reverse_bigrams_stemming N/A N/A rev N/A reverse_bigrams_stemming N/A N/A N/A ws N/A 0 N/A ws N/A 1 N/A N/A N/A terms append N/A N/A N/A rev N/A 0 N/A rev N/A 1 N/A N/A N/A exit N/A I can't reverse the bi-gram: The "%s" bi-gram was not stemmed w N/A N/A N/A w reverse_stemming N/A N/A rev N/A reverse_stemming N/A w N/A N/A N/A terms append rev N/A N/A N/A exit N/A I can't reverse the uni-gram: The "%s" uni-gram was not stemmed w N/A terms N/A reverse_stemming N/A N/A stem_map N/A N/A N/A rev_stemming N/A dict N/A s stem_map N/A N/A max 0 N/A aux None N/A w N/A stem_map N/A s N/A N/A N/A max N/A N/A stem_map N/A s N/A w N/A N/A max N/A N/A stem_map N/A s N/A w N/A aux w N/A N/A aux None N/A N/A Generate map: The word was not stemmed s N/A stem_map N/A N/A exit 0 N/A N/A rev_stemming N/A s aux N/A rev_stemming N/A reverse_stemming_bigrams N/A N/A stem_bigrams_map N/A N/A N/A N/A reverse_stemming stem_bigrams_map N/A save_lexicon N/A N/A output scored_terms term_freq stem score N/A N/A N/A ps N/A PorterStemmer N/A f1 N/A open output w N/A f2 N/A open N/A N/A output N/A 0 N/A N/A N/A len output N/A len N/A N/A N/A output split . N/A N/A N/A len N/A N/A output split . 1 1 N/A _with_scores_%s.txt score w N/A Saving the lexicon to file... N/A N/A i t N/A enumerate scored_terms N/A N/A f1 N/A t N/A 0 N/A f2 N/A %s,%s,%s N/A N/A t N/A 0 N/A t N/A 1 N/A term_freq N/A N/A stem N/A i N/A The Lexicon is ready! N/A N/A N/A __name__ __main__ N/A N/A parser N/A OptionParser N/A N/A N/A parser add_option -s --terms_scoring dest test help The statistical test used to score terms: pmi, chi2 or frequency (in the relevant documents) default pmi N/A N/A N/A parser add_option -r --hit_ratio dest hit_ratio help It normalizes the crisis score according to the number of relevant tweets the term matches on. action store_true default False N/A N/A N/A parser add_option -t --top_div dest optimization help It filters out the terms with lower crisis scores that frequently co-occur with with terms with higher scores action store_true default False N/A N/A N/A parser add_option -o --output dest output help Write output to FILE. The script will write one term per line metavar FILE default your_lexicon.csv N/A N/A N/A parser add_option -i --input dest input help Read from FILE_PATH. The FILE_PATH is expected to point to a director containing one sub-director per collection. It assumes that each collection file contains a header and on each line has: tweet id, tweet text, tweet class (which needs to be among the ones you set in the config.py file) metavar FILE default N/A N/A options args N/A N/A parser parse_args N/A tweets_cls N/A dict N/A tweets_terms N/A dict N/A wd_occ N/A dict N/A word_set N/A dict N/A fd N/A dict N/A N/A stem_map bigrams_map N/A N/A dict N/A dict N/A collections N/A set N/A scoring_options N/A pmi chi2 frequency N/A N/A lexicon Lexicon pmi_polarity_metric N/A N/A lexicon Lexicon chi2_metric N/A N/A lexicon Lexicon frequency_metric N/A N/A N/A discriminative_function N/A scoring_options N/A N/A options test N/A N/A N/A N/A N/A exit The terms scoring parameter accepts only the following options: pmi, chi2, frequency N/A Extracting the data... N/A labeled_collections N/A N/A glob glob N/A N/A options input /* N/A l labeled_collections N/A N/A N/A N/A N/A N/A os path isdir l N/A N/A N/A collection_path N/A N/A glob glob N/A l /* N/A collection_path N/A c collection_path N/A N/A c N/A labeled_data N/A open c r N/A N/A N/A collections add c N/A N/A N/A tweets_cls N/A c N/A tweets_terms N/A c N/A wd_occ N/A c N/A word_set N/A c N/A fd N/A c N/A N/A read get_terms labeled_data stem_map bigrams_map N/A Done with reading... N/A N/A term_weights occs fd_all N/A get_raw_lexicon collections tweets_terms word_set tweets_cls wd_occ fd get_aggregated_score discriminative_function N/A options hit_ratio N/A N/A options optimization N/A N/A optimized_terms N/A extract_max_weight_indep_terms_greedy term_weights occs fd_all N/A config lexicon_size N/A N/A optimized_terms N/A set N/A top_stemmed N/A N/A t N/A t N/A sorted term_weights key N/A term_weights get reverse True N/A N/A N/A options optimization N/A N/A options optimization N/A t optimized_terms N/A N/A config lexicon_size N/A top N/A reverse_stemmed_terms_set top_stemmed N/A reverse_stemming stem_map N/A reverse_stemming_bigrams bigrams_map N/A sorted_terms_weights N/A N/A N/A top N/A i N/A term_weights N/A t N/A N/A i t N/A enumerate top_stemmed N/A N/A save_lexicon N/A options output sorted_terms_weights fd_all top_stemmed N/A options test