Module ImportFrom alias Import alias Import alias FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore ListLoad Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad Assign NameStore ListLoad With Call NameLoad NameLoad Str NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Num For NameStore NameLoad body If CompareEq BinOpMod NameLoad Num Num body Print Str NameLoad AugAssignAdd NameStore Num Assign NameStore SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad SubscriptLoad NameLoad Index Str Assign NameStore BinOpAdd NameLoad ListLoad NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Print NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad For NameStore NameLoad body Assign NameStore SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index Num Assign NameStore ListLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad SubscriptLoad NameLoad Index Num attr If CompareEq NameLoad NameLoad body Assign NameStore Call NameLoad Call AttributeLoad Str attr NameLoad orelse Assign NameStore SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr For NameStore NameLoad body AugAssignAdd SubscriptStore NameLoad Index NameLoad Num Assign NameStore Dict Str Str Str NameLoad NameLoad Call NameLoad Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Return TupleLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore BinOpAdd NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore ListComp NameLoad comprehension NameStore NameLoad BoolOpAnd CompareGt Call NameLoad NameLoad Num CompareNotEq SubscriptLoad NameLoad Index Num Str Assign NameStore ListComp IfExp CompareNotEq SubscriptLoad NameLoad Index Num Str NameLoad SubscriptLoad NameLoad Slice Num comprehension NameStore NameLoad Assign NameStore Call AttributeLoad Str attr NameLoad Return Call AttributeLoad Call AttributeLoad NameLoad attr attr decorator_list
N/A collections defaultdict N/A csv N/A re build_data N/A N/A fname preprocess N/A True N/A N/A 
    Reads a CSV file with headers 'labels' and 'text' (containing label string and text respectively)
    and outputs sentences, vocab and labels
    :param fname: file name to read
    :param preprocess: should data be preprocessed
    :return: sentences is a list of dictionary (a format which NNTrainer accepts) [{'text': <text>, 'y':<label>},...]
     N/A sentences N/A N/A vocab N/A defaultdict float N/A labels N/A N/A rows N/A N/A N/A open fname rb f N/A N/A reader N/A N/A csv DictReader f N/A count 0 N/A line reader N/A N/A N/A N/A count 1000 0 N/A N/A Reading line no.  count N/A count 1 N/A label N/A line N/A labels N/A N/A N/A rows append N/A label N/A line N/A text N/A labels N/A labels N/A label N/A labels N/A list N/A set labels N/A N/A N/A labels sort N/A labels N/A labelIdToLabel N/A dict N/A zip labels N/A range N/A len labels N/A row rows N/A N/A y N/A labelIdToLabel N/A N/A row N/A 0 N/A rev N/A N/A N/A N/A rev append N/A N/A N/A row N/A 1 strip N/A N/A preprocess True N/A N/A orig_rev N/A clean_str N/A N/A join rev N/A N/A orig_rev N/A rev N/A 0 N/A words N/A set N/A N/A orig_rev split N/A word words N/A N/A N/A vocab N/A word 1 N/A datum N/A y text num_words y orig_rev N/A len N/A N/A orig_rev split N/A N/A N/A sentences append datum N/A N/A sentences vocab labels N/A clean_str N/A N/A str N/A N/A N/A 
    Tokenization/string cleaning. This is specific to tweets, but can be applied for other texts as well.
     N/A str N/A str N/A str N/A N/A re sub http[^ ]*[\\] \\ str N/A str N/A N/A re sub http[^ ]*  str N/A str N/A N/A str replace \n N/A arr N/A N/A re findall \w+(?:[-']\w+)*|'|[:)-.(]+|\S\w* str N/A arr N/A i N/A i arr N/A N/A N/A len i 1 N/A N/A i N/A 0 @ N/A arr N/A N/A N/A N/A i N/A 0 # i N/A i N/A 1 N/A i arr N/A res N/A N/A join arr N/A N/A N/A N/A N/A res lower strip N/A