Module Import alias ImportFrom alias ImportFrom alias ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body With Call AttributeLoad NameLoad attr NameLoad keyword Str NameStore body Assign NameStore ListComp Call AttributeLoad NameLoad attr comprehension NameStore NameLoad With Call AttributeLoad NameLoad attr NameLoad keyword Str NameStore body Assign NameStore ListComp Call AttributeLoad NameLoad attr comprehension NameStore NameLoad With Call AttributeLoad NameLoad attr NameLoad keyword Str NameStore body Assign NameStore ListComp Call AttributeLoad NameLoad attr comprehension NameStore NameLoad With Call AttributeLoad NameLoad attr NameLoad keyword Str NameStore body Assign NameStore ListComp Call NameLoad Call AttributeLoad NameLoad attr comprehension NameStore NameLoad Assert CompareEqEqEq Call NameLoad NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Return Dict Str Str Str Str NameLoad NameLoad NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults NameLoad body Return AttributeLoad NameLoad attr decorator_list decorator_list
N/A N/A codecs nltk wordpunct_tokenize marmot.representations.representation_generator RepresentationGenerator WordQEAndPseudoRefRepresentationGenerator N/A RepresentationGenerator N/A N/A 
    Generate the standard word-level format: 3 files, source, target, tags, one line per file, whitespace tokenized
    Also add the un-tokenized pseudo-references to the dataset
     __init__ N/A N/A self source_file target_file tags_file pseudo_ref_file N/A N/A N/A N/A self data N/A N/A self parse_files source_file target_file tags_file pseudo_ref_file N/A parse_files N/A N/A source_file target_file tags_file pseudo_ref_file N/A N/A N/A N/A N/A codecs open source_file encoding utf8 source N/A N/A source_lines N/A N/A N/A line split N/A line source N/A N/A N/A codecs open target_file encoding utf8 target N/A N/A target_lines N/A N/A N/A line split N/A line target N/A N/A N/A codecs open tags_file encoding utf8 tags N/A N/A tags_lines N/A N/A N/A line split N/A line tags N/A N/A N/A codecs open pseudo_ref_file encoding utf8 pseudo_ref N/A N/A pseudo_ref_lines N/A N/A wordpunct_tokenize N/A N/A line strip N/A line pseudo_ref N/A N/A N/A len source_lines N/A len target_lines N/A len tags_lines N/A len pseudo_ref_lines N/A N/A target source tags pseudo_ref target_lines source_lines tags_lines pseudo_ref_lines N/A staticmethod generate N/A N/A self data_obj N/A None N/A N/A N/A self data N/A N/A