Module Expr Str Import alias ImportFrom alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias Import alias ImportFrom alias ImportFrom alias Import alias Import alias Import alias Import alias Import alias identifier ImportFrom alias Import alias Import alias ClassDef bases Call NameLoad Str Str body FunctionDef arguments args NameParam defaults body Return BinOpAdd AttributeLoad NameLoad attr Call AttributeLoad Str attr ListComp BinOpAdd Str NameLoad comprehension NameStore AttributeLoad NameLoad attr decorator_list decorator_list Assign NameStore Call NameLoad Str Str Assign NameStore Call NameLoad Str Str Assign NameStore Call NameLoad Str Str ClassDef bases Call NameLoad Str Str body FunctionDef arguments args NameParam defaults body Return BinOpMod Str TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad Str attr ListComp BinOpAdd BinOpAdd NameLoad Str NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list FunctionDef arguments args NameParam defaults body Expr Str With Call NameLoad NameLoad NameStore body Return ListComp Call AttributeLoad NameLoad attr Str comprehension NameStore NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str With Call NameLoad NameLoad NameStore body Return ListComp Call AttributeLoad Call AttributeLoad NameLoad attr Str attr Str comprehension NameStore NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str With Call NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr GeneratorExp BinOpAdd NameLoad Str comprehension NameStore NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Return Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str TryExcept body With Call NameLoad BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad NameStore body Return Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Return NameLoad decorator_list Assign NameStore Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam defaults Num body Expr Str If UnaryOpNot NameLoad body Return NameLoad Assign NameStore SubscriptLoad NameLoad Index Num Assign NameStore Str Assign NameStore Dict Str NameLoad Assign NameStore Num While CompareLt NameLoad NameLoad body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad If CompareNotEq AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr body Return Call NameLoad SubscriptLoad NameLoad Slice Num NameLoad Assign NameStore Call AttributeLoad NameLoad attr If CompareNotIn Str SubscriptLoad NameLoad Index Str body Return Call NameLoad SubscriptLoad NameLoad Slice Num NameLoad Assign NameStore SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Num Return NameLoad handlers ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr body Pass Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Num NameLoad AugAssignAdd NameStore Num Raise Call NameLoad BinOpMod Str TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Num Num body Expr Str TryExcept body With Call NameLoad BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad NameStore body Return Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Pass Assign NameStore Num While CompareLt NameLoad NameLoad body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad If CompareEq AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr With Call NameLoad BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad orelse If BoolOpAnd CompareIn AttributeLoad NameLoad attr ListLoad Num Num Num Num CompareEq NameLoad BinOpSub NameLoad Num body Return NameLoad handlers ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr body Pass ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr body Return NameLoad ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr body Return NameLoad ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr body Pass ExceptHandler type AttributeLoad NameLoad attr body Pass Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Num NameLoad AugAssignAdd NameStore Num Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr keyword SubscriptLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Index Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Dict Str Str ListLoad Str Str ListLoad Str Str Assign NameStore Str Assign NameStore Str Assign NameStore Dict Str Str ListLoad BinOpMod Str NameLoad BinOpMod Str NameLoad BinOpMod Str NameLoad ListLoad BinOpMod Str NameLoad Assign NameStore Str Assign NameStore Dict Str Str ListLoad BinOpMod Str NameLoad BinOpMod Str NameLoad BinOpMod Str NameLoad ListLoad Str FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad Assign NameStore ListComp Call AttributeLoad Call AttributeLoad NameLoad attr attr Str comprehension NameStore NameLoad If BoolOpAnd CompareEq NameLoad Str CompareGtE Call NameLoad NameLoad Num CompareIn Str SubscriptLoad NameLoad Index Num body Expr Call AttributeLoad NameLoad attr Num Assign NameStore Call NameLoad AttributeLoad NameLoad attr NameLoad Assign NameStore ListComp NameLoad comprehension NameStore NameLoad BoolOpAnd NameLoad UnaryOpNot Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list For NameStore SubscriptLoad NameLoad Index NameLoad body For NameStore Call AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call NameLoad SubscriptLoad NameLoad Index NameLoad Assign NameStore Call NameLoad SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad Str attr NameLoad Return Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr With Call NameLoad BinOpMod Str TupleLoad NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad With Call NameLoad NameLoad NameStore body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr attr For NameStore Call AttributeLoad NameLoad attr Str body If UnaryOpNot NameLoad body Continue Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr TupleLoad Call NameLoad NameLoad BinOpAdd Call NameLoad NameLoad Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults SubscriptLoad NameLoad Index Num Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Return NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad For TupleStore NameStore NameStore NameLoad body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Slice NameLoad BinOpAdd NameLoad Num Return Call NameLoad AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad With Call NameLoad NameLoad NameStore body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr attr For NameStore Call AttributeLoad NameLoad attr Str body If UnaryOpNot NameLoad body Continue Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr TupleLoad Call NameLoad NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults SubscriptLoad NameLoad Index Num Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Return NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Num Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad Assign NameStore Dict Assign NameStore Num While CompareLt NameLoad Call NameLoad AttributeLoad NameLoad attr body If BoolOpAnd CompareLt NameLoad NameLoad CompareEq SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index Num NameLoad body Assign TupleStore NameStore NameStore NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad Call AttributeLoad Str attr SubscriptLoad AttributeLoad NameLoad attr Slice NameLoad BinOpAdd NameLoad Num attr Str Str AugAssignAdd NameStore Num Assign NameStore BinOpAdd NameLoad Num orelse Expr Call AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index NameLoad AugAssignAdd NameStore Num Assign NameStore Call AttributeLoad Call AttributeLoad Str attr NameLoad attr Str Assign NameStore SubscriptLoad NameLoad Index Num Assign NameStore SubscriptLoad NameLoad Slice Num Return Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad NameLoad decorator_list Assign NameStore Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListLoad If CompareGt BinOpAdd Call AttributeLoad AttributeLoad NameLoad attr attr Str Num NameLoad body Return NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr For NameStore AttributeLoad NameLoad attr body For NameStore Call AttributeLoad NameLoad attr NameLoad body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Slice NameLoad NameLoad If CompareNotIn NameLoad NameLoad body Continue Assign NameStore Call AttributeLoad BinOpMod Str TupleLoad SubscriptLoad NameLoad Slice Num NameLoad SubscriptLoad NameLoad Slice NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad With Call NameLoad BinOpMod Str TupleLoad NameLoad NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults Num AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num decorator_list FunctionDef arguments args NameParam defaults body AugAssignAdd AttributeStore NameLoad attr Num Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr If CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Print Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpMult Str AttributeLoad NameLoad attr Assign NameStore BinOpDiv BinOpMult Num AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr NameLoad NameLoad AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list Assign NameStore ListLoad Str Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str For NameStore NameLoad body Print BinOpMod Str NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad Assign NameStore ListLoad For NameStore NameLoad body If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign TupleStore NameStore NameStore NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Return TupleLoad NameLoad Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore NameLoad body Print BinOpMod Str NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore BinOpMod Str NameLoad If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Print Str Assign NameStore Call NameLoad Call AttributeLoad Call NameLoad NameLoad attr Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad Assign NameStore ListLoad TryExcept body For TupleStore NameStore NameStore NameLoad body If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Print Str Expr Call AttributeLoad NameLoad attr BinOpSub Call NameLoad NameLoad Call NameLoad NameLoad Expr Call NameLoad BinOpMod Str NameLoad NameLoad If NameLoad body Print BinOpMod Str TupleLoad Call NameLoad NameLoad NameLoad Print Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign TupleStore NameStore NameStore NameLoad Assign NameStore Call NameLoad NameLoad NameLoad If UnaryOpNot NameLoad body Return NameLoad Assign NameStore Call NameLoad NameLoad NameLoad Return Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body For NameStore NameLoad body Print BinOpMod Str NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad For NameStore NameLoad body If NameLoad body Expr Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign TupleStore NameStore NameStore NameStore NameLoad Assign NameStore Call NameLoad NameLoad NameLoad If UnaryOpNot NameLoad body Return NameLoad Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad NameLoad If UnaryOpNot NameLoad body Return NameLoad Assign NameStore Call NameLoad NameLoad NameLoad If UnaryOpNot NameLoad body Return NameLoad Return Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body Print BinOpMod Str NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad For NameStore NameLoad body If NameLoad body For NameStore NameLoad body Expr Call NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad Call NameLoad BinOpMod Str NameLoad For NameStore NameLoad body Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad For NameStore NameLoad body If CompareNotIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args defaults body Assign NameStore Call AttributeLoad NameLoad attr keyword Str Expr Call AttributeLoad NameLoad attr Str keyword ListLoad Str Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword ListLoad Str Str Str Str Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Num Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Num Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpMod Str AttributeLoad NameLoad attr If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpMod Str AttributeLoad NameLoad attr If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpMod Str AttributeLoad NameLoad attr If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Assign NameStore BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad If CompareEq AttributeLoad NameLoad attr Str body Expr Call NameLoad AttributeLoad NameLoad attr orelse If CompareEq AttributeLoad NameLoad attr Str body Expr Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr orelse If CompareEq AttributeLoad NameLoad attr Str body Expr Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr orelse If CompareEq AttributeLoad NameLoad attr Str body Expr Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr orelse If CompareEq AttributeLoad NameLoad attr Str body Expr Call NameLoad AttributeLoad NameLoad attr decorator_list If CompareEq NameLoad Str body Expr Call NameLoad
N/A N/A Script for downloading and generating question/answer pairs.
 N/A argparse collections namedtuple N/A hashlib itertools chain itertools izip itertools repeat N/A math multiprocessing.pool Pool multiprocessing.pool ThreadPool N/A os N/A re N/A sys N/A time N/A cchardet chardet lxml html N/A requests N/A socket Story N/A N/A namedtuple StoryBase url content highlights N/A ToString N/A N/A self N/A N/A N/A N/A N/A self content N/A N/A join N/A N/A 

@highlight

 highlight N/A highlight N/A self highlights N/A N/A N/A AnonymizedStory N/A namedtuple AnonymizedStory url content highlights anonymization_info N/A RawStory N/A namedtuple RawStory url html N/A TokenizedStory N/A namedtuple TokenizedStory url tokens QuestionContext N/A N/A namedtuple QuestionContextBase url context question answer anonymization_info N/A ToString N/A N/A self N/A N/A N/A N/A %s

%s

%s

%s

%s N/A N/A self url N/A self context N/A self question N/A self answer N/A N/A join N/A N/A N/A key : value N/A N/A key value N/A N/A N/A self anonymization_info iteritems N/A N/A ReadUrls N/A N/A filename N/A N/A N/A Reads a list of URLs.

  Args:
    filename: The filename containing the URLs.

  Returns:
    A list of URLs.
   N/A N/A open filename f N/A N/A N/A N/A N/A line strip N/A line f N/A ReadMultipleUrls N/A N/A filename N/A N/A N/A Reads a list of URL lists.

  Each line in the filename should contain a list of URLs separated by comma.

  Args:
    filename: The filename containing the URLs.

  Returns:
    A list of list of URLs.
   N/A N/A open filename f N/A N/A N/A N/A N/A N/A N/A line strip split , N/A line f N/A WriteUrls N/A N/A filename urls N/A N/A N/A Writes a list of URLs to a file.

  Args:
    filename: The filename to the file where the URLs should be written.
    urls: The list of URLs to write.
   N/A N/A open filename w f N/A N/A N/A N/A f writelines N/A N/A url N/A url urls N/A Hashhex N/A N/A s N/A N/A N/A Returns a heximal formated SHA1 hash of the input string.

  Args:
    s: The string to hash.

  Returns:
    A heximal formatted hash of the input string.
   N/A h N/A N/A hashlib sha1 N/A N/A N/A h update s N/A N/A N/A h hexdigest N/A ReadDownloadedUrl N/A N/A url corpus N/A N/A N/A Reads a downloaded URL from disk.

  Args:
    url: The URL to read.
    corpus: The corpus the URL belongs to.

  Returns:
    The content of the URL.
   N/A N/A N/A N/A open N/A %s/downloads/%s.html N/A corpus N/A Hashhex url f N/A N/A N/A N/A f read N/A N/A N/A IOError N/A N/A None N/A N/A wayback_pattern N/A N/A re compile web/([^/]*)/ WaybackUrl N/A N/A urls max_attempts N/A 6 N/A N/A Retrieves the URL for the latest historic copy using Wayback Machine.

  Args:
    urls: The URL for a specific page (canonical URL + forwarding URL's).
    max_attempts: The maximum attempts at requesting the URL.

  Returns:
    The URL or None if no copy is stored for the URL.

  Raises:
    RuntimeError: Failed to retrieve the URL.
   N/A N/A urls N/A N/A None N/A url N/A urls N/A 0 N/A index_collection_url http://archive.org/wayback/available N/A payload N/A url url N/A attempts 0 N/A N/A attempts max_attempts N/A N/A N/A N/A entry_req N/A N/A requests get index_collection_url params payload allow_redirects False N/A N/A N/A entry_req status_code N/A N/A requests codes ok N/A N/A N/A WaybackUrl N/A urls N/A 1 max_attempts N/A entry N/A N/A entry_req json N/A N/A closest N/A entry N/A archived_snapshots N/A N/A N/A WaybackUrl N/A urls N/A 1 max_attempts N/A wayback_url N/A N/A N/A entry N/A archived_snapshots N/A closest N/A url N/A wayback_url N/A N/A wayback_pattern sub web/\g<1>id_/ wayback_url 1 N/A wayback_url N/A N/A N/A N/A N/A requests exceptions ConnectionError N/A N/A N/A N/A N/A time sleep N/A N/A math pow 2 attempts N/A attempts 1 N/A N/A RuntimeError N/A Failed to download URL for %s after %d attempts. Please run the script again. N/A url max_attempts N/A DownloadUrl N/A N/A url corpus max_attempts timeout N/A 5 5 N/A N/A Downloads a URL.

  Args:
    url: The URL.
    corpus: The corpus of the URL.
    max_attempts: Max attempts for downloading the URL.
    timeout: Connection timeout in seconds for each attempt.

  Returns:
    The HTML at the URL or None if the request failed.
   N/A N/A N/A N/A open N/A %s/downloads/%s.html N/A corpus N/A Hashhex url f N/A N/A N/A N/A f read N/A N/A N/A IOError N/A N/A N/A attempts 0 N/A N/A attempts max_attempts N/A N/A N/A N/A req N/A N/A requests get url allow_redirects False timeout timeout N/A N/A N/A req status_code N/A N/A requests codes ok N/A N/A content N/A N/A N/A req text encode N/A req encoding N/A N/A open N/A %s/downloads/%s.html N/A corpus N/A Hashhex url w f N/A N/A N/A N/A f write content N/A content N/A N/A N/A N/A N/A req status_code N/A 301 302 404 503 N/A attempts N/A max_attempts 1 N/A N/A None N/A N/A N/A N/A N/A requests exceptions ConnectionError N/A N/A N/A N/A N/A N/A requests exceptions ContentDecodingError N/A N/A None N/A N/A N/A N/A requests exceptions ChunkedEncodingError N/A N/A None N/A N/A N/A N/A requests exceptions Timeout N/A N/A N/A N/A N/A socket timeout N/A N/A N/A N/A N/A time sleep N/A N/A math pow 2 attempts N/A attempts 1 N/A None N/A ParseHtml N/A N/A story corpus N/A N/A N/A Parses the HTML of a news story.

  Args:
    story: The raw Story to be parsed.
    corpus: Either 'cnn' or 'dailymail'.

  Returns:
    A Story containing URL, paragraphs and highlights.
   N/A parser N/A N/A html HTMLParser encoding N/A N/A N/A chardet detect N/A story html N/A encoding N/A tree N/A N/A html document_fromstring N/A story html parser parser N/A delete_selectors N/A cnn dailymail N/A //blockquote[contains(@class, "twitter-tweet")] //blockquote[contains(@class, "instagram-media")] N/A //blockquote[contains(@class, "twitter-tweet")] //blockquote[contains(@class, "instagram-media")] N/A cnn_exclude not(ancestor::*[contains(@class, "metadata")]) and not(ancestor::*[contains(@class, "pullquote")]) and not(ancestor::*[contains(@class, "SandboxRoot")]) and not(ancestor::*[contains(@class, "twitter-tweet")]) and not(ancestor::div[contains(@class, "cnnStoryElementBox")]) and not(contains(@class, "cnnTopics")) and not(descendant::*[starts-with(text(), "Read:")]) and not(descendant::*[starts-with(text(), "READ:")]) and not(descendant::*[starts-with(text(), "Join us at")]) and not(descendant::*[starts-with(text(), "Join us on")]) and not(descendant::*[starts-with(text(), "Read CNNOpinion")]) and not(descendant::*[contains(text(), "@CNNOpinion")]) and not(descendant-or-self::*[starts-with(text(), "Follow us")]) and not(descendant::*[starts-with(text(), "MORE:")]) and not(descendant::*[starts-with(text(), "SPOILER ALERT:")]) N/A dm_exclude not(ancestor::*[contains(@id,"reader-comments")]) and not(contains(@class, "byline-plain")) and not(contains(@class, "byline-section")) and not(contains(@class, "count-number")) and not(contains(@class, "count-text")) and not(contains(@class, "video-item-title")) and not(ancestor::*[contains(@class, "column-content")]) and not(ancestor::iframe) N/A paragraph_selectors N/A cnn dailymail N/A N/A //div[contains(@class, "cnnContentContainer")]//p[%s] cnn_exclude N/A //div[contains(@class, "l-container")]//p[%s] cnn_exclude N/A //div[contains(@class, "cnn_strycntntlft")]//p[%s] cnn_exclude N/A N/A //div[contains(@class, "article-text")]//p[%s] dm_exclude N/A he not(contains(@class, "cnnHiliteHeader")) and not(descendant::*[starts-with(text(), "Next Article in")]) N/A highlight_selectors N/A cnn dailymail N/A N/A //*[contains(@class, "el__storyhighlights__list")]//li[%s] he N/A //*[contains(@class, "cnnStryHghLght")]//li[%s] he N/A //*[@id="cnnHeaderRightCol"]//li[%s] he N/A //h1/following-sibling::ul//li ExtractText N/A N/A selector N/A N/A N/A Extracts a list of paragraphs given a XPath selector.

    Args:
      selector: A XPath selector to find the paragraphs.

    Returns:
      A list of raw text paragraphs with leading and trailing whitespace.
     N/A xpaths N/A map N/A tree xpath selector N/A elements N/A list N/A N/A chain from_iterable xpaths N/A paragraphs N/A N/A N/A N/A N/A e text_content encode utf-8 N/A e elements N/A N/A N/A corpus cnn N/A N/A len paragraphs 2 N/A (CNN) N/A paragraphs N/A 1 N/A N/A N/A N/A paragraphs pop 0 N/A paragraphs N/A map N/A str strip paragraphs N/A paragraphs N/A s N/A s paragraphs N/A s N/A N/A N/A str isspace s N/A paragraphs N/A N/A selector N/A delete_selectors N/A corpus N/A N/A bad N/A N/A tree xpath selector N/A N/A N/A N/A N/A N/A bad getparent remove bad N/A paragraphs N/A ExtractText N/A paragraph_selectors N/A corpus N/A highlights N/A ExtractText N/A highlight_selectors N/A corpus N/A content N/A N/A join paragraphs N/A N/A Story N/A story url content highlights N/A WriteStory N/A N/A story corpus N/A N/A N/A Writes a news story to disk.

  Args:
    story: The news story to write.
    corpus: The corpus the news story belongs to.
   N/A story_string N/A N/A story ToString N/A url_hash N/A Hashhex N/A story url N/A N/A open N/A %s/stories/%s.story N/A corpus url_hash w f N/A N/A N/A N/A f write story_string N/A LoadTokenMapping N/A N/A filename N/A N/A N/A Loads a token mapping from the given filename.

  Args:
    filename: The filename containing the token mapping.

  Returns:
    A list of (start, end) where start and
    end (inclusive) are offsets into the content for a token. The list is
    sorted.
   N/A mapping N/A N/A N/A open filename f N/A N/A line N/A N/A N/A N/A f readline strip N/A token_mapping N/A N/A line split ; N/A N/A N/A token_mapping N/A N/A N/A N/A start length N/A N/A token_mapping split , N/A N/A N/A mapping append N/A N/A int start N/A N/A int start N/A int length N/A N/A N/A mapping sort key N/A N/A N/A x N/A N/A x N/A 1 N/A mapping N/A Tokenize N/A N/A story corpus N/A N/A N/A Tokenizes a news story.

  Args:
    story: The Story.
    corpus: The corpus of the news story.

  Returns:
    A TokenizedStory containing the URL and the tokens or None if no token
    mapping was found for the URL.
   N/A s N/A N/A story ToString N/A url_hash N/A Hashhex N/A story url N/A mapping_filename N/A %s/tokens/%s.txt N/A corpus url_hash N/A N/A N/A N/A N/A os path exists mapping_filename N/A N/A None N/A mapping N/A LoadTokenMapping mapping_filename N/A tokens N/A N/A N/A start end mapping N/A N/A N/A N/A tokens append N/A s N/A start N/A end 1 N/A N/A TokenizedStory N/A story url tokens N/A LoadEntityMapping N/A N/A filename N/A N/A N/A Loads an entity mapping from the given filename.

  Args:
    filename: The filename containing the entity mapping.

  Returns:
    A list of (entity_index, start, end)
    where start and end (inclusive) are token offsets for an entity. The list
    is sorted.
   N/A mapping N/A N/A N/A open filename f N/A N/A line N/A N/A N/A N/A f readline strip N/A entity_mapping N/A N/A line split ; N/A N/A N/A entity_mapping N/A N/A N/A N/A entity_index start end N/A N/A entity_mapping split , N/A N/A N/A mapping append N/A N/A int entity_index N/A int start N/A int end N/A N/A N/A mapping sort key N/A N/A N/A x N/A N/A x N/A 2 N/A mapping N/A Anonymize N/A N/A tokenized_story corpus N/A N/A N/A Anonymizes a tokenized news story.

  Args:
    tokenized_story: A TokenizedStory.
    corpus: The corpus of the tokenized news story.

  Returns:
    A Story containing the URL, anonymized content and anonymized highlights or
    None if no entity mapping exists for the news story.
   N/A url_hash N/A Hashhex N/A tokenized_story url N/A mapping_filename N/A %s/entities/%s.txt N/A corpus url_hash N/A N/A N/A N/A N/A os path exists mapping_filename N/A N/A None N/A mapping N/A LoadEntityMapping mapping_filename N/A mapping_index 0 N/A mapping_len N/A len mapping N/A new_tokens N/A N/A anonymization_info N/A N/A i 0 N/A N/A i N/A len N/A tokenized_story tokens N/A N/A N/A N/A mapping_index mapping_len N/A N/A N/A mapping N/A mapping_index N/A 1 i N/A N/A N/A entity_index start end N/A mapping N/A mapping_index N/A anonymized_entity_name N/A @entity%d entity_index N/A N/A N/A new_tokens append anonymized_entity_name N/A N/A anonymization_info N/A anonymized_entity_name N/A N/A N/A N/A join N/A N/A tokenized_story tokens N/A start N/A end 1 replace  -  - N/A mapping_index 1 N/A i N/A end 1 N/A N/A N/A N/A new_tokens append N/A N/A tokenized_story tokens N/A i N/A i 1 N/A parts N/A N/A N/A N/A join new_tokens split  @ highlight  N/A content N/A parts N/A 0 N/A highlights N/A parts N/A 1 N/A N/A AnonymizedStory N/A tokenized_story url content highlights anonymization_info N/A N/A entity_pattern N/A N/A re compile @entity\d+ GenerateQuestionContexts N/A N/A anonymized_story context_token_limit N/A N/A N/A Generates a list of question/answer pairs given an anonymized news story.

  One question/answer pair is generated for each anonymized entity appearing in
  the question.

  Args:
    anonymized_story: The anonymized news story.
    context_token_limit: If the context of a news story is above the limit, the
        empty list will be returned.

  Returns:
    A list of QuestionContext containing questions and answers.
   N/A result N/A N/A N/A N/A N/A N/A N/A anonymized_story content count 1 context_token_limit N/A N/A result N/A entities_in_context N/A set N/A N/A entity_pattern findall N/A anonymized_story content N/A highlight N/A anonymized_story highlights N/A N/A match N/A N/A entity_pattern finditer highlight N/A N/A N/A start end N/A N/A match span N/A answer N/A highlight N/A start end N/A N/A answer entities_in_context N/A N/A N/A question N/A N/A N/A %s@placeholder%s N/A N/A highlight N/A 0 start N/A highlight N/A end lower N/A context N/A N/A N/A anonymized_story content lower N/A url N/A anonymized_story url N/A anonymization_info N/A anonymized_story anonymization_info N/A N/A N/A result append N/A QuestionContext url context question answer anonymization_info N/A result N/A WriteQuestionContext N/A N/A question_context corpus dataset N/A N/A N/A Writes a question/answer pair to disk.

  Args:
    question_context: The QuestionContext to write containing the question and
        answer.
    corpus: The corpus the question/answer belongs to.
    dataset: One of 'training', 'validation' and 'test'.
   N/A s N/A N/A question_context ToString N/A h N/A Hashhex s N/A N/A open N/A %s/questions/%s/%s.question N/A corpus dataset h w f N/A N/A N/A N/A f write s N/A ProgressBar N/A object N/A N/A Simple progress bar.

  Output example:
    100.00% [2152/2152]
   __init__ N/A N/A self total stream N/A 100 N/A sys stderr N/A N/A N/A self total total N/A N/A self stream stream N/A N/A self last_len 0 N/A N/A self curr 0 N/A Increment N/A N/A self N/A N/A N/A N/A self curr 1 N/A N/A N/A self PrintProgress N/A self curr N/A N/A N/A self curr N/A self total N/A N/A N/A PrintProgress N/A N/A self value N/A N/A N/A N/A N/A N/A self stream write N/A N/A self last_len N/A pct N/A N/A 100 N/A self curr N/A float N/A self total N/A out N/A N/A {:.2f}% [{}/{}] format pct value N/A self total N/A N/A self last_len N/A len out N/A N/A N/A N/A self stream write out N/A N/A N/A N/A self stream flush N/A N/A N/A datasets N/A training validation test UrlMode N/A N/A corpus request_parallelism N/A N/A N/A Finds Wayback Machine URLs and writes them to disk.

  Args:
    corpus: A corpus.
    request_parallelism: The number of concurrent requests.
   N/A dataset datasets N/A N/A N/A Finding Wayback Machine URLs for the %s set: dataset N/A old_urls_filename N/A %s/%s_urls.txt N/A corpus dataset N/A new_urls_filename N/A %s/wayback_%s_urls.txt N/A corpus dataset N/A urls N/A ReadMultipleUrls old_urls_filename N/A p N/A ThreadPool request_parallelism N/A results N/A N/A p imap_unordered WaybackUrl urls N/A progress_bar N/A ProgressBar N/A len urls N/A new_urls N/A N/A result results N/A N/A result N/A N/A N/A N/A new_urls append result N/A N/A N/A progress_bar Increment N/A N/A WriteUrls new_urls_filename new_urls N/A DownloadMapper N/A N/A t N/A N/A N/A Downloads an URL and checks that metadata is available for the URL.

  Args:
    t: a tuple (url, corpus).

  Returns:
    A pair of URL and content.

  Raises:
    RuntimeError: No metadata available.
   N/A N/A url corpus t N/A url_hash N/A Hashhex url N/A mapping_filename N/A %s/entities/%s.txt N/A corpus url_hash N/A N/A N/A N/A N/A os path exists mapping_filename N/A N/A N/A RuntimeError N/A No metadata available for %s. url N/A N/A url N/A DownloadUrl url corpus N/A DownloadMode N/A N/A corpus request_parallelism N/A N/A N/A Downloads the URLs for the specified corpus.

  Args:
    corpus: A corpus.
    request_parallelism: The number of concurrent download requests.
   N/A missing_urls N/A N/A dataset datasets N/A N/A N/A Downloading URLs for the %s set: dataset N/A urls_filename N/A %s/wayback_%s_urls.txt N/A corpus dataset N/A urls N/A ReadUrls urls_filename N/A missing_urls_filename N/A %s/missing_urls.txt corpus N/A N/A N/A N/A os path exists missing_urls_filename N/A N/A Only downloading missing URLs N/A urls N/A list N/A N/A N/A set urls intersection N/A ReadUrls missing_urls_filename N/A p N/A ThreadPool request_parallelism N/A results N/A N/A p imap_unordered DownloadMapper N/A izip urls N/A repeat corpus N/A progress_bar N/A ProgressBar N/A len urls N/A collected_urls N/A N/A N/A N/A N/A url story_html results N/A N/A story_html N/A N/A N/A N/A collected_urls append url N/A N/A N/A progress_bar Increment N/A N/A N/A KeyboardInterrupt N/A N/A Interrupted by user N/A N/A N/A missing_urls extend N/A N/A set urls N/A set collected_urls N/A N/A WriteUrls N/A %s/missing_urls.txt corpus missing_urls N/A missing_urls N/A N/A N/A %d URLs couldn't be downloaded, see %s/missing_urls.txt. N/A N/A len missing_urls corpus N/A Try and run the command again to download the missing URLs. N/A StoreMapper N/A N/A t N/A N/A N/A Reads an URL from disk and returns the parsed news story.

  Args:
    t: a tuple (url, corpus).

  Returns:
    A Story containing the parsed news story.
   N/A N/A url corpus t N/A story_html N/A ReadDownloadedUrl url corpus N/A N/A story_html N/A N/A None N/A raw_story N/A RawStory url story_html N/A N/A ParseHtml raw_story corpus N/A StoreMode N/A N/A corpus N/A N/A N/A dataset datasets N/A N/A N/A Storing news stories for the %s set: dataset N/A urls_filename N/A %s/wayback_%s_urls.txt N/A corpus dataset N/A urls N/A ReadUrls urls_filename N/A p N/A Pool N/A stories N/A N/A p imap_unordered StoreMapper N/A izip urls N/A repeat corpus N/A progress_bar N/A ProgressBar N/A len urls N/A story stories N/A N/A story N/A N/A N/A WriteStory story corpus N/A N/A N/A progress_bar Increment N/A GenerateMapper N/A N/A t N/A N/A N/A Reads an URL from disk and returns a list of question/answer pairs.

  Args:
    t: a tuple (url, corpus).

  Returns:
    A list of QuestionContext containing a question and an answer.
   N/A N/A url corpus context_token_limit t N/A story_html N/A ReadDownloadedUrl url corpus N/A N/A story_html N/A N/A None N/A raw_story N/A RawStory url story_html N/A story N/A ParseHtml raw_story corpus N/A tokenized N/A Tokenize story corpus N/A N/A tokenized N/A N/A None N/A anonymized N/A Anonymize tokenized corpus N/A N/A anonymized N/A N/A None N/A N/A GenerateQuestionContexts anonymized context_token_limit N/A GenerateMode N/A N/A corpus context_token_limit N/A N/A N/A dataset datasets N/A N/A N/A Generating questions for the %s set: dataset N/A urls_filename N/A %s/wayback_%s_urls.txt N/A corpus dataset N/A urls N/A ReadUrls urls_filename N/A p N/A Pool N/A question_context_lists N/A N/A p imap_unordered GenerateMapper N/A izip urls N/A repeat corpus N/A repeat context_token_limit N/A progress_bar N/A ProgressBar N/A len urls N/A question_context_list question_context_lists N/A N/A question_context_list N/A N/A question_context question_context_list N/A N/A N/A WriteQuestionContext question_context corpus dataset N/A N/A N/A progress_bar Increment N/A RemoveMode N/A N/A corpus N/A N/A N/A missing_urls N/A set N/A ReadUrls N/A %s/missing_urls.txt corpus N/A dataset datasets N/A N/A urls_filename N/A %s/wayback_%s_urls.txt N/A corpus dataset N/A urls N/A ReadUrls urls_filename N/A new_urls N/A N/A url urls N/A N/A N/A url missing_urls N/A N/A N/A N/A new_urls append url N/A N/A WriteUrls urls_filename new_urls N/A main N/A N/A N/A N/A N/A parser N/A N/A argparse ArgumentParser description Generates question/answer pairs N/A N/A N/A parser add_argument --corpus choices N/A cnn dailymail default cnn N/A N/A N/A parser add_argument --mode choices N/A store generate download urls remove default generate N/A N/A N/A parser add_argument --request_parallelism type int default 200 N/A N/A N/A parser add_argument --context_token_limit type int default 2000 N/A args N/A N/A parser parse_args N/A stories_dir N/A %s/stories N/A args corpus N/A N/A N/A N/A N/A os path exists stories_dir N/A N/A N/A N/A os mkdir stories_dir N/A downloads_dir N/A %s/downloads N/A args corpus N/A N/A N/A N/A N/A os path exists downloads_dir N/A N/A N/A N/A os mkdir downloads_dir N/A questions_dir N/A %s/questions N/A args corpus N/A N/A N/A N/A N/A os path exists questions_dir N/A N/A N/A N/A os mkdir questions_dir N/A dataset datasets N/A N/A dataset_dir N/A %s/questions/%s N/A N/A args corpus dataset N/A N/A N/A N/A N/A os path exists dataset_dir N/A N/A N/A N/A os mkdir dataset_dir N/A N/A N/A args mode store N/A N/A N/A StoreMode N/A args corpus N/A N/A N/A N/A args mode generate N/A N/A N/A GenerateMode N/A args corpus N/A args context_token_limit N/A N/A N/A N/A args mode download N/A N/A N/A DownloadMode N/A args corpus N/A args request_parallelism N/A N/A N/A N/A args mode urls N/A N/A N/A UrlMode N/A args corpus N/A args request_parallelism N/A N/A N/A N/A args mode remove N/A N/A N/A RemoveMode N/A args corpus N/A N/A N/A __name__ __main__ N/A N/A N/A main