Module Expr Str ImportFrom alias alias alias ImportFrom alias Import alias Import alias identifier Import alias ImportFrom alias Import alias ImportFrom alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad Str Str Str Str Str Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str keyword BinOpBitOr AttributeLoad NameLoad attr AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Expr Str Return Call NameLoad Lambda arguments args NameParam defaults Call AttributeLoad Call AttributeLoad NameLoad attr Num attr Str Str Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return Call NameLoad Lambda arguments args NameParam defaults Call AttributeLoad Call AttributeLoad NameLoad attr Num attr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return Call NameLoad Lambda arguments args NameParam defaults Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad NameLoad Str For NameStore Call AttributeLoad NameLoad attr Str body TryExcept body Assign NameStore SubscriptLoad NameLoad Index Str handlers ExceptHandler type NameLoad body Continue Expr Yield Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Str For NameStore Call AttributeLoad NameLoad attr Str body TryExcept body Assign NameStore SubscriptLoad NameLoad Index Str handlers ExceptHandler type NameLoad body Continue Expr Yield Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad For NameStore NameLoad body AugAssignAdd SubscriptStore NameLoad Index AttributeLoad Call NameLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad For NameStore NameLoad body For NameStore Call NameLoad NameLoad Call AttributeLoad AttributeLoad Call NameLoad NameLoad attr attr Str body AugAssignAdd SubscriptStore NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad Call NameLoad NameLoad attr Str TryExcept body Return Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr body TryExcept body Return Call AttributeLoad NameLoad attr Str NameLoad handlers ExceptHandler type NameLoad body Return NameLoad ExceptHandler type NameLoad body Return NameLoad ExceptHandler body Expr Call AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body If UnaryOpNot Call NameLoad NameLoad NameLoad body TryExcept body Return Call AttributeLoad NameLoad attr Str handlers ExceptHandler type NameLoad body Return Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list Assign NameStore Call AttributeLoad NameLoad attr Str keyword BinOpBitOr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str keyword AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Assign NameStore ListLoad For NameStore Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore SubscriptLoad NameLoad Slice SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index Num Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad If NameLoad body TryExcept body Assign NameStore Call NameLoad NameLoad handlers ExceptHandler body Continue Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad Return NameLoad decorator_list
N/A N/A ``dossier.models.features.basic`` provides simple transforms that
construct features.

.. This software is released under an MIT/X11 open source license.
   Copyright 2012-2015 Diffeo, Inc.

 __future__ absolute_import division print_function itertools imap N/A logging N/A regex re N/A traceback bs4 BeautifulSoup N/A urlnorm urlparse urlparse dossier.fc StringCounter N/A logger N/A N/A logging getLogger __name__ N/A REGEX_PHONES N/A \d-?\d{3}-?\d{3}-?\d{4} \d{3}-?\d{3}-?\d{4} \d{3}-?\d{4} \d{2}-?\d{11} \d{2,3}-?\d{7,15} N/A REGEX_PHONE N/A N/A re compile N/A N/A | join REGEX_PHONES N/A REGEX_EMAIL N/A N/A re compile \b\S+@\S+\.\S+\b N/A re IGNORECASE N/A REGEX_SKYPE N/A N/A re compile skype(\s|ID|handle|name|\:)*(?P<skype>[a-zA-Z][a-zA-Z0-9,-._]{5,31}) flags N/A N/A re UNICODE N/A re IGNORECASE phones N/A N/A text N/A N/A N/A Returns list of phone numbers without punctuation. N/A N/A imap N/A N/A N/A m N/A N/A N/A N/A N/A m group 0 replace - N/A N/A REGEX_PHONE finditer text N/A emails N/A N/A text N/A N/A N/A Returns list of phone numbers without punctuation. N/A N/A imap N/A N/A N/A m N/A N/A N/A N/A N/A m group 0 lower N/A N/A REGEX_EMAIL finditer text N/A skypes N/A N/A text N/A N/A N/A Returns list of skype handles. N/A N/A imap N/A N/A N/A m N/A N/A N/A m group skype N/A N/A REGEX_SKYPE finditer text N/A image_urls N/A N/A html N/A N/A N/A soup N/A BeautifulSoup html lxml N/A node N/A N/A soup find_all img N/A N/A N/A N/A src N/A node N/A src N/A N/A N/A KeyError N/A N/A N/A N/A N/A norm_url src N/A a_urls N/A N/A html N/A N/A N/A 
    return normalized urls found in the 'a' tag
     N/A soup N/A BeautifulSoup html lxml N/A node N/A N/A soup find_all a N/A N/A N/A N/A href N/A node N/A href N/A N/A N/A KeyError N/A N/A N/A N/A N/A norm_url href N/A host_names N/A N/A urls N/A N/A N/A 
    Takes a StringCounter of normalized URL and parses their hostnames

    N.B. this assumes that absolute URLs will begin with

    http://

    in order to accurately resolve the host name.
    Relative URLs will not have host names.
     N/A host_names N/A StringCounter N/A url urls N/A N/A N/A host_names N/A N/A N/A urlparse url netloc N/A urls N/A url N/A host_names N/A path_dirs N/A N/A urls N/A N/A N/A 
    Takes a StringCounter of normalized URL and parses them into
    a list of path directories. The file name is
    included in the path directory list.
     N/A path_dirs N/A StringCounter N/A url urls N/A N/A path_dir N/A filter None N/A N/A N/A N/A urlparse url path split / N/A N/A N/A path_dirs N/A path_dir N/A urls N/A url N/A path_dirs N/A norm_url N/A N/A url N/A N/A N/A url N/A N/A N/A uni url encode utf-8 N/A N/A N/A N/A N/A urlnorm norm url N/A N/A N/A N/A urlnorm InvalidUrl N/A N/A N/A N/A N/A N/A urlnorm norm_path url N/A N/A N/A UnicodeDecodeError N/A N/A url N/A N/A UnicodeDecodeError N/A N/A url N/A N/A N/A N/A N/A traceback print_exc N/A None N/A uni N/A N/A s N/A N/A N/A N/A N/A isinstance s unicode N/A N/A N/A N/A N/A N/A s decode utf-8 N/A N/A N/A UnicodeDecodeError N/A N/A N/A N/A s decode latin-1 N/A s N/A N/A REGEX_ICQ N/A N/A re compile icq(\s|number|\:)*(?P<icq>[0-9-]{3,9}(\s*[0-9-]{,9})*) flags N/A N/A re UNICODE N/A re IGNORECASE N/A REGEX_DIGITS N/A N/A re compile [^0-9] flags N/A re MULTILINE ICQs N/A N/A text N/A N/A N/A icqs N/A N/A match N/A N/A REGEX_ICQ finditer text N/A N/A sub N/A N/A match span icq N/A raw_string N/A text N/A N/A sub N/A 0 N/A sub N/A 1 N/A icq N/A N/A digits_re sub raw_string N/A icq N/A N/A N/A N/A icq N/A int icq N/A N/A N/A N/A N/A N/A N/A icqs append N/A str icq N/A icqs N/A