Module Import alias Import alias Import alias Import alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Str FunctionDef arguments args defaults body Expr Str Assign NameStore ListLoad Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Return Call AttributeLoad NameLoad attr NameLoad decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr ListLoad Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr Str Assign NameStore Num Assign AttributeStore NameLoad attr ListLoad While NameLoad body Assign NameStore AttributeLoad Call AttributeLoad NameLoad attr BinOpMod NameLoad NameLoad attr Assign NameStore Call AttributeLoad Call NameLoad NameLoad Str attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr Str keyword Str keyword Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str keyword Str keyword Call AttributeLoad NameLoad attr Str If UnaryOpNot NameLoad body Break For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Str Call AttributeLoad AttributeLoad NameLoad attr attr If Call AttributeLoad NameLoad attr Str NameLoad body If CompareGt Call NameLoad NameLoad Num body Assign NameStore SubscriptLoad NameLoad Index Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr Call NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AugAssignAdd NameStore Num Expr Call AttributeLoad NameLoad attr BinOpMod Str Call NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str TryExcept body Assign NameStore Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return NameLoad handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return NameLoad decorator_list decorator_list
N/A N/A random N/A requests N/A log N/A re bs4 BeautifulSoup N/A LOG N/A N/A log get_logger zxLogger N/A PROXY_POOL_URL http://proxy-list.org/english/search.php?search=CN&country=CN&type=any&port=any&ssl=any&p=%d get_AGENT N/A N/A N/A N/A N/A  randome choose a UserAgent string N/A Agents N/A Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2226.0 Safari/537.36 Mozilla/5.0 (Windows NT 6.4; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0 Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20130401 Firefox/31.0 Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0 Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/29.0 Mozilla/5.0 (X11; OpenBSD amd64; rv:28.0) Gecko/20100101 Firefox/28.0 Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 Mozilla/5.0 (Windows NT 6.1; rv:27.3) Gecko/20130101 Firefox/27.3 Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:27.0) Gecko/20121011 Firefox/27.0 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10 Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko ) Version/5.1 Mobile/9B176 Safari/7534.48.3 Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; de-at) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; da-dk) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1 Mozilla/5.0 (Linux; U; Android 4.0.3; ko-kr; LG-L160L Build/IML74K) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30 Mozilla/5.0 (Linux; U; Android 4.0.3; de-ch; HTC Sensation Build/IML74K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30 Mozilla/5.0 (Linux; U; Android 2.3; en-us) AppleWebKit/999+ (KHTML, like Gecko) Safari/999.9 Mozilla/5.0 (Linux; U; Android 2.3.5; zh-cn; HTC_IncredibleS_S710e Build/GRJ90) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; HTC Vision Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Mozilla/5.0 (Linux; U; Android 2.3.4; fr-fr; HTC Desire Build/GRJ22) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; T-Mobile myTouch 3G Slide Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Mozilla/5.0 (Linux; U; Android 2.3.3; zh-tw; HTC_Pyramid Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Mozilla/5.0 (Linux; U; Android 2.3.3; zh-tw; HTC_Pyramid Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari Mozilla/5.0 (Linux; U; Android 2.3.3; zh-tw; HTC Pyramid Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Mozilla/5.0 (Linux; U; Android 2.3.3; ko-kr; LG-LU3000 Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16 Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14 Mozilla/5.0 (Windows NT 6.0; rv:2.0) Gecko/20100101 Firefox/4.0 Opera 12.14 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0) Opera 12.14 Opera/12.80 (Windows NT 5.1; U; en) Presto/2.10.289 Version/12.02 Opera/9.80 (Windows NT 6.1; U; es-ES) Presto/2.9.181 Version/12.00 Opera/9.80 (Windows NT 5.1; U; zh-sg) Presto/2.9.181 Version/12.00 Opera/12.0(Windows NT 5.2;U;en)Presto/22.9.168 Version/12.00 Opera/12.0(Windows NT 5.1;U;en)Presto/22.9.168 Version/12.00 Mozilla/5.0 (Windows NT 5.1) Gecko/20100101 Firefox/14.0 Opera/12.0 Opera/9.80 (Windows NT 6.1; WOW64; U; pt) Presto/2.10.229 Version/11.62 Opera/9.80 (Windows NT 6.0; U; pl) Presto/2.10.229 Version/11.62 N/A N/A N/A random choice Agents N/A ProxyPool N/A object N/A N/A 
    read proxies servers list from websites
    and generate proxy from the lists
     __init__ N/A N/A self N/A N/A N/A N/A self proxies N/A N/A N/A N/A self load_list N/A N/A self it_proxy N/A iter N/A self proxies N/A load_list N/A N/A self N/A N/A N/A 
        read proxy list from website
         N/A N/A N/A LOG debug loading proxy list N/A page 1 N/A N/A self proxies N/A N/A True N/A N/A html N/A N/A N/A requests get N/A PROXY_POOL_URL page text N/A table N/A N/A N/A BeautifulSoup html html.parser find id proxy-table N/A proxy_tags N/A N/A table find_all li class_ proxy text N/A N/A re compile \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d+ N/A speed_tags N/A N/A table find_all li class_ speed text N/A N/A re compile [0-9.]k.*|- N/A N/A proxy_tags N/A N/A N/A speed_tag speed_tags N/A N/A speed N/A N/A re sub kb.* N/A N/A N/A speed_tag string strip N/A N/A N/A re match [0-9.]+ speed N/A N/A N/A N/A float speed 150 N/A N/A proxy_tag N/A proxy_tags N/A N/A N/A speed_tags index speed_tag N/A N/A N/A LOG debug N/A pick proxy: %s, speed %fkb N/A N/A proxy_tag string N/A float speed N/A N/A N/A N/A self proxies append N/A proxy_tag string N/A page 1 N/A N/A N/A LOG debug N/A proxy list parsed, total %d proxies N/A len N/A self proxies N/A add_proxy N/A N/A self proxy N/A N/A N/A N/A N/A N/A self proxies append proxy N/A del_proxy N/A N/A self proxy N/A N/A N/A N/A N/A N/A self proxies remove proxy N/A get_proxy N/A N/A self N/A N/A N/A 
        cycly generate the proxy server string
         N/A N/A N/A proxy N/A next N/A self it_proxy N/A N/A N/A LOG debug N/A fetched proxy : %s proxy N/A proxy N/A N/A N/A StopIteration N/A N/A N/A N/A LOG debug .... re-iterating proxies N/A N/A self it_proxy N/A iter N/A self proxies N/A proxy N/A next N/A self it_proxy N/A N/A N/A LOG debug N/A fetched proxy : %s proxy N/A proxy N/A N/A