Module Import alias Import alias Import alias Import alias Import alias ImportFrom alias alias ClassDef bases body Expr Str Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd NameLoad Str NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults Str Str Num Str Str Str body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Call NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword Str keyword NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr With AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr keyword Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore NameLoad Expr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr While BoolOpAnd UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr CompareLt BinOpSub Call AttributeLoad NameLoad attr NameLoad NameLoad UnaryOpNot AttributeLoad NameLoad attr body TryFinally body TryExcept body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam defaults body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str handlers ExceptHandler body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Return Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list Assign AttributeStore NameLoad attr ListLoad NameLoad While NameLoad body TryExcept body Expr Call AttributeLoad NameLoad attr keyword Num Break handlers ExceptHandler type AttributeLoad NameLoad attr body Pass ExceptHandler type AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad Assign AttributeStore NameLoad attr NameLoad If BoolOpOr Call AttributeLoad AttributeLoad NameLoad attr attr CompareGtE BinOpSub Call AttributeLoad NameLoad attr NameLoad NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Break handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Num ExceptHandler body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Num finalbody Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str While NameLoad body With AttributeLoad NameLoad attr body If CompareEq Call NameLoad AttributeLoad NameLoad attr Num body Break Expr Call AttributeLoad NameLoad attr Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpMult Num Num Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr While UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr body TryExcept body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad Str attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad With Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameStore body With Call AttributeLoad NameLoad attr NameLoad NameStore body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad Str attr NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults body FunctionDef arguments args NameParam defaults body If BoolOpOr Call AttributeLoad Call AttributeLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str attr attr Str SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str UnaryOpNot CompareIn Str SubscriptLoad SubscriptLoad NameLoad Index Str Index Str body Return Assign NameStore Dict Str Str Str Str SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str NameLoad NameLoad If CompareIn Str SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str body Assign SubscriptStore NameLoad Index Str SubscriptLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str Index Str orelse If CompareIn Str SubscriptLoad SubscriptLoad NameLoad Index Str Index Str body Assign NameStore SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str Assign SubscriptStore NameLoad Index Str SubscriptLoad NameLoad Slice Call AttributeLoad NameLoad attr Str orelse Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Assign SubscriptStore NameLoad Index Str Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr NameLoad NameLoad With AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Expr Call NameLoad NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad NameLoad NameLoad NameLoad TryFinally body TryExcept body Expr Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad Str attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr ExceptHandler body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad Str attr NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr finalbody Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args defaults body Expr Call NameLoad With AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr decorator_list Import alias Assign NameStore Call AttributeLoad Str attr AttributeLoad NameLoad attr Call AttributeLoad Str attr GeneratorExp Call AttributeLoad NameLoad attr Str comprehension NameStore Call NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad With AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad NameLoad attr decorator_list decorator_list
N/A N/A logging N/A time N/A threading N/A kombu N/A socket umbra.browser BrowserPool BrowsingException AmqpBrowserController N/A N/A N/A 
    Consumes amqp messages representing requests to browse urls, from the
    specified amqp queue (default: "urls") on the specified amqp exchange
    (default: "umbra"). Incoming amqp message is a json object with 3
    attributes:

      {
        "clientId": "umbra.client.123",
        "url": "http://example.com/my_fancy_page",
        "behaviorParameters": {"some":"parameter","another":"thing"},
        "metadata": {"arbitrary":"fields", "etc":4}
      }

    "url" is the url to browse.

    "clientId" uniquely identifies the client of umbra. Umbra uses the clientId
    as the amqp routing key, to direct information via amqp back to the client.
    It sends this information on the same specified amqp exchange (default:
    "umbra").

    "behaviorParameters" are used to populate the javascript behavior template.

    Each url requested in the browser is published to amqp this way. The
    outgoing amqp message is a json object:

      {
        "url": "http://example.com/images/embedded_thing.jpg",
        "method": "GET",
        "headers": {"User-Agent": "...", "Accept": "...", ...},
        "parentUrl": "http://example.com/my_fancy_page",
        "parentUrlMetadata": {"arbitrary":"fields", "etc":4, ...}
      }

    POST requests have an additional field, postData.
     N/A logger N/A N/A logging getLogger N/A N/A __module__ . __qualname__ __init__ N/A N/A self amqp_url chrome_exe max_active_browsers queue_name exchange_name routing_key N/A amqp://guest:guest@localhost:5672/%2f chromium-browser 1 urls umbra urls N/A N/A N/A self amqp_url amqp_url N/A N/A self queue_name queue_name N/A N/A self exchange_name exchange_name N/A N/A self routing_key routing_key N/A N/A self max_active_browsers max_active_browsers N/A N/A self _browser_pool N/A BrowserPool size max_active_browsers chrome_exe chrome_exe N/A start N/A N/A self N/A N/A N/A N/A self _browsing_threads N/A set N/A N/A self _browsing_threads_lock N/A N/A threading Lock N/A N/A self _exchange N/A N/A kombu Exchange name N/A self exchange_name type direct durable True N/A N/A self _reconnect_requested False N/A N/A self _producer None N/A N/A self _producer_lock N/A N/A threading Lock N/A N/A self _producer_lock N/A N/A N/A self _producer_conn N/A N/A kombu Connection N/A self amqp_url N/A N/A self _producer N/A N/A N/A self _producer_conn Producer serializer json N/A N/A self _consumer_thread N/A N/A threading Thread target N/A self _consume_amqp name AmqpConsumerThread N/A N/A self _consumer_stop N/A N/A threading Event N/A N/A N/A N/A self _consumer_thread start N/A shutdown N/A N/A self N/A N/A N/A N/A N/A N/A self logger info N/A N/A shutting down amqp consumer {} format N/A self amqp_url N/A N/A N/A N/A self _consumer_stop set N/A N/A N/A N/A self _consumer_thread join N/A shutdown_now N/A N/A self N/A N/A N/A N/A N/A N/A self _consumer_stop set N/A N/A N/A N/A self _browser_pool shutdown_now N/A N/A N/A N/A self _consumer_thread join N/A reconnect N/A N/A self N/A args kwargs N/A N/A N/A self _reconnect_requested True N/A N/A N/A N/A self _browser_pool shutdown_now N/A _wait_for_and_browse_urls N/A N/A self conn consumer timeout N/A N/A N/A start N/A N/A time time N/A browser None N/A N/A N/A consumer qos prefetch_count N/A self max_active_browsers N/A N/A N/A N/A N/A N/A self _consumer_stop is_set N/A N/A N/A N/A time time start timeout N/A N/A self _reconnect_requested N/A N/A N/A N/A N/A N/A browser N/A N/A N/A self _browser_pool acquire N/A N/A N/A browser start callback N/A N/A body message N/A N/A N/A N/A N/A client_id N/A N/A body get clientId N/A url N/A body N/A url N/A metadata N/A N/A body get metadata N/A behavior_parameters N/A N/A body get behaviorParameters N/A N/A N/A N/A N/A N/A N/A self logger error unable to decipher message %s message exc_info True N/A N/A N/A N/A self logger error discarding bad message N/A N/A N/A message reject N/A N/A N/A browser stop N/A N/A N/A N/A self _browser_pool release browser N/A N/A N/A N/A self _start_browsing_page browser message client_id url metadata behavior_parameters N/A N/A N/A consumer callbacks N/A callback N/A True N/A N/A N/A N/A N/A N/A conn drain_events timeout 0.5 N/A N/A N/A N/A N/A socket timeout N/A N/A N/A N/A N/A socket error N/A N/A N/A N/A N/A self logger error problem consuming messages from AMQP, will try reconnecting after active browsing finishes exc_info True N/A N/A self _reconnect_requested True N/A N/A N/A N/A N/A self _consumer_stop is_set N/A N/A N/A N/A time time start timeout N/A self _reconnect_requested N/A N/A N/A N/A browser stop N/A N/A N/A N/A self _browser_pool release browser N/A N/A N/A N/A KeyError N/A N/A N/A N/A time sleep 0.5 N/A N/A N/A N/A N/A N/A self logger critical problem with browser initialization exc_info True N/A N/A N/A time sleep 0.5 N/A N/A N/A consumer callbacks None N/A _wait_for_active_browsers N/A N/A self N/A N/A N/A N/A N/A N/A self logger info waiting for browsing threads to finish N/A True N/A N/A N/A self _browsing_threads_lock N/A N/A N/A N/A len N/A self _browsing_threads 0 N/A N/A N/A N/A N/A time sleep 0.5 N/A N/A N/A N/A self logger info active browsing threads finished N/A _consume_amqp N/A N/A self N/A N/A N/A RECONNECT_AFTER_SECONDS N/A 150 60 N/A url_queue N/A N/A kombu Queue N/A self queue_name exchange N/A self _exchange routing_key N/A self routing_key N/A N/A N/A N/A N/A self _consumer_stop is_set N/A N/A N/A N/A N/A N/A N/A self logger info N/A N/A connecting to amqp exchange={} at {} format N/A N/A self _exchange name N/A self amqp_url N/A N/A self _reconnect_requested False N/A N/A N/A kombu Connection N/A self amqp_url conn N/A N/A N/A N/A conn Consumer url_queue consumer N/A N/A N/A N/A self _wait_for_and_browse_urls conn consumer timeout RECONNECT_AFTER_SECONDS N/A N/A N/A self _wait_for_active_browsers N/A N/A N/A BaseException N/A e N/A N/A N/A N/A N/A self logger error N/A N/A caught exception {} format e exc_info True N/A N/A N/A time sleep 0.5 N/A N/A N/A N/A self logger error attempting to reopen amqp connection N/A _start_browsing_page N/A N/A self browser message client_id url parent_url_metadata behavior_parameters N/A N/A on_response N/A N/A chrome_msg N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A chrome_msg N/A params N/A response N/A url lower startswith data: N/A N/A N/A chrome_msg N/A params N/A response N/A fromDiskCache N/A N/A requestHeaders N/A N/A chrome_msg N/A params N/A response N/A N/A N/A payload N/A url headers parentUrl parentUrlMetadata N/A N/A N/A chrome_msg N/A params N/A response N/A url N/A N/A N/A chrome_msg N/A params N/A response N/A requestHeaders url parent_url_metadata N/A N/A :method N/A N/A N/A chrome_msg N/A params N/A response N/A requestHeaders N/A N/A N/A payload N/A method N/A N/A N/A N/A chrome_msg N/A params N/A response N/A requestHeaders N/A :method N/A N/A N/A requestHeadersText N/A N/A chrome_msg N/A params N/A response N/A N/A req N/A N/A N/A chrome_msg N/A params N/A response N/A requestHeadersText N/A N/A payload N/A method N/A req N/A N/A N/A req index N/A N/A N/A N/A N/A self logger warn unable to identify http method (assuming GET) chrome_msg=%s chrome_msg N/A N/A payload N/A method GET N/A N/A N/A N/A self logger debug sending to amqp exchange=%s routing_key=%s payload=%s N/A self exchange_name client_id payload N/A N/A self _producer_lock N/A N/A publish N/A N/A N/A self _producer_conn ensure N/A self _producer N/A N/A self _producer publish N/A N/A publish payload exchange N/A self _exchange routing_key client_id N/A browse_page_sync N/A N/A N/A N/A N/A N/A N/A N/A self logger info browser=%s client_id=%s url=%s behavior_parameters=%s browser client_id url behavior_parameters N/A N/A N/A N/A N/A N/A N/A browser browse_page url on_response on_response behavior_parameters behavior_parameters N/A N/A N/A message ack N/A N/A N/A BrowsingException N/A e N/A N/A N/A N/A N/A self logger warn N/A N/A browsing did not complete normally, requeuing url {} - {} format url e N/A N/A N/A message requeue N/A N/A N/A N/A N/A N/A self logger critical N/A N/A problem browsing page, requeuing url {}, may have lost browser process format url exc_info True N/A N/A N/A message requeue N/A N/A N/A N/A browser stop N/A N/A N/A N/A self _browser_pool release browser N/A browse_thread_run_then_cleanup N/A N/A N/A N/A N/A N/A browse_page_sync N/A N/A self _browsing_threads_lock N/A N/A N/A N/A N/A self _browsing_threads remove N/A N/A threading current_thread N/A N/A random N/A thread_name N/A N/A BrowsingThread{}-{} format N/A browser chrome_port N/A N/A join N/A N/A N/A random choice abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 N/A _ N/A range 6 N/A th N/A N/A threading Thread target browse_thread_run_then_cleanup name thread_name N/A N/A self _browsing_threads_lock N/A N/A N/A N/A N/A self _browsing_threads add th N/A N/A N/A th start N/A N/A