Module ImportFrom alias ImportFrom alias ImportFrom alias Import alias ImportFrom alias Import alias identifier ImportFrom alias alias ImportFrom alias alias alias ImportFrom alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Num Assign NameStore Num Assign NameStore Str Assign NameStore NameLoad FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad If AttributeLoad AttributeLoad NameLoad attr attr body Assign NameStore SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Num orelse Expr Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad BinOpAdd NameLoad Num Assign NameStore ListLoad Str Assign NameStore Call NameLoad keyword Str keyword Str keyword Str keyword Str keyword NameLoad keyword NameLoad keyword Num keyword Num Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr Str Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str Return NameLoad decorator_list ClassDef bases body Expr Str FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr Str Str attr Str Str attr Str Str attr Str Str If CompareLtE Call NameLoad NameLoad NameLoad body Return NameLoad orelse Return BinOpAdd BinOpAdd SubscriptLoad NameLoad Slice Call NameLoad BinOpDiv NameLoad Num Str SubscriptLoad NameLoad Slice BinOpSub UnaryOpUSub Call NameLoad BinOpDiv NameLoad Num Num decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Return Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr BinOpDiv NameLoad Num decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Dict Assign AttributeStore NameLoad attr Dict Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad Assign NameStore Call NameLoad Assign NameStore Call NameLoad NameLoad Str keyword NameLoad Assign NameStore Call NameLoad NameLoad keyword NameLoad keyword NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore Call NameLoad Num body TryExcept body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword Num Expr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad body Break Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Index Str If CompareEq SubscriptLoad NameLoad Index Str Str body Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str orelse If CompareEq SubscriptLoad NameLoad Index Str Str body If CompareIn NameLoad AttributeLoad NameLoad attr body Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad TupleLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Delete SubscriptDel AttributeLoad NameLoad attr Index NameLoad orelse Pass orelse Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign NameStore SubscriptLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Index Num If UnaryOpNot BoolOpOr AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Return Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad For TupleStore NameStore TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr NameLoad Slice Num Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore ListComp SubscriptLoad NameLoad Index NameLoad comprehension NameStore NameLoad If AttributeLoad NameLoad attr body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr NameLoad Slice Num Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore ListComp SubscriptLoad NameLoad Index NameLoad comprehension NameStore NameLoad Assign NameStore SubscriptLoad NameLoad Slice AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Slice AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Slice AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign AttributeStore AttributeLoad NameLoad attr attr NameLoad For NameStore AttributeLoad NameLoad attr body If CompareEq Call NameLoad NameLoad NameLoad body Assign AttributeStore NameLoad attr Str Break Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr Str Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr Str Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr Str Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr Str Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr Str Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr Str If AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr NameLoad If AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr NameLoad Assign NameStore Call NameLoad Call NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Assign AttributeStore NameLoad attr Call NameLoad keyword Str keyword Str keyword Str keyword Str keyword Str keyword Num Expr Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr If BoolOpAnd AttributeLoad NameLoad attr NameLoad body Assign NameStore SubscriptLoad NameLoad Slice AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Slice AttributeLoad NameLoad attr Assign NameStore Call NameLoad Call NameLoad keyword NameLoad keyword NameLoad Assign AttributeStore NameLoad attr Call NameLoad keyword Str keyword Str keyword Num keyword Str keyword Str keyword Num Expr Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr keyword NameLoad decorator_list decorator_list
N/A __future__ division __future__ print_function datetime datetime N/A json Queue Empty N/A numpy np bokeh.plotting figure output_server kombu Exchange Connection Queue bokeh.models.glyphs Segment bokeh.models.markers Circle bokeh.models Range1d ColumnDataSource bokeh.models.grids Grid bokeh.embed autoload_server bokeh.session Session bokeh.document Document N/A DEFAULT_NUM_URLS 25 N/A URL_CHAR_WIDTH 50 N/A EXCHANGE_NAME fetcher_log N/A PLOT_CIRCLES False init_plot N/A N/A crawl_name N/A N/A N/A session N/A Session N/A document N/A Document N/A N/A N/A session use_doc crawl_name N/A N/A N/A session load_document document N/A N/A N/A document context children N/A N/A plot N/A N/A N/A document context children N/A 0 N/A N/A N/A output_server crawl_name N/A current N/A N/A np datetime64 N/A N/A datetime now N/A xdr N/A Range1d current N/A current 1 N/A ydr N/A urls N/A plot N/A figure title Crawler Monitor tools hover x_axis_type datetime y_axis_location right x_range xdr y_range ydr width 1200 height 600 N/A N/A plot toolbar_location None N/A N/A N/A plot xgrid grid_line_color None N/A N/A N/A plot ygrid grid_line_color None N/A N/A N/A plot xaxis minor_tick_line_color None N/A N/A N/A plot xaxis major_tick_line_color None N/A N/A N/A plot xaxis major_label_text_font_size 0pt N/A N/A N/A plot yaxis minor_tick_line_color None N/A N/A N/A plot yaxis major_tick_line_color None N/A N/A N/A plot yaxis major_label_text_font_size 0pt N/A N/A N/A document add plot N/A N/A N/A session store_document document N/A script N/A autoload_server plot session N/A script N/A N/A script replace 'modelid': u' 'modelid': ' N/A script N/A NutchUrlTrails N/A N/A N/A 
    Class for managing URL Trails visualizations
     strip_url N/A N/A url N/A N/A N/A 
        Make a URL safe for visualization in Bokeh server
        :param url: a URL to be shortened/stripped
        :return: The stripped URL
         N/A stripped_url N/A N/A N/A N/A N/A N/A N/A N/A url replace https:// replace http:// replace : _ replace - _ N/A N/A N/A len stripped_url URL_CHAR_WIDTH N/A N/A stripped_url N/A N/A N/A N/A N/A stripped_url N/A N/A int N/A URL_CHAR_WIDTH 2 ... N/A stripped_url N/A N/A N/A N/A int N/A URL_CHAR_WIDTH 2 3 N/A staticmethod jtime_to_datetime N/A N/A t N/A N/A N/A 
        Convert a Java-format Epoch time stamp into np.datetime64 object
        :param t: Java-format Epoch time stamp (milliseconds)
        :return: A np.datetime64 scalar
         N/A N/A N/A np datetime64 N/A N/A datetime fromtimestamp N/A t 1000.0 N/A staticmethod __init__ N/A N/A self crawl_name num_urls N/A DEFAULT_NUM_URLS N/A N/A 
        Create a NutchUrlTrails instance for visualizing a running Nutch crawl in real-time using Bokeh
        :param name: The name of the crawl (as identified by the queue)
        :param num_urls: The number of URLs to display in the visualization
        :return: A NutchUrLTrails instance
         N/A N/A self crawl_name crawl_name N/A N/A self num_urls num_urls N/A N/A self open_urls N/A N/A N/A self closed_urls N/A N/A N/A self old_segments None N/A N/A self old_circles None N/A N/A self session N/A Session N/A N/A N/A N/A self session use_doc N/A self crawl_name N/A N/A self document N/A Document N/A con N/A Connection N/A exchange N/A Exchange EXCHANGE_NAME direct durable False N/A queue N/A Queue crawl_name exchange exchange routing_key crawl_name N/A N/A self queue N/A N/A con SimpleQueue name queue N/A handle_messages N/A N/A self N/A N/A N/A 
        Get and parse up to 250 messages from the queue then plot.  Break early if less.
         N/A i N/A range 250 N/A N/A N/A N/A m N/A N/A N/A self queue get block True timeout 1 N/A N/A N/A self parse_message m N/A N/A N/A Empty N/A N/A N/A N/A N/A self plot_urls N/A parse_message N/A N/A self message N/A N/A N/A 
        Parse a single message arriving from the queue.  Updates list of open/closed urls.
        :param message: A message from the queue
         N/A N/A print N/A message body N/A message N/A N/A json loads N/A message body N/A url N/A message N/A url N/A N/A N/A message N/A eventType START N/A N/A N/A N/A self open_urls N/A url N/A N/A NutchUrlTrails jtime_to_datetime N/A message N/A timestamp N/A N/A N/A N/A message N/A eventType END N/A N/A N/A url N/A self open_urls N/A N/A N/A N/A self closed_urls N/A url N/A N/A N/A self open_urls N/A url N/A N/A NutchUrlTrails jtime_to_datetime N/A message N/A timestamp N/A N/A N/A self open_urls N/A url N/A N/A N/A N/A N/A Exception Unexpected message type N/A plot_urls N/A N/A self N/A N/A N/A 
        Visualize crawler activity by showing the most recently crawled URLs and the fetch time.
         N/A N/A N/A N/A self session load_document N/A self document N/A plot N/A N/A N/A N/A self document context children N/A 0 N/A N/A N/A N/A self open_urls N/A self closed_urls N/A N/A N/A x0 N/A N/A x N/A N/A urls N/A N/A circles N/A N/A circle_urls N/A N/A current_time N/A N/A np datetime64 N/A N/A datetime now N/A N/A url start_t N/A N/A N/A self open_urls items N/A N/A url N/A N/A NutchUrlTrails strip_url url N/A N/A N/A x0 append start_t N/A N/A N/A x append current_time N/A N/A N/A urls append url N/A N/A url N/A start_t end_t N/A N/A N/A self closed_urls items N/A N/A url N/A N/A NutchUrlTrails strip_url url N/A N/A N/A x0 append start_t N/A N/A N/A x append end_t N/A N/A N/A circles append end_t N/A N/A N/A urls append url N/A N/A N/A circle_urls append url N/A x0 N/A N/A np asarray x0 N/A x N/A N/A np asarray x N/A circles N/A N/A np asarray circles N/A sort_index N/A N/A N/A np argsort x0 N/A -1 N/A x0 N/A x0 N/A sort_index N/A x N/A x N/A sort_index N/A urls N/A N/A urls N/A i N/A i sort_index N/A N/A self closed_urls N/A N/A circle_sort_index N/A N/A N/A np argsort circles N/A -1 N/A circles N/A circles N/A circle_sort_index N/A circle_urls N/A N/A circle_urls N/A i N/A i circle_sort_index N/A active_x0 N/A x0 N/A N/A self num_urls N/A active_x N/A x N/A N/A self num_urls N/A active_urls N/A urls N/A N/A self num_urls N/A min_x N/A min active_x0 N/A N/A N/A plot x_range start min_x N/A N/A N/A plot x_range end N/A N/A np datetime64 N/A N/A datetime now N/A N/A N/A plot y_range factors active_urls N/A r N/A plot renderers N/A N/A N/A N/A type r Grid N/A N/A N/A r grid_line_color black N/A N/A N/A N/A N/A plot right N/A 0 minor_tick_line_color black N/A N/A N/A N/A plot right N/A 0 major_tick_line_color black N/A N/A N/A N/A plot right N/A 0 major_label_text_font_size 12pt N/A N/A N/A N/A plot below N/A 0 minor_tick_line_color black N/A N/A N/A N/A plot below N/A 0 major_tick_line_color black N/A N/A N/A N/A plot below N/A 0 major_label_text_font_size 12pt N/A N/A self old_circles N/A N/A N/A N/A N/A plot renderers pop N/A N/A self old_circles None N/A N/A self old_segments N/A N/A N/A N/A N/A plot renderers pop N/A N/A self old_segments None N/A segment_source N/A ColumnDataSource N/A dict x0 active_x0 x1 active_x urls active_urls N/A N/A self old_segments N/A Segment x0 x0 y0 urls x1 x1 y1 urls line_color orange line_width 10 N/A N/A N/A plot add_glyph segment_source N/A self old_segments N/A N/A N/A self closed_urls PLOT_CIRCLES N/A N/A active_circles N/A circles N/A N/A self num_urls N/A active_circle_urls N/A circle_urls N/A N/A self num_urls N/A circle_source N/A ColumnDataSource N/A dict x active_circles urls active_circle_urls N/A N/A self old_circles N/A Circle x x y urls size 12 fill_color green line_color orange line_width 2 N/A N/A N/A plot add_glyph circle_source N/A self old_circles N/A N/A N/A N/A self session store_document N/A self document dirty_only False N/A N/A