Module Expr Str Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias alias Assign NameStore Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Str Str Str FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad NameLoad keyword Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Index Num TryExcept body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num handlers ExceptHandler type NameLoad body Assign NameStore Num Return TupleLoad NameLoad TupleLoad Num NameLoad BinOpPow NameLoad Num NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameLoad Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameLoad Assign NameStore Call NameLoad TupleLoad NameLoad NameLoad Assign NameStore Call NameLoad TupleLoad NameLoad NameLoad Assign NameStore BinOpAdd NameLoad NameLoad Assign NameStore BinOpAdd NameLoad NameLoad Assign NameStore BinOpAdd NameLoad NameLoad Return TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign TupleStore NameStore TupleStore NameStore NameStore NameStore NameStore NameStore NameLoad TryExcept body Assign NameStore BinOpDiv NameLoad Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr BinOpSub BinOpDiv BinOpSub NameLoad BinOpDiv BinOpPow NameLoad Num NameLoad NameLoad Num Return TupleLoad NameLoad TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad handlers ExceptHandler type NameLoad body Return TupleLoad NameLoad TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str decorator_list If CompareEq NameLoad Str body Assign NameStore Call AttributeLoad Call NameLoad attr NameLoad Assign NameStore Call NameLoad keyword NameLoad Expr Call NameLoad NameLoad
N/A N/A 
A Spark application that computes the summary statistics of arrival delays.
 N/A os N/A sys N/A csv N/A math functools partial StringIO StringIO pyspark SparkConf SparkContext N/A APP_NAME Summary Statistics of Arrival Delay by Airport N/A DATASET N/A N/A N/A os path join N/A N/A N/A os path dirname __file__ .. data ontime flights.csv counters N/A N/A line N/A N/A N/A 
    Splits the line on a CSV and parses it into the key and summary counters.
    A counter is as follows: (count, total, square, minimum, maximum).
     N/A reader N/A N/A csv reader N/A StringIO line delimiter , N/A row N/A N/A reader next N/A airport N/A row N/A 4 N/A N/A N/A delay N/A float N/A row N/A 8 N/A N/A N/A ValueError N/A N/A delay 0.0 N/A N/A airport N/A 1 delay N/A delay 2 delay delay N/A aggregation N/A N/A item1 item2 N/A N/A N/A 
    For an (airport, counters) item, perform summary aggregations.
     N/A N/A count1 total1 squares1 min1 max1 item1 N/A N/A count2 total2 squares2 min2 max2 item2 N/A minimum N/A min N/A min1 min2 N/A maximum N/A max N/A max1 max2 N/A count N/A count1 count2 N/A total N/A total1 total2 N/A squares N/A squares1 squares2 N/A N/A count total squares minimum maximum N/A summary N/A N/A aggregate N/A N/A N/A 
    Compute summary statistics from aggregation.
     N/A N/A airport N/A count total square minimum maximum aggregate N/A N/A N/A mean N/A total N/A float count N/A stddev N/A N/A math sqrt N/A N/A N/A square N/A N/A total 2 count count 1 N/A N/A airport N/A count mean stddev minimum maximum N/A N/A N/A Exception N/A N/A N/A airport N/A count None None minimum maximum N/A main N/A N/A sc N/A N/A N/A 
    Primary analysis mechanism for Spark application
     N/A delays N/A N/A N/A N/A sc textFile DATASET map counters N/A delays N/A N/A delays reduceByKey aggregation N/A delays N/A N/A delays map summary N/A N/A N/A delays saveAsTextFile delays-summary N/A N/A N/A __name__ __main__ N/A N/A conf N/A N/A N/A SparkConf setAppName APP_NAME N/A sc N/A SparkContext conf conf N/A N/A main sc