Module Import alias Import alias Import alias Import alias Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias Import alias Import alias identifier Import alias identifier Assign NameStore Call AttributeLoad NameLoad attr ClassDef bases AttributeLoad NameLoad attr body FunctionDef arguments args NameParam defaults body Assign NameStore Str Assign NameStore Str If CompareIn NameLoad AttributeLoad NameLoad attr body Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index NameLoad If CompareIsNot NameLoad Str body Assign NameStore SubscriptLoad NameLoad Slice Num AugAssignAdd NameStore BinOpAdd Str NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad orelse Expr Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Return Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad NameLoad attr NameLoad AttributeLoad Call AttributeLoad NameLoad attr attr keyword Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad Num Num Num body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str NameLoad NameLoad keyword BinOpDiv BinOpMult Num Num NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword Call AttributeLoad NameLoad attr NameLoad TryFinally body Assign NameStore BinOpAdd BinOpAdd Str Call AttributeLoad Str attr ListComp BinOpMod Str TupleLoad NameLoad NameLoad comprehension NameStore Call NameLoad Num BinOpAdd NameLoad Num Str Assign NameStore BinOpAdd NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Num Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign TupleStore NameStore NameStore NameStore AttributeLoad AttributeLoad NameLoad attr attr If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str keyword Num keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpAdd Str Call NameLoad NameLoad BinOpSub NameLoad NameLoad NameLoad orelse Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpAdd Str Call NameLoad NameLoad BinOpDiv BinOpSub NameLoad NameLoad BinOpDiv BinOpDiv BinOpDiv NameLoad Num Num Num NameLoad finalbody Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpMod Str BinOpDiv BinOpMult BinOpMult NameLoad Num Num Num Assign NameStore Call AttributeLoad NameLoad attr Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str NameLoad NameLoad keyword Num keyword BinOpDiv BinOpMult Num Num NameLoad keyword NameLoad keyword NameLoad keyword NameLoad TryFinally body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Num Assign NameStore BinOpSub Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad finalbody Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num Num Num Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num Num Num Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num Num Num Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num Num Num Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num Num Num Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Num keyword BinOpMult Num Num keyword Num keyword NameLoad keyword Num keyword NameLoad keyword Num keyword Call AttributeLoad NameLoad attr NameLoad TryFinally body Assign TupleStore NameStore NameStore NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str keyword Num keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Str BinOpSub Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str keyword Num keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Str BinOpSub Call AttributeLoad NameLoad attr NameLoad NameLoad finalbody Expr Call AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Num decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr BinOpMult BinOpMult Num Num Num decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr BinOpMult BinOpMult Num Num Num keyword Num keyword Num keyword Num decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr BinOpMult BinOpMult Num Num Num keyword Num keyword Num keyword Num decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults BinOpMult Num Num Num BinOpMult Num Num Num body Assign NameStore Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpAdd Str Call NameLoad Call AttributeLoad NameLoad attr TryFinally body Assign NameStore Call AttributeLoad NameLoad attr Str Call NameLoad NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword Num keyword NameLoad keyword Num keyword Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Str Str Str keyword Num Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Assert CompareEq Call AttributeLoad Call NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad finalbody TryExcept body Expr Call AttributeLoad Call NameLoad attr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr decorator_list decorator_list If CompareEq NameLoad Str body Import alias identifier Expr Call AttributeLoad NameLoad attr ListLoad NameLoad NameLoad
N/A N/A unittest N/A time N/A logging N/A traceback N/A ufora.native.Cumulus CumulusNative N/A ufora.FORA.python.FORA FORA N/A ufora.cumulus.test.InMemoryCumulusSimulation InMemoryCumulusSimulation N/A ufora.distributed.S3.InMemoryS3Interface InMemoryS3Interface N/A ufora.distributed.S3.ActualS3Interface ActualS3Interface N/A ufora.native.CallbackScheduler CallbackScheduler N/A ufora.test.PerformanceTestReporter PerformanceTestReporter N/A ufora.FORA.python.Runtime Runtime N/A uuid N/A os N/A ufora.distributed.Storage.S3ObjectStore S3ObjectStore N/A ufora.config.Setup Setup N/A callbackScheduler N/A N/A CallbackScheduler singletonForTesting BigboxPerformanceTest N/A N/A unittest TestCase N/A getTestDataBucket N/A N/A self N/A N/A N/A aws_az_key AWS_AVAILABILITY_ZONE N/A bucketName ufora-test-data N/A N/A aws_az_key N/A os environ N/A N/A az N/A N/A os environ N/A aws_az_key N/A N/A az N/A N/A region N/A az N/A -1 N/A bucketName N/A - region N/A N/A N/A logging info Resolved az: %s, region: %s az region N/A N/A N/A N/A logging info No availability zone resolved N/A bucketName N/A computeUsingSeveralWorkers N/A N/A self N/A args kwds N/A N/A N/A N/A InMemoryCumulusSimulation computeUsingSeveralWorkers args kwds N/A createObjectStore N/A N/A s3Service N/A N/A N/A N/A N/A S3ObjectStore S3ObjectStore s3Service N/A N/A N/A Setup config userDataS3Bucket prefix test_object_cache/ N/A staticmethod downloadTaxiData N/A N/A self filecount parse workers threadsPerWorker downloaderThreads N/A False 1 30 8 N/A N/A s3 N/A N/A ActualS3Interface ActualS3InterfaceFactory N/A bucketName N/A N/A self getTestDataBucket N/A N/A result simulation N/A N/A self computeUsingSeveralWorkers 1+1 s3 workers memoryLimitMb N/A N/A 45 1024 workers threadCount threadsPerWorker returnSimulation True ioTaskThreadOverride downloaderThreads useInMemoryCache False objectStore N/A N/A self createObjectStore s3 N/A N/A N/A dsText N/A N/A let ds =  N/A N/A + join N/A N/A datasets.s3("%s", "taxi_month_%s.csv") N/A bucketName ix N/A ix N/A range 1 N/A filecount 1 ; N/A text N/A dsText (ds, ds.sum(), size(ds)) N/A downloadTimeStart N/A N/A time time N/A result N/A N/A simulation compute text timeout 240 N/A N/A N/A self assertTrue N/A N/A result isResult N/A downloadTimeEnd N/A N/A time time N/A N/A ds dsSum bytecount N/A N/A result asResult result N/A parse N/A N/A parseTimeStart N/A N/A time time N/A result N/A N/A simulation compute size(parsing.csv(ds)) timeout 240 ds ds N/A parseTimeEnd N/A N/A time time N/A N/A N/A self assertTrue N/A N/A result isResult N/A N/A N/A PerformanceTestReporter recordTest N/A python.BigBox.LargeS3.ParseTaxidata. N/A str filecount N/A parseTimeEnd parseTimeStart None N/A N/A bytecount N/A bytecount pyval N/A N/A N/A PerformanceTestReporter recordTest N/A python.BigBox.LargeS3.TaxiSecondsPerGB. N/A str filecount N/A N/A downloadTimeEnd downloadTimeStart N/A N/A N/A bytecount 1024 1024.0 1024.0 None N/A N/A N/A N/A simulation teardown N/A test_parseTaxiData_1 N/A N/A self N/A N/A N/A N/A N/A self downloadTaxiData 1 parse True N/A test_downloadTaxiData_1 N/A N/A self N/A N/A N/A N/A N/A self downloadTaxiData 1 N/A test_downloadTaxiData_2 N/A N/A self N/A N/A N/A N/A N/A self downloadTaxiData 2 N/A test_downloadTaxiData_6 N/A N/A self N/A N/A N/A N/A N/A self downloadTaxiData 6 N/A dataCreationTest N/A N/A self totalMB workers threadsPerWorker testName N/A N/A N/A s3 N/A N/A InMemoryS3Interface InMemoryS3InterfaceFactory N/A text N/A size(Vector.range(%s, {_*_})) N/A N/A N/A totalMB 1024 1024 8 N/A t0 N/A N/A time time N/A N/A result simulation N/A N/A self computeUsingSeveralWorkers 1+1 s3 workers timeout 120 memoryLimitMb N/A N/A 55 1024 workers threadCount threadsPerWorker returnSimulation True useInMemoryCache False N/A N/A N/A t0 N/A N/A time time N/A result N/A N/A simulation compute text timeout 120 N/A totalTimeToReturnResult N/A N/A N/A time time t0 N/A N/A N/A PerformanceTestReporter recordTest testName totalTimeToReturnResult None N/A N/A N/A N/A simulation teardown N/A test_createData_4000_1_30_threads N/A N/A self N/A N/A N/A N/A N/A self dataCreationTest 4000 1 30 python.BigBox.VectorRange.4000_MB_1Worker_30_threads N/A test_createData_4000_1_2_threads N/A N/A self N/A N/A N/A N/A N/A self dataCreationTest 4000 1 2 python.BigBox.VectorRange.4000_MB_1Worker_2_threads N/A test_createData_40000_1_30_threads N/A N/A self N/A N/A N/A N/A N/A self dataCreationTest 40000 1 30 python.BigBox.VectorRange.40000_MB_1Worker_30_threads N/A test_createData_40000_1_30_threads_2 N/A N/A self N/A N/A N/A N/A N/A self dataCreationTest 40000 1 30 python.BigBox.VectorRange.40000_MB_1Worker_30_threads_2 N/A test_createData_40000_1_2_threads N/A N/A self N/A N/A N/A N/A N/A self dataCreationTest 40000 1 2 python.BigBox.VectorRange.40000_MB_1Worker_2_threads N/A test_byteToStringAndBackInDifferentPatterns N/A N/A self N/A N/A N/A s3 N/A N/A ActualS3Interface ActualS3InterfaceFactory N/A setupText 
            let ds = Vector.range(3000000000, {UInt8(_%100)});

            let dat = Vector.range(100, fun(block) {
                Vector.range(1000000, fun(o) { let base = block * 10000000 + o * 10; (base, base + 10) })
                });

            (ds, dat, dat.sum())
             N/A N/A setupResults simulation N/A N/A self computeUsingSeveralWorkers setupText s3 1 memoryLimitMb N/A 45 1024 threadCount 30 returnSimulation True ioTaskThreadOverride 8 useInMemoryCache False timeout 30 objectStore N/A N/A self createObjectStore s3 N/A N/A N/A N/A ds dat datSum N/A N/A setupResults asResult result N/A t0 N/A N/A time time N/A result N/A N/A simulation compute size(datSum ~~ { ds[_[0],_[1]].dataAsString }) == size(datSum) timeout 120 ds ds dat dat datSum datSum N/A N/A N/A PerformanceTestReporter recordTest python.BigBox.DataAsString.FlatVector N/A N/A N/A time time t0 None N/A t0 N/A N/A time time N/A result N/A N/A simulation compute size(dat ~~ {_ ~~ { ds[_[0],_[1]].dataAsString } }) == size(dat) timeout 120 ds ds dat dat datSum datSum N/A N/A N/A PerformanceTestReporter recordTest python.BigBox.DataAsString.NestedVector N/A N/A N/A time time t0 None N/A N/A N/A N/A simulation teardown N/A N/A N/A PerformanceTestReporter PerfTest python.BigBox.DataAsString.byteToStringAndBack test_write_to_s3_3mb N/A N/A self N/A N/A N/A N/A N/A self writeToS3Test 3000000 N/A N/A N/A PerformanceTestReporter PerfTest python.BigBox.WriteToS3.3MB test_write_to_s3_50mb N/A N/A self N/A N/A N/A N/A N/A self writeToS3Test N/A N/A 50 1000 1000 N/A N/A N/A PerformanceTestReporter PerfTest python.BigBox.WriteToS3.50MB test_write_to_s3_200mb_multibox N/A N/A self N/A N/A N/A N/A N/A self writeToS3Test N/A N/A 200 1000 1000 workers 4 memoryLimitMb 100 threadCount 1 N/A N/A N/A PerformanceTestReporter PerfTest python.BigBox.WriteToS3.200MB test_write_to_s3_2gb_multibox N/A N/A self N/A N/A N/A N/A N/A self writeToS3Test N/A N/A 1000 1000 1000 workers 4 memoryLimitMb 1000 threadCount 2 N/A N/A N/A PerformanceTestReporter PerfTest python.BigBox.WriteToS3.2GB writeToS3Test N/A N/A self bytecount pageSizeOverride workers memoryLimitMb threadCount N/A N/A 1024 1024 1 N/A 45 1024 30 N/A N/A text Vector.range(__bytecount__, {UInt8(_%100)}).paged N/A s3 N/A N/A ActualS3Interface ActualS3InterfaceFactory N/A keyGuid N/A bigbox-test-key- N/A str N/A N/A uuid uuid4 N/A N/A N/A setupText N/A N/A text replace __bytecount__ N/A str bytecount N/A N/A setupResults simulation N/A N/A self computeUsingSeveralWorkers setupText s3 workers memoryLimitMb memoryLimitMb threadCount threadCount returnSimulation True ioTaskThreadOverride 8 useInMemoryCache False timeout 30 objectStore N/A N/A self createObjectStore s3 pageSizeOverride pageSizeOverride N/A result N/A N/A simulation executeExternalIoTask N/A N/A N/A CumulusNative ExternalIoTask WriteCharBigvecToS3 N/A N/A N/A N/A setupResults asResult result getVectorBigvecGuid N/A N/A CumulusNative S3KeyAndCredentials N/A N/A self getTestDataBucket keyGuid timeout 60 N/A N/A N/A self assertTrue N/A N/A result isSuccess result N/A N/A N/A N/A N/A s3 getKeySize N/A N/A self getTestDataBucket keyGuid bytecount N/A N/A N/A N/A N/A N/A N/A s3 deleteKey N/A N/A self getTestDataBucket keyGuid N/A N/A N/A N/A N/A N/A logging warn Failed to cleanup the test key: %s N/A N/A traceback format_exc N/A N/A N/A N/A __name__ __main__ N/A N/A ufora.config.Mainline Mainline N/A N/A N/A Mainline UnitTestMainline N/A FORA Runtime