simulation-pipeline 0.0.1
Simulation Pipeline Python Package
Requires Python: >=3.6

# Simulation Pipeline Python Package


## Note
This is the python package for simulation pipeline. It's specifically designed for Krylov only.  
This package aims at helping data scientists traverse through training sets and extract data with more ease.

# Submit simulation job via krylov

## Create artifact
`krylovctl --verbose artifact create`

## Upload artifact
`krylovctl artifact upload ebay artifact-zhijzhang --path=artifact-zhijzhang.tar.gz --force`

## Submit job via REST API
* Request

`POST https://master.krylov-prod.svc.94.tess.io/api/core/v1/namespaces/ebay/jobs/pySparkTest2`
```json
{
    "jobName": "pySparkTest2",
    "artifact": "artifact-zhijzhang",
    "artifactTag": "latest",
    "mlApplication": "com.ebay.oss.krylov.workflow.app.GenericMLApplication",
    "executionParameters": {
        "image": "ecr.vip.ebayc3.com/krylov/krylov-workspace-jupyter:cuda10.1-cudnn7-py3.7-ubuntu18.04-base",
        "cpu": 4,
        "memory": 8,
        "hadoop": {
            "batchUser": "b_riskmgmt_insights",
            "hadoopCluster": "apollo-rno"
        }
    },
    "applicationParameters": {
        "tasks": {
            "spark_task": {
                "program": "com.ebay.oss.krylov.workflow.PythonProgram",
                "parameters": {
                    "file": "src/SimulationPipeline/simulation_krylov_task.py",
                    "args": [
                        "--simulation_jar",
                        "hdfs://apollo-rno/user/b_riskmgmt_insights/simulation/offline-simulation-pipeline-1.0.8-SNAPSHOT.jar",
                        "--arg_file_path",
                        "viewfs://apollo-rno/user/b_lms/simulation/argfile/20211206/argFile_68.json",
                        "--driver_set_file_path",
                        "hdfs://apollo-rno/user/b_riskmgmt_insights/yafan/upload/driver_set_poc_v3.csv",
                        "--driver_set_key_type",
                        "PurchaseOrderId",
                        "--keyspace_mock_file",
                        "hdfs://apollo-rno/user/b_riskmgmt_insights/yafan/upload/key_space_poc_mock.csv",
                        "--feature_set_file_path",
                        "hdfs://apollo-rno/user/b_riskmgmt_insights/yafan/temp_file/featureSet_1638847084879.txt"
                    ]
                }
            }
        }
    }
}
```
* Response
```json
{
    "message": "Job submitted successfully",
    "runId": "b0838db9-8185-4b08-9a55-668cafe01928"
}
```

## Get Krylov job status via REST API
* Request

`GET https://master.krylov-prod.svc.94.tess.io/api/core/v1/runs/b0838db9-8185-4b08-9a55-668cafe01928`

* Response
```json
{
    "runId": "b0838db9-8185-4b08-9a55-668cafe01928",
    "status": "completed",
    "jobName": "pySparkTest2",
    "namespace": "ebay",
    "artifact": "artifact-zhijzhang",
    "artifactTag": "latest",
    "submittedBy": "s_caucasus",
    "creationTime": 1639362104349,
    "lastUpdatedTime": 1639362498713,
    "duration": 394364,
    "actionHistory": [
        {
            "action": "Submit",
            "submittedBy": "s_caucasus",
            "submittedTime": 1639362104344
        }
    ],
    "stateHistory": [
        {
            "state": "inactive",
            "startTime": 1639362104344,
            "endTime": 1639362104399
        },
        {
            "state": "pending",
            "startTime": 1639362104399,
            "endTime": 1639362127926
        },
        {
            "state": "active",
            "startTime": 1639362127926,
            "endTime": 1639362498713
        },
        {
            "state": "completed",
            "startTime": 1639362498713,
            "endTime": 1639362498713
        }
    ],
    "tasks": [
        {
            "taskName": "spark_task",
            "status": "completed",
            "creationTime": 1639362127607,
            "lastUpdatedTime": 1639362463740,
            "duration": 336133,
            "executionParams": {
                "image": "ecr.vip.ebayc3.com/krylov/krylov-workspace-jupyter:cuda10.1-cudnn7-py3.7-ubuntu18.04-base",
                "cpu": 4,
                "memory": 8,
                "hadoop": {
                    "batchUser": "b_riskmgmt_insights",
                    "hadoopCluster": "apollo-rno"
                }
            },
            "predecessors": [
                "Not Applicable"
            ],
            "attempts": [
                {
                    "attempt": 1,
                    "status": "completed",
                    "creationTime": 1639362127607,
                    "lastUpdatedTime": 1639362463740,
                    "message": "Workflow task completed successfully.",
                    "stateHistory": [
                        {
                            "state": "inactive",
                            "startTime": 1639362127607,
                            "endTime": 1639362131108
                        },
                        {
                            "state": "pending",
                            "startTime": 1639362131108,
                            "endTime": 1639362177285
                        },
                        {
                            "state": "active",
                            "startTime": 1639362177285,
                            "endTime": 1639362463740
                        },
                        {
                            "state": "completed",
                            "startTime": 1639362463740,
                            "endTime": 1639362463740
                        }
                    ]
                }
            ]
        }
    ]
}
```

## Reference
https://94.aihub.krylov.vip.ebay.com/documentation/train-and-tune/serviceaccount