lisr 0.0.1
LisR
Requires Python: >= 3.9

# LisR Data Pipeline


## process 1 -- file ingestion, runs every 15 minutes from crontab
    rclone copy inbox raw/raw
    rclone lsl con1:constellr-lisr-raw/raw ==> get set A: {name, type, size, ts} identify type
    get set B from 'lisr_raw_file'
    insert (B - A) into lisr_raw_file with process_status='waiting'
    notify lisr_<type>

## process 2 - record processing :
    listen on lisr_record
    pick up 'waiting' type=record
    unpack tar in temp directory , catch tar stderr 
        * if tar didnt succeeded -- stop with status='failure', error_msg=stderr 
    rename untarred directory to record
    chmod +x record
    chmod +x record/v0
    read record/record_info.json
        {
        "sc": 0,
        "vc": 1,
        "record": 123,
        "timestamp": 1648212417018795740
        }
        * if not exists -- ERROR
    get record_id and timestamp, and the the whole json -> record_md 
        * if something missing -- ERROR
    get list of frames: ls -l record/v0/f*
    count num_of_frame_files in the list
    read version_info.json 
        {
        "sc": 0,
        "vc": 1,
        "record": 123,
        "version": 0,
        "frames": 5399,
        "timestamp": 1648212417019162372,
        "type": 32,
        "origin_version_id": 0,
        }
    make sure sc, vc, and record matche to record_md
        * if not -- ERROR
    get num_of_frames from "frames" field
        * if num_of_frame_files != num_of_frames -> ERROR
    !!! obtain list_of_frames = list of all f*.json in record/v0/info with frame sizes and timestamps
        * if len(the list of f*.json) != num_of_frames -> ERROR
    for frame in list_of_frames:
        read record/v0/info/<frame>.json
            {
            "sc": 0,
            "vc": 1,
            "record": 123,
            "version": 0,
            "frame": 100,
            "length": 55825,
            "width": 320,
            "height": 128,
            "pixel_format": 0,
            "endianess": 1,
            "image": {
                "sensor_w": 320,
                "sensor_h": 256,
                "offset_x": 0,
                "offset_y": 0,
                "timestamp": 1648212483804580129,
                "integration_time": 2250,
                "source_id": 0,
                "temperature": 0
            },
            "adcs": {
                "position_x": -1224927.877800,
                "position_y": -4459069.771200,
                "position_z": -4984862.572800,
                "velocity_x": 7017.646144,
                "velocity_y": 450.295194,
                "velocity_z": -2124.347355,
                "position_timestamp": 1648212480265625000,
                "position_valid": 1,
                "attitude_q0": 0.921950,
                "attitude_q1": -0.343474,
                "attitude_q2": 0.012096,
                "attitude_q3": 0.178573,
                "attitude_valid": 1,
                "body_rate_x": 0.000075,
                "body_rate_y": -0.001122,
                "body_rate_z": -0.000000,
                "body_rate_valid": 1,
                "iss_bad_1sec_status": 2,
                "iss_bad_10sec_status": 2,
                "time_error": -80748000
            }
            }
        * if record id doesnt match --> ERROR
        * if frame id doesnt match frame file -> ERROR
        * if frame size doesnt match the frame file size -> ERROR
        insert lisr_raw_frame
    insert lisr_raw_record
        * if record already exists -- ERROR
    commit transaction

## process 3 - telemetry processing:
    listen on lisr_telemetry
    pick up 'waiting' type=telemetry
    unpack tar in temp directory , catch tar stderr 
        * if tar didnt succeeded -- stop with status='failure', error_msg=stderr 
    read untarred json file
        [{"g":"sw","f":"watchdog_counter","t":1647878260035923565,"v":1},
        {"g":"adcs","f":"position_timestamp","t":1647878260137891685,"v":1647878257363281250},
        {"g":"adcs","f":"time_error","t":1647878260137893545,"v":31258000},
        {"g":"adcs","f":"position_x","t":1647878260137894955,"v":-2166907},
        {"g":"adcs","f":"position_y","t":1647878260137896065,"v":-3828835},
        ...
        * if json error --> ERROR
        * if root not list --> ERROR
    for each element e of the list:
        * if e isnt dict that contains g, f, t, v --> ERROR
        if doesnt exist in lisr_telemetry, insert
        notify lisr_leanspace
    commit
    
## process 4 - transmission to leanspace
    listen on lisr_leanspace
    pick up a batch of waiting or failure telemetry records: switch status to 'sending'
    transmit to leanspace
    switch status to 'success' or 'failure'