needle-track 0.1.6
Transient Recognition, Annotation, and Classification Kit
Requires Python: >=3.11


# **NEEDLE-TRACK**  
**Transient Recognition, Annotation, and Classification Kit**

## **System Overview**  
NEEDLE-TRACK is a local system (with future migration potential to cloud or other machines) for managing transient data from Lasair (ZTF). It provides structured data storage, update tracking, search capabilities, and a publicly available package for easy deployment.

## **Core Components**  
1. **Local System Management**  
   - Designed to run locally, with future migration capability to cloud or other machines.  
   - Configuration options allow users to set up the system on various environments.

2. **Database for Data Storage**  
   - Stores transient objects with changeable tags and associated comments for further annotation.  
   - Maintains an **Update List** to track objects whose data has changed or that have new incoming updates.  
   - Supports updates and tracking of object status changes.

3. **Terminal Interface**  
   - Users interact with the system through a command-line interface (CLI).  
   - Provides commands for ingestion, searching, commenting, and annotation.

4. **Search Function**  
   - Query objects by unique ZTF ID.  
   - Filter objects based on tags.  
   - Search based on annotation status (astronote presence).

5. **Public Package Distribution**  
   - The package, named `needle_track`, is hosted on GitHub for public access.  
   - Includes installation instructions, dependencies, and usage guidelines.

---

## **Data Ingestion**  
1. **Source**: Data exported from Lasair broker (ZTF).  
2. **Format**: JSON.  
3. **Update Mechanism**:  
   - Runs a script to fetch the latest data (default: past week, customizable).  
   - Checks for overlapping entries:  
     - **Overlap Check**: Compares incoming data with existing records.  
       - If overlaps are detected, the system updates the existing record and logs it into the **Update List**.  
       - New records are added directly to the database.  
       - Objects that are not interested are moved to the **Removed List**.  
   - Generates a report summarizing all updates and changes.


## **Data Storage**  
1. **SQL Database Structure**:  
   - **object table**: list of all objects in the database.

2. **Object Structure**:  
   - **ZTF ID**: Unique identifier from Lasair.  
   - **Object Properties**: Data and metadata from Lasair.  
   - **followup**: indicator if the object has been marked for followup.
   - **snoozed**: indicator if the object has been marked as snoozed.
   - **astronote**: indicator if the object has been marked as astronote.
   - **Comments**: User-added comments or notes for each object.  
   - **Link**: URL reference to the Lasair entry for further details.

---

## **Tutorial**
1. **Initialize the database**
   - `needle-track -i`
2. **Ingest data**
   - `needle-track ingest --slsn <path_to_data> --tde <path_to_data>`
3. **Search for objects**
   - `needle-track search --objectId/-o <object_id>`
   - `needle-track search --followup/-f`
   - `needle-track search --snoozed/-s`
   - `needle-track search --astronote/-a`
   - `needle-track search --list/-l`
4. **Update objects**
   - `needle-track update --objectId/-o <object_id> --followup/-f -c This is a comment`
   - `needle-track update --objectId/-o <object_id> --snoozed/-s -c This is a comment`
   - `needle-track update --objectId/-o <object_id> --astronote/-a -c This is a comment`
5. **Add comments**
   - `needle-track comment -o <object_id> This is a comment`