pipeline-eds 0.4.31
A Python API pipeline library and CLI toolkit for Emerson Ovation EDS systems — built for engineers, control technicians, and wastewater SCADA teams.
License: MIT
Requires Python: <3.15,>=3.9

# `pipeline-eds`

`pipeline-eds` is a Python project designed to simplify API access to Emerson Enterprise Data Server (EDS) machines. It facilitates seamless data exchange between Emerson's Ovation EDS system and various external parties, including third-party contractors and internal employees. The project is distributed on PyPI under the package name `pipeline-eds`.

---

## ⚡Quick Start

For detailed installation and usage instructions, see [QUICKSTART.md](https://github.com/City-of-Memphis-Wastewater/pipeline/blob/main/QUICKSTART.md).  
This guide includes step-by-step commands for Windows, Linux, Termux, and developer setups.

---

## 🚀 Getting Started

This detailed guide provides instructions for installing and running `pipeline-eds`. To ensure you follow the right path, first choose the method that best fits your needs.

---

### Choosing Your Installation Method

  * **For the absolute simplest setup (No Python required):** If you want to run the tool without installing Python or managing dependencies, use a pre-built binary. Follow **Method 1: Using Pre-Built Binaries**.
  * **For an easy CLI setup with simple updates:** If you are comfortable with the command line and want to easily keep the tool updated, `pipx` is the recommended approach. Follow **Method 2: CLI Installation with `pipx`**.
  * **For developing and contributing to the project:** If you plan to modify the source code, you need a full development environment. Follow **Method 3: Developer Setup with UV**.
  * **For installing from a Git clone without UV (Advanced):** If you have cloned the repository and need to install dependencies manually with `pip`, follow **Method 4: Installing from a Git Clone with `pip`**.

-----

<details>
<summary> Method 1: Using Pre-Built Binaries (`.exe`, `.elf`, `.pyz`)</summary>

This is the easiest way to get started, especially on systems where you don't have Python installed. These are standalone packages that you can download and run directly.

1.  **Download the appropriate binary** for your system from the project's [**GitHub Releases page**](https://github.com/City-of-Memphis-Wastewater/pipeline/releases).

    - `pipeline-eds*.exe`: For Windows.
    - `pipeline-eds* (ELF has no extension)`: For Linux and Termux on Android.
    - `pipeline-eds*.pyz`: A zipapp for any system that has Python installed.

2.  **Place the file** in a convenient location.
	```bash
	# On Termux (Android)
	termux-setup-storage
	cp storage/downloads/ . to copy the file from your Android downloads folder to your $HOME folder
	```
	On iSH, you launch by default in the `root` directory, and the executable can be copied here (if you would like) manually using the file browser.

3.  **Run the command** from your terminal. You may need to make the `.elf` file executable first (`chmod +x pipeline-eds-*`).

    ```bash
    # On Windows
    .\pipeline-eds-*.exe config

    # On Linux or Termux
    ./pipeline-eds-* config
    ```

For more details on the pros and cons of each binary type, see the **Distribution and Packaging** section below.

</details>

-----

<details>
<summary> Method 2: CLI Installation with `pipx` (Recommended for CLI Users)</summary>

`pipx` installs and runs Python applications in isolated environments. This is the best way to get easy updates and avoid conflicts with other Python packages.

**1. Install Python and `pip`**

If you don't have them, install them using your system's package manager or an official installer.
> **Windows Note:** If installing from the `.exe` installer from [python.org](https://www.python.org/downloads/), be sure to check the box for **"Add Python to PATH"** during setup.

```bash
# On Windows (using a package manager in PowerShell)
winget install Python.Python.3.11
# Or with Chocolatey:
# choco install python

# On Ubuntu/Debian
sudo apt update && sudo apt install python3 python3-pip python-is-python3

# On Termux (Android)
pkg update && pkg install python
pkg install python python-cryptography # python-numpy # numpy dependency removed for 0.3.50
pip install --upgrade setuptools wheel # Just to be sure.

# Some of these are likely overkill given prepackaged cryptography, but I want you to succeed, and I will continue testing.
pkg install clang make libffi openssl-dev libffi-dev
# pkg install rust # probably not necessary, with `python-cryptography` installed
# Alterative to python-cryptography, you could `pip install cryptography` after `pip install setuptools`

# On Alpine (iSH on iOS)
apk update && apk add python3 py3-pip
apk add py3-cryptography py3-numpy py3-virtualenv
apk add openssl-dev libffi-dev
```

**2. Install `pipx`**

Use `pip` to install `pipx` and add its scripts to your system's PATH.

```bash
python3 -m pip install --user pipx
python3 -m pipx ensurepath
```

*(You may need to restart your terminal for the PATH change to take effect.)*

**3. Install `pipeline-eds`**

Install the package from PyPI using `pipx`.

```bash
# For all systems (Linux, macOS, Windows)
pipx install pipeline-eds

# For Termux and iSH, which require dedicated system site packages like py3-cryptography, as seen above
pipx install --system-site-packages pipeline-eds

# For Windows users who want database features from the pyodbc library (the usefulness of this has not yet been developed).
pipx install "pipeline-eds[windows]"

# If you want non-browser plotting with Matplotlib (Linux, macOS, Windows)
pipx install "pipeline-eds[mpl]"
# With the `trend` command, use the `--webplot` flag to direct the plot to plotly HTML anyways and circumvent Matplotlib.
```

**4. Run Commands**

You can now use the `eds` alias directly in your terminal. There is also the `pipeline` alias and the `pipeline-eds` alias.

```bash
eds config
eds trend M100FI --start June3 --end June17
```

</details>

-----
<details>
<summary> Method 3: Developer Setup with UV</summary>

This method is for contributors who need a full development environment to modify the source code.
Learn more about [git](https://www.youtube.com/watch?v=qrD3z9_9DXU).
See [git guide](https://git-scm.com/docs).

** Clone the Repository**

```bash
git clone https://github.com/City-of-Memphis-Wastewater/pipeline-eds.git
cd pipeline-eds
```

** Install `uv`**

This project uses `uv` for dependency management.

** Run Commands**

```bash
uv run eds config
uv run eds ping
```

</details>

-----

<details>
<summary> Method 4: Installing from a Git Clone with `pip` (Advanced)</summary>

This method is for users who have cloned the repository but prefer to manage the environment with `pip` and `venv`. This is often necessary on platforms like **Termux** or **iSH (Alpine)**.
A use-case for this is for generating binaries on a system such that it is compatible with that system (Note that **iSH** emulates x86_64).

** Clone the Repository**

```bash
git clone https://github.com/City-of-Memphis-Wastewater/pipeline-eds.git
cd pipeline-eds
```

#### For Termux (Android)

1.  **Install System Build Dependencies:**
    ```bash
    pkg update && pkg upgrade -y
	pkg install python python-cryptography # python-numpy # numpy is no longer needed as of 0.3.50
    
	# The build tools (rust, clang, etc.) are generally not needed IF the 
    # Termux-installed packages satisfy the requirements.
    # The below line can likely be removed if --system-site-packages is used, 
    # but we will leave it for max compatibility.
    pkg install rust clang make openssl-dev libffi-dev
	pip install --no-cache-dir cryptography
    ```
2.  **Create and Activate a Virtual Environment:**
    ```bash
	# CRITICAL: Use the --system-site-packages flag to access Termux's pre-compiled packages.
    python -m venv --system-site-packages .venv
    source .venv/bin/activate
    ```
3.  **Install Python Dependencies:**
    ```bash
    pip install -r requirements.txt
    ```
4.  **Run Commands:**
    ```bash
    python -m pipeline_eds.cli config
    ```

#### For iSH / Alpine Linux (iOS)

1.  **Install System Build Dependencies:**
    ```bash
    apk update
	
	# Install the core Python environment tools and essential pre-compiled Python libraries
    # Installing 'py3-cryptography' and 'py3-numpy' via apk avoids difficult, lengthy compilation from source later.
    apk add python3 py3-pip py3-cryptography py3-numpy py3-virtualenv
    apk add gcc musl-dev build-base openssl-dev libffi-dev 
    
	```
2.  **Create and Activate a Virtual Environment:**
    ```bash
	# The '--system-site-packages' flag is crucial: it allows this venv to access the
    # pre-compiled Python packages (like py3-cryptography) installed in the previous step 
    # by the system package manager (apk). This satisfies their requirements without re-installing.
	python3 -m venv --system-site-packages .venv
    
	# Activate the virtual environment
	source .venv/bin/activate
    ```
3.  **Install Python Dependencies:**
    ```bash
	# Install all project-specific dependencies defined in the requirements file.
    # 'pip' will install these packages into the isolated '.venv', while still
    # using the system packages (if needed) due to the venv's configuration.
    pip install -r requirements.txt
    ```
4.  **Run Commands:**
    ```bash
	 # Execute the main application command using the Python interpreter from the activated venv.
    python3 -m pipeline_eds.cli trend M100FI
	
	# Deactivate the virtual environment. This resets the shell's PATH to the system's 
    # default Python environment.
    deactivate
    
    # NOTE: If you run the software after deactivating, the 'python3' command will only see
    # system-installed libraries, not the packages installed specifically for this project.
    # You can reactivate the environment anytime using 'source .venv/bin/activate' if you are 
    # in the project's directory.
    ```
	
</details>

---

<details>
<summary>📦 Distribution and Packaging</summary>

This project supports multiple packaging formats to make installation flexible across platforms.  
While some formats allow installation on systems without internet access, note that the **application itself requires internet connectivity** to call its API.

[How to Build Binaries](https://github.com/City-of-Memphis-Wastewater/pipeline/wiki/Build-Scripts,-How-and-Where)

### 🔹 Executables (`.exe`, `.elf`)
- **Generated by**: [`build_executable.py`](https://github.com/City-of-Memphis-Wastewater/pipeline/blob/main/build_executable.py)
- **Variants**:
  - **Windows `.exe`**: Tested on Windows 11. Runs standalone without requiring Python.
  - **Linux `.elf`**:
    - Built on **WSL2 Ubuntu** for general Linux systems.
    - Built on **Termux** for Android devices.  
      - Smoothest rollout on Termux: no need to install Python separately.
      - Avoids the `.shortcuts` widget permission error seen with `.pyz`.
- **Internet required for install**: ❌ (binaries can be copied directly)  
- **Internet required for use**: ✅ (API calls)  

### 🔹 Python Zip App (`.pyz` + `.bat`)
- **Generated by**: [`build_shiv.sh`](https://github.com/City-of-Memphis-Wastewater/pipeline/blob/main/build_shiv.py) on WSL2 Ubuntu
- **Best for**: Systems that already have Python installed.
- **Windows support**: A `.bat` launcher is provided for smoother execution.
- **Termux notes**:
  - Works, but calling `.pyz` from the Termux `.shortcuts` widget can trigger a **permission error**.
  - Requires Python to be installed on Termux.
- **Maintainability**: Easier to update regularly compared to static binaries.
- **Internet required for install**: ❌ (once `.pyz` is copied)  
- **Internet required for use**: ✅  

### 🔹 `pipx` Install
- **Best for**: Staying current with rolling changes.
- **Update shortcut**: On Termux, an update shortcut is available directly from the home screen widget.
- **Requirements**: Python and `pipx` installed.
- **Internet required for install/update**: ✅  
- **Internet required for use**: ✅  

### 🔹 Source Distributions (`.tar.gz`, `.whl`)
- **Generated by**: uv
- **Best for**: Developers or environments where building from source is preferred.
- **Internet required for install**: ✅ (to fetch dependencies)  
- **Internet required for use**: ✅  


---

### ⚠️ Notes and Limitations
- These packages simplify **installation** on disconnected systems, but the application itself requires internet access to function (API calls).

</details>

---

## ✨ Tips for Optimal Usage & Maintenance
To ensure a smooth and efficient experience with `pipeline-eds`, consider the following best practices:

### Keep `pipeline-eds` Updated: 
Regularly upgrade your `pipeline-eds` installation to benefit from the latest features, bug fixes, and performance improvements.
```bash
pipx upgrade pipeline-eds
```

### Understanding eds config and Credential Management:
The first time you execute a command requiring access to your EDS API (e.g., eds trend), `pipeline-eds` will guide you through a one-time configuration process. Your sensitive API credentials (URL, username, password) are securely stored using your operating system's native keyring service. This avoids storing plaintext passwords. If your credentials change, you can re-run eds config at any time to update them.

### Network Connectivity (VPN Essential):
If your EDS server is located on a private network (e.g., within your organization's internal network), you must be connected to the appropriate Virtual Private Network (VPN). Failure to do so will result in connection errors when `pipeline-eds` attempts to fetch data.

### Leveraging Flexible Date/Time Inputs:
The `eds trend` command offers highly flexible date and time parsing for its --start and --end options, thanks to the `pendulum` package. You can use a wide variety of natural language inputs, such as:

- `--start "2023-09-18"`
- `--start "Sept 18"`
- `--end "now"` 

Experiment with different formats to suit your query needs. Remember to use quotes around values if they contain spaces.

---


## ⚙️ Project Implementation & Use Cases

Aside from being a CLI tool for visualing sensor data at will, `pipeline` is designed to be deployed as a scheduled task on a Windows server, for hourly data transmission to a third party.

  * The project is executed by **Windows Task Scheduler**, which calls a PowerShell script (`main_eds_to_rjn_quiet.ps1`) as the entry point.
  * The iterative timing (e.g., hourly execution) is handled by the `Task Scheduler`, not by Python.
  * For these automated tasks, a standard `venv` is used, as `Task Scheduler` can run under different user accounts.

---

## 📱 Running on Android (`Termux`)

The `pipeline` project can be installed and run on Android devices using the **[Termux](https://termux.dev/)** terminal emulator.  
For most users, **CLI installation via `pipx` is the recommended method**, as development is not expected in this environment and `pipx` provides the smoothest way to stay up to date.

### Termux Installation Options

There are several ways to run `pipeline` inside Termux, depending on your needs:

- **`pipx` (Recommended)**  
  - Provides rolling updates and integrates with the Termux home‑screen widget.  
  - As of **v0.3.8**, you `eds setup` will generate a **shortcut button** in the Termux widget to update `pipeline-eds` without needing to open the terminal.  
  - Requires Python and `pipx` to be installed.  
  - Best choice if you want to keep up with frequent changes.

- **Native ELF Binary**  
  - A prebuilt `.elf` binary is available specifically for Termux.  
  - Smoothest rollout for users who just want the CLI tool without expecting updates.

- **`.pyz` Zipapp**  
  - ⚠️ Known limitation: launching `.pyz` directly from the Termux `.shortcuts` widget may trigger a **permission error**.

---


### 🌐 Termux and Web-Based Visuals (Plotly)
When using `pipeline-eds` in Termux to generate plots (e.g., with `eds trend`), the visuals are displayed as web-based HTML pages using libraries like Plotly. Instead of directly opening a graphical window (which is not typically supported by Termux's command-line environment), `pipeline-eds` serves these HTML files via a local web server (often on localhost).

[Why localhost](https://github.com/City-of-Memphis-Wastewater/pipeline/wiki/Why-localhost-on-Termux)

---

<details>
<summary> 🐍 Python Version Compatibility</summary>

The `pipeline-eds` project is designed to support a broad range of modern Python versions, from Python 3.8 up to the latest stable releases, ensuring accessibility across various operating environments (desktop, server, and mobile environments like Termux).

### Supported Python Versions

The project officially supports the following CPython versions:

| Python Version                | Status              | Key Dependency Notes                                                                                                                                                                                                                                                                           |
| ----------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **3.11 / 3.12 / 3.13 / 3.14** | ✅ Fully Supported   | Runs on the latest major versions of all dependencies (e.g., `keyring` v25+, `pendulum` v3+, `urllib3` v2+).                                                                                                                                                                                   |
| **3.10**                      | ✅ Fully Supported   | Stable; uses latest dependencies, with specific `numpy` and `matplotlib` pins.                                                                                                                                                                                                                 |
| **3.9**                       | ✅ Fully Supported   | Stable; this version marks the transition to modern dependency major versions.                                                                                                                                                                                                                 |


### Important Notes on Conditional Dependencies

To maintain compatibility across this range, we pin several major dependencies based on your Python version, ensuring maximum stability.

- **Python 3.9+ (Recommended)**: Installations on Python 3.9 and newer automatically receive the latest, feature-rich major versions of core libraries such as `keyring`, `pendulum`, `urllib3`, and the development tools `pytest` and `pytest-cov`.
    
- **Python 3.8 (Legacy)**: This project no longer support Python 3.8

### Recommended Version

**Python 3.11 or newer is highly recommended** for the best performance, security, and access to the latest features from all third-party libraries. If you are using the Developer Setup, please target Python 3.11.9 as specified in the getting started guide.

</details>

---

[Comparing the pipeline-eds Python Package to the Emerson Ovation EDS Excel Add-In and the EDS Portal Webapp](https://github.com/City-of-Memphis-Wastewater/pipeline/wiki/Comparison-of-%60pipeline%E2%80%90eds%60-Python-Package-vs-Emerson-Ovation-EDS-Excel-Add%E2%80%90In-vs-EDS-Portal-Webapp)