eclabfiles 0.4.1
Processing and converting of data files from BioLogic's EC-Lab.
Requires Python: >=3.9
Keywords: mpt,mpr,mps,biologic,ec-lab

# eclabfiles
This is a package to process and convert files from BioLogic's EC-Lab.
The parsers build on [Chris Kerr's `galvani` package](https://github.com/chatcannon/galvani)
and on the work of a previous civilian service member at Empa Lab 501,
Jonas Krieger.

## Installation
Use [pip](https://pip.pypa.io/en/stable/) to install eclabfiles.

```bash
> pip install eclabfiles
```

## Example Usage

### `process`: Processing Into Dictionaries
Process the data as it is stored in the corresponding file. The method
automatically determines filetype and tries to apply the respective
parser.

For `.mps` settings files you can specify the keyword `load_data` to
also load the data files from the same folder.

```python
import eclabfiles as ecf
data, meta = ecf.process("./mpt_files/test_01_OCV.mpt")
```

The returned data structure may look a bit different depending on which
filetype you read in.

See [Filetypes and Processed Data Structure](#filetypes-and-processed-data-structure).

### `to_df`: Processing Into Dataframe
Processes the file and converts it into a [Pandas `DataFrame`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html).
The `pd.DataFrame.attrs` will contain all the processed metadata.

```python
import eclabfiles as ecf
df = ecf.to_df("./mpr_files/test_02_CP.mpr")
```

If the given file is an `.mps`, all data files from the same folder will
be read into a `pd.DataFrame` with a [hierarchical index](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html#multiindex-advanced-indexing). The top-level index is the technique
number. The `pd.DataFrame.attrs` will contain `.mps` metadata, as well
as all techniques and their loaded metadata.

### `to_csv`: Converting to CSV
Process the file and write the data part into a `.csv` file at the
specified location.

```python
>>> import eclabfiles as ecf
>>> ecf.to_csv("./mpt_files/test_03_PEIS.mpt", csv_fn="./csv_files/test_PEIS.csv")
```

The `csv_fn` parameter is optional. If left away, the method writes a
`.csv` file into the same folder as the input file.

### `to_excel`: Converting to Excel
Process the file and write the data part into an Excel `.xlsx` file at
the specified location.

```python
>>> import eclabfiles as ecf
>>> ecf.to_excel("./experiment/test.mps")
```

The `excel_fn` parameter is optional. If left away, the method writes
a `.xlsx` file at the location of the input file.

## Filetypes and Processed Data Structure.
The file types that are implemented are:

| Filetype | Description                                                                          |
|----------|--------------------------------------------------------------------------------------|
| `.mpr`   | Raw data binary file, which also contains the current parameter settings             |
| `.mpt`   | Text format file generated when the user exports the raw `.mpr` file in text format. |
| `.mps`   | Settings file, which contains all the parameters of the experiment.                  |

### Processed `.mpr` Files
```python
data, meta = ecf.process("./test_01_OCV.mpr")
```

Any `data` returned by the `process` function for `.mpr` files is
structured into record dictionaries:
```python
[{column -> value}, ..., {column -> value}]
```

The `meta` processed from `.mpr` files looks like this:
```python
{
    "settings": {  # (optional) Settings if present.
        "technique": str,  # Technique name.
        "comments": str,  # Cell characteristics.
        "active_material_mass": float,
        "at_x": float,
        "molecular_weight": float,
        "atomic_weight": float,
        "acquisition_start": float,
        "e_transferred": int,
        "electrode_material": str,
        "electrolyte": str,
        "electrode_area": float,
        "reference_electrode": str,
        "characteristic_mass": float,
        "battery_capacity": float,
        "battery_capacity_unit": int
    },
    "params": [  # (optional) Technique parameter sequences.
        {"param1": float, "param2": str, ...},
        ...,
        {"param1": float, "param2": str, ...},
    ],
    "units": {  # Units of the data columns.
        "time": "s",
        "mode": None,
        ...,
        },
    "log": {  # (optional) Log if present.
        "channel_number": int,
        "channel_sn": int,
        "Ewe_ctrl_min": float,
        "Ewe_ctrl_max": float,
        "ole_timestamp": float,
        "filename": str,
        "host": str,
        "address": str,
        "ec_lab_version": str,
        "server_version": str,
        "interpreter_version": str,
        "device_sn": str,
        "averaging_points": int,
        "posix_timestamp": float,
    },
}
```

### Processed `.mpt` Files
```python
data, meta = ecf.process("./test_01_OCV.mpt")
```

Any `data` returned by the `process` function for `.mpr` files is
structured into record dictionaries:
```python
[{column -> value}, ..., {column -> value}]
```

The `.mpt` files generally contain a few more `data` columns than the
corresponding binary `.mpr` files from what I have seen.

The `meta` processed from `.mpt` files looks like this:

```python
{
    "raw": str,  # (optional) The raw file header if present.
    "settings": {  # (optional) Settings if the file has a header.
        "posix_timestamp": float,  # POSIX timestamp if present.
        "technique": str,  # Technique name.
    },
    "params": [  # (optional) Technique parameter sequences.
        {"param1": float, "param2": str, ...},
        {"param1": float, "param2": str, ...},
        ...,
    ],
    "units": {  # Units of the data columns.
        "time": "s",
        "mode": None,
        ...,
    },
    "loops": {  # (optional) Loops if present.
        "n_indexes": int,
        "indexes": list[int],
    }
}
```

### Processed `.mps` Files
```python
techniques, meta = ecf.process("./test.mps")
```

`.mps` files simply relate different `techniques` together and store no
data, while the other files contain the measurements.

For `.mps` settings files the `process` function returns the following
the linked `techniques` instead of the data (each technique can contain
data depending on `load_data`):

```python
{
    "1": {
        "technique": str,  # Technique name.
        "params": [  # (optional) Technique parameter sequences.
            {"param1": float, "param2": str, ...},
            ...,
            {"param1": float, "param2": str, ...},
        ],
        "data": list[dict]  # (optional) Data processed from data files.
        "meta": dict  # (optional) Metadata processed from data files.
    },
    ...
}
```

The `meta` processed from `.mpr` only contains the raw file header.
```python
{
    "raw": str
}
```

## Techniques
Detecting and processing the technique parameter sequences is not
implemented for all techniques as this is pretty tedious to do.
Currently, the following techniques are implemented:

| Short Name | Full Name                                       |
|------------|-------------------------------------------------|
| CA         | Chronoamperometry / Chronocoulometry            |
| CP         | Chronopotentiometry                             |
| CV         | Cyclic Voltammetry                              |
| GCPL       | Galvanostatic Cycling with Potential Limitation |
| GEIS       | Galvano Electrochemical Impedance Spectroscopy  |
| LOOP       | Loop                                            |
| LSV        | Linear Sweep Voltammetry                        |
| MB         | Modulo Bat                                      |
| OCV        | Open Circuit Voltage                            |
| PEIS       | Potentio Electrochemical Impedance Spectroscopy |
| WAIT       | Wait                                            |
| ZIR        | IR compensation (PEIS)                          |

### Implementing further techniques
In the best case you should have an `.mps`, `.mpr` and `.mpt` files
ready that contain the technique you would like to implement.

For the parsing of EC-Lab ASCII files (`.mpt`/`.mps`) you add a function
with a `list` of parameter names in `techniques.py` in the order they
appear in the text files. See `_wait_params()` to get an idea.

If the technique has a changing number of parameters in these ASCII
files, e.g. it contains a modifiable number of 'Limits' or 'Records',
you have to write a slightly more complicated function. Compare
`_peis_params()`.

Finally, add a case for the parsing function into `technique_params()`.

If you want to implement the technique in the `.mpr` file parser, you
will need to define a corresponding Numpy `np.dtype` in the
`techniques.py` module. I would recommend getting a solid hex editor
(e.g. Hexinator, Hex Editor Neo) to find the actual binary data type of
each parameter.

From the `.mpr` files I have seen, you will usually find the parameter
sequences at an offset of `0x1845` from the start of the data section in
the `VMP settings` module or somewhere around there. Compare the
parameter values in the binary data to the values in the corresponding
ASCII files.

As a rule of thumb, floats are usually 32bit little-endian (`<f4`),
integers are often 8bit (`|u1`) or 16bit (`<u2`) wide and units are
stored in 8bit integers. I have not gotten around to linking the integer
value with the corresponding unit yet.

If the technique has a changing number of parameters, make a list of
Numpy `dtype`s. Compare `_mb_params_dtypes` to see how this looks.

Finally, add your `np.dtype` or `list[np.dtype]` to the
`technique_params_dtypes` dictionary indexed by the technique ID. This
ID is the first byte value after the `VMP settings` module's header.

Good luck!