fdcodepy 0.1.1
A codebook solution for time series data compression and feature extraction considering rebound effect
License: MIT License
Requires Python: >=3.9

# fdcodepy

## Introduction

- FD_codepy is an open-source python package that can be used to extract time series in an interpretable manner, and use it for compression.
- The key idea is proposed specifically for metered data in energy sector, but can also be used with smart sensors and edge computing.
- Inspired by Codebook method, it breaks down the time series data into its constituent parts, i.e., the unique sub-patterns called Codewords, and the index of the Codewords, i.e., representations, allowing for efficient compression and analysis.
- Compared to resampling data into lower resolution, this lossy compression method takes similar data storage and transmission bandwidth, while preserving high frequency information and accumulative/average metered values.

- Get a high level idea of the problem from our article published by The Conversation:
  - [Smart meters haven’t delivered the promised benefits to electricity users. Here’s a way to fix the problems](https://theconversation.com/smart-meters-havent-delivered-the-promised-benefits-to-electricity-users-heres-a-way-to-fix-the-problems-225346#:~:text=Grid%20modernisation%20and%20smart%20meters,time%20isn't%20being%20provided)
- The FD_codepy source code is on GitHub: https://github.com/abc123yuanrui/FD_codepy/

  - An example is provided as notebook: FD_codepy\examples\Codebook_processing_for_energy_porfile.ipynb

## Key method for time series compression

- `Codebook`: key class for reconstructing long energy time series into unique partitions (codewords) and representations. Check examples for details.
  - It takes time series, window size, and distance metric types as inputs.
  - Four ensambled distance methods are:
    - Euclidean Distance (default or `'euclidean'`)
    - Dynamic Time Warping (`'DTW'`)
    - Wasserstein Distance (`'Wasserstein'`)
    - Flexibility Distance (`'flexibilityD'`)
  - `preprocessing` method will normalise the data into normalised series as attribute `normalized_arr`, with the scaler attribute `scaler_average`
  - `get_distance_matrix` method is a statistical analysis that computes the distance matrix for long time series (assuming we know historical data). It returns the matrix and quantile result for setting a similarity threshold (otherwise, the threshold can be set by an empirical value).
  - `desolve_time_series_thre` process the time seires into `codewords` and `representations`, return them.
  - `post_processing` reconstructe time series based on codewords and representations, the result stores as attribute `recovered_series`
- Flexbility distance: a novel distance metric that measures the similarity between time series data while taking into account both temporal and amplitude distance, and the rebound effect of the data.
  - `Codebook.flex_distance` is a static function for getting FD between the two given time series.
    - The default usage case is `fd = Codebook.flex_distance(series_a, series_b)` which provides fd by default settings
    - Get addtional routing inforamtion by `fd, row_index, col_index = Codebook.flex_distance(series_a, series_b, route = True)`. It probvides the reshaping strage from series a to series b
    - User can also customise the weighted matrix by `fd = Codebook.flex_distance(series_a, series_b, weighted_matix_a, weighted_matix_b, route = False)`
  - Given any two time series, user can generate a sample dashboard with the default settings following:
    - `from fdcodepy.utils.helpers import distance_method_routing_analysis`
    - `distance_method_routing_analysis(series_a, series_b, methods.Code_book, report = True)`
    - The report is gereated in working directorary, user can change it by modifying `export_dir` variable of the function `distance_method_routing_analysis`

## Installation

- Install using pip: `pip install fdcodepy`

## Usage

- Import the package: `import fdcodepy`
- Step by step example: Codebook processing for a given hourly time series and window size of 24, which decides the compression ratio (same with resmpling data from hourly into daily)
  - `from fdcodepy import methods`
  - `sample_series = np.random.uniform(0, 30, 365*24)`
  - `series_codebook = methods.Code_book(time_series, 24, 'flexibilityD')`
  - `series_codebook.pre_processing()`
  - `distance_matrix, quantiles = series_codebook.get_distance_matrix()`
  - `codewords, representations = series_codebook.desolve_time_series_thre(quantiles[0])`
  - `series_codebook.post_processing()`
  - `series_codebook.recovered_series` is the reconstruted data, computed from the representations with only lenght of `len(representations)`, compared to original series with length `len(sample_series)`
- The representations are the length of data needs to be communicated to data center, which is equal to the size of downsampled data, in this case, 365
- Use the Codebook processing result to generate a report
  - `from fdcodepy.utils.helpers import code_book_processing_analysis`
  - `code_book_processing_analysis(series_codebook, time_index, report = True, export_dir = '.')`
- Use the `FlexibilityDistance` to compute the flexibility distance between two time series datasets (with default settings).
  - `from fdcodepy import methods`
  - `Code_book.flex_distance(time_series_1, time_series_2)`

## Example analysis report

![Figure 1](https://github.com/abc123yuanrui/CompressionMethodsForSmartGridApplicatins/blob/main/examples/figs/report_demo.png?raw=true)

The figures can be zoomed to for checking details
![Figure 2](https://github.com/abc123yuanrui/CompressionMethodsForSmartGridApplicatins/blob/main/examples/figs/report_zoom_demo.png?raw=true)

## Reference

- Yuan, R., Pourmousavi, S. A., Soong, W. L., Black, A. J., Liisberg, J. A. R., & Lemos-Vinasco, J. (2024). Unleashing the benefits of smart grids by overcoming the challenges associated with low-resolution data. Cell Reports Physical Science, 5(2), 101830. https://doi.org/10.1016/j.xcrp.2024.101830
- Yuan, S. A. Pourmousavi, W. L. Soong, A. J. Black, J. A. R. Liisberg, and J. Lemos-Vinasco, “A
New Time Series Similarity Measure and Its Smart Grid Applications,” 2023. https://arxiv.org/abs/2310.12399