raman-hypmap 0.0.6
Raman hyperspectral processing for Horiba instrument
Requires Python: >=3.7

# raman_hypmap

Raman hyperspectral processing via python scripts tailoured for .txt output of Horiba instruments, e.g., the [LabSpec 6 Spectroscopy Suite Software](https://www.horiba.com/int/scientific/products/detail/action/show/Product/labspec-6-spectroscopy-suite-software-1843/) product. Theoretically this code will work on any hyperspectral Raman data if the .txt input is formatted correctly. 

Three python modules are available:
* bgr_process — automated background reduction of entire map
* filter_window — using defined peaks to create spectral intensity maps 
* hypmap_plot — interactive plot of intensity maps to create and save spectral masks

## Referencing

[![DOI](https://zenodo.org/badge/662148161.svg)](https://zenodo.org/doi/10.5281/zenodo.10473516)

If using this code, please reference the archive at [zenodo.org](https://doi.org/10.5281/zenodo.10473517)

Please reference Wang and Dai (2016) for the iterative polynomial smoothing method used for background reduction of spectra `bgr_process`. Reference: Wang, T. and Dai, L., 2017. Background subtraction of Raman spectra based on iterative polynomial smoothing. Applied spectroscopy, 71(6), pp.1169-1179. doi: https://doi.org/10.1177/0003702816670915.

The progress bar used in this code is from [user](https://stackoverflow.com/users/320726/6502) and full details are on [stackoverflow](https://stackoverflow.com/questions/6169217/replace-console-output-in-python)

The pixel selector used for the hyperspectral map `hypmap_plot` is adapted from a [matplotlib tutorial](https://matplotlib.org/stable/gallery/widgets/polygon_selector_demo.html) for creating a polygon selector. 

### License

The project is licensed under the GNU-v3 license.

## Installation

To install the package, a `.toml` and `.whl` are provided. Download, navigate to the folder where the . `.toml` or `.whl` file is located and run `pip install`. Alternatively, the files can be imported as modules locally. Module are defined within `hyprocess.py`. 

This package is also available at https://pypi.org/project/raman-hypmap/

## Usage

Import a .txt hyperspectral map file from the horiba software (*e.g.,* [LabSpec 6](https://www.horiba.com/int/scientific/products/detail/action/show/Product/labspec-6-spectroscopy-suite-software-1843/)). The file is structured as a tab deliminated file with the first row denoting the `Raman Shift` (cm<sup>-1</sup>) values. Note this row is offset two spaces, given that each hyperspectral pixel is then stored as `x-value`, `y-value`, and `spectral intensity-values` where the latter has the same length as the list of `Raman Shift` values, created pairs. An exampe .txt fie is provided in `example/example_data.txt` to illustrate this.

The .txt file should be imported into a pandas dataframe.

Although optional, setting the coordinates of the map to start at (0,0) will avoid complications.

```
import pandas as pd

file = 'example/example_data.txt'

file = file.rsplit('.',1)[0]
_df = pd.read_csv(file+'.txt', sep='\t', index_col=False, decimal=',',header=None)
_df = _df.replace(',','', regex=True)
# set origin to zero
_df[0] = _df[0].sub(_df[0].min())
_df[1] = _df[1].sub(_df[1].min())
```

The python modules can then be run in order, for example: 

```
from raman_hypmap import hyprocess

''' --- background reduction --- '''
_df_bgr = hyprocess.bgr_process(_df,file)
_df_bgr = pd.read_csv(file+'_bgr.csv', sep=',', index_col=False,header=None)
_df_raw = pd.read_csv(file+'_raw.csv', sep=',', index_col=False,header=None)

''' --- mineral filter --- '''
window_location = 'example/_window_ranges.csv'
hyprocess.filter_window(_df_bgr,file+'_bgr',window_location)

''' --- interactive hyperspectral map --- '''
_df_filtered = pd.read_csv(file+'_bgr_w_filter.csv', sep=',', index_col=False)
hyprocess.hypmap_plot(_df_filtered,_df_raw,file)
```

Note that each module imports the generated .csv file from the previous module as an input. These files will be created automatically when running the module to the local folder. 

An example script showing this code is also available in: `example/example.py`

### bgr_process

The input format is the standard .txt file generated by the Horiba [LabSpec 6](https://www.horiba.com/int/scientific/products/detail/action/show/Product/labspec-6-spectroscopy-suite-software-1843/) software. However, this format can be recreated via a tab-delimited .txt file as shown below:

| blank        | blank         | Raman-shift values (tab-delimited list)     |
| ------------ | ------------- |  ------------------------------------------ |
| x-coorindate | y-coordinate  | Raman-intensity values (tab-delimited list) |
| ...          | ...           | ...                                         |

Each spectra is background reduced via an iterative polynomial smoothing method Wang and Dai (2016) and written into a pandas dataframe. Three files are output and automatically saved as .csv files: 
* example_data_raw.csv — reformatted data input saved as a .csv file
* example_data_bg.csv — the background subtracted from each spectra saved as a spectral file
* example_data_bgr.csv — the resulting background reduced spectra saved as a spectral file

Each file is saved in a identical structure to the table shown above, but comma-delimited in a format that can be opened and viewed in Microsoft Excel. 

### filter_window

This module needs a .csv file formatted as explained in the section above to operate correctly. The code will itterate over every spectra provided and take the maximum intensity value from this window. It will then output a hyperspectral grid of values using this window. That hyperspecral grid will be saved as an image .png file with greyscale values stretched to the maximum and minimum intensity values recorded. 

The filter windows are imported from a standalone .csv file called `_window_ranges.csv` that can be open and edited in Microsoft Excel. The headings must not be changed. The format of the spredsheet is shown below: 

| phase    | char_min | char_max | comp_min | comp_max | i_min  | i_max  | 
| -------- | -------- |  ------- |  ------- |  ------- |  ----- |  ----- |
| graphite | 1510     | 1620     | 0        | 0        | 0      | 2000   |
| apatite  | 950      | 975      | 0        | 0        | 0      | 2000   |
| ...      | ...      | ...      | ...      | ...      | ...    | ...    |

These numbers are important as they define how the python code operates:
* 'Phase' defines the name of the species
* 'char_min' and 'char_max' define the window that the filter will operate over. A new hyperspectral map will be created for every row of values. 
* Sometimes spectra overlap and so 'comp_min' and 'comp_max' define a comparison window. the maximum intensity values defined in the 'char_' window must be *larger* than the maximum intensity value in the 'comp_' window otherwise a value of 0 will be given for that spectra. 
* 'i_min' and 'i_max' work in a similar way to the above. The maximum intensity of the spectra in the window defined by 'char_' must be within the intensity thresholds defined by the 'i_min' and 'i_max' variables. 

Each row of values will create and save a new hyperspectral map using that window. Examples derived from `example/example.py` using the current configuration of `_window_ranges.csv` are shown below:

| | | |
|:-------------------------:|:-------------------------:|:-------------------------:|
|<img width="300" alt="example_data_bgr_png_apatite.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_bgr_png_apatite.png">|<img width="300" alt="example_data_bgr_png_feldspar.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_bgr_png_feldspar.png">|<img width="300" alt="example_data_bgr_png_graphite.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_bgr_png_graphite.png">|
|<img width="300" alt="example_data_bgr_png_hematite.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_bgr_png_hematite.png">|<img width="300" alt="example_data_bgr_png_ilmenite.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_bgr_png_ilmenite.png">|<img width="300" alt="example_data_bgr_png_magnetite.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_bgr_png_magnetite.png">|

In addition, this module will create a new `_w_filter.csv` spreadsheet that stores these values. This is formatted so that the first two columns store the x and y coordinates and each subsequent column is added for every species  defined in `_window_ranges.csv`. An example of the output is provided `example/example_output/example_data_bgr_w_filter.csv`

### hypmap_plot

This module needs two .csv files, a `_w_filter.csv` spreadsheet output generated by the `filter_window` module and a spectrum data file formatted using the `bgr_process` module. The module will then create a hyperspectral image with a superposed grid of points aligned with each pixel in the hyperspectral image. These points can be selected by drawing circles using the mouse. When highlighted, they will be black. 

<img width="400" alt="example_data_spectra_map.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_spectra_map.png">

When pixels are selected in this way, pressing `ENTER` will generate a simple plot generated from every single pixel. Three lines are shown on the plot: 
* The **mean** with ± 2 standard deviations with grey bars
* The **mean** smoothed using a Savitzky–Golay filter (2nd order polynomial and 15-wide window).
* The **median** of all selected pixels (Orange)

<img width="400" alt="example_data_spectra_plot.png" src="https://github.com/ag00dwin/raman_hypmap/blob/main/example/_example%20output/example_data_spectra_plot.png">

The location of the highlighted pixels will be saved as a .png file and the spectral mask will be saved as a tab-delimited .txt file. In addition, the .txt file saves the **sum** of all spectra. A second text file consisiting of two columns: (1) Raman shift values, vs (2) mean intensity will also be generated. This is a suitable input for additional automatic fitting software, such as [IFORS](http://www.sediment.uni-goettingen.de/download/). An example of these .txt files are provided `example/example_output/example_data_spectra.txt`. 

This process can be repeated as long as a new selection is made. Every time you hit `ENTER` and plot the result of a new mask, the files automatically saving the spectra and location will be overwritten.