solo-epd-loader 0.4.4
Data loader for Solar Orbiter/EPD energetic charged particle sensors EPT, HET, and STEP.
Requires Python: >=3.9

solo-epd-loader
===============

|pypi Version| |conda version| |python version| |pytest| |codecov| |license| |repostatus| |zenodo doi|

.. |pypi Version| image:: https://img.shields.io/pypi/v/solo-epd-loader?style=flat&logo=pypi
   :target: https://pypi.org/project/solo-epd-loader/
.. |conda version| image:: https://img.shields.io/conda/vn/conda-forge/solo-epd-loader?style=flat&logo=anaconda
   :target: https://anaconda.org/conda-forge/solo-epd-loader/
.. |license| image:: https://img.shields.io/conda/l/conda-forge/solo-epd-loader?style=flat
   :target: https://github.com/jgieseler/solo-epd-loader/blob/main/LICENSE.rst
.. |python version| image:: https://img.shields.io/pypi/pyversions/solo-epd-loader?style=flat&logo=python
.. |zenodo doi| image:: https://zenodo.org/badge/446889843.svg
   :target: https://zenodo.org/badge/latestdoi/446889843
.. |pytest| image:: https://github.com/jgieseler/solo-epd-loader/workflows/pytest/badge.svg
.. |codecov| image:: https://codecov.io/gh/jgieseler/solo-epd-loader/branch/main/graph/badge.svg?token=Z8dueEWqKS
   :target: https://codecov.io/gh/jgieseler/solo-epd-loader
.. |repostatus| image:: https://www.repostatus.org/badges/latest/active.svg
   :alt: Project Status: Active – The project has reached a stable, usable state and is being actively developed.
   :target: https://www.repostatus.org/#active

Python data loader for Solar Orbiter's (SolO) `Energetic Particle Detector (EPD) <http://espada.uah.es/epd/>`_. At the moment provides level 2 (l2), level 3 (l3), and low latency (ll) data (`more details on data levels here <http://espada.uah.es/epd/EPD_data_overview.php>`_) obtained through CDF files from ESA's `Solar Orbiter Archive (SOAR) <http://soar.esac.esa.int/soar>`_ for the following sensors:

- Electron Proton Telescope (EPT)
- High Energy Telescope (HET)
- SupraThermal Electrons and Protons (STEP)

Current caveats:

- Only the standard ``rates`` data products are supported (i.e., no ``burst`` or ``high cadence`` data).
- For EPT and HET, only electrons, protons and alpha particles are processed (i.e., for HET He3, He4, C, N, O, Fe are omitted at the moment).
- For STEP, electron data needs to be calculated manually.
- The Suprathermal Ion Spectrograph (SIS) is not yet included. 

Disclaimer
----------
This software is provided "as is", with no guarantee. It is no official data source, and not officially endorsed by the corresponding instrument teams. **Please always refer to the** `official EPD data description <http://espada.uah.es/epd/EPD_data.php>`_ **before using the data!**

Installation
------------

solo_epd_loader requires python >= 3.9.

It can be installed either from `PyPI <https://pypi.org/project/solo-epd-loader/>`_ using:

.. code:: bash

    pip install solo-epd-loader

or from `Anaconda <https://anaconda.org/conda-forge/solo-epd-loader/>`_ using:

.. code:: bash

    conda install -c conda-forge solo-epd-loader

Usage
-----

The standard usecase is to utilize the ``epd_load`` function, which
returns Pandas dataframe(s) of the EPD measurements and a dictionary
containing information on the energy channels.

.. code:: python

   from solo_epd_loader import epd_load

   df_1, df_2, energies = epd_load(sensor, startdate, enddate=None, level='l2', viewing=None, path=None, 
                                   autodownload=False, only_averages=False)

Input
~~~~~

-  ``sensor``: ``'ept'``, ``'het'``, or ``'step'`` (string)
-  ``startdate``, ``enddate``: Datetime object (e.g., ``dt.date(2021,12,31)`` or ``dt.datetime(2021,4,15)``) or integer of the form yyyymmdd with empty positions filled with zeros, e.g. ``20210415`` (if no ``enddate`` is provided, ``enddate = startdate`` will be used)
-  ``level``: ``'l2'`` or ``'ll'`` (string); defines level of data product: level 2 (``'l2'``), level 3 (``'l3'``), or low-latency (``'ll'``). By default ``'l2'``.
-  ``viewing``: ``'sun'``, ``'asun'``, ``'north'``, ``'south'``, ``'omni'`` (string) or ``None``; not
   needed for ``sensor = 'step'``. ``'omni'`` is just calculated as the average of the other four viewing directions: ``('sun'+'asun'+'north'+'south')/4``
-  ``path``: directory in which Solar Orbiter data is/should be
   organized; e.g. ``'/home/userxyz/solo/data/'`` (string). See `Data folder structure`_ for more details.
-  ``autodownload``: if ``True``, will try to download missing data files
   from SOAR (bolean)
- ``only_averages``: If ``True``, will for STEP only return the averaged fluxes, and not the data of each of the 15 Pixels. This will reduce the memory consumption. By default ``False``.

Return
~~~~~~

-  For ``sensor`` = ``'ept'`` or ``'het'``:

   1. Pandas dataframe with proton fluxes and errors (for EPT also alpha
      particles) in ‘particles / (s cm^2 sr MeV)’
   2. Pandas dataframe with electron fluxes and errors in ‘particles /
      (s cm^2 sr MeV)’
   3. Dictionary with energy information for all particles:

      -  String with energy channel info
      -  Value of lower energy bin edge in MeV
      -  Value of energy bin width in MeV

-  For ``sensor`` = ``'step'``:

   1. Pandas dataframe with fluxes and errors in ‘particles / (s cm^2 sr
      MeV)’
   2. Dictionary with energy information for all particles:

      -  String with energy channel info
      -  Value of lower energy bin edge in MeV
      -  Value of energy bin width in MeV

SupraThermal Electron Proton (STEP) sensor electron measurements
----------------------------------------------------------------

Please note that the STEP electron measurements are not directly provided in the publically released data, but need to be calculated from them. This process is not straightforward, and the resulting data is prone to uncertainties (like contamination). **Thus it should only be used scientifically with caution! Please refer to the** `official EPD data description <http://espada.uah.es/epd/EPD_data.php>`_ **before using the data!**


Data folder structure
---------------------

The ``path`` variable provided to the module should be the base
directory where the corresponding cdf data files should be placed in
subdirectories. First subfolder defines the data product ``level``
(``l2``, ``l3``, or ``low_latency`` at the moment), the next one the
``instrument`` (so far only ``epd``), and finally the ``sensor``
(``ept``, ``het`` or ``step``).

For example, the folder structure could look like this:
``/home/userxyz/solo/data/l2/epd/het``. In this case, you should call
the loader with ``path='/home/userxyz/solo/data'``; i.e., the base
directory for the data.

You can use the (automatic) download function described in the following
section to let the subfolders be created initially automatically. NB: It might
be that you need to run the code with *sudo* or *admin* privileges in order to
be able to create new folders on your system.

Data download within Python
---------------------------

While using ``epd_load()`` to obtain the data, one can choose to automatically
download missing data files from `SOAR <http://soar.esac.esa.int/soar>`_
directly from within python. They are saved in the folder provided by the
``path`` argument (see above). For that, just add ``autodownload=True`` to the
function call:

.. code:: python

   from solo_epd_loader import epd_load

   df_protons, df_electrons, energies = \
       epd_load(sensor='het', level='l2', startdate=20200820,
                enddate=20200821, viewing='sun',
                path='/home/userxyz/solo/data/', autodownload=True)

   # plot protons and alphas
   ax = df_protons.plot(logy=True, subplots=True, figsize=(20,60))
   plt.show()

   # plot electrons
   ax = df_electrons.plot(logy=True, subplots=True, figsize=(20,60))
   plt.show()

Note: The code will always download the *latest version* of the file
available at SOAR. So in case a file ``V01.cdf`` is already locally
present, ``V02.cdf`` will be downloaded nonetheless.

Example 1 - low latency data
----------------------------

Example code that loads low latency (ll) electron and proton (+alphas)
fluxes (and errors) for EPT NORTH telescope from Apr 15 2021 to Apr 16
2021 into two Pandas dataframes (one for protons & alphas, one for
electrons). In general available are ‘sun’, ‘asun’, ‘north’, ‘south’, and ‘omni’
viewing directions for ‘ept’ and ‘het’ telescopes of SolO/EPD.

.. code:: python

   from matplotlib import pyplot as plt
   from solo_epd_loader import epd_load

   df_protons, df_electrons, energies = \
       epd_load(sensor='ept', level='ll', startdate=20210415,
                enddate=20210416, viewing='north',
                path='/home/userxyz/solo/data/')

   # plot protons and alphas
   ax = df_protons.plot(logy=True, subplots=True, figsize=(20,60))
   plt.show()

   # plot electrons
   ax = df_electrons.plot(logy=True, subplots=True, figsize=(20,60))
   plt.show()

Example 2 - level 2 data
------------------------

Example code that loads level 2 (l2) electron and proton (+alphas)
fluxes (and errors) for HET SUN telescope from Aug 20 2020 to Aug 20
2020 into two Pandas dataframes (one for protons & alphas, one for
electrons).

.. code:: python

   from matplotlib import pyplot as plt
   from solo_epd_loader import epd_load

   df_protons, df_electrons, energies = \
       epd_load(sensor='het', level='l2', startdate=20200820,
                enddate=20200821, viewing='sun',
                path='/home/userxyz/solo/data/')

   # plot protons and alphas
   ax = df_protons.plot(logy=True, subplots=True, figsize=(20,60))
   plt.show()

   # plot electrons
   ax = df_electrons.plot(logy=True, subplots=True, figsize=(20,60))
   plt.show()

Example 3 - partly reproducing `Fig. 2 <https://www.aanda.org/articles/aa/full_html/2021/12/aa39883-20/F2.html>`_ from Gómez-Herrero et al. 2021 [#]_
-----------------------------------------------------------------------------------------------------------------------------------------------------

.. code:: python

   from matplotlib import pyplot as plt
   from solo_epd_loader import epd_load
   import numpy as np

   # set your local path here
   lpath = '/home/userxyz/solo/data'

   # load ept sun viewing data
   df_protons_ept, df_electrons_ept, energies_ept = \
      epd_load(sensor='ept', level='l2', startdate=20200708, 
               enddate=20200724, viewing='sun', path=lpath, autodownload=True)

   # load step data             
   df_step, energies_step = \
      epd_load(sensor='step', level='l2', startdate=20200708,
               enddate=20200724, path=lpath, autodownload=True)

   # change time resolution to get smoother curve (resample with mean)
   resample = '60min'

   fig, axs = plt.subplots(2, sharex=True, figsize=(8, 10), dpi=200)
   axs[0].set_prop_cycle('color', plt.cm.Oranges_r(np.linspace(0,1,7)))
   axs[1].set_prop_cycle('color', plt.cm.winter(np.linspace(0,1,7)))

   # plot selection of ept electron channels
   for channel in [0, 8, 16, 26]:
      df_electrons_ept['Electron_Flux'][f'Electron_Flux_{channel}'].resample(resample).mean().plot(
         ax = axs[0], logy=True, label='EPT '+energies_ept["Electron_Bins_Text"][channel][0])

   # plot selection of step ion channels
   for channel in [8, 17, 33]:
      df_step[f'Magnet_Avg_Flux_{channel}'].resample(resample).mean().plot(
         ax = axs[1], logy=True, label='STEP '+energies_step["Bins_Text"][channel][0])

   # plot selection of ept ion channels
   for channel in [6, 22, 32, 48]:
      df_protons_ept['Ion_Flux'][f'Ion_Flux_{channel}'].resample(resample).mean().plot(
         ax = axs[1], logy=True, label='EPT '+energies_ept["Ion_Bins_Text"][channel][0])

   axs[0].set_ylim([0.3, 4e6])
   axs[1].set_ylim([0.01, 5e8])

   axs[0].set_ylabel("Electron flux\n"+r"(cm$^2$ sr s MeV)$^{-1}$")
   axs[1].set_ylabel("Ion flux\n"+r"(cm$^2$ sr s MeV)$^{-1}$")
   axs[0].legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
   axs[1].legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
   plt.subplots_adjust(hspace=0)
   fig.savefig("gh2021_fig_2.png", bbox_inches = "tight")
   plt.close('all')

**NB: This is just an approximate reproduction with different energy
channels, different time resolution, and different viewing direction!
Note also that the STEP data can not be used straightforwardly.**
|Figure|

Example 4 - partly reproducing `Fig. 2e <https://www.aanda.org/articles/aa/full_html/2021/12/aa40940-21/F2.html>`_ from Wimmer-Schweingruber et al. 2021 [#]_ 
-------------------------------------------------------------------------------------------------------------------------------------------------------------

.. code:: python

   from matplotlib import pyplot as plt
   from solo_epd_loader import epd_load
   import datetime
   import pandas as pd

   # set your local path here
   lpath = '/home/userxyz/solo/data'

   # load data
   df_protons_sun, df_electrons_sun, energies = \
       epd_load(sensor='ept', level='l2', startdate=20201210,
                enddate=20201211, viewing='sun',
                path=lpath, autodownload=True)
   df_protons_asun, df_electrons_asun, energies = \
       epd_load(sensor='ept', level='l2', startdate=20201210,
                enddate=20201211, viewing='asun',
                path=lpath, autodownload=True)
   df_protons_south, df_electrons_south, energies = \
       epd_load(sensor='ept', level='l2', startdate=20201210,
                enddate=20201211, viewing='south',
                path=lpath, autodownload=True)
   df_protons_north, df_electrons_north, energies = \
       epd_load(sensor='ept', level='l2', startdate=20201210,
                enddate=20201211, viewing='north',
                path=lpath, autodownload=True)

   # plot mean intensities of two energy channels; 'channel' defines the lower one
   channel = 6
   ax = pd.concat([df_electrons_sun['Electron_Flux'][f'Electron_Flux_{channel}'],
                   df_electrons_sun['Electron_Flux'][f'Electron_Flux_{channel+1}']],
                   axis=1).mean(axis=1).plot(logy=True, label='sun', color='#d62728')
   ax = pd.concat([df_electrons_asun['Electron_Flux'][f'Electron_Flux_{channel}'],
                   df_electrons_asun['Electron_Flux'][f'Electron_Flux_{channel+1}']],
                   axis=1).mean(axis=1).plot(logy=True, label='asun', color='#ff7f0e')
   ax = pd.concat([df_electrons_north['Electron_Flux'][f'Electron_Flux_{channel}'],
                   df_electrons_north['Electron_Flux'][f'Electron_Flux_{channel+1}']],
                   axis=1).mean(axis=1).plot(logy=True, label='north', color='#1f77b4')
   ax = pd.concat([df_electrons_south['Electron_Flux'][f'Electron_Flux_{channel}'],
                   df_electrons_south['Electron_Flux'][f'Electron_Flux_{channel+1}']],
                   axis=1).mean(axis=1).plot(logy=True, label='south', color='#2ca02c')

   plt.xlim([datetime.datetime(2020, 12, 10, 23, 0), 
             datetime.datetime(2020, 12, 11, 12, 0)])

   ax.set_ylabel("Electron flux\n"+r"(cm$^2$ sr s MeV)$^{-1}$")
   plt.title('EPT electrons ('+str(energies['Electron_Bins_Low_Energy'][channel])
             + '-' + str(energies['Electron_Bins_Low_Energy'][channel+2])+' MeV)')
   plt.legend()
   plt.show()

**NB: This is just an approximate reproduction; e.g., the channel
combination is a over-simplified approximation!** |image1|

Example 5 - EPT level 3 data
----------------------------

Example code that loads level 3 (l3) electron and ion fluxes (and errors) for the EPT sensor for the GLE event on Oct 28 2024.

Note that for EPT level 3 data, all particle species and viewing directions are saved in a single Pandas dataframe that also includes pitch-angle distributions. 
In addition, two additional dataframes are provided, which provide the particle flow directions (unit vector) in RTN coordinates as well as spacecraft coordinate information.
Also, next to a dictionary providing energy information, another dictionary is returned that contains the CDF file metadata.
See `data.serpentine-h2020.eu/l3data/solo/ <https://data.serpentine-h2020.eu/l3data/solo/>`_ for more details on the data product.

Also note that the **corrected electron fluxes** can contain **negative values**. Though the user probably wants to omit them while plotting, they **need to be included if the data is integrated over time!**

.. code:: python

   from matplotlib import pyplot as plt
   from solo_epd_loader import epd_load

   df, df_rtn, df_hci, energies, metadata = epd_load(sensor='ept', level='l3',
                                                     startdate=20211028, enddate=20211028,
                                                     autodownload=True, pos_timestamp='start',
                                                     path='/home/userxyz/solo/data/')

   # plot ions of south viewing (D stands for "down")
   ax = df.filter(like='Ion_Flux_D').plot(logy=True)
   plt.show()

   # plot electrons for sun viewing
   ax = df.filter(like='Electron_Corrected_Flux_S').plot(logy=True)
   plt.show()

   # plot pitch angles for all four viewings
   for v in ['Pitch_Angle_A', 'Pitch_Angle_S', 'Pitch_Angle_N', 'Pitch_Angle_D']:
      ax = df[v].plot(label=v)
   plt.legend()
   plt.show()


Contributing
------------

Contributions to this package are very much welcome and encouraged! Contributions can take the form of `issues <https://github.com/jgieseler/solo-epd-loader/issues>`_ to report bugs and request new features or `pull requests <https://github.com/jgieseler/solo-epd-loader/pulls>`_ to submit new code. 


References
----------

.. [#] First near-relativistic solar electron events observed by EPD onboard Solar Orbiter, Gómez-Herrero et al., A&A, 656 (2021) L3, https://doi.org/10.1051/0004-6361/202039883

.. [#] First year of energetic particle measurements in the inner heliosphere with Solar Orbiter’s Energetic Particle Detector, Wimmer-Schweingruber et al., A&A, 656 (2021) A22, https://doi.org/10.1051/0004-6361/202140940

.. |Figure| image:: https://github.com/jgieseler/solo-epd-loader/raw/main/examples/gh2021_fig_2.png
.. |image1| image:: https://github.com/jgieseler/solo-epd-loader/raw/main/examples/ws2021_fig_2d.png

License
-------

This project is Copyright (c) Jan Gieseler and licensed under
the terms of the BSD 3-clause license. This package is based upon
the `Openastronomy packaging guide <https://github.com/OpenAstronomy/packaging-guide>`_
which is licensed under the BSD 3-clause license. See the licenses folder for
more information.

Acknowledgements
----------------

The development of this software has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 101004159 (SERPENTINE).