ela 0.7.3
A Python package for exploratory lithology analysis
Keywords: lithology geophysics groundwater data-mining

ela
===

Exploratory Lithology Analysis
------------------------------

|pypi|\ |license| |Codacy Badge|

master |build-master|\ |codecov-master|

testing |build-testing|\ |codecov-testing|

|Getting started tutorial - Bungendore 3D lithology|

Analysing driller’s logs is a tedious and repetitive task in many
groundwater modelling projects. Automating the process of extracting
useful information from driller’s logs allows spending less time on
manual data wrangling, more time on its interpretation, and enhances the
reproducibility of the analysis.

This package combines features to:

-  perform natural language processing on lithology descriptions in the
   logs, to detect primary and secondary lithologies
-  apply supervised machine learning to interpolate lithologies across a
   3D grid
-  visualise interactively the 3D data

An introductory `sample
workflow <https://github.com/csiro-hydrogeology/pyela-doc/blob/master/tutorials/getting_started.ipynb>`__
gives an overview of some of these capabilities.

License
-------

MIT (see
`License.txt <https://github.com/csiro-hydrogeology/pyela/blob/master/LICENSE.txt>`__)

Documentation
-------------

The `documentation <https://pyela.readthedocs.io>`__ includes the API
documentation.

Some notebook tutorials are hosted `in a separate github repository
pyela-doc <https://github.com/csiro-hydrogeology/pyela-doc>`__.
pyela-doc includes quick setup instructions with conda.

Get a quick tour of `some legacy
visualisations <https://github.com/csiro-hydrogeology/pyela/blob/master/docs/visual_tour.md>`__.
Visualisation features are currently being reengineered.

Installation
------------

The quickest way with conda. We recommend installing conda packages
rather than pip packages whereever possible.

on Linux:

.. code:: bash

   wget https://raw.githubusercontent.com/csiro-hydrogeology/pyela/testing/configs/ela_environment.yml
   my_env_name=ELA
   # my_env_name=elauser
   conda env create -n $my_env_name -f ./ela_environment.yml python=3.7
   conda activate $my_env_name 

This should be all to get a working environment. If you want to use
‘ela’ from jupyter lab:

.. code:: bash

   conda install --name ${my_env_name} jupyterlab ipywidgets jupyter
   jupyter-labextension install @jupyter-widgets/jupyterlab-manager
   python -m ipykernel install --user --name ${my_env_name} --display-name "Py3 ELA"

Windows:

.. code:: bat

   call C:\Users\xxxyyy\AppData\Local\Continuum\anaconda3\Scripts\activate.bat
   REM curl should come with anaconda3. Perhaps not with miniconda though.
   where curl
   cd c:\tmp
   curl -o ela_environment.yml https://raw.githubusercontent.com/csiro-hydrogeology/pyela/testing/configs/ela_environment.yml
   set my_env_name=ELA
   conda env create -n %my_env_name% -f ela_environment.yml python=3.7
   conda activate %my_env_name% 

.. code:: bat

   This should be all to get a working environment. If you want to use 'ela' from jupyter lab:

.. code:: bat

   conda install -c conda-forge --name %my_env_name% jupyterlab ipywidgets jupyter
   jupyter-labextension install @jupyter-widgets/jupyterlab-manager
   python -m ipykernel install --user --name %my_env_name% --display-name "Py3 ELA"

Troubleshooting
~~~~~~~~~~~~~~~

pip packages specified from the ``environment.yaml`` files may have not
installed (under investigation). Check that pvgeo and “our” packages are
installed e.g.

.. code:: bat

   conda activate %my_env_name%
   conda list | grep pvgeo

if not present:

.. code:: bat

   REM make sure you have git in the PATH e.g.
   set PATH=C:\Users\per202\AppData\Local\Atlassian\SourceTree\git_local\mingw32\bin\;%PATH%
   where git

   conda activate %my_env_name%
   pip install --no-deps pvgeo
   pip install -e git+https://github.com/jmp75/striplog@master#egg=striplog
   pip install --no-deps -e git+https://github.com/jmp75/pyela@master#egg=ela

Manual installation
~~~~~~~~~~~~~~~~~~~

As of January 2019 `ela is on pypi <https://pypi.org/project/ela/>`__.
While ``pip install ela`` might work on some computers, it is *unlikely
that all python geospatial dependencies will install*. *We highly
recommend you set up a conda environment with all dependent packages*
prior to installing ela with pip or from source.

‘ela’ relies on several external packages, and some can be fiddly to
install depending on the version of Python and these external packages.
This section thus has fairly prescriptive instructions, given in the
hope of limiting the risk of issues.

Debian packages for spatial projections
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``cartopy`` and possibly other python packages require ``proj4`` version
4.9+ to be installed (libproj-dev). If your debian/ubuntu repo does not
suffice (older versions) you may try:

.. code:: bash

   sudo apt-get install -y libc6  
   wget http://en.archive.ubuntu.com/ubuntu/pool/universe/p/proj/proj-data_4.9.3-2_all.deb
   sudo dpkg -i proj-data_4.9.3-2_all.deb
   wget http://en.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj12_4.9.3-2_amd64.deb
   sudo dpkg -i libproj12_4.9.3-2_amd64.deb
   wget http://en.archive.ubuntu.com/ubuntu/pool/universe/p/proj/proj-bin_4.9.3-2_amd64.deb
   sudo dpkg -i proj-bin_4.9.3-2_amd64.deb
   wget http://en.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj9_4.9.2-2_amd64.deb 
   sudo dpkg -i libproj9_4.9.2-2_amd64.deb
   wget http://en.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj-dev_4.9.3-2_amd64.deb
   sudo dpkg -i libproj-dev_4.9.3-2_amd64.deb

Installation of python packages dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We recommend installing
`Anaconda <http://docs.continuum.io/anaconda/install>`__ to install
dependencies. Note that I recommend to **not** let anaconda change your
startup file and change the ``PATH`` environment. To activate Anaconda
you first need: ``source ~/anaconda3/bin/activate``. Then choose a conda
environment name.

Optionally, if your anaconda installation is a bit dated, you may want
to do ``conda update -n base conda`` and
``conda update -n base anaconda-navigator``

To create the conda environment for ela on Linux:

.. code:: bash

   # source ~/anaconda3/bin/activate
   my_env_name=ELA
   conda create --name ${my_env_name} python=3.7
   conda install --name ${my_env_name} rasterio cartopy geopandas pandas nltk scikit-learn scikit-image matplotlib vtk
   conda activate  ${my_env_name}

On Windows, using the DOS CMD prompt, assuming you installed Anaconda in
user mode.

.. code:: bat

   call %userprofile%\AppData\Local\Continuum\anaconda3\Scripts\activate.bat
   set my_env_name=ELA
   conda create --name %my_env_name% python=3.7
   REM if using conda activate  %my_env_name%  I seem to loose conda from the command line, so:
   conda install --name %my_env_name% conda 
   conda install --name %my_env_name% rasterio cartopy geopandas pandas nltk scikit-learn scikit-image matplotlib vtk
   conda activate  %my_env_name%

At this point we have installed all the python dependencies ela needs
that are available via ``conda``.

As of writing (2019-08) conda does not have pyqt5, and no suitable
version of mayavi for python3. We resort to use ``pip``. You may want to
do first:

.. code:: bash

   pip install --upgrade pip

For Python 3.x one needs to install pyqt5 for mayavi, as per `these
instructions <https://docs.enthought.com/mayavi/mayavi/installation.html>`__.
As of Jan 2019 be aware that there is a `known issue in mayavi visual
rendering with pyqt5 as a backend on
Linux <https://github.com/enthought/mayavi/issues/656>`__ and ‘ela’ is
affected by this. Nevertheless this is not a commplete blocker for most
‘ela’ features so installation instructions are kept here.

.. code:: bash

   pip search pyqt5 | sort -g
   pip search mayavi

.. code:: bash

   pip install pyqt5
   pip install mayavi

For users without admin rights，

.. code:: bash

   conda install pyqt5
   pip install mayavi --user

At this point all mandatory dependencies for ‘ela’ are installed.

Installing ELA
~~~~~~~~~~~~~~

There are three options to access ela:

-  use the latest available on pip,
-  clone and install with ``setup.py``
-  direct import of the package directory (this is done at runtime from
   e.g. a notebook)

.. code:: bash

   pip search ela
   pip install ela

Alternatively, from source with ``setup.py``

.. code:: bash

   pip install -r requirements.txt
   python setup.py install

Optional dependencies
^^^^^^^^^^^^^^^^^^^^^

As of 2019-05 you can find new features using deep learning for
classification in the submodule ``ela.experiment``. You will need the
additional dependencies:

.. code:: bash

   conda install --name ${my_env_name} gensim tensorflow keras
   pip install wordcloud

If reading xls files using pandas, need pkg ``xlrd`` with e.g.
``conda install xlrd``

using Jupyter-lab
~~~~~~~~~~~~~~~~~

You may use ela as you prefer; we recomment using “Jupyter Lab” to write
notebooks. See the `Jupyter lab
doc <https://jupyterlab.readthedocs.io/en/stable/>`__ for official
information.

The following should be enough otherwise to use ‘ela’

Linux:

.. code:: bash

   my_env_name=ELA
   conda install --name ${my_env_name} jupyterlab ipywidgets jupyter
   jupyter-labextension install @jupyter-widgets/jupyterlab-manager
   python -m ipykernel install --user --name ${my_env_name} --display-name "Py3 ELA"

Windows:

.. code:: bat

   set my_env_name=ELA
   conda install --name %my_env_name% jupyterlab ipywidgets jupyter
   jupyter-labextension install @jupyter-widgets/jupyterlab-manager
   python -m ipykernel install --user --name %my_env_name% --display-name "Py3 ELA"

Related Geoscience packages
---------------------------

‘ela’ aims to complement other Python packages for geoscience, in
particular for handling bore data . It already depends on the package
`‘striplog’ <https://github.com/agile-geoscience/striplog>`__ and is
likely to depend on `‘lasio’ <https://github.com/kinverarity1/lasio>`__
in the future.

You should also check the repository
`hydrogeol_utils <https://github.com/Neil-Symington/hydrogeol_utils>`__

Known issues
------------

3D interactive visualisation - Using mayavi 4.6+ on python 3.6+ may be
`visually buggy <https://github.com/enthought/mayavi/issues/656>`__ on
Linux. This appears to be a low level issue running on laptop with dual
Intel/NVIDIA Graphic cards.

.. _troubleshooting-1:

Troubleshooting
---------------

If in a conda environment trying to use ``pip`` you get:

.. code:: text

   ModuleNotFoundError: No module named 'pip._internal'

consider:

.. code:: bash

   curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
   python3 get-pip.py --force-reinstall

.. |pypi| image:: https://img.shields.io/pypi/v/ela.svg?logo=python&logoColor=white
   :target: https://pypi.org/project/ela/
.. |license| image:: http://img.shields.io/badge/license-MIT-blue.svg
   :target: https://github.com/csiro-hydrogeology/pyela/blob/master/LICENSE.txt
.. |Codacy Badge| image:: https://api.codacy.com/project/badge/Grade/40f01ca659f04fe5832bbfe7f939bf8b
   :target: https://www.codacy.com/app/jmp75/pyela?utm_source=github.com&utm_medium=referral&utm_content=csiro-hydrogeology/pyela&utm_campaign=Badge_Grade
.. |build-master| image:: https://img.shields.io/travis/csiro-hydrogeology/pyela.svg?branch=master
   :target: https://travis-ci.org/csiro-hydrogeology/pyela/branches
.. |codecov-master| image:: https://codecov.io/gh/csiro-hydrogeology/pyela/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/csiro-hydrogeology/pyela/branch/master
.. |build-testing| image:: https://img.shields.io/travis/csiro-hydrogeology/pyela.svg?branch=testing
   :target: https://travis-ci.org/csiro-hydrogeology/pyela/branches
.. |codecov-testing| image:: https://codecov.io/gh/csiro-hydrogeology/pyela/branch/testing/graph/badge.svg
   :target: https://codecov.io/gh/csiro-hydrogeology/pyela/branch/testing
.. |Getting started tutorial - Bungendore 3D lithology| image:: https://raw.githubusercontent.com/csiro-hydrogeology/pyela-doc/master/tutorials/img/3d_overlay_bungendore_clay_lithology_small.png
   :target: https://github.com/csiro-hydrogeology/pyela-doc/blob/master/tutorials/getting_started.ipynb