# -*- coding: utf-8 -*-

r"""
The :mod:`pyunlocbox.solvers` module implements a solving function (which will
minimize your objective function) as well as common solvers.

Solving
-------

Call :func:`solve` to solve your convex optimization problem using your
instantiated solver and functions objects.

Interface
---------

The :class:`solver` base class defines a common interface to all solvers:

.. autosummary::

    solver.pre
    solver.algo
    solver.post

Solvers
-------

Then, derived classes implement various common solvers.

.. autosummary::

    gradient_descent
    forward_backward
    douglas_rachford
    generalized_forward_backward

**Primal-dual solvers** (based on :class:`primal_dual`)

.. autosummary::

    mlfbf
    projection_based

.. inheritance-diagram:: pyunlocbox.solvers
    :parts: 2

"""

import time

import numpy as np

from pyunlocbox.functions import dummy, _prox_star
from pyunlocbox import acceleration

def solve(functions, x0, solver=None, atol=None, dtol=None, rtol=1e-3,
          xtol=None, maxit=200, verbosity='LOW', inplace=False):
    r"""
    Solve an optimization problem whose objective function is the sum of some
    convex functions.

    This function minimizes the objective function :math:`f(x) =
    \sum\limits_{k=0}^{k=K} f_k(x)`, i.e. solves
    :math:`\operatorname{arg\,min}\limits_x f(x)` for :math:`x \in
    \mathbb{R}^{n \times N}` where :math:`n` is the dimensionality of the data
    and :math:`N` the number of independent problems. It returns a dictionary
    with the found solution and some informations about the algorithm
    execution.

    Parameters
    ----------
    functions : list of objects
        A list of convex functions to minimize. These are objects who must
        implement the :meth:`pyunlocbox.functions.func.eval` method. The
        :meth:`pyunlocbox.functions.func.grad` and / or
        :meth:`pyunlocbox.functions.func.prox` methods are required by some
        solvers. Note also that some solvers can only handle two convex
        functions while others may handle more. Please refer to the
        documentation of the considered solver.
    x0 : array_like
        Starting point of the algorithm, :math:`x_0 \in \mathbb{R}^{n \times
        N}`.
    solver : solver class instance, optional
        The solver algorithm. It is an object who must inherit from
        :class:`pyunlocbox.solvers.solver` and implement the :meth:`_pre`,
        :meth:`_algo` and :meth:`_post` methods. If no solver object are
        provided, a standard one will be chosen given the number of convex
        function objects and their implemented methods.
    atol : float, optional
        The absolute tolerance stopping criterion. The algorithm stops when
        :math:`f(x^t) < atol` where :math:`f(x^t)` is the objective function at
        iteration :math:`t`. Default is None.
    dtol : float, optional
        Stop when the objective function is stable enough, i.e. when
        :math:`\left|f(x^t) - f(x^{t-1})\right| < dtol`. Default is None.
    rtol : float, optional
        The relative tolerance stopping criterion. The algorithm stops when
        :math:`\left|\frac{ f(x^t) - f(x^{t-1}) }{ f(x^t) }\right| < rtol`.
        Default is :math:`10^{-3}`.
    xtol : float, optional
        Stop when the variable is stable enough, i.e. when :math:`\frac{\|x^t -
        x^{t-1}\|_2}{\sqrt{n N}} < xtol`. Note that additional memory will be
        used to store :math:`x^{t-1}`. Default is None.
    maxit : int, optional
        The maximum number of iterations. Default is 200.
    verbosity : {'NONE', 'LOW', 'HIGH', 'ALL'}, optional
        The log level : ``'NONE'`` for no log, ``'LOW'`` for resume at
        convergence, ``'HIGH'`` for info at all solving steps, ``'ALL'`` for
        all possible outputs, including at each steps of the proximal operators
        computation. Default is ``'LOW'``.
    inplace : bool, optional
        If True and x0 is a numpy array, then x0 will be modified in place
        during execution to save memory. It will then contain the solution. Be
        careful to pass data of the type (int, float32, float64) you want your
        computations to use.

    Returns
    -------
    sol : ndarray
        The problem solution.
    solver : str
        The used solver.
    crit : {'ATOL', 'DTOL', 'RTOL', 'XTOL', 'MAXIT'}
        The used stopping criterion. See above for definitions.
    niter : int
        The number of iterations.
    time : float
        The execution time in seconds.
    objective : ndarray
        The successive evaluations of the objective function at each iteration.

    Examples
    --------
    >>> import numpy as np
    >>> from pyunlocbox import functions, solvers

    Define a problem:

    >>> y = [4, 5, 6, 7]
    >>> f = functions.norm_l2(y=y)

    Solve it:

    >>> x0 = np.zeros(len(y))
    >>> ret = solvers.solve([f], x0, atol=1e-2, verbosity='ALL')
    INFO: Dummy objective function added.
    INFO: Selected solver: forward_backward
    INFO: Forward-backward method
        dummy evaluation: 0.000000e+00
        norm_l2 evaluation: 1.260000e+02
    Iteration 1 of forward_backward:
        dummy evaluation: 0.000000e+00
        norm_l2 evaluation: 1.400000e+01
        objective = 1.40e+01
    Iteration 2 of forward_backward:
        dummy evaluation: 0.000000e+00
        norm_l2 evaluation: 2.963739e-01
        objective = 2.96e-01
    Iteration 3 of forward_backward:
        dummy evaluation: 0.000000e+00
        norm_l2 evaluation: 7.902529e-02
        objective = 7.90e-02
    Iteration 4 of forward_backward:
        dummy evaluation: 0.000000e+00
        norm_l2 evaluation: 5.752265e-02
        objective = 5.75e-02
    Iteration 5 of forward_backward:
        dummy evaluation: 0.000000e+00
        norm_l2 evaluation: 5.142032e-03
        objective = 5.14e-03
    Solution found after 5 iterations:
        objective function f(sol) = 5.142032e-03
        stopping criterion: ATOL

    Verify the stopping criterion (should be smaller than atol=1e-2):

    >>> np.linalg.norm(ret['sol'] - y)**2  # doctest:+ELLIPSIS
    0.00514203...

    Show the solution (should be close to y w.r.t. the L2-norm measure):

    >>> ret['sol']
    array([4.02555301, 5.03194126, 6.03832952, 7.04471777])

    Show the used solver:

    >>> ret['solver']
    'forward_backward'

    Show some information about the convergence:

    >>> ret['crit']
    'ATOL'
    >>> ret['niter']
    5
    >>> ret['time']  # doctest:+SKIP
    0.0012578964233398438
    >>> ret['objective']  # doctest:+NORMALIZE_WHITESPACE,+ELLIPSIS
    [[126.0, 0], [13.99999999..., 0], [0.29637392..., 0], [0.07902528..., 0],
    [0.05752265..., 0], [0.00514203..., 0]]

    """
    # to prevent any modification of the input
    if not(inplace):
        x0 = x0.copy()

    if verbosity not in ['NONE', 'LOW', 'HIGH', 'ALL']:
        raise ValueError('Verbosity should be either NONE, LOW, HIGH or ALL.')

    # Add a second dummy convex function if only one function is provided.
    if len(functions) < 1:
        raise ValueError('At least 1 convex function should be provided.')
    elif len(functions) == 1:
        functions.append(dummy())
        if verbosity in ['LOW', 'HIGH', 'ALL']:
            print('INFO: Dummy objective function added.')

    # Choose a solver if none provided.
    if not solver:
        if len(functions) == 2:
            fb0 = 'GRAD' in functions[0].cap(x0) and \
                  'PROX' in functions[1].cap(x0)
            fb1 = 'GRAD' in functions[1].cap(x0) and \
                  'PROX' in functions[0].cap(x0)
            dg0 = 'PROX' in functions[0].cap(x0) and \
                  'PROX' in functions[1].cap(x0)
            if fb0 or fb1:
                solver = forward_backward()  # Need one prox and 1 grad.
            elif dg0:
                solver = douglas_rachford()  # Need two prox.
            else:
                raise ValueError('No suitable solver for the given functions.')
        elif len(functions) > 2:
            solver = generalized_forward_backward()
        if verbosity in ['LOW', 'HIGH', 'ALL']:
            name = solver.__class__.__name__
            print('INFO: Selected solver: {}'.format(name))

    # Set solver and functions verbosity.
    translation = {'ALL': 'HIGH', 'HIGH': 'HIGH', 'LOW': 'LOW', 'NONE': 'NONE'}
    solver.verbosity = translation[verbosity]
    translation = {'ALL': 'HIGH', 'HIGH': 'LOW', 'LOW': 'NONE', 'NONE': 'NONE'}
    functions_verbosity = []
    for f in functions:
        functions_verbosity.append(f.verbosity)
        f.verbosity = translation[verbosity]

    tstart = time.time()
    crit = None
    niter = 0
    rtol_only_zeros = True

    # Solver specific initialization.
    solver.pre(functions, x0)

    # Evaluate the objective function at the begining
    objective = [solver.objective(x0)]

    while not crit:

        niter += 1

        if xtol is not None:
            last_sol = np.array(solver.sol, copy=True)

        if verbosity in ['HIGH', 'ALL']:
            name = solver.__class__.__name__
            print('Iteration {} of {}:'.format(niter, name))

        # Solver iterative algorithm.
        solver.algo(objective, niter)

        objective.append(solver.objective(solver.sol))
        current = np.sum(objective[-1])
        last = np.sum(objective[-2])

        # Verify stopping criteria.
        if atol is not None and current < atol:
            crit = 'ATOL'
        if dtol is not None and np.abs(current - last) < dtol:
            crit = 'DTOL'
        if rtol is not None:
            div = current  # Prevent division by 0.
            if div == 0:
                if verbosity in ['LOW', 'HIGH', 'ALL']:
                    print('WARNING: (rtol) objective function is equal to 0 !')
                if last != 0:
                    div = last
                else:
                    div = 1.0  # Result will be zero anyway.
            else:
                rtol_only_zeros = False
            relative = np.abs((current - last) / div)
            if relative < rtol and not rtol_only_zeros:
                crit = 'RTOL'
        if xtol is not None:
            err = np.linalg.norm(solver.sol - last_sol)
            err /= np.sqrt(last_sol.size)
            if err < xtol:
                crit = 'XTOL'
        if maxit is not None and niter >= maxit:
            crit = 'MAXIT'

        if verbosity in ['HIGH', 'ALL']:
            print('    objective = {:.2e}'.format(current))

    # Restore verbosity for functions. In case they are called outside solve().
    for k, f in enumerate(functions):
        f.verbosity = functions_verbosity[k]

    if verbosity in ['LOW', 'HIGH', 'ALL']:
        print('Solution found after {} iterations:'.format(niter))
        print('    objective function f(sol) = {:e}'.format(current))
        print('    stopping criterion: {}'.format(crit))

    # Returned dictionary.
    result = {'sol':       solver.sol,
              'solver':    solver.__class__.__name__,  # algo for consistency ?
              'crit':      crit,
              'niter':     niter,
              'time':      time.time() - tstart,
              'objective': objective}
    try:
        # Update dictionary for primal-dual solvers
        result['dual_sol'] = solver.dual_sol
    except AttributeError:
        pass

    # Solver specific post-processing (e.g. delete references).
    solver.post()

    return result

class solver(object):
    r"""
    Defines the solver object interface.

    This class defines the interface of a solver object intended to be passed
    to the :func:`pyunlocbox.solvers.solve` solving function. It is intended to
    be a base class for standard solvers which will implement the required
    methods. It can also be instantiated by user code and dynamically modified
    for rapid testing. This class also defines the generic attributes of all
    solver objects.

    Parameters
    ----------
    step : float
        The gradient-descent step-size. This parameter is bounded by 0 and
        :math:`\frac{2}{\beta}` where :math:`\beta` is the Lipschitz constant
        of the gradient of the smooth function (or a sum of smooth functions).
        Default is 1.
    accel : pyunlocbox.acceleration.accel
        User-defined object used to adaptively change the current step size
        and solution while the algorithm is running. Default is a dummy
        object that returns unchanged values.

    """

    def __init__(self, step=1., accel=None):
        if step < 0:
            raise ValueError('Step should be a positive number.')
        self.step = step
        self.accel = acceleration.dummy() if accel is None else accel

    def pre(self, functions, x0):
        """
        Solver-specific pre-processing. See parameters documentation in
        :func:`pyunlocbox.solvers.solve` documentation.

        Notes
        -----
        When preprocessing the functions, the solver should split them into
        two lists:
        * `self.smooth_funs`, for functions involved in gradient steps.
        * `self.non_smooth_funs`, for functions involved proximal steps.
        This way, any method that takes in the solver as argument, such as the
        methods in :class:`pyunlocbox.acceleration.accel`, can have some
        context as to how the solver is using the functions.

        """
        self.sol = np.asarray(x0)
        self.smooth_funs = []
        self.non_smooth_funs = []
        self._pre(functions, self.sol)
        self.accel.pre(functions, self.sol)

    def _pre(self, functions, x0):
        raise NotImplementedError("Class user should define this method.")

    def algo(self, objective, niter):
        """
        Call the solver iterative algorithm and the provided acceleration
        scheme. See parameters documentation in
        :func:`pyunlocbox.solvers.solve`

        Notes
        -----
        The method :meth:`self.accel.update_sol` is called before
        :meth:`self._algo` because the acceleration schemes usually involves
        some sort of averaging of previous solutions, which can add some
        unwanted artifacts on the output solution. With this ordering, we
        guarantee that the output of solver.algo is not corrupted by the
        acceleration scheme.

        Similarly, the method :meth:`self.accel.update_step` is called after
        :meth:`self._algo` to allow the step update procedure to act directly
        on the solution output by the underlying algorithm, and not on the
        intermediate solution output by the acceleration scheme in
        :meth:`self.accel.update_sol`.

        """
        self.sol[:] = self.accel.update_sol(self, objective, niter)
        self.step = self.accel.update_step(self, objective, niter)
        self._algo()

    def _algo(self):
        raise NotImplementedError("Class user should define this method.")

    def post(self):
        """
        Solver-specific post-processing. Mainly used to delete references added
        during initialization so that the garbage collector can free the
        memory. See parameters documentation in
        :func:`pyunlocbox.solvers.solve`.

        """
        self._post()
        self.accel.post()
        del self.sol, self.smooth_funs, self.non_smooth_funs

    def _post(self):
        raise NotImplementedError("Class user should define this method.")

    def objective(self, x):
        """
        Return the objective function at x.

        Necessitate `solver._pre(...)` to be run first.
        """
        return self._objective(x)

    def _objective(self, x):
        obj_smooth = [f.eval(x) for f in self.smooth_funs]
        obj_nonsmooth = [f.eval(x) for f in self.non_smooth_funs]
        return obj_nonsmooth + obj_smooth

class gradient_descent(solver):
    r"""
    Gradient descent algorithm.

    This algorithm solves optimization problems composed of the sum of
    any number of smooth functions.

    See generic attributes descriptions of the
    :class:`pyunlocbox.solvers.solver` base class.

    Notes
    -----
    This algorithm requires each function implement the
    :meth:`pyunlocbox.functions.func.grad` method.

    See :class:`pyunlocbox.acceleration.regularized_nonlinear` for a very
    efficient acceleration scheme for this method.

    Examples
    --------
    >>> import numpy as np
    >>> from pyunlocbox import functions, solvers
    >>> dim = 25
    >>> np.random.seed(0)
    >>> xstar = np.random.rand(dim)  # True solution
    >>> x0 = np.random.rand(dim)
    >>> x0 = xstar + 5*(x0 - xstar) / np.linalg.norm(x0 - xstar)
    >>> A = np.random.rand(dim, dim)
    >>> step = 1 / np.linalg.norm(np.dot(A.T, A))
    >>> f = functions.norm_l2(lambda_=0.5, A=A, y=np.dot(A, xstar))
    >>> fd = functions.dummy()
    >>> solver = solvers.gradient_descent(step=step)
    >>> params = {'rtol':0, 'maxit':14000, 'verbosity':'NONE'}
    >>> ret = solvers.solve([f, fd], x0, solver, **params)
    >>> pctdiff = 100 * np.sum((xstar - ret['sol'])**2) / np.sum(xstar**2)
    >>> print('Difference: {0:.1f}%'.format(pctdiff))
    Difference: 1.3%

    """

    def __init__(self, **kwargs):
        super(gradient_descent, self).__init__(**kwargs)

    def _pre(self, functions, x0):

        for f in functions:
            if 'GRAD' in f.cap(x0):
                self.smooth_funs.append(f)
            else:
                raise ValueError('Gradient descent requires each function to '
                                 'implement grad().')

        if self.verbosity == 'HIGH':
            print('INFO: Gradient descent minimizing {} smooth '
                  'functions.'.format(len(self.smooth_funs)))

    def _algo(self):
        grad = np.zeros_like(self.sol)
        for f in self.smooth_funs:
            grad += f.grad(self.sol)
        self.sol[:] -= self.step * grad

    def _post(self):
        pass

class forward_backward(solver):
    r"""
    Forward-backward proximal splitting (FISTA and ISTA) algorithm.

    This algorithm solves convex optimization problems composed of the sum of
    a smooth and a non-smooth function.

    See generic attributes descriptions of the
    :class:`pyunlocbox.solvers.solver` base class.

    Parameters
    ----------
    accel : :class:`pyunlocbox.acceleration.accel`
        Acceleration scheme to use.
        Default is :meth:`pyunlocbox.acceleration.fista`, which corresponds
        to the 'FISTA' solver. Passing :meth:`pyunlocbox.acceleration.dummy`
        instead results in the ISTA solver. Note that while FISTA is much more
        time-efficient, it is less memory-efficient.

    Notes
    -----
    This algorithm requires one function to implement the
    :meth:`pyunlocbox.functions.func.prox` method and the other one to
    implement the :meth:`pyunlocbox.functions.func.grad` method.

    See :cite:`beck2009FISTA` for details about the algorithm.

    Examples
    --------
    >>> import numpy as np
    >>> from pyunlocbox import functions, solvers
    >>> y = [4, 5, 6, 7]
    >>> x0 = np.zeros(len(y))
    >>> f1 = functions.norm_l2(y=y)
    >>> f2 = functions.dummy()
    >>> solver = solvers.forward_backward(step=0.5)
    >>> ret = solvers.solve([f1, f2], x0, solver, atol=1e-5)
    Solution found after 15 iterations:
        objective function f(sol) = 4.957288e-07
        stopping criterion: ATOL
    >>> ret['sol']
    array([4.0002509 , 5.00031362, 6.00037635, 7.00043907])

    """

    def __init__(self, accel=acceleration.fista(), **kwargs):
        super(forward_backward, self).__init__(accel=accel, **kwargs)

    def _pre(self, functions, x0):

        if self.verbosity == 'HIGH':
            print('INFO: Forward-backward method')

        if len(functions) != 2:
            raise ValueError('Forward-backward requires two convex functions.')

        if 'PROX' in functions[0].cap(x0) and 'GRAD' in functions[1].cap(x0):
            self.smooth_funs.append(functions[1])
            self.non_smooth_funs.append(functions[0])
        elif 'PROX' in functions[1].cap(x0) and 'GRAD' in functions[0].cap(x0):
            self.smooth_funs.append(functions[0])
            self.non_smooth_funs.append(functions[1])
        else:
            raise ValueError('Forward-backward requires a function to '
                             'implement prox() and the other grad().')

    def _algo(self):
        # Forward step
        x = self.sol - self.step * self.smooth_funs[0].grad(self.sol)
        # Backward step
        self.sol[:] = self.non_smooth_funs[0].prox(x, self.step)

    def _post(self):
        pass

class generalized_forward_backward(solver):
    r"""
    Generalized forward-backward proximal splitting algorithm.

    This algorithm solves convex optimization problems composed of the sum of
    any number of non-smooth (or smooth) functions.

    See generic attributes descriptions of the
    :class:`pyunlocbox.solvers.solver` base class.

    Parameters
    ----------
    lambda_ : float, optional
        A relaxation parameter bounded by 0 and 1. Default is 1.

    Notes
    -----
    This algorithm requires each function to either implement the
    :meth:`pyunlocbox.functions.func.prox` method or the
    :meth:`pyunlocbox.functions.func.grad` method.

    See :cite:`raguet2013generalizedFB` for details about the algorithm.

    Examples
    --------
    >>> import numpy as np
    >>> from pyunlocbox import functions, solvers
    >>> y = [0.01, 0.2, 8, 0.3, 0 , 0.03, 7]
    >>> x0 = np.zeros(len(y))
    >>> f1 = functions.norm_l2(y=y)
    >>> f2 = functions.norm_l1()
    >>> solver = solvers.generalized_forward_backward(lambda_=1, step=0.5)
    >>> ret = solvers.solve([f1, f2], x0, solver)
    Solution found after 2 iterations:
        objective function f(sol) = 1.463100e+01
        stopping criterion: RTOL
    >>> ret['sol']
    array([0. , 0. , 7.5, 0. , 0. , 0. , 6.5])

    """

    def __init__(self, lambda_=1, *args, **kwargs):
        super(generalized_forward_backward, self).__init__(*args, **kwargs)
        self.lambda_ = lambda_

    def _pre(self, functions, x0):

        if self.lambda_ <= 0 or self.lambda_ > 1:
            raise ValueError('Lambda is bounded by 0 and 1.')

        self.z = []
        for f in functions:
            if 'GRAD' in f.cap(x0):
                self.smooth_funs.append(f)
            elif 'PROX' in f.cap(x0):
                self.non_smooth_funs.append(f)
                self.z.append(np.array(x0, copy=True))
            else:
                raise ValueError('Generalized forward-backward requires each '
                                 'function to implement prox() or grad().')

        if self.verbosity == 'HIGH':
            print('INFO: Generalized forward-backward minimizing {} smooth '
                  'functions and {} non-smooth functions.'.format(
                      len(self.smooth_funs), len(self.non_smooth_funs)))

    def _algo(self):

        # Smooth functions.
        grad = np.zeros_like(self.sol)
        for f in self.smooth_funs:
            grad += f.grad(self.sol)

        # Non-smooth functions.
        if not self.non_smooth_funs:
            self.sol[:] -= self.step * grad  # Reduces to gradient descent.
        else:
            sol = np.zeros_like(self.sol)
            for i, g in enumerate(self.non_smooth_funs):
                tmp = 2 * self.sol - self.z[i] - self.step * grad
                tmp[:] = g.prox(tmp, self.step * len(self.non_smooth_funs))
                self.z[i] += self.lambda_ * (tmp - self.sol)
                sol += 1. * self.z[i] / len(self.non_smooth_funs)
            self.sol[:] = sol

    def _post(self):
        del self.z

class douglas_rachford(solver):
    r"""
    Douglas-Rachford proximal splitting algorithm.

    This algorithm solves convex optimization problems composed of the sum of
    two non-smooth (or smooth) functions.

    See generic attributes descriptions of the
    :class:`pyunlocbox.solvers.solver` base class.

    Parameters
    ----------
    lambda_ : float, optional
        The update term weight. It should be between 0 and 1. Default is 1.

    Notes
    -----
    This algorithm requires the two functions to implement the
    :meth:`pyunlocbox.functions.func.prox` method.

    See :cite:`combettes2007DR` for details about the algorithm.

    Examples
    --------
    >>> import numpy as np
    >>> from pyunlocbox import functions, solvers
    >>> y = [4, 5, 6, 7]
    >>> x0 = np.zeros(len(y))
    >>> f1 = functions.norm_l2(y=y)
    >>> f2 = functions.dummy()
    >>> solver = solvers.douglas_rachford(lambda_=1, step=1)
    >>> ret = solvers.solve([f1, f2], x0, solver, atol=1e-5)
    Solution found after 8 iterations:
        objective function f(sol) = 2.927052e-06
        stopping criterion: ATOL
    >>> ret['sol']
    array([3.99939034, 4.99923792, 5.99908551, 6.99893309])

    """

    def __init__(self, lambda_=1, *args, **kwargs):
        super(douglas_rachford, self).__init__(*args, **kwargs)
        self.lambda_ = lambda_

    def _pre(self, functions, x0):

        if self.lambda_ <= 0 or self.lambda_ > 1:
            raise ValueError('Lambda is bounded by 0 and 1.')

        if len(functions) != 2:
            raise ValueError('Douglas-Rachford requires two convex functions.')

        for f in functions:
            if 'PROX' in f.cap(x0):
                self.non_smooth_funs.append(f)
            else:
                raise ValueError('Douglas-Rachford requires each '
                                 'function to implement prox().')

        self.z = np.array(x0, copy=True)

    def _algo(self):
        tmp = self.non_smooth_funs[0].prox(2 * self.sol - self.z, self.step)
        self.z[:] = self.z + self.lambda_ * (tmp - self.sol)
        self.sol[:] = self.non_smooth_funs[1].prox(self.z, self.step)

    def _post(self):
        del self.z

class primal_dual(solver):
    r"""
    Parent class of all primal-dual algorithms.

    See generic attributes descriptions of the
    :class:`pyunlocbox.solvers.solver` base class.

    Parameters
    ----------
    L : function or ndarray, optional
        The transformation L that maps from the primal variable space to the
        dual variable space. Default is the identity, :math:`L(x)=x`. If `L` is
        an ``ndarray``, it will be converted to the operator form.
    Lt : function or ndarray, optional
        The adjoint operator. If `Lt` is an ``ndarray``, it will be converted
        to the operator form. If `L` is an ``ndarray``, default is the
        transpose of `L`. If `L` is a function, default is `L`,
        :math:`Lt(x)=L(x)`.
    d0: ndarray, optional
        Initialization of the dual variable.

    """

    def __init__(self, L=None, Lt=None, d0=None, *args, **kwargs):
        super(primal_dual, self).__init__(*args, **kwargs)

        if L is None:
            self.L = lambda x: x
        else:
            if callable(L):
                self.L = L
            else:
                # Transform matrix form to operator form.
                self.L = lambda x: L.dot(x)

        if Lt is None:
            if L is None:
                self.Lt = lambda x: x
            elif callable(L):
                self.Lt = L
            else:
                self.Lt = lambda x: L.T.dot(x)
        else:
            if callable(Lt):
                self.Lt = Lt
            else:
                self.Lt = lambda x: Lt.dot(x)

        self.d0 = d0

    def _pre(self, functions, x0):
        # Dual variable.
        if self.d0 is None:
            # The copy is necessary in case `L = lambda x: x`.
            self.dual_sol = self.L(np.asarray(x0).copy())
        else:
            self.dual_sol = self.d0

    def _post(self):
        self.d0 = None
        del self.dual_sol

    def _objective(self, x):
        obj_smooth = [f.eval(x) for f in self.smooth_funs]
        obj_nonsmooth = [self.non_smooth_funs[0].eval(x),
                         self.non_smooth_funs[1].eval(self.L(x))]
        return obj_nonsmooth + obj_smooth

class mlfbf(primal_dual):
    r"""
    Monotone+Lipschitz forward-backward-forward primal-dual algorithm.

    This algorithm solves convex optimization problems with objective of the
    form :math:`f(x) + g(Lx) + h(x)`, where :math:`f` and :math:`g` are proper,
    convex, lower-semicontinuous functions with easy-to-compute proximity
    operators, and :math:`h` has Lipschitz-continuous gradient with constant
    :math:`\beta`.

    See generic attributes descriptions of the
    :class:`pyunlocbox.solvers.primal_dual` base class.

    Notes
    -----
    The order of the functions matters: set :math:`f` first on the list,
    :math:`g` second, and :math:`h` third.

    This algorithm requires the first two functions to implement the
    :meth:`pyunlocbox.functions.func.prox` method, and the third function to
    implement the :meth:`pyunlocbox.functions.func.grad` method.

    The step-size should be in the interval :math:`\left] 0, \frac{1}{\beta +
    \|L\|_{2}}\right[`.

    See :cite:`komodakis2015primaldual`, Algorithm 6, for details.

    Examples
    --------
    >>> import numpy as np
    >>> from pyunlocbox import functions, solvers
    >>> y = np.array([294, 390, 361])
    >>> L = np.array([[5, 9, 3], [7, 8, 5], [4, 4, 9], [0, 1, 7]])
    >>> x0 = np.zeros(len(y))
    >>> f = functions.dummy()
    >>> f._prox = lambda x, T: np.maximum(np.zeros(len(x)), x)
    >>> g = functions.norm_l2(lambda_=0.5)
    >>> h = functions.norm_l2(y=y, lambda_=0.5)
    >>> max_step = 1/(1 + np.linalg.norm(L, 2))
    >>> solver = solvers.mlfbf(L=L, step=max_step/2.)
    >>> ret = solvers.solve([f, g, h], x0, solver, maxit=1000, rtol=0)
    Solution found after 1000 iterations:
        objective function f(sol) = 1.839060e+05
        stopping criterion: MAXIT
    >>> ret['sol']
    array([1., 1., 1.])

    """

    def _pre(self, functions, x0):
        super(mlfbf, self)._pre(functions, x0)

        if len(functions) != 3:
            raise ValueError('MLFBF requires 3 convex functions.')

        self.non_smooth_funs.append(functions[0])   # f
        self.non_smooth_funs.append(functions[1])   # g
        self.smooth_funs.append(functions[2])       # h

    def _algo(self):
        # Forward steps (in both primal and dual spaces)
        y1 = self.sol - self.step * (self.smooth_funs[0].grad(self.sol) +
                                     self.Lt(self.dual_sol))
        y2 = self.dual_sol + self.step * self.L(self.sol)

        # Backward steps (in both primal and dual spaces)
        p1 = self.non_smooth_funs[0].prox(y1, self.step)
        p2 = _prox_star(self.non_smooth_funs[1], y2, self.step)

        # Forward steps (in both primal and dual spaces)
        q1 = p1 - self.step * (self.smooth_funs[0].grad(p1) + self.Lt(p2))
        q2 = p2 + self.step * self.L(p1)

        # Update solution (in both primal and dual spaces)
        self.sol[:] = self.sol - y1 + q1
        self.dual_sol[:] = self.dual_sol - y2 + q2

class projection_based(primal_dual):
    r"""
    Projection-based primal-dual algorithm.

    This algorithm solves convex optimization problems with objective of the
    form :math:`f(x) + g(Lx)`, where :math:`f` and :math:`g` are proper,
    convex, lower-semicontinuous functions with easy-to-compute proximity
    operators.

    See generic attributes descriptions of the
    :class:`pyunlocbox.solvers.primal_dual` base class.

    Parameters
    ----------
    lambda_ : float, optional
        The update term weight. It should be between 0 and 2. Default is 1.

    Notes
    -----
    The order of the functions matters: set :math:`f` first on the list, and
    :math:`g` second.

    This algorithm requires the two functions to implement the
    :meth:`pyunlocbox.functions.func.prox` method.

    The step-size should be in the interval :math:`\left] 0, \infty \right[`.

    See :cite:`komodakis2015primaldual`, Algorithm 7, for details.

    Examples
    --------
    >>> import numpy as np
    >>> from pyunlocbox import functions, solvers
    >>> y = np.array([294, 390, 361])
    >>> L = np.array([[5, 9, 3], [7, 8, 5], [4, 4, 9], [0, 1, 7]])
    >>> x0 = np.array([500, 1000, -400])
    >>> f = functions.norm_l1(y=y)
    >>> g = functions.norm_l1()
    >>> solver = solvers.projection_based(L=L, step=1.)
    >>> ret = solvers.solve([f, g], x0, solver, maxit=1000, rtol=None, xtol=.1)
    Solution found after 996 iterations:
        objective function f(sol) = 1.045000e+03
        stopping criterion: XTOL
    >>> ret['sol']
    array([0, 0, 0])

    """

    def __init__(self, lambda_=1., *args, **kwargs):
        super(projection_based, self).__init__(*args, **kwargs)
        self.lambda_ = lambda_

    def _pre(self, functions, x0):
        super(projection_based, self)._pre(functions, x0)

        if self.lambda_ <= 0 or self.lambda_ > 2:
            raise ValueError('Lambda is bounded by 0 and 2.')

        if len(functions) != 2:
            raise ValueError('projection_based requires 2 convex functions.')

        self.non_smooth_funs.append(functions[0])   # f
        self.non_smooth_funs.append(functions[1])   # g

    def _algo(self):
        a = self.non_smooth_funs[0].prox(self.sol - self.step *
                                         self.Lt(self.dual_sol), self.step)
        ell = self.L(self.sol)
        b = self.non_smooth_funs[1].prox(ell + self.step * self.dual_sol,
                                         self.step)
        s = (self.sol - a) / self.step + self.Lt(ell - b) / self.step
        t = b - self.L(a)
        tau = np.sum(s**2) + np.sum(t**2)
        if tau == 0:
            self.sol[:] = a
            self.dual_sol[:] = self.dual_sol + (ell - b) / self.step
        else:
            theta = self.lambda_ * (np.sum((self.sol - a)**2) / self.step +
                                    np.sum((ell - b)**2) / self.step) / tau
            self.sol[:] = self.sol - theta * s
            self.dual_sol[:] = self.dual_sol - theta * t


#!/usr/bin/env python
# coding: utf-8

# # Week 1
#
# IPL (Indian Premier League) (cricket)

# # Imports

# In[1]:

# %load /Users/bartev/dev/github-bv/sporty/notebooks/imports.py
# %load /Users/bartev/dev/github-bv/sporty/notebooks/imports.py

## Where am I
get_ipython().system('echo $VIRTUAL_ENV')

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))

# magics
get_ipython().run_line_magic('load_ext', 'blackcellmagic')
# start cell with `%%black` to format using `black`

get_ipython().run_line_magic('load_ext', 'autoreload')
# start cell with `%autoreload` to reload module
# https://ipython.org/ipython-doc/stable/config/extensions/autoreload.html

# reload all modules when running
get_ipython().run_line_magic('autoreload', '2')

# In[2]:

# imports

import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
import seaborn as sns

from importlib import reload
from pathlib import Path

import matplotlib as mpl
import matplotlib.pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')

# https://plotnine.readthedocs.io/en/stable/

import plotnine as p9
from plotnine import ggplot, aes, facet_wrap

# In[30]:

get_ipython().system('pip install xlrd')

# In[32]:

get_ipython().system('pip install openpyxl')

# # Read data

# Need `IPL2018teams` data
#
# * https://www.kaggle.com/manasgarg/ipl

# ## archive
#
# 2008-2017
data_dir = Path('/Users/bartev/dev/github-bv/sporty/data/raw/wk1-IPL/archive')
matches = pd.read_csv(data_dir / 'matches.csv')

deliveries = pd.read_csv(data_dir / 'deliveries.csv')deliveries.head()matches.groupby('season').size()
# ## archive 2
# 2008 - 2019
data_dir2 = Path('/Users/bartev/dev/github-bv/sporty/data/raw/wk1-IPL/archive 2')
matches2 = pd.read_csv(data_dir2 / 'matches.csv')
deliveries2 = pd.read_csv(data_dir2 / 'deliveries.csv')matches2.groupby('season').size()deliveries2
# ## archive-3
#
# I found this data somewhere on Kaggle (didn't keep the link)
#
# More data than `archive 2`?

# In[157]:

data_dir = Path('/Users/bartev/dev/github-bv/sporty/data/raw/wk1-IPL/archive-3')
matches = pd.read_csv(data_dir3 / 'matches.csv')
deliveries = pd.read_csv(data_dir3/ 'deliveries.csv')
teams = pd.read_csv(data_dir3 / 'teams.csv')
teamwise = pd.read_csv(data_dir3 / 'teamwise_home_and_away.csv')
players = pd.read_excel(data_dir3 / 'Players.xlsx')

# In[158]:

matches.groupby('Season').size()

# In[63]:

deliveries.head()

# In[64]:

matches.head(2)

# In[159]:

teamwise

# In[ ]:

ipl18 = matches.query("Season == 'IPL-2018'").assign(
    hwin=lambda x: np.where(x["team1"] == x["winner"], 1, 0),
    awin=lambda x: np.where(x["team2"] == x["winner"], 1, 0),
    #                 htruns=lambda x: np.where(x['team1'] == x[''])
)
ipl18

# In[101]:

# get the match ids for season 2018
match_ids_2018 = ipl18["id"]

# Count runs scored by each team for season 2018
match_team_runs = (
    deliveries[["match_id", "inning", "batting_team", "bowling_team", "total_runs"]]
    .merge(match_ids_2018, left_on="match_id", right_on="id")
    .groupby(["match_id", "batting_team"])
    .agg({"total_runs": sum})
    .reset_index()
    .rename(columns={"match_id": "id"})
)
match_team_runs.head(5)

# In[133]:

ipl_cols = [
    "id",
    "Season",
    "date",
    "team1",
    "team2",
    "winner",
    "win_by_runs",
    "win_by_wickets",
    "hwin",
    "awin",
]

ipl18_exp = (
    ipl18[ipl_cols]
    .rename(columns={"team1": "home_team", "team2": "away_team"})
    .merge(
        match_team_runs, left_on=["id", "home_team"], right_on=["id", "batting_team"]
    )
    .drop(columns="batting_team")
    .rename(columns={"total_runs": "htruns"})
    .merge(
        match_team_runs, left_on=["id", "away_team"], right_on=["id", "batting_team"]
    )
    .drop(columns="batting_team")
    .rename(columns={"total_runs": "atruns"})
    .assign(
        my_hwin=lambda x: np.where(x["home_team"] == x["winner"], 1, 0),
        my_awin=lambda x: np.where(x["away_team"] == x["winner"], 1, 0),
    )
    #     .assign(match=lambda x: x['hwin'] == x['my_hwin'])
    .assign(count=1)
)

# In[134]:

ipl18_exp

# * Ph/Pa : number of games played at home/away

# In[143]:

iplhome = (
    ipl18_exp.groupby("home_team")["count", "hwin", "htruns", "atruns"]
    .sum()
    .reset_index()
    .rename(
        columns={
            "home_team": "team",
            "count": "Ph",
            "htruns": "htrunsh",
            "atruns": "atrunsh",
        }
    )
)
iplhome

# In[144]:

iplaway = (
    ipl18_exp.groupby("away_team")["count", "awin", "htruns", "atruns"]
    .sum()
    .reset_index()
    .rename(
        columns={
            "away_team": "team",
            "count": "Pa",
            "htruns": "htrunsa",
            "atruns": "atrunsa",
        }
    )
)
iplaway

# In[151]:

ipl18_combo = (
    iplhome.merge(iplaway, on="team")
    # aggregate home/away data for wins, games played, and runs
    .assign(
        W=lambda x: x["hwin"] + x["awin"],
        G=lambda x: x["Ph"] + x["Pa"],
        R=lambda x: x["htrunsh"] + x["atrunsa"],
        RA=lambda x: x["atrunsh"] + x["htrunsa"],
    )
    # get win percentage and pythagorean expectation
    .assign(wpc=lambda x: x['W'] / x['G'],
           pyth=lambda x: x['R']**2 / (x['R']**2 + x['RA']**2))
)

ipl18_combo

# In[152]:

sns.relplot(x='pyth', y='wpc', data=ipl18_combo)

# # Run a regression

# In[154]:

pyth_lm = smf.ols(formula = 'wpc ~ pyth', data=ipl18_combo).fit()
pyth_lm.summary()

# Interpretation
#
# * pyth
#     * coef = 2.5
#     * std err = 2.077
#     * t value = 1.235
#     * P-value = 0.263 >> 0.05
#     * P-value = probability that we'd observe the value 2.5 (for the coef) by chance if the true value were 0
#     * p-values > 0.05, so values are not considered statistically significant.
#         We have no confidence in this relationship.
#     * R^2 is low (20%)

# In[ ]:


'''
13 - Customizing your pandas import

The pandas package is also great at dealing with many  of the  issues  you  will
encounter when importing data as a data  scientist,  such  as comments occurring
in flat files, empty lines and missing values. Note that missing values are also
commonly referred to as NA or NaN. To  wrap  up this chapter, you're now going to
import a slightly corrupted copy of the Titanic dataset titanic_corrupt.txt, which

    - contains comments after the character '#'

    - is tab-delimited.

Instructions:

- Complete the `sep` (the pandas version of delim), comment and na_values arguments of
  pd.read_csv(). comment takes characters that comments occur after in the file, which
  in this case is '#'. na_values takes a list of strings to recognize as  NA/NaN,  in
  this case the string 'Nothing'.

- Execute the rest of the code to print the head of the resulting DataFrame and plot
  the histogram of the 'Age' of passengers aboard the Titanic.
'''
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Import matplotlib.pyplot as plt
import matplotlib.pyplot as plt

# Assign filename: file
file = 'titanic_corrupt.txt'

# Import file: data
data = pd.read_csv(file, sep='\t', comment='#', na_values='Nothing')

# Print the head of the DataFrame
print(data.head())

# Plot 'Age' variable in a histogram
pd.DataFrame.hist(data[['Age']])
plt.xlabel('Age (years)')
plt.ylabel('count')
plt.show()

'''
data.head()
---------------------------------------------------------------------------------------------------------
      PassengerId  Survived  Pclass     Sex   Age  ...  Parch            Ticket    Fare  Cabin Embarked
    0            1         0       3    male  22.0  ...      0         A/5 21171   7.250    NaN       S
    1            2         1       1  female  38.0  ...      0          PC 17599     NaN    NaN      NaN
    2            3         1       3  female  26.0  ...      0  STON/O2. 3101282   7.925    NaN        S
    3            4         1       1  female  35.0  ...      0            113803  53.100   C123        S
    4            5         0       3    male  35.0  ...      0            373450   8.050    NaN        S

    [5 rows x 11 columns]
---------------------------------------------------------------------------------------------------------
'''

14_Introduction to Importing Data in Python-(part-1)/01_Introduction and flat files/13_customizing-your-pandas-import.py


#!/usr/bin/env python
# coding: utf-8

# In[1]:

import numpy as np
import panel as pn

pn.extension("katex")

# # Lecture 08 - Wells*
#
# (_The contents are based on the class lecture materials of Prof. R. Liedl. Modifications mostly to fit this specific format were done by Prof. Liedl and Dr. P. K. Yadav._)
#
# ---
#
# ## Motivation
#
# In the last lecture ({doc}`/contents/flow/lecture_07/17_quantify_flow` we derived system equations for different groundwater flow problems. We realized of the difficulties associated with solving flow problems specially at higher dimensions (2D/3D). Numerical methods are mostly used for solving groundwater problems but direct (analytical) solutions are also possible for some problems.
#
# **Wells** are the most common and also most extensively used method of utilizing (or affecting) groundwater. Thus, _wells_ represent a very common groundwater problem. We can now use our understanding of aquifer properties and groundwater system equations to analyze effect of _wells_ on the natural groundwater flow. We will however restrict our extent, in this lecture, to problems that can be directly solved. After learning numerical methods (in last part of this course), we will apply it to evaluate more complex groundwater problems also associated with wells.
#
# We begin this lecture recalling groundwater storage property _transmissivity._ Introducing _wells_ we then derive few relations that can help us understand the effect of wells in groundwater flow. To conclude, we will use the _wells_ to characterize aquifers, i.e., using so called pumping test.
#
# (**_The contents are based on the class lecture materials of Prof. R. Liedl. Modifications mostly to fit this specific format were done by Prof. Liedl and Dr. P. K. Yadav._**)

# ## Transmissivity ##
#
# When discussing storage properties in {doc}`/contents/flow/lecture_03/13_gw_storage`, we saw that aquifers or single layers may frequently be treated as two-dimensional systems. This is justified because the lateral extension of aquifers is usually much larger than the vertical extension. Thus, vertical variations of storage properties can be replaced by some average value without adversely affecting the quantification of groundwater storage.
#
# Similar things can be done with regard to conductivity properties and this brings us to the geohydraulic parameter of transmissivity ($T$,  L$^2$T$^{-1}$). The idea is to neglect vertical variations of hydraulic conductivity and to use vertically averaged values instead. This procedure does not eliminate horizontal variability, so transmissivity may still depend on horizontal coordinates  $(x, y)$.
#
# The vertically averaged $K$ value is then multiplied by the water-saturated thickness to obtain transmissivity. The concept of water-saturated thickness (or water-saturated depth) requires to distinguish whether _confined_ or _unconfined flow_ conditions prevail.
#
# In general, water-saturated thickness is the distance from the aquifer bottom to a level up to which all pores are filled with water. For _confined aquifers,_ this level is equal to aquifer top and water-saturated thickness is tantamount to aquifer thickness. For _unconfined aquifers,_ however, water-saturated thickness corresponds to the distance between aquifer bottom and groundwater level. We will see some illustrations below when we try to quantify transmissivity.
#
# Let us have a closer look at the confined case first. The black cuboid in {numref}`Trans_c_2D`  illustrates that water-saturated thickness extends from aquifer bottom to aquifer top. So, it is equal to aquifer thickness $m$. Transmissivity is calculated by $T_x = K_x \cdot m$ and $T_y = K_y \cdot m$. Here we allow for horizontal aquifer anisotropy with different hydraulic conductivities in $x-$ and $y-$ direction $(K_x\neq K_y)$. For horizontally isotropic aquifers $(K_x = K_y = K)$, transmissivity is given by $T = K \cdot m$.
#
# ```{figure} images/L08_f1.png
# ---
# scale: 40%
# align: center
# name: Trans_c_2D
# ---
# The transmissivity cuboid in confined aquifer.
# ```
# Things are a bit more complicated for unconfined aquifers.  {numref}`Trans_u_2D` illustrates that water-saturated thickness extends from the aquifer bottom to the groundwater table. It is important to note that transmissivity of unconfined aquifers depends on the vertical position of the groundwater table. For instance, if the groundwater table is lowered during to a draught period, transmissivity is decreasing. This is fundamentally different from the confined case where the water-saturated thickness is given by aquifer geometry only and is not affected by hydraulic head changes.
#
# ```{figure} images/L08_f2.png
# ---
# scale: 40%
# align: center
# name: Trans_u_2D
# ---
# The transmissivity cuboid in unconfined aquifer.
# ```
#
# Computing transmissivity of unconfined aquifers requires to determine the difference of hydraulic head h and the elevation of aquifer bottom $z_{bot}$. Based on this, transmissivity is given by $T_x = K_x\cdot(h - z_{bot})$ and $T_y = K_y\cdot(h - z_{bot})$. As above, we are allowing for horizontal aquifer anisotropy. For an isotropic unconfined aquifer we get $T = K\cdot(h - z_{bot})$.

# Two more remarks appear to be appropriate: First, transmissivity may be computed by the given equations even if the aquifer bottom is not horizontal. This case is not covered by the {numref}`Trans_u_2D`. Second, textbooks frequently present the equation **$T = K\cdot h$** for transmissivity of unconfined aquifers. It is to be noted that this equation only holds if two conditions are fulfilled:
#
# ```{admonition} Required conditions when $T = K\cdot h$ is valid
# :class: note
# 1. The aquifer bottom must be horizontal, and
#
# 2. hydraulic head values are expressed with respect to the elevation of aquifer bottom (= reference datum).
# ```
# Finally, we can try to compute transmissivity for isotropic aquifers and check how the result depends on several quantities like aquifer bottom, aquifer top, and hydraulic head.

# ### Example problem  ###
#
# ```{admonition} Transmissivity
# Find if the aquifer is confined or unconfined, and then calculate transmissivity of the aquifer.
# ```

# In[2]:

#print("\n\033[1m Provided are:\033[0m\n") # \033[1m \033[0m bold font, \n - new line

K_a = 8.5e-05 # m/s, Hydraulic conductivity
Z_bot = 120 # m, aquifer bottom
Z_top = 150 # m, aquifer top
h_a   =  139 # m, hydraulic head in aquifer

# interim calculation
A_t = Z_top-Z_bot # m, Aquifer thickness
A_wt = h_a - Z_bot # m, water_table level
S_t = min(A_t, A_wt) # m, saturated thickness

# result
if h_a<Z_top:
    print("It is Unconfined Aquifer \n")
else:
    print("It is Confined Aquifer \n")

T_a = K_a*S_t # m^2/s, transmissivity

print("The required transmissivity is {0:1.2e}".format(T_a), "m\u00b2/s")

# ## Wells - Overview ##
#
# ### What is a Well? ###
#
# A **well** is a shaft or a hole that has been sunk, dug or drilled into the earth to extract water (source: [Glossary of Hydrology](https://hydrologie.org/glu/HINDEN.HTM)).
#
# ```{figure} images/L08_f3.png
# ---
# scale: 50%
# align: center
# name: Well
# ---
# Well and its components.
# ```
#
# ### Using wells ###
#
# Wells are very extensively used around the globe. The contents below only highlights few of the use of wells.
#
# ```{margin}
# <img src=" images/L08_f4.png" alt="gw-sediment" class="bg-primary" width="300px">
# ```
#
# > Water supply: e.g., for households, agriculture, industry
#
# > Lowering the groundwater level: e.g., for excavations, open-pit mining
#
# > Remediation of aquifer contamination: e.g., applying pump and treat method.
#
# > Aquifer characterisation: e.g., using pumping test (this lecture)
#
# Apart from aquifer characterisation, wells are usually operated at steady-state i.e., at constant pumping rate. The figure below presents different uses of wells. {numref}`Well-ex` shows the case of lowering the groundwater level at the excavation site. The lowering is very often observed at the works that requires sub-surface construction works, e.g., high-rise building, tunnels. For this wells are placed close to excavation works and water is pumped out at higher discharge rates compare to the groundwater replenishing rate. This leads to decline of water level at the excavation site.
#
# ```{figure} images/L08_f5.png
# ---
# scale: 40%
# align: center
# name: Well-ex
# ---
# Well at the excavation site
# ```
#
# {numref}`Well-refuse` presents the case of using wells to delineate contamination site from the groundwater. These are part of the development of sanitary landfills sites or industries that pose threat to groundwater quality. Wells are used to lower groundwater table such that seepage of refuse is contained in the limited region.
#
# ```{figure} images/L08_f6z.png
# ---
# width: 30 cm
# height: 8 cm
# align: center
# name: Well-refuse
# ---
# Well at the refuse site
# ```

# ### Fully versus Partially Penetrating Wells ###
#
# Fully penetrating wells:
# : The fully penetrating wells are those which extends through the whole saturated depth of an aquifer and are constructed in such a manner that water is permitted to the well screen over its length [Glossary of Hydrology](https://hydrologie.org/glu/HINDEN.HTM)).
#
# {numref}`full-pen` shows a schematic of water flow in a fully penetrating wells.
#
# ```{figure} images/L08_f7x.png
# ---
# scale: 40%
# align: center
# name: full-pen
# ---
# Flownet in a fully penetrating wells
# ```
#
# Partially penetrating Wells:
# : The partially penetrating Wells are those in which the length of water entry is less than the thickness of the saturated aquifer which it penetrates ([Glossary of Hydrology](https://hydrologie.org/glu/HINDEN.HTM)).
#
# Partially penetrating wells are constructed when aquifer depths are very high; in which case a fully penetrating well may not also be economical. {numref}`par-pen` presents a schematic of a flownet that is likely to be observed in the partially penetrating well.
#
# ```{figure} images/L08_f7y.png
# ---
# scale: 40%
# align: center
# name: par-pen
# ---
# Flownet in a partially penetrating wells
# ```
#
# Comparing the Flownets in {numref}`full-pen` and {numref}`par-pen`, it can be observed that the vertical flow components can be significant in the partially penetrating case. This then lead to a 3D groundwater problem compared to the fully penetrating case, which can be treated as 2D groundwater problem as vertical flow component has limited effect.
#
# This course we will deal only with the fully penetrating steady-state cases.
#

# ## Groundwater Flow Near Wells Operated at Steady State ##
#
# As stated earlier wells are mostly operated under steady-state condition. Here will attempt to quantify the case. We will first attempt to identify the most relevant problems associated with the steadily pumping wells and then define our approach to solve them. The **two** most important points that need to be addressed here  are:
#
# > Which relevant quantities are needed to describe steady-state flow
# towards a well?
#
# > What is the quantitative relationship between the hydraulic
# parameters under steady-state conditions?
#
# To answer the above questions, we follow the following approach:
#
# > Find an appropriate way to apply the _law of continuity_
# (conservation of volume) and _Darcy‘s law._
#
# > We delineate the problem, e.g., study the _confined_ and the _unconfined_ cases separately.
#

# ### Cone of Depression in a Confined Aquifer ###
#
# A **cone of depression** (or drawdown cone) will result when the well pumps groundwater groundwater is pumped from the well. {numref}`cone-con-un` presents the schematic with relevant quantities of a well pumping case in both confined and unconfined aquifers. In both cases the well is fully penetrating. As can be understood from the figure that in the unconfined case the drawdown ($s$) magnitude depends on the water table level. This in the case of confined aquifer is dependent on the hydraulic head.
#
#
# ```{figure} images/L08_f8-9.png
# ---
# scale: 25%
# align: center
# name: cone-con-un
# ---
# Cone of depression in a (a) confined Aquifer and (b) unconfined aquifer (right)
# ```
#
# ```{margin} Relevant quantities in the figure:
# + pumping rate $Q$ [L$^3$T$^{-1}$]
# % free space
# + aquifer thickness $m$ [L]
# % free space
# + hydraulic conductivity $K$ [LT$^{-1}$]
# % free space
# + water level at rest $H$ [L]
# % free space
# + water-level in the well $h$ [L]
# % free space
# + Radius of influence $R$ [L]
# % free space
# + well radius $r_w$ (incl. gravel pack!)[L]
# % free space
# + drawdown $s = H - h$ [L]
# ```
#
# The Radius of influence ($R$) in the figure is the distance from the well at which the drawdown becomes negligible or unobservable. Thus $R$ delineates the influence of the well on the normal groundwater flow.
#
# #### Evolution of Cone of Depression with Time ####
#
# Transient models have to be used to observe the evolution of cone of depression. These are mostly only possible through use of numerical models (to be discussed in the later end of this course). The animation provides the computer simulation of evolving cone of depression as a function of time. From the figure it is easier to obtain the radius of influence ($R$). It is to be noted that $R$ will be maximum at the steady-state condition.

# In[3]:

video1 = pn.pane.Video("images/L08_f9X.mp4", width=400, height=150, loop=False)
spacer = pn.Spacer(width=50)
image = pn.pane.PNG("images/L08_f9Y.png", width=200)
pn.Row(video1, spacer, image)

# ### Law of Continuity ###
#
# Next we attempt to quantify the cone of depression. For this, and it is common in any hydraulics study, we begin with the _Law of Continuity._ Overall law of continuity implies:
#
# ```{admonition} Law of Continuity
# Discharge $Q_w = $ constant
# ```
#
# ```{figure} images/L08_f10X.png
# ---
# scale: 40%
# align: center
# name: law-cont
# ---
# Graphic visualization of Law of quantity.
# ```
#
# This, also seen in {numref}`law-cont`, implies that the pumping rate $Q$ corresponds to the discharge $Q_w$ near the well. Further, under steady-state conditions the law of continuity
# implies that there is the same discharge at all cross sections which
# completely surround the well ("mantle of a cylinder").
#
# ### Darcy's Law for Flow Towards a Well in a Confined Aquifer ###
#
# The Darcy's law, as we have known so far,  relates the Darcy's velocity $v_f$ and the hydraulic gradient $i$. If the hydraulic gradient was constant in space, the hydraulic gradient would be (also see {numref}`law-cont` )
#
# $$
# i = \frac{\Delta h}{\Delta r}
# $$
#
# where, $r$ represents the radial distance from the well axis. In this case the hydraulic gradient $i$ depends on the distance $r$ within a cone of depression {numref}`law-cont`). Qualitatively, it can be understood that $i$ is decreasing with increasing distance from the well. Since the hydraulic gradient is not constant in space, the ratio $\frac{\Delta h}{\Delta r}$ has to be replaced by
#
# $$
# i(r) = \frac{\textrm{d} h (r)}{\textrm{d} r} \tag{C1}
# $$
#
# With this defined, the Darcy's law should be expressed as function of $r$ as
#
# $$
# v_f (r) = - K\cdot i(r) = - K \cdot \frac{\textrm{d} h (r)}{\textrm{d} r} \tag{C2}
# $$
#
# And, now introducing the continuity equation we obtain:
#
# $$
# Q_w = A\cdot v_f (r) = - K\cdot i(r) = - K \cdot A\cdot\frac{\textrm{d} h (r)}{\textrm{d} r} \tag{C3}
# $$
#
# with $A$ is the area of a cross section at distance $r$ from the well axis. Therefore $A$ is
#
# $$
# A = 2\cdot \pi \cdot r \cdot m \tag{C4}
# $$
#
# $A$ is now inserted to eq. (C3), This results to
#
# $$
# Q_w =  - 2\cdot \pi \cdot r \cdot m \cdot K \cdot \frac{\textrm{d} h (r)}{\textrm{d} r} \tag{C5}
# $$
#
# ```{margin}
# <img src="images/L08_f8.png" alt="Confined Aquifer" class="bg-primary" width="400px">
# ```
#
# Eq (C5) is a first-order differential equation for hydraulic head $h(r)$. It can be solved by separation of variable. Doing that eq. (C5) becomes
# $$
# \textrm{d}h(r) = \frac{Q_w}{2\cdot \pi \cdot m \cdot K }\cdot \frac{\textrm{d}r}{r} \tag{C6}
# $$
#
# We now integrate eq. (C6). The limit of integration (see figure) along the vertical direction ($h$) will be from $h$ to $H$, and that along the radial axis will be from $r_w$ and $R$, i.e., we get
#
# $$
# \int\limits_h^H\textrm{d}h(r) = \frac{Q_w}{2\cdot \pi \cdot m \cdot K }\cdot \int\limits_{r_w}^R\frac{\textrm{d}r}{r} \tag{C7}
# $$
#
# The integrals in eq. (C7) are direct integrals and can be obtained from the table of integrals. The eq. (C7) with the indefinite integrals is thus
#
# $$
# \big[h(r)\big]_h^H = - \frac{Q_w}{2\cdot\pi\cdot m \cdot K}\cdot \big[\ln r\big]_{r_w}^R
# $$
#
# after inserting the limits of integration, we get
#
# $$
# H - h = - \frac{Q_w}{2\cdot\pi\cdot m \cdot K}\cdot (\ln R - \ln r_w) \tag{C8}
# $$
#
# Finally, we can solve for the discharge $Q_w$ and obtain
#
# $$
# Q_w = - \frac{2\cdot\pi\cdot m \cdot K \cdot (H-h)}{(\ln R - \ln r_w)}\tag{C9}
# $$
#
# The **negative** sign on the right-hand side of eq. (C9) indicates that flow is anti-parallel to the direction of the coordinate axis. Frequently, the pumping rate $Q$ is used instead of $Q_w$ and the negative sign is omitted. Thus we get a well known solution called Theim equation after Thiem (1906)<sup>[^Theim]</sup>
#
# ````{panels}
# :container: container pb-4
# :column: col-lg-8 p-2
#
# **The discharge from the well in confined aquifer after Thiem (1906)**
# ^^^
# $$
# Q = \frac{2\cdot\pi\cdot m \cdot K \cdot (H-h)}{(\ln R - \ln r_w)}  =  \frac{2\cdot\pi\cdot m \cdot K \cdot (H-h)}{\ln (R/r_w)}
# $$
# ````
#
# with decadic logarithm, we get,
#
# $$
# Q_w = \frac{2\cdot\pi\cdot m \cdot K \cdot (H-h)}{2.3\cdot(\log R - \log r_w)}  =  \frac{2\cdot\pi\cdot m \cdot K \cdot (H-h)}{2.3 \cdot\log (R/r_w)}
# $$
#
# [^Theim]: Thiem, G. (1906), _Hydrologische Methoden_, 56 pp., Gephardt, Leipzig, Germany.

# ### Example problem  ###
#
# ```{admonition} Well discharge from confined aquifer
# From the provided data, calculate the transmissivity of the aquifer.
# ```

# #### Solution ####
#
# For confined aquifer $T = K\cdot m$, the Thiem equation can be modified as:
#
# $$
# Q = \frac{2\cdot\pi\cdot T \cdot (H-h)}{\ln (R/r_w)}
# $$

# In[4]:

print("\n\033[1m Provided are:\033[0m\n")

Q = 9 # m^3/min, Given discharge
r1 = 8 # m, distance from well to point 1
h1 = 9 # m, head at well 1
R2 = 22 # m, distance from well to point 2
H2 = 10 # m, head at well 2

print(" The given dscharge is: {}".format(Q), "m\u00b3/min \n")
print(" The distance to Well 1 and well 2 are: {}m and {}m  \n".format(r1, R2))
print(" The head at Well 1 and well 2 are: {}m and {}m".format(h1, H2))

#interim calculation
Q_min = Q * 1440 # m^3/d

#Calculation

T = Q/(2*np.pi*(H2-h1))*np.log(R2/r1) # m^2/d, Transmissivity - inverting Theim equation

print("\n\033[1m Result:\033[0m\n")
print("The transmissivity in the aquifer is {0:0.2f} m\u00b2/d".format(T))

# ## Flow Towards a Well in an Unconfined Aquifer ##
#
# In the unconfined aquifer, the hydraulic head is function of $r$. Thus the discharge in the aquifer:
#
# $$
# Q_w = A \cdot v_f = - A \cdot K \frac{\textrm{d}h}{\textrm{d}r}(r) \tag{U1}
# $$
#
# ```{margin}
# <img src="images/L08_f9.png" alt="Unconfined Aquifer" class="bg-primary" width="400px">
# ```
# Also in this case, the cross section area $A$ also is dependent on $r$, i.e.,
#
# $$
# A = 2 \cdot \pi \cdot r \cdot h(r) \tag{U2}
# $$
#
# Next, we insert eq (U2) in eq. (U1) and get
#
# $$
# Q_w = A \cdot v_f = - 2 \cdot \pi \cdot r \cdot h(r)  \cdot K \frac{\textrm{d}h}{\textrm{d}r}(r) \tag{U3}
# $$
#
# Eq. (U3) is a first-order differential equation for hydraulic head $h(r)$, and it can be solved by separation of variables. Separating the variables of eq. (U3) leads to
#
# $$
# h(r) \cdot \textrm{d}h(r) = - \frac{Q_w}{2 \cdot \pi \cdot K }\cdot \frac{\textrm{d}r}{r} \tag{U4}
# $$
#
# Next we integrate eq. (U4) from limits $r_w$ to $R$ in the right-hand side, and from $h$ to $H$ (left-hand side), i.e.,
#
# $$
# \int\limits_h^H h(r) \cdot \textrm{d}h(r) = - \frac{Q_w}{2 \cdot \pi \cdot K }\cdot \int\limits_{r_w}^{R}\frac{\textrm{d}r}{r} \tag{U5}
# $$
#
# The integral in eq. (U5) can be obtained from the standard table of integrals. With that we get
#
# $$
# \frac{1}{2}(H^2 - h^2) = - \frac{Q_w}{2 \cdot \pi \cdot K }\cdot (\ln R - \ln r_w) \tag{U6}
# $$
#
# The discharge in aquifer $Q_w$ can be obtained from eq. (U6) and the expression is
#
# $$
# Q_w = -\frac{\pi\cdot K \cdot (H^2-h^2)}{\ln R - \ln r_w}\tag{U6}
# $$
#
# The negative sign on the right-hand side of the eq. (U6) indicates that flow is anti-parallel to the orientation of the coordinate axis. Frequently, the pumping rate $Q$ is used instead of $Q_w$ and the negative sign can thus be
# omitted, and we get
#
# ````{panels}
# :container: container pb-4
# :column: col-lg-10 p-2
#
# **Discharge $Q$ from well in unconfined aquifer modified from Thiem (1906)**
# ^^^
# $$
# Q_= \frac{\pi\cdot K \cdot (H^2-h^2)}{\ln R - \ln r_w} = \frac{\pi\cdot K \cdot (H^2-h^2)}{\ln (R/r_w)}
# $$
# ````
#
# In the decadic logarithm, the discharge from the well is:
#
# $$
# Q_= \frac{\pi\cdot K \cdot (H^2-h^2)}{2.3(\log R - \log r_w)} = \frac{\pi\cdot K \cdot (H^2-h^2)}{2.3\log (R/r_w)}
# $$

# ### Example problem  ###
#
# ```{admonition} Well discharge from unconfined aquifer
# From the provided data, calculate discharge of the aquifer.
# ```

# In[5]:

print("\n\033[1m Provided are:\033[0m\n")

K = 24.50 # m/d, conductivity
r_1 = 0.23 # m, distance from well to point 1
h_1 = 12 # m, head at well 1
R_2 = 275 # m, distance from well to point 2
H_2 = 18 # m, head at well 2

print(" The given conductivity is: {}".format(K), "m/d \n")
print(" The distance to Well 1 and well 2 are: {} m and {} m are \n".format(r_1, R_2))
print(" The head at Well 1 and well 2 are: {} m and {} m".format(h_1, H_2))

#Calculation

Q_1 = (np.pi*K*(H_2**2-h_1**2))/(np.log(R_2/r_1)) # m^2/d, Transmissivity - inverting Theim equation

print("\n\033[1m Result:\033[0m\n")
print("Discharge from the well is {0:0.2f} m\u00b3/d".format(Q_1))

# ### Radius of Influence ###
#
# The radius of influence ($R$) can also (instead of using numerical simulations) be obtained from empirical equations. The table below provides a list of few equations.
#
# ```{margin} The symbols in equations are:
# + $s$ = drawdown in pumping well,
# + $t$ = pumping time,
# + $N$ = groundwater recharge,
# + $S$ = storage coefficient,
# + $K$ = hydraulic conductivity,
# + $H$ = water level at rest (unconfined aquifer).
#
# In confined aquifers, $H$ has to be replaced by the aquifer thickness $m$.
# ```
#
# | Source                             |   | Equation                                  |
# |------------------------------------|---|-------------------------------------------|
# | Lembke (1886, 1887)                |   | $R = H (K/2\cdot N)^{1/2}$                |
# | Weber (Schultze, 1924)             |   | $R = 2.45 (H\cdot K\cdot t/S)^{1/2}$      |
# | Kusakin (Aravin and Numerov, 1953) |   | $R = 1.9 (H \cdot K \cdot t/s)^{1/2}$     |
# | Siechardt (Certousov, 1962)        |   | $R = 3000\cdot s \cdot K^{1/2}$           |
# | Kusakin (Certousov, 1949)          |   | $R = 575\cdot s \cdot (H \cdot K)^{1/2} $ |
#
#
# **Siechardt** and **Kusakin** equation are among the preferred equations by practitioners. In both equations, $K$ has to be expressed in _m/s_ and all other quantities must be expressed in _m._ _R_ depends on drawdown $s= H-h$ in both equations. Trial and error or iterative strategies have to be used to determine $R$ and $h$.
#
#

# ## Aquifer Characterisation by Pumping Tests ##
#
# Pumping tests are used to estimate aquifer properties such as hydraulic conductivity $(K)$, transmissivity $(T)$ or storativity $(S)$. Pumping results in an evolving cone of depression as was discussed earlier (see {numref}`cone-con-un`). The decrease in hydraulic head (or increase in drawdown) with time is recorded in one or more observation wells (and sometimes also in the pumping well itself).
#
# ```{margin}
# <img src="images/L08_f13.png" alt="Theis" class="bg-primary" width="200px">
# ```
#
# A variety of different schemes exist to evaluate pumping test data. The appropriate method has to be selected according to the specific setting (confined or unconfined, layered system, horizontal or inclined aquifer bottom etc.). A well known approach to derive $T$ and $S$ from pumping test data was developed by Theis (1935)<sup>[^Theis]</sup>
#
# ### Applicability of the Theis Method ###
#
# Pumping test data can be evaluated according to Theis (1935) if the following assumptions are (approximately) justified:
#
# + The aquifer is confined, homogeneous and isotropic.
# %
# + The aquifer thickness is uniform.
# %
# + The aquifer bottom is horizontal.
# %
# + The well is fully penetrating.
# %
# + The well radius is very small as compared to the radius of influence.
# %
# + The pumping rate is constant within the measurement period.
# %
# + There is no vertical flow component.
# %
# + The evolution of the cone of depression is not influenced by other hydraulic factors (surface water, impermeable boundaries etc.).
#
# [^Theis]: Theis, C.V., 1935. The relation between the lowering of the piezometric surface and the rate and duration of discharge of a well using groundwater storage, Am. Geophys. Union Trans., vol. 16, pp. 519-524.

# ### Drawdown According to Theis (1935) ###
#
# Theis (1935) deals with the transient flow (as opposed to steady-state methods discussed above) of water to a pumping well. The time-dependent drawdown $s$ in an observation well, which is a distance $r$ apart from the pumping well, is given by
#
# ```{margin}
# + $s$ is drawdown [L]
# %
# + $Q$ is pumping rate [L$^3$T$^{-1}$]
# %
# $T$ is transmissivity [L$^2$T$^{-1}$]
# %
# $W(u)$ is the well function
# ```
#
# $$
# s(r,t) = \frac{Q}{4\pi T}\cdot W(u) \tag{T1}
# $$
#
# where, $W(u)$ is the well function, which is given as
#
# $$
# W(u) = \int\limits_u^\infty \frac{\textrm{e}^{-\widetilde{u}}}{\widetilde{u}}\textrm{d}\widetilde{u}
# $$
#
# in which $u$ is defined by
#
# $$
# u = \frac{Sr^2}{4Tt} \tag{T2}
# $$
#
# where $S$ [-] is storage coefficient. For application Eq (T1) and eq. (T2), these equation are log-transformed. Eq (T1) then becomes
#
# $$
# \log s(r,t) = \log \frac{Q}{4\pi T}\cdot \log W(u) \tag{T3}
# $$
#
# and upon log-transformation and rearrangement of eq. (T2) results to
#
# $$
# \log \frac{t}{r^2} = \log\frac{S}{4T} + \log\frac{1}{u} \tag{T4}
# $$
#
# These two equations are used to derive $T$ and $S$ from drawdown data. This can either be done by applying special computer software or manually by a graphical method, which is discussed next.
#

# ### Manual Comparison of Data and Type Curve ###
#
# Theis (1935) provides a graphical approach to use the  Theis equation. The following steps are to be followed for using the graphical approach:
#
# + The logarithm of drawdown ($\log s$) is plotted against the $\log(t/r^2)$ in the data sheet
#
# + The logarithm of the well function $(\log W(u))$ is plotted against $\log (1/u)$ in a type curve sheet.
#
# ```{sidebar} Type Curve and Data sheet
# <img src="images/L08_f11.png" alt="Type-Curve" class="bg-primary" width="1000px">
# ```
#
# + Both sheets are put on top of each other such that the data coincide with some part of the type curve.
#
# + The shifts along the vertical and the horizontal axes correspond to the constant terms in the equations
#
# $$
# \log s = \log\frac{Q}{4\cdot \pi \cdot T} + \log W(u)
# $$
#
# $$
# \log \frac{t}{r^2} = \log\frac{S}{4 T} + \log \frac{1}{u}
# $$
#
# + The constant term in the upper equation ($\log\frac{Q}{4\cdot \pi \cdot T}$) can then be solved for $T$.
#
# + Finally, the constant term $(\log\frac{S}{4 T})$ in the lower equation can be used to solve for $S$.
#
# ```{important}
# The Type curve is independent from aquifer properties.
# ```

# ### Example  ###
#
# ```{admonition} Using Type curve
# For the provided pumping data, find the Transmissivity and Storage coefficient when the steady-discharge was 26.7 L/s.
# ```

# #### Solution ####
#
# The practical application of the Theis method is facilitated by selecting a match point in the range of the data such that corresponding values $W_A$
# and $1/u_A$ are "simple".
#
# <img src="images/L08_f14.png" alt="Type-Curve-Example" class="bg-primary" width="600px">
#
# In the example:
#
# $$
# W_A = 10^0 = 1
# $$
#
# $$
# 1/u_A = 10^2 = 100
# $$
#
# Next, values for $s$ and $t/r^2$ at the match points are determined. In this example:
#
# $$
# S = 0.2
# $$
#
# $$
# t/r^2 = 0.57
# $$
#
# Next, we obtain $T$ from $T = \frac{Q}{4\cdot\pi\cdot s }\cdot W_A$. For $Q = 26.7$ L/s, we obtain $T = 1.06 \cdot 10^{-2} $ m<sup>2</sup>/s. Finally, $S$ is obtained from $S = \frac{4 T\cdot t/r^2}{1/u_A}$. In this example:
# $S = 2.42 \cdot 10^{-4}$

# ### Computer-Based Comparison of Data and Type Curve ###
#
# The data in the type-curve can now easily be fitted using computer simulations. The {numref}`Type-Cur-com` is from the {doc}`/contents/tutorials/tutorial_07/tutorial_07`. The simulation tool {doc}`/contents/tools/type_curve_fit` provided can be used to fit user data to the type curve.
#
# ```{figure} images/L08_f15.png
# ---
# scale: 20%
# align: center
# name: Type-Cur-com
# ---
# Computationally fitted data to the Type curve
# ```

# In[ ]:


#!/usr/bin/env python
# coding: utf-8

# In[1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from xlrd import *
import ipysheet as ips
import panel as pn
get_ipython().run_line_magic('matplotlib', 'inline')
from scipy import stats
pn.extension('katex')

# ## Groundwater Exam Solution -  2019-2020 ##
#
# _(The contents presented in this section were re-developed principally by Dr. P. K. Yadav with supervision from Prof. Rudolf Liedl)_

# **Q1. Aquifer Types**   (ca. 5 pts.)
#
#
# a. Differentiate between Aquifer, Aquitard and Aquiclude (3 points)
#
# b. Schematically present a confined aquifer (vertical cross-section) providing essential features with their legends (2 points)

# **Solution 1. a.**
#
# See slide: L03/08
#
# An **aquifer** or a groundwater reservoir can store and transmit significant (= exploitable) amounts of groundwater.
#
# An **aquitard** can store and transmit groundwater but to a much lesser extent than an (adjacent) aquifer.
#
# An **aquiclude** can store groundwater but cannot transmit groundwater.
#

# **Solution 1b** - (L03/11)
#
#
#
#
#
# **Confined Aquifer**
#
# 1. The essential feature of confined aquifer is provided in the figure above.
#

# **Q2. Groundwater storage (3 pts.)**
#
# Dry season in Dresden (2018-19) led to intense extraction of groundawater in rural areas. At one location in a confined aquifer (storage coeff. 4·10-4, total porosity = 30%),  the pressure head was lowered by 150 m. The thickness of the aquifer was measured to be 90 m before the beginning of extraction and the compressibility of the porous medium in that region is estimated 6·10-8 m2/N. Density of water can be assumed to be 1000 kg/m3.
#
# (Hint: $\Delta V_T = \alpha_{pm}\cdot\rho_w \cdot g \cdot  V_T \cdot  \Delta \psi $).
#
#
# a. Approximately how much water was extracted? (1 point)
#
# b. How much land subsidence due to water extraction is expected? (2 point)
#

# **Solution 2**
#
# Given relation:
#
# For part a. (see Tut 02/P4)
#
# $
# S_s  = \frac{\Delta V_w}{V_T\cdot \Delta \psi}
# $
#
# In confined aquifer $S$ is used, which is obtained from:
#
# $S = S_s \cdot m$
#
# $
#  \frac{S}{m} = \frac{\Delta V_w}{A_T\cdot m \cdot \Delta \psi}
# $
#
# So,
#
# $  \frac{\Delta V_w}{A}  = S \cdot \Delta \psi$
#
#
# For part b.
#
#
# $\Delta V_T = \alpha_{pm}\cdot\rho_w \cdot g \cdot  V_T \cdot  \Delta \psi $
#
# $V_T = A\times h$, with $A$ surface area and $h$ aquifer thickness.
#
# $\Delta V_T = A \times \Delta h$, with $\Delta h$ change in thickness.
#
# $A \times \Delta h= \alpha_{pm}\cdot\rho_w \cdot g \cdot  A \cdot h \cdot  \Delta \psi $
#
# $\Delta h = \alpha_{pm}\cdot\rho_w \cdot g \cdot h \cdot  \Delta \psi$, with $\Delta h$ being the land subsidence

# In[2]:

#  solution 2 a

# Given

A = 1 # m², assuming 1 m² aquifer area
h = 90 # m, aquifer height before extraction
d_psi = 150 # m, change in pressure head
S_2 = 4*10**-4 #  specific storage
rho_w = 1000 # Kg/m³, density of water
a_pm = 6* 10**-8 # m²/N = m-s²/kg, compressibility of porous medium
g = 9.81 # m/s², gravity factor

#Solution

d_V_w  = S_2*A*d_psi

print("The water abstraction volume per m\u00b2 aquifer is {0:0.3f}".format(d_V_w), "m\u00b3")

# In[3]:

#  solution 2 b

# Given

A = 1 # m², assuming 1 m² aquifer area
h = 90 # m, aquifer height before extraction
d_psi = 150 # m, change in pressure head
Ss = 4*10**-4 # specific storage
rho_w = 1000 # Kg/m³, density of water
a_pm = 6* 10**-8 # m²/N = m-s²/kg, compressibility of porous medium
g = 9.81 # m/s², gravity factor

#interim calculation
V_T = A*h # m³, Aquifer volume before extraction

#Solution

d_h  = a_pm*rho_w*g*h*d_psi

print("The water abstraction volume is {0:0.2f}".format(d_h), "m")

# **Q3. Aquifer Properties** (ca. 10 pts.)
#
# The hydraulic conductivity of a sample (length 15 cm, diameter 5 cm) is to be determined using a constant-head permeameter. For that 250 ml water is passed through the sample in 30 s while maintaining the head difference of 2.5 cm. Properties of water provided are:
# density of water at 20°C: 1000 kg/m3;                                                  dynamic viscosity of water at 20°C: 1.0087·10-3 Pa·s
#
# a.  Sketch the problem as accurately as possible providing essential features with legends (3 points)
#
#
# b. What will be the conductivity of the sample? (4 points)
#
# What is the intrinsic permeability of the sample? (2 points)
#
# c. What soil type is likely the sample? (1 point)
#
# (Hint: For the calculation of permeability, dynamic viscosity/density ratio is required.)
#
#
#

# **Solution 3** -
#
# **Solution 3a** (L05/15)
#
#
#
#
#

# In[4]:

#Solution 3b** (L05/15)

# Given

L_c = 15 # cm, column length
Dia_c = 5 # cm, diameter column
V_in= 250 # mL, water entering the column
t_c = 30 # s, time required to pass
d_3h = 2.5 # cm, head difference

# interim calculation
A_c = np.pi*Dia_c**2/4 # cm², Area of column
Q_c = V_in/t_c # cm³/s, assume 1mL = 1 cm³, Discharge out of column

#solution
K_c = (Q_c*L_c)/(A_c*d_3h) # cm/s, conductivity

print("The area of the column is {0:0.2f}".format(A_c), "cm\u00b2")
print("The discharge from the aquifer is {0:0.2f}".format(Q_c), "cm\u00b3/s")
print("The conductivity of the sample is {0:0.2f}".format(K_c), "cm/s")
print("The conductivity of the sample is {0:0.4f}".format(K_c/100), "m/s")

# In[5]:

#Solution 3c** (L05/18)

# Given

K_m = K_c/100 # m/s, conductivity
rho_3w = 1000 # Kg/m³, density of water
nu_w = 1.0087*10**-3 # Pa-s = Kg/m-s, dynamic viscosity
g = 9.81 # m/s², gravity factor

#solution
k_c = K_m*nu_w/(rho_3w*g)

print("The permeability of the sample is {0:0.10f}".format(k_c), "m\u00b2")
print("The permeability of the sample is {0:0.2E}".format(k_c), "m\u00b2")

# **Solution 3d**
# (L05/11)
#
# The sample in the column is likely gravel or coarse sand.

# **Q4. Sieve Analysis (ca. 6 pts.)**
#
# Sieve experiments were performed with the bore samples and the following observations were obtained:
#
# | mesh diameter [mm] 	| residue in the sieve [g] 	| Σ total 	| Σ/Σtotal 	|
# |:------------------:	|:------------------------:	|:-------:	|:--------:	|
# |          6         	|             0            	|         	|          	|
# |          2         	|            40            	|         	|          	|
# |         0.6        	|            250           	|         	|          	|
# |         0.2        	|            150           	|         	|          	|
# |        0.06        	|            60            	|         	|          	|
# |     <0.06 (cup)    	|            10            	|         	|          	|
#
#
# a. Draw the granulometric curve in the diagram below.			(ca. 5 pts.)
#
# b. Briefly characterise the sediment.						(ca. 1 pt.)

# In[6]:

#Solution 4 -(L03/18)

dia = [6,2,0.6,0.2, 0.06, 0.001] # mm, diameter <0.06 (cup)= 0.001
mass = [0, 40, 250, 150, 60, 10] # g, the residue in seive

# Calculation steps - filling table
Total_mass = np.sum(mass)  # add the mass column to get total mass
retain_per = mass/Total_mass*100   # retain percentage
retain_per_cumsum = np.cumsum(retain_per) # get the cummulative sum of the reatined
passing_per = 100 - retain_per_cumsum # substract 100-cummsum to get passing %

data = {"mesh diameter [mm]": dia, "residue in the sieve [g]": mass, "Σtotal": retain_per, "Σ/Σtotal": passing_per }

df1= pd.DataFrame(data)
df1

# In[7]:

# plotting
plt.rcParams['axes.linewidth']=2
#plt.rcParams["axes.edgecolor"]='white'
plt.rcParams['grid.linestyle']='--'
plt.rcParams['grid.linewidth']=1
x = np.append([10],dia) # adding data to extend over 6 mm dia
y = np.append([100],passing_per) # adding 100% to plot

fig = plt.figure(figsize=(9,6))
plt.semilogx(x, y, 'x-', color='red')
tics=x.tolist()

plt.grid(which='major', color='k', alpha=0.7)
plt.grid(which='minor', color='k', alpha=0.3)
plt.xticks(x, tics);
plt.yticks(np.arange(0,110,10));
plt.title('grain size distribution');
plt.xlabel('grain size d [mm]');
plt.ylabel('grain fraction < d ins % of total mass');

# **solution 4b**
#
# The sample can be considered uniformly distributed as over 70% of sample falls in the sand size (0.2 mm-2 mm). Therefore, the sample can be considered sandy.

# **Q5. Aquifer characterization** (ca. 8 pts.)
#
# Water levels in m a.s.l. were measured at three observation wells (see figure).
#
#
#
#
# a. Sketch hydraulic head isolines for increments of 0.5 m. (ca. 3 points.)
#
# b. Gravel layer (thickness (t1) = 1.5 m, and conductivity (K1) = 3.7 10-3 is embedded between two sandy layers (t2 = 2 m, K2 = 3·10-4 m/s; and t3 = 3 m, K3 = 4·10-4 m/s). If the hydraulic gradient is 1% and overall discharge is 1 m3/d per unit width of the aquifer, find the effective hydraulic conductivity considering a parallelly layered aquifer.
#
# (Hint: $K_{eff} = \frac{m}{\sum_{i=1}^n \frac{m_i}{K_i}}$ or $K_{eff} =  \sum_{i=1}^n\frac{m_i\cdot K_i}{m}$  ) (ca. 2 points)
#
# c. Distinguish between homogeneity and heterogeneity, and isotropy and anisotropy (ca. 3 points)
#
#
#

# **Solution 5a**
# (L07/08-09)
#
# The isolines and flow direction is provided in the figure below.
#
#
#

# In[8]:

# Solution 5b (L06/08-13)

# Given:

G_t1 = 2 # m, sandy layer top
G_t2 = 1.5 # m, gravel layer middle
G_t3 = 3 # m, sandy layer bottom
K_1 =  3.0*10**-4 # m/s cond. in G_t1
K_2 = 3.7*10**-3 # m/s cond. in G_t2
K_3 =  4.0*10**-4 # m/s cond. in G_t3
i = 1/100 # (), hydraulic gradient 1%
Q_5 = 1 # m³/d per-W, discharge per unit width

#intermediate calculation
m = G_t1+G_t2+G_t3 # m, total aq. thickness

K_ef_h = (1/m) * (G_t1*K_1 + G_t2*K_2 + G_t3*K_3) # m/s, eff. horizontal cond.
K_ef_v = m/(G_t1/K_1 + G_t2/K_2 + G_t3/K_3) # # m/s, eff. vertical cond.

print("The thickness of the aquifer is {0:0.3f}".format(m), "m")
print("The effective horizontal conductivity of the aquifer is {0:0.2E}".format(K_ef_h), "m/s")
print("The effective vertical conductivity of the aquifer is {0:0.2E}".format(K_ef_v), "m/s")

# **Solution 5c** (L06/23)
#
# **Homogeneity**: An aquifer is homogeneous when its parameters are constant throughout the porous medium, i.e. the properties of the medium are independent of space
#
# **Heterogeneity**: Heterogeneous aquifer have its properties varies in space or the properties are space dependent.
#
# **Isotropy**: This relates to properties of aquifer being independent of direction, i.e., $K_v = K_h$
#
# **Anisotropy**: In this case the aquifer properties are direction dependent, i.e., $K_v \neq K_h$.

# **Q6. Well** (ca. 5 pts.)
#
#
# a. Sketch the pumping scenario of an unconfined aquifer (vertical cross section) and label all possible quantities (ca 3 pts.)
#
#
# b. The conductivity of a confined aquifer (8 m thick) is estimated to be $4\cdot 10^{-4}$ m/s. If the steady-state discharge 50 m3/s, using the Theis equation ($s = Q/4\pi T·W(u)$, with $W(u) = 15$), find the drawdown in the aquifer. (2 points)
#

# **Solution 6a**
# (L08/16)
#
# Figure below presents the scenario of a well in an _unconfined_ aquifer.
#
#
#
#

# In[9]:

#Solution 6b

#Given

Q_6 = 50 # m³/s, discharge
K_6 = 4*10**-4 # m/s, conductivity
m_6 = 8 # m, thickness
W_u = 15 # (), well function

# interim cal.
T_6 = K_6 * m_6 # m²/s, Transmissivity T = K*m

# solution
s_6 = (Q_6/(4*np.pi*T_6)) * W_u # m, drawdown

print("The Transmissivity of the aquifer is {0:0.5f}".format(T_6), "m\u00b2/s")
print("The drawdown in the well is {0:0.2f}".format(s_6), "m")

# **Q7. Conservative Transport**  (ca. 7 pts.)
#
# a. How is reactive transport different to conservative transport in the aquifers. (2 points)
#
# b. With suitable sketch distinguish between advective flux and dispersive flux. (2 points)
#
#
# c. A column (L = 1.2 m and Ø = 5 cm) was packed with sandy soil (ne= 35%  K= 0,0002 m/s). The hydraulic head at the inlet and the outlet was set to 230 m and 235 m, resp. The NaCl solution with conc. 10 mg/L was steadily introduced to the column after saturating it with distilled water. The experiment condition was such that diffusive flow could be neglected.  You may make justified assumption for any missing information.
#
# c.i. What will be the advective mass flux at the outlet of the column? (1.5 points)
#
# c.ii. Considering initial concentration difference between inlet and outlet to be 10 mg/L, what    will be the dispersive mass flux at the outlet? (1.5 points)
#
# (Hint: Dispersive and Advective fluxes are either of $ n_e \cdot v\cdot C$ and $n_e\cdot \alpha \cdot v \cdot \Delta C/L$)

# **Solution 7a** (L09/05)
#
# A chemical in groundwater is subject to conservative transport processes if there is:
#
# + no interaction with the solid material,
# + no interaction with other chemicals,
# + no interaction with microbes.
#
# When either of the above are part of the groundwater, the transport process is reactive.
#

# **Solution 7b** (L09/09)
#
# The sketch below distinguish between advective and dispersive fluxes. The figure in the left is of advective process and that in the right results to dispersive flux.
#
#

# In[10]:

#Solution 7c

L_7 = 1.2 # m, col. length
Dia_7 = 5 # cm, col. diameter
ne_7 = 0.35 # (), effective porosity
K_7 = 0.0002 # m/s, conductivity
H_7in = 235 # m, head inlet
H_7out = 230 # m, head outlet
C_7 = 10 # mg/L, NaCl concentration
al_7 = 1 # m, assumed
C_7d = 10 # mg/L

#intermediate calc.
i_7 = (H_7in-H_7out)/L_7 # (), head gradient
v_7dar = K_7*i_7 # m/s, darcy velocity
v_7av = v_7dar/ne_7 # m/s, average linear velocity

# Solution
F_7ad = ne_7*v_7av*C_7 # mg-m/L-s, advective flux
F_7dis = ne_7*al_7*v_7av*C_7d/L_7 # mg-m/L-s, dispersive flux

print("The hydraulic gradient is {0:0.4f}".format(i_7), "")
print("The Darcy velocity is {0:0.4f}".format(v_7dar), "m/s")
print("The average linear velocity is {0:0.4f}".format(v_7av), "m/s")
print("The advective flux is {0:0.10f}".format(F_7ad), "mg-m/L-s")
print("The dispersive flux is {0:0.10f}".format(F_7dis), "mg-m/L-s")

# **Q8. Sorption Isotherms** (ca. 10 pts)
#
# Five batch tests (different initial concentrations – see table below) were performed to determine the sorption properties of a sediment. For each batch 20 g of sediment in 30 mL of water were used. The measured equilibrium solute concentrations are also provided in the table.
#
# a)	Complete the above table								(ca. 3 pts.)
#
# b)	Plot the results in the diagram below and draw a Henry isotherm			(ca. 3 pts.)
#
# c)	How is retardation related to isotherm (ca. 2 points)
# (value and unit!)		(ca. 2 pts.).
#
#
#
#
#

# In[11]:

head = ["Batch nr. ", "Initial Conc. (mg/L) ", "Equi. Conc. (mg/L)", "Sorbed mass (g)", "Sorbed mass/solid (mg/g)" ]
bn = np.array([1,2,3,4,5])
C_0 = np.array([5, 10, 15, 20, 25])# mg/L, initial conc.
C_eq = np.array([2.5, 4.9, 8, 9.8, 13.2])# mg/L, equilibrium conc.
s2 = ips.sheet(rows=6, columns=5, row_headers=False, column_headers=head)
ips.column(0, bn, row_start=0)
ips.column(1, C_0, row_start=0)
ips.column(2, C_eq, row_start=0);
s2

# In[12]:

# SOlution of Problem 10 a (T07/HP9)

# Given
v_ml = 30 # ml of water used in expt.
v_l = v_ml/1000 # L, unit conversion
m_s = 20 # g, solid mass used in expt.

bn = np.array([1,2,3,4,5])
C_0 = np.array([5, 10, 15, 20, 25])# mg/L, initial conc.
C_eq = np.array([2.5, 4.9, 8, 9.8, 13.2])# mg/L, equilibrium conc.
s_m = (C_0-C_eq)*v_l
m_m = s_m/m_s# mg/g, mass ratio

#output
d8 = {"Batch Nr": bn, "Initial Conc. (mg/L)": C_0, "Equi. Conc. (mg/L)": C_eq, "Sorbed mass (g)": s_m, "Sorbed mass/solid (mg/g)" :m_m}
df9 = pd.DataFrame(d8); df9

# In[13]:

# Solution of proble 10 (b) (T07/HP9)
# fit
slope, intercept, r_value, p_value, std_err = stats.linregress(C_eq, m_m) # linear regression

#plot and fit
fig = plt.figure(); plt.plot(C_eq, m_m, 'bo', label=' provided data');
pred = intercept + slope*C_eq # fit line
plt.plot(C_eq, pred, 'r', label='y={:.2E}x+{:.2f}'.format(slope,intercept)) ;
plt.xlabel(r"$C_{eq} $ mg/L"); plt.ylabel(r"$C_{a} $ mg/g");
plt.grid(); plt.legend(fontsize=11);  plt.text(2.2, 0.014,'$R^2 = %0.2f$' % r_value)
plt.text(2.2, 0.012,'$C_a = K_{d}\cdot C_{eq}$');

# **solution 8c**
# (L10/13)
#
# The following relation relates Retardation ($R$) with linear isotherm ($K_d$)
#
# $$
# R = 1+ \frac{1-n_e}{n_e}\rho_s K_d
# $$
#
# with effective porosity $n_e$, solid density $\rho_s$.

# **Q9. Groundwater Modelling** (ca. 8 points.)
#
# a. Distinguish between conceptual model and mathematical model; and between analytical solution and empirical solution (ca. 4 points).
#
# b. Draw a conceptual model for a rectangular aquifer 100 m long and 20 m wide. Discretize the domain with 1/10 of the length length-wise and 1/5 of the width width-wise. Assure that flow in the model is from left to right direction (ca. 3 points).
#
# c. How is a no-flow boundary condition mathematically defined? (ca. 1 point)
#
#

# **Solution 9a**
# (L11/04-06)
#
# A model or also a _conceptual model_ is a representation, an image or a description of a real system.
#
# example for a real system: porous medium with water flowing through the pores (Darcy experiment)
#
# A _mathematical model_ provides a quantitative representation of the relevant system components, processes and impacts in the area of investigation. The quantitative representation is based on mathematical equations.
#
# _Analytical solution_ : These are exact mathematical expressions solving the model equations.
#
# _Emperical solution_ : These are solution based on experimental results.
#
#
#

# **Solution 9b** - (L14/12)
#
#
#

# **Solution 9c**
# (L13/16)
#
# A no-flow boundary condition is special case of second type or Neumann boundary condition. For no flow condition head gradient is equated to zero, i.e., there is no gradient and thus no flow (water flows from high to low head). Mathematically, this is:
#
# $\frac{dh}{dx} = 0 $ for no-flow along $x-$axis, with $h$ representing head.
#
#

# Good Luck.