r""".. role:: html(raw)
   :format: html

Understanding the Haar Measure
==============================

.. meta::
    :property="og:description": Learn all about the Haar measure and how to randomly sample quantum states.

    :property="og:image": https://pennylane.ai/qml/_images/spherical_int_dtheta.png

.. related::

    tutorial_unitary_designs Unitary designs
    quantum_volume Quantum volume
    qsim_beyond_classical Beyond classical computing with qsim
    tutorial_barren_plateaus Barren plateaus

*Author: PennyLane dev team. Posted: 22 March 2021. Last updated: 22 March 2021.*

If you've ever dug into the literature about random quantum circuits,
variational ansatz structure, or anything related to the structure and
properties of unitary operations, you've likely come across a statement like the
following: "Assume that :math:`U` is sampled uniformly at random from the Haar
measure".  In this demo, we're going to unravel this cryptic statement and take
an in-depth look at what it means. You'll gain an understanding of the general
concept of *measure*, the Haar measure and its special properties, and you'll
learn how to sample from it using tools available in PennyLane and other
scientific computing frameworks. By the end of this demo, you'll be able to
include that important statement in your own work with confidence!

.. note::

   To get the most out of this demo, it is helpful if you are familiar with
   `integration of multi-dimensional functions
   <https://en.wikipedia.org/wiki/Multiple_integral>`__, the `Bloch sphere
   <https://en.wikipedia.org/wiki/Bloch_sphere>`__, and the conceptual ideas
   behind `decompositions
   <https://en.wikipedia.org/wiki/Matrix_decomposition>`__ and factorizations of
   unitary matrices (see, e.g., 4.5.1 and 4.5.2 of [#NandC2000]_).

Measure
-------

`Measure theory <https://en.wikipedia.org/wiki/Measure_(mathematics)>`__ is a
branch of mathematics that studies things that are measurable---think length,
area, or volume, but generalized to mathematical spaces and even higher
dimensions. Loosely, the measure tells you about how "stuff" is distributed and
concentrated in a mathematical set or space. An intuitive way to understand
measure is to think about a sphere. An arbitrary point on a sphere can be
parametrized by three numbers---depending on what you're doing, you may use
Cartesian coordinates :math:`(x, y, z)`, or it may be more convenient to use
spherical coordinates :math:`(\rho, \phi, \theta)`.

Suppose you wanted to compute the volume of a solid sphere with radius
:math:`r`.  This can be done by integrating over the three coordinates
:math:`\rho, \phi`, and :math:`\theta`. Your first thought here may be to simply
integrate each parameter over its full range, like so:

.. math::

    V = \int_0^{r} \int_0^{2\pi} \int_0^{\pi} d\rho~ d\phi~ d\theta = 2\pi^2 r

But, we know that the volume of a sphere of radius :math:`r` is
:math:`\frac{4}{3}\pi r^3`, so what we got from this integral is clearly wrong!
Taking the integral naively like this doesn't take into account the structure of
the sphere with respect to the parameters. For example, consider
two small, infinitesimal elements of area with the same difference in
:math:`\theta` and :math:`\phi`, but at different values of :math:`\theta`:

.. figure:: /demonstrations/haar_measure/spherical_int_dtheta.png
    :align: center
    :width: 50%

    |

Even though the differences :math:`d\theta` and :math:`d\phi` themselves are the
same, there is way more "stuff" near the equator of the sphere than there is
near the poles. We must take into account the value of :math:`\theta` when
computing the integral! Specifically, we multiply by the function
:math:`\sin\theta`---the properties of the :math:`\sin` function mean that the
most weight will occur around the equator where :math:`\theta=\pi/2`, and the
least weight near the poles where :math:`\theta=0` and :math:`\theta=\pi`.

Similar care must be taken for :math:`\rho`.  The contribution to volume of
parts of the sphere with a large :math:`\rho` is far more than for a small
:math:`\rho`---we should expect the contribution to be proportional to
:math:`\rho^2`, given that the surface area of a sphere of radius :math:`r` is
:math:`4\pi r^2`.

On the other hand, for a fixed :math:`\rho` and :math:`\theta`, the length of
the :math:`d\phi` is the same all around the circle. If put all these facts
together, we find that the actual expression for the integral should look like
this:

.. math::

    V = \int_0^r \int_0^{2\pi} \int_0^{\pi} \rho^2 \sin \theta~ d\rho~ d\phi~
    d\theta = \frac{4}{3}\pi r^3

These extra terms that we had to add to the integral, :math:`\rho^2 \sin
\theta`, constitute the *measure*. The measure weights portions of the sphere
differently depending on where they are in the space. While we need to know the
measure to properly integrate over the sphere, knowledge of the measure also
gives us the means to perform another important task, that of sampling points in
the space uniformly at random. We can't simply sample each parameter from the
uniform distribution over its domain---as we experienced already, this doesn't
take into account how the sphere is spread out over space. The measure describes
the distribution of each parameter and gives a recipe for sampling them in order
to obtain something properly uniform.

The Haar measure
----------------

Operations in quantum computing are described by unitary matrices.
Unitary matrices, like points on a sphere, can be expressed in terms of a fixed
set of coordinates, or parameters. For example, the most general single-qubit rotation
implemented in PennyLane (:class:`~.pennylane.Rot`) is expressed in terms of three
parameters like so,

.. math::

    U(\phi, \theta, \omega) = \begin{pmatrix} e^{-i(\phi + \omega)/2}
                        \cos(\theta/2) & -e^{i(\phi - \omega)/2} \sin(\theta/2)
                        \\ e^{-i(\phi - \omega)/2} \sin(\theta/2) & e^{i(\phi +
                        \omega)/2} \cos(\theta/2) \end{pmatrix}.

For every dimension :math:`N`, the unitary matrices of size :math:`N \times N`
constitute the *unitary group* :math:`U(N)`. We can perform operations on
elements of this group, such as apply functions to them, integrate over them, or
sample uniformly over them, just as we can do to points on a sphere. When we do
such tasks with respect to the sphere, we have to add the measure in order to
properly weight the different regions of space. The *Haar measure* provides the
analogous terms we need for working with the unitary group.

For an :math:`N`-dimensional system, the Haar measure, often denoted by
:math:`\mu_N`, tells us how to weight the elements of :math:`U(N)`. For
example, suppose :math:`f` is a function that acts on elements of :math:`U(N)`,
and we would like to take its integral over the group. We must write this
integral with respect to the Haar measure, like so:

.. math::

    \int_{V \in U(N)} f(V) d\mu_N(V).

As with the measure term of the sphere, :math:`d\mu_N` itself can be broken down
into components depending on individual parameters.  While the Haar
measure can be defined for every dimension :math:`N`, the mathematical form gets
quite hairy for larger dimensions---in general, an :math:`N`-dimensional unitary
requires at least :math:`N^2 - 1` parameters, which is a lot to keep track of!
Therefore we'll start with the case of a single qubit :math:`(N=2)`, then show
how things generalize.

Single-qubit Haar measure
~~~~~~~~~~~~~~~~~~~~~~~~~

The single-qubit case provides a particularly nice entry point because we can
continue our comparison to spheres by visualizing single-qubit states on the
Bloch sphere. As expressed above, the measure provides a recipe for sampling
elements of the unitary group in a properly uniform manner, given the structure
of the group. One useful consequence of this is that it provides a method to
sample quantum *states* uniformly at random---we simply generate Haar-random
unitaries, and apply them to a fixed basis state such as :math:`\vert 0\rangle`.

We'll see how this works in good time. First, we'll take a look at what happens
when we ignore the measure and do things *wrong*. Suppose we sample quantum
states by applying unitaries obtained by the parametrization above, but sample
the angles :math:`\omega, \phi`, and :math:`\theta` from the flat uniform
distribution between :math:`[0, 2\pi)` (fun fact: there is a measure implicit in
this kind of sampling too! It just has a constant value, because each point is
equally likely to be sampled).

"""

import pennylane as qml
import numpy as np
import matplotlib.pyplot as plt

# set the random seed
np.random.seed(42)

# Use the mixed state simulator to save some steps in plotting later
dev = qml.device('default.mixed', wires=1)

@qml.qnode(dev)
def not_a_haar_random_unitary():
    # Sample all parameters from their flat uniform distribution
    phi, theta, omega = 2 * np.pi * np.random.uniform(size=3)
    qml.Rot(phi, theta, omega, wires=0)
    return qml.state()

num_samples = 2021

not_haar_samples = [not_a_haar_random_unitary() for _ in range(num_samples)]

######################################################################
# In order to plot these on the Bloch sphere, we'll need to do one more
# step, and convert the quantum states into Bloch vectors.
#

X = np.array([[0, 1], [1, 0]])
Y = np.array([[0, -1j], [1j, 0]])
Z = np.array([[1, 0], [0, -1]])

# Used the mixed state simulator so we could have the density matrix for this part!
def convert_to_bloch_vector(rho):
    """Convert a density matrix to a Bloch vector."""
    ax = np.trace(np.dot(rho, X)).real
    ay = np.trace(np.dot(rho, Y)).real
    az = np.trace(np.dot(rho, Z)).real
    return [ax, ay, az]

not_haar_bloch_vectors = np.array([convert_to_bloch_vector(s) for s in not_haar_samples])

######################################################################
# With this done, let's find out where our "uniformly random" states ended up:

def plot_bloch_sphere(bloch_vectors):
    """ Helper function to plot vectors on a sphere."""
    fig = plt.figure(figsize=(6, 6))
    ax = fig.add_subplot(111, projection='3d')
    fig.subplots_adjust(left=0, right=1, bottom=0, top=1)

    ax.grid(False)
    ax.set_axis_off()
    ax.view_init(30, 45)
    ax.dist = 7

    # Draw the axes (source: https://github.com/matplotlib/matplotlib/issues/13575)
    x, y, z = np.array([[-1.5,0,0], [0,-1.5,0], [0,0,-1.5]])
    u, v, w = np.array([[3,0,0], [0,3,0], [0,0,3]])
    ax.quiver(x, y, z, u, v, w, arrow_length_ratio=0.05, color="black", linewidth=0.5)

    ax.text(0, 0, 1.7, r"|0⟩", color="black", fontsize=16)
    ax.text(0, 0, -1.9, r"|1⟩", color="black", fontsize=16)
    ax.text(1.9, 0, 0, r"|+⟩", color="black", fontsize=16)
    ax.text(-1.7, 0, 0, r"|–⟩", color="black", fontsize=16)
    ax.text(0, 1.7, 0, r"|i+⟩", color="black", fontsize=16)
    ax.text(0,-1.9, 0, r"|i–⟩", color="black", fontsize=16)

    ax.scatter(
        bloch_vectors[:,0], bloch_vectors[:,1], bloch_vectors[:, 2], c='#e29d9e', alpha=0.3
    )

plot_bloch_sphere(not_haar_bloch_vectors)

######################################################################
# You can see from this plot that even though our parameters were sampled from a
# uniform distribution, there is a noticeable amount of clustering around the poles
# of the sphere. Despite the input parameters being uniform, the output is very
# much *not* uniform. Just like the regular sphere, the measure is larger near
# the equator, and if we just sample uniformly, we won't end up populating that
# area as much. To take that into account we will need to sample from the proper
# Haar measure, and weight the different parameters appropriately.
#
# For a single qubit, the Haar measure looks much like the case of a sphere,
# minus the radial component. Intuitively, all qubit state vectors have length
# 1, so it makes sense that this wouldn't play a role here. The parameter that
# we will have to weight differently is :math:`\theta`, and in fact the
# adjustment in measure is identical to that we had to do with the polar axis of
# the sphere, i.e., :math:`\sin \theta`. In order to sample the :math:`\theta`
# uniformly at random in this context, we must sample from the distribution
# :math:`\hbox{Pr}(\theta) = \sin \theta`. We can accomplish this by setting up
# a custom probability distribution with
# `rv_continuous <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_continuous.html#scipy.stats.rv_continuous>`__
# in ``scipy``.

from scipy.stats import rv_continuous

class sin_prob_dist(rv_continuous):
    def _pdf(self, theta):
        # The 0.5 is so that the distribution is normalized
        return 0.5 * np.sin(theta)

# Samples of theta should be drawn from between 0 and pi
sin_sampler = sin_prob_dist(a=0, b=np.pi)

@qml.qnode(dev)
def haar_random_unitary():
    phi, omega = 2 * np.pi * np.random.uniform(size=2) # Sample phi and omega as normal
    theta = sin_sampler.rvs(size=1) # Sample theta from our new distribution
    qml.Rot(phi, theta, omega, wires=0)
    return qml.state()

haar_samples = [haar_random_unitary() for _ in range(num_samples)]
haar_bloch_vectors = np.array([convert_to_bloch_vector(s) for s in haar_samples])

plot_bloch_sphere(haar_bloch_vectors)

######################################################################
# We see that when we use the correct measure, our qubit states are now
# much better distributed over the sphere. Putting this information together,
# we can now write the explicit form for the single-qubit Haar measure:
#
# .. math::
#
#    d\mu_2 = \sin \theta d\theta \cdot d\omega \cdot d\phi.
#
# Show me more math!
# ~~~~~~~~~~~~~~~~~~
#
# While we can easily visualize the single-qubit case, this is no longer
# possible when we increase the number of qubits. Regardless, we can still
# obtain a mathematical expression for the Haar measure in arbitrary
# dimensions. In the previous section, we expressed the Haar measure in terms of
# a set of parameters that can be used to specify the unitary group
# :math:`U(2)`. Such a parametrization is not unique, and in fact there are
# multiple ways to *factorize*, or decompose an :math:`N`-dimensional unitary
# operation into a set of parameters.
#
# Many of these parametrizations come to us from the study of photonics.  Here,
# arbitrary operations are broken down into elementary operations involving only
# a few parameters which correspond directly to parameters of the physical
# apparatus used to implement them (beamsplitters and phase shifts). Rather than
# qubits, such operations act on modes, or *qumodes*. They are expressed as
# elements of the :math:`N`-dimensional `special unitary group
# <https://en.wikipedia.org/wiki/Special_unitary_group>`__. This group, written
# as :math:`SU(N)`, is the continuous group consisting of all :math:`N \times N`
# unitary operations with determinant 1 (essentially like :math:`U(N)`, minus
# a potential global phase).
#
#
# .. note::
#
#     Elements of :math:`SU(N)` and :math:`U(N)` can still be considered as
#     multi-qubit operations in the cases where :math:`N` is a power of 2, but
#     they must be translated from continuous-variable operations into qubit
#     operations. (In PennyLane, this can be done by feeding the unitaries to
#     the :class:`~.pennylane.QubitUnitary` operation directly. Alternatively,
#     one can use *quantum compilation* to express the operations as a sequence
#     of elementary gates such as Pauli rotations and CNOTs.)
#
# .. admonition:: Tip
#
#    If you haven't had many opportunities to work in terms of qumodes, the
#    `Strawberry Fields documentation
#    <https://strawberryfields.ai/photonics/concepts/photonics.html>`__ is a
#    good starting point.
#
# For example, we saw already above that for :math:`N=2`, we can write
#
# .. math::
#
#    U(\phi, \theta, \omega) = \begin{pmatrix} e^{-i(\phi + \omega)/2}
#                        \cos(\theta/2) & -e^{i(\phi - \omega)/2} \sin(\theta/2)
#                        \\ e^{-i(\phi - \omega)/2} \sin(\theta/2) & e^{i(\phi +
#                        \omega)/2} \cos(\theta/2) \end{pmatrix}.
#
#
# This unitary can be factorized as follows:
#
# .. math::
#
#    U(\phi, \theta, \omega) =
#        \begin{pmatrix}
#          e^{-i\omega/2} & 0 \\ 0 & e^{i\omega/2}
#        \end{pmatrix}
#        \begin{pmatrix}
#          \cos(\theta/2) & -\sin(\theta/2) \\ \sin(\theta/2) & \cos(\theta/2)
#        \end{pmatrix}
#       \begin{pmatrix}
#          e^{-i\phi/2} & 0 \\ 0 & e^{i\phi/2}
#        \end{pmatrix}
#
# The middle operation is a beamsplitter; the other two operations are phase
# shifts.  We saw earlier that for :math:`N=2`, :math:`d\mu_2 = \sin\theta
# d\theta d\omega d\phi`---note how the parameter in the beamsplitter
# contributes to the measure in a different way than those of the phase
# shifts. As mentioned above, for larger values of :math:`N` there are multiple
# ways to decompose the unitary. Such decompositions rewrite elements in
# :math:`SU(N)` acting on :math:`N` modes as a sequence of operations acting
# only on 2 modes, :math:`SU(2)`, and single-mode phase shifts.  Shown below are
# three examples [#deGuise2018]_, [#Clements2016]_, [#Reck1994]_:
#
# .. figure:: /demonstrations/haar_measure/unitaries.png
#    :align: center
#    :width: 95%
#
#
# In these graphics, every wire is a different mode. Every box represents an
# operation on one or more modes, and the number in the box indicates the number
# of parameters.  The boxes containing a ``1`` are simply phase shifts on
# individual modes. The blocks containing a ``3`` are :math:`SU(2)` transforms
# with 3 parameters, such as the :math:`U(\phi, \theta, \omega)` above. Those
# containing a ``2`` are :math:`SU(2)` transforms on pairs of modes with 2
# parameters, similar to the 3-parameter ones but with :math:`\omega = \phi`.
#
# Although the decompositions all produce the same set of operations, their
# structure and parametrization may have consequences in practice.  The first [#deGuise2018]_
# has a particularly convenient form that leads to a recursive definition
# of the Haar measure. The decomposition is formulated recursively such that an
# :math:`SU(N)` operation can be implemented by sandwiching an :math:`SU(2)`
# transformation between two :math:`SU(N-1)` transformations, like so:
#
# |
#
# .. figure:: /demonstrations/haar_measure/sun.svg
#    :align: center
#    :width: 80%
#
# |
#
# The Haar measure is then constructed recursively as a product of 3
# terms. The first term depends on the parameters in the first :math:`SU(N-1)`
# transformation; the second depends on the parameters in the lone :math:`SU(2)`
# transformation; and the third term depends on the parameters in the other
# :math:`SU(N-1)` transformation.
#
# :math:`SU(2)` is the "base case" of the recursion---we simply have the Haar measure
# as expressed above.
#
# |
#
# .. figure:: /demonstrations/haar_measure/su2_haar.svg
#    :align: center
#    :width: 25%
#
# |
#
# Moving on up, we can write elements of :math:`SU(3)` as a sequence of three
# :math:`SU(2)` transformations. The Haar measure :math:`d\mu_3` then consists
# of two copies of :math:`d\mu_2`, with an extra term in between to take into
# account the middle transformation.
#
# |
#
# .. figure:: /demonstrations/haar_measure/su3_haar.svg
#    :align: center
#    :width: 80%
#
# |
#
# For :math:`SU(4)` and upwards, the form changes slightly, but still follows
# the pattern of two copies of :math:`d\mu_{N-1}` with a term in between.
#
# |
#
# .. figure:: /demonstrations/haar_measure/su4_premerge.svg
#    :align: center
#    :width: 90%
#
# |
#
# For larger systems, however, the recursive composition allows for some of the
# :math:`SU(2)` transformations on the lower modes to be grouped. We can take
# advantage of this and aggregate some of the parameters:
#
# |
#
# .. figure:: /demonstrations/haar_measure/su4_triangle_merge.svg
#    :align: center
#    :width: 100%
#
# |
#
# This leads to one copy of :math:`d\mu_{N-1}`, which we'll denote as
# :math:`d\mu_{N-1}^\prime`, containing only a portion of the full set of terms
# (as detailed in [#deGuise2018]_, this is called a *coset measure*).
#
# |
#
# .. figure:: /demonstrations/haar_measure/su4_haar.svg
#    :align: center
#    :width: 100%
#
# |
#
# Putting everything together, we have that
#
# .. math::
#
#    d\mu_N = d\mu_{N-1}^\prime \times \sin \theta_{N-1}
#    \sin^{2(N-2)}\left(\frac{\theta_{N-1}}{2}\right) d\theta_{N-1} d\omega_{N-1} \times d\mu_{N-1}
#
# The middle portion depends on the value of :math:`N`, and the parameters
# :math:`\theta_{N-1}` and :math:`\omega_{N-1}` contained in the :math:`(N-1)`'th
# :math:`SU(N)` transformation. This is thus a convenient, systematic way to
# construct the :math:`N`-dimensional Haar measure for the unitary group. As a
# final note, even though unitaries can be parametrized in different ways, the
# underlying Haar measure is *unique*. This is a consequence of it being an
# invariant measure, as will be shown later.
#
# Haar-random matrices from the :math:`QR` decomposition
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# Nice-looking math aside, sometimes you just need to generate a large number of
# high-dimensional Haar-random matrices. It would be very cumbersome to sample
# and keep track of the distributions of so many parameters; furthermore, the
# measure above requires you to parametrize your operations in a fixed way.
# There is a much quicker way to perform the sampling by taking a (slightly
# modified) `QR decomposition
# <https://en.wikipedia.org/wiki/QR_decomposition>`__ of complex-valued
# matrices.  This algorithm is detailed in [#Mezzadri2006]_, and consists of the
# following steps:
#
# 1. Generate an :math:`N \times N` matrix :math:`Z` with complex numbers :math:`a+bi`
#    where both :math:`a` and :math:`b` are normally distributed with mean 0 and variance 1
#    (this is sampling from the distribution known as the *Ginibre ensemble*).
# 2. Compute a QR decomposition :math:`Z = QR`.
# 3. Compute the diagonal matrix :math:`\Lambda = \hbox{diag}(R_{ii}/|R_{ii}|)`.
# 4. Compute :math:`Q^\prime = Q \Lambda`, which will be Haar-random.
#
#

from numpy.linalg import qr

def qr_haar(N):
    """Generate a Haar-random matrix using the QR decomposition."""
    # Step 1
    A, B = np.random.normal(size=(N, N)), np.random.normal(size=(N, N))
    Z = A + 1j * B

    # Step 2
    Q, R = qr(Z)

    # Step 3
    Lambda = np.diag([R[i, i] / np.abs(R[i, i]) for i in range(N)])

    # Step 4
    return np.dot(Q, Lambda)

######################################################################
# Let's check that this method actually generates Haar-random unitaries
# by trying it out for :math:`N=2` and plotting on the Bloch sphere.
#

@qml.qnode(dev)
def qr_haar_random_unitary():
    qml.QubitUnitary(qr_haar(2), wires=0)
    return qml.state()

qr_haar_samples = [qr_haar_random_unitary() for _ in range(num_samples)]
qr_haar_bloch_vectors = np.array([convert_to_bloch_vector(s) for s in qr_haar_samples])
plot_bloch_sphere(qr_haar_bloch_vectors)

######################################################################
# As expected, we find our qubit states are distributed uniformly over the
# sphere.  This particular method is what's implemented in packages like
# ``scipy``'s `unitary_group
# <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.unitary_group.html>`__
# function.
#
# Now, it's clear that this method works, but it is also important to
# understand *why* it works.  Step 1 is fairly straightforward---the base of our
# samples is a matrix full of complex values chosen from a typical
# distribution. This isn't enough by itself, since unitary matrices also
# have constraints---their rows and columns must be orthonormal.
# These constraints are where step 2 comes in---the outcome of a generic
# QR decomposition consists of an *orthonormal* matrix :math:`Q`, and and upper
# triangular matrix :math:`R`. Since our original matrix was complex-valued, we end
# up with a :math:`Q` that is in fact already unitary. But why not stop there? Why
# do we then perform steps 3 and 4?
#
# Steps 3 and 4 are needed because, while the QR decomposition yields a unitary,
# it is not a unitary that is properly Haar-random. In [#Mezzadri2006]_, it is
# explained that a uniform distribution over unitary matrices should also yield
# a uniform distribution over the *eigenvalues* of those matrices, i.e., every
# eigenvalue should be equally likely. Just using the QR decomposition out of
# the box produces an *uneven* distribution of eigenvalues of the unitaries!
# This discrepancy stems from the fact that the QR decomposition is not unique.
# We can take any unitary diagonal matrix :math:`\Lambda`, and re-express the decomposition
# as :math:`QR = Q\Lambda \Lambda^\dagger R = Q^\prime R^\prime`. Step 3 removes this
# redundancy by fixing a :math:`\Lambda` that depends on :math:`R`, leading to a unique
# value of :math:`Q^\prime = Q \Lambda`, and a uniform distribution of eigenvalues.
#
# .. admonition:: Try it!
#
#    Use the ``qr_haar`` function above to generate random unitaries and construct
#    a distribution of their eigenvalues. Then, comment out the lines for steps 3 and
#    4 and do the same---you'll find that the distribution is no longer uniform.
#    Check out reference [#Mezzadri2006]_ for additional details and examples.

######################################################################
# Fun (and not-so-fun) facts
# --------------------------
#
# We've now learned what the Haar measure is, and both an analytical and
# numerical means of sampling quantum states and unitary operations uniformly at
# random. The Haar measure also has many neat properties that play a role in
# quantum computing.
#
# Invariance of the Haar measure
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# Earlier, we showed that the Haar measure is used when integrating functions over
# the unitary group:
#
# .. math::
#
#    \int_{V \in U(N)} f(V) d\mu_N(V).
#
# One of the defining features of the Haar measure is that it is both left and
# right *invariant* under unitary transformations. That is,
#
# .. math::
#
#    \int_{V \in U(N)} f(\color{red}{W}V) d\mu_N(V) =  \int_{V \in U(N)} f(V\color{red}{W}) d\mu_N(V) =  \int_{V \in U(N)} f(V) d\mu_N(V).
#
# This holds true for *any* other :math:`N\times N` unitary :math:`W`! A
# consequence of such invariance is that if :math:`V` is Haar-random, then so is
# :math:`V^T,` :math:`V^\dagger,` and any product of another unitary matrix and
# :math:`V` (where the product may be taken on either side).
#
# Another consequence of this invariance has to do with the structure of the entries
# themselves: they must all come from the same distribution. This is because the
# measure remains invariant under permutations, since permutations are unitary---
# the whole thing still has to be Haar random no matter how the entries are ordered,
# so all distributions must be the same.  The specific distribution is complex
# numbers :math:`a+bi` where both :math:`a` and :math:`b` has mean 0 and variance
# :math:`1/N` [#Meckes2014]_ (so, much like Ginibre ensemble we used in the QR decomposition
# above, but with a different variance and constraints due to orthonormality).
#
# Concentration of measure
# ~~~~~~~~~~~~~~~~~~~~~~~~
#
# An unfortunate (although interesting) property of the Haar measure is that it
# suffers from the phenomenon of `concentration of measure
# <https://en.wikipedia.org/wiki/Concentration_of_measure>`__. Most of the
# "stuff" in the space concentrates around a certain area, and this gets worse
# as the size of the system increases. You can see the beginnings of by looking
# at the sphere. For the 3-dimensional sphere, we saw graphically how there is
# concentration around the equator, and how the measure takes that into account
# with the additional factor of :math:`\sin \theta`. This property becomes
# increasingly prominent for `higher-dimensional spheres
# <https://en.wikipedia.org/wiki/N-sphere>`__.
#
# .. important::
#
#    The concentration described here is not referring to what we witnessed
#    earlier on, when we sampled quantum states (points on the Bloch sphere)
#    incorrectly and found that they clustered around the poles. However, that
#    is not unrelated. Concentration of measure refers to where the measure
#    itself is concentrated, and which parts of the space should be more heavily
#    weighted. For the case of the sphere, it is the equatorial area, and when
#    we didn't sample properly and take that concentration into account, we
#    obtained an uneven distribution.
#
# Let's consider an :math:`N`-dimensional unit sphere. Points on the sphere, or
# vectors in this space, are parametrized by :math:`N-1` real coordinates.
# Suppose we have some function :math:`f` that maps points on that sphere to
# real numbers. Sample a point :math:`x` on that sphere from the uniform
# measure, and compute the value of :math:`f(x)`. How close do you think the
# result will be to the mean value of the function, :math:`E[f]`, over the
# entire sphere?
#
# A result called `Levy's lemma
# <https://en.wikipedia.org/wiki/Concentration_of_measure#Concentration_on_the_sphere>`__
# [#Gerken2013]_, [#Hayden2006]_ expresses how likely it is that :math:`f(x)` is a specific
# distance away from the mean. It states that, for an :math:`x` selected
# uniformly at random, the probability that :math:`f(x)` deviates from
# :math:`E[f]` by some amount :math:`\epsilon` is bounded by:
#
# .. math::
#
#    \hbox{Pr}(|f(x) - E[f]| \ge \epsilon) \leq 2 \exp\left[-\frac{N\epsilon^2}{9\pi^3 \eta^2}\right].
#
# A constraint on the function :math:`f` is that it must be `Lipschitz
# continuous <https://en.wikipedia.org/wiki/Lipschitz_continuity>`__, where
# :math:`\eta` is the *Lipschitz constant* of the function. The important aspect
# here is the likelihood of deviating significantly from the mean by an amount
# :math:`\epsilon` decreases exponentially with :math:`\epsilon.` Furthermore,
# increasing the dimension :math:`N` also makes the deviation exponentially less
# likely.
#
# Now, this result seems unrelated to quantum states---it concerns higher-
# dimensional spheres. However, recall that a quantum state vector is a complex
# vector whose squared values sum to 1, similar to vectors on a sphere. If you
# "unroll" a quantum state vector of dimension :math:`N = 2^n` by stacking its
# real and complex parts, you end with a vector of length :math:`2 \cdot 2^{n}`
# which ends up behaving just like a unit vector on the sphere in this
# dimension. Given that measure concentrates on spheres, and quantum state
# vectors can be converted to vectors on spheres, functions on random quantum
# states will also demonstrate concentration.
#
# This is bad news! To do useful things in quantum computing, we need a lot of
# qubits. But the more qubits we have, the more our randomly sampled states will
# look the same (specifically, random states will concentrate around the
# maximally entangled state [#Hayden2006]_). This has important consequences for
# near-term algorithms (as detailed in the next section), and any algorithm that
# involves uniform sampling of quantum states and operations.
#
# Haar measure and barren plateaus
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# Suppose you are venturing out to solve a new problem using an algorithm such
# as the :doc:`variational quantum eigensolver </demos/tutorial_vqe>`. A
# critical component of such methods is the choice of :doc:`variational ansatz
# </glossary/circuit_ansatz>`. Having now learned a bit about the properties of
# the Haar measure, you may think it would make sense to use this for the
# parametrization. Variational ansaetze are, after all, parametrized quantum
# circuits, so why not choose an ansatz that corresponds directly to a
# parametrization for Haar-random unitaries?  The initial parameter selection
# will give you a state in the Hilbert space uniformly at random. Then, since
# this ansatz spans the entire Hilbert space, you're guaranteed to be able to
# represent the target ground state with your ansatz, and it should be able to
# find it with no issue ... right?
#
# Unfortunately, while such an ansatz is extremely *expressive* (i.e., it is
# capable of representing any possible state), these ansaetze actually suffer
# the most from the barren plateau problem [#McClean2018]_, [#Holmes2021]_.
# :doc:`Barren plateaus </demos/tutorial_barren_plateaus>` are regions in the
# cost landscape of a parametrized circuit where both the gradient and its
# variance approach 0, leading the optimizer to get stuck in a local minimum.
# This was explored recently in the work of [#Holmes2021]_, wherein closeness to
# the Haar measure was actually used as a metric for expressivity. The closer
# things are to the Haar measure, the more expressive they are, but they are
# also more prone to exhibiting barren plateaus.
#
#
# .. figure:: /demonstrations/haar_measure/holmes-costlandscapes.png
#    :align: center
#    :width: 50%
#
#    Image source: [#Holmes2021]_. A highly expressive ansatz that can access
#    much of the space of possible unitaries (i.e., an ansatz capable of
#    producing unitaries in something close to a Haar-random manner) is very
#    likely to have flat cost landscapes and suffer from the barren plateau
#    problem.
#
# It turns out that the types of ansaetze know as *hardware-efficient ansaetze*
# also suffer from this problem if they are "random enough" (this notion will be
# formalized in a future demo). It was shown in [#McClean2018]_ that this is a
# consequence of the concentration of measure phenomenon described above. The
# values of gradients and variances can be computed for classes of circuits on
# average by integrating with respect to the Haar measure, and it is shown that
# these values decrease exponentially in the number of qubits, and thus huge
# swaths of the cost landscape are simply and fundamentally flat.
#
# Conclusion
# ----------
#
# The Haar measure plays an important role in quantum computing---anywhere
# you might be dealing with sampling random circuits, or averaging over
# all possible unitary operations, you'll want to do so with respect
# to the Haar measure.
#
# There are two important aspects of this that we have yet to touch upon,
# however. The first is whether it is efficient to sample from the Haar measure---given
# that the number of parameters to keep track of is exponential in the
# number of qubits, certainly not. But a more interesting question is do we
# *need* to always sample from the full Haar measure?  The answer to this is
# "no" in a very interesting way. Depending on the task at hand, you may be able
# to take a shortcut using something called a *unitary design*. In an upcoming
# demo, we will explore the amazing world of unitary designs and their
# applications!
#
# References
# ----------
#
# .. [#NandC2000]
#
#     M. A. Nielsen, and I. L. Chuang (2000) "Quantum Computation and Quantum Information",
#     Cambridge University Press.
#
# .. [#deGuise2018]
#
#     H. de Guise, O. Di Matteo, and L. L. Sánchez-Soto. (2018) "Simple factorization
#     of unitary transformations", `Phys. Rev. A 97 022328
#     <https://journals.aps.org/pra/abstract/10.1103/PhysRevA.97.022328>`__.
#     (`arXiv <https://arxiv.org/abs/1708.00735>`__)
#
# .. [#Clements2016]
#
#     W. R. Clements, P. C. Humphreys, B. J. Metcalf, W. S. Kolthammer, and
#     I. A. Walmsley (2016) “Optimal design for universal multiport
#     interferometers”, \ `Optica 3, 1460–1465
#     <https://www.osapublishing.org/optica/fulltext.cfm?uri=optica-3-12-1460&id=355743>`__.
#     (`arXiv <https://arxiv.org/abs/1603.08788>`__)
#
# .. [#Reck1994]
#
#    M. Reck, A. Zeilinger, H. J. Bernstein, and P. Bertani (1994) “Experimental
#    realization of any discrete unitary operator”, `Phys. Rev. Lett.73, 58–61
#    <https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.73.58>`__.
#
# .. [#Mezzadri2006]
#
#     F. Mezzadri (2006) "How to generate random matrices from the classical compact groups".
#     (`arXiv <https://arxiv.org/abs/math-ph/0609050>`__)
#
# .. [#Meckes2014]
#
#     E. Meckes (2019) `"The Random Matrix Theory of the Classical Compact Groups"
#     <https://case.edu/artsci/math/esmeckes/Haar_book.pdf>`_, Cambridge University Press.
#
# .. [#Gerken2013]
#
#     M. Gerken (2013) "Measure concentration: Levy's Lemma"
#     (`lecture notes <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.679.2560>`__).
#
#
# .. [#Hayden2006]
#
#     P. Hayden, D. W. Leung, and A. Winter (2006) "Aspects of generic
#     entanglement", `Comm. Math. Phys. Vol. 265, No. 1, pp. 95-117
#     <https://link.springer.com/article/10.1007%2Fs00220-006-1535-6>`__.
#     (`arXiv <https://arxiv.org/abs/quant-ph/0407049>`__)
#
# .. [#McClean2018]
#
#     J. R. McClean, S. Boixo, V. N. Smelyanskiy, R. Babbush, and H. Neven
#     (2018) "Barren plateaus in quantum neural network training
#     landscapes", `Nature Communications, 9(1)
#     <http://dx.doi.org/10.1038/s41467-018-07090-4>`__.
#     (`arXiv <https://arxiv.org/abs/1803.11173>`__)
#
# .. [#Holmes2021]
#
#     Z. Holmes, K. Sharma, M. Cerezo, and P. J. Coles (2021) "Connecting ansatz
#     expressibility to gradient magnitudes and barren plateaus". (`arXiv
#     <https://arxiv.org/abs/2101.02138>`__)
#
#


#!usr/bin/env python3
import numba
from numba.typed import List

# JIT, Just-In-Time compiling, is a technology that has been used very
# successfully in several dynamic languages to gain performance.
# The idea behind Just-In-Time is that we collect information on the code at
# run time and according to this information, we generate specific machine
# code for these cases. If you're function runs only once, JIT will not help.
# But in most cases, bottlenecks are in functions that are called many times.
# Numba, this is a compiler kit called LLVM, under the hood, to generate
# machine code for Python functions.
# LLVM is an acronym that stands for low level virtual machine.
# It also refers to a compiling technology called the LLVM project, which is a
# collection of modular and reusable compiler and toolchain technologies.

def poly(coeffs, n):
    """Compute value of polynomial given coefficients"""
    total = 0
    for i, c in enumerate(coeffs):
        total += c * n ** i
    return total

@numba.jit
def poly_j(coeffs, n):
    """Compute value of polynomial given coefficients - JIT"""
    total = 0
    for i, c in enumerate(coeffs):
        total += c * n ** i
    return total

if __name__ == '__main__':
    # previously we had coeffs = [4, 8, 15, 16, 23, 42] but we were getting
    # a warning:
    # http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types
    coeffs = List()
    [coeffs.append(x) for x in [4, 8, 15, 16, 23, 42]]

# In [7]: %run src/using_jit.py
#
# In [8]: poly(coeffs, 7)
# Out[8]: 767400
#
# In [9]: poly_j(coeffs, 7)
# Out[9]: 767400
#
# In [10]: %timeit poly(coeffs, 7)
# 18.6 µs ± 2.08 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)
#
# In [11]: %timeit poly_j(coeffs, 7)
# 1.67 µs ± 36.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)


#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.4'
#       jupytext_version: 1.1.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # s_stock_short_horizon [<img src="https://www.arpm.co/lab/icons/icon_permalink.png" width=30 height=30 style="display: inline;">](https://www.arpm.co/lab/redirect.php?code=s_stock_short_horizon&codeLang=Python)
# For details, see [here](https://www.arpm.co/lab/redirect.php?permalink=ExerStockShort).

# +
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import datetime as dt
from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()

from arpym.tools import add_logo
# -

# ## [Input parameters](https://www.arpm.co/lab/redirect.php?permalink=s_stock_short_horizon-parameters)

# day, month and the year of the plotted value
day = 2
month = 9
year = 2015

# ## [Step 0](https://www.arpm.co/lab/redirect.php?permalink=s_stock_short_horizon-implementation-step00): Load data

# loading data from 2015-05-27 to 2015-12-07
path = '../../../databases/global-databases/high-frequency/db_stock_NOK_intraday/'
df_nokia_stock = pd.read_csv(path + 'data.csv',
                             header=0)
# convert column 'date' from string to datetime64
df_nokia_stock['date'] = pd.to_datetime(df_nokia_stock.date, dayfirst=True)

# ## [Step 1](https://www.arpm.co/lab/redirect.php?permalink=s_stock_short_horizon-implementation-step01): Select the data to be plotted

t_first = dt.datetime(year, month, day, 9, 30)  # starting time
t_last = dt.datetime(year, month, day, 16, 0)  # ending time
# select data
v_stock = df_nokia_stock[(df_nokia_stock.date >= t_first) &
                         (df_nokia_stock.date <= t_last)]

# ## Plots

# +
plt.style.use('arpm')
# extract values from dataframe
t = v_stock.date.values
v_t_stock = v_stock.price.values

number_of_xticks = 6
tick_array = np.linspace(0, t.shape[0]-1, number_of_xticks, dtype=int)
myFmt = mdates.DateFormatter('%H:%M:%S')

fig = plt.figure()
plt.plot_date(t, v_t_stock, '-')
plt.gca().xaxis.set_major_formatter(myFmt)
plt.xticks(t[tick_array])
plt.xlabel('Time')
plt.ylabel('Value')
plt.title(f'NOKIA intraday value on {dt.date(year, month, day)}')
add_logo(fig)


#!/usr/bin/env python
# coding: utf-8

# # 5 - Medium Level Example - Modeling Carports and Canopies + Sampling accross a Module!
#
# This journal shows how to model a carport or canopy ~ a fixed structure, usually at a high clearance from the ground, with more than one bifacial solar module in the same inclined-plane to create a "shade" for the cars/people below.
#
# We assume that bifacia_radiacne is already installed in yoru computer. This works for bifacial_radiance v.3 release.
#
# These journal outlines 4 useful uses of bifacial_radiance and some tricks:
#
#

#
 Creating the modules in the canopy/carport

#
 Adding extra geometry for the pillars/posts supporting the carport/canopy

#
 Sampling the rear irradiance with more resolution (more sensors)

#
 and hacking the sensor position to obtain an irradiance map of rear-irradiance.

#
 Adding an object to simulate a car with a specific reflectivity.

#

#
# This is what we will create:
# ![Carport Image We will create](../images_wiki/Carport.png)
#
# ### Steps:
#
#

#
  Setup of Variables through Making OCT Axis

#
  Adding the pillars

#
  Analysis of the collector width

#
  Mapping the irradiance througout all the Carport

#
  Adding a "Car"

#

#

#

# ### 1. Setup of Variables through Making OCT Axis
#
# We've done this before a couple times, no new stuff here.
#
# The magic is that, for doing the carport we see in the figure, we are going to do a 4-up configuration of modules (**numpanels**), and we are going to repeat that 4-UP 7 times (**nMods**)

# In[2]:

from bifacial_radiance import *
import numpy as np

# In[4]:

from pathlib import Path
testfolder = str(Path().resolve().parent.parent / 'bifacial_radiance' / 'TEMP')

timestamp = 4020 # Noon, June 17th.
simulationname = 'HotelCarport'

# MakeModule Parameters
moduletype='PrismSolar'
numpanels = 4  # Carport will have 4 modules along the y direction (N-S since we are facing it to the south) .
x = 0.95
y = 1.95
xgap = 0.15 # Leaving 15 centimeters between modules on x direction
ygap = 0.10 # Leaving 10 centimeters between modules on y direction
zgap = 0 # no gap to torquetube.
sensorsy = 10*numpanels  # this will give 70 sensors per module.

# Other default values:

# TorqueTube Parameters
axisofrotationTorqueTube=False
torqueTube = False
cellLevelModule = False

# SceneDict Parameters
gcr = 0.33   # We are only doing 1 row so this doesn't matter
albedo = 0.28  #'concrete'     # ground albedo
clearance_height = 4.3 # m
nMods = 7 # six modules length.
nRows = 1  # only 1 row

azimuth_ang=180 # Facing south
tilt =20 # tilt.

# Now let's run the example

demo = RadianceObj(simulationname,path = testfolder)  # Create a RadianceObj 'object'
demo.setGround(albedo) # input albedo number or material name like 'concrete'.  To see options, run this without any input.
epwfile = demo.getEPW(40.0583,-74.4057) # NJ lat/lon 40.0583° N, 74.4057
metdata = demo.readEPW(epwfile) # read in the EPW weather data from above
demo.gendaylit(4020)  # Use this to simulate only one hour at a time.
# This allows you to "view" the scene on RVU (see instructions below)
# timestam 4020 : Noon, June 17th.
#demo.genCumSky(demo.epwfile) # Use this instead of gendaylit to simulate the whole year

# Making module with all the variables
moduleDict=demo.makeModule(name=moduletype,x=x,y=y,numpanels = numpanels, xgap=xgap, ygap=ygap)
# create a scene with all the variables
sceneDict = {'tilt':tilt,'pitch': round(gcr/moduleDict['sceney'],3),'clearance_height':clearance_height,'azimuth':azimuth_ang, 'module_type':moduletype, 'nMods': nMods, 'nRows': nRows}
scene = demo.makeScene(moduletype=moduletype, sceneDict=sceneDict) #makeScene creates a .rad file with 20 modules per row, 7 rows.
octfile = demo.makeOct(demo.getfilelist())  # makeOct combines all of the ground, sky and object fil|es into a .oct file.

# If you view the Oct file at this point, you should see the array of 7 modules, with 4 modules each along the collector widt.

#

# ### 2. Adding the pillars
#
# We will add 4 pillars at roughly the back and front corners of the structure. Some of the code below is to calculate the positions of where the pillars will be at.
#
# We are calculating the location with some math geometry

# In[5]:

xright= x*4
xleft=  -xright

#centerhubheight = (1.9*3+1.9/2)*np.sin(tilt*np.pi/180)
y2nd = -(y*numpanels/2)*np.cos(tilt*np.pi/180) + (y)*np.cos(tilt*np.pi/180)
y6th=  -(y*numpanels/2)*np.cos(tilt*np.pi/180) + (y*numpanels)*np.cos(tilt*np.pi/180)
z2nd = (y*np.sin(tilt*np.pi/180))+clearance_height
z6th =  (y*numpanels)*np.sin(tilt*np.pi/180)+clearance_height

name='Post1'
text='! genbox black cuteBox 0.5 0.5 {} | xform -t -0.25 -0.25 0 -t {} {} 0'.format(z2nd, xleft, y2nd)
print (text)
customObject = demo.makeCustomObject(name,text)
demo.appendtoScene(radfile=scene.radfiles, customObject=customObject, text="!xform -rz 0")

name='Post2'
text='! genbox black cuteBox 0.5 0.5 {} | xform -t -0.25 -0.25 0 -t {} {} 0'.format(z2nd, xright, y2nd)
customObject = demo.makeCustomObject(name,text)
demo.appendtoScene(scene.radfiles, customObject, '!xform -rz 0')

name='Post3'
text='! genbox black cuteBox 0.5 0.5 {} | xform -t -0.25 -0.25 0 -t {} {} 0'.format(z6th, xright, y6th)
customObject = demo.makeCustomObject(name,text)
demo.appendtoScene(scene.radfiles, customObject, '!xform -rz 0')

name='Post4'
text='! genbox black cuteBox 0.5 0.5 {} | xform -t -0.25 -0.25 0 -t {} {} 0'.format(z6th, xleft, y6th)
customObject = demo.makeCustomObject(name,text)
demo.appendtoScene(scene.radfiles, customObject, '!xform -rz 0')

octfile = demo.makeOct(demo.getfilelist())  # makeOct combines all of the ground, sky and object files into a .oct file.

# ### View the geometry with the posts on :
#
# ## rvu -vf views\front.vp -e .01 -pe 0.4 -vp 3.5 -20 22 HotelCarport.oct
#
# -pe sets the exposure levels, and -vp sets the view point so the carport is centered (at least on my screen. you can play with the values). It should look like this:
#
# ![Carpport with posts](../images_wiki/Carport.png)
#
# The post should be coindient with the corners of the array on the high-end of the carport, and on the low end of the carport they should be between the lowest module and the next one. Cute!
#
#

#
#

# ### 3. Analysis of the collector width
#
# Now let's do some analysis along the slope of the modules. Each result file will contain irradiance for the 4 modules that make up the slope of the carport. You can select which "module" along the row you sample too.
#
# We are also increasign the number of points sampled accross the collector width, with the  variable **sensorsy** passed to **moduleanalysis**

# In[4]:

analysis = AnalysisObj(octfile, demo.name)  # return an analysis object including the scan dimensions for back irradiance
modWanted = 1
rowWanted = 1
frontscan, backscan = analysis.moduleAnalysis(scene, modWanted=modWanted, rowWanted=rowWanted, sensorsy=sensorsy)

analysis.analysis(octfile, simulationname+"Mod1", frontscan, backscan)  # compare the back vs front irradiance
print('Annual bifacial ratio average:  %0.3f' %( sum(analysis.Wm2Back) / sum(analysis.Wm2Front) ) )
print("")

# This is the module analysis and an image of the results file
# ![This is the module analysed.](../images_wiki/Carport_analysis.PNG)
#
# You can repeat the analysis for any other module in the row:
#
#

# Notice we are passing a CUSTOM simulation name so the results are generated in separate csv files.
#

#

# In[5]:

modWanted = 2
rowWanted = 1
frontscan, backscan = analysis.moduleAnalysis(scene, modWanted=modWanted, rowWanted=rowWanted, sensorsy=sensorsy)

analysis.analysis(octfile, simulationname+"Mod2", frontscan, backscan)  # compare the back vs front irradiance
print('Annual bifacial ratio average:  %0.3f' %( sum(analysis.Wm2Back) / sum(analysis.Wm2Front) ) )

modWanted = 3
rowWanted = 1
frontscan, backscan = analysis.moduleAnalysis(scene, modWanted=modWanted, rowWanted=rowWanted, sensorsy=sensorsy)

analysis.analysis(octfile, simulationname+"Mod3", frontscan, backscan)  # compare the back vs front irradiance
print('Annual bifacial ratio average:  %0.3f' %( sum(analysis.Wm2Back) / sum(analysis.Wm2Front) ) )

#

# ### 4. Mapping the irradiance througout all the Carport ("Hack" the sensors)
#
# You can "hack" the sensors starting locations to obtain an irradinace distribution map. This is easier when the modules are facing South, or East/West. Below is an example, you'll have to repeat over all the modules/ all the surface area with as much resolution as you have patience to see edge-effects.

# In[6]:

# HACK Frontscan and Backscan
frontscan['xstart']=-1.2

analysis.analysis(octfile, simulationname+"Mod3_point2", frontscan, backscan)  # compare the back vs front irradiance
print('Annual bifacial ratio average:  %0.3f' %( sum(analysis.Wm2Back) / sum(analysis.Wm2Front) ) )

#

# The printed Annual bifacial ratio does not include cleaning of the sensors for the material. Some of the sensros might fall in the spacing between the modules (ygaps) or in the structures added, torquetubes, etc. For a real bifacial ratio gain, use the load and clean functions in load.py.
#
# (This process might be automated in a future release TBD)
#

#
#

#

# ### 5. Adding a "Car"
#
# Add a surface (just like we added the pillars) with a specific reflectivity to represent a car. If you are doing hourly simulation you can compare how much the irradiance increases with and without the car, and if you keep track of your parking lot comings/goings this could make an interesting toy-problem: how much are your employees contributing to your rear irradiance production?

# In[7]:

name='Car_1'
carpositionx=-2
carpositiony=-1
text='! genbox white_EPDM HondaFit 1.6 4.5 1.5 | xform -t -0.8 -2.25 0 -t {} {} 0'.format(carpositionx, carpositiony)
customObject = demo.makeCustomObject(name,text)
demo.appendtoScene(scene.radfiles, customObject, '!xform -rz 0')

octfile = demo.makeOct(demo.getfilelist())  # makeOct combines all of the ground, sky and object files into a .oct file.

# Viewing with:
# ## rvu -vf views\front.vp -e .01 -pe 0.019 -vp 1.5 -14 15 HotelCarport.oct
#
#
# ![Behold the Honda-fit sized cube](../images_wiki/Carport_with_car.PNG)

# In[ ]:

docs/tutorials/5 - Medium Level Example - Bifacial Carports and Canopies + sampling across a module!.py


# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.13.8
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# # Raster-vector interactions {#raster-vector}
#
# ## Prerequisites

#| echo: false
import pandas as pd
import matplotlib.pyplot as plt
pd.options.display.max_rows = 6
pd.options.display.max_columns = 6
pd.options.display.max_colwidth = 35
plt.rcParams["figure.figsize"] = (5, 5)

# Let's import the required packages:

import numpy as np
import matplotlib.pyplot as plt
import geopandas as gpd
import rasterio
import rasterio.mask
from rasterio.plot import show

# and load the sample data:

src_srtm = rasterio.open("data/srtm.tif")
zion = gpd.read_file("data/zion.gpkg")
zion_points = gpd.read_file("data/zion_points.gpkg")

# ## Introduction
#
# ## Raster cropping
#
# Many geographic data projects involve integrating data from many different sources, such as remote sensing images (rasters) and administrative boundaries (vectors). Often the extent of input raster datasets is larger than the area of interest. In this case raster **cropping** and **masking** are useful for unifying the spatial extent of input data. Both operations reduce object memory use and associated computational resources for subsequent analysis steps, and may be a necessary preprocessing step before creating attractive maps involving raster data.
#
# We will use two objects to illustrate raster cropping:
#
# * The `srtm.tif` raster representing elevation (meters above sea level) in south-western Utah
# * The `zion.gpkg` vector layer representing the Zion National Park
#
# Both target and cropping objects must have the same projection. The following reprojects the vector layer `zion` into the CRS of the raster `src_srtm`:

zion = zion.to_crs(src_srtm.crs)

# To mask the image, i.e., convert all pixels which do not intersect with the `zion` polygon to "No Data", we use the `rasterio.mask.mask` function as follows:

out_image_mask, out_transform_mask = rasterio.mask.mask(
    src_srtm,
    zion["geometry"],
    crop=False,
    nodata=9999
)

# Note that we need to specify a "No Data" value in agreement with the raster data type. Since `srtm.tif` is of type `uint16`, we choose `9999` (a positive integer that is guaranteed not to occur in the raster).
#
# The result is the `out_image` array with the masked values:

out_image_mask

# and the new `out_transform`:

out_transform_mask

# Note that masking (without cropping!) does not modify the raster spatial configuration. Therefore, the new transform is identical to the original:

src_srtm.transform

# Unfortunately, the `out_image` and `out_transform` object do not contain any information indicating that `9999` represents "No Data". To associate the information with the raster, we must write it to file along with the corresponding metadata. For example, to write the cropped raster to file, we need to modify the "No Data" setting in the metadata:

out_meta = src_srtm.meta
out_meta.update(nodata=9999)
out_meta

# Then we can write the cropped raster to file:

new_dataset = rasterio.open("output/srtm_masked.tif", "w", **out_meta)
new_dataset.write(out_image_mask)
new_dataset.close()

# Now we can re-import the raster:

src_srtm_mask = rasterio.open("output/srtm_masked.tif")

# The `.meta` property contains the `nodata` entry. Now, any relevant operation (such as plotting) will take "No Data" into account:

src_srtm_mask.meta

# Cropping means reducing the raster extent to the extent of the vector layer:
#
# * To crop *and* mask, we can use the same in `rasterio.mask.mask` expression shown above for masking, just setting `crop=True` instead of `crop=False`.
# * To just crop, *without* masking, we can derive the extent polygon and then crop using it.
#
# For example, here is how we can obtain the extent polygon of `zion`, as a `shapely` geometry object:

bb = zion.unary_union.envelope
bb

# The extent can now be used for masking. Here, we are also using the `all_touched=True` option so that pixels partially overlapping with the extent are included:

out_image_crop, out_transform_crop = rasterio.mask.mask(
    src_srtm,
    [bb],
    crop=True,
    all_touched=True,
    nodata=9999
)

# Figure ... shows the original raster, and the cropped and masked results.

fig, axes = plt.subplots(ncols=3, figsize=(9,5))
show(src_srtm, ax=axes[0])
zion.plot(ax=axes[0], color="none", edgecolor="black")
show(src_srtm_mask, ax=axes[1])
zion.plot(ax=axes[1], color="none", edgecolor="black")
show(out_image_crop, transform=out_transform_crop, ax=axes[2])
zion.plot(ax=axes[2], color="none", edgecolor="black")
axes[0].set_title("Original")
axes[1].set_title("Mask")
axes[2].set_title("Crop");

# ## Raster extraction
#
# From points...
#
# From line...
#
# From polygon (srtm)...
#
# From polygon (nlcd)...
#
# ## Rasterization
#
# ...
#
# ## Spatial vectorization
#
# Spatial vectorization is the counterpart of rasterization (Section ...), but in the opposite direction. It involves converting spatially continuous raster data into spatially discrete vector data such as points, lines or polygons.
#
# There are three standard methods to convert a raster to a vector layer:
#
# * Raster to polygons
# * Raster to points
# * Raster to contours
#
# The most straightforward form of vectorization is the first one, converting raster cells to polygons, where each pixel is represented by a rectangular polygon. The second method, raster to points, has the additional step of calculating polygon centroids. The third method, raster to contours, is somewhat unrelated. Let us demonstrate the three in the given order.

src = rasterio.open("data/grain.tif")

# To polygons... ...
#
#
# FIGURE 6.9: Illustration of vectorization of raster (left) into polygons (dissolve = FALSE; center) and aggregated polygons (dissolve = TRUE; right).

src = rasterio.open("data/elev.tif")

# To points...

src = rasterio.open("data/elev.tif")

# To contours...
#
# ...
#
# Another common type of spatial vectorization is the creation of contour lines representing lines of continuous height or temperatures (isotherms) for example. We will use a real-world digital elevation model (DEM) because the artificial raster elev produces parallel lines (task for the reader: verify this and explain why this happens). Contour lines can be created with the terra function as.contour(), which is itself a wrapper around filled.contour(), as demonstrated below (not shown):
#
# Contours can also be added to existing plots with functions such as contour(), rasterVis::contourplot() or tmap::tm_iso(). As illustrated in Figure 6.8, isolines can be labelled.
#
# The final type of vectorization involves conversion of rasters to polygons. This can be done with terra::as.polygons(), which converts each raster cell into a polygon consisting of five coordinates, all of which are stored in memory (explaining why rasters are often fast compared with vectors!).
#
# This is illustrated below by converting the grain object into polygons and subsequently dissolving borders between polygons with the same attribute values (also see the dissolve argument in as.polygons()).
#
#
#
# ## Exercises
#


#!/usr/bin/python

'''
This is assignment 3, group submission by :
    Student Name : Ravi Mishra Student ID: 21249928
    Student Name : Prasad Deshpande Student ID: 21249530

    Link to Github Repository : https://github.com/ravi3990/ARC.git

'''

import os, sys
import json
import numpy as np
import re
import math

### YOUR CODE HERE: write at least three functions which solve
### specific tasks by transforming the input x and returning the
### result. Name them according to the task ID as in the three
### examples below. Delete the three examples. The tasks you choose
### must be in the data/training directory, not data/evaluation.

'''
solve_d4a91cb9:

Difficulty Level according to us : Easy to Medium

Logic :

The basic logic for solving this problem through the naked eye and to a layman
person would be get the coordinates of the two colours ie sky blue and red.The
values for these in our list is 2 and 8.And then connect the 2 points throug
it's row and column by two lines of yellows till it intersects.

We used the numpy where function to find the coordinates of these points.The main
problem in solving this issue was to find out in what direction the line has to be plotted.
For this we used the if else to determine if blue has higher column value or red.
Based on that we update the corresponding values in the given row and column to 4.

Basic python and numpy used.Was able to solve all inputs and outputs in the json
file successfully.

'''

def solve_d4a91cb9(x):
    #Changing to Numpy array in case the input is a list
    b=np.array(x)
   #Getting the position of 8 and 2 in the array
    eight_pos=np.where(b==8)[0][0],np.where(b==8)[1][0]
    two_pos=np.where(b==2)[0][0],np.where(b==2)[1][0]
    #Changing the column values to 4
    if eight_pos[0]<two_pos[0]:
        for i in range (eight_pos[0]+1,two_pos[0]+1):
            b[i][eight_pos[1]]=4
    elif eight_pos[0]>two_pos[0]:
        for i in range (two_pos[0],eight_pos[0]):
            b[i][eight_pos[1]]=4

    #Changing the row  values to 4
    if eight_pos[1]<two_pos[1]:
        for i in range (eight_pos[1]+1,two_pos[1]):
            b[two_pos[0]][i]=4
    elif eight_pos[1]>two_pos[1]:
        for i in range (two_pos[1]+1,eight_pos[1]+1):
            b[two_pos[0]][i]=4
    return b

'''
solve_f5b8619d:

Difficulty Level according to us : Easy

Logic :

On a first look it looked a little difficult to us.However we could quickly understand
that it is a mirror effect after some basic changes to the inital array.
In the initial array we have to identify the columns where the value is non zero.
We have to change the value of all the rows for that column without touching the original
values to 8.Once it is completed we have to mirror the array along the horizontal
axis once and then the vertical axis once.This can be done interchangeably as well.

Basic python and numpy has been used.

All the test inputs and outputs were successfully validated and passed.

'''

def solve_f5b8619d(x):
    new_arr=np.array(x)
    #Getting the columns where the value is non zero and changing rest of the column values to 8
    for mn in np.where(new_arr!=0)[1]:
        for i in range(new_arr.shape[1]):
            if new_arr[i][mn]==0:
                new_arr[i][mn]=8
     #Mirroring the array horizontally and vertically
    new_arr=np.concatenate((new_arr,new_arr),axis=0)
    new_arr=np.concatenate((new_arr,new_arr),axis=1)
    return new_arr

'''
solve_0a938d79:

Difficulty Level according to us : Easy

We found Task 0a938d79 simple to understand and implement. Looking at task demonstration
we can see alternate coloured line pattern repeats until end of array. If number of rows are
more than the columns then line patterns are repeated until last row. If the number of
columns are more than the rows then the line patterns are repeated until end of last column.
Count for intermediate black rows are dependent on number of columns/rows between given
two colour cells.

Solver function checked shape of array, colours of non-black locations and loop through
the rows/columns until max number of rows/columns reached. Coloured locations/cells are
used as reference to paint the location to create the line in X or Y axis and new array
return after transformation.

All training and test grids solved correctly.

'''

def solve_0a938d79(x):
    '''empty list to collect coloured cells'''
    non_zero = []

    ''' copy input array'''
    new_arr=np.array(x)

    ''' get number of rows are colums with shape'''
    num_rows, num_cols = new_arr.shape

    '''find the non-zero coordinates '''
    non_zero = np.nonzero(new_arr)

    '''find the colour at non-zero coordinates '''
    colour_1 = new_arr[non_zero[0][0], non_zero[1][0]]
    colour_2 = new_arr[non_zero[0][1], non_zero[1][1]]

    '''if number of columns are greater than rows then patterns need to be
    formed in x-axis else in y-axis'''

    if(num_cols > num_rows):
        black_mid_segments = abs(non_zero[1][1] - non_zero[1][0])
        while((non_zero[1][0] or non_zero[1][1]) < num_cols):
            '''fill the colour at non-zero column'''
            for i in range(num_rows):
                if(non_zero[1][0] < num_cols):
                    new_arr[i,non_zero[1][0]] = colour_1
                if(non_zero[1][1] < num_cols):
                    new_arr[i,non_zero[1][1]] = colour_2
            '''increment coloumn index based number of empty col. in pattern'''
            non_zero[1][0] += (black_mid_segments*2)
            non_zero[1][1] += (black_mid_segments*2)
    else:
        black_mid_segments = abs(non_zero[0][0] - non_zero[0][1])
        while((non_zero[0][0] or non_zero[0][1]) < num_rows):
            '''fill the colour at non-zero rows'''
            for i in range(num_cols):
                if(non_zero[0][0] < num_rows):
                    new_arr[non_zero[0][0], i] = colour_1
                if(non_zero[0][1] < num_rows):
                    new_arr[non_zero[0][1], i] = colour_2
            '''increment row index based number of empty col. in pattern'''
            non_zero[0][0] += (black_mid_segments*2)
            non_zero[0][1] += (black_mid_segments*2)

    ''' return the modified array'''
    return new_arr

'''
solve_1a07d186(x)

Difficulty Level according to us : Medium to difficult

We found Task 1a07d186 medium to be difficult. On the first look, task seems easy
i.e. just moving the coloured cells adjacent to respective lines. But it is also important to
identify line as object, move coloured cells in right direction and to remove other
coloured cells which doesn’t have coloured line present. The same logic then needs to be
defined for vertical and horizontal lines (single/multiple).

We decided to implement the solver function with basic logic and with less use of
library APIs. Numpy is used only where necessary, so the code looks big. It identified
colour in array and marked the coordinates. Then identified the line object with traversing
in X/Y axis from edge to edge to confirm the line position. Once all these information
is available, then look for nearest line coordinate for coloured cell and paint next
location in same row/column. The original colour location is then overwritten with black.
In case of cells with colours other than line colour, the location is overwritten with black.

All training and test grids solved correctly.

'''

def solve_1a07d186(x):

    ''' copy input array'''
    new_arr=np.array(x)

    ''' get number of rows are colums with shape'''
    num_rows, num_cols = new_arr.shape

    '''find the non-zero coordinates '''
    non_zero = np.nonzero(new_arr)

    ''' index for checking all coordinates around the colour'''
    Dy = [0, -1, 0, 1]
    Dx = [1, 0, -1, 0]

    ''' empty lists to log if Hlines or Vlines found in image'''
    H_line = []
    V_line = []

    ''' loop to check colour around all identified colour co-ordinates '''
    for x, y  in zip(non_zero[0],non_zero[1]):
        colour = new_arr[x][y]

        for dy, dx in zip(Dy, Dx):
            ''' if same colour found then move either x or y axis to find if it is a line'''
            x_, y_ = x+dx, y+dy

            ''' loop through number of cols or rows to find lines and log into list '''
            if(x_ < num_rows and y_ < num_cols):
                if(colour == new_arr[x_][y_]):
                    if(abs(x_ - x)):
                        #vertical line
                        for k in range(num_rows):
                            if(colour == new_arr[k][y]):
                                V_line.append(y)
                                v_line_flag = True

                    if (abs(y_-y)):
                        #horizontal line
                        for k in range(num_cols):
                            if(colour == new_arr[x][k]):
                                H_line.append(x)
                                v_line_flag = False

    ''' consider only unique items to give exact number of lines and their one
    coordinates location. Other location can be assumed zero as it is a line'''
    H_line = np.unique(H_line)
    V_line = np.unique(V_line)

    ''' loop through all coloured cellls to either associate with line or if
    not then make it zero'''
    for x, y  in zip(non_zero[0],non_zero[1]):
        if(v_line_flag == True):
            ''' if y is not in Vline then proceed '''
            if(y not in V_line):
                colour_flag = False
                colour = new_arr[x][y]

                for i in V_line:
                    if(colour == new_arr[0, i]):
                        if(i < y):
                            new_arr[x,i+1] = colour
                        else:
                            new_arr[x,i-1] = colour

                        new_arr[x][y] = 0
                        colour_flag = True
                ''' if colour cell is not associated with line colour then
                make it zero '''
                if(colour_flag == False ):
                    new_arr[x][y] = 0

        if(v_line_flag == False):
            '''if x is not in Hline then proceed '''
            if(x not in H_line):
                colour_flag = False
                colour = new_arr[x][y]

                for i in H_line:
                    if(colour == new_arr[i, 0]):
                        if(i < x):
                            new_arr[i+1, y] = colour
                        else:
                            new_arr[i-1, y] = colour

                        new_arr[x][y] = 0
                        colour_flag = True
                ''' if colour cell is not associated with line colour then
                make it zero '''
                if(colour_flag == False ):
                    new_arr[x][y] = 0

    ''' return the modified array'''
    return new_arr

'''
def solve_00d62c1b(x)

Difficulty Level according to us : Medium to difficult

We found task 00d62c1b medium to be difficult. The challenge was in finding the enclosed
shape to fill the space with yellow colour. The logic which is applied here is same
as previous task 1a07d186, instead of lines, here is to identify the space enclosed
with green colour.

Same logic of x, y location +/-1 for checking all coordinates around the coloured
cell used. Code optimized with the use of list as queue and multiple conditions in loops.
‘Searched’ array created with padding of rows and columns. Based on search the true/False
marked and resultant used for filling yellow colour.
The in-line comments give more details of logic.

All training and test grids solved correctly.

'''

def solve_00d62c1b(x):

    ''' copy input array'''
    new_arr=np.array(x)

    ''' get number of rows are colums with shape'''
    num_rows, num_cols = new_arr.shape

    green = 3
    yellow = 4

    ''' index for checking all coordinates around the colour'''
    Dy = [0, -1, 0, 1]
    Dx = [1, 0, -1, 0]

    '''
    https://numpy.org/doc/stable/reference/generated/numpy.pad.html

    ((1,1),(1,1)) -Number of values padded to the edges of each axis. ((before_1, after_1)
    pads with 'Constant' value of zero [False]

    '''

    arr_padded = np.pad(new_arr, ((1,1),(1,1)), "constant", constant_values=0)

    '''
    input array padded on x and y axis at the edge i.e. num_col and num_rows
    increased by 2 and created searched array with default False in each element'''
    search_empty_sq = np.zeros(arr_padded.shape, dtype=bool)
    search_empty_sq[0, 0] = True

    queue = [(0, 0)]
    while queue:
        ''' take each element/cell from matrix'''
        j, i = queue.pop()

        ''' loop for near by coordinates in x, y axis '''
        for dy, dx in zip(Dy, Dx):
            y_, x_ = j+dy, i+dx

            if not 0 <= y_ < num_rows+2 or not 0 <= x_ < num_cols+2:
                ''' skip the next logic for edge elements/cells'''
                continue
            ''' in case of identical elements change value at searched to True
                else maintain the value at False. False indicates yellow colour
                to add
            '''
            if not search_empty_sq[y_][x_] and arr_padded[y_][x_]==0:
                '''add to queue for next operation '''
                queue.append((y_, x_))
                search_empty_sq[y_, x_] = True

    '''remove the padding'''
    res = search_empty_sq[1:-1, 1:-1]

    ''' check the green and keep it as it is'''
    res |= new_arr==green

    ''' invert the searched results and fill true with yellow '''
    new_arr[~res] = yellow

    ''' return the modified array'''
    return new_arr

'''
def solve_d0f5fe59(x)

Difficulty level according to us : Medium

Logic:
The idea behind solving this problem was to get the number of islands within a
particular array.This was done through a depth first search.Once a particulat
island is visited it is not counted twice.For this we created an internal functions
called dfs.

After iterating through the entire array we found the number of islands.We then
created a numpy zero array with the same dimensions as the number of islands.
We then update the diagonal of the array with the value 8.

We were able to succsfully validate and test all the inputs with their given outputs
as in the json file.

'''

def solve_d0f5fe59(x):
    row=len(x)
    col=len(x[0])
    count=0
   #Creating a function for Depth first search for finding the islands
    def dfs(a,row,col,m,n):
        if a[m][n]==0:
            return
        a[m][n]=0

        if m!=0:
            dfs(a,row,col,m-1,n)

        if m!=row-1:
            dfs(a,row,col,m+1,n)

        if n!=0:
            dfs(a,row,col,m,n-1)
        if n!=col-1:
            dfs(a,row,col,m,n+1)
            #Iterating to count the number of islands
    for i in range(row):
        for j in range(col):
            if x[i][j]==8:
                dfs(x,row,col,i,j)
                count+=1
     #Creating a zero numpy array with the same dimensions as the number of islands
    b=np.zeros((count, count), int)
    #Filling the diagonal with the value 8
    np.fill_diagonal(b,8)

    return b

'''
def solve_4093f84a(x)

Difficulty Level  according to us : Medium to Difficult

Logic :
4093f84a.json looked to be simple on the testing interface initially.Logically
we needed to find the rectangle with 5's and then determine whether it was a horizontal
one or a vertical one.After that close to the the rectangle  we had to fill the 5 in
the respective rows or columns.

We tried to have a simple approach of finding the bigger island with the rectangle first.
After that we made all the other positions as zero.We noted the positions of the other
values.And then added them to the rectangle in their respective rows and columns.

The given code use basic python and numpy functions.
While testing this it passed for the test demonstration 1.However there was a
slight difference in demosntraions 2 and 3 as there were multiple non 5 values in the
same row or column.This made it to not completely satisfy the outputs for 2 and 3 and test input grid.

The idea to solve this would be to get the positions ond the number of such colours
in a particular row or column.Once we get the number we can add to the same side of
the rectangle the cells and this would result in the success of the function

'''

def solve_4093f84a(x):
    x=np.array(x)
    #Getting positions where the value is non zero
    mn=np.where(x!=0)
    #Getting positions where the value is five
    yz=np.where(x==5)
    emp_lst=[]
    #Adding to an empty list the positions of the non 5 and non zero values
    for i in range(len(mn[0])):
        if x[mn[0][i]][mn[1][i]]!=5:
            emp_lst.append([mn[0][i],mn[1][i]])
    #Making all the non island positions 0 before actioning
    for m in emp_lst:
        x[m[0],m[1]]=0
    #Action to alter the value to 5 for the respective postions near to the island
    if np.amin(yz[1])==0:
        for m in emp_lst:
            if m[0]<np.amin(yz[0]):
                x[np.amin(yz[0])-1][m[1]]=5
            else :
                x[np.amax(yz[0])+1][m[1]]=5
    else :
        for m in emp_lst:
            if m[1]<np.amin(yz[1]):
                x[m[0]][np.amin(yz[1])-1]=5
            else :
                x[m[0]][np.amax(yz[1])+1]=5
    return x

'''
def solve_5ad4f10b(x)

Difficulty Level according to us : Difficult

Logic:

For solving 5ad4f10b.json we tried to look at various ways for coming up.It seemed
a little difficult.The main challenge was to find the area for the section which was
to be contracted and then the values updated with the other colour outliers.

Following is the approach we followed for this section :

1.Identify the n*n squares(start with 3) in the given array,their coordinates as
well and store their values in separate lists.
2.Get the max and min values of the rows and column numbers for the areas containing squares.
3.Create a new array with only the rows and columns which  was obtained from Step 2
4.Clear any other colour values than 0 and the squares and convert it to 0.
5.Get the other colour value from the original array and create a new array with the specific positions.

We have used basic numpy and math library to get this output.

The one limitation to this part of the code is that it is currently useful for
3*3 squares only(Training Input and output 2 currently. We can use a similar version of
the same code by altering the for loops a little to make it working for 4*4 squares as well.
Due to timing limitations we have not tried it. However we believe the logic would be near
about the same and would work for other training and test grids as well with slight modifications.
'''

def solve_5ad4f10b(x):
    a=np.array(x)
    row,colmn=a.shape[0],a.shape[1]

     #Code to get the 3*3 squares inside the array

    emp_lst1=[]
    range_lst1=[]
    #comp=0
    for i in range(row):
        for j in range(colmn):
            emp_lst=[]
            range_lst=[]
            comp=0
            if i-1>0 and j-1>0 and i+1<row and j+1<colmn:
                for m in range(i-1,i+2):
                    for n in range(j-1,j+2):
                        emp_lst.append(a[m][n])
                        range_lst.append ((m,n))

            if len(emp_lst)>0 and emp_lst[0]!=0:
                comp=1
                #print (emp_lst )
                for mn in emp_lst:
                    if emp_lst[0]!=mn:
                        comp+=1
            #print(comp)
            if comp==1 and len(emp_lst)>0:
                #print(emp_lst)
                #print(range_lst)
                emp_lst1.append(emp_lst)
                range_lst1.append(range_lst)

    #Getting the minimum and maximum section of the array to be cut

    min_row=range_lst1[0][0][0]

    for m in range_lst1:
        for n in m:
            if n[0]<min_row:
                min_row=n[0]

    min_col=range_lst1[0][0][1]

    for m in range_lst1:
        for n in m:
            if n[1]<min_col:
                min_col=n[1]
    max_row=range_lst1[0][0][0]

    for m in range_lst1:
        for n in m:
            if n[0]>max_row:
                max_row=n[0]

    max_col=range_lst1[0][0][1]

    for m in range_lst1:
        for n in m:
            if n[1]>max_col:
                max_col=n[1]

    new_arr=a[min_row:max_row+1,min_col:max_col+1]
    #Clear the array for any oher number which are outliers
    new_row,new_col=new_arr.shape[0],new_arr.shape[1]

    for i in range(new_row):
        for j in range(new_col):
            if new_arr[i][j]!=emp_lst1[0][0]:
                new_arr[i][j]=0

    new_mod_arr=np.zeros((int(math.sqrt(new_row)),int(math.sqrt(new_col))))
    row_alt,col_alt=new_mod_arr.shape[0],new_mod_arr.shape[1]

    #Getting the colour value which has to be updated
    non_zero_val_loc=np.where(a!=0)

    for m in non_zero_val_loc[0]:
        for n in non_zero_val_loc[1]:
            if a[m][n]!=emp_lst1[0][0]:
                val_to_update=a[m][n]

    #Updating the new modified array with the intended values
    for i in range(0,new_row,3):
        for j in range(0,new_col,3):
            if new_arr[i][j]!=0:
                new_i=int(i/3)
                new_j=int(j/3)
            #print (new_i,new_j)
                new_mod_arr[new_i,new_j]=val_to_update
    return new_mod_arr

'''

A short summary/reflection, commenting on the Python features and libraries you used in the solve *

In first semester of MSc. while learning and working on assignments we could see maturity
of available ML packages which can trained on different kind of tasks e.g., NLP task for
suggestion mining or regression tasks for prediction of home prices etc. but while working
with ARC, we realized that we’re still very far away from building a human like intelligence.
All the tasks (.JSON files) when looked in browser, one can identify the patterns in seconds
just looking at few samples, but it is not easy for ML/programming to generalize this type
of intelligence. We didn’t find much help on generalised pattern recognition available,
and ARC is an upcoming new topic in field of AI. We both found it is super interesting
and addictive i.e., we initially though to just complete 3-4 patterns and submit the assignment
but end up coding 8-9 patterns!

We did not use any ML package or specific libraries but used Numpy and pure python to
translate our understanding of patterns into code. We found that, in future there can
be few common functions/libraries be made to recognize the patterns e.g., find specific
shape objects or enclosed spaces, highlight repetitive patterns, symmetry etc. or correlate
the colours in base image to transformed image. These functions in addition to math
libraries can be then basis for creating hypothesise in terms of hyper parameters or
support functions. We still have limited understanding of this topic and acknowledge
that writing the code to solve all sorts of problems i.e., generalisation is not easy
task.

Commonalities between the problems :

->Pattern Recognition is very important.
->Many problems include finding the position of enclosed areas and islands.
->In many problems we had dimensionality reshaping as well as revaluation of the array.
->Most of the problems can be solved via if else loops or basic libraray functions.
  However as mentioned before  there is no one off solution to all problems

Github has been thoroughly used by us for core sharing and working as a team of 2.
Readme file has been altered as well for reflecting the ARC understanding.

As mentioned by François Chollet in his paper 'The Measure of Intelligence' the
solutions can be generalised only to a limited extent because of some of the limitations
which we could see while solving our given problems.The major limitations were a
generalisation of the problem and limited set of inputs and outputs for training
on the same problem. We could only test what was given to use.We are however aware
that there will be many variations in the given formations.

No doubt there are still many leaps to be taken by Arificial Intelligence!

'''

def main():
    # Find all the functions defined in this file whose names are
    # like solve_abcd1234(), and run them.

    # regex to match solve_* functions and extract task IDs
    p = r"solve_([a-f0-9]{8})"
    tasks_solvers = []
    # globals() gives a dict containing all global names (variables
    # and functions), as name: value pairs.
    for name in globals():
        m = re.match(p, name)
        if m:
            # if the name fits the pattern eg solve_abcd1234
            ID = m.group(1) # just the task ID
            solve_fn = globals()[name] # the fn itself
            tasks_solvers.append((ID, solve_fn))

    for ID, solve_fn in tasks_solvers:
        # for each task, read the data and call test()
        directory = os.path.join("..", "data", "training")
        json_filename = os.path.join(directory, ID + ".json")
        data = read_ARC_JSON(json_filename)
        test(ID, solve_fn, data)

def read_ARC_JSON(filepath):
    """Given a filepath, read in the ARC task data which is in JSON
    format. Extract the train/test input/output pairs of
    grids. Convert each grid to np.array and return train_input,
    train_output, test_input, test_output."""

    # Open the JSON file and load it
    data = json.load(open(filepath))

    # Extract the train/test input/output grids. Each grid will be a
    # list of lists of ints. We convert to Numpy.
    train_input = [np.array(data['train'][i]['input']) for i in range(len(data['train']))]
    train_output = [np.array(data['train'][i]['output']) for i in range(len(data['train']))]
    test_input = [np.array(data['test'][i]['input']) for i in range(len(data['test']))]
    test_output = [np.array(data['test'][i]['output']) for i in range(len(data['test']))]

    return (train_input, train_output, test_input, test_output)

def test(taskID, solve, data):
    """Given a task ID, call the given solve() function on every
    example in the task data."""
    print(taskID)
    train_input, train_output, test_input, test_output = data
    print("Training grids")
    for x, y in zip(train_input, train_output):
        yhat = solve(x)
        show_result(x, y, yhat)
    print("Test grids")
    for x, y in zip(test_input, test_output):
        yhat = solve(x)
        show_result(x, y, yhat)

def show_result(x, y, yhat):
    print("Input")
    print(x)
    print("Correct output")
    print(y)
    print("Our output")
    print(yhat)
    print("Correct?")
    if y.shape != yhat.shape:
        print(f"False. Incorrect shape: {y.shape} v {yhat.shape}")
    else:
        print(np.all(y == yhat))

if __name__ == "__main__": main()