#!/usr/bin/env python
# coding: utf-8

# In the [last notebook](https://github.com/caiomiyashiro/RecommenderSystemsNotebooks/blob/master/Month%202%20Part%20I%20-%20User%20User%20Collaborative%20Filtering.ipynb), we took a look at the Nearest Neighboor User User CF, a form of recommender system that looked at the similarity between users to define which item should be suggested to a new customer.
#
# We saw the benefits from User User CF when comparing it to non personalised and content based recommendations, but we also saw that it comes with one difficulty.
#
# - It doesn't scale well. Even in a big e-commerce dataset, the amount of intersecting items between 2 users is not as big as it could be when User User CF was created. Because of this, when a user bought an item that intersected now with another different customer, the new similarity could drastically change, causing the system's owners to recalculate the new similarities very often.
#
# - If they don't update the matrix often, they can also lose profits over it because it wouldn't map the user's short term interests which, on the internet, can vary quite a lot.
#
# All in all, short term interest and sparse mutual interest space make the User User CF inapt for high scale companies.
#
# # Item Item Collaborative Filtering (CF)
#
# [Item Item CF](https://en.wikipedia.org/wiki/Item-item_collaborative_filtering) was created by ([Sarwar et all, 1998](https://patentimages.storage.googleapis.com/41/80/fb/07d4d9e61e7431/US6266649.pdf)) in partneship with Amazon in order to fix the problems with the User User CF. In Item Item perspective, as the name suggests, changes the perspective from User centered to a Item centered view, *i.e.*, instead of having a User User similarity matrix, they started to use a item item similarity matrix. Then, when a user $u$ bought and liked an item $i_{1}$ and $i_{1}$ was similar to item $i_{2}$, then we predicted that $u$ would also like $i_{2}$. Take a look at the image below:
#
#
#
# Why this simple change in perspective helped to solve the inneficiency problems present in the User User CF?
#
# By considering an enviroment of a big e-commerce company, we end with the number of users >> number of items.
# In this case, even if a single user hasn't given many reviews, the chances are that many users have given a review to a specific item.
# By having a big number of reviews, an item relationship to other items doesn't change too much by receiving a few more reviews, *i.e.*, item item relationship are more stable. Therefore, by being more stable, the similarity matrix doesn't have to be recalculated often, as in the User User CF.
#
# An extra perfomance improvement comes also from the prediction calculation. In the Item Item CF, a new prediction for a user $u$ for a product $p$ is made by retrieving the items similarities and calculating a weighted average. The number of neighboors for this calculation is only the item that $u$ has liked or bought in the past and this number is often small enought. Therefore, we don't need to search the big user user similarity matrix to find the best $k$ neighboors.
#
# # Item Item Steps
#
# As always, we're going to work with one of the datasets from the [Coursera's Specialization on Recommender Systems](https://www.coursera.org/specializations/recommender-systems). This dataset is from the last week in the course of [Nearest Neighboors CF](https://www.coursera.org/learn/collaborative-filtering) for Item Item CF. Dataset is [here](https://d396qusza40orc.cloudfront.net/umntestsite/on-demand_files/A5/Assignment%205.xls) (Coursera's page) and [here](https://github.com/caiomiyashiro/RecommenderSystemsNotebooks/blob/master/data/Item%20Item%20Collaborative%20Filtering%20-%20Ratings.csv) (personal Github account).
#
# The steps taken to evaluate and recommend are similar to User User CF, with some different calculations in the prediction step, as we've said.
#
# - Load traditional input - User Item Review dataset
# - Create similarity matrix
# - Make predictions
#
# Lets go!
#
# ## Example Dataset
#
# The dataset is a matrix with size 25 users x 25 movies and each cell $c_{u,m}$ contains the rating user $u$ gave to movie $m$. If user $u$ didn't rate movie $m$, the cell is empty. As the float values were stored with commas and consequently were being casted as strings, I had to process it a little bit to replace the commas for dots and then convert the column to floats

# In[1]:

import pandas as pd
import numpy as np

# In[2]:

df = pd.read_csv('data/Item Item Collaborative Filtering - Ratings.csv', index_col=0, nrows=20)

df.drop('Mean', axis=1, inplace=True) # remove mean column that comes at the end

# replace commas for dots and convert previous string column into float
def processCol(col):
    return col.astype(str).apply(lambda val: val.replace(',','.')).astype(float)
df = df.apply(processCol)

print('Dataset shape: ' + str(df.shape))
df.head()

# ## Create Similarity Matrix
#
#
# ### Similarity Function
#
# As for the User User CF, we have a few possibilities to choose from when deciding how we're going to define if an item is similar to another item. Again, ([Herlocker et all, 2002](https://grouplens.org/site-content/uploads/evaluating-TOIS-20041.pdf)) did an analysis on the performance of these metrics on Item Item CF and realised that, for this case, the [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) was the best performant metric. So we're going with them this time. On the **next notebook**, we try to analyse these metrics and see why it performs better in certain cases and others not.
#
#
# ## Calculating User User Similarity with Cosine Similarity:
#
# One important point here is on the calculation of the denominator of the cosine similarity. Even though we make the dot product only with values existing in both arrays, the norm of the individual vectors are considering all values, and not the intersections between array1 and array2.

# In[3]:

def cos_similarity(item1, item2):
    item1Values = ~np.isnan(item1)
    item2Values = ~np.isnan(item2)
    allValues = np.logical_and(item1Values,item2Values) # get only existent elements of both vectors
    return np.dot(item1[allValues], item2[allValues])/(np.linalg.norm(item1[item1Values]) * np.linalg.norm(item2[item2Values]))

def pre_cos_similarity(item1, df):
    return df.apply(lambda item2: cos_similarity(item1, item2))

df_corr = df.apply(lambda item1: pre_cos_similarity(item1, df))
df_corr.head()

# ## Predictions Calculation
#
# By now we already know which items are more similar to each other. This will help us when predicting a new rating, by giving higher weights for more similar items than other the user has bough.
#
# The way we're going to calculate the new predictions is the same we used for User User CF, *i.e.*, a weighted average:
#
# $$\frac{\sum_{n=1}^{k} r_{n}w_{n}}{\sum_{n=1}^{k} w_{n}}$$
#
# The difference is that we don't have the neighboors anymore, so the $n$ in the summation is considering **all** the items user $u$ has rated and $w$ is still the similarities, but now *item similarity*.

# In[4]:

def predictRating(userRatings, itemSimilarity):
    userHasRating = ~np.isnan(userRatings)
    return np.dot(userRatings[userHasRating], itemSimilarity[userHasRating])/np.sum(itemSimilarity[userHasRating])

def pre_predictRating(userRatings, df_corr):
    return df_corr.apply(lambda itemSimilarity: predictRating(userRatings, itemSimilarity))

predictions = df.apply(lambda userRatings: pre_predictRating(userRatings, df_corr), axis=1)
predictions.head()

# ## Mean Normalised Weighted Average
#
# As in the same way of the User User CF, we can calculate predictions using the absolute value of the reviews or from the mean centralised values of it. The advantages are the same: consider the scale variability of reviewers when attributing a final score for a item of interest:
#
# $$\bar{r_{u}} + \frac{\sum_{n=1}^{k} (r_{n} - \bar{r_{n}})w_{n}}{\sum_{n=1}^{k} w_{n}}$$
#
# We took the same function as above, but added two extra parameters:
# - $userMeanRating$: mean average ratings for a specific user
# - $neighboorsMeanRating$: mean average rating for all the nearest neighboors for a specific user

# In[5]:

# mean normalise
def subtractFromMean(col, meanCol):
    result = np.array([np.nan] * col.shape[0])
    isValidValue = ~np.isnan(col)
    result[isValidValue] = col.values[isValidValue] - meanCol.values[isValidValue]
    return result
userMeanRatings = df.apply(np.mean, axis=1)
df_ratings_norm = df.apply(lambda col: subtractFromMean(col, userMeanRatings))

# similarity matrix
df_corr_norm = df_ratings_norm.apply(lambda item1: pre_cos_similarity(item1, df_ratings_norm))

# ### Remove negative correlations
#
# In this example, we are replacing the negative correlations by 0, as we can interpret as a maximum weight for unwanted items:
#

# In[6]:

def replaceNegative(col):
    col[col < 0] = 0
    return col
df_corr_norm2 = df_corr_norm.apply(replaceNegative)

# ### Predict!

# In[7]:

predictions_norm = df.apply(lambda userRatings: pre_predictRating(userRatings, df_corr_norm2), axis=1)
predictions_norm.head()

# * I didn't quite understand why we didn't use the not normalised item's ratings in this calculation. Ideally, we would use the mean centered user ratings as well..

# # Comparison Between Approaches
#
# The comparison follows the same guidelines we used when evaluating the User User CF.
#
#
# # Final Considerations on User User CF
#
# When we mean centered the user's rating for the User User CF, the objective was clear, we wanted to take into account that users rate in different parts of the scale. But what about mean centering for the Item Item CF? This evaluation I'll leave it to the next notebook, where we evaluate and compare the main metrics used for similarity calculation in a CF system.
#
# --
#
# Item Item CF brings efficiencies steps forward from the User User CF schema. With it, we bring personalised recommendations and in a way that is computationally efficient to scale for giant e-commerce companies, such as Amazon or Netflix. But Item Item CF isn't a gold system, where we can implement it and always get good results. It has a few premisses:
#
# - First, it has the premisse that number of users >> number of items. This is a prerequisite to have stable entities, items in this case, and doesn't need to recalculate the similarity matrix often, as in the User User CF.
#
#
# - Secondly, and this is an interesting feature, Item Item CF is better when the item ratings are stable, *i.e.*, they have lots of evaluations. This means that the user's items are probably going to have a lot of influence from these popular items and, at the end, receiving popular items recommendations. This is good when you want to be safe about your recommendations, such as expensive services or products or rarely bought, such as houses or cars. However, this lack of '*serendipity*' is missed when we want to enable users to find that particular rare item and amazingly matched with your tastes. As an example, If we take Spotify, we don't want to receive recommendations such as 'Hey, as you listened to Mozart, here is what we think you'd like: Bach'. Spotify greatness works on the premisse of finding the bands and songs that can surprise you, so they wouldn't be effective by working on the Item Item CF schema. Of course, we are going to see more advance techniques in the future where these companies apply modern algorithms to have good recommendations and still be performatic, but the idea now was to show how we can't rely on one algorithm as the best of them all.
#
#
#
#
# In the [next notebook](https://github.com/caiomiyashiro/RecommenderSystemsNotebooks/blob/master/Month%202%20Part%20III%20-%20Notes%20on%20Similarity%20Metrics%20for%20CF.ipynb), we finalise the discussion over Collaborative Filtering by investigating a little more on how the similarity metrics work and try to find out some of its features such as:
#
# * Why pearson end up being better for User User CF and Cosine Similarity better for Item Item CF?
# * What are the strenghts and weakness when thinking on using one of the evaluated metrics?
# * Some filosophies on what they represent and how we can think about them geometrically
#
# Stay tuned :)


# -*- coding: utf-8 -*-
# Copyright (c) 2013 - 2019, Gorka Zamora-López <gorka@Zamora-Lopez.xyz>
#
# Released under the Apache License, Version 2.0 (the "License");
# you may not use this software except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0

"""
Graph Analysis Library
======================

A package for the analysis of graphs and complex networks in Python.
Compatible with Python 2.7 and 3.X

GAlib treats networks as adjacency matrices, represented as 2D NumPy arrays,
thus taking advantage of the faster operations over pure Pythen data types.
The core of the library consists of three modules:

metrics
    Basic graph metrics (degrees, clustering, graph distance, etc)
models
    Generation of synthetic networks and randomization.
tools
    Miscelaneous helper functions.

The rest of modules provide additional functionalities:

metrics_numba
    Uses package Numba to accelerate calculation of some metrics.
models_numba
    Uses package Numba to accelerate generation of some graph models.
extra.py
    Additional measures and functionalities related to network analysis.

USING pyGAlib
^^^^^^^^^^^^^^
The library is organised in the following modules:

- *metrics.py*: Common graph metrics (degrees, clustering, graph distance, etc)
- *models.py*: Generation of synthetic networks and randomization.
- *tools.py*: Miscelaneous helper functions.
- *metrics_numba.py*: Uses the Numba package to accelerate calculation of some
metrics.
- *models_numba.py*: Uses the Numba package to accelerate generation of some
graph models.
- *extra.py*: Additional measures and functionalities related to network analysis.

Getting started
***************
Since pyGAlib depends on NumPy, it is recommended to import NumPy first.
Although this is not necessary for loading pyGAlib, NumPy functionalities and
array manipulation will be often needed. Try importing pyGAlib: ::

    >>> import numpy as np
    >>> import galib

..note::
    Importing galib imports also all functions in module *metrics.py*
    into its namespace. The rest of modules are imported separately. Therefore,
    if the import is relative those functions can be called as, e.g.,

::

    >>> import galib
    >>> ...
    >>> deg = galib.Degree(net)
    >>> C, Cnodes = galib.Clustering(net)

See that we did not have to call ``galib.metrics.Degree(net)``. In the case of
an absolute import (using an asterisk ``*``) all functions in *metrics.py* are
imported to the base namespace:  ::

    >>> from galib import *
    >>> ...
    >>> deg = Degree(net)
    >>> C, Cnodes = Clustering(net)

Example
*******
Let's generate a random graph following the Erdos-Renyi model, G(N,p), with
*N = 100* nodes and link probability *p = 0.1*:  ::

    >>> import galib
    >>> import galib.models
    >>> N = 100; p = 0.1
    >>> net = galib.models.ErdosRenyiGraph(N, p, directed=False)

Here, *net* is the adjacency matrix of the random graph represented as a
2-dimensional NumPy array. Let's calculate some basic properties.  ::

    >>> galib.Density(net)
    0.09838383838383838

As expected, the density of an Erdos-Renyi random graph is close to the
*p = 0.1* value given. We now calculate the degree of every node:  ::

    >>> deg = galib.Degree(net)
    >>> deg
    array([10,  7, 10, 10, 11,  7,  5, 11, 13, 12, 14, 13,  8, 10,  9,  8,  7,
       10, 11,  9, 11, 11,  8, 10,  5,  9, 13, 10, 13, 12, 12, 11, 11,  7,
       13, 11,  7, 10, 10,  6, 12, 10,  6, 10,  7,  6,  9, 10,  9,  9,  7,
        9,  8, 13, 10,  9,  7,  7, 11,  8, 13,  6,  7, 12, 14,  6,  5, 11,
        5, 12, 14, 14, 13,  8,  7, 12,  4, 19,  9, 13,  7, 10, 15, 15,  4,
        9,  7, 12,  7,  8, 12,  4, 11, 12,  6, 13,  6, 12, 16, 12])

The degree is returned as a numpy array of rank 1, of integer type. The function
``Clustering()`` returns both the global clustering coefficient and the local
clustering of every node:  ::

    >>> C, Cnodes = galib.Clustering(net)
    >>> C
    0.096051227321238
    >>> Cnodes
    array([0.13333333, 0.04761905, 0.04444444, 0.08888889, 0.10909091,
       0.19047619, 0.3       , 0.12727273, 0.08974359, 0.06060606,
      	... ... ...
      	... ... ...
       0.04545455, 0.        , 0.10909091, 0.13636364, 0.        ,
       0.07692308, 0.13333333, 0.09090909, 0.08333333, 0.10606061])

We compute the pair-wise graph distance between all the nodes using the
Floyd-Warshall algorithm, which is returned as a matrix ``dij`` (numpy array of
rank 2):  ::

	>>> dij = galib.FloydWarshall(net)
	>>> avlen = (dij.sum() - dij.trace()) / (N*(N-1))
	>>> avlen
	2.248080808080808

.. note::
    For calculating the distance matrix of larger networks please use
    the version of the function located in the module *metrics_numba.py*. Here,
    the function ``FloydWarshall_Numba()`` works the same but makes use of the
    Numba library to significantly speed the calculation.

Most network generators and graph metrics in pyGAlib work with directed graphs
as well. Check for the optional parameter ``directed``. Following the example
above, we generate a directed Erdos-Renyi graph and calculate its input and
output degrees for every node:  ::

    >>> net = galib.models.ErdosRenyiGraph(N, p, directed=True)
    >>> galib.Density(net)
    0.10272727272727272
    >>> indeg, outdeg = galib.Degree(net, directed=True)
    >>> indeg
    array([17,  7,  9,  8, 11, 10,  9,  8, 13, 13,  5,  9, 13,  9, 10, 10, 13,
       10,  9,  9,  7, 11, 13, 10,  4, 15, 11, 11, 10,  6,  6,  8,  8,  8,
       11,  8,  4, 12,  8, 13, 13, 14, 12,  5,  6,  5, 16, 12,  5, 10,  9,
       13,  8,  9,  7,  8, 13, 14,  9, 18,  7, 11,  5,  4, 12,  8,  8, 10,
        7,  9, 15, 12, 14,  9, 15, 11, 13, 12, 15, 10, 11, 11, 15,  7, 10,
       13,  7, 14,  9, 16, 11, 11,  6, 18,  7,  4, 14, 12, 12, 10])
    >>> outdeg
    array([ 9, 10,  7,  9, 12,  9, 19,  9, 11, 16, 11, 12, 11, 15, 11,  6,  9,
        8, 11, 12,  9, 13,  9,  8, 11,  6,  7, 11, 11, 12, 10,  8, 11, 12,
       10, 12, 13,  8, 18, 11,  8, 13, 10,  8, 10, 10, 11,  8, 11, 11, 11,
       10, 11, 10,  9, 12,  6, 10,  7, 10, 10, 11, 15, 12, 11,  7, 10,  8,
        5, 11,  7, 11, 13,  8,  5,  6, 13, 11, 10, 13,  7,  6, 13, 11,  8,
        8, 10,  6, 10,  9, 12, 15, 11,  9, 15, 11,  7,  8, 11, 10])

Data I/O
********
Since GAlib is based on NumPy arrays, saving and reading of adjacency matrices,
as well as any other output of GAlib functions, can be performed using the usual
data I/O functionalities of NumPy. See for example the documentation for
functions: ``loadtxt()``, ``savetxt()``, ``load()``, ``save()`` and ``savez()``.
The *tools.py* module in pyGAlib provides also some data conversion
functionalities.

How to find further documentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
While working in an interactive session, after importing a module, the built-in
``help()`` function will show further details:  ::

    >>> help(modulename)

The help for galib (``help(galib)``) shows the general summary of the package
and a list of all the modules in the library. The help for each module,
e.g., ``help(galib.metrics)`` or ``help(galib.models)`` will display module
specific information and a list of all the functions in the module. For further
details regarding each function, type:  ::

    >>> help(galib.modulename.functionname)

For IPython and Jupyter notebook users the help command is replaced by a
question mark after the module's or function's name, e.g.:  ::

    >>> modulename?
    >>> functionname?

For questions, bug reports, etc, please write to <galib@Zamora-Lopez.xyz>, or
open an issue in GitHub.

License
-------

Copyright (c) 2013 - 2019, Gorka Zamora-López <gorka@Zamora-Lopez.xyz>

Released under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

or see the LICENSE.txt file.

.. note::
    Please, use the logos provided in the Branding/ folder whenever possible.

"""
from __future__ import absolute_import

from . import metrics
from .metrics import*

__author__ = "Gorka Zamora-Lopez"
__email__ = "galib@Zamora-Lopez.xyz"
__copyright__ = "Copyright 2013-2020"
__license__ = "Apache License version 2.0"
__version__ = "1.1.2"
__update__="15/06/2020"

#


# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.4'
#       jupytext_version: 1.2.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

#

#

#

#
Scientific Python

#
A super quick crash course

#

#

#

# Seismo-Live: http://seismo-live.org
#
# ##### Authors:
# * Lion Krischer ([@krischer](https://github.com/krischer))
#
# ---

# This notebook is a very quick introduction to Python and in particular its scientific ecosystem in case you have never seen it before. It furthermore grants a possibility to get to know the [IPython/Jupyter notebook](http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261). [See here for the official documentation](http://nbviewer.jupyter.org/github/jupyter/notebook/blob/master/docs/source/examples/Notebook/Notebook%20Basics.ipynb) of the Jupyter notebook - a ton more information can be found online.
#
#
# A lot of motivational writing on *Why Python?* is out there so we will not repeat it here and just condense it to a single sentence: **Python is a good and easy to learn, open-source, general purpose programming language that happens to be very good for many scientific tasks (due to its vast scientific ecosystem).**
#
#
# #### Quick Reference on How to Use This Notebook
#
#
#
#
# * `Shift + Enter`: Execute cell and jump to the next cell
# * `Ctrl/Cmd + Enter`: Execute cell and don't jump to the next cell
#
#
# #### Disclaimer
#
# The tutorials are employing Jupyter notebooks but these are only one way of using Python. Writing scripts to text files and executing them with the Python interpreter of course also works:
#
# ```bash
# $ python do_something.py
# ```
#
# Another alternative is interactive usage on the command line:
#
# ```bash
# $ ipython
# ```
#
# ## Notebook Setup
#
# First things first: In many notebooks you will find a cell similar to the following one. **Always execute it!** They do a couple of things:
# * Make plots appear in the browser (otherwise a window pops up)
# * Printing things works like this:
#
# ```python
# print("Hello")
# ```
#
# This essentially makes the notebooks work under Python 2 and Python 3.
#
# * Plots look quite a bit nicer (this is optional).
#

# +
# Plots now appear in the notebook.
# %matplotlib inline

import matplotlib.pyplot as plt
plt.style.use('ggplot')                            # Matplotlib style sheet - nicer plots!
plt.rcParams['figure.figsize'] = 12, 8             # Slightly bigger plots by default
# -

# ---
#
# ## Useful Links
#
# Here is collection of resources regarding the scientific Python ecosystem. They cover a number of different packages and topics; way more than we will manage today.
#
# If you have any question regarding some specific Python functionality you can consult the official [Python documenation](http://docs.python.org/).
#
# Furthermore a large number of Python tutorials, introductions, and books are available online. Here are some examples for those interested in learning more.
#
# * [Learn Python The Hard Way](http://learnpythonthehardway.org/book/)
# * [Dive Into Python](http://www.diveintopython.net/)
# * [The Official Python Tutorial](http://docs.python.org/2/tutorial/index.html)
# * [Think Python Book](http://www.greenteapress.com/thinkpython/thinkpython.html)
#
# Some people might be used to Matlab - this helps:
#
# * [NumPy for Matlab Users Introdution](http://wiki.scipy.org/NumPy_for_Matlab_Users)
# * [NumPy for Matlab Users Cheatsheet](http://mathesaurus.sourceforge.net/matlab-numpy.html)
#
#
# Additionally there is an abundance of resources introducing and teaching parts of the scientific Python ecosystem.
#
# * [NumPy Tutorial](http://wiki.scipy.org/Tentative_NumPy_Tutorial)
# * [Probabilistic Programming and Bayesian Methods for Hackers](http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/): Great ebook introducing Bayesian methods from an understanding-first point of view with the examples done in Python.
# * [Python Scientific Lecture Notes](http://scipy-lectures.github.io/): Introduces the basics of scientific Python with lots of examples.
# * [Python for Signal Processing](http://python-for-signal-processing.blogspot.de/): Free blog which is the basis of a proper book written on the subject.
# * [Another NumPy Tutorial](http://www.loria.fr/~rougier/teaching/numpy/numpy.html), [Matplotlib Tutorial](http://www.loria.fr/~rougier/teaching/matplotlib/matplotlib.html)
#
# You might eventually have a need to create some custom plots. The quickest way to success is usually to start from some example that is somewhat similar to what you want to achieve and just modify it. These websites are good starting points:
#
# * [Matplotlib Gallery](http://matplotlib.org/gallery.html)
# * [ObsPy Gallery](http://docs.obspy.org/gallery.html)
# * [Basemap Gallery](http://matplotlib.org/basemap/users/examples.html)
#
#
# ---

# ## Core Python Crash Course
#
# This course is fairly non-interactive and serves to get you up to speed with Python assuming you have practical programming experience with at least one other language. Nonetheless please change things and play around an your own - it is the only way to really learn it!
#
# The first part will introduce you to the core Python language. This tutorial uses Python 3 but almost all things can be transferred to Python 2. If possible choose Python 3 for your own work!
#
#
# ### 1. Numbers
#
# Python is dynamically typed and assigning something to a variable will give it that type.

# +
# Three basic types of numbers
a = 1             # Integers
b = 2.0           # Floating Point Numbers
c = 3.0 + 4j      # Complex Numbers, note the use of j for the complex part

# Arithmetics work as expected.
# Upcasting from int -> float -> complex
d = a + b         # (int + float = float)
print(d)

e = c ** 2        # c to the second power, performs a complex multiplication
print(e)
# -

# ### 2. Strings

# Just enclose something in single or double quotes and it will become a string. On Python 3 it defaults to unicode strings, e.g. non Latin alphabets and other symbols.

# +
# You can use single or double quotes to create strings.
location = "New York"

# Concatenate strings with plus.
where_am_i = 'I am in ' + location

# Print things with the print() function.
print(location, 1, 2)
print(where_am_i)

# Strings have a lot of attached methods for common manipulations.
print(location.lower())

# Access single items with square bracket. Negative indices are from the back.
print(location[0], location[-1])

# Strings can also be sliced.
print(location[4:])
# -

# #### Exercise
#
# Save your name in all lower-case letters to a variable, and print a capitalized version of it. Protip: [Google for "How to capitalize a string in python"](http://www.google.com/search?q=how+to+capitalize+a+string+in+python). This works for almost any programming problem - someone will have had the same issue before!

# + {"tags": ["exercise"]}

# + {"tags": ["solution"]}
name = "lion"
print(name.capitalize())
# -

# ### 3. Lists

# Python has two main collection types: List and dictionaries. The former is just an ordered collection of objects and is introduced here.

# +
# List use square brackets and are simple ordered collections of things.
everything = [a, b, c, 1, 2, 3, "hello"]

# Access elements with the same slicing/indexing notation as strings.
# Note that Python indices are zero based!
print(everything[0])
print(everything[:3])
print(everything[2:-2])

# Negative indices are counted from the back of the list.
print(everything[-3:])

# Append things with the append method.
everything.append("you")
print(everything)
# -

# ### 4. Dictionaries
#
# The other main collection type in Python are dictionaries. They are similiar to associative arrays or (hash) maps in other languages. Each entry is a key-value pair.

# +
# Dictionaries have named fields and no inherent order. As is
# the case with lists, they can contain anything.
information = {
    "name": "Hans",
    "surname": "Mustermann",
    "age": 78,
    "kids": [1, 2, 3]
}

# Acccess items by using the key in square brackets.
print(information["kids"])

# Add new things by just assigning to a key.
print(information)
information["music"] = "jazz"
print(information)

# Delete things by using the del operator
del information["age"]
print(information)

# -

# ### 5. Functions
#
# The key to conquer a big problem is to divide it into many smaller ones and tackle them one by one. This is usually achieved by using functions.

# +
# Functions are defined using the def keyword.
def do_stuff(a, b):
    return a * b

# And called with the arguments in round brackets.
print(do_stuff(2, 3))

# Python function also can have optional arguments.
def do_more_stuff(a, b, power=1):
    return (a * b) ** power

print(do_more_stuff(2, 3))
print(do_more_stuff(2, 3, power=3))

# For more complex function it is oftentimes a good idea to
#explicitly name the arguments. This is easier to read and less error-prone.
print(do_more_stuff(a=2, b=3, power=3))
# -

# ### 6. Imports
#
# To use functions and objects not part of the default namespace, you have import them. You will have to do this a lot so it is necessary to learn how to do it.

# +
# Import anything, and use it with the dot accessor.
import math

a = math.cos(4 * math.pi)

# You can also selectively import things.
from math import pi

b = 3 * pi

# And even rename them if you don't like their name.
from math import cos as cosine
c = cosine(b)
# -

# How to know what is available?
#
# 1. Read the [documentation](https://docs.python.org/3/library/math.html)
# 2. Interactively query the module

print(dir(math))

# Typing the dot and the TAB will kick off tab-completion.

# math.

# In the IPython framework you can also use a question mark to view the documentation of modules and functions.

# math.cos?

# ### 7. Control Flow
#
# Loops and conditionals are needed for any non-trivial task. Please note that **whitespace matters in Python**. Everything that is indented at the same level is part of the same block. By far the most common loops in Python are for-each loops as shown in the following. While loops also exist but are rarely used.

# +
temp = ["a", "b", "c"]

# The typical Python loop is a for-each loop, e.g.
for item in temp:
    # Everything with the same indentation is part of the loop.
    new_item = item + " " + item
    print(new_item)

print("No more part of the loop.")
# -

# Useful to know is the range() function.
for i in range(5):
    print(i)

# The second crucial control flow structure are if/else conditional and they work the same as in any other language.

# +
# If/else works as expected.
age = 77

if age >= 0 and age < 10:
    print("Younger than ten.")
elif age >= 10:
    print("Older than ten.")
else:
    print("Wait what?")

# +
# List comprehensions are a nice way to write compact loops.
# Make sure you understand this as it is very common in Python.

a = list(range(10))
print(a)
b = [i for i in a if not i % 2]
print(b)

# Equivalant loop for b.
b = []
for i in a:
    if not i % 2:
        b.append(i)
print(b)

# -

# ### 8. Error Messages
#
# You will eventually run into some error messages. Learn to read them! The last line is often the one that matters - reading upwards traces the error back in time and shows what calls led to it. If stuck: just google the error message!

# +
def do_something(a, b):
    print(a + b + something_else)

# do_something(1, 2)
# -

# ## The Scientific Python Ecosystem
#
# The [SciPy Stack](https://www.scipy.org/stackspec.html) forms the basis for essentially all applications of scientific Python. Here we will quickly introduce the three core libraries:
#
# * `NumPy`
# * `SciPy`
# * `Matplotlib`
#
# The SciPy stack furthermore contains `pandas` (library for data analysis on tabular and time series data) and `sympy` (package for symbolic math), both very powerful packages, but we will omit them in this tutorial.

# ### 9. NumPy
#
# Large parts of the scientific Python ecosystem use NumPy, an array computation package offering N-dimensional, typed arrays and useful functions for linear algebra, Fourier transforms, random numbers, and other basic scientific tasks.

# +
import numpy as np

# Create a large array with with 1 million samples.
x = np.linspace(start=0, stop=100, num=1E6, dtype=np.float64)

# Most operations work per-element.
y = x ** 2

# Uses C and Fortran under the hood for speed.
print(y.sum())

# FFT and inverse
x = np.random.random(100)
large_X = np.fft.fft(x)
x = np.fft.ifft(large_X)
# -

# ### 10. SciPy
#
# `SciPy`, in contrast to `NumPy` which only offers basic numerical routines, contains a lot of additional functionality needed for scientific work. Examples are solvers for basic differential equations, numeric integration and optimization, spare matrices, interpolation routines, signal processing methods, and a lot of other things.

# +
from scipy.interpolate import interp1d

x = np.linspace(0, 10, num=11, endpoint=True)
y = np.cos(-x ** 2 / 9.0)

# Cubic spline interpolation to new points.
f2 = interp1d(x, y, kind='cubic')(np.linspace(0, 10, num=101, endpoint=True))
# -

# ### 11. Matplotlib
#
# Plotting is done using `Matplotlib`, a package for greating high-quality static plots. It has an interface that mimics Matlab which many people are familiar with.

# +
import matplotlib.pyplot as plt

plt.plot(np.sin(np.linspace(0, 2 * np.pi, 2000)), color="green",
         label="Some Curve")
plt.legend()
plt.ylim(-1.1, 1.1)
plt.show()
# -

# ## Exercises
#
# #### Functions, NumPy, and Matplotlib
#
# A. Write a function that takes a NumPy array `x` and `a`, `b`, and `c` and returns
#
# $$
# f(x) = a x^2 + b x + c
# $$
#
# B. Plot the result of that function with matplotlib.

# + {"tags": ["exercise"]}

# + {"tags": ["solution"]}
import matplotlib.pyplot as plt
import numpy as np

def simple_poly(x, a, b, c):
    return a * x ** 2 + b * x + c

plt.plot(simple_poly(np.linspace(-5, 5), 10, 2, 2))
plt.show()
# -

# #### 99 Bottles of Beer
#
# *(stolen from http://www.ling.gu.se/~lager/python_exercises.html)*
#
#
# "99 Bottles of Beer" is a traditional song in the United States and Canada. It is popular to sing on long trips, as it has a very repetitive format which is easy to memorize, and can take a long time to sing. The song's simple lyrics are as follows:
#
# ```
# 99 bottles of beer on the wall, 99 bottles of beer.
# Take one down, pass it around, 98 bottles of beer on the wall.
# ```
#
# The same verse is repeated, each time with one fewer bottle. The song is completed when the singer or singers reach zero.
#
# Your task here is write a Python program capable of generating all the verses of the song.
#

# + {"tags": ["exercise"]}

# + {"tags": ["solution"]}
print("99 bottles of beer on the wall, 99 bottles of beer.")
for i in range(98, -1, -1):
    print("Take one down, pass it around, %i bottles of beer on the wall." % i)
# -

# #### Ceasar Cipher
#
# *(stolen from http://www.ling.gu.se/~lager/python_exercises.html)*
#
# In cryptography, a Caesar cipher is a very simple encryption techniques in which each letter in the plain text is replaced by a letter some fixed number of positions down the alphabet. For example, with a shift of 3, A would be replaced by D, B would become E, and so on. The method is named after Julius Caesar, who used it to communicate with his generals. ROT-13 ("rotate by 13 places") is a widely used example of a Caesar cipher where the shift is 13. In Python, the key for ROT-13 may be represented by means of the following dictionary:
#
# ```python
# key = {'a':'n', 'b':'o', 'c':'p', 'd':'q', 'e':'r', 'f':'s', 'g':'t', 'h':'u',
#        'i':'v', 'j':'w', 'k':'x', 'l':'y', 'm':'z', 'n':'a', 'o':'b', 'p':'c',
#        'q':'d', 'r':'e', 's':'f', 't':'g', 'u':'h', 'v':'i', 'w':'j', 'x':'k',
#        'y':'l', 'z':'m', 'A':'N', 'B':'O', 'C':'P', 'D':'Q', 'E':'R', 'F':'S',
#        'G':'T', 'H':'U', 'I':'V', 'J':'W', 'K':'X', 'L':'Y', 'M':'Z', 'N':'A',
#        'O':'B', 'P':'C', 'Q':'D', 'R':'E', 'S':'F', 'T':'G', 'U':'H', 'V':'I',
#        'W':'J', 'X':'K', 'Y':'L', 'Z':'M'}
# ```
#
# Your task in this exercise is to implement an decoder of ROT-13. Once you're done, you will be able to read the following secret message:
#
# ```
# Pnrfne pvcure? V zhpu cersre Pnrfne fnynq!
# ```
#
# **BONUS:** Write an encoder!

# + {"tags": ["exercise"]}

# + {"tags": ["solution"]}
sentence = "Pnrfne pvcure? V zhpu cersre Pnrfne fnynq!"

key = {'a':'n', 'b':'o', 'c':'p', 'd':'q', 'e':'r', 'f':'s', 'g':'t', 'h':'u',
       'i':'v', 'j':'w', 'k':'x', 'l':'y', 'm':'z', 'n':'a', 'o':'b', 'p':'c',
       'q':'d', 'r':'e', 's':'f', 't':'g', 'u':'h', 'v':'i', 'w':'j', 'x':'k',
       'y':'l', 'z':'m', 'A':'N', 'B':'O', 'C':'P', 'D':'Q', 'E':'R', 'F':'S',
       'G':'T', 'H':'U', 'I':'V', 'J':'W', 'K':'X', 'L':'Y', 'M':'Z', 'N':'A',
       'O':'B', 'P':'C', 'Q':'D', 'R':'E', 'S':'F', 'T':'G', 'U':'H', 'V':'I',
       'W':'J', 'X':'K', 'Y':'L', 'Z':'M'}

result = ""
for letter in sentence:
    if letter not in key:
        result += letter
    else:
        result += key[letter]
print(result)


#!/usr/bin/env python
# coding: utf-8

# <!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
# doconce format html hw2.do.txt --no_mako -->
# <!-- dom:TITLE: PHY321: Classical Mechanics 1 -->

# # PHY321: Classical Mechanics 1
# **Homework 2, due January 28 (Midnight)**
#
# Date: **Jan 18, 2022**

# ### Practicalities about  homeworks and projects
#
# 1. You can work in groups (optimal groups are often 2-3 people) or by yourself. If you work as a group you can hand in one answer only if you wish. **Remember to write your name(s)**!
#
# 2. Homeworks are available 10 days  before the deadline.
#
# 3. How do I(we)  hand in?  You can hand in the paper and pencil exercises as a  scanned  document. For this homework this applies to exercises 1-5. The scanned document should be uploaded to D2L. Alternatively, you can hand in everyhting (if you are ok with typing mathematical formulae using say Latex) as a jupyter notebook at D2L. The numerical exercise(s) (exercise 6 here) should always be handed in as a jupyter notebook by the deadline at D2L.

# ### Exercise 1 (10 pt), Forces, discussion questions, test your intuition
#
# * 1a (2pt) Single force. Can an object affected only by a single force have zero acceleration?
#
# * 1b (2pt) Zero velocity. If you throw a ball vertically it has zero velocity at its maximum point. Does it also have zero acceleration at this point?
#
# * 1c (3pt) Acceleration of gravity. You measure the acceleration of gravity in an elevator moving at a velocity of 9.8m/s downwards. What will you measure?
#
# * 1d (3pt) Air resistance. You throw a ball straight up and measure the velocity as it passes you on its way down. Will the velocity be larger, the same, or smaller if you did the same experiment in vacuum?

# ### Exercise 2 (10 pt), setting up forces, Newton's second law
#
# Useful material here to read is
# 1. Taylor chapters 1.3 and 1.4 and
#
# 2. Malthe-Sørenssen chapters 5.1, 5.2 and 5.3
#
# A person jumps from an airplane, falling freely for several seconds before the person pulls the cord of her parachute and the parachute unfolds.
# * 2a (3pt)  Identify the forces acting on the parachuter and draw a free-body diagram of the parachuter before the person has pulled the cord.
#
# * 2b (3pt)  Identify the forces acting on the parachuter and draw a free-body diagram of the parachuter after the person has pulled the cord.
#
# * 2c (4pt)  Sketch the net force acting on the parachuter as a function of time, F(t).

# ### Exercise 3 (10 pt), Space shuttle with air resistance
#
# Useful material here to read is
# 1. Malthe-Sørenssen chapters 5.1, 5.2 and 5.3
#
# During lift-off of the space shuttle the engines provide a force of $35\times 10^{6}$ N. The mass of the shuttle is approximately
# $2\times 10^6$ kg.
# * 3a (3pt) Draw a free-body diagram of the space shuttle immediately after lift-off.
#
# * 3b (3pt)  Find an expression for the acceleration of the space shuttle immediately after lift-off.
#
# Let us assume that the force from the engines is constant, and that the mass of the
# space shuttle does not change significantly over the first 20 s.
# * 3c (4pt) Find the velocity and position of the space shuttle after 20 s if you ignore air resistance.

# ### Exercise 4 (15 pt), now hitting a golf ball
#
# Useful material here to read is
# 1. Taylor chapters 1.3-1.6 and
#
# 2. Malthe-Sørenssen chapter 6.3-6.4 and 7.1-7.3
#
# **Taylor exercise 1.35**. The formulae you obtain here will be useful for the numerical exercises below (see exercise 6 below).

# ### Exercise 5 (15 pt), hitting a puck instead
#
# Taylor exercise 1.38.

# ### Exercise 6 (40pt), Numerical elements, moving to more than one dimension
#
# **This exercise should be handed in as a jupyter-notebook** at D2L. Remember to write your name(s).
#
# Last week we:
# 1. Analytically mapped 1D motion over some time
#
# 2. Gained practice with functions
#
# 3. Reviewed vectors and matrices in Python
#
# This week we will:
# 1. Practice using Python syntax and variable manipulation
#
# 2. Utilize analytical solutions to create more refined functions
#
# 3. Work in two, three or even higher dimensions
#
# This material will then serve as background for the numerical part of homework 3. The first part is a simple warm-up, with hints and suggestions you can use for the code to write below.

# In[1]:

get_ipython().run_line_magic('matplotlib', 'inline')

# As usual, here are some useful packages we will be using. Feel free to use more and experiment as you wish.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
get_ipython().run_line_magic('matplotlib', 'inline')

# In class (the falling baseball example) we used an  analytical expression for the height of a falling ball.
# In the first homework we used instead the position from experiment (Usain Bolt's 100m record run) and stored this
# information with one-dimensional arrays in Python.
#
# Let us get some practice with this. The cell below creates two arrays,
# one containing the times to be analyzed and the other containing the $x$
# and $y$ components of the position vector at each point in time.  This is a two-dimensional object. The
# second array is initially empty. Then we define  the initial
# position to be $x=2$ and $y=1$. Take a look at the code and comments
# to get an understanding of what is happening. Feel free to play around with it.

# In[2]:

tf = 4 #length of value to be analyzed
dt = .001 # step sizes
t = np.arange(0.0,tf,dt) # Creates an evenly spaced time array going from 0 to 3.999, with step sizes .001
p = np.zeros((len(t), 2)) # Creates an empty array of [x,y] arrays (our vectors). Array size is same as the one for time.
p[0] = [2.0,1.0] # This sets the inital position to be x = 2 and y = 1

# Below we are printing specific values of our array to see what is being
# stored where. The first number in the array $r[]$ represents which array
# iteration we are looking at, while the number after the  represents
# which listed number in the array iteration we are getting back.

# In[3]:

print(p[0]) # Prints the first array
print(p[0,:]) # Same as above, these commands are interchangeable

# In[4]:

print(p[3999]) # Prints the 4000th array

# In[5]:

print(p[0,0]) # Prints the first value of the first array

# In[6]:

print(p[0,1]) # Prints the second value of first array
print(p[:,0]) # Prints the first value of all the arrays

# Then try running this cell. Notice how it gives an error since we did not implement a third dimension into our arrays

# In[7]:

print(p[:,2])

# In the cell below we want to manipulate the arrays.
# In this example we make each vector's $x$ component valued the same as their respective vector's position in the iteration and the $y$ value will be twice that value, except for  the first vector, which we have already set.
# That is we have $p[0] = [2,1], p[1] = [1,2], p[2] = [2,4], p[3] = [3,6], ...$
#
# Here we set up an array for $x$ and $y$ values.

# In[8]:

for i in range(1,3999):
    p[i] = [i,2*i]
# Checker cell to make sure your code is performing correctly
c = 0
for i in range(0,3999):
    if i == 0:
        if p[i,0] != 2.0:
            c += 1
        if p[i,1] != 1.0:
            c += 1
    else:
        if p[i,0] != 1.0*i:
            c += 1
        if p[i,1] != 2.0*i:
            c += 1

if c == 0:
    print("Success!")
else:
    print("There is an error in your code")

# You could also think of an alternative way of storing the above information. Feel free to explore how to store
# multidimensional objects.
#
# Last week we studied Usain Bolt's 100m run and in class we studied a falling baseball. We made basic plots of the baseball
# moving in one dimension. This week we will be working with a three-dimensional variant. This will be useful for our next homeworks and numerical projects.
#
# Assume we have a soccer ball moving in three dimensions with the following trajectory:
#
# 1. $x(t) = 10t\cos{45^{\circ}} $
#
# 2. $y(t) = 10t\sin{45^{\circ}} $
#
# 3. $z(t) = 10t - \dfrac{9.81}{2}t^2$
#
# Now let us create a three-dimensional (3D) plot using these equations. In the cell below
# we write the equations into their respective labels. We fix a final time in the code below.
#
# Important Concept: Numpy comes with many mathematical packages, some
# of them being the trigonometric functions sine, cosine, tangent. We
# are going to utilize these this week. Additionally, these functions
# work with radians, so we will also be using a function from Numpy that
# converts degrees to radians.

# In[9]:

tf = 2.04  # The final time to be evaluated
dt = 0.1  # The time step size
t = np.arange(0,tf,dt) # The time array
theta_deg = 45 # Degrees
theta_rad = np.radians(theta_deg) # Converts degrees to their radian counterparts
x = 10*t*np.cos(theta_rad) # Equation for our x component, utilizing np.cos() and our calculated radians
y = 10*t*np.sin(theta_rad) # Put the y equation here
z = 10*t-9.81/2*t**2# Put the z equation here

# Then we plot it

# In[10]:

## Once you have entered the proper equations in the cell above, run this cell to plot in 3D
fig = plt.axes(projection='3d')
fig.set_xlabel('x')
fig.set_ylabel('y')
fig.set_zlabel('z')
fig.scatter(x,y,z)

# * 6a (8pt) How would you express $x(t)$, $y(t)$, $z(t)$ for this problem as a single vector, $\boldsymbol{r}(t)$?
#
# Then run the code and plot using the array $r$

# In[11]:

## Run this code to plot using our r array
fig = plt.axes(projection='3d')
fig.set_xlabel('x')
fig.set_ylabel('y')
fig.set_zlabel('z')
fig.scatter(r[0],r[1],r[2])

# * 6b (8pt) What do you think the benefits and/or disadvantages are from expressing our three equations as a single array/vector? This can be both from a computational and physics stand point. Use the **Numpy** package to also print the maximum $x$, $y$ and $z$ components from $\boldsymbol{r}$.
#
# Complete Exercise 4 above (Taylor exercise 1.35) before moving further. (Recall that the golf ball was hit due east at an angle $\theta$ with respect to the horizontal, and the coordinate directions are $x$ measured east, $y$ north, and $z$ vertically up.)
#
# * 6c (8pt) What is the analytical solution for our theoretical golf ball's position $\boldsymbol{r}(t)$ over time from Exercise 4?  Also what is the formula for the time $t_f$ when the golf ball hits the ground? Use this to develop a program with a function called for example Golfball that utilizes our analytical solutions. This program should take in an initial velocity and the angle $\theta$ that the golfball was hit with in degrees. It should also produce  a 3D graph of the motion. You need also to find the maximum values for $x$, $y$ and $z$.
#
# * 6d (8pt) Given initial values of $v_i = 90 m/s$, $\theta = 30^{\circ}$, what would our maximum x, y and z components be?
#
# * 6e (8pt) Given initial values of $v_i = 45 m/s$, $\theta = 45^{\circ}$, what would our maximum x, y and z components be?


r"""
Elements of posets, lattices, semilattices, etc.
"""
#*****************************************************************************
#       Copyright (C) 2008 Peter Jipsen <jipsen@chapman.edu>,
#                          Franco Saliola <saliola@gmail.com>
#
#  Distributed under the terms of the GNU General Public License (GPL)
#
#    This code is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
#    General Public License for more details.
#
#  The full text of the GPL is available at:
#
#                  http://www.gnu.org/licenses/
#*****************************************************************************
from sage.structure.element import Element
from sage.structure.element import have_same_parent

class PosetElement(Element):

    def __init__(self, poset, element, vertex):
        r"""
        Establish the parent-child relationship between ``poset``
        and ``element``, where ``element`` is associated to the
        vertex ``vertex`` of the Hasse diagram of the poset.

        INPUT:

        - ``poset`` -- a poset object

        - ``element`` -- any object

        - ``vertex`` -- a vertex of the Hasse diagram of the poset

        TESTS::

            sage: from sage.combinat.posets.elements import PosetElement
            sage: P = Poset([[1,2],[4],[3],[4],[]], facade = False)
            sage: e = P(0)
            sage: e.parent() is P
            True
            sage: TestSuite(e).run()
        """
        Element.__init__(self, poset)
        if isinstance(element, self.parent().element_class):
            self.element = element.element
        else:
            self.element = element
        self.vertex = vertex

    def __hash__(self):
        r"""
        TESTS::

            sage: P = Poset([[1,2],[4],[3],[4],[]], facade = False)
            sage: e = P(0)
            sage: hash(e)
            0
        """
        return hash(self.element)

    def _repr_(self):
        """
        TESTS::

            sage: Poset([[1,2],[4],[3],[4],[]], facade = False)(0)._repr_()
            '0'
        """
        return "%s" % str(self.element)

    def _latex_(self):
        r"""
        Return the latex code of the poset element.

        EXAMPLES::

            sage: m = matrix(2,[1,2,3,4])
            sage: m.set_immutable()
            sage: P = Poset(([m],[]), facade = False)
            sage: [e] = P
            sage: type(e)
            <class 'sage.combinat.posets.posets.FinitePoset_with_category.element_class'>
            sage: latex(e)                 #indirect doctest
            \left(\begin{array}{rr}
            1 & 2 \\
            3 & 4
            \end{array}\right)
        """
        from sage.misc.latex import latex
        return latex(self.element)

    def __eq__(self, other):
        """
        TESTS::

            sage: P = Poset([["a","b"],["d"],["c"],["d"],[]], facade = False)
            sage: Q = Poset([["a","b"],["d"],["c"],[],[]], facade = False)
            sage: P(0).__eq__(P(4))
            False
            sage: from sage.combinat.posets.elements import PosetElement
            sage: PosetElement(P,0,"c") == PosetElement(P,0,"c")
            True
            sage: PosetElement(P,0,"c") == PosetElement(Q,0,"c")
            False
            sage: PosetElement(P,0,"b") == PosetElement(P,0,"c")
            False

        .. warning:: as an optimization, this only compares the parent
           and vertex, using the invariant that, in a proper poset
           element, ``self.element == other.element`` if and only
           ``self.vertex == other.vertex``::

            sage: PosetElement(P,1,"c") == PosetElement(P,0,"c")
            True

        Test that :trac:`12351` is fixed::

            sage: P(0) == int(0)
            False
        """
        # This should instead exploit unique representation, using
        # self is other, or best inherit __eq__ from there. But there
        # are issues around pickling and rich comparison functions.
        return have_same_parent(self, other) \
            and self.vertex == other.vertex

    def __ne__(self, other):
        r"""
        TESTS::

            sage: P = Poset([[1,2],[4],[3],[4],[]])
            sage: P = Poset([["a","b"],["d"],["c"],["d"],[]])
            sage: P(0).__ne__(P(4))
            True
            sage: from sage.combinat.posets.elements import PosetElement
            sage: PosetElement(P,0,"c") != PosetElement(P,0,"c")
            False
            sage: PosetElement(P,0,"b") != PosetElement(P,0,"c")
            True

        For this one, see comment in :meth:`__eq__`::

            sage: PosetElement(P,1,"c") != PosetElement(P,0,"c")
            False
        """
        return not self == other

    def _cmp(self, other):
        """
        TESTS::

            sage: P = Poset([[1,2],[4],[3],[4],[]], facade = False)
            sage: P(0)._cmp(P(4))
            -1
            sage: P(4)._cmp(P(0))
            1
            sage: P(0)._cmp(P(0))
            0
            sage: P(1)._cmp(P(2))

        """
        return self.parent().compare_elements(self, other)

    def __lt__(self, other):
        """
        TESTS

        ::

            sage: dag = DiGraph({0:[2,3], 1:[3,4], 2:[5], 3:[5], 4:[5]})
            sage: P = Poset(dag, facade = False)
            sage: P(0) < P(1)
            False
            sage: P(4) < P(1)
            False
            sage: P(0) < P(0)
            False
        """
        return self._cmp(other) == -1 or False

    def __le__(self, other):
        """
        TESTS

        ::

            sage: dag = DiGraph({0:[2,3], 1:[3,4], 2:[5], 3:[5], 4:[5]})
            sage: P = Poset(dag, facade = False)
            sage: P(1) <= P(0)
            False
            sage: P(0) <= P(1)
            False
            sage: P(0) <= P(3)
            True
            sage: P(0) <= P(0)
            True
        """
        return self == other or self._cmp(other) == -1 or False

    def __gt__(self, other):
        """
        TESTS

        ::

            sage: dag = DiGraph({0:[2,3], 1:[3,4], 2:[5], 3:[5], 4:[5]})
            sage: P = Poset(dag)
            sage: P(0).__gt__(P(5))
            False
            sage: P(5).__gt__(P(0))
            True
            sage: P(0).__gt__(P(0))
            False
        """
        return self._cmp(other) == 1 or False

    def __ge__(self, other):
        """
        TESTS

        ::

            sage: dag = DiGraph({0:[2,3], 1:[3,4], 2:[5], 3:[5], 4:[5]})
            sage: P = Poset(dag)
            sage: P(0).__ge__(P(5))
            False
            sage: P(5).__ge__(P(0))
            True
            sage: P(0).__ge__(P(0))
            True
        """
        return self == other or self._cmp(other) == 1 or False

class MeetSemilatticeElement(PosetElement):
    def __mul__(self, other):
        r"""
        Return the meet of ``self`` and ``other`` in the lattice.

        EXAMPLES::

            sage: D = posets.DiamondPoset(5,facade=False)
            sage: D(1) * D(2)
            0
            sage: D(1) * D(1)
            1
            sage: D(1) * D(0)
            0
            sage: D(1) * D(4)
            1
        """
        return self.parent().meet(self, other)

class JoinSemilatticeElement(PosetElement):
    def __add__(self, other):
        r"""
        Return the join of ``self`` and ``other`` in the lattice.

        EXAMPLES::

            sage: D = posets.DiamondPoset(5,facade=False)
            sage: D(1) + D(2)
            4
            sage: D(1) + D(1)
            1
            sage: D(1) + D(4)
            4
            sage: D(1) + D(0)
            1
        """
        return self.parent().join(self, other)

class LatticePosetElement(MeetSemilatticeElement, JoinSemilatticeElement):
    pass


"""
        Assignment-2
        Name - Gajraj Singh Chouhan
        Roll No - B19130
        Branch - DSE
        Mobile No - +91-9351159849
"""

import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from functions import properties, plot_rmse

# imports

data = pd.read_csv('pima_indians_diabetes_miss.csv')
original_data = pd.read_csv('pima_indians_diabetes_original.csv')

# Question 1

missing_val = data.isna().sum()
# We can count the missing values by using isna(), adding all the cells in a column which would a NaN value (isna() would give each cell 1 or 0 whether it is or is-not a NaN value).
missing_val.plot.bar(rot=0)
# plotting the missing values.
# adding the labels and title.
plt.title("Missing Values in the Attributes")
plt.xlabel("Attributes")
plt.ylabel("Missing Values")
plt.savefig("Q1-Barplot.png")
plt.close()

# Question 2a
print('Question 2a', end='\n\n')

col = data.shape[1] # number of columns
to_drop = 3 # inverse of 1/3rd rows to drop
filtered = data.isna().sum(axis=1)
# finding how many NaN values are in each row by summing the boolean values for each row. (using axis=1 will do sum of each row)
# Now we have a dataframe consisting of number of NaN values in that row

filtered = filtered[(filtered * to_drop) >= col]
# We can filter the rows to drop, by using the condition that number of NaN values must be greater or equal to 1/3rd of no of columns.
# This will give us True or False Boolean Value if that row is to be dropped or not.

print(f"Total number of tuples deleted = {filtered.size}")

filtered = filtered.index
data = data.drop(filtered, axis=0)
# Now we can drop the rows in the original DataFrame using their indexes we got in the previous step.

print(f"Indices of rows dropped (0-based): ")
print(filtered.to_list());print()

# Question 2b
print('Question 2b', end='\n\n')

filtered = data[data['class'].isna()]
# rows whose 'class' attribute is NaN using conditional filtering
filtered = filtered.index
# getting index of rows which we will delete

print(f"Total number of tuples deleted = {filtered.size}")

data = data.drop(filtered, axis=0) # remove the rows

print(f"Indices of rows dropped (0-based): ")
print(filtered.to_list());print()

# Question 3
print('Question 3', end='\n\n')

count_of_NaN = data.isna().sum()
# again we can use isna() for number of null values.

print("Number of missing values in each attributes: ")
print(count_of_NaN.to_frame().T)
# printing number of missing value per column
# ignore the first 0, it just the first row number as I am displaying the Series in horizontal way.
print(f"Total number of missing values in the dataFrame : {count_of_NaN.sum()}");print()

# Question 4a
print('Question 4a', end='\n\n')
column_mean = data.mean()
data_withmean = data.fillna(column_mean)
# fillna() will fill the NaN values for each attribute by the mean of that column

print('RMSE of Original and Mean Filled data')
plot_rmse(data_withmean, original_data, np.where(pd.isnull(data)), count_of_NaN);print()

# plotting the rmse after replacing the nan values.
plt.title("RMSE of original and mean filled data")
plt.savefig("Q4-MeanFilled.png")
plt.close()

# Question 4b
print('Question 4b', end='\n\n')

data_with_interpolate = data.interpolate()
# by default each column will be filled with data interpolated with linear method.

print('RMSE of Original and interpolated data')
plot_rmse(data_with_interpolate, original_data, np.where(pd.isnull(data)), count_of_NaN);print()

plt.title("RMSE of original and interpolated data")
plt.savefig("Q4-Interpolated.png")
plt.close()

# Question 4
print('Question 4 (comparing the filled data and original) : ', end='\n\n')

# Comparing the properties (mean, median, mode, standard-dev...) of the
# original data and after we filled the values in 4a and 4b.
print('Original Data : ')
print(properties(original_data).to_string())
print('\nAfter filling data with mean : ')
print(properties(data_withmean).to_string())
print('\nAfter interpolating the data : ')
print(properties(data_with_interpolate).to_string());print()

# Question 5
print('Question 5 (Outliers)', end='\n\n')

data_with_interpolate = data.interpolate()
columns = ["Age", "BMI"]
quartiles = data_with_interpolate.quantile([0.25, 0.5, 0.75]).loc[:, columns]
# quantile() will give the required quartiles (0.25, 0.5, 0.75) for required columns
data.loc[:, columns].plot.box()
plt.title("Box-Plot of data before replacing outliers with median")
plt.savefig("Q5-before-BoxPlot.png")
plt.close()

for col in columns:
    q1, median, q3 = quartiles.loc[:, col].to_list()
    # quartiles for one column
    print(f"Column : {col}")
    check_outlier = lambda num : (q1 - (1.5 * (q3 - q1))) < num < (q3 + (1.5 * (q3 - q1)))
    # this will return boolean value whether a number ("num") would be outlier according to the quartiles.
    for ind, x in enumerate(data_with_interpolate.loc[:, col]):
        # iterating over the column and checking if it's a outlier.
        if not check_outlier(x):
            # if it is a outlier
            data.loc[data_with_interpolate.index[ind], col] = median
            # we can change value of a cell to the median of that column from the column name and row index.
            # data_with_interpolate.index[ind] - row index
            print(x, end=' ')
    print()

data.loc[:, columns].plot.box()
plt.title("Box-Plot of data after replacing outliers with median")
plt.savefig("Q5-after-BoxPlot.png")
plt.close()

print('properties after replacing with median : ')
print(properties(data.loc[:, columns]));print()


"""
Module file containing functions that allow to reproduce FIG. 1 of the article

    Interaction of "Solitons" in a Collisionless Plasma and the Recurrence of Initial States,
    N. J. Zabusky and M. D. Kruskal,
    Phys. Rev. Lett. 15, 240 (1965)

This module was prepared as a part of the scientific short course

  A brief guide to publication-ready scientific figures using Python's matplotlib

held during the 2020 seminar week of the Ultrafast Laser Laboratory
at Institute of Quantum Optics at Leibniz University Hannover.

Author: O. Melchert
Date: 2020-09-08
"""
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib.gridspec import GridSpec

__author__ = 'Oliver Melchert'
__date__ = '2020-09-09'

def fetch_data(path):
    """fetch data

    Reads in data from file in numpy npz-format

    Args:
      path (str): path to npz-file

    Returns: (x, t, uxt)
      x (1D array): x samples
      t (1D array): t samples
      uxt (2D array): wave profile u(x,t)
    """
    dat = np.load(path)
    return dat['x'], dat['t'], dat['uxt']

def set_style():
    """set figure style

    Function that customizes the default style to be conform with the Physical
    Review style and notation guide [1]. For instructions on how to set the
    default style using style sheets see [2].

    Notes:
    - main font size is chosen as 8pt, matching the fontsize of figure captions
    - fontsize of legends and auxiliary text labels are set to 6pt, exceeding
      the minimally required pointsize of 4.25 pt. (1.5 mm)
    - default rc (rc = "run commands", i.e. startup information) settings are
      changed dynamically
    - the custom font-scheme 'type2' depends on your latex installation and
      is not guaranteed to run on your specific system

    Refs:
      [1] https://journals.aps.org/prl/authors
      [2] https://matplotlib.org/3.3.1/tutorials/introductory/customizing.html
    """

    fig_width_1col = 3.4        # figure width in inch
    fig_width_2col = 7.0        # figure width in inch
    fig_aspect_ratio = 0.66     # width to height aspect ratio
    font_size = 8               # font size in pt
    font_size_small = 6         # font size in pt
    font_scheme = 'type2'       # options:
                                #   None    - default matplotlib fonts
                                #   'type1' - text: Helvetica, math: Computer modern
                                #   'type2' - text: Helvetica, math: Helvetica

    mpl.rcParams['figure.figsize'] = fig_width_1col, fig_aspect_ratio*fig_width_1col
    mpl.rcParams['axes.labelsize'] = font_size
    mpl.rcParams['font.size'] = font_size
    mpl.rcParams['legend.fontsize'] = font_size_small
    mpl.rcParams['xtick.labelsize'] = font_size
    mpl.rcParams['ytick.labelsize'] = font_size
    mpl.rcParams['xtick.direction'] = 'out'
    mpl.rcParams['ytick.direction'] = 'out'
    mpl.rcParams['lines.linewidth'] = 1.0
    mpl.rcParams['axes.linewidth'] =  0.5

    if font_scheme == 'type1':
        mpl.rcParams['text.usetex'] = True
        mpl.rcParams['font.family'] = 'sans-serif'
        mpl.rcParams['font.sans-serif'] = 'Helvetica'
        mpl.rcParams['mathtext.fontset'] = 'cm'

    if font_scheme == 'type2':
        mpl.rcParams['text.usetex'] = True
        mpl.rcParams['text.latex.preamble'] = [
           r'\usepackage{siunitx}',
           r'\sisetup{detect-all}',
           r'\usepackage{helvet}',
           r'\usepackage{sansmath}',
           r'\sansmath'
        ]

def set_circle(ax, x0, y0, label):
    """set circle

    Function that generates a circle with text-label at its center.
    For more options on scatter plots, see [1], for more options on
    setting text, see [2]

    Refs:
      [1] https://matplotlib.org/3.3.1/api/_as_gen/matplotlib.pyplot.scatter.html
      [2] https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.text.html

    Args:
      ax (object): figure part for which the labeled circle is intended
      x0 (float): x-position for center of circle
      y0 (float): y-position for center of circle
      label (str): text that should be displayed within circle
    """
    # -- scatter plot consisting of a single object
    ax.scatter(x0, y0, s=50, linewidth=0.75, facecolors='none', edgecolor='black', zorder=10)
    # -- place text at the center of the scatter plot object
    ax.text(x0, y0, label, backgroundcolor='none', ha='center', va='center', color='black', zorder=10, fontsize=6)

def set_legend(ax, lines):
    """set legend

    Function generating a custom legend, see [1] for more options

    Refs:
      [1] https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.legend.html

    Args:
      ax (object): figure part for which the legend is intended
      lines (list): list of  Line2D objects
    """
    # -- extract labels from lines
    labels = [x.get_label() for x in lines]
    # -- customize legend
    ax.legend(lines,                # list of Line2D objects
              labels,               # labels
              title = '',           # title shown on top of legend
              loc = 0,              # location of the legend
              ncol = 1,             # number of columns
              labelspacing = 0.3,   # vertical space between handles in font-size units
              borderpad = 0.3,      # distance to legend border in font-size units
              handletextpad = 0.6,  # distance between handle and label in font-size units
              handlelength = 2.0,   # length of handle in font-size units
              frameon = False       # remove background patch
              )

def set_grid(ax):
    """set grid

    Function generating a custom grid, see [1] for more options

    Refs:
      [1] https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.grid.html

    Args:
      ax (object): figure part for which the grid is intended
    """
    #ax.set_facecolor('lightgray')
    ax.grid(which = 'major',        # grid lines at major ticks only
            linestyle = '-',        # solid line
            linewidth = 0.5,        # width of gridlines
            color = 'lightgray',    # color of gridlines
            zorder = 0
            )

def save_figure(fig_format = None, fig_name = 'test'):
    """ save figure

    Function that saves figure or shows interactive plot

    Note:
    - if no valid option is provided, an interactive plot is shown

    Args:
      fig_format (str): format to save figure in (options: png, pdf, svg)
      fig_name (str): name for figure (default: 'test')
    """
    if fig_format == 'png':
        plt.savefig(fig_name+'.png', format='png', dpi=600)
    elif fig_format == 'pdf':
        plt.savefig(fig_name+'.pdf', format='pdf', dpi=600)
    elif fig_format == 'svg':
        plt.savefig(fig_name+'.svg', format='svg')
    else:
        plt.show()

def generate_figure(x, t, uxt, fig_format=None, fig_name='fig01'):
    """generate figure

    Function generating a figure reproducing FIG. 1 of [1].

    Refs:
      [1] Interaction of "Solitons" in a Collisionless Plasma and the Recurrence of Initial States,
          N. J. Zabusky and M. D. Kruskal,
          Phys. Rev. Lett. 15, 240 (1965)

    Args:
      x (1D array): x samples
      t (1D array): t samples
      uxt (2D array): wave profile u(x,t)
      fig_format (str): format for output figure
                        (choices: png, pdf, svg; default: interactive figure)
      fig_name (str): name for output figure wihtout suffix (default='fig01')
    """

    # (1) SET A STYLE THAT FITS THE TARGET JOURNAL
    set_style()

    # (2) SET FIGURE LAYOUT
    fig = plt.figure()
    plt.subplots_adjust(left = 0.12,    # pos. of subplots left border
                        bottom = 0.16,  # pos. of subplots bottom border
                        right = 0.97,   # pos. of subplots right border
                        top = 0.98      # pos. of subplots top border
                        )
    gs00 = GridSpec(nrows = 1, ncols = 1)   # set geometry of subplot grid
    ax01 = fig.add_subplot(gs00[:,:])       # add subplot

    # (3) SET AXES CONTENTS
    # -- custom constants and local functions
    tB = 1./np.pi                                   # breakdown time
    _ux = lambda t0: uxt[np.argmin(np.abs(t-t0))]   # wave form for array index closest to t0

    # -- plot real-vaued field at selected times
    l1 = ax01.plot(x, _ux(0.0),    color='blue',  dashes=[1,1], zorder=100, label=r'$t=0$')
    l2 = ax01.plot(x, _ux(1.0*tB), color='green', dashes=[3,1], zorder=101, label=r'$t=t_{\mathrm{B}}$')
    l3 = ax01.plot(x, _ux(3.6*tB), color='black', zorder=102, label=r'$t=3.6\,t_{\mathrm{B}}$')

    # -- add legend
    set_legend(ax01, l1 + l2 + l3)

    # -- set data curve labels
    ax01.text(1.27, -0.9,               # x,y pos. in data coordinates
                r'A',                   # text label
                color='blue',           # text color
                backgroundcolor='none', # backgroundcolor
                ha='left',              # horizontal alignment
                va='bottom',            # vertical alignment
                zorder=10               # layering order in figure
                )
    ax01.text(0.60, -0.9, r'B', color='green', backgroundcolor='none',
                    ha='left', va='bottom', zorder=10)
    ax01.text(1.75, -0.9, r'C', color='black', backgroundcolor='none',
                    ha='left', va='bottom', zorder=10)

    # -- set auxiliary dash-dotted lines
    _f = lambda x: -1.5+1.5*np.where(x>0.8, x, x+2.)
    ax01.plot(x[x<0.75], _f(x[x<0.75]), color='black',
                    dashes = [15,1,2,1,2,1], linewidth=0.75)
    ax01.plot(x[x>0.80], _f(x[x>0.80]), color='black',
                    dashes = [15,1,2,1,2,1], linewidth=0.75)

    # -- set circles with labels
    for  idx, x0 in enumerate([0.643, 0.362, 0.098, 0.920, 1.14, 1.37, 1.6, 1.845]):
        set_circle(ax01, x0, _f(x0) + 0.2, '$%d$'%(idx+1))

    # (4) SET AXIS DETAILS
    # -- customize x-axis
    x_lim = (0, 2)
    x_ticks = (0., 0.5, 1.0, 1.5, 2.0)
    # -- major ticks
    ax01.tick_params(
        axis = 'x',         # axis for which to set ticks
        direction = 'out',  # place ticks outside
        length = 3.5,       # tick length in pts.
        pad = 2,            # tick to label distance in pts.
        top=False           # no ticks on top spine
        )
    ax01.set_xlim(x_lim)
    ax01.set_xticks(x_ticks)
    # -- minor ticks
    ax01.tick_params(
        axis = 'x',         # axis for which to set ticks
        which = 'minor',    # address minor ticks
        length = 2.         # tick length in pts.
        )
    ax01.set_xticks(
        np.linspace(x_lim[0],x_lim[1], int((x_lim[1]-x_lim[0])/0.1), endpoint=False),
        minor=True
        )
    # -- label
    ax01.set_xlabel(r'Normalized distance $x$')

    # -- customize y-axis
    y_lim = (-1., 3.)
    y_ticks = (-1., 0., 1., 2., 3.)
    # -- major ticks
    ax01.tick_params(axis='y', direction='out', length=3.5, pad=2, right=False)
    ax01.set_ylim(y_lim)
    ax01.set_yticks(y_ticks)
    # -- minor ticks
    ax01.tick_params(axis='y', which='minor', length=2.)
    ax01.set_yticks(np.linspace(y_lim[0], y_lim[1], int((y_lim[1]-y_lim[0])/0.2), endpoint=False), minor=True)
    # -- label
    ax01.set_ylabel(r'Real-valued field $u(x,t)$')

    # (5) SET BACKGROUND GRID
    set_grid(ax01)

    # (6) SAVE FIGURE
    save_figure(fig_format, fig_name)

def main():
    x, t, uxt = fetch_data('KdV_raw_data.npz')
    generate_figure(x, t, uxt, fig_format='png', fig_name='fig01')

if __name__ == '__main__':
    main()