# Real User Monitoring

## Summary

Websites and mobile applications are constantly looking to improve end user’s experience with their product, be it stability, performance or usability. By passively observing every user’s interaction with the application, product teams can get deep insights into the user experience and take corrective action. This passive monitoring process is called Real User Monitoring (RUM) or End User Monitoring.

Traditional methods of Application Performance Management focus on downtime or response time. These are from product features perspective rather than end-user perspective. RUM gives a transparent, real-time view of every single user transaction for IT managers to pre-emptively detect and resolve user issues.

RUM tools typically insert small bits of JavaScript code into the client application for continuous data collection. Events such as DNS resolution, TCP connect time, SSL encryption negotiation, first-byte transmission, navigation display, page render time, TCP out-of-order segments, and user think-time are monitored.

## Discussion

### What are the key defining features of RUM?

    + **User Journey Mapping** - Study of user interactions with product at various stages of familiarity with the product (User Lifecycle). Intuitive analysis to understand each user action and the motivation behind those actions. Trace user movement within the product – what they like to do, where they falter, what features they avoid, why they exit.
    + **Real-time Alerts** - Using AI based analysis, problems detected in the user session are compiled into actionable notification alerts, even while user session is in progress. Example alert – “Error rate has been up for over 10 hours in the check-out page of ecommerce app”.
    + **Individual User Session Monitoring** - Collection of chronological events for a particular end user from start of a web session to a configurable period of inactivity. During each user session, pages/components visited and timing information are collected.
    + **Data Visualization and Analysis** - RUM collects data points from a high volume of users over wide range of metrics. Visualizations such as bar graphs, time series charts and area graphs make these large volumes of data human-readable. Actionable insights can be derived from the data.

### What are the objectives of performing RUM?

RUM is generally used by product managers and owners to discover key bottlenecks to meeting their business objectives – ROI, trial to subscription conversion rates, app downloads/uninstall rates and so on. By closely observing how users navigate their way through the app/website, it is possible to unearth the specific reasons for poor user engagement.

RUM helps meet the following objectives:

    + Debug a user’s navigation path in case of failed operations to surface hidden problems.
    + Why a user experiences slow page load times - browser, network, server, or content download that is taking more processing time.
    + Real-time measurement of key targets by tracking actual visits and delivering top-level data on actual use cases.
    + Collect UX metrics such as NPS (Net Promoter Score), Usability Scale, App ratings, Time on Task, Click to Action.
    + User segmentation and testing based on actual operating system, browser, user location, language, device or network settings. Helps in troubleshooting deployment specific issues.

### How is RUM different from web analytics mechanisms such as Google Analytics?

RUM and analytics tools like Google Analytics serve different purposes. Analytics are for collecting high level performance data from product perspective. They do not focus on individual user experience at all. These tools aggregate trends captured across web sessions and show them up as performance statistics.

For example, collect traffic data across the span of a single day, a week or a month. For high traffic scenarios, analytics tools use sampled traffic data for reporting purposes.

The main purpose of Google Analytics is to collect and analyse web data for SEO, lead generation and conversion tracking, from a marketing perspective. RUM instead focuses more on the actual root causes of performance issues that users may experience.

It is impossible to debug issues faced by an individual user using analytics. Or verify feature implementation across product versions. RUM enables monitoring user sessions across deployed devices/browsers or trace unhandled exceptions in a particular crash scenario. RUM captures 100% webpage hits, thus giving a very clear and close view of the user–product relationship.

Analytics is meant to be a post-mortem of web activity, whereas RUM is about real-time analysis and rectification.

### What is synthetic monitoring and how does it compare with RUM?

Synthetic monitoring is the process of analysing a website’s user experience and performance without actual users, instead by simulating them. It is especially effective in the pre-deployment phase, during in-house testing before going live. It's essential for load testing of high-traffic websites.

Synthetic monitoring can answer queries such as:

    + Is my website up and running?
    + Are response times acceptable, especially with high load?
    + Are all transactions working?
    + If there is a slow down or failure where is it in the infrastructure?
    + Are my third party components operating correctly?
    + How is the performance versus cost?Unlike RUM, synthetic monitoring cannot used to find what the real user is experiencing. In real-time deployment scenarios, synthetic test cases may all pass, but a real user might still face transaction outage or speed issues. This can be identified only using RUM.

It’s almost impossible to simulate practical failure cases such as those related to geographical location access, specific user device settings, unexpected network outage.

The general practice is to run synthetic monitoring test cases before going live, and real user monitoring after live deployment, at regular intervals. The two methods complement each other.

### How is an RUM solution deployed?

RUM tool usage is divided into two phases – Deployment and Execution.

During deployment, two steps are performed:

    + JavaScript code RUM integration with client-side product code
    + Install RUM agent on target server on all deployed systems. It discovers all technologies and applications running on it (JS, PHP, IIS, .Net)RUM tool deployment requires full stack development skills.

Now the tool is ready for usage. Its execution is navigated from the tool dashboard. User-specific access level permissions control data access.

### What is the typical execution process flow for RUM?

    + **Data Collection** - captures data about requests for pages, JSON, and other resources from browser to web servers, even when requested content is hosted on third-party sites.
    + **User Session Analysis** - captured data regrouped into session-wise record of pages, components, timing information. User journey analysed for registered/anonymous users, new/recurring users. JavaScript errors checked for show stoppers or mild irritants.
    + **User Performance Analysis** – Various response times as perceived by user are studied at front/back end. Metrics optimised as per user bandwidth availability.
    + **User Behavior Analysis** – Frequent user activities, entry/exit actions, landing pages, last page viewed before exit, bounce rate, user leaving app after first visit are studied.
    + **Problem Detection** - undesirable behaviour such as slow response time, system problems, page load errors, transaction failures, web navigation errors, analyzed for different pages, objects, sessions.
    + **AI-powered Root Cause Analysis** - DevOps team perform RCA and map user issues with underlying root cause. Analytics model understands the context of each metric.
    + **Reporting and Notification** - Various data visualization formats and problem reports can be accessed. When data points cross configured thresholds, notification alerts can be configured to be sent.

### What are the popular RUM tools used?

RUM tools are testing and analysis platforms that enable product teams to study whether applications are working as per their intended design.

RUM tools are generally deployed within the product environment by embedding JavaScript snippets into product code. They are not plug and play tools, they require code-level integration with the product. Most of the tools support React, AJAX, AngularJS web applications. For RUM on mobile apps, code is embedded into the app source code for Android/iOS apps.

RUM tool dashboard can then be used to monitor user sessions at real-time and observe key metrics such as page rendering time, environment specific performance, error rates, and bounce rates. Alert notifications can be set for key touch-points.

Some tools also support Single Page Application monitoring (where web apps dynamically rewrite current page content itself every time, instead of loading new pages for every operation). Such apps mostly use AJAX requests to pull content dynamically and create a fluid user experience.

Some of the popular RUM tools include Sematext, Monitis from TeamViewer, SmartBear, Dynatrace, SOASTA mPulse from Akamai, Raygun, AppDynamics from Cisco, and CXOptimize by Cognizant (Open Source).

### What are the limitations and concerns in using RUM?

RUM is useful in several scenarios, but has some limitations. There are also concerns of performance overhead and security. Some are listed below:

    + **Not effective with limited traffic** – Before product launch or just after launch, websites/apps don’t generate much traffic. RUM would not detect any meaningful issues at this stage. It works best when there is steady traffic volume.
    + **Generates too much of data** - Ironically, during heavy usage, large number of datasets are collected. Small companies won't have sufficient DevOps resources for analysing this.
    + **Trial and error visible to users** - Synthetic monitoring is done using simulated environments without involving the actual user. Whereas, RUM tool runs in actual production environment. If issues are found regularly, users would notice frequent product updates. Product may appear immature.
    + **Exposure risk at code level** - Product teams must be careful before integrating RUM script code into product source code. Critical product functionality should not get exposed to external threats.
    + **Not useful for benchmarking** - RUM data is inherently random because it depends entirely on user activity which is unpredictable. So fixed interval monitoring of data may not yield accurate benchmarking results.

## Milestones

2005

Google launches its **Web Analytics** solution that tracks and reports website traffic.

2006

Dynatrace, Sematext and other software intelligence companies introduce their **Application Performance Management** products of which RUM is a part. Over the next few years, these tools employ Big Data and Cloud Based services to help internet companies monitor their user traffic.

2008

ISO Standard for measuring User experience was established (ISO 9241-210) as early as 1999. With its introduction in 2008, **ISO 13407** is revised for human-centred design for interactive systems. The goal is to establish standards and metrics to measure user experience and engagement with web applications.

2016

Gartner defines end-user experience in three dimensions - (a) End User Experience Monitoring (RUM); (b) Application discovery, tracing and diagnostics (ADTD); (c) Application analytics (AA).

Feb
2017

In a blog post, LinkedIn engineer describes how they used RUM to improve the performance of a Single Page Application (SPA). RUM data helped them identify bottlenecks. Traditional RUM libraries rely on Navigation Timing API and Resource Timing API. But these fail to capture JavaScript execution times. To address this, they use User Timing API. To improve performance, they lazy load data and do lazy rendering. Their web application is now faster by 20%.

Real User Monitoring


# GitHub Actions

## Summary

In any software project, there are repetitive tasks that can benefit from automation. Code reviews, branch management, pull request labelling, regression testing, issue triaging, release management, npm package publication, and cloud deployment are examples of such tasks. These are hard to do manually or consistently. GitHub Actions is feature of GitHub to automate these tasks.

Before 2018, third-party tools and services such as CircleCI, Jenkins and Travis CI were used to automate some of these tasks. Bots automated repetitive tasks. With GitHub Actions, it's now possible to do these more easily within the GitHub platform for code hosted at GitHub.

The GitHub Marketplace has thousands of free GitHub Actions shared by developers. This improves productivity for developers and teams. Actions can be executed for private/public repositories under free/paid plans, though limits differ.

## Discussion

### What are the use cases for GitHub Actions?

While there are thousands of tasks that could be automated with GitHub Actions, we mention a few. The notable use case is CI/CD in which every commit triggers a build-and-test cycle so that code merges become simpler and errors are caught proactively.

When making a software release, with Actions we can build the software for multiple architectures in parallel.  Likewise, for software testing or code coverage, we can exercise the software across all variations of platform versions, browsers, operating systems, etc.

Before publishing container images to a registry, it's a good idea to scan them for security vulnerabilities. After scanning, the resulting metadata is checked against security policies for compliance. These tasks can be part of a GitHub Actions workflow.

GitHub itself uses Actions to track vulnerabilities in open source components they use. When an issue is raised, a workflow updates an internal database and reports it as per security policies.

### What are the components of GitHub Actions?

We describe the following components:

    + **Workflow**: Encapsulates a specific automation. Defined in a configurable YAML file at `.github/workflows` within the repository. A repository can have multiple workflows.
    + **Event**: Triggers a workflow. Pull request, new commit and new issue are examples of events. A REST API call, a manual trigger or a defined schedule can also trigger workflows. See the full list of events.
    + **Job**: A sequence of steps that define the actual execution of a workflow. A shell script or an action is executed. A workflow can have one or more jobs. Jobs can be defined to run in parallel or in sequence. Steps within a job are sequential and can share data.
    + **Action**: A custom application for the GitHub Actions platform. Encapsulates repetitive tasks and enables code reuse across workflows. Actions are steps in a job. Configured in a YAML file at `.github/actions`.
    + **Runner**: A server that runs a workflow. A runner runs a single job. GitHub provides Ubuntu Linux, Microsoft Windows, and macOS runners. Projects can also use self-hosted runners. Each workflow run creates a new virtual machine.

### Could you describe some important features of GitHub Actions?

**Reusability** is an essential feature. A workflow can reuse actions defined in the same repository, a public repository or a published image on Docker Hub. A workflow itself can be reused if defined in the same repository or a public repository.

Actions can use **variables**. These are name-value pairs under `env`, which can be scoped to the entire workflow, a job or a step. In addition, workflows can use default environment variables. Examples include `GITHUB_ACTION_PATH`, `GITHUB_EVENT_NAME`, `GITHUB_RUN_ID`, `RUNNER_ARCH`, `RUNNER_OS`, and many more.

**Expressions** are those within `${{…}}`. In conjunction with `if`, they can be used to execute jobs or steps conditionally. Expressions include literals (`null`, `true`, `false`, numbers, strings), operators (index, compare, logical, etc.) and functions (`contains`, `startsWith`, `join`, `toJSON`, etc.).

**Contexts** are objects that contain useful information. Available contexts include `github`, `env`, `job`, `steps`, `runner`, `secrets`, `strategy`, `matrix`, `needs`, and `inputs`. Default environment variables exist only on the runner. In other parts of the workflow, use contexts. For example, `if: ${{ github.ref == 'refs/heads/main' }}` executes the job only if the current branch is `main`.

### What's the mechanism to pass data among jobs and workflows?

To set an environment variable, here's an example showing two ways to do it: `echo "action_state=yellow" >> $GITHUB_ENV` within a `run` or `action_state: yellow` within `env` context. Subsequent steps of the job can access it via `env.action_state`. In shells, it can be accessed as `$action_state` (Linux) or `$env::action_state` (PowerShell).

Suppose *job2* depends on *job1*. Via `outputs` property, *job1* can pass data to *job2*, which can access it via the `needs` context. **Workflow artifacts** allow for persistent data across jobs of the same workflow. Common artifacts include log files, core dumps, test results, code coverage results, screenshots, and binary files.

To pass inputs to a workflow, the property `inputs` can be used. Each input has a name and sub-properties `description`, `type`, `default` and `required`. Within the workflow, inputs can be accessed via `github.event.inputs`.

### What's the platform and language support for GitHub Actions?

GitHub Actions is supported on Linux, Windows and macOS platforms. Workflows can be configured to use specific versions of these platforms or the latest version. This is specified via the job property `runs-on`. Another useful feature **build matrix**. With a single job definition, developers can execute variations of the job in parallel, such as for different OS or Node versions. This is specified with the job property `strategy.matrix`.

For programming actions, GitHub Actions supports C, C++, C#, Go, Java, JavaScript, PHP, Python, Ruby, Scala, and TypeScript.

Within a Docker environment any platform and any language can be used. The job property `container` includes container configuration including image, environment, ports and volumes to use. A developer can releases Docker images for various architectures.To facilitate this, Docker has published actions for QEMU setup (`docker/setup-qemu`), enhanced build capability (`docker/setup-buildx`), Docker Hub authentication (`docker/login-action`), and build and push images to Docker Hub (`docker/build-push-action`).

### What are some best practices for GitHub Actions?

Keep actions minimal to reduce wait times and favour reuse. Install only necessary dependencies. In fact, leverage GitHub's feature that can cache dependencies. In case of Node.js, publish the entire `node_modules` folder. Don't pollute the global environment with lots of variables. Variables can be scoped to narrower levels such as a job or a step. Don't overwrite default environment variables. To attribute action to authors, use the `author` metadata in YAML file. Version actions properly. Test actions before releasing them.

From a security standpoint, avoid self-hosted runner in a public repository. Someone could take a fork and submit a pull request with malicious code. Use GitHub web interface to manage secrets. Never hardcode these actions in a YAML file. Don't dump context data into logs since this could expose sensitive information.

Many Marketplace actions are not certified by GitHub. Have a security policy to either fully block them or allow them via forking. When reusing actions, pick a stable version rather than the latest on the master branch.

### What tools can help developers create GitHub Actions?

For many common use cases, developers need not create actions from scratch. The GitHub Marketplace has 12,000+ actions shared by developers and organizations. Of these, nearly 300 are from verified creators. Marketplace actions are grouped into many categories: API management, code quality, continuous integration, dependency management, IDEs, mobile, monitoring, security, testing, and more.

Beginners can start with the official documentation. The GitHub Actions guides has tutorials and overviews on specific use cases. Take a look at the course material of GitHub Actions: Hello World. For a summary, consult the Actions Cheat Sheet.

Details of VMs used by GitHub-hosted runners are available.

Those using other CI/CD tools can read the migration guides. These are available for Azure Pipelines, CircleCI, GitLab CI/CD, Jenkins and Travis CI.

To help developers build actions easily, toolkits are available. The official toolkit supports JavaScript, TypeScript and Docker. An alternative is Etcovitch's toolkit. A curated list of GitHub Actions is also available online.

## Milestones

Apr
2008

**GitHub** is launched as an online software repository that uses Git, a decentralized version control system. By July, GitHub hosts 10,000 repositories. It's particularly popular among open source and Rails communities.

Oct
2018

Microsoft is in the process of acquiring GitHub. About the same time, **GitHub Actions is announced**. This makes GitHub more than just a code repository. Workflows can be automated. Sam Lambert, GitHub's platform head, describes this as "the biggest shift we've had in the history of GitHub."

Nov
2018

A study of 351 sampled open source projects on GitHub finds that 93 (26%) use bots. This excludes the newly launched GitHub Actions. Bots are used for ensuring license agreement signing, reporting integration failures, reviewing code, reviewing/merging pull requests, assigning reviewers, welcoming newcomers, running automated tests, building, creating issues, and more.

Nov
2019

Due to popular demand from developers, support for **CI/CD** becomes generally available. An example CI/CD pipeline would be to trigger a build and deploy to Heroku whenever a new commit happens on the master branch. Based on the project, GitHub can suggest suitable Actions workflows. It's now easy to publish/consume packages from GitHub Package Registry and other registries. Beta version of this feature was announced in August.

Sep
2020

With the release of GitHub Enterprise Server 2.22, GitHub Actions is for the first time **supported in GitHub Enterprise Server**. This includes support for enterprise features such as centralized self-hosted runners, runner groups to manage access controls, and custom workflow templates. By now, GitHub Actions is the most popular CI solution on GitHub.com.

Mar
2021

Kinsman et al. publish results of a study on how developers are using GitHub Actions on open source projects. The study considers the period Nov 2019 to Aug 2020. Of the 416k+ active repositories, only 0.7% are using actions. Python, Java and Ruby repositories are the top adopters. Common actions include repository checkout or environment setup. Most actions are in the categories of continuous integration, utilities and deployment. They also find that Actions doesn't result in faster merges of pull requests.

Oct
2021

Beta version of **reusable workflows** is introduced. Repetitive tasks can be encapsulated into actions or workflows. This reduces duplication and brings consistency. Like actions, workflows support inputs and outputs to pass data. In November, this feature becomes generally available.

Feb
2022

At the GitHub Marketplace, there are now 12,000+ Actions that developers can readily use.

GitHub Actions


# Control and User Plane Separation

## Summary

In traditional 4G LTE networks, control plane and user plane functionalities were combined into the same network elements. The Serving Gateway (SGW), the PDN Gateway (PGW) and the Traffic Detection Function (TDF) are examples. In Release 14, 3GPP published a modified architecture is which the two planes were separated. Called **Control and User Plane Separation (CUPS)**, SGW got split into SGW-C and SGW-U. Likewise, we obtained PGW-C, PGW-U, TDF-C and TDF-U. CUPS was subsequently adopted in the design of the 5G System, both in the 5G Core and in the NG-RAN.

CUPS leads to better scalability and performance. For the more demanding use cases of 5G, CUPS may play a significant role. Control and user plane specifications can now evolve independently. However, implementing CUPS has its challenges.

## Discussion

### Could you give an overview of CUPS?

The figure shows an example of CUPS as applied in 4G/LTE EPC (Evolved Packet Core). SGW and PGW are split into their respective control plane and user plane parts. These parts communicate via the PFCP protocol. Each plane has a well-defined functionality. For example, packet forwarding and transport level packet marking are exclusively user plane features.

When a PDU session is established, the CP function gets involved. CP associates with a UP function and configures the latter to process user plane traffic. Subsequently, when user plane traffic flows, UP function is actively involved. UP function occasionally updates CP function as necessary. UP bandwidth per session is in the order of Gbps whereas CP bandwidth requirement is perhaps only a few Mbps. UP latency for URLLC is 1ms whereas for CP its 10ms. With these differing performance requirements, it makes sense to separate CP and UP functions.

In 5G CUPS, SMF takes care of the signalling while UPF is the user plane counterpart. SMF and UPF communicate using PFCP over the N4 interface.

### Is CUPS applicable for the RAN?

In 5G's Next-Generation Radio Access Network (NG-RAN), gNB is no longer a monolith. It's disaggregated into Central Unit (CU), Distributed Unit (DU) and Radio Unit (RU). Furthermore, the CU is split into CU-CP and CU-UP for the control plane and user plane respectively.  A gNB consists of one CU-CP and one or more CU-UP/DU/RU each.

Due to CUPS applied to CU, we have these new interfaces: E1 interface between CU-CP and CU-UP, F1-c interface between CU-CP and DU, and F1-u interface between CU-UP and DU. Moreover, old X2, Xn and NG interfaces are disaggregated into X2-C/X2-U, Xn-C/Xn-U and NG-C/NG-U respectively. The E1 interface is purely a control plane interface.

CU/DU split can be done in many ways. For standardization work, 3GPP adopted what's called *Option 2* in which RRC/PDCP-C are in CU-CP, SDAP/PDCP-U are in CU-UP, and RLC/MAC/PHY are in DU.

In 4G/LTE, ng-eNB is disaggregated into ng-eNB-CU and ng-eNB-DU. Between these two, W1-C and W1-U interfaces are defined. However, unlike in 5G, ng-eNB-CU is not split into separate control and user plane entities.

### What are the key benefits of CUPS?

Due to CUPS, CP and UP functions can be scaled independently. During peak periods, more UP nodes can be added without increasing the CP nodes. UP functions can be located closer to UEs that they serve, thus reducing latency. A UE may be served by a single CP function and many UP functions, one for each PDU session.

For operators, CUPS can result in lower operating cost. By processing data at the edge, backhaul costs can be reduced. Each UP function can be specialized for the application use case it serves. CUPS allows operators deliver new services and handle traffic growth without sacrificing customer experience.

CP and UP functions can evolve independently. For example, CP nodes may be upgraded without upgrading UP nodes, provided the evolution is backward compatible.

Likewise, CUPS in NG-RAN gives operators flexibility, and optimize for cost and/or performance.

### What are the possible deployments due to CUPS?

The figure shows three possible deployments. For low latencies, SGW-U/PGW-U can be deployed closer to cell sites. In urban areas with a large subscriber base, high performance and high capacity SGW-U/PGW-U may be deployed. If we need resilience to disasters and congestion, SGW-U/PGW-U may be deployed with redundancy. In other words, if one SGW-U/PGW-U goes down, another in hot standby mode will take over. For that matter, we can have similar redundancy in the control plane as well.

In the RAN, we can co-locate CU-CP and CU-UP in the cloud. An alternative is to co-locate CU-UP and DU at the edge while CU-CP remains centralized in the cloud. This can reduce the midhaul latency but at the expense of control plane signalling overhead.

### How does a CP function select a UP function?

In traditional 4G EPC, MME selects SGW and PGW using DNS. This is based on UE's location and Access Point Name (APN). With CUPS, MME selects SGW-C/PGW-C and the latter select SGW-U/PGW-U. SGW-C/PGW-C will use UE's location, APN, UP performance and load status. Thus, with CUPS, SGW-C/PGW-C uses dynamic information in the selection of SGW-U/PGW-U.

Suppose a UE has an IMS voice session and a data session. Before CUPS, an EPC wishing to avoid the S5 hop would select the same node for SGW and PGW of both sessions. With CUPS, a combined SGW-C/PGW-C plus two specialized combined SGW-U/PGW-U nodes can be selected.

In some scenarios, selecting the right SGW-U/PGW-U can be a problem. For example, SGW-C and PGW-C may select UP functions that are not co-located. This results in two hops in the data path. The figure shows a UE in 5G-to-4G handover. SGW-C doesn't know which UP function is being currently used.

### What's the migration path from traditional deployment to CUPS-based deployment?

A traditional 4G transport network may not get the full benefits of CUPS. Migrating EPC towards CUPS enables smooth evolution towards 5G Core. Cisco has proposed a three-stage migration: inline CUPS where CP and UP functions are logically separated; co-located CUPS where CP and UP functions are separate nodes but co-located in the same data center; remote CUPS where the functions are in different data centers.

## Milestones

Jul
2015

At the SA WG2 Meeting #S2-110, a study item titled *Feasibility Study on Control and User Plane Separation of EPC nodes* is proposed. This is subsequently approved in September at the 3GPP TSG SA Meeting #69. The document notes that such a separation shall not affect the functionality of the nodes being split. It shall not affect UE or RAN. New reference points shall be introduced only among the split parts of SGW, PGW and TDF.

Jun
2016

3GPP publishes **TR 23.714** titled *Study on control and user plane separation of EPC nodes* as part of Release 14. The document identifies the different issues and possible solutions. The selected solution is a functional split of user plane and control plane functions, including the case of combined SGW/PGW. User plane functions shall be selected by the respective control plane functions.

Sep
2016

3GPP publishes **TS 23.214** that specifies the architectural changes to support CUPS in LTE EPC. This is for Release 14. This evolves for other releases: V15.0.0 (Sep 2017), V16.0.0 (Jun 2019) and V17.0.0 (Jun 2021).

Dec
2017

3GPP approves the first specifications for 5G, called "early drop" of Release 15. This release specifies CUPS in 5G Core in **TS 29.244** (Jun 2017) and CUPS in NG-RAN in **TS 38.401** (Dec 2017). Thus, unlike in 4G, 5G has adopted CUPS from the outset.

Jan
2020

The O-RAN Alliance publishes documents detailing the O-RAN architecture. These documents show CUPS applied to the RAN. In particular, the Open Distributed Unit (O-DU) and the Open Central Unit (O-CU) are split. E1 interface connects O-CU-CP and O-CU-UP. F1-c and F1-u interfaces connect O-CU-CP and O-CU-UP to the O-DU. These interfaces are in fact specified by 3GPP and adopted by O-RAN.

Control and User Plane Separation


# R Data Structures

## Summary

R is an object-oriented language and all data structures are objects. R doesn't provide programmers direct access to memory and all data must be accessed via symbols or variables that refer to objects.

Since vectorized operation is an important aspect of R, R does not have any scalars. The most basic data structure is a `vector`, which is a sequence of data items. Thus, a single integer value is treated as an integer vector of unit length. The most versatile data structure is the `list` while the most common one used for data analysis is the `data.frame`.

The terms *data type* and *mode* usually refers to what is stored (integer, character, etc.). The term *data structure* usually refers to how data is stored, that is, the containers (vector, list, etc.).

## Discussion

### What data types are available in R?

Data types are many. The common ones include `integer`, `real`, `complex`, `logical` and `character`. Types `integer` and `real` are termed as `numeric`. There's no separate "string" type. Instead, `character` type is sufficient to denote strings.

Integers are specified with a suffix "L", such as `23L` or `-2L`. Real numbers are specified without this suffix, such as `2.3` or `23`. Examples of complex numbers are `-2+3i` and `-45i`. Logical type can take values `TRUE` or `FALSE`. These have shortforms `T` and `F`. Character type can be specified by a matching pair of single or double quotes, such as `"R"`, `'R'` or `"This is R!"`.

### Could you compare vector, matrix, array, list and data.frame?

The following data structures are common in R:

    + `vector`: Contains a sequence of items of the same type. This is most basic structure. Items of a vector can be accessed using `[]`. Function `length` can be called to know the number of items.
    + `list`: Represented as a vector but can contain items of different types. Different columns can contain different lengths. Items of a list can be accessed using `[[]]`. This is a recursive data type: lists can contain other lists.
    + `array`: An n-dimensional structure that expands on a vector. Under the hood, this has `dim` and optionally `dimnames` attributes, which don't exist for vectors. Like vectors, all items must be of the same underlying type.
    + `matrix`: A two-dimensional array.
    + `data.frame`: While all columns of a matrix have same type, with data frames, different columns can have different types.Formally, vectors can be said to be of two types: atomic vectors (items of same type) and lists. In practice, when we say vectors we are referring to atomic vectors.

### What are factors?

Consider the sex of a person. This variable can have only two possibilities or categories: male or female. We call this categorical data and *factors* are used to represent such data. Roughly equivalent to an "enum" type in C, factor represents a finite set of values.

Under the hood, these are nothing more than integer vectors with each integer representing one category. In R, these possible values are called *levels*.

Thus, though sex may contain values "male" or "female", these are not characters but integers. In addition, factors can be ordered or unordered. For example, sex may be defined as unordered factor. Olympics medal may be defined as ordered factor such that Bronze < Silver < Gold.

### Is the NULL object a special data type?

R documentation states that "the NULL object has no type and no modifiable properties". Attributes don't apply to NULL. When you want to indicate absence, NULL can be used. A vector or list of zero length is not the same as NULL.

### How to interpret the functions class, mode, typeof and storage.mode?

All these functions can be called on R with differing results. Function `class` represents the object's abstract type whereas `typeof` is the object's specific type. A good example is factors: its class `factor` but its type is `integer`. Another example is a data frame: its class is `data.frame` but its type if `list`.

Function `mode` is similar to `typeof` and it exists for compatibility with R's predecessor, the S language. Function `storage.mode` also exists for compatibility with S. It's useful when interfacing to code in other languages. For example, consider a vector of integers. Functions typeof, mode and storage.mode will respectively return `integer`, `numeric` and `integer`. In S, both integers and reals have the same mode and hence storage.mode becomes useful.

Hadley Wickham has commented that it's best to avoid using mode and storage.mode in R. If we need the underlying type, calling `typeof` should be preferred over `storage.mode`.

### What are some basic operations on R vectors?

Here are some basic operations on vectors:

    + **Combining**: We can combine vectors into a single vector. Eg. `v <- c(v1, v2)` to combine v1 and v1 into v.
    + **Indexing**: Indexing starts from 1. Negative numbers imply selecting all others except those specified. Eg. `v[1]` for first element; `v[-3]` for elements except the third one. Indexing may be treated as a special case of subsetting.
    + **Subsetting**: We can select a subset of a vector by using integer vectors for indexing. Eg. `v[c(1,3,5)]` to select first, third and fifth element; `v[1:3]` to select the first three elements. We can also use a logical vector to subset a vector. Eg. `v[v > 5]` to select elements with value greater than 5.
    + **Coercing**: Since vectors contain elements of a single type, values are coerced to a single type if they are different. Eg. `v <- c(12L, 2.2, TRUE)` coerces to doubles [12.0, 2.2, 1.0]; `v <- c(2.2, TRUE, "Hi")` coerces to characters ["2.2", "TRUE", "Hi"].
    + **Converting**: May be called *explicit coercing*. Convert the type. Eg. `as.integer(c(3, 2.2, TRUE))` becomes [3, 3, 1]; `as.numeric(c(2.2, TRUE, "Hi"))` becomes [2.2, NA, NA], where NA stands for "Not Available".

### Are there datasets to understand the different data structures?

R comes with many datasets for experimental analysis and learning. These can be listed by typing `data()` in the R console. Details of each dataset can be obtained by using `?` or `help`. For example, for help on "rivers" dataset type either `?rivers` or `help(rivers)`. It's been said that datasets "mtcars", "iris", "ToothGrowth", "PlantGrowth" and "USArrests" are commonly used by researchers.

An example of vector is the "rivers" dataset. An example of a vector with names given to each observation is "precip". There are plenty of examples for data.frame: "airquality", "mtcars", "iris". An example of a list in "state.center" dataset.

In "CO2", "Plant" variable is an ordered factor whereas "Type" variable is an unordered factor. Dataset "Titanic" is of class `table`, which is a type of array. This data structure records counts of combinations of factor levels.

An object can belong to multiple classes. As an example, try `class(CO2)` and `str(CO2)`. It's a data.frame but also belongs to other classes.

### Can you give examples of data structures beyond the core ones given by R?

Developers can create their own data structures that can build on top of the basic ones. One popular one is called **data.table**, which is based on data.frame. It offers a simplified and consistent syntax for handling data.

Another example is **tibble**, which retains the effective parts of data.frame and does less work so that developers can catch problems early on.

If you want to display data frames in HTML with conditional formatting (like in Microsoft Excel), **formattable** is a suitable package to use.

Another package named **dplyr** isn't exactly a data structure. Rather, it offers a number of functions for manipulating data. It comes as part of the *tidyverse* collection of R packages targetted towards data science.

## Milestones

2006

Version 1.0 of **data.table** is released.

Apr
2013

R version 3.0.0 is released. This version supports vectors longer than `2^31 - 1` elements. This applies to raw, logical, integer, double, complex and character vectors, as well as lists. Elements of character vectors are limited to `2^31 - 1` bytes.

Jan
2014

Hadley Wickam releases version 0.1 of **dplyr**, which is meant "to provide a consistent set of verbs that help you solve the most common data manipulation challenges." Version 1.0.0 is release in May 2020.

Apr
2020

R version 4.0.0 is released. This version uses `stringsAsFactors = FALSE` as default when reading tabular data via `data.frame()` or `read.table()`. Previous versions used to convert strings to factors by default.

R Data Structures


# Edge Computing

## Summary

With the growth of the Internet of Things (IoT) billions of devices are generating huge amounts of data. But to store or analyze all that data in real time is almost impossible. This is where edge computing becomes relevant.

With edge computing, we process data closer to the source, such as an IoT device, an IoT gateway, an edge server, a smartphone, or a user's computer. The idea is to move intelligence to the edge and let edge nodes/devices do **real-time analytics**. This reduces application latency and saves network bandwidth. The architecture is **distributed**, as opposed to centralizing all processing in the cloud. It minimizes long-distance client-sever communication.

Gartner defines edge computing as,

> part of a distributed computing topology where information processing is located close to the edge, where things and people produce or consume that information.

## Discussion

### Could you explain edge computing with an example?

Consider a building security application that has many networked cameras. Cameras capture high-definition video streams. Assume that the cameras are 'dumb'. They simply capture and transmit raw video to a cloud server. The cloud server analyzes the video streams for motion detection. We note two problems in this scenario: strain on network bandwidth and strain on the cloud server that has to process videos from multiple cameras.

With edge computing, we enable each camera to run motion detection locally. Cameras are equipped with some storage and sufficient compute power to do this. Cameras have now become 'smart'. Only important video segments are sent to the cloud for storage or deeper analysis. Both network bandwidth and cloud storage/processing are saved. In turn, the cloud server can now support many more cameras.

### What are some use cases of edge computing?

In autonomous driving, if a pedestrian crosses the road, the vehicle may have to brake immediately. Waiting for the cloud to make this decision may prove fatal. Vehicles can also communicate directly with one another.

In healthcare, there are glucose monitors, fitness trackers, and other health-monitoring wearables. Some monitoring devices locally analyze pulse data or sleep patterns without involving the cloud. Edge computing enables timely care for remote patient monitoring, in-patient care and healthcare management in hospitals.

In smart factories, the lower latency due to edge computing enables more timely actions to control manufacturing workflows. If analytics determines that a machine is about to fail, immediate action can be triggered to stop the machine. Robots that process their own data will be more self-sufficient and reactive. Edge computing enhances safety and efficiency on the factory floor.

In agriculture, connectivity is an issue at remote locations. This leads to high investment towards fibre, microwave or satellite connections. Edge computing presents a more cost-effective alternative. Yield can be improved and food wastage reduced.

Many more use cases exist in other sectors such as consumer electronics, defence, telecom, oil & gas, energy, retail and finance.

### Which are the main benefits of edge computing?

These are some benefits of edge computing:

    + **Reduced Latency**: For delay-critical applications, the longer it takes to process data, the less relevant it becomes. Edge computing avoids roundtrip delay.
    + **Better Security**: Centralized cloud systems are vulnerable to DDoS attacks. Edge computing allows us to filter sensitive information locally.
    + **Cost Savings**: By retaining and processing data closer to source, edge computing saves on connectivity costs. Data can be categorized and handled suitably: stored locally, stored in cloud, or processed and discarded. Redundant storage is avoided. Data management solutions also cost less.
    + **Greater Reliability**: Connectivity to the cloud is never perfect. Edge devices/nodes can deal with temporary outages by storing or processing data locally.
    + **Scalability**: While cloud infrastructure is built for scalability, data still needs to be sent to datacentres and stored centrally. Edge computing complements this by scaling in a distributed manner.
    + **Interoperability**: Edge nodes can act as intermediaries to interface legacy and modern machines.

### Which are the technologies that enable computing at the edge of networks?

**Cloud computing** itself is an enabler for edge computing. Edge computing doesn't replace cloud computing. They complement each other. Based on application requirements, engineers must decide what data is best processed at the edge and what should go into the cloud.

**Fog computing** extends the cloud while also being closer to edge devices. With fog computing, we place compute and storage in the most logical and efficient location between the cloud and the origin of data. As part of the fog computing infrastructure, there are *cloudlets* and *micro datacentres*, which are simply edge servers clustered together to serve local storage or compute requirements. They are suited for resource intensive or interactive applications.

**Multi-Access Edge Computing (MEC)** is another enabler. In cellular networks such as 4G/5G, Radio Access Network (RAN) refers to the part that handles radio or wireless resources and communication. MEC places compute and storage resources within the RAN to improve network efficiency and content delivery.

More generally, dropping cost of electronics has brought more compute power to smaller devices. Tools for real-time analytics, particularly on embedded devices, have also made edge computing possible.

### How does edge computing differ from fog computing?

Sensors, IoT devices and even IoT gateways are **edge devices**. On the other hand, **edge nodes** belong to the fog network that interfaces edge devices to the cloud. Therefore, fog computing depends on edge computing but the reverse is not true.

For example, consider a train fitted with sensors. A sensor giving engine status is processed locally on the train. This is edge computing. Suppose data from sensors attached to wheels and brakes need to be aggregated over time and processed. These can be sent to an edge node (fog computing) without saving that data in a centralized cloud.

There's no universal definition of fog computing. Some regard edge devices, edge nodes and even localized datacentres as belonging to the edge network, not preferring to use the terms fog computing or fog network.

Some others regard edge computing as any processing that's close to the origin of data. Any processing that happens on devices connected to the LAN or on LAN hardware is seen as fog computing. Investing in fog computing makes sense if data has to be aggregated from many edge devices.

### Which organizations are standardizing edge computing?

Many organizations are standardizing edge computing. There's no single standard that currently covers all aspects of edge computing.

As early as 2017, many organizations got interested in this space: The European Telecommunication Standards Institute (ETSI), OpenFog Consortium, Open Networking Foundation (ONF), Open Edge Computing and Facebook's Telecom Infra Project (TIP). TM Forum, CNCF (Cloud Native Computing Foundation), ONAP (Open Network Automation Platform) and LF Edge are also involved.

**LF Edge** is an umbrella organization within the Linux Foundation. It hosts many edge-specific projects.

AECC (Automotive Edge Computing Consortium) and 5G-ACIA (5G Alliance for Connected Industries and Automation) focus on specific verticals that matter to them.

By 2020, ETSI's **Multi-Access Edge Computing (MEC)** was emerging as an important standard. 5G Future Forum and Wireless Broadband Alliance are two organizations who have adopted MEC specifications.

### What hardware is available to implement edge computing?

Important considerations for edge computing hardware include processing power, power source, memory, wireless connectivity, variety of ports/interfaces, reliability and ruggedness.

For the edge, RISC processors such as ARM, ARC, Tensilica, and MIPS are preferred over CISC. While ARM Cortex is suitable, ARM also offers Neoverse specifically for edge use cases. ARM Cortex-M55 and Ethos-U55 are AI edge computing chips.

NVIDIA Jetson GPUs are designed for the edge. For example, Jetson Nano is a 128-core GPU. Jetson TX2 and Jetson Xavier are for industrial and robotic use cases. There's also NVIDIA EGX Platform that offers GPU edge servers.

Intel has Movidius and Myriad 2. The latter is also part of Intel's Neural Compute Stick (NCS) that draws power from host device via USB.

Mainflux Labs offers MFX-1 IoT Edge Gateway. Huawei's Atlas AI Computing Platform of AI accelerators, edge stations and servers is based on its Ascend AI processors. Scale Computing offers HC3 Edge and HE500 systems. APC's EcoStruxure Micro Data Centre promises physical security, standardized deployment, and remote cloud-based monitoring. There are many more focusing on micro datacentres.

### How are cloud providers enabling edge computing?

Google uses TPUs in its datacentres. In 2018, it started offering **Edge TPU**, an ASIC for AI inferencing at the edge. TensorFlow Lite models can run on an Edge TPU.

Microsoft's **Azure Stack Edge** is a cloud-managed edge appliance with compute, storage and intelligence. Processors are Intel Xeon. For machine learning workloads, Intel Arria10 FPGA is the hardware accelerator.

AWS has a number of services for the edge. **AWS Outposts** extends AWS infrastructure and services to any other datacentre or on-premise facility. **AWS Snow Family** offers small portable rugged edge devices to be deployed as close as possible to sensors collecting data. **AWS Wavelength** brings single-digit millisecond latencies to mobile devices and end users. It does this by bringing AWS compute and storage to telecom datacentres at the edge of 5G networks.

## Milestones

1988

Mark Weiser at Xerox PARC coins the term **Ubiquitous Computing**. Unlike desktop computing, the term implies that computing can happen in any device, in any location. It can happen in laptops, mobile phones, sensor devices, and everyday objects such as refrigerators, umbrellas or alarm clocks. In the late 1990s, a similar term **Pervasive Computing** is coined.

Aug
1998

Akamai is incorporated "to intelligently route and replicate content over a large network of distributed servers." This is what we call a **Content Delivery Network (CDN)**. Akamai delivers first live traffic in February 1999 and launches commercial service in April. Yahoo! becomes one of their customers. By 2019, Akamai is said to have 240,000 edge servers, many of which are located in ISPs or mobile data towers. A user request is served by the closest available edge server that also caches content.

1999

Napster is launched as a file sharing software to download and distribute music. The common way to download on the internet is to connect to a server and download files to the client machine. Napster is a **peer-to-peer (P2P) network** in which machines are both clients and servers. One machine can download parts of file from another nearby machine. This distributed architecture makes P2P network scalable, faster and resilient to failures. This idea of cooperative file sharing can be traced to USENET (1979).

2006

Amazon launches Amazon Web Services (AWS). It's Elastic Compute Cloud service enables users to use datacentre infrastructure to run programs. The same year Google Docs is launched. Spreadsheets and documents can be edited and saved online. This is the beginning of **cloud computing**. However, Saleforce pioneered this model back in 1999.

2009

Satyanarayanan et al. coin the term **cloudlet**. They recognize that network latency hurts user experience for highly interactive applications. A response time of more than a second becomes annoying. Mobile devices are also resource constrained and relying on the cloud is too slow. Cloudlets are a solution. They provide the compute power for nearby edge devices. They connect to edge devices via low-latency high-bandwidth one-hop wireless links.

2011

Myoonet pioneers the concept of modular **micro datacentres**. A year later AOL follows this trend with indoor micro datacentres for enterprises.

2012

With IoT in mind, and to handle real-time low-latency applications, Cisco engineers coin the term **fog computing** to imply a distributed cloud infrastructure. Their paper is titled *Fog Computing and Its Role in the Internet of Things*. Fog computing encompasses edge processing plus the network connections that bring data from the edge to the cloud. In 2015, the OpenFog Consortium is founded. In 2019, this merges with Industrial Internet Consortium.

2013

Ha et al. note the emergence of new resource-intensive interaction-intensive applications: face recognition, speech recognition, object and pose identification, mobile augmented reality, and physical simulation and rendering. In their experiments they find that face recognition can be done on an average server one hop away from the edge device. But speech recognition is more intensive and would need cloud processing unless the edge server is upgraded.

Oct
2016

The First IEEE/ACM Symposium on Edge Computing is organized in Washington DC. In May 2017, there's IEEE International Conference on Fog and Edge Computing in Madrid, Spain. In June 2017, there's IEEE International Conference on Edge Computing in Hawaii. These events highlight the growing interest in edge computing.

2025

In 2019, Gartner research shows that only 10% of enterprise data was created and processed outside cloud infrastructure. They predict that by 2025 this will increase to 75%. This underscores the importance of edge computing. In another research, McKinsey estimates the value of edge computing to be $175-$215 billion by 2025.

Edge Computing


# Python Distributions and Packaging

## Summary

The Python Software Foundation does regular releases of Python, which we call the standard distribution. This is a good starting point for beginners. However, there are alternatives distributions. Developers may opt for these alternatives to suit their specific requirements or simplifying the installation process.

Third-party packages are usually distributed via a central repository called the *Python Package Index (PyPI)*. For developers who wish to release their software to the Python community, Python provides many tools to package their software in a standard way and release the same on PyPI. As of February 2018, PyPI has over 130,000 packages.

## Discussion

### What's the standard Python distribution and what does it include?

The Python Software Foundation (PSF) releases the standard distribution of Python periodically at Python.org. This includes Python 3.x and Python 2.x until the latter is retired. Installables are available for Windows and Mac OS X. It's also possible to install Python on Windows using a package manager such as Chocolatey.

For Linux/UNIX, source code is available although most of them come pre-installed with Python that can also be updated with their respective package managers. Python is also available for other OS such as IBM i 7.1+, iOS, VMS, HP-UX, and more; but these lag behind the official release.

The standard implementation of Python is in C and is named *CPython*. This includes implementation of the entire standard library.

### Are there implementations of Python other than CPython?

Yes. Implementation refers to that of the interpreter itself that takes Python code and converts into bytecode. With *CPython*, the interpreter is written in C. This enables easy interfacing of C code with Python code.

Likewise, we have *IronPython* for .NET and *Jython* for JVM. *PyPy* uses a JIT compiler for faster execution. *Stackless Python* augments CPython to support microthreads. *MicroPython* is a slim version of CPython that can run on microcontrollers.

These alternative implementations lag CPython releases. For example, as of February 2018, IronPython is not available for Python 3 and its Python 2 release hasn't been updated since December 2016. Jython is available only for Python 2. PyPy is available for both 2 and 3 though lagging the latest version of CPython.

### What are some alternatives to the standard Python distribution?

There are a number of alternative Python distributions. ActivePython is a commercial distribution that bundles many useful third-party packages along with the standard library. There's also a free Community Edition. Anaconda and Enthought are two distributions customized for scientific computing and analysis. While Anaconda is free, Enthought is commercial. Intel offers an enhancement of Anaconda.

Python(x,y) is based on Qt and Spyder. This may be good place to start for MATLAB/IDL/C/C++/Fortran programmers who wish to switch to Python. Other alternatives include WinPython, Conceptive Python, PyIMSL, and more. Stackless Python is limited by the heap and not the stack for recursion, and it consumes less memory.

The purpose of all of these distributions is to simplify the process of installation and package updates. Commercial distributions also offer support such porting Python to an exotic platform or maintaining Python 2.x for legacy codebases.

What if you wish to use Python without bothering with any installation at all? PythonAnywhere allows you to code and execute on the cloud from a web browser. It's ideal for teachers and students.

### What's the purpose of PyPI?

PyPI serves as a central repository for sharing third-party packages. Developers can go to PyPI and search for packages that they might find useful in their applications. If source code is open (in sites outside PyPI), they can also improve the package. This is often a better approach than developing something from scratch. Note two variants of PyPI:

    + Production site: Main download repository, powered by a codebase called *Warehouse*.
    + Test site: Site where developers can test their packaging before releasing to production site.While it's possible to download a package manually (as source or binary), build and install the same, having packages on PyPI makes the process easier. Packages are easier to find. We can use a standard tool such as `pip` to manage packages and `pip` automatically knows where to look for packages.

### As a Python user, how can I install a package?

The tool *pip* can be used to install a package from PyPI. It replaces an earlier tool named *easy\_install*. If pip itself is missing on your system, use system tools to install the same; or download get-pip.py and execute `python get-pip.py`. While pip is sufficient to install pre-built binaries, to build from source you will also need *setuptools* and *wheel*, which can installed with the command `pip install setuptools wheel`.

If you're working on multiple Python projects, each on a different version of Python, then it's recommended to set up *virtual environments* and install required packages within each environment. Run `pip install virtualenv` and then create/activate it using `virtualenv myproj; source myproj/bin/activate`. It's common to list all project requirements in a file such as requirements.txt and then install all of them by running `pip -r requirements.txt`. This file typically mentions each package name followed by a suitable version in a format specified in PEP 440.

### I don't find pip suitable for my workflow. What are my options?

An easier approach to using `pip` and `virtualenv` separately, or updating requirements.txt independently, is to use the package `pipenv`, which manages and tracks dependencies using `Pipfile` and `Pipfile.lock` files.

If you wish to create custom workflows, `pip-tools` is recommended. Other alternatives include `hatch` and `poetry`.

### What's the definition of a package, packaging and distribution?

*Package* is a bundle of one or more Python modules that can be shared, downloaded and installed. A package is really nothing more than a folder that contains its module files (and optionally sub-packages) and a __init__.py. The process and tools used in making and uploading packages is called *packaging*. The term *distribution* is usually reserved for a Python distribution and must not be confused for package distribution.

### What are Eggs and Wheels?

Both are built distribution formats for packages. Wheels have replaced Eggs as the preferred format. Installer pip supports only Wheels while Eggs can be installed with the older *easy\_install* tool. Wheels don't include compiled Python files. In a built distribution, files and metadata are included. The files are simply moved to correct locations while installing. There's no separate build step before installation.

Python packaging specifies how to express dependencies and integration in the database of Python distributions. Wheels implements these but not Eggs. Wheels have a richer naming convention. Their binary format is standardized. For these reasons, Wheels are preferred.

There are three types of Wheels:

    + **Universal Wheel**: Project is pure Python and natively supports both Python 2 and 3.
    + **Pure Python Wheel**: Project is pure Python and does not natively supports both Python 2 and 3.
    + **Platform Wheel**: Project contains compiled extensions.Since pip will prefer a wheel over source distribution, avoid making universal wheels if your project contains extensions. If you're not providing the source and your project is not pure Python, then multiple wheels will need to be provided for every platform that you wish to support.

### Can you guide me in packaging and distributing my own package?

The following are needed and can be installed using pip:

    + **setuptools**: This extends the older *distutils*. The project specification file is setup.py, which can further be controlled with setup.cfg.
    + **wheel**: This enables creation of wheels.
    + **twine**: It's possible to upload your package to PyPI by executing setup.py (register and upload). However, Twine is preferred. This enables you to upload your package to PyPI securely. Control of where to upload can be specified in file .pypirc in your home folder.
    + **tox**: This enables local installation and testing of your package in different environments. Configuration is usually given in a file named tox.ini. A similar testing can be done on the cloud using a service such as Travis CI.The command `python setup.py clean --all sdist bdist_wheel` will perform the build: sdist for source distribution and bdist\_wheel for creating wheels. This will create two folders build/ and dist/. The former contains temporary files used during the build process. The latter contains the final source archive (\*.tar.gz) and/or binary wheels (\*.whl).

## Milestones

2000

Python adds *disutils* to the standard library as a cross-platform build and distribution system. A replacement to this called *disutils2* was worked on during 2009-2012 but this didn't make it into Python 3.3.

2003

PyPI goes online.

2004

Package installer *easy\_install* is released as part of *setuptools*. Problem with easy\_install is that it isn't easy to uninstall packages.

2008

Package installer *pip* is released as an alternative to *easy\_installer*. This uses *Wheel* rather than *Egg* as the built distribution format. It's possible to use this to save package requirements to a file (`pip freeze > reqs.txt`) and recreate the same environment on another system (`pip install -r reqs.txt`). Wheel itself is published later in 2012-13.

Feb
2011

*The Python Packaging Authority (PyPA)* is created. It's job is to drive standardization, maintain tools and documentation related to Python packaging. It recognizes that the Python packaging and distribution infrastructure needs improvements.

Jul
2012

Anaconda 0.8.0. In February 2018, Anaconda 5.1.0 is released.

Sep
2012

WinPython v0.6 is released. In November 2017, WinPython 3.6.3 is released.

Mar
2014

Python 3.4 is released and it includes *pip* installer by default.

Apr
2018

Kenneth Reitz releases version 11.10.1 of **pipenv** at PyPI. Meanwhile, **Warehouse** replaces legacy code to power PyPI. This means that requests to `pypi.python.org` are redirected to `pypi.org` and they both share the same data store.

Python Distributions and Packaging


# Cross-Validation

## Summary

Cross-validation is a statistical method that estimates how well a trained model will work on unseen data. The model's efficiency is validated by training it on a subset of input data and testing on a different subset. Cross-validation helps in building a generalized model. Due to the iterative nature of modeling, cross-validation is useful for both performance estimation and model selection.

The three steps involved in cross-validation are:

i. Divide the dataset into two parts: one for training and other for testing.

ii. Train the model with the training dataset.

iii. Evaluate the model's performance using the testing set. If the model doesn't perform well with the testing set, check for issues.

If the model performs well on unseen data, it's consistent and can predict with good accuracy for a wide range of input data; this model is stable. Cross-validation helps evaluate the stability of machine learning models.

## Discussion

### What's the need for cross-validation?

In order to build a generalized model that works well for unseen data, cross-validation is needed. This is how it's done:

Split the data into 3 random parts: Training data (65%), Validation data (20%), and Test data (remaining 15%). The model building doesn't involve test data; it is used as 'unseen' data to verify and declare the model accuracy.

Let's say, a model (kNN) is built using Training data and is optimized (optimum k is arrived at) using Validation data. Only 65% of the entire available data is used for model building, which isn't a good sign. With cross-validation, 85% (Training + Validation data) can be used to build the model. Here's how:

k-fold concept is applied now by dividing the Training and Validation data into equal parts, say 5 parts of about 17% each. The model is trained 5 times, each time with 17% Validation data and rest 68% Training data. An aggregation mechanism like average value is applied to arrive at the final optimal model. The final model, built with 85% data, is checked for accuracy with Test data. This ensures a 'generalized' model is built that works well for unseen data too.

### What is the difference between Training Data, Validation Data, and Test Data?

For training and testing the model, the dataset must be split into three distinct parts:

    + **Training Data**: The model is trained to learn the hidden features/patterns of the dataset with the training data. The model evaluates the data repeatedly to learn more about the data's behavior, following which, it adjusts itself to serve the intended purpose. It's basically used to fit the models.
    + **Validation Data**: This is used to validate the model performance during training. It helps tune the model's hyper-parameters and configurations accordingly. The validation data estimates the prediction error for model selection. An over-fitting model is prevented with validation data.
    + **Test Data**: After completion of training, the test data validates that the trained model can make accurate predictions. It's used for assessment of the generalization error of the final chosen model.

### What are the main assumptions behind cross-validation?

The learning dataset that is used to build and evaluate a predictive model is assumed to be a sample from the population of interest. With random sub-sampling methods, the training set and test set are generated from the learning set. A supervised prediction method is only expected to learn how to predict on unseen samples that are drawn from the same distribution as training samples; an evaluation of its performance ought to respect this assumption, as in the case of cross-validation with random partitions.

Random Cross-Validation assumes that a randomly selected set of samples comprising the test set, well represents unseen data. This assumption doesn't hold true when samples are obtained from different experimental conditions.

### Which are the commonly used cross-validation techniques?

    + **Hold-out method**: The data is separated into training and testing sets. The proportion of training data has to be larger than test data. This is used on large datasets, since the model is trained only once and is computationally inexpensive.
    + **Leave one out cross-validation (LOOCV)**: The test data is a single observation from the dataset. Everything else is training data to train the model. In each iteration, a different sample is chosen as test data; the remaining are training data. This is repeated \(n\) times (\(n\) - number of samples). The average of all iterations gives the test set error estimate.
    + **k-fold**: The data is divided into \(k\) sets of near-equal sizes. The first set is the test set; the model is trained on the remaining \(k-1\) sets. Test error rate is calculated after fitting the model to the test data. In the second iteration, the second set is the test set and remaining \(k-1\) sets are the training data. This process continues for all \(k\) sets and error is calculated for each iteration. The mean of these errors gives the test error estimate.

### How do we choose \(k\) value for k-fold cross-validation?

The key-configuration parameter for **k-fold cross-validation** is \(k\) - the number of folds that the given dataset must be split into. Commonly, \(k\)-value is chosen as follows:

    + **Representative**: The \(k\)-value is chosen such that each train/test group of data samples is large enough to be statistically representative of the broader dataset.
    + By performing a sensitivity analysis for different \(k\) values, the optimal value can be determined. This implies evaluating the performance of the same model on the same dataset for different values of \(k\) and see how they compare.
    + To compare classifiers with similar bias, \(k=2\) works best as it has lowest variance. To measure error, \(k=5\) or \(k=10\) are less biased than \(k=2\).
    + Most common values chosen for \(k\) are `3`, `5`, and `10`, with the most popular one being `10`, experimentally found to provide good trade-off of low computational cost and low bias in an estimate of model performance.
    + Typically, low \(k\) values result in a noisy estimate of model performance, while large \(k\) values result in a less noisy estimate.The computation time increases almost exponentially with higher values of \(k\), particularly with large datasets.

### What's nested cross-validation?

**Nested cross-validation** works with a double loop: an outer loop that computes an unbiased estimate of the expected accuracy of the algorithm and an inner loop for hyper-parameter selection. These two loops are independent of each other.

From the example shown, the outer loop is repeated 5 times, generating 5 different test sets. In each iteration, the outer train set is split (into 4 folds here). With 5 outer folds and 4 inner folds (shown in the figure), a total of 20 models are trained.

The outer layer is used to estimate the quality of models trained on the inner layer. The inner layer is used for selecting the best model (including best set of hyper parameters). This way, you're not just assessing the quality of the model, but also the quality of procedure for model selection. For each iteration of the outer loop, one and only one inner model is selected that will be evaluated on the test set for the outer fold. After you vary the outer test set, you'll have 5 estimates that can be averaged to better assess quality of the models.

### What are the use cases of cross-validation?

Cross-validation can be used for comparison of performances of a set of predictive modeling procedures. For example, for optical character recognition, if Support Vector Machine or k-nearest neighbors are considered to predict the true character from an image of a handwritten character, the use of cross-validation can objectively compare these two methods in terms of their respective fractions of misclassified characters. Simply comparing the methods based on their in-sample error rates, one method might appear to perform better than the other.

The use of cross-validation is widespread in medical research. Consider the use of the expression levels of a certain number of proteins, say 15 for example for predicting if a cancer patient will respond to a specific drug. The ideal way would be to determine which subset of the 15 features produce the ideal predictive model. Using cross-validation, you can determine the exact subset that provides the best results.

Data analysts have used cross-validation in medical statistics, with these procedures being useful for meta-analysis.

### What are the challenges with cross-validation?

Cross-validation simply provides one additional mapping from training sets to models. Any mapping of this kind constitutes an inductive bias; hence like any other classification strategy, the performance of cross-validation depends on the environment in which it is applied.

For ideal conditions, it provides optimum output. But with inconsistent data, it may produce drastic result. This is one of the biggest disadvantages of cross-validation as there is no certainty of the type of data in machine learning.

In predictive modeling, data evolves over a period, and it may face the differences between training set and validation sets. For example, if a model has been created to predict stock market values, by training it on stock values of the previous 5 years, the realistic future values for the next 5 years could be drastically different.

While k-fold cross-validation is typically the method of choice to estimate the generalization error for small sample sizes, there exists no universal (valid under all distributions) unbiased estimator of the variance of this technique.

### What are some tips when implementing cross-validation?

Tip #1: While splitting the data into train-test set, a good rule of thumb is to use 25% of the data-set for testing. Generally, the ratio can be 80:20, 75:25, 90:10, etc. It's the machine learning engineer who has to take this decision based on the amount of available data.

Tip #2: The Data Science community has a general rule based on empirical evidence and different researches that suggest 5- and 10-fold cross-validation should be preferred over LOOCV.

Tip #3: In Deep Learning, the normal tendency is to avoid cross-validation due to the cost associated with training \(k\) different model. Instead of doing **k-fold** or other cross-validation techniques, you could use a random subset of your training data as a hold-out for validation purposes.

Tip #4: In case the data is of medical or financial nature, it should be split by person. Avoid having data for one person both in training and the test set, since it could be considered as data leak.

### What software packages help implement cross-validation?

Cross-validation techniques can be implemented using Python and open-source Sci-kit learn. For **k-fold cross-validation**, `sklearn.model_selection.KFold` can be used.

Alternatively, MATLAB supports cross-validation. Some of these cross-validation techniques can be used with the Classification Learner App and the Regression Learner App of MathWorks.

The Keras deep learning library allows you to pass one of two parameters for fit function that performs learning. This includes the validation\_split and validation\_data. The same approach is used in official tutorials of other DL frameworks such as PyTorch and MxNet, where they suggest splitting the data into three parts: training, validation, and testing.

Cross-validation can be easily implemented using \(R\) programming language. The statistical metrics used to evaluate the accuracy of regression models are:

    + Root Mean Squared Error (RMSE) gives the average prediction error made by the model. Decreased RMSE value leads to increase in accuracy of the model.
    + Mean Absolute Error (MAE) gives the absolute difference between actual values and values predicted by the model for the target variable. Less MAE value makes better models.
    + R2 Error reflects the relationship strength between target variable and model. High R2 value gives a better model.

## Milestones

1931

Larson **divides the dataset into two groups**, estimates the regression coefficients from one group and then predicts the criterion scores from the second group. His work is towards a study of the actual amount of shrinkage in the field of psychological testing.Theoretical statisticians previously showed that the coefficient of multiple correlation \(R\), derived for a given dataset, has a deceptively large value. If the equation is applied to another dataset, the yield (except sampling errors) is less than the first. An increase in the number of variables in the regression equation leads to greater shrinkage.

1951

Mosier presents five distinct designs closely related to cross-validation: 1) cross-validation, 2) validity-generalization, 3) validity extension, 4) simultaneous validation, and 5) replication. The purpose is to evaluate the **predictive validity of linear regression** equations used to forecast a performance criterion from scores on a battery of tests. The multiple correlation coefficient in the original sample used to assign values of regression weights gives an optimistic impression of the predictive effectiveness of the regression equation when applied to future observations.

1968

Mosteller and Turkey develop the idea of cross-validation. Their work comes close to what would later be called **k-fold cross-validation**.

1974

For the choice and assessment of statistical predictions, Stone uses a cross-validation criterion. A cross-validatory paradigm with a simple structure is presented. He omits single observations, a method that's later named **Leave-One-Out Cross-Validation (LOOCV)**. While it's assumed that major problems might be encountered in the execution of the cross-validatory paradigm, it's expected that the status of such problems won't be as ambiguous as those associated with the conventional paradigm.

1975

Geisser presents the method of **predictive sample reuse** around the same time as M. Stone's cross-validatory method. Geisser's method uses multiple observational omissions (unlike Stone's LOOCV), yielding a desirable degree of flexibility. He gives more relevance to prediction than parameter estimation for inference since prediction can be adequately assessed in real situations, unlike parameter estimation. He develops a highly flexible and versatile low structure predictivistic approach that serves as a complement to the tightly structured Bayes approach. This method while assuming less, yields less.

1994

Moody and Utans develop a model for rating bonds (for corporate bond rating prediction) as a case study of architecture selection procedures. With limited data availability and lack of complete a priori information, they attempt to select a good neural network architecture to model any specific dataset. Their bond rating study shows that nonlinear networks outperform a linear regression model for a financial application.

2000

Browne reviews many cross-validation methods, considering the original applications in multiple linear regressions first. He assesses structural models for moment matrices. Upon investigating single-sample and two-sample validation indices, it's seen that the optimal number of parameters suggested by both these indices depend on sample size. It's shown how predictive accuracy depends on sample size and the number of predictor variables.

2015

To select the appropriate model from available data, cross-validation is used. Zhang and Yang focus on selecting a modeling procedure in the regression context through cross-validation. They investigate the relationship between cross-validation performance and the ratio of splitting the data, in terms of modeling procedure selection. In comparing the predictive performance of two modeling procedures, they ensure that a large evaluation set accounts for randomness in the prediction assessment. The relative performance for a reduced sample size is made to resemble that for a full sample size.

2020

Li et al. derive a cheap and theoretically guaranteed **auxiliary/augmented validation technique**. It trains models on the given dataset once, making the selection of model quite efficient. It's also suitable for a wide range of learning settings owing to the independence of augmentation and out-of-sample estimation on the learning process. The augmented validation set plays a key role to select the ideal model. Their validation approach is not only computation-efficient, but also effective for validation and is easy for application.

2021

Zhang et al. proposes a **Targeted Cross-Validation (TCV)** approach for model or procedure selection based on a general weighted loss. TCV is shown to be consistent for selection of the best performing candidate and is potentially advantageous over global cross-validation or use of local data for modelling a local region. TCV is used to find a candidate method with the best performance for a local region. The flexible framework allows the best candidate to switch with varying sample sizes, and can be applied to high-dimensional data and complex ML scenarios with dynamic relative performances of modelling procedures.

Cross-Validation


# Pandas DataFrame Operations

## Summary

DataFrame is an essential data structure in Pandas and there are many way to operate on it. Arithmetic, logical and bit-wise operations can be done across one or more frames.

Operations specific to data analysis include:

* **Subsetting**: Access a specific row/column, range of rows/columns, or a specific item.
* **Slicing**: A form of subsetting in which Python slicing syntax `[:]` is used. Rows/columns are numbered from zero. Negative integers imply traversal from the last row/column.
* **Filtering**: Obtain rows that fulfil one or more conditions.
* **Reshaping**: Reorganize data such that the number of rows and columns change.
* **Merging**: A DataFrame is merged with another. This can be a simple concatenation of frames or database-style joins.

**Indexing** is a general term for subsetting, slicing and filtering.

## Discussion

### What are some ways to access values in a Pandas DataFrame?

Consider a simple DataFrame with index values 10-12 and columns A-C. A row or column can be accessed by its index value or column label respectively. Column label is used directly in `[]`: `df['A']`. Index values are used via `.loc[]`: `df.loc[10]`. We can also access a row by its position using `.iloc[]`: `df.iloc[0]`.

In our example, there are many ways to get the last item, a scalar value:

    + Row first: `df.iloc[2]['C']`, `df.loc[12]['C']`, `df.loc[12, 'C']`
    + Column first: `df['C'][12]`, `df['C'].loc[12]`, `df['C'].iloc[2]`To access multiple rows use `df.loc[[10,11]]` or `df.iloc[0:2]`. For multiple columns, use `df[['A','B']]`. To get last two columns of last two rows, we can write `df.loc[[11,12], ['B','C']]`, `df.iloc[1:][['B','C']]`, `df[['B','C']].iloc[1:]` or `df[['B','C']].loc[[11,12]]`. Integer lists `df.iloc[[1,2],[1,2]]` or slicing `df.iloc[1:,1:]` or `df.iloc[-2:, -2:]` can be used.

Positional and label-based indexing can be combined: `df.loc[df.index[[0, 2]], 'A']`.

**Callable** can be used for indexing. Function takes a Series or DataFrame and must return a valid index. For example, `df.loc[:, lambda df: ['A','B']]` or `df[lambda df: df.columns[0]]`.

### What is multi-indexing on a Pandas DataFrame?

Hierarchical indexing or MultiIndex allows us to work with higher dimensional datasets using 2-D DataFrame. A MultiIndex could be seen as an array of unique tuples. In fact, it can be created from list of arrays, array of tuples, a DataFrame or from a product of iterables.

Referring to the figure, rows can be accessed using `df.loc['a']` or `df.loc['a',:]`. For multi-level indexing, use `df.loc[('a','one')]`, `df.loc[('a','one'),:]` or `df.loc[(slice(None),'one'), :]`. Likewise for columns, we have `df['A']`, `df.loc[:,'A']` or `df.loc[:, (slice(None), 'x')]`. Multi-indexing across rows and columns can be done with `df['A']['x']`, `df['A','x']` or `df.loc[:,('A','x')]`. To get specific items, use `df.loc[('a','one'),('B','x')]` or `df.loc[('a','one'),'B']`.

Partial slicing is done using `df.loc[('a', 'one'):('b', 'two')]` (result has four rows). Reindexing is done with `df.loc[[('a', 'one'), ('b', 'two')]]` (result has two rows).

With two levels on index, `df.swaplevel()` will change the order of the levels.

Note that `df.loc[('a','one'),:]` is valid but `df.loc[['a','one'],:]` is not. In this context, lists and tuples have different semantics:

> For indexing, a tuple is a multi-level key whereas a list specifies several keys.

### How can I filter a Pandas DataFrame?

Boolean arrays of same size as index can be used for filtering, that is, select rows indexed with `True`. For example, `df.loc[~df.index.isin([11])]` ignores a specific index; `df[df['A']>0]` selects only positive values of column A; `df[(df['A'] > 2) & (df['B'] < 3)]` shows a complex expression where use of parenthesis is important.

The method `query()` can simplify the syntax for complex expressions. Thus, for columns a-d, we can write `df.query('a not in b')` instead of `df[~df['a'].isin(df['b'])]`; `df.query('a in b and c < d')` instead of `df[df['b'].isin(df['a']) & (df['c'] < df['d'])]`.

Whereas `df[df.A>0]` will return rows that match the condition, `df.where(df.A>0)` will return a DataFrame of same shape as the original. Where the condition is False, NaN will be returned. Optional second argument to `where()` is a replacement value. The inverse boolean operation of `where()` is `mask()`, that is, it returns rows that don't match the condition.

To select only rows with null values, use `df[df.isnull().any(axis=1)]`. Use `df[~df.isnull().any(axis=1)]` to do the reverse.

For MultiIndex DataFrame, `isin()` method can be used. For example, `df.loc[df.index.isin([('a','one'), ('b','two')])]` or `df.loc[df.index.isin(['a','c'], level=0)]`.

### How do I do GroupBy operations on a Pandas DataFrame?

GroupBy uses a **split-apply-combine** workflow. Data is split using index or column values. A function is applied to each group independently. The results are combined into a single data structure. In the apply step, data can be **aggregated** (sum, mean, min, etc), **transformed** (normalize data, fill missing values, etc) or **filtered** (discard small groups, filter data based on group mean, etc).

To group by single column 'A' or index 'a', use `df.groupby('A')` or `df.groupby('a')`. Index grouping can also be done by position: `df.groupby(level=0)`. To group by multiple columns or indices, use `df.groupby(['a', 'A'])`. To group by all levels except 'b', use `df.groupby(level=df.index.names.difference('b'))`. If a column and index share the same name 'b', use `df.groupby([pd.Grouper(level='b'), 'A'])` (index 'b') or `df.groupby([pd.Grouper('b'), 'A'])` (column 'b').

To aggregate, simply call the function after grouping: `df.groupby('A').sum()`; or `df.groupby('A').agg([np.sum, np.mean])` to aggregate in many ways. Each column can be aggregated differently: `df.groupby('A').agg({'C': np.sum, 'D': lambda x: np.std(x, ddof=1)})`.

Methods `transform()`, `filter()`, `resample()`, `expanding()`, `rolling()` and `emw()` can be used after `df.groupby()`. For grouping datetime types, `pd.Grouper(key, level, freq, axis, sort)` is useful.

### How can I reshape a Pandas DataFrame?

When data is in "record" form, `pivot()` spreads rows into columns. Method `melt()` does the opposite. Methods `stack()` and `unstack()` are designed for MultiIndex objects and they're closely related to pivot. They can be applied on Series or DataFrame. Here are a few examples:

    + `df.pivot(index='foo', columns='bar', values='baz')`: Column 'foo' becomes the index, 'bar' values become new columns and values of 'baz' becomes values of the new DataFrame. A more generalized API is `df.pivot_table()` that allows for duplicate values of an index/column pair.
    + `df.melt(id_vars=['A','B'])`: Two columns are retained and other columns are spread into rows. Two new columns named 'variable' (with old column names as values) and 'value' are introduced. Original index can be retained but values will be duplicated.
    + `df.stack()`: Columns becomes part of a new inner-most index level. If DataFrame has hierarchical column labels, level can be specified as argument.
    + `df.unstack('second')` or `df.unstack(1)`: Values of index 'second' are spread into new columns. If no argument is supplied, the inner-most index is spread.These methods when combined with GroupBy and aggregations can be expressive. Examples: `df.stack().mean(1).unstack()`, `df.groupby(level=1, axis=1).mean()`, `df.stack().groupby(level=1).mean()` and `df.mean().unstack(0)`.

### What are the ways to combine two datasets in Pandas?

Perhaps the simplest is to understand is concatenating two or more frames that share the same column labels. Rows of one are concatenated to the other: `pd.concat([df1, df2])`. To concatenate column-wise, use `pd.concat([df1, df2], axis=1)`. If the index values don't match, we re-index before concatenating: `pd.concat([df1, df2.reindex(df1.index)], axis=1)`. Argument `join` takes values `outer` (default) and `inner`. An outermost level of multi-index can be created with `pd.concat([df1, df2], keys=['A','B'])`.

Database-style joins are possible with `df.join(df1)` in which the argument `how` takes four values: `left`, `right`, `outer`, `inner`. By default, joins are based on index. With `on` argument, we can choose to join column or index level of `df` with index of `df1`. Joins based on MultiIndex are supported.

Method `df.merge()` is more flexible than join since index levels or columns can be used. If merging on only columns, indices are ignored. Unlike join, cross merge (a cartesian product of both frames) is possible. Methods `pd.merge()`, `pd.merge_ordered()` and `pd.merge_asof()` are related.

Examples of merge, join and concatenate are available in the user guide.

### Could you share some best practices or tips for using Pandas DataFrame?

Here are some tips and pointers relevant to DataFrame operations:

    + While `df['A']` is used to access a column, `df.A` is a handy short form. This is called **attribute access**. Column name must be a valid Python identifier and it shouldn't conflict with existing method/attribute. Thus, `df.1` and `df.first name` (has a whitespace) are not allowed. `df.min` and `df.index` conflict with existing method/attribute.
    + Syntax `df['A']` results in a Series data structure whereas `df[['A']]` returns another DataFrame.
    + On a MultiIndex DataFrame, consider `dfmi['one']['second']` versus `dfmi.loc[:, ('one','second')]`. Both give the same result but the latter is faster. The former form is actually two operations: first `dfmi['one']` returns a DataFrame, then `['second']` is **chained** to it.
    + To create an explicit copy, write `df.copy()`. The statements `df1 = df; df1[0:3] = 0` doesn't create a copy and modifies the original DataFrame.
    + Remember that `loc[]` is for label-based indexing and `iloc[]` is for integer-based indexing. Integers may be used for the former but they're treated as labels.
    + When filtering by datetime values, their attributes can be used, such as, `df.loc[(df['Date'].dt.month==3) | (df['Date'].dt.month==11)]`.

## Milestones

Jul
2019

In Pandas 0.25.0, **Named Aggregation** is introduced. This uses a named tuple `pd.NamedAgg()` with two fields `column` and `aggfunc`. The latter can be set to a callable or a string alias. For a more compact code, plain tuples can be used instead. This is now the recommended replacement for using "dict-of-dicts", which was deprecated in version 0.20.0 (May 2017).

Jul
2020

In Pandas 1.1.0, `ignore_index` argument is added to `melt()` with default value True. If set to False, index is retained although duplicate values will be present in the result. For GroupBy, it's now possible to retain `NA` values in group keys. For example, `df.groupby(by=["b"], dropna=False).sum()` will retain `NA` values in column 'b'.

Dec
2020

In Pandas 1.2.0, **exponentially weighted window** operations are supported. Thus, we can write for example `df.groupby('A').ewm(com=1.0).mean()`.

Pandas DataFrame Operations