# IoT Alliances and Consortiums

## Summary

Alliances and consortiums are shaping the IoT landscape to reach a level of standardization, maturity and acceptance. They allow a reach which no individual organization can achieve. They give confidence to the market that there's larger commitment to the goals of the alliance or consortium.

Alliances and consortiums also enable mass adoption of technology. Organizations like IEEE leverage their brand and mass reach to promote generic frameworks which cut across competing technologies. They also help to consolidate an otherwise fragmented marketspace. This helps businesses reach economies of scale to make the price of solutions affordable to the common man or enable the industry to deploy systems in a large scale.

While many alliances/consortiums existed in 2014, later years saw mergers and collaborations, leading to less market fragmentation.

## Discussion

### What's the role of IoT alliances and consortiums?

Alliances and consortiums provide for the following:

    + Help to promote standards and product certifications in a particular area (Examples: RFID Consortium, NFC Forum, Wi-Fi Alliance, Zigbee Alliance, LoRa Alliance )
    + Facilitate joint R&D efforts, influence direction of the industry, promote best practices, and provide testbeds (Example: IIC)
    + Help new entrants or startups to start quickly by providing required licenses, ecosystem or access to standards so that they can concentrate on building their differentiatorsStandards organizations can sometimes be guilty of catering only to those vendors who are on their committees. Alliances and consortiums, being non-profit, help drive requirements for standardization, and subsequent testing and certification in a vendor neutral manner. But when they also create the standards and fail to make them public, the entire industry suffers.

### What should we look for when joining a particular IoT alliance or consortium?

We can evaluate based on the following:

    + **Openness**: What are their policies on intellectual property? Are there royalties involved in implementing their standards? Is it a closed group that requires paid membership?
    + **Availability**: Is their work (standards, white papers, guidelines) already available or is it still a work in progress?
    + **Adoption**: Are people in industry already adopting and using their work?Back in 2014, Ian Skerrett evaluated a number of IoT alliances and consortiums based on the above metrics and published a descriptive analysis, which is a good read.

In 2018, an online survey of 502 participants showed that Eclipse IoT, Apache Foundation, W3C, and IEEE are among the top consortiums that developers considered important in the IoT space. However, we should note that this survey was conducted by Eclipse IoT, AGILE IoT, IEEE and OMA. The survey targeted only members of their communities.

### What are the major IoT alliances and consortiums?

Among the many IoT alliances and consortiums, some important ones are the following:

    + Industrial Internet Consortium (IIC)
    + IEEE
    + Open Connectivity Foundation (OCF)
    + Thread Group
    + LoRa AllianceIn one study from early 2018, the following important alliances were listed: OMA, Genivi, IIC, IoT Consortium, TrustedIoTAlliance, OneM2M, AIOTI, and OCF.

Some alliances and consortiums are vertically focused on a particular industry. For example, Thread Group is focused on connected homes; Apple's HealthKit is about health and fitness; EnOcean Alliance is about building automation; Open Automotive Alliance is about connected cars; HART Communication Foundation looks at the industrial IoT space.

### How does Industrial Internet Consortium (IIC) contribute to IoT?

The IIC was founded by AT&T, Cisco, General Electric, IBM, and Intel in March 2014. Though it's parent company is the Object Management Group, IIC itself is not a standardization body. It had around 155 members as of August 2020.

It was formed to bring together industry players — from multinational corporations, small and large technology innovators to academia and governments — to accelerate the development, adoption and widespread use of Industrial Internet technologies. IIC members work to create an ecosystem for insight and thought leadership, interoperability and security via reference architectures, security frameworks and open standards, and real world implementations to vet technologies and drive innovations (called testbeds).

As per Compass Intelligence, IIC won the *top IoT organization influencer of the year* award for 2018.

### What role does IEEE play in IoT?

The IEEE IoT Initiative was launched in 2014. It aims to help engineering and technology professionals learn, share knowledge and collaborate around IoT.

IEEE has a variety of programs that can help solution providers that are trying to understand IoT, including the IEEE IoT Technical Community, which is made up of members involved in research, application and implementation of IoT. IEEE also heads the IoT Scenarios Program, which is an interactive platform to demonstrate use cases, business models and service descriptions.

In March 2020, IEEE published an architectural framework for IoT, **IEEE 2413-2019**. IEEE also publishes a number of peer-reviewed IoT papers across its different publications, including *IEEE Internet of Things Journal (IoT-J)* that's dedicated to IoT. IEEE also organizes many conferences and events on IoT around the world, in particular the IEEE World Forum on Internet of Things (WF-IoT).

### What are the major developments in the Open Connectivity Foundation (OCF)?

Open Interconnect Consortium (OIC) in 2016 changed its name to Open Connectivity Foundation (OCF) when Microsoft and Qualcomm joined this group. **The AllSeen Alliance** was developing the **AllJoyn** standard, which was first created by Qualcomm. In October 2016, the AllSeen Alliance and OCF merged together to become the new OCF. The merged group continues working on the open-source IoTivity (OCF) and AllJoyn (AllSeen) projects under the auspices of the Linux Foundation, eventually merging them into a single **IoTivity** standard.

Developing interoperability standards is one of the key aims of the OCF, a single IoTivity implementation that offers the best of the previous two standards. Current devices running on either AllJoyn or IoTivity are expected to be interoperable and backward-compatible with the unified IoTivity standard. There are millions of AllJoyn-enabled products on the market.

### What's the role of Thread Group among IoT alliances?

Thread is mesh network built on open standards and IPv6/6LoWPAN protocols to simply and securely connect products around the house. 6LoWPAN is a power-efficient personal area network protocol with underlying standards of IPv6 and IEEE 802.15.4. Thread is also capable of running for a long period of time from a long-lasting battery.

Members of the Thread Group include ARM, Nordic, NXP, Nest, Qualcomm, Tridonic, Texas Instruments, Silicon Labs, and many more. Thread Group members get practical resources to help grow the world of connected devices by joining a global ecosystem of technology innovators. The group also offers a certification program based on Thread 1.1 specification. Thread can run multiple applications layers including Dotdot, LWM2M, OCF or Weave.

Thread Group liaises with other IoT alliances and consortiums like Open Connectivity Foundation and Zigbee Alliance and tries to provide a one-stop shop for the home IoT.

### What is special about Lora Alliance group?

The LoRa Alliance is a fast growing technology alliance. It's a non-profit association of more than 500 member companies, committed to enabling large scale deployment of Low Power Wide Area Networks (LPWAN) IoT through the development and promotion of the **LoRaWAN** open standard.

Members benefit from a vibrant ecosystem of active contributors offering solutions, products & services, which create new and sustainable business opportunities.

Through standardisation and the accredited certification scheme the LoRa Alliance delivers the interoperability needed for LPWA networks to scale, making LoRaWAN the premier solution for global LPWAN deployments. The speciality of Lora Alliance is that it operates primarily on the WAN side of the IoT space.

### What are the other alliances and consortiums to keep a watch on?

In 2015, some companies got together to address security aspects of IoT. They defined the Open Trust Protocol (OTrP). It's overseen by the **OTrP Alliance** while collaborating with IETF and Global Platform. The key insight is that system-level root of trust is necessary. Security must be addressed at all levels, from firmware to applications.

Trusted IoT Alliance applied blockchain technology to IoT. In January 2020, this alliance merged into IIC.

Founded in 2008, IPSO Alliance promoted the Internet Protocol for "smart object" communications, advocating for IP networked devices in health care, industrial and energy applications. In 2018, it merged with Open Mobile Alliance (OMA) to form OMA SpecWorks.

## Milestones

Dec
2013

**AllSeen Alliance** is formed. Premier level members include Haier, LG Electronics, Panasonic, Qualcomm, Sharp, Silicon Image and TP-LINK.

2014

This is the year when alliances and consortiums start appearing. The Thread Group, Open Interconnect Consortium (OIC) and Industrial Internet Consortium (IIC) are all started in 2014. IEEE IoT Initiative is also launched. Thread Group opens for membership in October.

Mar
2015

Open, non-profit **LoRa Alliance** becomes operational. It's mission is to promote global adoption of the LoRaWAN standard. By 2018, it has 500+ members.

Nov
2015

To accelerate distributed computing, networking and storage for IoT, ARM, Cisco, Dell, Intel, Microsoft and Princeton University Edge Laboratory establish **OpenFog Consortium**. Focus is to build frameworks and architectures for end-to-end scenarios with capabilities pushed to network edges.

2016

In February, Open Interconnect Consortium (OIC) is renamed into **Open Connectivity Foundation (OCF)**. In October, AllSeen Alliance merges into OCF. OCF will now sponsor both IoTivity and AllJoyn open source projects at The Linux Foundation.

Oct
2017

Industrial Internet Consortium (IIC) and EdgeX Foundry merge with a mandate "to align efforts to maximize interoperability, portability, security and privacy for the industrial Internet". EdgeX Foundry, a Linux-backed, open-source project has been "building a common interoperability framework to facilitate an ecosystem for IoT edge computing".

Dec
2019

Apple, Google, Amazon and Zigbee come together to form a new working group under the Zigbee Alliance called **Connected Home over IP (CHIP)**. It's based on IPv6 and meant to work with any PHY or MAC layer for any application or product. However, CHIP leaves out OCF from the discussions, which might create interoperability issues.

IoT Alliances and Consortiums


# Go kit

## Summary

Building an app based on microservices architecture is not trivial. We would need to implement many specialized parts such as RPC safety, system observability, infrastructure integration, program design, and more. Go language's standard library itself isn't adequate to implement these easily. This is where Go kit becomes useful.

Using Go kit means that developers can focus on their application rather than grapple with the challenges of building a distributed system. Go kit is positioned as a **toolkit for microservices**. It's not a framework. It doesn't force developers to use everything it provides. It's also lightly opinionated about what components or architecture that developers should use. Go kit is really a library. Developers can choose only what they want.

Go kit is open source and follows the MIT License.

## Discussion

### What's the relevance of Go kit in the context of microservices?

When an app is composed of dozens or even hundreds of microservices, there's lot more to be done than just defining and building those services. We need consistent logging across services. Metrics have to be collected both at infrastructure layer and application layer. When a request comes in, we may need to trace how this request moves across multiple services. For robustness, we need rate limiting and circuit breaking. Each service will likely be running multiple instances that are dynamically created and destroyed. Service discovery and load balancing are therefore essential.

While Go has a much lower resource footprint compared to Ruby, Scala, Clojure or Node, back in 2014, it didn't have anything for a microservices architecture. Go kit's inventor, Peter Bourgon at SoundCloud, found that teams were choosing Scala and JVM because of better support for "microservices plumbing". In fact, SoundCloud adopted Twitter's Finagle. Go didn't have an equivalent to this. Thus was born the idea for Go kit. It's really,

> a set of standards, best practices, and usable components for doing microservices in Go

### Which are the key components of Go kit?

Go kit microservices have three layers: Transport, Endpoint, Service. Requests come in at transport layer, move through endpoint layer and arrive at service layer. Responses take the reverse route.

Go kit supports a number of transports. Thus, legacy transports and modern transports can be supported within the same service. Some supported transports include NATS, gRPC, Thrift, HTTP, AMQP and AWS Lambda.

The primary messaging in Go kit is RPC. An endpoint is an RPC method that maps to a service method. A single endpoint can be exposed via multiple transports.

Service layer is where all the business logic resides. This is the concern of application developers who need not know anything about endpoints or transports, which are taken care of by Go kit. A service is modelled as an interface and its implementation contains the business logic.

Due to this separation of concerns, Go kit encourages you to adopt SOLID design principles and clean architecture. In fact, Go kit can be used to build elegant monoliths as well.

### What's the concept of middleware in Go kit?

Go kit takes the idea of "separation of concerns" further with the use of middleware. Rather than bloat core implementations with lots of functionality, additional functionality is provided through middleware using the **decorator pattern**. Go kit provides middleware for endpoint layer while application developers can write middleware for the service layer. We can also chain multiple middleware.

Here's a sample of middleware:

    + Endpoint: Load balancing, safety, operational metrics, circuit breaking, rate limiting.
    + Service: Includes anything that needs knowledge of business domain including app-level logging, analytics, and instrumentation.As can be expected of the decorator pattern, an endpoint middleware accepts an endpoint and returns an endpoint. A service middleware accepts a service and returns a service.

Go kit doesn't force you to use all the middleware the come with it. For example, if you're using Istio that already comes with circuit breaking and rate limiting, then you ignore their equivalent Go kit middleware.

### Which are the packages available in Go kit?

Without being exhaustive, here are some Go kit packages:

    + **Authentication**: Basic, casbin, JWT.
    + **Circuit Breaker**: Hystrix, GoBreaker, and HandyBreaker.
    + **Logging**: Provide an interface for structured logging. Recognizes that logs are data. They need context and semantics to be useful for analysis. Supported formats are logfmt and JSON.
    + **Metrics**: Provides uniform interfaces for service instrumentation. Comes with counters, gauges, and histograms. Has adapters for CloudWatch, Prometheus, Graphite, DogStatsD, StatsD, expvar, and more.
    + **Rate Limit**: Uses Go's token bucket implementation.
    + **Service Discovery**: Consul, DNS SRV, etcd, Eureka, ZooKeeper, and more.
    + **Tracing**: OpenCensus, OpenTracing, and Zipkin.
    + **Transport**: AMQP, AWS Lambda, gRPC, HTTP, NATS, Thrift.

### Could you share some best practices for using Go kit?

Don't change your platform or infrastructure to suit Go kit. Remember that Go kit is only lightly opinionated. It will integrate well with whatever platform or infrastructure you're already using.

Go kit adopts domain-driven design and declarative composition. It uses interfaces as contracts. You should too when implementing your services.

Each middleware should have a single responsibility.

Other frameworks might use dependency injection. In Go kit, the preferred approach is to wire up the entire component graph in `func main`. This forces you to pass them explicitly to components via constructors. Avoid using global state.

Errors can be encoded in two ways: as an error field in response struct, or an error return value. For services, prefer the former method. For endpoints, prefer the latter since these are recognized by middleware such as circuit breaker.

Individual services should not collect or aggregate logs. This is the job of the platform. Your services should write to stdout/stderr.

Access to databases will likely be in a service implementation. Better still, use an interface to model persistence operations while an implementation wraps the database handle.

### What are some criticisms or limitations of Go kit?

It's been said that Go kit is **too verbose**. Adding an API to a microservice involves a lot of boilerplate code. While Go kit nicely separates the business logic, endpoint and transport layers, this abstraction comes at a cost. The code is **harder to understand**.

However, Developers who don't wish to write boilerplate code can make use of code generators. Two generators are *kitgen* and Kujtim Hoxha's *GoKit CLI*.

Inspired by Go kit, the engineering team at Grab created **Grab-Kit**. They felt that Go kit still requires a lot of manual work, something Grab-Kit aims for solve. The idea is to codify best practices. Grab-Kit standardizes APIs, SDKs, error handling, etc. It uses a consistent middleware stack across services including automatic profiling and execution traces.

## Milestones

Feb
2015

Peter Bourgon, while working at SoundCloud, makes the first commit of Go kit code. However, the codebase is not versioned at this point. First tagged version v0.1.0 happens only in June 2016.

Jan
2018

Based on GitHub stars, Go kit is seen as the most popular toolkit, followed by Micro, Gizmo and Kite.

Mar
2018

Version 0.7.0 is released. This includes **kitgen** code generator, **JSON-RPC** transport, and **etcdv3** support in service discovery.

Nov
2018

Version 0.8.0 is released. This includes **NATS** and **AMPQ** transports.

Jun
2019

Version 0.9.0 is released. This supports **AWS Lambda** as a transport.

Go kit


# Time Series Analysis

## Summary

Time series data is an ordered sequence of observations of well-defined data items at regular time intervals. Examples include daily exchange rates, bank interest rates, monthly sales, heights of ocean tides, or humidity. Time Series Analysis (TSA) finds hidden patterns and obtains useful insights from time series data. TSA is useful in predicting future values or detecting anomalies across a variety of application areas.

Historically, TSA was divided into time domain versus frequency domain approaches. The time domain approach used autocorrelation function whereas the frequency domain approach used Fourier transform of the autocorrelation function. Likewise, there are also Bayesian and non-Bayesian approaches. Today these differences are of less importance. Analysts use whatever suits the problem.

While most methods of TSA are from classical statistics, since the 1990s artificial neural networks have been used. However, these can excel only when sufficient data is available.

## Discussion

### What are the main objectives of time series analysis?

TSA has the following objectives:

    + **Describe**: Describe the important features of the time series data. The first step is to plot the data to look for the possible presence of trends, seasonal variations, outliers and turning points.
    + **Model**: Investigate and find out the generating process of the time series.
    + **Predict**: Forecast future values of an observed time series. Applications are in predicting stock prices or product sales.

### What are some applications of time series analysis?

TSA used in numerous practical fields such as business, economics, finance, science, or engineering. Some typical use cases are Economic Forecasting, Sales Forecasting, Budgetary Analysis, Stock Market Analysis, Yield Projections, Process and Quality Control, Inventory Studies, Workload Projections, Utility Studies, and Census Analysis.

In TSA, we collect and study past observations of a time series data. We then develop an appropriate model that describes the inherent structure of the series. This model is then used to generate future values for the series, that is, to make forecasts. Time series analysis can be termed as the act of predicting the future by understanding the past.

Forecasting is a common need in business and economics. Besides forecasting, TSA is also useful to see how a single event affects the time series. TSA can also help towards quality control by pointing out data points that are deviating too much from the norm. Control and monitoring applications of TSA are more common in science and industry.

### What are the main components of time series data?

There are many factors that result in variations in time series data. The effects of these factors are studied by following four major components:

    + **Trends**: A trend exists when there is a long-term increase or decrease in the data. It doesn't have to be linear. Sometimes we will refer to a trend as "changing direction" when it goes from an increasing trend to a decreasing trend.
    + **Seasonal**: A seasonal pattern exists when a series is influenced by seasonal factors (quarterly, monthly, half-yearly). Seasonality is always of a fixed and known period.
    + **Cyclic Variation**: A cyclic pattern exists when data exhibits rises and falls that are not of fixed period. The duration of these cycles is more than a year. For example, stock prices cycle between periods of high and low values but there's no set amount of time between those fluctuations.
    + **Irregular**: The variation of observations in a time series which is unusual or unexpected. It's also termed as a *Random Variation* and is usually unpredictable. Floods, fires, revolutions, epidemics, and strikes are some examples.

### What is a stationary series and how important is it?

Given a series of data points, if the mean and variance of all the data points remain constant with time, then we call it a stationary series. If these vary with time, we call it a non-stationary series.

Most prices (such as stock prices or price of Bitcoins) are not stationary. They are either drifting upward or downward. Non-stationary data are unpredictable and cannot be modeled or forecasted. The results obtained by using non-stationary time series may be spurious in that they may indicate a relationship between two variables where one doesn't exist. In order to receive consistent, reliable results, non-stationary data needs to be transformed into stationary data.

### Given a non-stationary series, how can I make it stationary?

The two most common ways to make a non-stationary time series curve stationary are:

    + **Differencing**: In order to make a series stationary, we take a difference between the data points. Suppose the original time series is \(X\_1, X\_2, X\_3, \ldots X\_n\). Series with difference of degree 1 becomes \(X\_2-X\_1, X\_3-X\_2, X\_4-X\_3, \ldots, X\_n-X\_{n-1}\). If this transformation is done only once to a series, we say that the data has been **first differenced**. This process essentially eliminates the trend if the series is growing at a fairly constant rate. If it's growing at an increasing/decreasing rate, we can apply the same procedure and difference the data again. The data would then be **second differenced**.
    + **Transformation**: If the series can't be made stationary, we can try transforming the variables. Log transform is probably the most commonly used transformation for a diverging time series. However, it's normally suggested to use transformation only when differencing is not working.

### What are the different models used in Time Series Analysis?

Some commonly used models for TSA are:

    + **Auto-Regressive (AR)**: A regression model, such as linear regression, models an output value based on a linear combination of input values. \(y = \beta\_0 + \beta\_1x + \epsilon\). In TSA, input variables are observations from previous time steps, called *lag variables*. For p=2, where p is the order of the AR model, *AR(p)* is \( x\_t = \beta\_0 + \beta\_1 x\_{t-1} + \beta\_2 x\_{t-2}\)
    + **Moving Average (MA)**: This uses past forecast errors in a regression-like model. For q=2, *MA(q)* is \(x\_t = \theta\_0 + \theta\_1 \epsilon\_{t-1} + \theta\_2 \epsilon\_{t-2}\)
    + **Auto-Regressive Moving Average (ARMA)**: This combines both AR and MA models. *ARMA(p,q)* is \(\begin{align}x\_t = &\beta\_0 + \beta\_1 x\_{t-1} + \beta\_2 x\_{t-2} + \ldots + \beta\_p x\_{t-p} + \\ &\theta\_0 + \theta\_1 \epsilon\_{t-1} + \theta\_2 \epsilon\_{t-2} + \ldots + \theta\_q \epsilon\_{t-q} \end{align}\)
    + **Auto-Regressive Integrated Moving Average (ARIMA)**: The above models can't handle non-stationary data. *ARIMA(p,d,q)* handles the conversion of non-stationary data to stationary: *I* refers to the use of differencing, *p* is lag order, *d* is degree of differencing, *q* is averaging window size.

### What are autocorrelations in the context of time series analysis?

Autocorrelations are numerical values that indicate how a data series is related to itself over time. It measures how strongly data values separated by a specified number of periods (called the **lag**) are correlated to each other. **Auto-Correlation Function (ACF)** defines autocorrelation for a specific lag.

Autocorrelations may range from +1 to -1. A value close to +1 indicates a high positive correlation while a value close to -1 implies a high negative correlation. These measures are most often evaluated through graphical plots called **correlogram**. A correlogram plots the auto-correlation values against lag. Such a plot helps us choose the order parameters for ARIMA model.

In addition to suggesting the order of differencing, ACF plots can help in determining the order of MA(q) models. **Partial Auto-Correlation Function (PACF)** correlates a variable with its lags, conditioned on the values in between. PACF plots are useful when determining the order of AR(p) models.

### How do I build a time series model?

ARMA or ARIMA are standard statistical models for time series forecast and analysis. Along with its development, the authors Box and Jenkins also suggested a process for identifying, estimating, and checking models. This process is now referred to as the **Box-Jenkins (BJ) Method**. It's an iterative approach that consists of the following three steps:

    + **Identification**: Involves determining the order (p, d, q) of the model in order to capture the salient dynamic features of the data. This mainly leads to use graphical procedures such as time series plot, ACF, PACF, etc.
    + **Estimation**: The estimation procedure involves using the model with p, d and q orders to fit the actual time series and minimize the loss or error term.
    + **Diagnostic checking**: Evaluate the fitted model in the context of the available data and check for areas where the model may be improved.

### How do we handle random variations in data?

Whenever we collect data over some period of time there's some form of random variations. Smoothing is the technique to reduce the effect of such variations and thereby bring out trends and cyclic components. There are two distinct groups of smoothing methods:

**Averaging Methods**

    + Moving Average: we forecast the next value by averaging 'p' previous values.
    + Weighted Average: we assign weights to each of the previous observations and then take the average. The sum of all the weights should be equal to 1.**Exponential Smoothing Methods**: It assigns exponentially decreasing weights as the observation get older. In other words, recent observations are given relatively more weight in forecasting than the older observations. There are several varieties of this method:

    + Simple exponential smoothing for series with no trend and seasonality: the basic formula for simple exponential smoothing is \(S\_{t+1} = \alpha y\_t + (1-\alpha)S\_t, 0 < \alpha <=1, t > 0\)
    + Double exponential smoothing for series with a trend and no seasonality.
    + Triple exponential smoothing for series with both trend and seasonality.

## Milestones

1662

John Graunt publishes a book titled *Natural and Political Observations … Made upon the Bills of Mortality*. The book contains the number of births and deaths recorded weekly for many years starting from early 17th century. It also includes the probability that a person dies by a certain age. Such tables of life expectancy later become known as **actuarial tables**. This is one of the earliest examples of time series style of thinking applied to medicine.

1861

Robert FitzRoy coins the term "weather forecast". Such forecasts start appearing in *The Times* from August 1861. Atmospheric data collected from many parts of England are relayed by telegraph to London, where FitzRoy analyzes the data (along with past data) to make forecasts. His forecasts forewarn sailors of impending storms and directly contribute to reducing shipwrecks.

1887

Augustus D. Waller, a doctor by profession, records what is possibly the first electrocardiogram (ECG). As practical ECG machines arrive in the early 20th century, TSA is applied to estimate the risk of cardiac arrests. In the 1920s, electroencephalogram (EEG) is introduced to measure brain activity. This gives doctors more opportunities to apply TSA.

1927

Yule applies **harmonic analysis** and **regression** to determine the periodicity of sunspots. He separates periodicity from superposed fluctuations and disturbances. Yule's work starts the use of statistics in TSA. In general, application of autoregressive models is due to Yule and Walker in the 1920s and 1930s.

1960

Muth establishes a statistical foundation for **Simple Exponential Smoothing (SES)** by showing that it's optimal for a random walk plus noise. Further advances to exponential smoothing happen in 1985: Gardner gives a comprehensive review of the topic; Snyder links SES to innovation state space model, where *innovation* refers to the forecast error.

1969

Bates and Granger show that by **combining forecasts** from two independent models, we can achieve a lower mean squared error. They also propose how to derive the weights in which the two original forecasts are to be combined. The same year, David Reid publishes his PhD thesis that's probably the first non-trivial study of time series forecast accuracy.

1970

Box and Jenkins publish a book titled *Time Series Analysis: Forecasting and Control*. This work popularizes the ARIMA model with an iterative modelling procedure. Once a suitable model is built, forecasts are conditional expectations of the model using mean squared error (MSE) criterion. In time, this model is called the **Box-Jenkins Model**.

1978

Through the 1970s, many statisticians continue to believe that there's a single model waiting to be discovered that can best fit any given time series data. However, empirical evidence show that an ensemble of models give better results. These debates cause George Box to famously remark,

> All models are wrong but some are useful

1979

Makridakis and Hibon use 111 time series data and compare the performance of many forecasting methods. Their results claim that a combination of simpler methods can outperform a sophisticated method. This causes a stir within the research community. To prove the point, Makridakis and Hibon organize a competition, called **M-Competition** starting from 1982: 1001 series (1982), 29 series (1993), 3003 series (2000), 100,000 series (2018), and 42,840 series (2020).

1980

Although Kalman filtering was invented in the 1960, it's only in the 1980s that statisticians use **state-space parameterization** and **Kalman filtering** for TSA. The recursive form of the filter enables efficient forecasting. An ARIMA model can be put into a state-space model. Similarly, a state-space model suggests an ARIMA model.

1982

Robert Engle develops the **Autoregressive Conditional Heteroskedasticity (ARCH)** model to account for time-varying volatility observed in economics time series data. In 1986, his student Time Bollerslev develops the Generalized ARCH (GARCH) model. In general, variance of the error term depends on past error terms and their variance. ARCH and GARCH are non-linear generalizations of the Box-Jenkins model.

1987

Engle and Grange propose **cointegration** as a technique for multivariate TSA. Cointegration is a linear combination of marginally unit-root nonstationary series to yield a stationary series. This becomes a popular method in econometrics due to long-term relationship between variables. An earlier method of multivariable TSA is **Vector Autoregressive (VAR)** model.

1998

Zhang et al. publish a survey of **neural networks** applied to forecasting. They note an early work by Lapedes and Farber (1987) who proposed multi-layer feedforward networks. However, the use of ANNs for forecasting happens mostly in the 1990s. In general, feedforward or recurrent networks are preferred. At most two hidden layers are used. Number of input nodes correspond to the number of lagged observations needed to discover patterns in data. Number of output nodes correspond to the forecasting horizon.

2019

Sánchez-Sánchez et al. highlight many issues in using neural networks for TSA. There's no clarity on how to select the number of input or hidden neurons. There's no guidance on how best to partition the data into training and validation sets. It's not clear if data needs to be preprocessed or if seasonal/trend components have to be removed before data goes into the model. In 2018, Hyndman commented that neural networks perform poorly due to insufficient data. This is likely to change as data becomes more easily available.

Time Series Analysis


# Types of Regression

## Summary

Regression is widely used for prediction or forecasting where given one or more independent variables we try to predict another variable. For example, given advertising expense, we can predict sales. Given a mother's smoking status and the gestation period, we can predict the baby's birth weight.

There are many types of regression models, one source mentioning as many as 35 different models. An analyst or statistician must select a model that makes sense to the problem. Models differ based on the number of independent variables, type of the dependent variable and how these two are related to each other.

Regression comes from statistics. It's one of many techniques used in machine learning.

## Discussion

### Could you introduce regression?

Suppose there's a dependent or response variable \(Y\_i\) and independent variables or predictors \(X\_i\). The essence of regression is to estimate the function \(f(X\_i,\beta)\) that's a model of how the dependent variable is related to the predictors. Adding an error term or residual \(\epsilon\_i\), we get \(Y\_i = f(X\_i,\beta) + \epsilon\_i\), for scalar \(Y\_i\) and vector \(X\_i\).

The residual is not seen in data. It's the difference between the observed value \(Y\_i\) and what the model predicts. With the goal of minimizing the residuals, regression estimates model parameters or coefficients \(\beta\) from data. There are many ways to do this and the term **estimation** is used for this process.

Regression modelling also makes important assumptions. The sampled data should represent the population. There are no measurement errors in the predictor values. Residuals have zero mean (when conditioned on \(X\_i\)) and constant variance. Residuals are also uncorrelated with one another. More assumptions are used depending on the model type and estimation technique.

Regression uncovers useful relationships, that is, how predictors are correlated to the response variable. Regression makes no claim that predictors influence or cause the outcome. Correlation should not be confused for causality.

### How do you classify the different types of regression?

Regression techniques can be classified in many ways:

    + **Number of Predictors**: We can distinguish between *Univariate Regression* and *Multivariate Regression*.
    + **Outcome-Predictors Relationship**: When this is linear, we can apply *Linear Regression* or its many variants. If the relationship is non-linear, we can apply *Polynomial Regression* or *Spline Regression*. More generally, when the relationship is known it's *Parametric Regression*, otherwise it's *Non-parametric Regression*.
    + **Predictor Selection**: With multiple predictors, sometimes not all of them are important. *Best Subsets Regression* or *Stepwise Regression* can find the right subset of predictors. We could penalize too many predictors in the model using *Ridge Regression*, *Lasso Regression* or *Elastic Net Regression*.
    + **Correlated Predictors**: If predictors are correlated, one approach is to transform them into fewer predictors by a linear combination of the original predictors. *Principal Component Regression (PCR)* and *Partial Least Squares (PLS) Regression* are two approaches to do this.
    + **Outcome Type**: When predicting categorical data, we can apply *Logistic Regression*. When outcome is a count variable, we can apply *Poisson Regression* or *Negative Binomial Regression*. In fact, a suitable method of regression can be inferred from the distribution of the dependent variable.

### What are the types of linear regression models?

**Simple Regression** involves only one predictor. For example, \(Y\_i = \beta\_0 + \beta\_{1}X\_{1i} + \epsilon\_i\).

If we generalize to many predictors, the term **Multiple Linear Regression** is used. Consider a bivariate linear model \(Y\_i = \beta\_0 + \beta\_{1}X\_{1i} + \beta\_{2}X^2\_{2i} + \epsilon\_i\). Although there's a square term, the model is still linear in terms of the parameters.

To represent many Multiple Linear Regression models in a compact form we can use the **General Linear Model**. This generalization allows us to work with many dependent variables dependent on the same independent variables. This also incorporates different statistical models including ANOVA, ANCOVA, OLS, t-test and F-test.

The General Linear Model makes the assumption that \(Y\_i ∼ N(X^T\_i\beta,\sigma^2)\), that is, response variable is normally distributed with a mean that's a linear combination of predictors. A larger class of models is called **Generalized Linear Model (GLM)** that allows \(Y\_i\) to be any distribution of the exponential family of distributions. The General Linear Model is a specialization of the GLM.

If response is affected by randomness, the **Generalized Linear Mixed Model (GLMM)** can be used.

### Could you compare linear and logistic regression?

Since logistic regression deals with categorical outcomes, it predicts the probability of an outcome rather than a continuous value. Predictions should therefore be restricted to the range 0-1. This is done by transforming the linear regression equation to the **logit scale**. This is the natural log of the odds of being in one category versus the other categories.

For this reason, logistic regression may be seen as a particular case of GLM. Logit is used as the *link function* that relates predictors to the outcome.

Logistic regression shares with linear regression many of the assumptions: independence of errors, linearity (but in the logit scale), absence of multicollinearity among predictors, and lack of influential outliers.

There are three types of logistic regressions:

    + **Binary**: Only two outcomes. Example: predict that a student passes a test. When all predictors are categorial, we call them *logit models* .
    + **Nominal**: More than two outcomes. Also called *Multinominal Logistic Regression*. Example: predict the colour of an iPhone model a customer is likely to buy.
    + **Ordinal**: More than two ordered outcomes. Example: predicting a medical condition (good, stable, serious, critical).

### Could you explain parametric versus non-parametric regression?

Linear models and even non-linear models are parametric models since we know (or make an educated guess) about how the outcome relates to predictors. Once the model is fixed, the task is to estimate the parameters \(\beta\) of the model. If we have problems in this estimation, we can revise the model and try again.

Non-parametric regression is more suitable when we have no idea how the outcome relates to the predictors. Usually when the relationship is non-linear, we can adopt non-parametric regression. For example, one study attempting to predict the logarithm of wage from age found that non-parametric regression approaches outperformed simple linear and polynomial regression methods.

Parametric models have a finite set of parameters that try to capture everything about observed data. Model complexity is bounded even with unbounded data. Non-parametric models are more flexible because the model gets better as more data is observed. We can view them as having infinite parameters or functions that we attempt to estimate. Artificial neural networks with infinitely many hidden units is equivalent to non-parametric regression.

### What are some specialized regression models?

We note a few of these with brief descriptions:

    + **Robust Regression**: This is better suited than linear regression in handling outliers or influential observations. Observations are weighted.
    + **Huber Regression**: To handle outliers better, this optimizes a combination of squared error and absolute error.
    + **Quantile Regression**: Linear regression predicts the mean of the dependent variable. Quantile regression predicts the median. More generally, it predicts the nth quantile. For example, predicting the 25th quantile of a house price means that there's 25% chance that the actual price is below the predicted value.
    + **Functional Sequence Regression**: Sometimes predictors affect the outcome in a time-dependent manner. This model includes the time component. For example, onion weight depends on environmental factors at various stages of the onion's growth.
    + **Regression Tree**: Use a decision tree to split the predictor space at internal nodes. Terminal nodes or leaves represent predictions, which are the mean of data points in each partitioned region.

### Could you share examples to illustrate a few regression methods?

In a production plant, there's a linear correlation between water consumption and amount of production. Simple regression suffices in this case, giving the fit as `Water = 2273 + 0.0799 Production`. Thus, even without any production, 2273 units of water are consumed. Every unit of production increases water consumption by 0.0799 units. Both predictor and outcome are continuous variables.

As an example of multiple linear regression, let's predict the birth weight of a baby (continuous variable) based on two predictors: mother is a smoker or non-smoker (categorial variable) and gestation period (continuous variable). We represent non-smokers as 0 and smokers as 1. The regression equation is `Wgt = - 2390 + 143.10 Gest - 244.5 Smoke`. If we plot this, we'll actually see two parallel lines, one for smokers and one for non-smokers.

One study looked at the number of cigarettes college students smoked per day. They predicted this count from gender, birth order, education level, social/psychological factors, etc. The study used poisson regression, negative binomial regression, and many others.

### With so many types of regression models, how do I select a suitable one?

To apply linear regression, the main assumptions must be met: linearity, independence, constant variance and normality. Linearity can be checked via graphical analysis. A plot of residuals versus predicted values can show non-linearity, or use goodness of fit test. Non-linear relations can be made linear using transformations of predictors and/or the outcome. These could be log, square root or power transformations. Try adding transformations of current predictors. Try semi or non-parametric models.

In practice, linear regression is sensitive to outliers and cross-correlations. Piecewise linear regression, particularly for time series data, is a better approach. Non-parametric regression can be used when there's an unknown non-linear relationship. SVR is an example of non-parametric regression.

When overfitting is a problem, use cross validation to evaluate models. Ridge, lasso and elastic net models can help tackle overfitting.  They can also handle multicollinearity. Quantile regression is suited to handle outliers.

For predicting counts, use negative binomial regression if variance is larger than the mean. Poisson regression can be used only if variance equals the mean.

### What are some tips to analyze model statistics?

Well-known model performance metrics include R-squared (R2), Root Mean Squared Error (RMSE), Residual Standard Error (RSE) and Mean Absolute Error (MAE). We also have metrics that penalize additional predictors: Adjusted R2, Akaike's Information Criteria (AIC) and Bayesian Information Criteria (BIC) and Mallows Cp. Higher the R2 or Adjusted R2, better the model. For all other metrics, lower value implies a better model.

A high t-statistic implies coefficient is probably non-zero. A low p-value on the t-statistic gives confidence on the estimate. Low coefficients and low p-value for the model as a whole can imply multicollinearity. While t-test is applied to individual coefficients, F-test is applied to the overall model.

Two models can be compared graphically. For example, the coefficients and their confidence intervals can be plotted and compared visually.

### What software packages support regression?

In R, functions `lm()`, `summary()`, `residuals()` and `predict()` in the `base` package enable linear regression. For GLM, we can use `glm()` function. Use `quantreg` package for quantile regression; `glmnet` for ridge, lasso and elastic net regression; `pls` for principal component regression; `plsdepot` for PLS regression; `e1071` for Support Vector Regression (SVR); `ordinal` for ordinal regression; `MASS` for negative binomial regression; `survival` for cox regression. Other useful packages are `stats`, `car`, `caret`, `sgd`, `BLR`, `Lars`, and `nlme`.

In Python, `scikit-learn` provides a number of modules and functions for regression. Use module `sklearn.linear_model` for linear regression including logistic, poisson, gamma, huber, ridge, lasso, and elastic net; `sklearn.svm` for SVR; `sklearn.neighbors` for k-nearest neighbours regression; `sklearn.isotonic` for isotonic regression; `metrics` for regression metrics; `sklearn.ensemble` for ensemble methods for regression.

## Milestones

1795

Regression starts with Carl Friedrich Gauss with the **method of least squares**. He doesn't publish the method until much later in 1809. In 1805, Adrien-Marie Legendre invents the same approach independently. Legendre uses it to predict the orbits of comets.

1877

Francis Galton plots in 1877 what may be called the first **regression line**. It concerns the size of sweet-pea seeds. It correlated the size of daughter seeds against that of mother seeds. Such an analysis came about in the course of investigating Darwin's mechanism for heredity. By these experiments, Galton also introduces the concept of "reversion to the mean", later called **regression to the mean**.

1915

R.A. Fisher gives the exact sampling distribution of the coefficient of correlation, thus marking the beginning of **multivariate analysis**. Fisher then simplifies it to a form via **z-transformation**. In the early 1920s, he introduces the **F distribution** and **maximum likelihood** method of estimation.

1957

Hotelling proposes **Principal Component Regression (PCR)** in an attempt to reduce the number of explanatory variables (predictors) in the regression model. PCR itself is based on Principal Component Analysis (PCA) that was invented independently by Pearson (1901) and Hotelling (1930s).

1960

Although the logistic function was invented by Verhulst in the 1830s, it's only in the 1960s that it's applied to regression analysis. D.R. Cox is among the early researchers to do this. Many researchers, including Cox, independently develop **Multinomial Logistic Regression** through the 1960s.

1970

Hoerl and Kennard note that least squares estimation is unbiased and this can give poor results if there's multicollinearity among the predictors. To improve the estimation they propose a biased estimation approach that they call **Ridge Regression**. Ridge regression uses standardized variables, that is, outcome and predictors are subtracted by mean and divided by standard deviation. By introducing some bias, variance of the least squares estimator is controlled.

1972

D.R. Cox applies regression to life-table analysis. Among the sampled individuals, he observes either the time to "failure" or that the individual is removed from the study (called censoring). Moreover, the distribution of survival times is often skewed. For these reasons, linear regression is not suitable. Cox instead uses a hazard function that incorporates age-specific failure rate. In later years, this approach is simply called **Cox Regression**.

1972

Nelder and Wedderburn introduce the **Generalized Linear Model (GLM)**. As examples, they relate GLM to normal, binomial (probit analysis), poisson (contingency tables), and gamma (variance components) distributions. However, it's only in the 1980s that GLM becomes popular due to the work of McCullagh and Nelder.

1978

Koenker and Bassett introduce **Quantile Regression**. This uses weighted least absolute error rather than least squares error common in linear regression.

1981

Huber proposes an estimator that's quadratic in small values and grows linearly for large values. It's later named **Huber Regression**.

1996

Tibshirani proposes **Lasso Regression** that uses the least squares estimator but constrains the sum of absolute value of coefficients to a maximum. This forces some coefficients to zero or low values, leading to more interpretable models. This is useful when we start with too many predictors.

2002

De'ath proposes the **Multivariate Regression Tree (MRT)**. The history of regression trees goes back to the 1960s. With the release of CART (Classification and Regression Tree) software in 1984, they became more well known. However, CART is limited to a single response variable. MRT extends CART to multivariate response data.

2005

Zou and Hastie propose **Elastic Net Regression**. This combines elements of both ridge regression and lasso regression.

Types of Regression


# Full Stack Developer

## Summary

Any technological solution to a real-world problem consists of several IT components interacting with each other. The entire basket of software platforms, tools, services, and even hardware or networking devices employed in the development of an IT application is called *Technology Stack*. A developer whose skills cover the entire range of the technology stack, both at client and system end is a Full Stack Developer. It's more of a coinage indicating a programmer who is jack of all arts and master of one or two.

These professionals can easily understand most programming languages and can help to bring the company's minimum viable product into the market quickly. This is especially important for web or mobile app start-ups. Full stack developers, due to their wider system understanding, are able to contribute better to system design in addition to development.

## Discussion

### How did the sudden demand for full stack developers occur?

The whole narrative of full stack developer emerged from the IT start-up boom. Earlier, large IT service companies or product MNCs were keen on specialists who knew one thing well. These traditional roles were GUI developer, C/Java programmer, database specialist/admin, network engineer, test or automation engineer, and so on. A typical IT application would be an integrated hierarchical solution requiring most of these skill-sets.

But in start-ups, the team is looking to build a minimum viable product that can showcase their basic idea. This helps them seek funding and then make expansion plans. This has to be achieved with minimum number of developers and limited investment. Time to market is also very short. Hence, the need arose to recruit developers capable of programming in the entire spectrum of technologies.

A 2018 LinkedIn survey listed "Full Stack Developers" among the top 10 hard skills that developers need to possess in the IT industry.

While full stack developers are presently the trend, it doesn't imply that experts are no longer vital. As the product grows and scales, specialists are required.

### What are the technologies that a full stack developer is expected to know?

Skillset of full-stack developers resembles the T-model. They have knowledge across wide-breadth of technologies but in-depth knowledge of a couple of those.

Knowledge of at least one language/platform in each technology layer is a must. In the web development context, popular full stack combinations include:

    + **LAMP stack**: JavaScript - Linux - Apache - MySQL - PHP
    + **LEMP stack**: JavaScript - Linux - Nginx - MySQL - PHP
    + **MEAN stack**: JavaScript - MongoDB - Express - AngularJS - Node.js
    + **Django stack**: JavaScript - Python - Django - MySQL
    + **Ruby on Rails**: JavaScript - Ruby - SQLite - RailsVendor-specific full stack expertise is also quite prevalent:

    + **Microsoft**: .NET (WPF, WCF, WF), Visual Studio, C#, ASP.NET/ MVC/ MVVM, REST/ Web API, Azure, GIT/ SVN, Web (HTML5/CSS3/JS), jQuery / Typescript, Bootstrap / Angular, MS SQL Server
    + **Amazon**: AWS Amplify (Web UI), Amazon Cognito (web browser), Amazon API gateway, AWS Lambda, Amazon DynamoDBHowever, developers must always remember that the technology choice is dependent on what works best for the product under design, not the other way round.

### What is a company really asking for while recruiting a full stack developer?

Technology used in products are never static. Companies may decide to migrate to a new version, platform or vendor that enters the market. Therefore, there is no point in recruiting a full stack developer with rigid set of skills. By asking for full stack developers, companies are actually looking for the following skills and attitude in a candidate:

    + At least know one language or platform in each technology function - user Interface, backend processing, middleware, business logic, DB/storage, networking/communication, testing.
    + Self-learning ability to master a new technology quickly.
    + Ability to make a demonstrable product or prototype.
    + Can independently perform debugging and customer support for applications, with quick turnaround time. The bug could be anywhere in the system.
    + Work in teams and relate to the problems faced by developers working on other modules.
    + Understand the big picture, translate the customer requirement into a system design, which would encompass several technologies across layers. A full stack developer will make a good tech lead or system architect.
    + Good at integration testing and system testing from customer perspective.

### Are there multiple types of technology stacks?

There is no text book definition for what constitutes a technology stack. Any hierarchy of interdependent modules built using different technologies, frameworks and languages is a technology stack.

For instance, the concept of a protocol stack has existed for decades now - OSI Layers, TCP/IP stack and other communication/control protocols.

A mobile phone device can be an example of a hardware technology stack – body, network processor chip, peripherals, memory, battery, LCD screen all stacked one above the other.

A MEAN stack refers to a stack for building web apps and websites. It consists of MongoDB for database storage, Express.js as the application framework, AngularJS MVC framework, and Node.js as the server-side framework.

The system design document prepared by a product development team would document the customised technology stack to be used for its own products.

Many large IT companies openly declare what technology stack is used in their development.

### How to choose a technology stack for a product/application?

Technology choices are made only in the product design phase, after the product requirements are finalised. This involves evaluating various technology alternatives to make up the stack. The factors considered are:

    + **Meets the requirements entirely**: For example, if the requirement is to build a military application, then platforms with highest data security and reliability are chosen. Limited network connectivity, multi-language support, accessibility for the disabled are examples of specialised requirements that influence the choice of stack.
    + **Scalable to support future requirement additions**: Product requirements are always changing based on customer feedback. So the technology choice must support product growth for at least 3-5 years.
    + **Cost considerations**: When budgets are limited, companies tend to prefer open source options. Or if an older project has a pre-purchased software license, the same may continue into the new one. This is not optimal, but happens a lot in the industry.
    + **Skillset of existing workforce**: This goes against the idea of designing for requirements. But very often, due to HR constraints and inability to reskill, companies decide to stick to a particular technology stack.

### Can you list the technology stacks used in popular IT applications?

Some product MNCs and start-ups swear by the efficacy of hiring full stack developers. They openly publicise the technology stack used in their solutions, such as:

    + **Facebook**: PHP, React, GraphQL, Memcached, Cassandra, Hadoop, MySQL, Swift, C++, PHP, JavaScript, JSON, HTML, CSS.
    + **Amazon**: Java, Perl, Angular JS, MySQL, Amazon EC2 container service, DynamoDB and a host of other Amazon frameworks.
    + **Google**: Python, Java, Android SDK, Go, C++, Preact, Angular JS, Kubernetes, TensorFlow and a host of other Google frameworks.
    + **Dropbox**: REDIS, C#, .NET, MS SQL Server
    + **StackOverflow**: NGINX, Amazon, MySQL, Python
    + **Airbnb**: Javascript, MySQL, Java, Ruby on Rails
    + **Fitbit**: Node.JS, Javascript, Objective C, JavaHowever many technology experts and recruiters call the idea as a passing fad which breeds superficial programmers, who lack the ability to build deep expertise in anything. Without knowing the nuances of a language, the best implementations are not possible, they claim.

## Milestones

2008

One of the earliest mentions of the term "Full Stack Developer" is in a blog written by Randy Schmidt for Forge38 magazine. It's clearly used in the context of web development.

2010

The first Google search for the term "Full Stack Developer" happens. So it's a fairly recent phenomenon.

2012

Front-end development takes a rapid leap. Full stack developers for web development become a popular choice. Earlier, web browsers were poor at interpreting a lot of JavaScript. Adding complex functionality with JS wasn't always a good idea. As browsers became more powerful, JavaScript became versatile with extensions such as AngularJS and jQuery.

2014

The number of layers in the stack is steadily increasing. In 2010, a full-stack developer perhaps needed to know PHP, jQuery, HTML, CSS, and FTP to transfer files to the hosting server. Today, a full-stack developer needs a wider spectrum of skills from modular frameworks to CSS pre-processors, from responsive UI design to cloud cluster management.

2020

Facebook announces a new update to their technology stack. They are rebuilding their tech stack for Facebook.com, moving beyond a simple PHP website. Their stack includes React (a declarative JavaScript library for building user interfaces) and Relay (a GraphQL client for React).

Oct
2020

The position "Full stack developer" has over 14,000 listings in the US and 6,000 India on the Indeed Job portal.

Full Stack Developer


# IEEE 802.11ac

## Summary

IEEE 802.11ac is a Wi-Fi standard for Very High Throughput (VHT) applications. It's a significant improvement over the earlier IEEE 802.11n while remaining backward compatible. It's designed only for the 5 GHz band.

It provides higher data rates, reaching a maximum of 6.9 Gbps. Explicit beamforming and MU-MIMO are two important features of 802.11ac that improves network capacity and efficiency.

It has a migration path towards IEEE 802.11ax.

## Discussion

### What are the typical use cases for 802.11ac?

Video consumption is on the rise. A 720p uncompressed video at 60 frames/sec needs 1.3 Gbps. A H.264 lightly compressed video needs 70-200 Mbps. IEEE 802.11n offer a theoretical 600 Mbps but practical rates available for application are lot less.

The number of devices in the home or office is also increasing. There's a need for multiple devices to connect to the same access point and utilize the channels more efficiently. This is particularly true for bring-your-own-device (BYOD) scenarios where each employee may bring multiple Wi-Fi devices to the office.

In general, 802.11ac aims to provide high data rates for video streaming, low latency experience and more efficient multiplexing of multiple clients.

### What are main features of 802.11ac contributing to higher data rate?

While 802.11n offers a maximum data rate of 600 Mbps, 802.11ac can offer 10x data rate due many improvements:

    + **Channel Bonding**: Each channel is 20 MHz wide but with 802.11ac we can combine 8 of these to obtain 160 MHz channel for a single client. If available, contiguous channels are easily combined although standard defines 80+80 MHz mode to combine non-contiguous channels.
    + **Higher Modulation**: The use of 256QAM is possible, 4x denser than 64QAM of 802.11n. This means that 4x more bits can be carried per symbol.
    + **Spatial Streams**: Up to 8 spatial streams are possible, although Wave 2 certification covers only 4 spatial streams.The maximum data rate of 6.9 Gbps is obtained when using 160 MHz channels, 256 QAM, eight spatial streams and 400 ns guard interval. A handy reference comparing 802.11n and 802.11ac rates relative to SNR and RSSI is available online.

### How is 802.11ac able to achieve better efficiency?

For better channel utilization and network efficiency, the following are useful:

    + **5 GHz Band**: The use of 2.4 GHz band is avoided where interference is higher due to cordless phones, microwaves and other devices. The 5 GHz band is cleaner.
    + **Enhanced RTS/CTS**: To avoid collisions due to the use of wider channels, RTS/CTS mechanism is extended.
    + **A-MPDU**: For higher MAC layer throughput, all MAC frames are sent as Aggregate MAC Protocol Data Unit (A-MPDU), which was introduced in 802.11n for selective use.
    + **MU-MIMO**: With Single-User Multiple-Input and Multiple-Output (SU-MIMO), only one client could send/receive at a particular time. Multi-User Multiple-Input and Multiple-Output (MU-MIMO) is able to multiplex multiple clients at the same time, thus reducing latency and improving overall network efficiency.
    + **Beamforming**: Due to the use of MIMO and multiple antennas, beamforming is possible. Transmission is steered towards each client. This is made more efficient via explicit feedback from clients–using Null Data Packet (NDP)–for better channel estimation.

### What are some trade-offs involved in using 802.11ac?

Decision to trade-off one metric with another will depend on real-time network conditions. Moving from 40 MHz to 80 MHz aggregate bandwidth will increase data rate, but since the same power is spread across many more subcarriers, range will reduce. Obtaining a free channel 160 MHz wide is also difficult, especially in enterprise use cases. It's easier with 80+80 MHz mode but this requires twice as many RF chains.

Moving from 64QAM to 256QAM is possible only over short distances (good signal-to-noise ratio) since the constellation is tighter and more sensitive to errors.

Beamforming using Explicit Compressed Feedback (ECFB) gives a precise channel estimate but this feedback comes with a lot of overhead. Beamforming and MU-MIMO become less effective when clients are moving.

In terms of spatial streams, each stream requires its own antenna and RF chain. Although 8 streams are defined in the standard, often this is impractical in mobile devices since each antenna must be sufficiently spaced.

### Could you explain 802.11ac MU-MIMO?

Since 802.11n, SU-MIMO allows routers to send/receive multiple streams of data to/from clients. MU-MIMO is introduced in 802.11ac in the downlink. An access point can send data to multiple clients at the same time. For example, with three clients A, B and C, two streams may be sent to A, one stream to B and one stream to C simultaneously. Clients receiving single streams need not have multiple antennas or RF chains, which is often the case with small devices and smartphones.

With MU-MIMO, we get better network capacity utilization because a single client with not much to send can be multiplexed with other clients. Because multiple clients can be allowed to receive at the same time, latency also drops. Even non-MIMO clients will benefit since they can access the channel more easily. More device clients can be supported on the Wi-Fi network.

In 802.11ac, MU-MIMO is available only in the downlink. Another limitation is that access points might fallback to SU-MIMO if they detect that clients are moving, since MU-MIMO may not work well.

### What are 802.11ac Wave 1 and Wave 2?

The idea of identifying two "waves" of 802.11ac products was to allow vendors to release their first 802.11ac products into the market quickly. Wave 2 is complex due to MU-MIMO, 160 MHz bandwidth support and four spatial streams. By defining Wave 1 without these features, we can still benefit from having 256QAM and 80 MHz bandwidth support that 802.11n lacked.

First Wave 1 products started arriving in 2013. Certification for these also started in mid-2013. From 2014, all new products started supporting Wave 1. Early Wave 2 products arrived in 2015. Certification for Wave 2 started in mid-2016.

## Milestones

Sep
2008

Work on 802.11ac standardization formally commences with the approval of Project Allocation Request.

2012

Draft 2.0 of 802.11ac is released in January. A refined draft 3.0 is released in May.

Jun
2013

Wi-Fi Alliance announces certification process for 802.11ac Wave 1 products.

Dec
2013

**IEEE 802.11ac-2013** standard is published. Along with IEEE 802.11ad-2012 that was standardized a year earlier, both are the result of the **Very High Throughput (VHT)** study group.

2014

Quantenna becomes the first to have a chipset that's capable of 4x4 MU-MIMO but without 160 MHz support. In April, Qualcomm Atheros also starts offering chipsets for MU-MIMO. In general, 802.11ac Wave 2 support arrives in 2014 whereas 2013 saw only 802.11ac Wave 1 support.

Jan
2015

At CES 2015, more chipsets and products capable of 802.11ac Wave 2 are announced.

May
2015

What's probably the world's first 802.11ac router with MU-MIMO support, Linksys releases its EA8500. For best results, clients also need to support MU-MIMO. Some existing devices might support it with a firmware upgrade since their underlying chipsets have MU-MIMO support already. An example of this is ASUS RT-AC87U router released in August 2014.

Jun
2016

Wi-Fi Alliance announces certification process for 802.11ac Wave 2 products.

2018

CNET identifies some of the best 802.11ac routers. This includes brands Asus, Synology, D-Link and Netgear. Asus has multiple routers highly recommended by CNET.

IEEE 802.11ac


# Text Clustering

## Summary

The amount of text data being generated in the recent years has exploded exponentially. It's essential for organizations to have a structure in place to mine actionable insights from the text being generated. From social media analytics to risk management and cybercrime protection, dealing with textual data has never been more important.

Text clustering is the task of grouping a set of unlabelled texts in such a way that texts in the same cluster are more similar to each other than to those in other clusters. Text clustering algorithms process text and determine if natural clusters (groups) exist in the data.

## Discussion

### What's the principle behind text clustering?

The big idea is that documents can be represented numerically as vectors of features. The similarity in text can be compared by measuring the distance between these feature vectors. Objects that are near each other should belong to the same cluster. Objects that are far from each other should belong to different clusters.

Essentially, text clustering involves three aspects:

    + Selecting a suitable distance measure to identify the proximity of two feature vectors.
    + A criterion function that tells us that we've got the best possible clusters and stop further processing.
    + An algorithm to optimize the criterion function. A greedy algorithm will start with some initial clustering and refine the clusters iteratively.

### What are the use cases of text clustering?

We note a few use cases:

    + **Document Retrieval**: To improve recall, start by adding other documents from the same cluster.
    + **Taxonomy Generation**: Automatically generate hierarchical taxonomies for browsing content.
    + **Fake News Identification**: Detect if a news is genuine or fake.
    + **Language Translation**: Translation of a sentence from one language to another.
    + **Spam Mail Filtering**: Detect unsolicited and unwanted email/messages.
    + **Customer Support Issue Analysis**: Identify commonly reported support issues.

### How is text clustering different from text classification?

Classification is a supervised learning approach that maps an input to an output based on example input-output pairs. Clustering is a unsupervised learning approach.

    + **Classification**: If the prediction value tends to be category like yes/no or positive/negative, then it falls under classification type problem in machine learning. The different classes are known in advance. For example, given a sentence, predict whether it's a negative or positive review.
    + **Clustering**: Clustering is the task of partitioning the dataset into groups called clusters. The goal is to split up the data in such a way that points within single cluster are very similar and points in different clusters are different. It determines grouping among unlabelled data.

### What are the types of clustering?

Broadly, clustering can be divided into two groups:

    + **Hard Clustering**: This groups items such that each item is assigned to only one cluster. For example, we want to know if a tweet is expressing a positive or negative sentiment. *k-means* is a hard clustering algorithm.
    + **Soft Clustering**: Sometimes we don't need a binary answer. Soft clustering is about grouping items such that an item can belong to multiple clusters. *Fuzzy C Means (FCM)* is a soft clustering algorithm.

### What are the steps involved in text clustering?

Any text clustering approach involves broadly the following steps:

    + **Text pre-processing**: Text can be noisy, hiding information between stop words, inflexions and sparse representations. Pre-processing makes the dataset easier to work with.
    + **Feature Extraction**: One of the commonly used technique to extract the features from textual data is calculating the frequency of words/tokens in the document/corpus.
    + **Clustering**: We can then cluster different text documents based on the features we have generated.

### What are the steps involved in text pre-processing?

Below are the main components involved in pre-processing.

    + **Tokenization**: Tokenization is the process of parsing text data into smaller units (tokens) such as words and phrases.
    + **Transformation**: It converts the text to lowercase, removes all diacritics/accents in the text, and parses html tags.
    + **Normalization**: Text normalization is the process of transforming a text into a canonical (root) form. Stemming and lemmatization techniques are used for deriving the root word.
    + **Filtering**: Stop words are common words used in a language, such as 'the', 'a', 'on', 'is', or 'all'. These words do not carry important meaning for text clustering and are usually removed from texts.

### What are the levels of text clustering?

Text clustering can be document level, sentence level or word level.

    + **Document level**: It serves to regroup documents about the same topic. Document clustering has applications in news articles, emails, search engines, etc.
    + **Sentence level**: It's used to cluster sentences derived from different documents. Tweet analysis is an example.
    + **Word level**: Word clusters are groups of words based on a common theme. The easiest way to build a cluster is by collecting synonyms for a particular word. For example, WordNet is a lexical database for the English language that groups English words into sets of synonyms called *synsets*.

### How do I define or extract textual features for clustering?

In general, *words* can be used to represent a common class of feature. *Word characteristics* are also features. For example, capitalization matters: US versus us, White House versus white house. *Part of speech* and *grammatical structure* also add to textual features. *Semantics* can be a textual feature: buy versus purchase.

The mapping from textual data to real-valued vectors is called feature extraction. One of the simplest techniques to numerically represent text is **Bag of Words (BOW)**. In BOW, we make a list of unique words in the text corpus called vocabulary. Then we can represent each sentence or document as a vector, with each word represented as *1 for presence* and *0 for absence*.

Another representation is to count the number of times each word appears in a document. The most popular approach is using the **Term Frequency-Inverse Document Frequency (TF-IDF)** technique.

More recently, word embeddings are being used to map words into feature vectors. A popular model for word embeddings is **word2vec**.

### How can I measure similarity in text clustering?

Words can be similar lexically or semantically:

    + **Lexical similarity**: Words are similar lexically if they have a similar character sequence. Lexical similarity can be measured using string-based algorithms that operate on string sequences and character composition.
    + **Semantic similarity**: Words are similar semantically if they have the same meaning, are opposite of each other, used in the same way, used in the same context or one is a type of another. Semantic similarity can be measured using corpus-based or knowledge-based algorithms.Some of the metrics for computing similarity between two pieces of text are *Jaccard coefficient*, *cosine similarity* and *Euclidean distance*.

### Which are some common text clustering algorithms?

Ignoring neural network models, we can identify different types:

    + **Hierarchical**: In the *divisive* approach, we start with one cluster and split that into sub-clusters. Example algorithms include DIANA and MONA. In the *agglomerative* approach, each document starts as its own cluster and then we merge similar ones into bigger clusters. Examples include BIRCH and CURE.
    + **Partitioning**: k-means is a popular algorithm but requires the right choice of *k*. Other examples are ISODATA and PAM.
    + **Density**: Instead of using a distance measure, we form clusters based on how many data points fall within a given radius. DBSCAN is the most well-known algorithm.
    + **Graph**: Some algorithms have made use of knowledge graphs to assess document similarity. This addresses the problem of polysemy (ambiguity) and synonymy (similar meaning).
    + **Probabilistic**: A cluster of words belong to a topic and the task is to identify these topics. Words also have probabilities that they belong to a topic. *Topic Modelling* is a separate NLP task but it's similar to soft clustering. pLSA and LDA are example topic models.

### How can I evaluate the efficiency of a text clustering algorithm?

Measuring the quality of a clustering algorithm has shown to be as important as the algorithm itself. We can evaluate it in two ways:

    + **External quality measure**: External knowledge is required for measuring the external quality. For example, we can conduct surveys of users of the application that includes text clustering.
    + **Internal quality measure**: The evaluation of the clustering is compared only with the result itself, that is, the structure of found clusters and their relations to one another. Two main concepts are compactness and separation. *Compactness* measures how closely data points are grouped in a cluster. *Separation* measures how different the found clusters are from each other. More formally, compactness is intra-cluster variance whereas separation is inter-cluster distance.

### What are the common challenges involved in text clustering?

Document clustering is being studied for many decades. It's far from trivial or a solved problem. The challenges include the following:

    + Selecting *appropriate features* of documents that should be used for clustering.
    + Selecting an *appropriate similarity measure* between documents.
    + Selecting an *appropriate clustering method* utilising the above similarity measure.
    + *Implementing the clustering algorithm* in an efficient way that makes it feasible in terms of memory and CPU resources.
    + Finding ways of *assessing the quality* of the performed clustering.

## Milestones

1971

Text mining research in general relies on a vector space model. Salton first proposes it to model text documents as vectors. Features are considered to be the words in the document collection and feature values come from different term weighting schemes, the most popular of which is the **Term Frequency-Inverse Document Frequency (TF-IDF)**.

1983

Massart et al. in the book *The Interpretation of Analytical Chemical Data by the Use of Cluster Analysis* introduces various clustering methods, including hierarchical and non-hierarchical methods. They show how clustering can be used to interpret large quantities of analytical data. They discuss how clustering is related to other pattern recognition techniques.

1992

Cutting et al. adapt partition-based clustering algorithms to cluster documents. Two of the techniques are Buckshot and Fractionation. **Buckshot** selects a small sample of documents to pre-cluster them using a standard clustering algorithm and assigns the rest of the documents to the clusters formed. **Fractionation** finds k centres by initially breaking N documents into N/m buckets of a fixed size m > k. Each cluster is then treated as if it's an individual document and the whole process is repeated until there are only K clusters.

1997

Huang introduces **k-modes**, an extension to the well-known k-means algorithm for clustering numerical data. By defining the mode notion for categorical clusters and introducing an incremental update rule for cluster modes, the algorithm preserves the scaling properties of k-means. Naturally, it also inherits its disadvantages, such as dependence on the seed clusters and the inability to automatically detect the number of clusters.

2008

Sun et al. develop a novel hierarchal algorithm for document clustering. They use cluster overlapping phenomenon to design cluster merging criteria. The system computes the overlap rate in order to improve time efficiency.

Text Clustering


# JSON Web Token

## Summary

JSON is a data format commonly used in web applications. JSON Web Token (JWT) is a mechanism that brings **security to JSON data**.

JSON grew in adoption from the mid-2000s. This influenced the adoption of JWT. Compared to alternatives such as XML or SAML, app developers found JWT easier to implement and use. JWTs are less verbose and more secure. By the late 2010s, JWTs were widely used in the world of cloud computing and microservices.

JWT is available in two formats: JSON Web Signature (JWS) and JSON Web Encryption (JWE). JWS offers protection against data tampering (integrity). JWE prevents others from reading the data (confidentiality). Moreover, developers have a choice of various keys and algorithms to protect JSON data in either of these formats.

IETF has published the main RFCs that cover JWTs. There are also plenty of open source implementations in many languages.

## Discussion

### How does JWT bring security to the web?

Consider an application consisting of many services exposed to clients via APIs. We certainly don't want clients to authenticate with each service. Authentication is done by a specific service or server. Once authenticated successfully, the client should be able to access any of the services without further authentication. This is where JWTs can help.

For example, in AWS, Amazon Cognito does authentication. An authenticated client is issued a JWT. Whenever the client makes an API request, it presents this token. The API gateway validates the token before allowing the client to access the requested service. Thus, all relevant information is within the JWT. The API gateway need not contact the authentication server to determine if the client should be allowed access.

For authorized access, privileges can be set within the token. For example, the name-value pair `admin:true` could be set to allow deletion of records and other admin operations. Moreover, such privileges are set when the token is issued. The client or third-party hackers can't tamper with the token.

### What are some use cases where JWT can be used?

The common use case of JWTs is **authorization**. For example, APIs often require an access token and this could be a JWT. Systems implementing Single Sign-On (SSO) can issue JWTs to allow the user to access various services.

Assume a server authenticates a user and issues a single-use short-lived JWT. User uses this token to download a file from another server. In this example, JWT temporarily authorizes the user to download a protected resource. In microservices architecture, JWTs are used to pass authorization across services. OAuth 2.0 access tokens are JWTs.

JWTs can be used for **authentication**. For example, in OpenID Connect (OIDC), users login with a JWT. Another example is to authenticate a SOAP request with JWT rather than SAML2 assertion. In Oracle Cloud, API gateway authenticates an API client based on the JWT it receives. Once validated, claims in the token are used to authorize the client. JWTs can be used to authenticate SPAs.

Due to its protection against tampering and snooping, JWTs are a means to **exchange information securely**.

### What are the main components of a signed JWT?

A JWT has two essential components: header and payload. In practice, JWTs are signed, that is, they include a signature. This is what we call *JWS*. Thus, the three main components of a signed JWT are:

    + **Header**: Specifies the type of token (typically `JWT`) and the algorithm used.
    + **Payload**: The main content of the token that includes a set of claims.
    + **Signature**: This is computed from header and payload to protect the integrity of the token.Header and payload are JSON objects. However, these are not transmitted as such. They are Base64-URL encoded, which is similar to Base64 encoding except that characters special to URLs are replaced: `+` becomes `-`, `/` becomes `_`.

Signature is computed as `BASE64URL(UTF8(JWS Protected Header)) || '.' || BASE64URL(JWS Payload)` and then Base64-URL encoded.

The JWS is constructed by concatenating header, payload, and signature. Period character separates the fields. We can represent this as `BASE64URL(UTF8(JWS Protected Header)) || '.' || BASE64URL(JWS Payload) || '.' || BASE64URL(JWS Signature)`. This format of JWS is called *JWS Compact Serialization*. There's also *JWS JSON Serialization* that can have multiple signatures.

### What are JWT claims and how to specify such claims?

The payload of a JWT has a set of claims. A claim is a name-value pair. It states a fact about the token and its subject such as username or email address. Claims are not mandatory since JWTs are meant to be compact. It's up to applications to include claims that matter.

Some claim names are **registered** with IANA. Examples include "iss" (issuer), "sub" (subject), and "aud" (audience). Some registered claim names are of datetime type: "exp" (token expires at this time), "nbf" (token can't be used before this time), and "iat" (time when token was issued). For example, `"exp":1300819380` says that the token expires at the specified timestamp.

There are also **public** or **private** claim names. These are application specific and their semantics are agreed between producer and consumer of the token. Public names must be collision-resistant. For example, a name based on domain name or Universally Unique IDentifier (UUID) is unlikely to collide with another public name.

### Is it possible to encrypt the payload in a JWT?

A signed token can be read by anyone. The purpose of signature is to prevent hackers from tampering with the header or payload. Any changes to these would result in a different signature, which the attacker can't create without the secret key. Such tampering would cause a failure during signature verification.

If the intention is to send sensitive payload, signature alone is inadequate. There's a need to encrypt the payload. This is where JWE format becomes relevant.

Encryption of content uses symmetric keys. However, these keys need not be shared in advance. They can be generated dynamically and exchanged within JWE. However, the shared symmetric key is encrypted using asymmetric private-public key pair.

It's also possible to do both, such as a JWS within a JWE or vice versa. Such as token is called **Nested JWT**. For example, we could create a JWS as usual and encrypt it. This then becomes the encrypted payload of a JWE. A simpler approach is to use only encryption algorithms mentioned in JWA since they also provide integrity protection.

### Which are the various algorithms supported by JWT?

Both symmetric and asymmetric algorithms are supported for signing and encrypting tokens. In fact, this support for a variety of algorithms is perhaps one reason for the wider adoption of JWTs.

For JWS, at the minimum, an implementation should support HMAC using SHA-256. This uses a shared symmetric key. For the "alg" header parameter, its value is "HS256". Apart from HMAC for signature, we can have digital signatures using asymmetric keys with RSASSA-PKCS1-v1\_5, ECDSA, or RSASSA-PSS. Signing is done with the private key. Verification happens with the public key.

For content encryption in JWE, at the minimum, an implementation should support A128CBC-HS256 and A256CBC-HS512. A128CBC-HS256 does AES encryption in CBC mode with 128-bit IV value, plus HMAC authentication using SHA-256 and truncating HMAC to 128 bits. Encryption key is called Content Encryption Key (CEK).

The CEK itself is encrypted using other algorithms and included in the JWE. Some of these are RSA-OAEP, A128KW, ECDH-ES, ECDH-ES+A128KW, and more. It's also possible to use a shared symmetric key as CEK.

### Can I use JWTs as a replacement to session objects?

With session objects, server maintains the state about each logged-in user, who gets a session ID via a HTTP cookie. Subsequent requests contain the session ID. Server uses it to retrieve the session object and serve the client. Thus, stateless HTTP calls are strung together into a stateful session.

With JWTs, server doesn't need to store session state. All relevant information is contained in the JWT. This also makes it convenient to deploy on a distributed architecture. One server might issue the JWT. Subsequent client requests could be served by another server via a load balancer.

While JWTs seem attractive, a JWT takes more space compared to a session ID. Even without session objects, most client requests will still need access to the database, implying that JWTs don't improve performance. Most web frameworks automatically sign session cookies, implying that signing a JWT isn't really an advantage. JWTs stored on local storage can be less secure. We can't invalidate individual JWTs or deal with stale claims in the token. For these reasons, use of JWTs as an alternative to session IDs is not recommended.

### Which are the known vulnerabilities of JWT?

Most attacks on JWT are due to implementations rather than its design.

Some early implementations used the "alg" value in header to verify the signature. An attacker could set the value to "none"; or change "RS256" to "HS256" and use the public key as the shared secret key to generate a valid signature. Based on the "alg" value, the consumer would skip verification or incorrectly find that the signature is valid.

Brute force attacks on HS256 are simple if the shared secret key is too short. Another possible oversight in implementations is not verifying the claims or not including adequate claims. For example, a token without audience is issued to Org1 but the attacker could present the same token to Org2 can gain access if that username exists in both organizations.

Don't store sensitive information in JWS, that is, in unencrypted form. Don't also assume that encrypted data can't be tampered with.

RFC8725 details many vulnerabilities and best practices to overcome the same.

### What are some best practices when using JWT?

Pick strong keys. These are often long and created by cryptographic-quality pseudo-random number generators (PRNGs). Don't use human-readable shared secret keys. For federated identity, or when third-party services are involved, it's inconvenient and unsafe to use a shared secret. Instead, use public-private key pair.

Don't rely on the header to select the algorithm for verifying the signature. Use libraries that allow for explicit selection of algorithm.

It's a good practice to verify applicable claims. For example, verify that token has not expired. In AWS we might verify that the audience claim matches the app client ID created in the Amazon Cognito user pool. Where nested tokens are used, verify signature on all tokens, not just on the outermost token.

Use different validation rules for each token. Avoid key reuse for different tokens. For example, we could use different secret keys for each subsystem. Using "kid" claim, we could identify which secret key is used by the token.

Keep the lifetime of tokens short, says for a few minutes or hours. In addition, we could include a nonce in the token to prevent replay attacks (which is what OpenID Connect does).

### Could you mention some resources concerning JWT?

IETF documents relevant to JWT include RFC7519: JSON Web Token (JWT), RFC7515: JSON Web Signature (JWS), RFC7516: JSON Web Encryption (JWE), RFC7517: JSON Web Key (JWK), RFC7518: JSON Web Algorithms (JWA), and RFC7797: JSON Web Signature (JWS) Unencoded Payload Option.

IANA's JOSA page contains lists of registered header parameter names and algorithm names.

Peyrott's JWT Handbook is worth reading. This book includes JavaScript code with explanations, which is a useful reference for developers. Another useful reference is a JWT cheatsheet published by Pragmatic Web Security.

For a simpler approach, developers can use third-party JWT libraries. These are available in many languages. Lists of JWT implementations are available at OpenID and at jwt.io.

Site jwt.io offers a debugger to paste a JWT and view its decoded form. Optionally, signature verification is possible if you include the secret key.

## Milestones

Apr
2001

Douglas Crockford and Chip Morningstar send out what is historically the first JSON message. Since JSON is nothing more than plain JavaScript, Crockford himself states that probably this message format was in use as early as 1996. In July 2006, Crockford describes in RFC 4627 the JSON format and its MIME media type `application/json`.

2005

From the mid-2000s, the growth of Web 2.0 and the use REST APIs in web apps lead to wider adoption of JSON. The term AJAX itself is coined in 2005 but it's clear than AJAX is not limited to XML: JSON can be used instead. By late 2000s, with the increasing use of JSON on the web, it's recognized that standards are needed to offer security services in JSON format.

Dec
2010

As an Internet-Draft, **JSON Web Token (JWT)** is published at IETF. This document goes through multiple revisions, with the final draft revision appearing in December 2014. In May 2015, it becomes RFC7519.

Sep
2011

At IETF, the **Javascript Object Signing and Encryption (JOSE) Working Group** is formed. For better interoperability, the group aims to standardize the mechanism for integrity protection (signature and MAC), encryption, the format of keys and algorithm identifiers.

May
2015

IETF publishes **RFC7519: JSON Web Token (JWT)** as a Proposed Standard. Other relevant RFCs are also published the same month: RFC7515: JWS, RFC7516: JWE, RFC7517: JWK, RFC7518: JWA.

Jul
2015

On Auth0 blog, a new logo for JWT is announced along with a redesigned website at jwt.io. The blog post also notes that interest in JWT has been increasing since 2013. By now, there are 972 JWT-related GitHub repositories and 2600+ threads on StackOverflow. It's also claimed that,

> If you use Android, AWS, Microsoft Azure, Salesforce, or Google then chances are that you are already using JWT.

Feb
2016

IETF publishes **RFC7797: JSON Web Signature (JWS) Unencoded Payload Option** as a Proposed Standard. Typically, JWT payload is Base64-URL encoded. This document gives the option of skipping this encoding step. For example, a payload `$.02` is sent as it is rather than sending its encoded form of `JC4wMg`. Header parameter "b64" controls the use of this option and "crit" parameter facilitates backward compatibility.

Jul
2019

JWTs stored in local or session storage are vulnerable to XSS attacks. On the other hand, JWTs stored in cookies are vulnerable to CSRF attacks. One blogger proposes to mitigate the risks by storing the signature in a HttpOnly, SameSite, Secure cookie. JWT header and payload are in local storage and transferred in HTTP header as a bearer token. The application server has to assemble the complete JWT from its parts. HttpOnly cookies are inaccessible to JavaScript.

JSON Web Token