# Attention Mechanism in Neural Networks

## Summary

In machine translation, the encoder-decoder architecture is common. The encoder reads a sequence of words and represents it with a high-dimensional real-valued vector. This vector, often called the **context vector**, is given to the decoder, which then generates another sequence of words in the target language. If the input sequence is very long, a single vector from the encoder doesn't give enough information for the decoder.

Attention is about giving more contextual information to the decoder. At every decoding step, the decoder is informed how much "attention" it should give to each input word.  While attention started this way in sequence-to-sequence modelling, it was later applied to words within the same sequence, giving rise to self-attention and transformer architecture.

Since the late 2010s, attention mechanism has become popular, sometimes replacing CNNs, RNNs and LSTMs.

## Discussion

### Could you explain attention with an example?

Consider an example from machine translation. The sentence "The agreement on the European Economic Area was signed in August 1992" is to be translated to French, which might be "L'accord sur la zone économique européenne a été signé en août 1992". We can see that "Economic" becomes "économique" and "European" becomes "européenne", but their positions are swapped. The phrase "was signed" becomes "a été signé". Thus, translation depends not just on individual words but also their context within the sentence. Attention is meant to capture this context.

In this example, attention is passed from the encoder to the decoder. The decoder generates the translated words one by one. Each output word is influenced by all input words in different amounts. Attention captures these weights.

We can also visualize attention via heatmaps. In the figure, we map English words to translated French words. We note that sometimes a translated word is attended to by multiple English words. Lighter colours represent higher attention.

### Could you describe the architecture of attention?

Let's consider machine translation as explained by Bahdanau et al. (2014). Encoder is a bidirectional RNN while the decoder is an RNN. The input sequence is fed into the encoder whose hidden states are exposed to the decoder via the attention layer. More specifically, the backward and forward hidden encoder states are concatenated. These states are weighted to give a context vector that's used by the decoder. Attention weights are calculated by aligning the decoder's last hidden state with the encoder hidden states.

The decoder's current hidden state is a function of it's previous hidden state, previous output word and the context vector. Attention is passed via the context vector, which itself is based on the alignment of encoder and decoder states.

Luong et al. proposed a slightly different architecture. Their encoder and decoder are each a 2-layer LSTM. It also uses a feedforward network for the final output. In Google's Neural Machine Translation, 8-layer LSTM is used in encoder and decoder. The first encoder layer is bidirectional. Both encoder and decoder include some residual connections.

### What do you mean by "alignment" in the context of attention mechanism?

Bahdanau et al. align the decoder's sequence with the encoder's sequence. An alignment score quantifies how well output at position *i* is aligned to the input at position *j*. The context vector that goes to the decoder is based on the weighted sum of the encoder's RNN hidden states \(h\_j\). These weights come from the alignment. Mathematically, given an alignment model *a*, alignment energy *e*, context vector *c*, and weights α, we have:

$$e\_{ij} = a(s\_{i-1},h\_j)\\\alpha\_{ij} = exp(e\_{ij})/\sum\_{k=1}^{T\_x}{exp(e\_{ik})}\\c\_i = \sum\_{j=1}^{T\_x}{\alpha\_{ij}h\_j}$$

The decoder's hidden state is based on it's previous hidden state \(s\_{i-1}\), the previous predicted word and the current context vector. At each time step, the context vector is adjusted via the alignment model and attention. Thus, at step step, the decoder selectively attends to the input sequence via the encoder hidden states.

Bahdanau et al. concatenated the forward and backward encoder hidden states and added these with decoder hidden state. Luong et al. proposed many other alternative alignment scores. Vaswani et al. proposed the scaled dot product.

### What is self-attention?

Self-attention is about attending to words within the sequence, such as within the encoder or decoder. By seeing how one word attends to other words in the sequence, we're able to capture syntactical structures.

Consider the sentence "The animal didn't cross the street because it was too tired". The word "it" refers to the animal. What happens if we replace "tired" with "wide"? The word "it" now refers to the street. Attention understands this. In the former case there's high attention linking "it" and "animal" but in the latter case high attention shifts to "street".

Self-attention was earlier applied together with RNNs. Later, self-attention came to stand on its own. Vaswani et al.'s paper titled *Attention is all you need* showed how we can get rid of CNNs and RNNs. RNNs in particular are hard to parallelize on GPUs, which is a problem solved by self-attention.

### Could you describe some applications of attention mechanism?

Beyond its early application to machine translation, attention mechanism has been applied to other NLP tasks such as sentiment analysis, POS tagging, document classification, text classification, and relation classification. One research used human eye-tracking corpora to derive attention and enhance NLP tasks. In another study, semantic role labelling was improved using linguistically-informed self-attention.

By combining CNN with self-attention, the Google Brain team achieved top results for image classification and object detection. In Visual Question Answering (VQA), where there's a need to focus on small areas or details of the image, attention mechanism is useful. Attention is also useful for image captioning.

In speech recognition, attention aligns characters and audio.

In one medical study, higher attention was given to abnormal heartbeats from ECG readings to more accurately detect specific heart conditions. In another study based on ICU data, feature-level attention was used rather than attention on embeddings. This provided physicians better interpretability.

### What's the difference between global and local attention?

The distinction between global versus local attention originated in Luong et al. (2015). In the task of neural machine translation, global attention implies we attend to all the input words, and local attention means we attend to only a subset of words.

It's said that local attention is a combination of hard and soft attentions. Like hard attention, it focuses on a subset. Like soft attention, it's differentiable and hence easier to implement and train. It's computationally simpler than global or soft attentions.

Given the decoder's current state, local attention first selects the best aligned position \(p\_t\) in the input sequence. Note that selecting \(p\_t\) is not directly influenced by the encoder's states. Local attention is also called **window-based attention** because it's about selecting a window of input tokens for attention distribution. This window is centred on \(p\_t\). To keep the approach differentiable, a Gaussian distribution is applied on the window. Attention \(a\_t\) is therefore focused around \(p\_t\).

## Milestones

Jun
2014

Even before attention mechanism becomes popular via NLP in later years, it's used in computer vision. Mnih et al. propose a method to focus on important parts of an image that are then processed at high resolution. Instead of processing the entire image at once, it's processed sequentially, attending to different locations as is relevant to the task.

Sep
2014

Bahdanau et al. apply the concept of **attention** to the seq2seq model used in machine translation. This helps the decoder to "pay attention" to important parts of the source sentence. Encoder is a bidirectional RNN. Unlike earlier seq2seq models that use only the encoder's last hidden state, attention mechanism uses all hidden states of encoder and decoder to generate the context vector. It also aligns the input and output sequences, with alignment score parameterized by a feed-forward network.

Feb
2015

Xu et al. propose the use of visual attention to the task of image captioning. They distinguish between **soft attention** and **hard attention**. Soft deterministic attention is smooth and differentiable, and is trained by standard back propagation. Hard stochastic attention is trained by maximizing an approximate variational lower bound. Soft attention is similar to Bahdanau et al.'s proposal.

Mar
2015

Sukhbaatar et al. propose the concept of **multi-hop attention**. Each hop or layer contains attention weights. Input and output from each layer is fed to the next higher layer. Thus, no hard decisions are taken in each layer. Outputs from each layer are passed on in a "soft" manner until prediction after the last layer.

Aug
2015

Luong et al. distinguish between **global attention** versus **local attention**. In global attention, we attend to all the input words. In local attention, we attend to only a subset of words.

Nov
2015

Attention mechanism has been applied to computer vision. For **Visual Question Answering (VQA)**, Chet et al. propose Attention-Based Configurable Convolutional Neural Network (ABC-CNN). The query text guides the model to pay attention to relevant image regions. In traditional VQA models, visual processing and question understanding are done separately.

Jun
2016

Yang et al. propose the **Hierarchical Attention Network (HAN)**. This comes from their insight that documents have a hierarchical structure (words, sentences, document). HAN has two levels of attention: word level and sentence level. This enables the model to more accurately do document classification. It improves on the state-of-the-art when tested on Yelp, IMDb and Amazon datasets. Kränkel and Lee provide a good description of HAN.

Nov
2016

Matthew Honnibal describes what he calls as the new neural network playbook for NLP. It's a four-step approach: **embed, encode, attend, predict**. This highlights the importance and usefulness of attention mechanism. Word embeddings do the embed part at word level. Bidirectional RNNs do the encode part at sequence level. Using a context vector, attend part produces a vector that's given to a feed-forward network. The predict part is done by this network.

Nov
2016

For the task of machine reading, Cheng et al. propose the use of both inter-attention (between encoder and decoder) and intra-attention (within encoder or decoder). **Intra-attention** (later to be called self-attention) is about attending to tokens within a sequence, thus uncovering lexical relations between tokens.

Jun
2017

Vaswani et al. propose the **transformer** model in which they use a seq2seq model without RNN. The transformer model relies only on **self-attention**. Self-attention is about attending to different tokens of the sequence. Self-attention leads to powerful language models including BERT (2018) and GPT-2 (2019).

Oct
2017

Veličković et al. apply attention to nodes in a graph. Nodes attend to neighbours. The computation can be parallelized.

Attention Mechanism in Neural Networks


# Exploratory Data Analysis

## Summary

Exploratory Data Analysis (EDA) consists of techniques that are typically applied to gain insight into a dataset before doing any formal modelling.

EDA helps us to uncover the underlying structure of the dataset, identify important variables, detect outliers and anomalies, and test underlying assumptions. With EDA, we identify relevant variables, their transformations, and interaction among variables with respect to the model we want to build. EDA can also point out missing data as may be relevant to building desired models.

EDA uses techniques of *statistical graphics* but has a broader scope. It's an approach rather than just a set of techniques. The general idea is,

> Let the data speak for themselves... Exploratory Data Analysis is not “fishing” or “torturing” the data set until it confesses.

## Discussion

### What's the recommended process for doing Exploratory Data Analysis?

One can follow these steps:

    + Look at the structure of the data: number of data points, number of features, feature names, data types, etc.
    + When dealing with multiple data sources, check for consistency across datasets.
    + Identify what data signifies (called measures) for each of data points and be mindful while obtaining metrics.
    + Calculate key metrics for each data point (summary analysis): a. Measures of central tendency (Mean, Median, Mode); b. Measures of dispersion (Range, Quartile Deviation, Mean Deviation, Standard Deviation); c. Measures of skewness and kurtosis.
    + Investigate visuals: a. Histogram for each variable; b. Scatterplot to correlate variables.
    + Calculate metrics and visuals per category for categorical variables (nominal, ordinal).
    + Identify outliers and mark them. Based on context, either discard outliers or analyze them separately.
    + Estimate missing points using *data imputation techniques*.

### What are the data types used in EDA?

In statistics and Machine Learning, data types are also called *levels of measurement*. Four common ones are used:

    + **Nominal**: This is qualitative, not quantitative; eg. Religious Preference: 1 = Buddhist, 2 = Muslim, 3 = Christian, 4 = Jewish, 5 = Other.
    + **Ordinal**: An ordinal scale that indicates ordering or direction in addition to providing nominal information; eg. Low/Medium/High or Faster/Slower are examples of ordinal levels of measurement. Ranking an experience as a "nine" on 1-10 scale tells us that it was higher than an experience ranked as a "six".
    + **Interval**: Interval scales provide information about order, and also ability to compare ranges; eg. temperature measured either on a Fahrenheit or Celsius scale: measured in Fahrenheit units, the difference between a temperature of 46 and 42 is the same as the difference between 72 and 68.
    + **Ratio**: In addition to possessing the qualities of nominal, ordinal, and interval scales, a ratio scale has an absolute zero, a point where none of the quality being measured exists; eg. income, years of work experience, number of children.

### What are measures of central tendency?

A measure of central tendency is a single value that attempts to describe a set of data by identifying the central position within that set of data. These include the following:

    + **Mean**: Mean is equal to the sum of all the values in the data set divided by the number of values in the data set. This is also called *arithmetic mean*. Other means such as *geometric mean* and *harmonic mean* are also sometimes useful.
    + **Median**: Median is the middle score for a set of data that has been arranged in order of magnitude. For example, given an ordered list of student marks, [14 35 45 55 55 56 58 65 87 89 92], median is 56 because it is the middle mark since there are 5 items before it, 5 items after it.
    + **Mode**: Mode is the most frequent score in our data set. For the above data set of student marks, mode is 55 because 55 is repeated for the maximum number of times.

### What are measures of dispersion?

Measures of dispersion are important for describing the spread of the data, or its variation around a central value.

**Range** is the difference between the smallest value and the largest value in the data set. This is the simplest measure but it's based on extreme values and tells nothing about the data in between.

**Standard Deviation** is therefore a better measure. A value within ±1 SD from mean is considered normal; a value beyond ±3 SD is considered extremely abnormal. One alternative to this is a simple measure called **Mean Absolute Deviation (MAD)**. Another alternative, often used as a measurement of error, is **Root Mean Square Anomaly (RMSA)**.

If one desires the spread of data around the central region of data, **Quartile Deviation** is a good measure. This is half of what's called **Interquartile Range (IQR)**. A variation of this that considers all data is called **Median Absolute Deviation (MAD)**.

### What is Skewness And Kurtosis?

**Skewness** is a measure of symmetry, or more precisely, the lack of symmetry. A distribution, or data set, is symmetric if it looks the same to the left and right of the central point.

**Kurtosis** is a measure of whether the data are heavy-tailed or light-tailed relative to a normal distribution. That is, data sets with high kurtosis tend to have heavy tails, or outliers. Data sets with low kurtosis tend to have light tails, or lack of outliers.

### Can measures of central tendency, dispersion, skewness and kurtosis be the same for different datasets?

Yes, it's possible. Statistician Francis Anscombe came up with four datasets to illustrate the importance of graphing data before analyzing it, and to show the effect of outliers on statistical properties. This is now called *Anscombe's quartet*. It comprises of four datasets that have nearly identical simple statistical properties, yet appear very different when graphed. Each dataset consists of eleven (x,y) points.

Anscombe's quartet emphasizes the importance of looking at your data, not just the summary statistics and parameters you compute from it.

### What are outliers and how to handle outliers?

Any observation that appears to deviate markedly from other observations in the sample is considered an outlier. Identifying an observation as an outlier depends on the underlying distribution of the data. Determining whether an observation is an outlier or not is a subjective exercise.

Context dictates whether to focus on or get rid of outliers. For example, in an income distribution, a luxury brand company would focus on the outliers (the rich people) while a Government public distribution system would choose to get rid of the outliers. It's recommended that you generate a normal probability plot of the data before applying an outlier test.

Outliers can also come in different flavours, depending on the environment: point outliers, contextual outliers, or collective outliers.

### What are the visual aids for exploratory analysis?

Data can be represented visually in many ways with programming languages and visualization packages. Programming languages such as R, Python, Matlab, SAS, etc. provide libraries for creating data visuals. In JavaScript, we have D3.js, NVD3, FusionCharts and Chart.js. In Python, we have Matplotlib, Seaborn, Bokeh and Plotly.

There are dedicated visualization platforms such as Tableau, Qlikview, and PowerBI in the market that even non-programmers and traditional data analysts can use to make visuals.

Histograms and scatterplots are widely used for exploratory analysis to quickly understand the structure of data and inter-relations of variables. However, numerous other charts can be used to create visuals that have repeat purpose and long shelf life.

### What should we look for in a histogram or a distribution?

**Histogram** represents the underlying structure in the form of a *frequency distribution*; that is, how often a particular value occurs. Visually, a histogram is similar to a bar chart. While a bar chart has bars for individual values, in a histogram it's more common to group together a range of values into a single *bin*. Often 5-15 bins should be considered depending on the range of values in the dataset. With too few bins, the graph will not be detailed enough to interpret the distribution.

In fact, due to binning, histograms can plot both categorical and continuous variables. Bar charts are only for categorical variables.

Histograms help us see data symmetry, peaks, outliers or data error through omission. In the figure, two peaks imply two distinct classes. Additional data informs us that the peaks are due to gender differentiation. If we split the data by gender, we will get two histograms, each with a single peak. Thus, when we see multimodal histograms (more than one peak), there's room to split the data. For every peak, we can build a different model.

### What should we look for in a scatterplot?

**Scatterplot** is a mechanism to plot two variables and see the *underlying relationship* between them. A scatterplot can reveal data symmetry, clusters, correlation between variables, and extreme values or outliers. The plot is a series of dots "scattered" in two dimensions. Often a line is drawn across these dots. The line doesn't connect the actual points unlike a line graph. The line, often called **regression line**, shows the trend and can be used as a predictive tool.

A scatterplot is two-dimensional (two variables) while a histogram is one-dimensional (one variable). Hence we should pay more attention to outliers in scatterplots. For example, in the accompanying image, Employee #2 and Employee #19 are both outliers when we consider their test scores and sales performance. However, if we analyze the data in either of these variables separately, they will not appear as outliers.

In technical jargon, histogram provides *Univariate Visualization*. Scatterplot provides *Bivariate Visualization*.

### What's a pair plot and what's its utility?

**Pair plot** is a plot that helps comprehend the underlying structure of a variable and its relationship with other variables in a single visual. Basically, it's a combination of histogram and scatterplot in one visual. This can help us notice patterns that may not be obvious when analyzed separately.

### How do we handle missing data?

Data is rarely complete and may have missing points. Data can be missing due to various reasons: not captured, captured but may not be available, etc. In such circumstances, it's normal to estimate the missing value and proceed with analysis. This process is called **imputation**. There are many standard imputation procedures and algorithms to estimate missing data.

## Milestones

1855

John Snow uses a dot plot on a map of London to analyze the 1854 cholera outbreak. He suspects water contamination at the Broad Street pump. The mapped data presents a compelling visual that this could be true. Although not strictly EDA, this is an example of using data visualization to confirm a hypothesis.

1869

Dmitri Mendeleev organizes known chemical elements into a periodic table. This visual suggests some undiscovered elements. This is a good example of EDA leading to new discoveries.

1885

Francis Galton creates a bivariate frequency chart that evolves later to today's more familiar correlation diagram. He uses it to analyze the relationship between the heights of parents and adult children. In earlier experiments from the 1870s, he did a similar correlation study with sweet-pea seeds.

1905

Karl Pearson proposes the *kurtosis coefficient* as a way to measure the degree of flatness of frequency distributions. Along with *skewness coefficient* proposed earlier, he challenges the notion that most distributions are normal or should be transformed to normality. Instead, we should accurately represent observed data.

1973

Statistician Francis Anscombe constructs the *Anscombe's quartet* to demonstrate the importance of graphing data before analyzing it and the effect of outliers on statistical properties.

1977

John W. Tukey, often considered the father of EDA, publishes "Exploratory Data Analysis" at a time when computer-aided visualization was still nascent. He introduces new plots such as the stem-leaf plot and the five-point boxplot. He implies that *Confirmatory Data Analysis (CDA)* can suffer from confirmation bias due to predetermined hypothesis. EDA is a more open-minded approach to discover patterns in data and to answer specific scientific questions.

1999

Just as languages have grammar, Leland Wilkinson formalizes a grammar for making graphs. Called **Grammar of Graphics**, it defines a structure to combine graph elements so that data can be shown in meaningful ways. This later inspires others to implement the same in popular languages (R, Python, Julia, D3).

Exploratory Data Analysis


# Web Annotation

## Summary

In traditional print media, it's common practice among readers to underline text, highlight sections or write comments in the margins. Such annotations allow readers to express their opinions. **Web Annotation** is a standard for creating similar annotations on the web for digital content. It's standardized by the *W3C Web Annotation Working Group*.

Since the early years of the Web, there have been many proprietary systems to enable online annotations. Web Annotation offers a standardized data model, vocabulary, protocol and embeddings to create, organize and share annotations. Web Annotation allows anyone to express their opinions freely without being censored by content authors or publishers. It reuses concepts and tools from Semantic Web.

## Discussion

### Without Web Annotation, how do people annotate content on the web?

Web content involves authors, content publishers and readers. To engage readers, content publishers enable comments on their site. However, they could moderate the comments and even delete them if they didn't like them. This control from publishers meant that the system was not truly open from a reader's perspective. In fact, some publishers don't even allow comments.

Published comments often appear at the end of the article and therefore remain disconnected from a particular line or paragraph to which the comment applies. There's also no distinction between a valuable comment and frivolous chit-chat. Readers also have no way to archive their comments or take their comments from one platform to another. Readers may be identified only loosely and there's no reputation model to say who's a more valuable commenter. The lack of a reputation model was one reason why project Dispute Finder failed.

Before Web Annotation, solutions were proprietary and closed systems. They didn't partner with organizations such as W3C to take a standards-driven approach.

### What are some use cases and current adoptions of Web Annotation?

Annotations can be used to "provide a trace of use; third party commentary; information sharing; information filtering; semantic labeling of document content; and enhanced search". Annotations can help readers discover new content by subscribing to annotation feeds. They can also share annotations, thereby creating communities of common interests. Publishers can use annotations to add value to their content.

Many systems are using annotations for online collaboration. Examples include Kami and GoodReader. For greater insight, DocumentCloud promises to turn documents into data.

For engaging online communities in open discussions, RapGenius started in 2009 to annotate rap lyrics but later expanded to other topics and changed its name to Genius in 2014. Publisher John Wiley & Sons, Inc. and IIIF Consortium have adopted annotations. Mendeley (acquired by Elsevier) provides research management system that includes annotations.

To flag fake news, annotations can help in building an ecosystem for fact checking. Web Annotation can enable "decentralized, trustworthy mechanism for fact checking and public discussion". Beyond facts, Climate Feedback is using annotations for content reasoning and argumentation.

### What annotation tools or services are currently available?

Since the early 1990s, many systems have come and gone to support annotations on the web. Hypothes.is has shared a list of annotation efforts. Among the more widely used services are Diigo, Mendeley, DocumentCloud, RapGenius, Good Reader and Notable PDF. A curated list of ten annotation tools from April 2017 includes zipBoard, UserSnap, PageProofer, JIRA Capture, BugHerd, Scrible, Hypothesis, Diigo, TrackDuck and Twiddla.

Mosaic browser that was released in 1993 had support for annotations. Third Voice was an annotation service during 1999-2001. Predating this, there have been other open-source annotation apps: CritSuite, JotBot, ComMentor and Xanadu. Angel-funded Fleck existed during 2006-2008. At the 2013 I Annotate Conference, many new services were presented: Domeo, Maphub, Pelagios, Authorea, dotdotdot, Hypothes.is.

### What's the architecture behind annotations?

There are two ways to implement annotations:

    + **Proxy-based**: A proxy server merges web content and annotations. Browsers see the merged content. CritLink and InterNote are examples.
    + **Browser-based**: Original web content and annotations are merged at the browser end. This may be done via a browser plugin or by JavaScript served from the annotation server. Third Voice is an example of a plugin. JotBot uses a Java applet. Yawas uses internal DOM events to display annotations.Proxy-based approach is restrictive since readers must access a IRI different from the original one. Browser-based approach is preferred. As examples, Hypothes.is and Pundit take the latter approach. It's expected that eventually browsers will natively support Web Annotation.

It's important to note that annotations are not necessarily stored at the publisher's server. They could be stored in a separate annotation server. Any third party can offer an annotation service that collects and organizes annotations. Such a service must be neutral to realize the ideals of an open collaborative web.

### Could you briefly explain the Web Annotation Data Model?

An annotation is a rooted directed graph that establishes relationships among resources. A resource can be either a **Body** or a **Target**. Annotations can have 0 or more bodies and 1 or more targets. Content of the body are "about" the target. An annotation, body or target may have its own properties and relationship such as creation or descriptive information.

Since resources are distributed on the web, they are identified using IRI. In some cases, Body is just textual content and may be included as part of the annotation. In such cases, separate IRI is not required for the Body.

We often want to select part of a resource and not the entire content at the IRI. This is called **Segment (of Interest)**. A **Selector** is used to extract the segment from a resource. For example, selectors are available to select some region of an image; an exact quote; content that matches a CSS or XPath rule; some text by its start and end positions; and so on.

### What are the W3C Recommendations for Web Annotation?

We have the following Recommendations and Working Group Notes:

    + Web Annotation Data Model, REC: Describes the underlying Annotation Abstract Data Model as well as a JSON-LD serialization.
    + Web Annotation Vocabulary, REC: The Vocabulary which underpins the Web Annotation Data Model.
    + Web Annotation Protocol, REC: The HTTP API for publishing, syndicating, and distributing Web Annotation.
    + Embedding Web Annotation in HTML, WG Note: Various ways annotations can be added to an HTML file using current specifications like JSON-LD or RDFa.
    + Selectors and States, WG Note: An extract from Web Annotation Data Model to make the use of these classes easier and more broadly usable by other specifications.

### What are the tools available to implement Web Annotation?

Annotator is an open-source JavaScript library that can be added to any website and thus enable annotations on it. Annotator can also be extended with plugins. Annotorious is one such plugin for image annotation. Annotator is being used in many projects: Hypothes.is, Harvard's Open Video Annotation Project, EdX, MIT's Annotation Studio, WritingPod, Crunched Book and many more.

Hypothesis is based on Annotator and is also open source. While Annotator used to store annotations at *annotateit.org*, this service closed in March 2017. Hypothesis is also a service that stores annotations but you can deploy your own servers instead.

Pundit Annotator is based on AngularJS. It comes with open source client code. You can be download and install the server code.

### What are the elements of an interoperable annotations layer?

Annotations may be considered as an extra layer of information on top of web content. To accelerate the development of a pervasive interoperable annotation layer, the *Annotating All Knowledge Coalition* identified what such a layer should contain.

An interoperable annotations layer should be standardized. It should be an open framework. Annotations should be as granular as possible: parts of an image, specific lines of text, etc. It should support discovery and linking of annotations across different content formats (HTML or PDF). Likewise, content may have different versions and annotations should persist across versions.

Annotations can be private or public. Identities should be managed. To allow discovery and sharing of annotations, standard identifiers should be used. Notifications should be possible, such as notifying publishers when their content is annotated.

Readers should be able to selectively choose which annotations they wish to see when reading content. Annotations should be portable across services. It should be possible to link to other resources or programmatically annotate. Further tools can enable tagging and filtering of annotations.

### What are the challenges ahead for Web Annotation?

Interoperatibility is a challenge. Annotations created with one tool should be compatible with another. In other words, can we export annotations from one and import into another? When a browser has multiple tools installed, these tools should coexist in terms creating, editing and anchoring annotations. One study annotated a webpage with both Hypothesis and Pundit. It then found that the tools are incompatible and somewhat conflicting.

## Milestones

1989

Tim Berners-Lee publishes his ideas about the World Wide Web and hypertext. This paper includes thoughts about annotations, some of which could be kept private. Thus, the idea of annotating web content is not something that came later.

1993

The first widely distributed web browser Mosaic is released by Marc Andreessen and Eric Bina. **Mosaic v1.1** includes client-side support for annotations. However, this is considered as a "nice-to-have" feature and is not developed further.

1999

As a browser plugin, **Third Voice** is launched. It allows users to annotate any website and thereby have open discussions without being controlled by content publishers. However, this service gathers bad reputation for enabling "web graffiti". Getting users to download and install the plugin becomes a barrier. The service closes in April 2001.

Jul
2001

Where Third Voice fails, W3C attempts with **Annotea** that's built into W3C's Amaya browser. W3C proposes to adopt and reuse concepts and tools from Semantic Web, such as RDF and XML.

2009

Two groups, Annotation Ontology and the Open Annotation Collaboration, independently start work on annotation specifications. In 2011, they join together to form W3C Open Annotation Community Group.

Jul
2011

Version 1.0.0 of **Annotator** is released. This is a JavaScript library for building annotation applications in browsers. Version 1.2.10 is released in February 2015.

Oct
2011

As an open source non-profit platform, **Hypothes.is** is announced. It aims to be community moderated, neutral and transparent.

Feb
2013

Open Annotation Community Group at W3C publishes a Community Draft titled Open Annotation Data Model. This specifies an RDF-based data model for exchanging annotations between applications. This is not a standard but forms the starting point for future standards.

Apr
2013

What is today an annual event, the first I Annotate Conference is held in San Francisco. Dan Whatley of Hypothes.is introduces,

> Annotation as the 3D printing of the web: the promise of decentralization of knowledge creation online.

Oct
2014

Built on Annotator, **Hypothesis** application is launched as a Chrome extension. An alpha version was launched earlier in 2013. In April 2018, they achieve a milestone of 3 million annotations.

Feb
2017

W3C Web Annotation Working Group publishes three Recommendations that define Web Annotation. In addition, two Working Group Notes are also published. This Working Group was formed in 2014.

Web Annotation


# CSS Design Patterns

## Summary

To become a good web designer/developer, it's not enough to know CSS properties and their values. We need to know the contexts in which each property-value pair is applicable, how results differ in different contexts, and how various properties interact. Rather than memorize hundreds of rules and their exceptions, it's more efficient to get familiar with contexts of usage and combinations of properties applicable in those contexts. This is what CSS design patterns is all about.

Design patterns are common in programming, such as in object-oriented programming. In HTML/CSS, design patterns are those that work across browsers and devices. They manage complexity, mitigate code bloat, and improve productivity. They're the building blocks that facilitate the creative process: the designer picks known patterns, adjusts them to current context and combines them for a final result.

## Discussion

### What are the design patterns pertaining to the CSS Box Model?

Common ways to display an element are `inline`, `inline-block` and `block`. Special ways include `list-item` and `table`. We can also take out an element from its normal flow using CSS properties `position` and `float`. More sophisticated ways to layout elements include CSS Flexbox, CSS Grid, and CSS Multi-column.

Bowers et al. talk about three patterns that affect box dimensions:

    + **Sizing**: manually assign absolute or relative sizes
    + **Shrinkwrapping**: shrink to fit the content
    + **Stretching**: stretch to fill its parent containerThese can be combined with positioning patterns **indenting**, **offsetting**, and **aligning**. Specifically, a stretched element can be indented and this affects its dimensions. A sized or shrinkwrapped element can be offset or aligned without affecting its dimensions.

Much of this is achieved by applying CSS rules to parent and grandparent containers of the element. These rules mainly specify dimensional properties, positional properties and margins.

### What are some traditional CSS patterns for layout design?

Two patterns are not recommended: fixed-width layouts since measurements are absolute and tables since table cells "can't flow" when viewport changes.

One design pattern is to use columns whose widths adjust to viewport width. On smaller screens, columns reflow into rows. This is called **fluid layout** and is achieved using properties `float`, `width`, `min-width` and `max-width`.

In **elastic layout**, overall width is set relative to some design element such as font size. A **hybrid layout** is a combination fixed and fluid/elastic layouts.

Property `width` is actually an element's inner width. Sometimes we wish to fix the outer width in design and then adjust the margin, border and padding without affecting the layout. This is possible by having an element within another element, called **outside-in** pattern. The inner element includes margin, border and padding but the outer element has none.

In fluid layouts, mixing fixed values and percentages is not ideal but can't be avoided. Using percentages isn't possible for borders. It's common to use fixed values for margins and padding, and percentages for width. This problem is solved by the outside-in pattern.

### What responsive layout patterns are possible with CSS Flexbox and CSS Grid?

In 2019, LePage used Flexbox and media queries to implement five layout patterns:

    + **Mostly Fluid**: On small screens, content reflows and columns stack vertically. On bigger screens, margins are adjusted.
    + **Column Drop**: Columns stack vertically on smaller screens.
    + **Layout Shifter**: Content moves around when screen sizes change. Most responsive with many breakpoints but also more complex to maintain.
    + **Tiny Tweaks**: Only small changes involving font sizes, image dimensions, or content offsets.
    + **Off Canvas**: On smaller screens, less frequently used content is kept off-screen but accessible with a single click.Rachel Andrew has shared a number of patterns based on CSS Grid. Typical layouts contain header, footer, sidebar and main content area. An example of nested grid shows positioning of media along with text and caption. She also gives examples of using named lines and areas that are easer for developers to work with. Bryan Robinson has shared a good example of a responsive layout with CSS Grid.

### What CSS design patterns help control spaces?

CSS gives designers plenty of options to control spaces. Margin adds space on the outside of an element. Padding adds space between the element's border and its content. For both of these, each boundary (top, right, bottom, left) can be individually set. Negative margins can remove spaces and cause elements to overlap.

In the flow of text, spacing can be controlled with `letter-spacing`, `word-spacing`, `line-height`, and `text-indent`. In addition, `text-align:justify` will stretch out inter-word spaces. Property `text-indent` doesn't work on inline elements. For hanging indent, we can use negative `text-indent` and positive `padding-left`.

Whitespace can be preserved with `white-space:pre`, which is useful for displaying code snippets in monospaced font. To preserve whitespace and also wrap long lines, use `white-space:pre-wrap`, `white-space:pre-line`, or `white-space:break-spaces`. In addition, `word-break:break-all` and `word-break:break-word` give more control.

### Which are the different categories of CSS design elements?

Sometimes it's helpful to think of CSS design patterns in terms of UI components or widgets. A typical webpage has header, footer, menu, buttons, breadcrumbs, forms, tables, pagination, etc. Common ways of using CSS for these purposes can be gathered into CSS patterns.

The page UI Patterns on CodePen documents many such patterns, although some of them are not pure CSS (they include JavaScript as well).

Phuoc's CSS Layout is another place for CSS patterns. Close to 100 patterns are featured here under the categories of layout, navigation, input, display, and feedback.

### Is it a good idea to adopt a CSS framework?

**CSS frameworks** offer a faster route to benefit from CSS design patterns. Such frameworks often provide many patterns that can be mixed and matched to achieve an intended design. They save us the effort of hand-coding dozens of rules. Well-known frameworks include Bootstrap, Foundation, Bulma, UIkit, Semantic UI, Susy, Materialize, Pure, Skeleton, and Milligram.

Some prefer to choose a small lightweight framework rather than one that tries to do too many things. Smaller frameworks and libraries may also be easier to customize. In any case, be wary of frameworks in which abstractions leak. Some lightweight frameworks include Pure, Milligram, Picnic, Wing, and Chota.

**CSS preprocessors** are also useful in managing the complexity of CSS styles. Features include variables, nesting, mixins, functions and mathematical operations. Well-known preprocessors include Sass, LESS and Stylus. These can be compiled into CSS. In fact, many CSS frameworks make use of preprocessors.

Even when CSS preprocessors are used, developers can't sacrifice a solid understanding of CSS. Preprocessors complement CSS, not replace it.

### What are some CSS anti-patterns that I should avoid?

We note a few CSS anti-patterns:

    + **Undoing Styles**: Use of `float:none`, `padding:0`, or `border-bottom:0` are examples where we're attempting to undo styles set elsewhere. It shows that we added styles too early.
    + **Magic/Hard-coded Numbers**: These make the design brittle to changes and confusing for other developers. Examples, `top:37px` or `line-height:32px`. The latter could be made relative, such as `line-height:1.5em`.
    + **Qualified Selectors**: Selector `ul.nav` means that style can't be reused for navigation menus on other element names. Selector `.nav` is more reusable. On the flip side, we could have selectors that are too broad, such as `ul`. Instead, prefer a class selector.
    + **Loose Class Names**: Selector `.card` doesn't suggest its intended purpose. Instead, `.credit-card-image` is better. Another poorly named selector is `.teal`. It describes a visual attribute and has no semantic value.
    + **Repetitive Key Selectors**: Same key selector appearing in many CSS rules suggests poor design. There's no single source of truth. Instead, use BEM convention.
    + **Complex Rules**: A rule such as `.slider {height:27px; float:left;}` is trying to do too many things. Instead, we could break it up into two separate rules `.slider` and `.left`.

## Milestones

1997

At a workshop, Jenifer Tidwell presents *Common Ground*, a pattern language for human-computer interface design. She later develops this work and publishes it as a book titled *Designing Interfaces* in 2005. This is just an example to highlight that CSS design patterns are related to **UI patterns**.

2001

A simple three-column layout with a top header that works for all browsers is surprising difficult to design. Owen Briggs builds such a layout and shares his styles online under the title *Box Lessons*. In time, this becomes a useful reference for many CSS developers. This is an early example of CSS design patterns.

Dec
2005

Holzschlag writes about the constraints of table-based layouts. She proposes the use of CSS to create more flexible layouts. CSS enables designers design for discrete, semantic elements rather than be limited to grid designs.

Oct
2006

As a CSS preprocessor, **Sass** v0.1.0 is released to help developers manage complex styles. Subsequently, other preprocessors are released: LESS v1.0 (Apr 2010) and Stylus v0.02 (Jan 2011). Adoption of preprocessors grows through the 2010s. A survey from 2016 reveals that about 86% of respondents use a CSS preprocessor in their development workflow.

Mar
2009

Marcotte proposes **Fluid Grids** as a layout pattern in which all design elements are sized in proportion to font size. This overcomes the assumption of "minimum screen resolution" and creating fixed layouts. In later years, this is called **elastic layout**.

Aug
2011

Twitter open sources **Bootstrap**, what's possibly the world's first framework for frontend design. For pragmatic developers who wish to get things done, patterns in Bootstrap speed up development. Subsequently, Bootstrap 2 (Jan 2012) adds responsive functionality, Bootstrap 3 (Aug 2013) becomes responsive by default and mobile first, Bootstrap 4 (Jan 2018) moves from LESS to Sass and uses CSS Flexbox, and Bootstrap 5 Alpha (Jun 2020) replaces jQuery with vanilla JS and card decks with CSS Grid.

2011

Bowers et al. publish the book *Pro HTML5 and CSS3 Design Patterns*. They describe more than 350 design patterns, each pattern being modular, customizable, and reusable. Patterns come with working code in HTML and CSS.

Mar
2012

Based on a study of layout examples on Media Queries website, Wroblewski identifies five high-level patterns: Mostly Fluid, Column Drop, Layout Shifter, Tiny Tweaks, and Off Canvas. Years later, in 2019, LePage showcases how these patterns can be implemented using CSS Flexbox and media queries.

Sep
2012

W3C publishes the first Candidate Recommendation of **CSS Flexible Box Layout Module** (first draft in 2009). In May 2017, W3C publishes **CSS Grid Layout Module Level 1** as a Candidate Recommendation (first draft in 2012). Both Flexbox and Grid greatly simplify CSS rules for layouts. In time, developers share patterns for these two approaches.

Sep
2018

Bob Myers argues that BEM, CSS preprocessors, CSS frameworks, and obsessive separation of content and presentation are all CSS anti-patterns. CSS engines are faster today than when BEM came out. Preprocessors complicate build pipelines and require developers learn their syntax. Frameworks such as Bootstrap are not needed now for layout when we have CSS Flexbox and CSS Grid.

CSS Design Patterns


# Quantum Algorithm

## Summary

A quantum computer is a machine that employs quantum mechanics to perform tasks that would be quite challenging for a machine based solely on classical physics laws to accomplish. Quantum computing's future applications include everything from cracking cryptographic systems to developing novel treatments. These applications are based on quantum algorithms, which run on a quantum computer and achieve a speedup or other efficiency advantage over any other classical method.

Quantum computers use quantum algorithms to outperform traditional computers. Cryptography, search and optimization, quantum system modelling, and solving huge systems of linear equations are all areas where quantum methods can be used.

**Shor's algorithm for factoring** and **Grover's algorithm for searching** an unstructured database or an unordered list are two of the most popular search methods. Shor's algorithms are faster (probably more powerful) than the well-known traditional factoring algorithm, the general number field sieve.

## Discussion

### Why do we need Quantum Algorithm?

To do factoring, the well-known quantum algorithm is **Shor's Algorithm**. This method of quantum factoring takes about the same amount of time to function as a whole number of digits, while the older advanced algorithm takes about the exposure time. The notion that complex factoring is critical to functioning of modern cryptosystems, allows for most e-commerce. In these cryptosystems, the user can encrypt information and "eavesdroppers" can only determine the information by entering a large number. This strategy is useful for keeping information confidential as long as there are no quick fixes.

Modeling of chemical and physical interactions is the another known quantum computer program. Because the amount of space required to track a quantum system increases rapidly with the number of particles, quantum mechanics are often difficult to duplicate using a standard computer. This has made understanding of the operation of many important systems and interactions, from advanced superconductors to photosynthesis, difficult for scientists and engineers. Quantum computers, for example, are expected to improve and accelerate drug research and industrial chemical processes by allowing 'in silico'(an experiment that is performed on computer) of chemical compounds and performance.

### What are the Characteristics of a Quantum Algorithm?

Quantum computing techniques can be accomplished using classical computer infrastructure by applying quantum algorithms. It is critical to analyze and categorise quantum computing technologies using quantum algorithms to describe characteristics. Parallelism, aggregate count of accessible qubits, topologies, algorithms for locating qubits, and qubit operations are all algorithmic aspects of quantum computing technology.

Because parallel implementation of quantum gates is essential to either prevent or decrease qubit decoherence, parallelism is a key characteristic. Another characteristic that contributes to the quantum computer's stability and scalability is the total number of qubits available. The topologies of a quantum computer's architecture are the multiple conceivable groupings of distinct physical devices.

The fundamental objective is architecture optimization, which allows for a seamless flow of data and information among the many physical parts of the system. The addressing mechanism for identifying a single qubit is quite complicated logically. This capability allows for a more detailed examination of qubit states in terms of quantum computer physical implementation. Moreover, in order to conduct any operation on qubits, they must be transported from the address where they are stored to the place where the qubit gates are acting on them.

### How can the Quantum Algorithms be classified?

**Quantum Fourier Transform-based algorithms:**

Quantum Fourier transform is a quantum version of the discrete Fourier transform that is utilised in a variety of quantum techniques. In the field F2, the Hadamard transform is an example of a quantum Fourier transform over an n-dimensional vector space. Only a polynomial number of quantum gates are required to accomplish the quantum Fourier transform on a quantum computer.

**Amplitude Amplification-based algorithms:**

Amplitude amplification is a method that enables a quantum state's subdomain to be amplified. It frequently results in quadratic speedups when compared to traditional techniques. That can be regarded as a generalisation of  Grover's algorithm.

**Quantum Walk-based algorithms:**

Quantum walks are a powerful and general framework for creating rapid quantum algorithms. A quantum walk is based on the simulated coherent quantum evolution of a particle moving on a graph, similar to how a random walk method is based on the simulated motion of a particle moving randomly within some underlying graph structure.

**Hybrid quantum/classical algorithms:**

They are predicted to be the first useful applications for quantum computing, as they are well-suited for execution on NISQ (noisy intermediate-scale quantum) devices by integrating quantum computers with classical computers.

### What is a Variational Quantum Algorithm?

Variational Quantum Algorithm(VQA) is a Hybrid type of algorithm, as it is a combination quantum state preparation and measurement with classical optimization. This methodology can be used to locate low-energy eigenstates as well as to minimise any objective function.

The variational method is a classical method for obtaining low energy states of a quantum system in quantum theory. The basic idea behind this method is to build a trial wave function (also known as an ansatz) as a function of some parameters, and then determine the values of these parameters that minimise the energy's expectation value in relation to these parameters. The expectation value serves as an upper bound on the energy of the ground state, and the reduced ansatz is an approximation to the lowest energy eigenstate. As a result, a new type of algorithm called as Variational Quantum Algorithm(VQA) has emerged.

### How can a Quantum Algorithm be designed?

**Yao** is a quantum computation research software that solves real-world challenges. Given the limits of noisy intermediate-scale quantum circuits in the near term, treating quantum devices as co-processors and enhancing their capabilities with classical computing resources is useful. VQAs have emerged as a potential study field. A quantum circuit with configurable gate parameters and a classical optimizer are frequently used in these techniques.

For designing and testing quantum algorithms in these difficult domains, efficient quantum software is critical. Yao is an open-source framework for designing quantum algorithms with exceptional extensibility and efficiency. Quantum circuits can be programmed in a general and differentiable way with Yao. It simulates tiny to intermediate-sized quantum circuits with state-of-the-art performance that is important to near-term applications.

The quantum block intermediate representation of quantum circuits, a built-in automated differentiation engine tuned for reversible computing, and batched quantum registers with GPU(graphics processing unit) acceleration are among the design principles and critical methodologies underpinning Yao.

Besides increasing the number of qubits in experiments, further research objectives ask for quantum software with a low overhead for repetitive feedback control, convenient circuit structure modifications, and rapid gradient calculation.

### How can Quantum Algorithms solve Dynamic Programming Problems?

Quantum algorithms can be used to solve dynamic programming issues with finite and infinite horizons. For the similar jobs, the query complexity lower bounds for classical randomised algorithms are examined, and a polynomial separation between the query complexity of quantum algorithms and the best-case query complexity of conventional randomised algorithms can be demonstrated as a result. Quantum algorithms can provide a quadratic advantage in terms of the number of states and actions in the dynamic programming problem up to polylogarithmic factors. However, the speedup achieved comes at the expense of the introduction of other polynomial factors in the scaling of the algorithm, which contribute to the solution's precision. The quantum algorithm framework is concerned with discrete and combinatorial optimization issues that have traditionally been tackled with dynamic programming techniques.

## Milestones

1992

The **Deutsch–Jozsa algorithm** is one of the first examples of a quantum algorithm that is exponentially faster than any possible deterministic classical algorithm. This is perhaps the earliest result in the computational complexity of quantum computers, proving that they were capable of performing some well-defined computational task more efficiently than any classical computer.

1994

**Peter Shor** of Bell Laboratories develops a quantum algorithm for factoring integers that has the potential to decrypt RSA-encrypted communications, a widely-used method for securing data transmissions.  **Isaac Chuang and Yoshihisa Yamamoto** propose a quantum-optical realization of a quantum computer to implement Deutsch's algorithm. Their work introduces dual-rail encoding for photonic qubits.

1996

**Lov Grover** of Bell Laboratories invents the quantum database search algorithm. The quadratic speedup is not as dramatic as the speedup for factoring, discrete logs, or physics simulations. However, the algorithm can be applied to a much wider variety of problems. Any problem that has to be solved by random, brute-force search, can take advantage of this quadratic speedup (in the number of search queries).

1998

**First experimental demonstration of a quantum algorithm**. A working 2-qubit NMR quantum computer is used to solve Deutsch's problem by Jonathan A. Jones and Michele Mosca at Oxford University and shortly after by Isaac L. Chuang at IBM's Almaden Research Center and Mark Kubinec and the University of California, Berkeley together with coworkers at Stanford University and MIT.[Contributors 2005]
 First execution of Grover's algorithm on an NMR computer.
 Hidetoshi Nishimori & colleagues from Tokyo Institute of Technology showed that quantum annealing algorithm can perform better than classical simulated annealing.

2003

Implementation of the Deutsch–Jozsa algorithm on an **ion-trap quantum computer** at the University of Innsbruck.

2007

First use of Deutsch's Algorithm in a **cluster state quantum computer**.

2009

Quantum algorithm developed for **differential equation systems**.

2013

First resource analysis of a large-scale quantum algorithm using explicit fault-tolerant, error-correction protocols was developed for **factoring**.

2016

Physicists led by Rainer Blatt joined forces with scientists at MIT, led by Isaac Chuang, to efficiently implement Shor's algorithm in an ion-trap based quantum computer.

Quantum Algorithm


# OpenWrt

## Summary

OpenWrt is a Linux-based customizable operating system for embedded devices. Instead of being a static firmware, it's a flexible Linux distribution that allows applications to be added/removed through a package management system without having to rebuild the entire firmware.

OpenWrt started as a means to give users and developers control over router firmware. The suffix "WRT" itself comes from Wireless RouTer. Today, OpenWrt can be used in various embedded devices including Wi-Fi routers, wired routers, residential gateways, smartphones, laptops and even x86-based PCs.

## Discussion

### Why should I use OpenWrt?

Hardware manufacturers normally ship their products with their own firmware. However, such a firmware may be unstable or lack features that you desire. One Reddit user noted many problems with the default firmware: broken port forwarding, security flaws, missing IPv6 support, limited routing support, no patches, no SSH access, etc. Manufacturers may not bother patching old device models. Their firmware may also have government backdoors.

This is where OpenWrt comes in as a suitable alternative. Users can download pre-built binaries and customize the same with necessary packages. Developers can built firmware from source and develop new packages. The code is open source and is maintained by a community of developers. Because source is open, OpenWrt is a useful tool for students and researchers. Moreover, one can get enterprise-class features by installing OpenWrt on a cost-effective hardware.

As of January 2017, OpenWrt is supported for close to 700 different models across brands. In February 2011, OpenWrt had about 2000 packages. In 2017, this number was 3500.

### How are people customizing OpenWrt for their routers?

With OpenWrt, users/developers can use their router to run a BitTorrent client, enable VPN, create a guest Wi-Fi network, analyze network traffic, do traffic-shaping or apply QoS rules on packets. The router can also run servers: SSH (and do SSH tunneling), IRC server, HTTP server, FTP server, etc. Mesh networking, port knocking, firewalling, wireless bridging, file sharing and real-time monitoring are some other useful features. When configured as a public hotspot, OpenWrt provides a number of functions to manage the hotspot.

OpenWrt can also connect to printers, webcams, modems and soundcards. In general, an OpenWrt device can work with any hardware that has Linux support.

### Could you share some technical details of OpenWrt?

OpenWrt is a Linux distribution but because it's optimized for embedded systems, many of the libraries have been customized. uClibc is the C standard library used. Utilities from BusyBox are used though procd has replaced some of them for process management. For package management, opkg is used. It's been claimed that a stripped version of OpenWrt can run with only 8 MiB of main memory and non-volatile storage of 2 MiB.

The firmware is composed of image (kernel + rootfs) and optional packages. Linux, BSD or MacOS are recommended for the host system building the firmware. Robbie Cao has shared a detailed discussion of OpenWrt's build process, installation, configuration, package management system, boot up sequence and memory layout.

### What are some open source alternatives to OpenWrt?

Many of the alternatives themselves are derived from OpenWrt: LEDE, Gargoyle, Fon, libreCMC, Freifunk, LibreMesh, etc. Other alternatives include DD-WRT, pfSense, ClearOS, Tomato (and its derivatives), LibreMesh, telehash, Gluon and OpenWISP. FreeWRT and Chilifire are also available.

While OpenWrt is more of a customizable Linux distribution, DD-WRT is a monolithic build. OpenWrt takes a little more work than DD-WRT for the user but OpenWrt has more features. Tomato is lightweight and strikes a balance between performance and features. However, it supports fewer device variants unlike DD-WRT. DD-WRT is claimed to be the most popular open source router firmware.

### Are there any concerns with using OpenWrt?

Flashing your router with OpenWrt will void the manufacturer's warranty. In some exceptional cases, due to incompatibility, the device may become unusable (bricked) after loading OpenWrt. If device has a button to revert to factory settings, this could be an option to recover it. In any case, flashing a replacement firmware should be avoided via a wireless link. Also, ensure that availability of uninterrupted power supply during the process.

Some routers using open source firmware have been blamed for using 5 GHz spectrum without using Dynamic Frequency Selection (DFS). As a result, they interfere with weather radar systems. Since June 2016, in the US, devices are required to operate within allowed bands, modulations and power levels. Some manufacturers such as TP-Link took the easy path of preventing the use of open source firmware while Linksys still allows them with necessary checks.

Some have claimed that one can boost power levels with open source firmware. However, if done wrongly, this may violate local limits and regulations.

### What tools are available to build, install or configure OpenWrt and its packages?

The development and build environment is called *OpenWrt Buildroot*, which is derived from *Buildroot* system. This allows developers to cross-compile for different target architectures and automate the build process using makefiles and scripts.

*OpenWrt SDK* is needed to build userspace packages. We can download a pre-built SDK or build one ourselves. Package manager *opkg* is used to install/remove/update pre-built packages. If we are building a custom package, OpenWrt SDK can also be used to manage the package.

Configuration is centralized via *Unified Configuration Interface (UCI)*. We can configure OpenWrt from a command console or from a web interface named *LuCI*. Alternative web interfaces include *X-Wrt* and *Gargoyle Router Management Utility*. LuCI includes a package manager page. In fact, some packages have their own configuration pages in LuCI.

### As a developer, how can I contribute to OpenWrt?

OpenWrt is completely open source and community driven. It's released under GPL v2.0 license. The code is available on GitHub. You can either contribute to the main distribution or to the various community-maintained packages. Documentation is on OpenWrt Wiki.

Tasks for developers typically include the following:

    + Update the Linux kernel to more recent versions
    + Fix bugs in kernel or packages: these may include security fixes
    + Update to drivers and networking protocols
    + Support new hardware and new platforms
    + Write new packages
    + Participate in code reviews and testing
    + Improve build, packaging and development tools

## Milestones

2003

Linksys releases router WRT54G with support for IEEE 802.11g standard. Some folks in open source community investigate the firmware and find that it uses open source Linux components. Under pressure from community, Linksys releases firmware source code in July 2003. This is the genesis of OpenWrt, and custom router firmware in general. Sveasoft's Alchemy and Talisman, and BrainSlayer's DD-WRT are based on this firmware.

Jan
2004

OpenWrt project is started. First version is released the same year using Linksys open source firmware for the Linksys wireless router WRT54G.

2008

LuCI project is started to provide a web user interface for embedded devices. It's based on Lua programming language. OpenWrt adopts this interface in Kamikaze 8.09 release. Today, LuCI source code is part of OpenWrt.

Oct
2015

OpenWrt Summit is organized for the first time to discuss OpenWrt technology and support the movement. It happens in Dublin, Ireland.

Mar
2016

Chaos Calmer 15.05.1 is released. This is a maintenance release on Chaos Calmer 15.05 released in September 2015. Note that the number "15.05" indicate that release branch was created in May 2015, although the release was finalized only in September. Next OpenWrt release named "Designated Driver" is under development.

May
2016

*Linux Embedded Development Environment (LEDE)* is born as a fork of OpenWrt. LEDE is started by many active OpenWrt developers unhappy with OpenWrt's processes and lack of transparency, among other things. LEDE promises more open communication, liberal merge policy, automated testing, simplified release process, predictable release cycles and simple infrastructure that requires little maintenance.

May
2017

OpenWrt and LEDE projects meet and plan for a merger.

Jan
2018

OpenWrt code is replaced by code from LEDE as part of efforts to merge the two under the original brand name of OpenWrt.

OpenWrt


# Flask

## Summary

Flask is a simple and minimalist web framework written in Python. It has no database abstraction layer, form validation, or other components that a web app might require. However, Flask can be enhanced with extensions that can add application features as if they were implemented in Flask itself. It's open source under a BSD license.

Flask is easy to set up and learn. One can write a Flask app in as few as seven lines of code and extend it to thousands. Among the big companies that use Flask are LinkedIn, Pinterest, Reddit, and many more.

Flask along with Bootstrap and SQLite can be used to easily develop full-functioning web apps.

## Discussion

### With so many frameworks out there, what's special about Flask?

Flask uses Python and is lightweight. It consumes minimum resources to get the work done. It's very easy to learn once you have a sound knowledge of Python. Flask enables rapid prototyping of your app and works very efficiently for small applications. It enables you to focus more on other jobs than being stuck with web server management.

Among the important features of Flask are a built-in web server and debugger, unit testing support, RESTful request dispatching, secure cookies, WSGI compliance, Unicode support and good documentation.

### How does Flask compare to Django?

Django is a full-featured MVC framework while Flask is a micro-framework. What this means is that Django takes a "batteries-included" approach by providing many packages typically used by web apps. Flask on the other hand let's you bring in packages as required for your project.

Django is a stricter and more mature framework than Flask. Django has it's own ways of doing things, while Flask gives you all the freedom you want. For example, Django comes with an ORM but with Flask you can use any ORM of your choice.

Flask is easier to learn due to its minimalism. Beginners looking to learn a Python-based web framework can start with Flask. For simple apps, Flask is more than adequate. For complex apps or to make a polished product, you can consider migrating to Django. Because Django includes lots of useful stuff by default, it might be faster to develop complex apps. For example, Django comes with an administrator interface that can be customized. This can be useful when building a Content Management System (CMS).

### As a beginner, how do I get started with Flask?

One can refer to the Flask official documentation. The User's Guide contains a quickstart guide, detailed tutorials and useful patterns in Flask.

Week 7 of Harvard's CS50 course is useful if you want a proper classroom type tutoring with assignments. Flask Tutorial Playlist by Corey Schafer is also a good starting point.

Use `pip` to install Flask: `pip install -U Flask`

### Which websites use Flask?

There's an curated list of sites using Flask. Another list mentions big names: Red Hat, Airbnb, Netflix, Lyft, Reddit, Uber, and Samsung. Other big names include Pinterest, Twilio and LinkedIn.

### What are some popular extensions used along with Flask?

Flask can be used to implement an Model-View-Controller (MVC) pattern of web framework: Flask (controller), SQLite (model), Bootstrap (view). Jinja2 is popular for rendering HTML templates and is part of the view. Flask can be used with front-end frameworks such as VueJS but you would have to learn JavaScript.

These are some commonly used extensions:

    + Flask-Cors: A Flask extension for handling Cross-Origin Resource Sharing (CORS), making cross-origin AJAX possible.
    + Flask-User: Customizable user authentication, user management, password recovery, and more.
    + Flask-PyMongo: This bridges Flask and PyMongo. Provides some convenient helpers. PyMongo is a MongoDB driver.
    + Flask-SQLAlchemy: Database abstraction layer and Object Relational Mapper (ORM) popular for Flask apps.

### How do I deploy a Flask application to production?

Flask comes with a simple built-in server. This runs on the default port 5000. This is commonly used for development but not recommended for production because it doesn't scale well.

For production, many cloud providers document procedures on how to deploy Flask on their platforms. If you wish to deploy Flask on your own, `mod_wsgi` is an essential module for Apache server. If Nginx server is used, it will typically serve static files and the rest will be handed over to Gunicorn server, which becomes the WSGI entry point for the Flask application. In fact, Nginx can reverse proxy a HTTP request to any WSGI-compliant server including Apache, Gunicorn or uWSGI.

## Milestones

2004

Pocco is formed as an international group of Python Enthusiasts.

Apr
2010

Coders from Pocco release version 0.1 of Flask.  Flask is really a combination of two other projects developed at Pocco: Werkzeug (a web programming toolkit) and Jinja2 (a templating engine). It's initially called Denied–The next generation micro-web-framework.

Sep
2016

Harvard's popular *CS50: Introduction to Computer Science* course incorporates Flask to teach web development, thus shifting focus from PHP to Python. It becomes available on edX as CS50x from 2017.

Apr
2018

Flask 1.0 is launched. Python versions 2.7 and 3.3 are no longer supported. Thus, Flask requires Python 2.7, Python 3.4 and above, or PyPy.

May
2018

Flask 1.0.2 is released.

Flask


# Package Manager

## Summary

Package Managers are used to automate the process of installing, upgrading, configuring, and removing programs. There are many package managers today for Unix/Linux-based systems. By mid-2010s, package managers made their way to Windows as well. Package managers are also used for installing and managing modules for languages such as Python, Ruby, etc.

A package is simply an archive that contains binaries of software, configuration files, and information about dependencies.

The general workflow starts with the user requesting a package using the package manager (PM) available in the system. The PM then finds the requested package from a known location and downloads it. The PM then installs the package and advises on any manual steps that it finds necessary.

## Discussion

### Why is the package manager required in the first place?

Unix began its journey by being a programmer's OS. This means that every time a new program was written it had to be compiled, linked and run.

Unix got the ability to use libraries ("shared objects"), ELF executables, etc. To solve the task of building more complicated software easily, *make* was developed. Source code was getting shipped with a *Makefile* (the file that's used by *make*). But it was still a laborious task as the developer or the maintainer had to take care of the dependencies.

Instead of running `make` commands every time on every machine having the same configuration, it was thought that we can have a package manager to ship the executable and also the dependencies to other machines. Hence, the earliest PMs started evolving with this idea.

Today's Linux distributions contain thousands of packages. This has come about due to its modular design, code reuse, and collaborative code creation. However, there's a trade-off between code reuse and incompatible dependencies. Package managers solve this complexity by streamlining the process.

### What are the basic functions of a package manager?

The basic functions of the PM are the following:

    + Working with file archivers to extract package archives
    + Ensuring the integrity and authenticity of the package by verifying their digital certificates and checksums
    + Looking up, downloading, installing or updating existing software from a software repository or app store
    + Grouping packages by function to reduce user confusion
    + Managing dependencies to ensure a package is installed with all packages it requires, thus avoiding *dependency hell*The user interface of a PM may be a command line, a graphical interface, or both. Often users can search for packages by name or category. Some even show user reviews or ratings of packages. Batch installation is also possible with PM. Some may support "safe upgrading" (retain existing versions) or "holding" (lock package to a specific version).

### What exactly is Dependency Hell?

Dependency Hell is colloquial term that developers use to indicate their frustration in managing complex inter-dependencies among packages. In Windows, the equivalent term could be *DLL Hell*.

When a package depends on another package as a prerequisite, it will either not install or work incorrectly if the latter is missing or incorrectly set up. A developer may attempt to install the dependency, which in turn may depend on yet more packages. This could quickly become unmanageable if the developer tries to install all these dependencies manually.

It could also happen that a dependent package is installed but it's of an older incompatible version. Two packages A and B might require different versions of package C. This could be a problem if only one version can be installed in the system. Circular dependencies could also cause problems.

Package managers solve this problem by resolving dependencies. Because every package comes with metadata, the PM knows what are the dependencies and what versions of those dependencies ought to be used. Package managers, if properly used, solve the problem of dependency hell.

### From where do packages gets downloaded?

Packages gets downloaded from **software repositories**, often simply called **repos**. Alternatives terms include **sources** and **feeds**. These repos are available online at well-defined locations and they serve as a central distribution point for packages.

For performance and redundancy, these repos may be mirrored by many other locations worldwide. As an example, Cygwin uses mirror sites. Local repos may also mirror remote official repos for saving bandwidth or tighter privacy. Ubuntu's `apt-mirror` provides this feature.

While most developers will use these repos to download packages, advanced developers can also contribute or upload packages to be hosted at these repos. All repos publish the process that developers need to follow to upload packages. Official repos have a strict review and approval process. Community-managed repos may have a more relaxed process. In all cases, repositories are meant to be malware free.

### How would a package manager know the location of the repository?

Every package manager has associated configuration files that point to repository locations. For example, in Ubuntu, `/etc/apt/sources.list` contains the locations of repositories. This would include the official repos but users can also update this file for getting packages from other repos. Likewise, configuration for Fedora and CentOS distributions are at `/etc/yum.conf` for YUM and `/etc/dnf/dnf.conf` for DNF. For Arch Linux, it is at `/etc/pacman.conf` when pacman is used.

When adding third-party repos to a package manager, users must take care to check that those repos can be trusted. This is important so that you don't end up with a malware infecting your system. In fact, this is one of the problems solved by trusted repos. Instead of downloading software from a third-party website, downloading it via the package manager from a trusted repo is a more secure practice.

### What's in a package?

A package includes the concerned software, which may be an application or shared library. If it's a development package, it will include source files (such as header files) to build your software that depends on a library. Packages are meant for specific distributions and therefore installation paths, desktop integration and startup scripts are set up to the targeted distribution. Package formats could include \*.tgz (for source code archives), \*.deb (for Debian) or \*.rpm (for Red Hat).

Packages include metadata as well. This will include summary, description, list of files, version, authorship, targeted architecture, file checksums, licensing, and dependent packages. This metadata is essential for the package manager to do its job correctly.

### Could you give examples of Linux package managers?

All Linux distributions don't use the same package manager. The Debian family, which includes Ubuntu, uses `apt-get` and `dpkg`. Where possible, apt-get should be preferred since it will resolve all dependencies; dpkg doesn't resolve dependencies but it can work directly with \*.deb files. Also, apt-get invokes dpkg for low-level operations. Other relevant Debian tools are `apt-cache` and `aptitude`.

In RedHat, Fedora and SUSE distributions, `rpm` is the low-level package manager. Arch Linux uses pacman. Slackware distribution includes `pkgtools` and `slackpkg` but neither of these resolve dependencies. Slackware takes a unique approach. They distribute packages as intended by original creators. They give full control to administrators by not resolving dependencies.

It's common for distributions to offer graphical interfaces for those users who aren't comfortable with remembering or typing commands. YaST (openSUSE) and Synaptic (Debian) are two examples of GUIs. For those comfortable with commands, look up this handy reference.

### Could you give examples of language package managers?

Modern languages are delivered as a core part that comes with the default installation plus a wide range of optional packages that can be installed when necessary. Those that manage these add-ons are called *language package managers*. Within the scope of a project or application, the term *dependency manager* is used. The term package manager is used at system/language level whereas dependency manager is used at project level. For example, in PHP, PEAR can be called a package manager while Composer is a dependency manager.

Let's say, you're working on a Python project. This may depend on many other Python packages for correct execution. Moreover, another Python project will have its own dependencies. A dependency manager helps developers manage these dependencies and share their project settings in a consistent manner with others.

Here are some examples, in the format of "language: (manager, repository)":

    + Python: (pip, PyPI)
    + PHP: (Composer, Packagist)
    + Ruby: (RubyGems, RubyGems), (Bundler, Bundler)
    + Node.js and JavaScript: (NPM, NPM), (Yarn, (Yarn, NPM))
    + Web frontend: (Bower, Bower), (Yarn, (Yarn, NPM))
    + Java: (Maven, Maven), (Gradle, (Ivy, Maven, etc.))

### What package managers are available for MAC?

Homebrew, MacPorts and Fink are just a few examples of package managers for MAC.

### Does Windows have a package management system?

In 2006, Microsoft Vista got a package manager. Years later, there was Windows Update (with Microsoft Store as repo). These were not very useful since they could manage only Microsoft software. For example, they couldn't update your Node.js or Firefox installation. To solve this, third-party package managers are available: Allmyapps, Chocalatey, Intel AppUp, OpenWrap, NPanday, Chewie, Ninite, and Npackd.

In 2014, Microsoft introduced *OneGet*, which was later renamed to *PackageManagement*. Via the Windows PowerShell interface, this can install and manage even non-Microsoft software. OneGet was available in Windows 8.1 but it's included by default in Windows 10. PackageManagement is able to handle different installer technologies (MSI, MSU, APPX, etc.) and different sources. The default repo used is called PowerShell Gallery.

For .NET developers, *NuGet* is the package manager to use. In addition, when using Visual Studio Team Services and Team Foundation Server, it's easy for developers to manage NuGet, npm, and Maven packages for their project requirements.

### If there are options, how should I choose a package manager?

Here are a few things to consider:

    + **Ease of use**: A graphical interface can be great for beginners. For command-line interface, commands have to be intuitive.
    + **Features**: Look for more than just managing packages: list available/installed packages, search, filter, remote/local installs, wildcard support, source/binary package support, etc.
    + **Customization**: Check if it supports customization such as interactive mode to let user decide on the next step during installation.
    + **Speed**: Faster the better and this might depend on how well caching is done.
    + **Ease of development**: For package developers, the workflow should be easy including a simple upload to a repository.

## Milestones

1978

The concept of using *make* and *Makefile* is invented in Bell Labs by Stuart Feldman. This simplifies the process of building software. The *make* system can read dependencies and thus lays the conceptual framework for package managers that are to appear years later.

1991

David Mackenzie starts work on a tool called *autoconf*. It can look at a system and generate the appropriate configuration/header files and Makefile. These can then be used to build the software. Rather than relying on just versions, it relies on features to check for dependencies.

1992

Some users of TeX (used for typesetting) start working on *Comprehensive TeX Archive Network (CTAN)* for distributing Tex-related material and software. This may be one of the earliest examples of distributing packages over the Internet from a central point. Prior to this, it was common to share software via newsgroups.

1993

FreeBSD 1.0 is released and it comes with what's called *ports tree*, a package building framework that uses *make*. This release includes `pkg_install` and other `pkg_*` tools. In the world of Linux, this is probably the earliest package management system. Support for using remote repos comes in 1999 with version 3.1.

1994

Version 1.0 of Bogus Linux distribution includes a package manager named *Package Management System (PMS)*.

1995

RedHat's well-known *RedHat Package Manager (RPM)* is released with version 2.0 of the distribution. The same year the *Comprehensive Perl Archive Network (CPAN)* is launched as a repository for Perl.

1998

Initial version of Debian's *apt* is released. This is an acronym for *Advanced Packaging Tool*.

2014

Windows introduces *OneGet* that can manage even non-Microsoft software. This later becomes *PackageManagement* and is accessible via the PowerShell interface.

Package Manager


# Database Compression

## Summary

Database compression is a set of techniques that reorganizes database content to save on physical storage space and improve performance speeds. Compression can be achieved in two primary ways:

* **Lossless**: Original data can be fully reconstructed from the compressed data.
* **Lossy**: Reduction in data size due to deliberate compromise in quality.

Depending on the nature and extent of redundancy in the data patterns, the compression method might work within a data field, across a row/column of data, across an entire data page or in hierarchical data structures. Compression methods applied for text/primitive data are different from those for audio-video data.

While most compression algorithms are open standards, companies like Oracle employ proprietary algorithms too. Run-length encoding, prefix encoding, compression using clustering, sparse matrices, dictionary encoding are all popular standard compression techniques used in database applications.

## Discussion

### What are the benefits of database compression?

    + **Size** - The most obvious reason for database compression is reduction in the overall database storage footprint of the organization. It applies relational (tables), unstructured (files), indices, data transfer over the network and backup data. Depending on the data cardinality (extent of repetition in data values), compression can reduce storage consumption from 60% to as low as 20% of the original space. Sparsely populated tables (lots of zeros, spaces in data) compress much better.
    + **Speed** – DB Read operations become much faster (up to 10X) as smaller amounts of physical data need to be moved from disk to memory. However, performance of write operations is marginally affected as there is an extra step of decompression involved.
    + **Resource Utilization** - More data will fit into a page in the disk, in memory or in the buffer pool, thereby minimizing I/O costs. Because I/O occurs at the page level, a single I/O will retrieve more data. This increases the likelihood that data resides in the cache because more rows fit on a physical page. Compression also results in massive reduction in backup/restore times.

### What are the disadvantages of database compression? When should we avoid it?

Most compression algorithms build an internal encoding dictionary to manage the compression keywords. **When database size is small**, compressed files might be larger than uncompressed files due to additional dictionary creation.

For any DB, **compression and decompression is a task overhead** above its regular DML/DDL operations. It consumes additional CPU/memory. So compression must be attempted only when the gain in CPU/memory due to optimised page reads is much larger than the compression overhead.

While compression might happen in parallel as a background task, **decompression introduces client-side latency** as it happens in the foreground after a client query. However, commercial databases like Oracle support advanced compression techniques where DB read can happen without decompression.

Compression is not recommended **when the cardinality of data is poor**. For numerical data and non-repetitive strings, don't compress your data. For data types like `BLOB` data (images, audio), depending on the compression algorithm, storage size can either reduce or increase.

In summary, it's important to estimate likely storage and performance benefits for any algorithm before implementing on live databases.

### What's the effect of compression on common data types?

Compression is a DDL function that can be selectively applied to tables, indexes or partitions with `CREATE`, `ALTER` and `BACKUP` commands.

Data compression applies to these database objects – heaps, clustered indexes, non-clustered indexes, partitions, indexed views.

Row-level compression converts fixed length data types into variable length types. Fields created as fixed length types, such as `char 100`, might not fill the entire 100 characters for every record. So this works well for fixed length text and numeric fields (`char`, `int`, `float`). For example, storing 23 in an `int` column will require just 1 byte when compressed instead of the entire 4 bytes allocated. No space is consumed for NULL or 0 values.

Page compression is more advanced. It internally invokes row compression. Commonality in page data is extracted and encoded, one column at a time or all columns together. Actual data is then replaced by the codes. Large objects are not directly compressed. Instead they are stored in a separate page for direct retrieval (`LOB` or `BLOB` types).

Unicode values are compressed into 2 bytes, regardless of locale. This results in nearly 50% savings for most locales.

### What are some popular database compression techniques?

Databases avoid lossy compression at the backend. However, applications might accept a lower quality using lossy techniques, especially for images, audio, videos.

Among lossless techniques, all compressed data is stored in binary. **Run-Length Encoding** is the simplest. In this technique, sequential data is scanned for repeated symbols, such as pixels in an image and replaced by short codes called "run". For a gray scale image, run length code is represented by `{Si, Ci}`, where `Si` is the symbol or intensity of pixels and `Ci` is the count or the number of repetitions of `Si`.

**Prefix encoding** is another basic technique that works well only on homogeneous data. The first part of the data fields are matched for commonality, not the entire field. Hence, the name 'prefix' encoding.

**Dictionary compression** is another page-level technique where common codes are consolidated into a dictionary and stored under the header row. The algorithm calculates the number of bits (X) needed to encode a single attribute of the column from unique attribute values. It then calculates how many of these X-bit encoded values can fit into 1, 2, 3, or 4 bytes.

### How does database compression vary from generic data compression techniques?

Data compression applies to within a field, how size of a single data item can be minimised. Here both lossless and lossy techniques are popular. On the other hand, database compression is an aggregate technique. It aims to compress data across values in a row, column or page. Along with applying compression at DB level, it's quite common to apply data compression at field level too.

In image compression, for example, a lossy BMP-to-JPG conversion is done using sampling and quantisation at field level. Then, BLOB data fields might be compressed at DB level using lossless methods.

While NULL value cannot be further compressed individually, when a series of NULL values occur in a DB, compression techniques such as the Sparse Column can optimise storage of the NULLs collectively.

Database compression techniques not only compress actual data, they also act on derived entities like indices, views, clusters and heaps. However, for derived entities like indices, there's a small CPU overhead to reconstruct the key column values during index lookup or scans. This can be minimized by keeping the prefixes locally in the block.

### How do compression techniques compare between relational and hierarchical databases?

There is no major variation in the compression algorithms applied on relational and hierarchical DBs. It's just a difference in nomenclature. Hierarchical DBs have segments similar to tables in RDBMS. And fields in the segment compare with columns.

However, in hierarchical databases, segments are implicitly joined with each other. Because of this, I/O paging process would vary, so algorithms must consider the page sizes.

### How do large-scale software applications handle database compression?

Facebook uses a real-time compression algorithm **Zstandard** with super fast decoders plus a dictionary compression for small data.

Google uses **Snappy**, a high speed compressor algorithm in their BigTable and MapReduce systems.

Commercial DBMS like SQL Server use row, column and page level compression. SQL Server uses a proprietary **XPRESS** algorithm for backup compression.

Oracle uses an internal compression technique called **Oracle Advanced Compression**.

Open source MySQL uses **LZ77** algorithm for its InnoDB table compressions.

## Milestones

1940

As early as the 1940s, **Information Theory** establishes the theoretical background behind lossless and lossy compression.

1970

In the 1970s, major relational databases–Ingres, MS SQL Server and Sybase–enter the commercial market. The support primitive **keyword-based compression** techniques built into their systems.

1977

The **LZ77** lossless compression algorithm, as the name suggests, is released. This forms the basis of compression schemes for PNG and ZIP formats. Implementations of the algorithm spread across most commercial databases. Compression in MySQL uses LZ77 via the zlib library.

2007

Oracle Database 11g Release 1 introduces OLTP Table compression, now called **Advanced Row Compression**, which maintains compression during all types of data manipulation operations. Oracle has been implementing compression techniques in their DBs from much earlier. But after acquisition of Sun, the techniques become more robust and hardware integrated.

2013

Designed for the cloud, Oracle releases Oracle Database 12c. It includes compression methods used in earlier Oracle Database 11g release and adds new ones: heat map, automatic data optimization, temporal optimization, hybrid columnar compression. **Heat map** method monitors data usage and selects the most suitable compression method.

Database Compression