# Part-of-Speech Tagging

## Summary

In the processing of natural languages, each word in a sentence is tagged with its part of speech. These tags then become useful for higher-level applications. Common parts of speech in English are noun, verb, adjective, adverb, etc.

The main problem with POS tagging is **ambiguity**. In English, many common words have multiple meanings and therefore multiple POS. The job of a POS tagger is to resolve this ambiguity accurately based on the context of use. For example, the word "shot" can be a noun or a verb. When used as a verb, it could be in past tense or past participle.

POS taggers started with a linguistic approach but later migrated towards a statistical approach. State-of-the-art models achieve accuracy better than 97%. POS tagging research done with English text corpus has been adapted to many other languages.

## Discussion

### Could you give an overview of POS tagging?

A POS tagger takes in a phrase or sentence and assigns the most probable part-of-speech tag to each word. In practice, input is often pre-processed. One common pre-processing task is to tokenize the input so that the tagger sees a sequence of words and punctuations.  Other tasks such as stop word removals, punctuation removals and lemmatization may be done before tagging.

The set of predefined tags is called the **tagset**. This is essential information that the tagger must be given. Example tags are NNS for a plural noun, VBD for a past tense verb, or JJ for an adjective. A tagset can also include punctuations.

Rather than design our own tagset, the common practice is to use well-known tagsets: 87-tag Brown tagset, 45-tag Penn Treebank tagset, 61-tag C5 tagset, or 146-tag C7 tagset. In the architecture diagram, we have shown the 45-tag Penn Treebank tagset. Sketch Engine is a place to download tagsets.

### What's the relevance of POS tagging for NLP?

POS tagging is a basic task in NLP. It's an essential pre-processing task before doing syntactic parsing or semantic analysis. It benefits many NLP applications including information retrieval, information extraction, text-to-speech systems, corpus linguistics, named entity recognition, question answering, word sense disambiguation, and more.

If a POS tagger gives poor accuracy, this has an adverse effect on other tasks that follow. This is commonly called **downstream error propagation**. To improve accuracy, some researchers have proposed combining POS tagging with other processing. For example, joint POS tagging and dependency parsing is an approach to improve accuracy compared to independent modelling.

### What are the sources of information for performing POS tagging?

Sometimes a word on its own can give useful clues. For example, 'the' is a determiner. Prefix 'un-' suggests an adjective, such as 'unfathomable'. Suffix '-ly' suggests adverb, such as 'importantly'. Capitalization can suggest proper noun, such as, 'Meridian'. Word shapes are also useful, such as '35-year' that's an adjective.

A word can be tagged based on the neighbouring words and the possible tags that those words can have. Word probabilities also play a part in selecting the right tag to resolve ambiguity. For example, 'man' is rarely used as a verb and mostly used as a noun.

In a statistical approach, we can count tag frequencies of words in a tagged corpus and then assign the most probable tag. This is called **unigram tagging**. A much better approach is **bigram tagging**. This counts the tag frequency given a particular preceding tag. Thus, a tag is seen to have dependence on previous tag. We can generalize this to **n-gram tagging**. In fact, it's common to model a sequence of words and estimate the sequence of tags. This is done by the **Hidden Markov Model (HMM)**.

### Which are the main types of POS taggers?

We note the following types of POS taggers:

    + **Rule-Based**: A dictionary is constructed with possible tags for each word. Rules guide the tagger to disambiguate. Rules are either hand-crafted, learned or both. An example rule might say, "If an ambiguous/unknown word X is preceded by a determiner and followed by a noun, tag it as an adjective."
    + **Statistical**: A text corpus is used to derive useful probabilities. Given a sequence of words, the most probable sequence of tags is selected. These are also called stochastic or probabilistic taggers. Among the common models are n-gram model, Hidden Markov Model (HMM) and Maximum Entropy Model (MEM).
    + **Memory-Based**: A set of cases is stored in memory, each case containing a word, its context and suitable tag. A new sentence is tagged based on best match from cases stored in memory. It's a combination of rule-based and stochastic method.
    + **Transformation-Based**: Rules are automatically induced from data. Thus, it's a combination of rule-based and stochastic methods. Tagging is done using broad rules and then improved or transformed by applying narrower rules.
    + **Neural Net**: RNN and Bidirectional LSTM are two examples of neural network architectures for POS tagging.

### In machine learning terminology, is POS tagging supervised or unsupervised?

POS taggers can be either supervised or unsupervised. **Supervised** taggers rely on a tagged corpus to create a dictionary, rules or tag sequence probabilities. They perform best when trained and applied on the same genre of text. **Unsupervised** taggers induce word groupings. This saves the effort of pre-tagging a corpus but word clusters are often coarse.

A combination of both approaches is also common. For example, rules are automatically induced from an untagged corpus. The output from this is corrected by humans and resubmitted to the tagger. The tagger looks at the corrections and adjusts the rules. Many iterations of this process may be necessary.

In 2016, it was noted that a completely unsupervised approach is not yet mature. Instead, weakly supervised approaches are adopted by aligning text, using translation probabilities (for machine translation) or transferring knowledge from resource-rich languages. Even a small amount of tagged corpus can be generalized to give better results.

### In HMM, given a word sequence, how do we determine the equivalent POS tag sequence?

We call this the **decoding problem**. We can observe the word sequence but the sequence of tags is hidden. We're required to find out the most probable tag sequence given the word sequence. In other words, we wish to maximize \(P(t^{n}|w^{n})\) for an n-word sequence.

An important insight is that parts of speech (and not words) give language its structure. Thus, using Bayes' Rule, we recast the problem to the following form, \(P(t^{n}|w^{n})=P(w^{n}|t^{n})\,P(t^{n})/P(w^{n})\). \(P(w^{n}|t^{n})\) is called **likelihood**. \(P(t^{n})\) is called **prior probability**.

Since we're maximizing over all tag sequences, the denominator can be ignored. We also make two assumptions: each word depends only on its own tag, and each tag depends only on its previous tag. We therefore need to maximize \(\prod\_{i=1}^{n}P(w\_i^{n}|t\_i^{n})\,P(t\_i|t\_{i-1})\). In HMM, the terms are called **emission** probabilities and **transition** probabilities.

These probabilities are estimated from the tagged text corpus. The standard solution is to apply **Viterbi algorithm**, which is a form of dynamic programming. In the example figure, we see two non-zero paths and we select the more probable one.

### What are some practical techniques for POS tagging?

In any supervised statistical approach, it's recommended to divide your corpus into training set, development set (for tuning parameters) and testing set. An alternative is to use the entire corpus for training but do **cross-validation**. Moreover, if the corpus is too general the probabilities may not suit a particular domain; if it's too narrow, it may not generalize well across domains. To analyse where your model is failing, you can use confusion matrix or contingency table.

When **unknown words** are seen, one approach is to assign a suffix and calculate the probability that the suffixed word with a particular tag occurs in a sequence. Another approach is to assign a set of default tags and calculate the probabilities. Or we could look at the word's internal structure, such as assigning NNS for words ending in 's'.

To deal with sparse data (probabilities are zero), there are **smoothing** techniques. A naïve technique is to add a small frequency count, say 1, to all counts. The Good-Turing method along with Katz's backoff is a better technique. Linear interpolation is another technique.

### How have researchers adapted standard POS tagging methods for custom applications?

Learner English is English as a foreign language. Such text often contain spelling and orthographic errors. The use of neural networks, **Bidirectional LSTM** in particular, is found to give better accuracy than standard POS taggers. Word embeddings, character embeddings and native language vectors are used.

Historical English also present tagging challenges due to differences in spelling, usage and vocabulary. A combination of **spelling normalization** and a **domain adaptation** method such as feature embedding gives better results. Other approaches to historical text include neural nets, conditional random fields and self-learning techniques.

Techniques have been invented to tag Twitter data that's often sparse and noisy. In mathematics, POS taggers have been adapted to handle formulae and extract key phrases in mathematical publications. For clinical text, tagged corpus for that genre is used. However, it was found that it's better to share annotations across corpora than simply share a pretrained model.

### Could you describe some tools for doing POS tagging?

In Python, nltk.tag package implements many types of taggers. pattern is a web mining module that includes ability to do POS tagging. Unfortunately it lacks Python 3 support. It's also available in R as pattern.nlp. TextBlob is inspired by both NLTK and Pattern. spaCy is another useful package.

Implemented in TensorFlow, SyntaxNet is based on neural networks. Parsey McParseface is a parser for English and gives good accuracy.

Parts-of-speech.info is an online tool for trying out POS tagging for any text input. Useful open source tools are Apache OpenNLP, Orange and UDPipe.

Samet Çetin shows how to implement your own custom tagger using a logistic regression model. There's also a commercial tool from Bitext.

Datacube at the Vienna University of Economics and Business is a place to download text corpora, and taggers (OpenNLP or Stanford) implemented in R. Stanford tagger is said to be slow. Treetagger is limited to non-commercial use. Another R package is RDRPOS tagger. A Java implementation of a log-linear tagger from Stanford is available.

## Milestones

1963

Klein and Simmons describe a rule-based method with a focus towards initial categorical tagging rather than part-of-speech disambiguation. They identify 30 categories and achieve 90% accuracy, which may be because fewer categories implies less ambiguity.

1964

W. Nelson Francis and Henry Kučera at the Department of Linguistics, Brown University, publish a computer-readable general corpus to aid linguistic research on modern English. The corpus has 1 million words (500 samples of about 2000 words each). Revised editions appear later in 1971 and 1979. Called **Brown Corpus**, it inspires many other text corpora. Brown Corpus is available online.

1971

Greene and Rubin develop the **TAGGIT** system to tag the Brown Corpus from a set of 86 tags. It uses rules for tagging and obtains 77% accuracy. Human experts then do post-editing.

1976

In one of the earliest departures from rule-based method to statistical method, Bahl and Mercer apply **HMM** to the problem of POS tagging and use Viterbi algorithm for decoding. In 1992, a research team at Xerox led by Doug Cutting apply HMM in two ways: they apply the Baum-Welch algorithm to obtain the maximum likelihood estimate of the model parameters; next they apply Viterbi algorithm to decode a sequence of tags given a sequence of words.

1983

The **CLAWS** algorithm uses the **co-locational probabilities**, that is, likelihood of co-occurrence of ordered pairs of tags. These probabilities are estimated from the tagged Brown Corpus. Steven J. DeRose improves on this work in 1988 with the **VOLSUNGA** algorithm that's more efficient and achieves 96% accuracy. CLAWS and VOLSUNGA are N-gram taggers. By late 1980s, statistical approaches become popular. Rather than build complex and brittle hand-coded rules, statistical models learn these rules from text corpora.

1992

Started in 1989 at the University of Pennsylvania, the **Penn Treebank** is released in 1992. It's an annotated text corpus of 4.5 million words of American English. The corpus is POS tagged. Over half of it is also annotated with syntactic structure. **Treebank II** is released in 1995. The original release had 48 tags. Treebank II merges some punctuation tags and results in a 45-tag tagset. Apart from POS tags, the corpus includes chunk tags, relation tags and anchor tags.

1992

At a time when stochastic taggers are performing better than rule-based taggers, Eric Brill proposes a rule-based tagger that performs as well as stochastic taggers. It works by assigning the most likely estimates from a corpus but without any contextual information. It then improves on the estimate by applying patching rules, which are also learned from the corpus. One test shows 5.1% error rate with only 71 patches. This method is later named **Transformation-Based Learning (TBL)**.

1994

Called Net-Tagger, Helmut Schmid uses a **Multi-Layer Perceptron (MLP)** network to solve the POS tagging problem. He notes that neural networks were used previously for speech recognition. Though Nakamura et al. (1990) used a 4-layer feed-forward network for tag prediction, Net-Tagger is about tag disambiguation rather than prediction. Accuracy of 96.22% is achieved with a 2-layer model.

1996

**Maximum Entropy Model** was previously used for problems such as language modelling and machine translation. A. Ratnaparkhi applies the model to POS tagging and achieves state-of-the-art word accuracy of 96.6%; and 85.6% for unknown words. OpenNLP POS tagger uses such a model.

2011

Christopher Manning, NLP researcher at Stanford University, comments that POS tagging has reached 97.3% token accuracy and 56% sentence accuracy. Further gains in accuracy might be possible with improved descriptive linguistics. He also argues that accuracy of 97% claimed to be achieved by humans might be an overestimate. Thus, automatic taggers are already surpassing humans.

2018

Since 2015, many neural network based POS taggers have shown better than 97% accuracy. In 2018, **Meta-BiLSTM** achieves an accuracy of 97.96%.

Part-of-Speech Tagging


# Containerization

## Summary

Containerization is a technique that allows software to run reliably regardless of the computing environment. By encapsulating software within isolated environments called **containers**, we can more reliably port software across operating systems and hardware infrastructures.

Let's say, most of the development or testing is done on a developer's laptop. The software may work as expected on the laptop but when deployed on the server the software fails. This could be because the server is using different versions of libraries, has a different configuration or interfaces differently to other components of the system. Containerization solves this problem by providing a consistent and isolated runtime environment regardless of the underlying OS or hardware infrastructure.

For developers, what this means is that we're no longer deploying just the application software but deploying **container images** that contain the app along with its dependencies.

## Discussion

### What's a container and what's in it?

Containers consists of the runtime environment: an application, dependencies, libraries, binaries, and configuration files needed to run an application, bundled into one package. By containerizing the application platform and its dependencies, differences in OS distributions and underlying infrastructure are abstracted.

Essentially, a container includes the application and all of its dependencies. It shares the OS kernel with other containers. It's not tied to a specific infrastructure: it only needs Docker Engine (or equivalent) installed on host. Thus, containers isolate the application process in user space on the host OS from other application processes.

### What are the benefits of containers?

A survey commissioned by Red Hat in 2015 showed that containers are seen to bring better security, efficiency, portability, flexibility and speed. More specifically, we can identify the following benefits:

*Benefits to Applications*

    + Portable
    + Packaged in a standard way
    + Automated testing, packaging and integrations
    + Support newer microservices architectures
    + Alleviate platform compatibility issues*Benefits to Deployment*

    + Easy
    + Repeatable
    + Reliable deployments: improved speed and frequency of deployments
    + Consistent application lifecycle: configure once and run multiple times
    + Consistent environments: no more process differences between local, dev and staging environments
    + Simple scaling: Fast deployments ease the addition of workers and permit workload to grow and shrink for on-demand use cases

### How is Virtualization different from Containerization?

With virtualization technology, the package that can be passed around is a **Virtual Machine (VM)**. It includes an OS as well as the application. A server running three VMs would have a hypervisor and three separate operating systems running on top of it. By contrast, a server running three containerized applications runs on top of a single OS, all containers sharing the same OS kernel. Shared parts of the operating system are read only, while each container has its own mount (i.e., a way to access the container) and volumes for reading from and writing to the file system. This means that the containers are much more lightweight and use far fewer resources than virtual machines.

Containerization has been called an **operating system level virtualization**, an operating system feature in which the OS kernel allows the existence of multiple isolated user-space instances. Instances are created from images that we can build and share. Instances created from images are called containers, partitions, virtualization engines or jails (FreeBSD). Applications running inside a container can only see the container's contents and devices assigned to the container.

### What's the typical size of a container?

Containers can only be tens of megabytes in size. For example, the Docker Alpine image is about 4 MB. For comparison, Fedora version 25 is about 231 MB. A virtual machine with its entire OS may be several gigabytes in size. For this reason, a single server can host far more containers than virtual machines.

Virtual machines may take several minutes to boot up their operating systems and start running the applications they host. However, containerized applications can be started almost instantly. This also means that containers can be created as needed and destroyed quickly, thereby using resources efficiently.

### What are some types of containers?

The common use case is to separately run applications within **Application Containers**. Code that developers create run within these containers, which greatly simplify the deployment of apps. Docker and OCI-based containers are examples.

Another use case of containers is to provide a virtual operating system. What if we wish to deploy Ubuntu, CentOS and RHEL on top of a common host OS? This is where **Operating System Containers** can be used. LXC and LXD are examples. It's interesting that Docker and OCI-based containers can partly achieve this functionality by running `systemd`.

Suppose you're building a specialized app. You want the flexibility to customize your app but at the same time benefit from the automation and tooling of the container ecosystem. You can use **Pet Containers**.

Administrators can benefit from **Super Privileged Containers (SPC)** for kernel module loading, monitoring, backups, etc. SPCs usually have a tighter coupling with the host kernel.

### Could you describe specific examples of container usage?

Let's take the example of a web application using Nginx as the load balancer, Node.js for the app and Postgres for database. Traditionally, all of these would be deployed on a single machine. These can't be deployed, scaled or managed independently. Instead, if we adopt a three-tiered architecture, each one is delivered as a separate application container image. Each one can be deployed independently, on different machines if needed.

Perhaps we what to run our app within a specific OS. In this case, we can use an OS container image of that OS, install the necessary dependencies and bring up the app within the container in a consistent manner.

Application containers are used in smartphones. Nexus One uses LXC on the Android kernel. McAfee provides a Secure Container for Android. Apple iPhones use containers to compartmentalize applications and their data.

### Which are the technical enablers for implementing containers?

**Linux Containers (LXC)** is a modern method of virtualizing an application. LXC leverages *cgroups* to isolate the CPU, memory, file/block I/O and network resources. LXC also uses *kernel namespaces* to isolate the application from the operating system and separates the process trees, network access, user IDs, and file access. LXC is considered a technique that falls between chroot and VM. In version 1.0 of LXC, unprivileged containers are more secure because they run as regular unprivileged users.

To enable containers to start quickly, the container image is not copied but shared. A copy is made only when data is modified. This is called the **Copy-on-Write (CoW)** mechanism. File-level CoW is easier to backup, more space efficient and simpler to cache than block-level CoW on whole-system virtualizers.

### What are some essential terms to know in relation to containers?

A container is specified by one or more files called **Container Image**. Often, a hierarchy of image layers, tagged and stored with metadata, together form what's called a **Repository**. Some use the two terms interchangeably.

A container image is stored on a **Registry Server**, whose location is known to anyone wishing to pull and use the image. The system on which a container runs is called **Container Host**. Running containers are also called **Containerized Processes** because ultimately they're nothing but processes.

Thus, a container starts as an image at rest and ends up as a process during execution. The job of processing user requests, pulling images from the registry and initiating execution of the container is done by the **Container Engine**. The engine itself doesn't execute containers. This is done by the **Container Runtime**, which gets the image mount point and metadata from the engine, communicates with the kernel, and sets up relevant permissions.

Since a registry may have lots of repositories, **Namespaces** help in logically separating them. These can be names of persons, organizations, products, etc. These should not be confused with kernel namespaces.

### Could you name some container implementations?

**Docker** is the most popular one. Others include Linux OpenVZ, Linux-VServer, FreeBSD Jails, AIX Workload Partitions (WPARs), HP-UX Containers (SRP), and Solaris Containers. Docker offers a complete container ecosystem including image management, deployment assistance, automation, API for building Platform as a Service (PaaS), and more.

The Open Container Initiative (OCI) has standardized container runtime. Its reference implementation is called **runc**. This is used by Docker, CRI-O and others. Docker previously used LXC as the runtime. Then it created its own runtime called libcontainer, which eventually became runc. Other OCI-compliant runtimes include crun, railcar, and katacontainers.

Among the different container engines are Docker, RKT, CRI-O, and LXD. In addition, cloud providers may provide their own built-in container engines. For interoperability, many engines accept Docker or OCI-compliant images.

### Which Linux distributions are suitable for use as a container host?

Most Linux distributions are unnecessarily feature-heavy if their intended use is simply to act as a container host to run containers. For that reason, a number of Linux distributions have been designed specifically for running containers. Here are some examples:

    + Container Linux — formerly called CoreOS Linux, it's one of the first lightweight container operating systems built for containers.
    + RancherOS — a simplified Linux distribution built from containers, specifically for running containers.
    + Photon OS — a minimal Linux container host, optimized to run on VMware platforms.
    + Project Atomic Host — Red Hat's lightweight container OS has versions that are based on CentOS and Fedora, and there's also a downstream enterprise version in Red Hat Enterprise Linux.
    + Ubuntu Core — the smallest Ubuntu version, Ubuntu Core is designed as a host operating system for IoT devices and large-scale cloud container deployments.
    + Alpine Linux — is a very tiny Linux distribution focused on security.

## Milestones

1979

During the development of Unix V7 in 1979, the *chroot* system call is introduced, changing the root directory of a process and its children to a new location in the filesystem. In BSD Unix, this feature is introduced in 1982.

2000

FreeBSD Jails allow administrators to partition a FreeBSD computer system into several independent, smaller systems called *jails*, with the ability to assign an IP address for each system and configuration. In 2001, something similar is done by Linux VServer to partition resources (file systems, network addresses, memory) on a computer system.

2004

Oracle releases a Solaris Container called Solaris Zones that combines system resource controls and boundary separation provided by *zones*, which are able to leverage features like snapshots and cloning from ZFS.

2005

OpenVZ is an operating system-level virtualization technology for Linux that uses a patched Linux kernel for virtualization, isolation, resource management and checkpointing. Back then, the code is not released as part of the official Linux kernel.

2006

Process Containers is launched by Google. It's designed for limiting, accounting and isolating resource usage (CPU, memory, disk I/O, network) of a collection of processes. It's renamed **Control Groups (cgroups)** a year later and eventually merged to Linux kernel 2.6.24.

2008

LXC (LinuX Containers) is the first, most complete implementation of Linux container manager. It's implemented using cgroups and Linux namespaces. It works on a single Linux kernel without requiring any patches.

2013

Let Me Contain That For You (LMCTFY) is started as an open-source version of Google's container stack, providing Linux application containers. Applications can be made "container aware," creating and managing their own subcontainers. Active deployment in LMCTFY stopped in 2015 after Google started contributing core LMCTFY concepts to *libcontainer*, which is now part of the Open Container Foundation.

2013

Docker is released and containers explode in popularity. Docker uses LXC in its initial stages and later replaces that container manager with its own library, *libcontainer*. Later, Docker separates itself from the pack by offering an entire ecosystem for container management.

Containerization


# Market Basket Analysis

## Summary

Market basket analysis is a data mining technique, generally used in the retail industry in an effort to understand purchasing behaviour. It looks for combinations of items that frequently occur in the same transaction. In other words, it gives insights into items that may have some association or affinity.

For example, customers purchasing flour and sugar are also likely to buy eggs. The outcome of the analysis is to derive a set of rules that can be understood as "if this, then that". Retailers can use these insights to do product placements or offer discounts.

In fact, market basket analysis is being applied outside retail. Therefore, it's more generally called **Affinity Analysis**.

## Discussion

### Could you give examples of market basket analysis?

Market basket analysis uncovers associations between products by looking for combinations of products that frequently co-occur in transactions. Thus, supermarkets can identify relationships between products that people buy.

For example, customers who buy a pencil and paper are likely to buy an eraser or ruler. A customer in an English pub buying a pint of beer without a bar meal is more likely to buy crisps/chips than somebody who didn't buy beer. Someone who buys shampoo is likely to buy conditioner. Retailers can use this information to modify the store layout or offer discount on shampoo but not on conditioner.

Online retails such as Amazon make product purchase recommendations. If you add any item to your cart, Amazon will recommended other items that other customers often bought together with your selected item.

### What are the applications of market basket analysis?

Within retail, market basket analysis helps determine purchasing behaviour, build recommendation engines, customize loyalty programs, cross-sell, up-sell, place products in stores in the right places, offer right combinations of discounts, and so on.

Beyond retail, affinity analysis has been used on medical data. Does high BMI and smoking lead to greater chance of high blood pressure? Affinity analysis answers such questions.

In web browsing, it enables click stream analysis. For example, given the last two clicks, how likely is the user to click a specific link? It can be used to detect intrusions.

In banking, credit card purchases are analysed to detect frauds and cross-sell. In insurance, user profiles formed via market basket analysis can be used to flag fraudulent claims. In telecom, market basket analysis can suggest the right bundle of services to retain customers or analyse calling patterns.

### What's the typical data pipeline for market basket analysis?

Data for market basket analysis typically comes from point-of-sale (POS) transaction data or invoices. These usually include list of products purchased, unit price and quantity of each item. To be statistically significant, the dataset must be large. One analyst reported a dataset of 32 million records of 50K unique items.

The next step is to look for combinations of items that occur most often together within a transaction. The selection of the right algorithm is important here. Otherwise, we will end up with too many combinations of items (perhaps in millions) that may be computationally difficult to analyse.

Once frequently occurring item combinations are identified, we next look for associations. This step is often called **Association Rule Mining**.

The last step is to pick out strong association rules, seek explanations as to why such associations exists and drive business decisions. The usefulness or "interestingness" of a rule is application dependent.

### Could you give more details on Association Rule Mining?

Association Rule Mining counts the frequency of items that occur together across a large collection of items or actions. The goal is to find associations that take place together far more often than you would find in a random sampling of possibilities.

Output of above will be a set of association rules in the following form: `IF {item 1, item 2} THEN {item 3}`. This states that when items 1 and 2 are purchased, then item 3 is likely to be purchased with a certain probability. The first part of the rule is called **antecedent**. The second part is called **consequent**.

For example, a customer who buys pencil and paper (antecedent) is likely to buy an eraser (consequent). But how likely is such a customer to buy an eraser? To quantify this, we have a few measures: support, confidence, and lift. These tell us how important or reliable is an association rule.

### Could you explain the terms Support, Confidence and Lift?

These are common quantitative measures to identify most important association rules:

    + **Support**: Given all transactions, support is the percentage of transactions that contain a given item combination. Often combination that fall below a support threshold are ignored in further analysis. When dataset has thousands of items and millions of transactions, a threshold of 0.01% is reasonable.
    + **Confidence**: Given item A is purchased, what's the chance that customer will buy item C? This question is answered by the confidence measure. Thus, rather than looking at just probability of purchasing item C (which support does), confidence looks at conditional probability.
    + **Lift**: Suppose data shows that items A and C are occurring together in many transactions. Do A and C have an association or are they occurring together purely by chance? This question is answered by the lift measure.Association rules must satisfy both minimum support and minimum confidence values. To filter the results further to a smaller list, lift is a popular measure.

### Could you give example calculations of Support, Confidence and Lift?

For example, given a million transactions, 24K transactions contain {flour,sugar}; 30K transactions contain {eggs}; 20K transactions contain both {flour,sugar} and {eggs}. Thus,

    + Support({flour,sugar}) = 24K / 1M = 0.024
    + Support(eggs) = 30K / 1M = 0.03
    + Support({flour,sugar}, eggs) = 20K / 1M = 0.02
    + Confidence({flour,sugar}->eggs) = Support({flour,sugar}, eggs) / Support({flour,sugar}) = 0.02 / 0.024 = 0.83
    + Confidence(eggs->{flour,sugar}) = Support({flour,sugar}, eggs) / Support(eggs) = 0.02 / 0.03 = 0.66
    + Lift({flour,sugar}, eggs) = Support({flour,sugar}, eggs) / (Support({flour,sugar})\*Support(eggs)) = 0.02 / (0.024\*0.03) = 27.8Note that Confidence is directional but Lift is not. In the above example, a purchase of {flour,sugar} drives purchase of eggs more strongly than eggs driving purchase of {flour,eggs}.

A lift value of 1 implies there's no association. A value more than 1 implies a positive association. In our example, the denominator value (0.024\*0.03) = 0.00072 = 0.072% is how often both items would occur together if they had no relationship. Lift is giving us a measure of association relative to being random.

### What are the tools or packages available to do market basket analysis?

R language has the **arules** package for association rule mining. This includes C implementations of Apriori and Eclat algorithms. The **arulesViz** package has useful visualizations that can help in exploratory analysis. It includes visualizations of support, confidence and lift.

**KNIME** offers a tool for market basket analysis. It provides a graphical block-diagram-based interface that can be ideal for non-programmers. It offers the Apriori algorithm in traditional as well as the more optimized Borgelt implementation.

In Python, we can use the **MLxtend** package. Christian Borgelt has also released a C implementation that can be compiled for the Python environment. He calls this **PyFIM**, where FIM stands for Frequency Itemset Mining.

## Milestones

1980

This decade sees progress in barcode technology that becomes widely used in the retail industry. It therefore becomes easy to collect vast amounts of purchasing data. This leads to what can be called information-driven marketing.

1991

Piatetsky-Shapiro analyzes and presents strong rules discovered in databases using different measures of interestingness.

1992

Market basket analysis of 1.2 million market baskets from 25 Osco Drug stores brings out an unexpected association between diapers and beer. Subsequently, this example becomes commonly used in data mining literature. In reality, the managers at Osco did not exploit this "interesting" (and not significant) association but they became aware of what's possible with market basket analysis.

1993

Based on the concept of strong rules, Agrawal, Imielinski, and Swami introduce the problem of mining association rules from transaction data. The term **association rules** can be attributed to Agrawal and team. They also propose an algorithm for discovering these rules.

1994

Rakesh Agrawal and Ramakrishnan Srikant publish the **Apriori algorithm**. This outperforms earlier algorithms, particularly when datasets are large. At the IEEE International Conference on Data Mining (ICDM) in December 2006, this is identified as one of the ten most important algorithms in the field of data mining. Apriori uses support and confidence but not lift. Lift is something we use as a final step.

2003

Christian Borgelt provides C implementation of the Apriori algorithm.

Dec
2018

In R language, for association rule mining, version 1.6 of **arules** package is published on CRAN.

Market Basket Analysis


# Amazon Alexa

## Summary

Amazon Alexa is cloud-based voice service to bring natural voice experiences to users. It can seen as a virtual assistant with the capability to interact with users using voice rather than traditional computer interfaces of keyboard, mouse or monitor.

Initially, Alexa was released as the backend support service for Amazon's Echo device, a voice-activated connected speaker. Since then, Alexa has become smarter. It's been interfaced to a number of products beyond Amazon's Echo series of devices. It's moved from being just a smart speaker to central controller of your smart home.

## Discussion

### What are typical uses of Amazon Alexa?

When Amazon Echo was released in 2014, Alexa could answer basic questions, add items to a shopping cart or play your favourite music album. Today, developers can add new capabilities to Alexa. Capabilities exist for managing chores, telling stories, quizzing, and gaming.

Alexa can integrate with web services for ordering something, getting information, etc. It can control or query smart devices connected to the cloud. For example, you can ask to see the feed from a remote camera or dim the lights. In fact, Alexa's importance in the smart home is set to grow as it aims to handle complete home automation.

In terms of some specific examples, Xbox One console will soon support Alexa so that gamers can issue voice commands. A number of lighting solutions can be controlled via Alexa: Philips Hue LED, Lifx, TP-Link, Osram, Stack LED. In healthcare space, Alexa has been used to get information on drug side-effects, medication schedule or gather patient survey.

### As a developer, what can I build with Alexa?

With Alexa, three things are possible:

    + Develop capabilities to Alexa for new use cases. These capabilities are called *skills*. **Alexa Skills Kit (ASK)** is a set of APIs, tools and documentation to help you develop skills.
    + If you wish to create a hands-free voice interface in your own products, use the **Alexa Voice Service (AVS)**. Your products can then leverage Alexa's features. A simple example is to make your Raspberry Pi work with Alexa as explained in a detailed tutorial.
    + Let your customers control their connected devices in smart homes and other gadgets. Devices can reach Alexa via the Internet or via a local hub. Developers need to use **Smart Home Skill API** and **Gadgets API**.

### What are the languages supported by Alexa?

As of August 2018, Alexa's skills can support English, French, German, Italian, Japanese and Spanish. A device (such as Echo) when set to a particular language can access any skill available in that language. As a developer, you can publish your skill in multiple languages to target a wider user base.

From a programming perspective, if the business logic of your skill is deployed on AWS Lambda, then any programming language that Lambda supports can be used: Node.js, Java, Python, or C#. Or you can deploy it as a web service on any cloud infrastructure. This means that you can code in any language, though the data is JSON via REST.

### What's the architecture of Alexa?

Alexa resides in the cloud. Access is via the Alexa Voice Service API. Alexa receives requests as speech streams. Alexa analyzes the speech and identifies the requested skill. A structured representation of the skill and intent is created.

Now there are two possible paths. If the skill code is deployed in AWS Lambda, that service will be triggered within AWS. If it's a web service accessed via HTTPS then the REST API call is made. In either case, Alexa will receive the response as text. Optionally, the response may have images. Alexa will convert the text into speech, which gets streamed to the user via their current device. Images will be served where the device has a display.

A skill is identified by an *invocation name* and mapped to a backend service. A skill can be designed to do multiple things, each of which is called an *intent*. Each intent can be identified by one or more words or phrases, each of which is called an *utterance*. For example, you may create a travel-related skill that has three intents: rent a car, book a flight, or book a hotel.

### Could you define some Alexa-specific terminology?

Here are a few essential terms and more are available in the Alexa Skills Kit Glossary:

    + **Skills**: A capability of Alexa. Alexa comes with built-in skills, such as playing music. Developers can also build new skills.
    + **Intents**: The main request or action associated with the user's command for a custom skill.
    + **Slots**: Arguments to an intent that provide Alexa more information. These can be required or optional.
    + **Utterances**: Words users say to convey what they want Alexa to do or in response to a question.
    + **Interaction Model**: The words and phrases users can say to make a skill do what they want. It determines the requests the skill can handle and the words users say to invoke those requests.
    + **Invocation**: The act of beginning an interaction with a particular Alexa ability.
    + **Wake word**: A word that "wakes up" Alexa for a task. Typically, this word is "Alexa".
    + **Cards**: Elements displayed in a visual interface to describe or enhance voice interaction.

### What technologies power Alexa under the hood?

Two fundamental technologies required to make Alexa work are speech-to-text and text-to-speech conversions. To make these work reliably, Alexa employs deep learning algorithms. More specifically, we have two technologies, both of which were opened for developers in November 2016:

    + **Amazon Lex**: This does the speech-to-text conversion using techniques of *Automatic Speech Recognition* and *Natural Language Understanding*. It can be used independently of Alexa to build conversational interfaces (such as chatbots) using a mix of voice and text.
    + **Amazon Polly**: This does the text-to-speech conversion by synthesizing speech in various languages and accents. In May 2018, Alexa developers got access to eight different voices from Amazon Polly. In fact, we can use *Speech Synthesis Markup Language (SSML)* to generate natural-sounding speech. One researcher showed how SSML can be used to teach Alexa to speak with a Boston accent.

### What resources are available to learn about programming for Alexa?

A good place to start is the Voice Design Guide. This includes a design checklist and a glossary. There are also guides on using Amazon Skill Builder.

As a tutorial, you can start with this 6-step tutorial to build a fact skill. An easier way to get started is to use Alexa Skill Blueprints. These are templates that you can customize to make your own skill. Alexa on GitHub contains open source code.

There are some useful developer tools. There are also open source frameworks for building voice apps. Two such frameworks are Jovo and Violet. These frameworks are cross-platform; that is, from a single codebase they can target Amazon Alexa, Google Assistant, etc. Though not open source, Alexa Flow is a tool to easily create, manage and host your custom skills without coding knowledge. An alternative is Storyline that offers a visual drag-n-drop design interface, previews, deployment and analytics. Many more tools, SDKs and platforms are listed in Wolter's blog.

Blogs specific to Alexa are useful to get latest news or learn best practices. From the main Alexa website you can access all Alexa-specific documentation.

## Milestones

Nov
2014

**Amazon Echo** becomes available by invite only to select Amazon Prime customers. This is a voice-activated connected speaker that can act on user requests or answer basic questions. It's actually powered in the cloud by a virtual assistant called **Alexa**.

Jun
2015

To enable third-party integration of Alexa into their connected devices, **Alexa Voice Service (AVS)** is released. In April 2016, Invoxia becomes the first maker to release Alexa service on a non-Amazon device. Called *Triby*, it's a kitchen device that can set timers, play music or read sports scores.

Apr
2016

The **Alexa Smart Home API** is launched. In May 2017, with the **Smart Home Skill API**, both users and developers can use appliance categories (camera, light, smartlock, etc.) to easily identify device types.

Apr
2017

Amazon brings out the **Alexa Skill Builder**, a more intuitive interface to build skills including the interaction model and multi-turn dialogues. Work on this started back in October 2016.

Apr
2018

Amazon releases **Alexa Skill Blueprints**. These are skill templates that you can customize rather than develop custom skills from scratch.

May
2018

Developers cannot sell Alexa skills but Amazon enables **In-Skill Purchasing (ISP)**. If a skill drives user engagement, Amazon also offers rewards to developers. Amazon releases a preview in which developers can choose any of eight U.S. English voices to narrate skills. This is powered by **Amazon Polly**.

Aug
2018

Amazon releases **Alexa Auto SDK** on GitHub. Automobile makers can integrate Alexa directly into their vehicles. Amazon announces **Quick Start Templates** to accelerate skill development for popular skill samples.

Amazon Alexa


# IPsec

## Summary

IPsec (IP Security) is a suite of security protocols added as an extension to the IP layer in networking. IPsec can ensure a secure connection between two computing devices over unprotected IP networks, such as the Internet. The nature of security threats which IPsec prevents are varied and constantly changing—such as man-in-the-middle attacks, sniffing, replay attacks. IPsec finds application in three security domains: Virtual Private Networks (VPNs), application-level Security and routing security.

IPsec protocols cover the entire communication process – (1) Initial authentication of the two connecting devices (2) Confidentiality of transmitted data during actual communication.

IPsec is a capability built over IP (IPv4 and IPv6) by means of additional headers. It consists of three distinct functions – authentication, confidentiality and key management.

Similar to IP itself, IPsec is an open standard maintained by the Internet Engineering Task Force (IETF).

## Discussion

### How does IPsec compare with other security techniques that exist at different OSI layers?

Network server systems protect data in the network by supporting a variety of cryptography-based network security protocols. Key differentiating features of these protocols are:

    + **IPsec:** Supports network-level peer and data origin authentication, data integrity, data encryption, and protection.
    + **SSL (Secure Sockets Layer):** Doesn't require pre-shared key like in IPsec, uses public key cryptography to negotiate handshake and securely exchange encryption keys. Useful for bypassing firewalls and port-based traffic blocking.
    + **TLS (Transport Layer Security):** Cryptographic algorithms based on cipher suite that server and client negotiate. Uses SSL protocol.
    + **HTTPS:** Sets up encrypted link using TLS. Data transfer is encrypted between browser and web server, preventing cyber criminals from reading or altering data.
    + **SNMP (Simple Network Management Protocol):** User-based Security Model (USM) which provides different levels of security based on the user accessing managed information. Uses authentication and data encryption for privacy.
    + **OSPF (Open Shortest Path First) Authentication:** Dynamic routing protocol. Supports message authentication and integrity of OSPF routing messages. Unauthorized IP resource cannot inject messages into network without detection. Ensures integrity of routing tables in the OSPF routing network.

### What's the architecture of IPsec protocol suite?

IPsec specifies procedures, components, their inter-relationship and the general process required to provide security services at IP layer.

IPsec is a suite of three transport-level protocols used for authenticating the origin and content of IP packets and optionally for data payload encryption. Applications can invoke IPsec on IP datagrams that are security enabled in the IPsec global policy file on system-wide or per-socket level.

Key components of IPsec architecture include:

    + **Security Associations (SA):** Specifies security properties recognized by communicating hosts. Hosts require two SAs to communicate securely, one for each direction. Prior to transmission of protected datagrams, they agree upon specific security protections, cryptographic algorithms, secret keys to be applied and specific types of traffic to be protected.
    + **IPsec Protocols:** AH (Authentication Header) and ESP (Encapsulating Security Payload) protocols are for authentication. They contain proof of data source, data integrity and anti-replay protection. The third protocol IKE (Internet Key Exchange) is a hybrid protocol used for peer authentication and key exchange processes before actual data transfer begins.
    + **IPsec modes:** Both protocols (AH & ESP) support two modes - transport mode & tunnel mode.

### What are the services offered by IPsec?

IPsec offers a number of services: access control, connectionless integrity, data origin authentication, rejection of replayed packets, confidentiality (encryption), and limited traffic flow confidentiality.

### What are the primary applications of IPsec?

IPsec enables secure communication over the Internet, independent from the application or higher protocols. It supports network layer security, but it's compatible with schemes providing security at application layer. It provides secure communication across LAN, private/public WAN and the Internet.

(1) An enterprise may establish a secure VPN connection over a public WAN or Internet. IPsec minimises the need for a private network and hence saves costs. Ensures authentication, confidentiality and encryption. Application scenarios include:

    + Secure connection between head office and branches (client-server)
    + Different branches (peers) of same company
    + Workers connect to company intranet remotely
    + Intranet or extranet connectivity with partners (partner organizations, payment gateways, third party transactions)(2) Security for e-commerce applications, mainly B2B. IPsec enhances existing application-level security features. Customer-oriented commerce is primarily web based and uses SSL.

(3) Network diagnostics for debugging connections. A tunnel is initiated with a ping from the remote device.

(4) IPsec usage in a 4G network, simple end-to-end implementation between the mobile device and the services network peer (security gateway, server or another peer).

### What are the security protocols in the IPsec suite?

IPsec protocol suite mainly consists of these protocols:

    + **Authentication Header (AH):** AH is an extension header to provide data integrity, authentication and anti-replay but it doesn't provide encryption. Anti-replay protection ensures partial sequence integrity and protects against unauthorized transmission of packets. Implements strong hashing algorithms to provide data integrity. AH doesn't provide data confidentiality.
    + **Encapsulating Security Payload (ESP):** ESP consists of an encapsulating header and trailer. Provides all that AH offers (data integrity, encryption, authentication and anti-replay). It additionally provides authentication for payload data using symmetric key encryption. Uses encryption algorithms such as GCM, DES, 3DES, and AES. Set of services provided depends on options selected at the time of Security Association (SA) establishment and location of the implementation in a network topology.
    + **Internet Key Exchange (IKE):** IETF-specific key managing protocol for exchanging and negotiating security policies. Supports manual or dynamic association of management in cryptographic keys. Can be used outside IPsec as well.
    + **Cryptographic algorithms:** For encryption, message authentication, pseudorandom functions (PRFs), and cryptographic key exchange.

### Can you explain the different authentication modes in IPsec?

For authentication, both AH and ESP support two modes of use: transport and tunnel mode. Authentication can be applied to the entire original packet (tunnel mode) or to the packet contents except the IP header (transport mode).

    + **Transport Mode**: Client to client. Covers most of the original packet. Original IP packet is used with ESP and AH Headers, then original IP Header is reused in front of ESP and AH Headers. This mode doesn't change the IP packet header. Only IP protocol field is changed to 51 (AH) or 50 (ESP), and checksum of IP packet header is recalculated. This mode is used when we have end-to-end control of the network, so that there will be no packet manipulation through the network.
    + **Tunnel Mode**: Covers entire original packet. Inserts the original packet in a new IP packet, a new IP Header is added in front of the ESP and AH Headers. Original IP packet header is hidden. This mode is used in communications between VPN gateways or between a host and a VPN gateway. Tunnel Mode is the default option.

### How does IPsec implementations vary between IPv4 and IPv6?

Security concerns were raised over IPv4. As a result, IPsec was first developed for the newest version of Internet Protocol (IPv6), then retrospectively added for IPv4. It was included as a mandatory feature in the IPv6 standards whereas it's optional and must be supported externally in IPv4.

By design, IPv6 is more secure from IP address scanning attacks, as its address space is huge. So IP scanning techniques in networks may not work to find possible computers with security vulnerabilities, as it could take years.

IPsec in IPv4 is widely used for VPNs. These are terminated at the edge of networks. In IPv4, IPsec is generally not used to secure end-to-end traffic because of the widespread use of Network Address Translation in IPv4, called *NAT44*. NAT mangles the IPv4 headers and breaks IPsec. This restriction doesn't exist in IPv6, so that using IPsec end-to-end becomes more practical.

### What are the disadvantages of IPsec?

    + **Over-reliance on public keys:** If a network has poor key management or the integrity of the keys is compromised then the IPsec security factor is lost.
    + **Performance overhead:** High CPU usage because all data that passes through the server has to be encrypted and decrypted. When data packet size is small, network performance suffers due to large IPsec overhead. May need to use hardware appliances such as VPN Concentrators.
    + **Wide access range of privileges:** Giving access to a single device in IPsec-based network can give access privileges for other devices too. So if any device in a home network is compromised, it might affect the corporate network that is connected from the home network (through the IPsec tunnel).
    + **Implementation complexity:** IPsec contains many options and high flexibility, which makes it popular. But also adds to its complexity.
    + **Network compatibility issues:** When we are connected to an IPsec-based VPN, we can't connect to another network due to restrictions in firewalls. Also, IPsec doesn't provide support for multi-protocol and IP multicast traffic.

## Milestones

1980

The first formal standard (or public) version of IPv4 is created as **TCP/IP v4** and defined in RFC 760.

1992

The IETF group for defining the **IP security protocol (IPsec)** is constituted. It actively starts functioning in 1993.

1994

**Network Address Translator (NAT)** is published in RFC 1631. NAT is a technology used to prolong IPv4 availability. It works by translating a private address in an internal network into legal public addresses.

1995

**IPsec** protocols are formally defined and published in **RFC 1825 through RFC 1829**.

1998

IETF publishes IPv6 specification to overcome the limitations of IPv4. IETF initiated the design and development of IPv6 back in 1994. Also in 1998, IPsec protocol definitions are superseded by **RFC 2401** and **RFC 2412**. The mutual authentication and key exchange protocol Internet Key Exchange (IKE) is added to IPsec to create and manage Security Associations.

2005

Latest version of IPsec protocol suite is released with important updates to IKE and ESP protocols, as defined in **RFC 4301** and **RFC 4309**. These documents standardize the abbreviation of IPsec to uppercase "IP" and lowercase "sec".

2008

An IPsec Maintenance and Extensions (ipsecme) working group is formed at the IETF.

IPsec


# AWS Well-Architected Framework

## Summary

Cloud platforms offer a number of services. A typical application can use one or more of these services, each of which can be configured in a number of different ways. Applications deployed on dedicated on-premise servers usually need to be re-architected before migrating them to the cloud. Each application is different and therefore deploying an application to the cloud is usually not a trivial task.

Amazon Web Services (AWS) is one of the big cloud platforms. **AWS Well-Architected Framework** offers a set of guidelines and best practices to help practitioners migrate, manage and optimize their applications and their operations on the AWS cloud. By adopting this framework, we can build and deploy faster, lower or mitigate risks, make informed decisions, and learn AWS best practices.

## Discussion

### What exactly does the AWS Well-Architected Framework provide?

The framework is made of pillars, design principles and questions.

Pillars are foundational to produce stable and efficient systems. By paying attention to these pillars, we can meet expectations and requirements. Moreover, we can focus on our applications and its functional requirements rather than fight with infrastructure issues. For each pillar, the framework lists a number of design principles.

For a practitioner, the framework is,

> A set of foundational questions that will allow you to measure your architecture against these best practices and to learn how to address any shortcomings.

Once shortcomings are identified, along with possible solutions, how much and how quickly you implement these will depend on the complexity of the task and the required skill sets.

### What are the pillars of AWS Well-Architected Framework?

There are six pillars:

    + **Operational Excellence**: Align with business objectives. Give alerts and respond to them in an automated manner. Perform operations with code. Make incremental changes. Learn from failures. Test for responses to unexpected events. Document current procedures.
    + **Security**: Protect information, systems and assets. Do risk assessments. Have mitigation strategies. Secure at all layers. Enable traceability. Implement a principle of least privilege. Automate best practices. Areas include Identity and Access Management (IAM), detective controls, infrastructure protection, data protection, and incident response.
    + **Reliability**: Automatically recover from infrastructure or service disruptions. Test recovery procedures. Scale horizontally to increase availability. Stop guessing capacity. Choose instance type based on application needs. Use multiple availability zones.
    + **Performance Efficiency**: Make efficient use of resources. Enable latency-based routing. Adopt latest technologies and architectures. Experiment more often.
    + **Cost Optimization**: Avoid unneeded costs. Assess resource utilization. Analyze and attribute expenditure. Match supply and demand. Optimize over time. Delete unused resources. Use consolidated billing, spot instances, and reserved instances. Rightsize before/after migrations.
    + **Sustainability**: Reduce the environmental impact of cloud workloads. Understand impact. Maximize resource utilization.

### Which AWS services can help in implementing AWS Well-Architected Framework?

Here we mention a few of them without being exhaustive:

    + **Operational Excellence**: Services are available in areas of preparation, operations, responses. AWS Developer Tools, RunCommand, AWS Batch, AWS CloudFormation and AWS Config can be used in all three areas. Use AWS CloudTrail and AWS CloudWatch for operations and responses.
    + **Security**: IAM, MFA Token, Amazon VPC, AWS CloudFormation, AWS Config, AWS CloudTrail, AWS CloudWatch, Elastic Load Balancing, Amazon S3, Amazon RDS, and AWS Key Management Service are some services to use.
    + **Reliability**: Use AWS CloudFormation and Multi-AZ for failure management. For change management, use Auto Scaling.
    + **Performance Efficiency**: Use AWS Lambda instead of running EC2 instances. Use Amazon CloudFront and Route 53 to reduce latency.
    + **Cost Optimization**: Release Elastic IPs, EBS volumes or RDS instances if not attached to other services. Move archived data from S3 to Glacier. Use EC2 Scheduler to automatically manage start/stop of instances. Use AWS CloudFormation to automate and save time. Amazon SNS can help with expenditure awareness. Use reserved/spot instances. Use AWS GuardDuty for low-cost security monitoring.

### What are the design principles and best practice areas within AWS Well-Architected Framework?

While there are design principles specific to each pillar, some general principles include the following:

    + Stop guessing your capacity needs.
    + Test systems at production scale.
    + Automate to make architectural experimentation easier.
    + Allow for evolutionary architectures.
    + Build data-driven architectures.
    + Improve through game days.A blog article online explains the above and cites relevant AWS services for each.

### Could you share some user stories that show the benefits of applying AWS Well-Architected Framework?

National Instruments started developing on AWS in 2008. In 2013, they started adopting the AWS Well-Architected Framework. Since 2015, all their cloud products follow the framework.

Once they adopted the framework, their scaling latency dropped from 30 minutes to 5 minutes; they optimized from overprovisioning; they removed dependency on data center; and increased developer efficiency.

Before adopting the framework, they took too long to deploy code and scale operations. Since there wasn't much automation, manual intervention was high and operators suffered from alert fatigue. For better security, they adopted IAM rather than give use Root API key. Their roadmap includes multi-region disaster recovery.

Among the tools used or they plan to use are CloudTrail, Amazon Inspector, AWS WAF, AWS Certificate Manager, AWS Config, CloudFormation, Elastic Load Balancing, VPC, CloudFront, Route 53, Multi-AZ, and more.

### What capabilities does AWS offer to implement the AWS Well-Architected Framework?

The **AWS Well-Architected Tool** reviews a workload against best practices and guidance set by the pillars of the Framework. A workload is nothing more than code and resources used by a cloud application. By applying the tool to a workload, we get to know the problem areas and the steps to solve those problems. This tool can be accessed from the AWS Management Console. There's no additional charge to using this tool.

Workloads come in different types or are part of different domains. Each industry or technology domain may require specialized reviews. This is exactly where **AWS Well-Architected Lenses** become relevant. Each lens is customized to a specific workload type. Each lens provides reference architectures, design principles and walkthroughs. AWS comes with many such lenses: Data Analytics Lens, Healthcare Industry Lens, Container Build Lens, SAP Lens, Serverless Applications Lens, and many more. Moreover, users can create custom lenses.

There's also **AWS Well-Architected Guidance**. It focuses on specific use cases or implementation scenarios. This is often in the form of whitepapers.

### Could you name some AWS partners or firms offering consulting on AWS Well-Architected Framework?

There are many vendors providing managed services for AWS. Some of them are recognized by AWS as AWS Well-Architected Framework Review (WAFR) partners. These partners typically ask questions to evaluate your architecture in terms of the framework and suggest ways to improve on the five pillars.

Reviews may use the AWS Well-Architected Tool. A review should focus on only one workload. Each pillar is addressed and prioritized as per business needs. Various stakeholders can participate during the review but there must be a solution architect who's an expert in AWS.

A small sample of such partners include BJSS, Contino, Endava, Foghorn Consulting, Idexcel, KCOM, Logicworks, nClouds, Onica, Piksel Group, Relium, Steamhaus, Telefonica, and Version1.

## Milestones

2012

Within AWS Principal Engineering community, an initiative called **Well-Architected** is started. The aim is to share with customers best practices for architecting in the cloud.

Oct
2015

In a blog post, **AWS Well-Architected Framework** is announced. The author asks, "Have you chosen a cloud architecture that is in alignment with the best practices for the use of AWS?" The framework consists of four pillars.

Sep
2016

Enterprise Support customers get access to a Well-Architected Review for business critical workloads. This review, delivered by an AWS Solutions Architect, provides guidance and best practices to design reliable, secure, efficient, and cost-effective systems in the cloud.

Nov
2016

The framework introduces **Operational Excellence** as the fifth pillar.

Nov
2017

At the AWS re:Invent event, **AWS Well-Architected Partner Program** is launched. Partners can use the principles and best practices of the framework to review a customer's use of AWS services. They guide them towards building secure, performant, and resilient infrastructure to support their applications. Also in November, AWS announces **AWS Well-Architected Lenses** while also releasing two lenses: one for High-Performance Computing (HPC) and one for serverless applications. In November 2021, AWS allows the creation and use of custom lenses.

Nov
2018

AWS blog reports the launch of **AWS Well-Architected Partner Program** at re:Invent event. At this time, there are more than 30 approved partners. Also in November, the **AWS Well-Architected Tool** is introduced. This tool reviews workloads against best practices and gives suggestions. API integration with this tool comes in 2020.

Dec
2021

The framework introduces **Sustainability** as the sixth pillar. This pillar focuses on reducing the environmental impact of any cloud workload via more optimized resource usage and increased efficiency.

AWS Well-Architected Framework


# O-RAN

## Summary

Traditionally, mobile network operators relied on network equipment vendors for RAN hardware. Vendors are few in number and their equipment is specialized and expensive. With 4G/5G, small cell deployments are more common, which is driving up deployment and maintenance costs.

The real problem is that operators are at the mercy of vendors. Vendors implement proprietary protocols and technologies resulting in vendor lock-in. O-RAN changes this by enabling multi-vendor deployments and open interoperable interfaces. O-RAN also enables virtualization and AI-based optimization.

O-RAN is a set of standards that specifies open RAN interfaces, nodes, profiles and services. The operator-led O-RAN Alliance oversees the development of O-RAN. It also facilitates interoperability testing. Operators can therefore buy O-RAN compliant hardware from any vendor. To speed up adoption and innovation, O-RAN Alliance sponsors open source development of RAN software.

## Discussion

### Who started O-RAN and how are they organized?

The main organization behind O-RAN standards is the **O-RAN Alliance**. This was started in February 2018 by five mobile network operators AT&T, China Mobile, Deutsche Telekom, NTT DOCOMO and Orange. In fact, the Alliance was a merger of xRAN Forum (led by NTT DOCOMO) and C-RAN Alliance (led by China Mobile).

By mid-2020, O-RAN Alliance expanded to 24 global operators and 148 contributors. Contributors include chipset manufacturers, network equipment manufacturers, radio element manufacturers, software vendors, service providers, and academic institutions. Despite this diversity, the Alliance is led by operators. Operators are the only members who get to vote at the AGM.

O-RAN Alliance defines RAN standards and supports members towards interoperability testing. In cooperation with the Linux Foundation, it also sponsors the **O-RAN Software Community (SC)** for open source development for the RAN. Thus, O-RAN Alliance defines open architecture and specifications that O-RAN SC uses to develop open source software for industry deployment.

Ultimately, O-RAN Alliance's mission is,

> To reshape the RAN industry towards more intelligent, open, virtualised and fully interoperable mobile networks.

### Could you clarify the terms Open RAN, OpenRAN,  O-RAN, vRAN and C-RAN?

**Open RAN** is a generic term that refers to open RAN architectures including open interfaces, virtualization, and use of AI.

**OpenRAN** is a project initiated by the Telecom Infra Project (TIP). It's an attempt to realize the Open RAN concept. Its work covers 2G/3G/4G/5G. As inputs, OpenRAN uses 3GPP and O-RAN specifications.

**O-RAN** can refer to the O-RAN Alliance or standards created by the Alliance. It complements 3GPP specifications by defining interface profiles, new open interfaces and new nodes. O-RAN addresses 5G RAN interfaces including the X2 interface between 4G and 5G base stations.

**vRAN (Virtualized RAN)** makes the RAN software-defined and programmable. Whereas Open RAN focuses on openness, vRAN is really about moving functionality from hardware to software. vRAN started in 4G with proprietary interfaces, later extended to 2G/3G (OpenRAN) and 5G (OpenRAN or O-RAN).

**C-RAN (Cloud RAN)** is vRAN built on cloud native technologies, such as microservices, containers and CI/CD. Confusingly, C-RAN is also sometimes used to mean Centralized RAN where processing is centralized in CU or DU, away from RU.

### When 3GPP has already standardized RAN, why do we need O-RAN?

NG-RAN nodes are not designed as monoliths. They are disaggregated such that multiple Radio Units (RUs) connect to a Distributed Unit (DU) and multiple DUs connect to a Centralized Unit (CU). The RU-DU link is called **fronthaul**. The DU-CU link is called **midhaul**. The CU-Core link is called **backhaul**.

3GPP has standardized many NG-RAN interfaces: NG (NR-RAN and 5GC), Xn (gNB and gNB/ng-eNB), X2 (gNB and eNB), F1 (gNB-DU and gNB-CU), W1 (ng-eNB-DU and ng-eNB-CU), and E1 (gNB-CU-CP and gNB-CU-UP). However, fronthaul interfaces have not been standardized, thus forcing operators to buy RU and DU hardware from the same vendor. Moreover, X2 was seen as an optional interface. Vendors often added proprietary techniques to this interface. Therefore, when expanding to 5G NR, a 4G/LTE operator with eNB equipment is forced to buy gNB equipment from the same vendor.

O-RAN standardizes the fronthaul. It opens up new interfaces including A1/O1/O2/E2. It specifies aspects of RAN virtualization. It also standardizes F1/W1/E1/X2/Xn interfaces already covered by 3GPP but the intention is to complement 3GPP standards. For example, O-RAN specifies profiles.

### What's the O-RAN architecture?

O-RAN reference architecture disaggregates the RAN into RU, DU and CU. Moreover, user plane (UP) and control plane (CP) are disaggregated. RRC, SDAP and PDCP are in CU. Given a lower layer functional split, RLC, MAC and higher-PHY are in DU; lower-PHY and RF are in RU.

A key functional module is the RAN Intelligent Controller (RIC). This is decomposed into two parts based on real time (RT) capabilities:

    + **RIC non-RT**: Execution exceeding 1s. Part of the Orchestration/NMS layer. Functions include service and policy management, RAN analytics and model training for RIC near-RT. Core algorithms are developed by operators.
    + **RIC near-RT**: Execution within 1s. Next generation RRM that uses AI/ML models trained and passed on from RIC non-RT. Functions include load balancing, RB management, interference detection/mitigation, QoS management, connectivity management and seamless handover control. A robust, secure and flexible platform for third-party applications.RIC near-RT uses the **Radio-Network Information Base (RNIB)**, which gets near-RT state of the radio network via E2 and commands from RIC non-RT via A1. RIC near-RT and Multi-RAT CU platform are often virtualized so that capacity can be distributed across multiple network elements.

### Which are the interfaces relevant to O-RAN?

The NG interface connects NG-RAN to 5G Core. It's completely specified by 3GPP. Otherwise, O-RAN interfaces include:

    + **Open Fronthaul**: DU-RU lower layer functional split. Consists of Control, User and Synchronization (CUS) plane and Management plane. Although Common Public Radio Interface (CPRI) is a standard commonly used at fronthaul, vendors often modified it in proprietary ways. O-RAN standardizes this interface.
    + **A1**: Carries network or UE-level information from eNB/gNB to RIC non-RT to optimize network and ensure SLAs.
    + **E2**: Interfaces RIC near-RT with CU/DU. Carries measurements from CU/DU and configuration commands to CU/DU.
    + **F1/W1/E1/X2/Xn**: Existing 3GPP interfaces but enhanced by O-RAN for multi-vendor interoperation. Interfaces to Multi-RAT CU platform. F1 is the CU-DU higher layer functional split.
    + **O1**: An Operations and Maintenance (OAM) interface. Functions include management of provisioning, fault supervision, performance assurance, tracing, files, heartbeat, and Physical Network Function (PNF) software.
    + **O2**: Service provider functionality resides in Service Management and Orchestration (SMO) while cloud provider functionality resides in O-Cloud. This interface connects the two. It's decomposed into cloud infrastructure management and deployments on the infrastructure.

### What are O-RAN Workgroups and Focus Groups?

The work of O-RAN is organized into workgroups. These include **WG1**: Use Cases and Overall Architecture, **WG2**: The Non-real-time RAN Intelligent Controller and A1 Interface, **WG3**: The Near-real-time RIC and E2 Interface, **WG4**: The Open Fronthaul Interfaces, **WG5**: The Open F1/W1/E1/X2/Xn Interface, **WG6**: The Cloudification and Orchestration, **WG7**: The White-box Hardware Workgroup, **WG8**: Stack Reference Design, and **WG9**: Open X-haul Transport.

In addition, there are a few focus groups:

    + **Standard Development Focus Group (SDFG)**: Strategizes standardization effort. Coordinates and liaises with other standard organizations.
    + **Test & Integration Focus Group (TIGF)**: Defines test and integration specifications across workgroups.
    + **Open Source Focus Group (OSFG)**: Successfully established O-RAN SC. Otherwise, it's currently dormant.

### In O-RAN, what is meant by white-box hardware?

White-box implies general purpose vendor-neutral hardware, often called **Commercial Off-the-Shelf (COTS) hardware**. White-box hardware is common in IT. With O-RAN, it's coming to telecom.

White-box hardware is expected to bring down deployment costs. It enables decoupling of hardware and software. It opens up the market to small players. Hardware vendors can make equipment conforming to O-RAN white-box specifications without worrying about the complexity of software. Operators and software vendors can focus on software with the foreknowledge of the hardware.

O-RAN specifies "high performance, spectral and energy efficient white-box base station hardware". It publishes reference hardware designs and software architecture for both O-DU and O-CU, where the prefix "O-" signifies O-RAN specification.

O-RAN has defined hardware requirements for different deployment scenarios (indoor pico, outdoor pico/micro/macro, IAB) and different architectures (split vs integrated). Performance requirements are specified in terms of peak data rate, peak spectral efficiency, bandwidth, latency, and mobility. Among the different split architectures, WG7 has considered options 6 (all PHY in O-RU), 7-2 (lower PHY in O-RU, higher PHY is O-DU) and 8 (all PHY in O-DU).

### Could you share details on O-RAN slicing?

Network slicing is defined primarily by 3GPP but there are aspects studied by ONAP, ETSI and GSMA. O-RAN includes a slicing framework and architecture to realize 3GPP's network slicing. In particular, there's impact on RIC, O-CU, O-DU, and A1/E2/O1/O2 interfaces.

A network slice has two parts, RAN slice and CN slice. O-RAN slicing is concerned with only RAN slices. Non-RT RIC should be aware of RAN slices, slice configuration and performance metrics. These become inputs to AI/ML models for slice assurance and optimization that's done over O1 interface.

Dynamic slice optimization is done by Near-RT RIC to prevent SLA violations. Near-RT RIC collects slice performance metrics over E2 interface, applies its algorithms and optimizes slices over E2.

3GPP defines Network Slice Management Function (NSMF) and Network Slice Subnet Management Function (NSSMF). For these functions, O-RAN provides multiple deployment options. They can be within or outside the SMO.

### What are some challenges when it comes to adoption of O-RAN or Open RAN in general?

Even if operators introduce O-RAN compliant hardware for 4G/5G, their earlier investments in legacy equipment prevent them from reaping the benefits of a single virtualized and flexible RAN architecture. Lack of fibre or cost-efficient transport can be bottleneck for deployment of open fronthaul and midhaul.

Some network vendors such as Ericsson and Huawei are holding out against O-RAN. Huawei hasn't joined the O-RAN Alliance. It believes that its integrated offering is more performant that can't be matched by disaggregated white-box hardware. However, Huawei supports the ONAP initiative for better interoperability.

Detractors of O-RAN believe that disaggregated hardware can compromise overall network security.

It's been said that the "main issue with disaggregation is aggregation". While disaggregated multi-vendor hardware is attractive, it's not trivial to integrate and achieve an end-to-end interoperable solution. Operators have the challenge of selecting the right hardware from multiple vendors and also build in-house integration expertise. On the other hand, there's a new opportunity for system integrators.

### What are some useful resources to know more about O-RAN?

O-RAN specifications can be downloaded for free but requires acceptance of O-RAN ALLIANCE Adopter License Agreement.

To follow the status of O-RAN open source implementation, visit the homepage of O-RAN SC. The site includes documentation of the latest release.

## Milestones

2016

The **Telecom Infra Project (TIP)** is founded to address the lack of innovation, high costs and a closed ecosystem ruling the telecom world. It's starts the **OpenRAN** initiative. OpenRAN is meant to spur innovation, enable supplier diversity and reduce costs. First OpenRAN trials start in 2017, in India and Latin America, particularly in rural areas where user base is low and Average Revenue Per User (ARPU) is also low.

Feb
2018

Operators AT&T, China Mobile, Deutsche Telekom, NTT DOCOMO and Orange come together and start the **O-RAN Alliance**. O-RAN Alliance creates standards, unlike TIP that doesn't create standards but promotes OpenRAN.

2019

In an online article, Tamaskar of Radisys writes that 2019 may be the year for Open RAN. She notes the work of both O-RAN Alliance and Telecom Infra Project (TIP). We may see more collaboration between these two groups. Just as Self-Organizing Network (SON) brought automation to RAN management, planning and optimization, O-RAN will use AI to automate operations and reduce costs.

Feb
2019

Rakuten announces the world's first virtualized, cloud-native greenfield 4G network. However, it's not based on O-RAN standards. It's based on Nokia's X2 interface, that Nokia opened up for integration with another vendor.

Jun
2019

O-RAN WG5 publishes **O-RAN NR profiles for EN-DC (E-UTRA New Radio Dual Connectivity)** for both C-plane and U-plane. Initial 5G deployments are expected to be in *Non-Standalone (NSA)* mode, in which a UE will connect to LTE/eNB as master node and via Dual Connectivity connect to 5G/gNB as slave node. This is referred to as *EN-DC* or *option 3*. This deployment option was first standardized by 3GPP in Release 15 (early drop) in December 2017.

Sep
2019

NTT DOCOMO claims to be the world's first to deploy multi-vendor RAN conforming to O-RAN specifications. Its new pre-commercial 5G service is deployed with its existing 4G network. This is what we call 5G NSA network that makes use of X2 interface between 4G and 5G base stations.

Nov
2019

The O-RAN Software Community releases the first version (Amber Release) of its **open source software**. Since O-RAN specifications continue to be developed, this must be considered as pre-specification software with limited capabilities. Subsequent releases include Bronze Release (June 2020) and Cherry Release (December 2020).

Feb
2020

Mavenir founds the **Open RAN Policy Coalition (ORPC)**. This is a coalition to promote policies leading to open interoperable RAN solutions. It's not a standardization body. In May, Nokia joins ORPC. In January 2021, ORPC joins a coalition of coalitions that includes BSA, GSMA, CableLabs, and TIP.

May
2020

O-RAN Alliance and GSMA announce partnership to accelerate the adoption of Open RAN and thereby benefit from new virtualization architectures. This follows an earlier announcement in February when O-RAN Alliance partnered with TIP.

Jul
2020

Nokia and Samsung announce **OpenRAN products for 5G**. Meanwhile, Ericsson and Huawei make no similar announcements, though Ericsson is a contributor in O-RAN Alliance.

Dec
2020

The O-RAN Software Community makes the **Cherry Release** of O-RAN open source software. It's aligned with latest O-RAN specifications and brings the implementation closer to commercial deployments.

O-RAN


# Information Retrieval

## Summary

Information Retrieval (IR) refers to finding out relevant information from any kind of data. It may be audio, video, document, article or an image. Traditional ways of information retrieval consist of breaking down data into subsets or clusters across dimensions and finding relevant information according to problem statement.

There are various ways to retrieve information from any unstructured data. It started with rule-based models. More recently, neural networks are being used.

The way humans understand is far more complex than how machines understand. Machines follow the logic of finding out similarities, differences or links with the input data. But it's hard to interpret many-to-many relationships. Humans are able to do this for all previous information stored in memory whenever an image, text or video appears.

## Discussion

### What's the difference between Information Retrieval (IR) and Information Extraction (IE)?

IR is about searching and retrieving unstructured documents that fulfil a specific search query. IE is about obtaining structured representation of key information from unstructured documents. For example, IR can help us find all documents about fishing whereas IE can tell who went fishing and what type of fish was caught.

Information retrieval is the process of gathering information or sources that are appropriate to the topic from a collection of raw text data.

Information extraction is a kind of automated process where rule-based algorithm is applied to structured data after it is obtained from any unstructured source. It involves NLP techniques of cleaning and arranging it in a matrix. It also involves activities like automatic annotation or content extraction.

### Could you mention some applications of IR?

Web search is an example of IR. A search engine might search billions of web documents to respond to a query such as "Italian restaurants near me" or "flights to London tomorrow". A more personal example is to search through emails. IR in this case is typically executed by the mail program such as GMail.

Searching through online discussion forums or Q&A archives is another IR application. Text classification is typically used to classify posts, questions and answers.

In any enterprise system, we might need to retrieve patent documents, research papers or other publications based on a keyword or phrase. In a library, we might want to obtain books with specific words in its title, or by author name, or by genre. In an e-commerce site, we might want to apply various criteria and see a filtered list of products.

As an example of IR for multimedia, MPEG-7 is a standard that's used to describe multimedia content. This metadata can be searched to retrieve audio/video clips that are of interest to the user.

### What are some essential terms used in information retrieval?

While the term "record" is common in databases, the equivalent term in IR is **document**. It refers to a basic unit of data stored in the system. A document can be a newspaper article, an encyclopaedia entry, a paragraph in a report, a web page, an entire website, etc. It really depends on the application.

A set of documents indexed and stored is called a **collection**. Alternative names include archive, corpus, and digital library.

**Query** is what the user submits to the system to meet her information need. A query is composed of one or more terms. A **term** is a lexical item but it could be a phrase as well.

To search through each document is a slow inefficient process. Instead, each document is indexed in advance. The **index** is a structured representation of the document. Searching through this representation is much faster. A similar representation of the query aids the retrieval function.

In practice, IR systems use **inverted index** that maps each index term to a list of documents in which it occurs.

### Does information retrieval search through structured or unstructured documents?

Information retrieval typically concerns itself with **unstructured documents**. IR started with a focus on text. More recently, due to machine learning and improved processing capability, other types such as image, audio, and video can also be retrieved.

While documents may be unstructured, often they are accompanied by **structured metadata**. Metadata is data about the document. For example, an email will have fields date, from, to, subject, and body. This is useful structured information. Another example is a photograph taken with a digital camera. Useful metadata might include date, camera model, exposure setting, and picture dimensions.

Sometimes documents are **semi-structured** in the sense that specific information are in "standard" locations that heuristic algorithms can identify. Text documents with sections and sub-sections are also semi-structured. XML documents can be seen as semi-structured that IR systems can exploit.

### What are the various models of IR?

IR models can be categorized into Classic, Structured and Browsing models. Classic models include Boolean, Vector Space and Probabilistic models.

Boolean primitives (AND, OR, NOT) can be used in queries according to user's information need. This requires some skills for any complex IR. Different vocabularies will result in problems of string matching. This is the most used approach.

Vector space models use vectors to map documents, phrases or terms. Words/terms can be categorized to high or low resolving power weights according to vector dimension of document. This helps in finding out positive/negative or similar/opposite matches and retrieve information.

Probability distribution models use different distribution types to find out similarity between documents. It can be one of two types: similarity-based and expected-utility-based. TF-IDF and HMM can be used in different frameworks to generate multiple words within the same model, making comparison easier. HMM gives better accuracy than traditional TF-IDF models.

Two broad matching strategies are:

    + **Literal Term**: Vector Space Model (VSM), Hidden Markov Model (HMM), Language Model (LM)
    + **Concept**: Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (PLSA), Topical Mixture Model (TMM)

### How is information retrieval represented?

Information is any unstructured form of data which consists of insights of any particular domain. It needs to be converted to a format that can then be parsed/extracted into structured form. Information in text format can be represented as document representation or query representation.

The content of a document can be represented as a collection of terms such as terms, phrases or other units. Every term will be having their weights which gives its importance to that document. A proper term weighting is needed according to various aspects of words, phrases, names, statistical, linguistic and other means for better IR.

Query representation is based on conventional approach to formulate a query based on a keyword or set of keywords. But in several cases there's no relevant context for user to select appropriate keywords. In that case, maximal frequent sequences is used to retrieve relevant documents from an input document. A **bag-of-words model** is used to identify keywords and extract similar documents.

### Which are some basic techniques used in information retrieval?

Let's say a query has the term 'processed'. A strict match for this word will leave out documents that contain variants such as 'processing' and 'process'. **Stemming** or **lemmatization** solve this but they create their own problems.

Often it doesn't make sense to index words such as 'to', 'be', 'for', etc. A **stop list** of these high-frequency words are often omitted for indexing.

Suppose the user searches for 'high blood pressure'. The IR system may also search for 'hypertension' and 'hypertension, renal'. This is called **query expansion**. Query expansion usually involves pre-generating a suitable thesaurus using **term clustering**.

When a query has multiple terms, some terms may be more important than others. This is done by **term weighting**. **TF-IDF** is a popular approach.

To improve IR performance, **relevance feedback** can be used, typically in VSM. User is shown a small set of retrieved documents to get feedback on what's relevant and what's not. The IR system then refines the query to improve performance.

### What are some useful resources on Information Retrieval?

Greengrass (2000) offers a comprehensive survey of IR. Nyamisa et al. (2017) is another survey paper. Zhou et al. (2006) introduces basic terms and concepts. There's also a comprehensive online IR glossary.

Stanford University's CS276 course on IR is worth studying. Among the books for IR, you can look into Manning et al. (2009), Croft et al. (2015) and Frakes et al. (1992).

For IR research, you'll need collections. Wiki Small/Large and CACM collections are available for free download. TIPSTER Complete is another useful collection. Other important collections include the Cranfield collection, TREC, GOV2, Reuters-21578, Reuters-RCV1, and 20 Newgroups. For cross-language IR, there's NTCIR and CLEF.

To keep track of latest developments, ACM's Special Interest Group on Information Retrieval (SIGIR) is the place to visit.

## Milestones

Dec
1931

Emanuel Goldberg receives a US patent for a machine that does information retrieval. The machine is opto-electric in nature. Light is passed through a negative search plate containing the search terms. Document matches are picked up by activating a photoelectric cell. All matches are recorded on a photographic plate.

Mar
1950

Calvin Mooers coins the term **Information Retrieval**.

1952

H. P. Luhn describes an early IR system from IBM. It makes use of punched cards and a photoelectric scanning unit. Information is indexed and encoded into **record cards**. A query is input as a **search card**. Plug connections on a switchboard control how search terms are to be combined. This is therefore one of the earliest implementations of the **Boolean Model** for IR.

1957

H. P. Luhn proposes a **statistical approach** to IR. Content can be identified based on the frequency of occurrence of words. This also means that documents can be indexed automatically based on the words they contain. This is therefore the first attempt at **automatic indexing**. The system uses statistical information to group words into "notional" families.

1959

Cavin Mooers notes that often using information is not rewarded. Those who are at work are seen as getting the job done rather than fussing about information. Having information means you have to read it, understand it, store it carefully and make decisions based on it. These observations lead Mooers to define **Mooers' Law**,

> An information retrieval system will tend not to be used whenever it is more painful and troublesome for a customer to have information than for him not to have it.

1960

Maron and Kuhns publish a paper titled *On Relevance, Probabilistic Indexing and Information Retrieval*. Where previous IR systems returned matching documents, the idea in this paper is to rank documents by computing **probability of relevance**. The paper also introduces the notion of **query expansion**. In years to come, this goes on to become one of the most influential papers in the field of IR.

1965

Salton and Lesk propose **SMART** retrieval system. It goes beyond Luhn's work of 1957. It uses the original search words along with their word stems. It uses synonym dictionaries to account for variations in vocabulary. It uses relations between words and surrounding words to determine the nature of content. Over the years, the SMART system develops or uses many important concepts such as vector model, term weighting, relevance feedback, and cosine similarity measure.

1976

Robertson and Jones implement a **probabilistic model** to IR. From the distribution of index terms in documents, they determine term weights.  This work is perhaps the first real application of the probability to IR as proposed by Maron and Kuhns in 1960.

1982

Salton et al. propose the **extended Boolean model**. Output from a traditional Boolean model is hard to control and retrieved documents are not ranked. The extended model assigns weights to terms in both queries and documents. This enables the model to rank retrieved documents based on similarity to queries.

1988

Dumais et al. note two common problems in IR: same word has many meanings, different words express the same concept. These result in irrelevant documents or missed relevant documents respectively. They therefore propose a novel method that maps terms and documents into a lower dimensional **semantic space**. In later years, this field is named **Latent Semantic Analysis (LSA)**.

1992

In the US, **Text Retrieval Conference (TREC)** is launched to facilitate research and collaboration in IR. It goes on to become an almost annual event. The growth of the web and the need for large-scale IR makes TREC all the more relevant. TREC produces test collections that are useful for IR research.

Aug
1998

At SIGIR'98, Ponte and Croft propose the use of **Language Modelling (LM)** to the task of IR. They note that typical IR systems lack a good indexing model. Existing indexing models make unwarranted parametric assumptions. They therefore propose a single non-parametric model for both indexing and retrieval. They show better performance compared to TF-IDF weighting. In 2001, Zhai and Lafferty study different smoothing techniques for such language models.

Sep
1998

Google is founded for web search. By now, many other search engines are already popular: Lycos (1994), Yahoo! Search (1995), Excite (1995), AltaVista (1995), Yandex (1997). These engines become the best instantiations of IR. They use features that until now were only experimental.

2016

There's growing interest in applying neural networks to IR. This is called **Neural IR**. The network learns representations of queries and documents. In one application of Neural IR from 2019, two deep models are trained, one for image and one for text. This allows a user to retrieve images based on a textual query, or vice versa. However, early work on content-based image retrieval can be traced to the work of Wan et al. (2014).

Information Retrieval