# Question Generation

## Summary

Given some content, the goal of Question Generation (QG) is to automatically generate a set of questions that can be answered by that content. This content can be in the form of sentences, paragraphs, documents, databases or even images.

A common application of question generation is to automatically prepare questions for quizzes, assessments or even FAQs that present the content in a more readable form.

Traditionally, rules and templates were used to generate questions. Since the mid-2010s, there's been greater interest in using statistical methods, particularly neural networks. Question generation is also closely linked to other NLP tasks such as question answering. In fact, early neural network models were adapted from machine translation, text summarization and image captioning.

## Discussion

### What are some applications of question generation?

In education, question generation helps in assessing reading comprehension. Students can use it for self-assessment. It can be used to create quizzes and online tests without manual effort from educators. By one estimate, teachers spend 50% of their time towards student assessments.

Many versions of tests can be created to prevent cheating. For online courses and adaptive learning, these variations are helpful. A solution-oriented approach generates questions based on skills and concepts. A template-based approach uses variables, constraints and templates.

For training question answering (QA) or dialogue systems, QG can produce question-answer pairs. Chatbots trained on QG can ask relevant questions in an ongoing dialogue.

Often questions are generated from available answers. But it's possible to generate questions that seek more information or clarification. Such a model can be trained using conversational QA datasets.

Applications also include help systems and multi-modal conversations involving virtual agents. One research applied QG for user authentication in online systems.

### What are the different types of questions generated by QG systems?

**Multiple Choice Questions (MCQs)** are commonly generated for student assessments. Along with question, the correct answer and a few incorrect answers (called *distractors*) are generated. For English vocabulary assessment, QG systems could use WordNet, web searches and Word Sense Disambiguation.

**Fill-in-the-blank questions** are simpler than MCQs since there's no need to generate distractors. Fill-in-the-blank statements are first generated, from which questions are framed via NLP analysis.

**Wh-questions** are questions that ask "what, where, when, which, who, why" and "how". They could also include imperative statements that start with "name, tell, find, define or describe".

**Factoid questions** are based on simple facts. **Non-factoid questions** require deeper analysis of the source text. These could involve causation, inference, reasoning or interpretation. For example, "What colour is the sky?" is a factoid question. "Why is the sky blue?" is an non-factoid question. Similarly, we distinguish between closed questions and open questions. Open questions are of the form "To what extent…", "Why…", "Should…", etc.

### What are the typical challenges in generating questions?

QG is quite unlike many NLP tasks. Whereas NLG generates sentences from some semantic input, the input to QG is often in natural language. Unlike machine translation, both input and output in QG are in the same language, and often there's no one-to-one correspondence. QG also significantly reorders and rephrases words.

Whereas Question Answering (QA) is often *extractive* (selecting text spans from the input), question generation is often *abstractive* (generating text not necessarily present in the input). Moreover, a variety of questions can be framed from the many relations among words and phrases in the input. In one example, a single sentence generated 2000+ questions. QG systems must therefore weed out silly questions from important ones.

Often a generated question might require improvements to its grammar, form or simplicity. QG systems need to figure out long-distance dependencies to do this.

### What's a typical data pipeline in a question generation system?

QG systems can be classified based on the input type. The common ones are text-based QG, visual QG, and structured data-based QG. Structured QG uses knowledge graphs and semi-structured tables, where input is subject-object-relation triplet.

QG pipelines might depend on the type of input, type of questions and the nature of the model. In general, we expect three steps:

    + **Pre-processing**: Convert complex sentences into simpler ones, perhaps using parse trees. Using POS, dependency labels and SRL, classify sentences to determine type of question. Select content based on relevance to the final task. From web sources, mine and predict question patterns.
    + **Question Construction**: Generate correct answers and distractors for MCQs. Select gap position for fill-in-the-blank questions. Transform assertive sentences into interrogative forms. Control the difficulty of questions.
    + **Post-processing**: Construct the final surface form of questions. Rank questions to prioritize high-quality questions.Visual QG systems might detect/classify objects and identify features via CNN (colour, size, shape, relations). Features may be object-specific or on the entire image. These are then used to generate questions.

### Which are the neural network models for question generation?

Du et al. adopted the encoder-decoder architecture of a **seq2seq** model. Decoder was an LSTM that generated the question. Both an input sentence and its containing paragraph were encoded via separate BiLSTM and then concatenated. Via an **attention layer**, the decoder learned to pay attention to more relevant parts of the encoded input representation. Encoders and decoder had two layers each.

Later models included target answer at the input to avoid questions such as "What is mentioned?" Position embeddings may be used to give more attention to answer words closer to context words. Some decoders predict question words (when, how, why, etc.) before generating the question.

Seq2seq models struggle to capture paragraph-level context that's needed to generate high quality questions. Zhao et al. extended the seq2seq model with answer tagging, **maxout pointer mechanism** and a gated self-attention encoder. Another approach uses **multi-stage attention**.

**Transformer-based models** also capture longer context. One research pre-processed the input with NER. Other models have adapted BERT and GPT-2 for QG.

### How is question answering relevant to question generation?

About 2000-2010, interest in QG was to aid QA systems. Later, motivated by applications, QG became an important task on its own. However, the synergy between QG and QA continues. In 2017, papers were published that showed that QG and QA are dual tasks.  Similarly, visual QG and visual QA are considered dual tasks.

Joint probability between questions and answers can be applied. The probabilistic correlation between QA and QG can be used to regularize the training process. A QA model evaluates if a question generated by a QG model can be answered. A QA-specific signal feeds into the QG loss function. The QG model estimates the probability of a question given the answer, which helps QA. It's possible to simultaneously training both models. Tang et al. used seq2seq model for QG and RNN for QA.

For supervised training of any end-to-end QA model (using neural networks) we require lots of training data. Where data is limited, QG helps by automatically creating questions from any given input. Duan et al. achieved this by crawling community-QA websites such as Quora or Yahoo!Answers. They also generated questions from passages using CNN and RNN.

### Which datasets are useful for building QG models?

Datasets assembled for Question Answering (QA) or Machine Comprehension (MC) can be reused in QG. For example, MC datasets typically have document-question-answer triples with the goal of predicting the answer. QG models are instead trained to predict the question.

WikiQA is a text-based dataset. WebQuestions and SimpleQuestions are knowledge-based datasets. SQuAD and MS MARCO are for MC. SQuAD has 150K question-answer pairs and another 50K questions without answers. NewsQA has 120K question-answer pairs gathered from CNN news. Based on Freebase, one corpus has 30M factoid question-answer pairs. SciQ has 13.7K MCQs on biology, chemistry, earth science, and physics. Some of these datasets were crowdsourced via Amazon Mechanical Turk (AMT).

Specifically for QG, QG-STEC dataset was created in 2010. It has 180 questions from sentences and 390 questions from paragraphs. Medical CBQ is a corpus for the medical domain. MCQL has 7.1K MCQs from web crawling on topics of biology, physics, and chemistry.

For non-factoid questions, community-driven question answering websites could be harvested for a large volume of training data. Yahoo!Answers and Quora are examples.

### How can I evaluate question generation models?

**Intrinsic evaluation** analyzes the QG model's output based on criteria such as grammaticality or fluency. **Extrinsic evaluation** is about measuring system or application performance in which the QG model is used. The trend has been towards automatic and intrinsic evaluation. However, there's no common framework that makes it easy to compare different QG models.

Suppose students' language proficiency is evaluated from two sets of questions, one machine-generated and the other human-generated. This is extrinsic evaluation. Instead, if experts subjectively evaluate the machine-generated questions, it's intrinsic evaluation.

With **human evaluation**, criteria to consider are relevance, syntactic correctness, fluency, ambiguity, question type, and variety. These are quality judgements. It's also important for annotators to agree with one another.

For **automatic evaluation**, ROUGE, precision, recall, and F1 are some useful intrinsic metrics. Perplexity based on a language model is a metric to measure fluency. In general, models with high scores could perform poorly on human evaluation. *Question answerability* may be a better metric compared to n-gram similarity metrics such as BLEU, METEOR, NIST.

## Milestones

1973

In an article titled *Conditions on transformations* Chomsky proposes a linguistic approach in which questions are really transformations of canonical declarative sentences.

May
2003

Mitkov and Ha **generate multiple choice questions** using "transformational rules, a shallow parser, automatic term extraction, word sense disambiguation, a corpus and WordNet". Sentences with domain-specific terms are considered for question generation. Distractors are selected to be semantically close to the correct answer. For example, if the correct answer is 'syntax', 'semantics' and 'pragmatics' are better choices than 'football' or 'chemistry'. An example rule is to generate "Which HVO" (H for hypernym) for an SV-type sentence.

Jul
2003

Echihabi and Marcu generate factoid question-answer pairs by mining structured or semi-structured databases such as World Fact Book, Biography.com or WordNet. They apply **extraction patterns** from information retrieval to manually define **template pairs**. However, their end goal is not question generation but question answering. In fact, research on question generation in this decade is mostly motivated by the task of **question answering**.

2009

Chen et al. note that self-questioning improves reading comprehension among learners. In this context, they automatically generate questions from natural text. However, narrative text is different from informational text. The former involves characters, behaviour and mental states. The latter involves descriptions and explanations. For each question type, they use rules and question templates. Their approach uses **Semantic Role Labelling (SRL)**. Lindberg et al. (2013) also use SRL and templates.

Jun
2010

Heilman and Smith propose a **rule-based QG system** for generating factoid questions. Rules are based on linguistic knowledge. Rules are generic and not dependent on sentence types. Their two-step process first simplifies input sentence and then transforms it into a question. Answer phrases (noun phrases or prepositional phrases) are identified and then replaced with question phrases. They over-generate questions and then rank them using a logistic regression model.

Jul
2010

Following the annual tasks run by CoNLL, the first *Shared Task Evaluation Challenge on Question Generation (QG-STEC)* is organized. Although inspired by NLG, researchers note that QG is currently seen as a discourse processing task rather than an NLG task. This shared task aims to provide questions in an application-independent manner based on core ideas in the input text. They focus on two categories: QG from sentences and QG from paragraphs. For the latter, questions are of type "who, where, when, which, what, why, how many/long, yes/no." Input sources are Wikipedia, OpenLearn, and Yahoo!Answers.

2011

While previous research work mostly considered sentence-level context, Agarwal et al. consider **paragraph-level context**. Discourse connectives connect two clauses or sentences to establish temporal, causal, elaboration, contrast, or result relations. They make use of these connectives to identify question types and generate questions of the type "why, when, give an example, and yes/no". They use QGSTEC-2010 and Wikipedia datasets.

2015

Chali and Hasan propose **topic-to-question** generation in which input texts are about a specific topic. Generated questions are therefore topic-focused. Named entities, semantic role labels and predicate argument structures are used to generate questions. Using Latent Dirichlet Allocation (LDA), they identify sub-topics to rank questions by relevance. They also evaluate the syntactic correctness of generated questions. They argue that this approach enables a QA system to answer a complex question from simpler questions.

2017

**Sequence-to-sequence** neural network models with **attention**, first applied to machine translation in 2014, is applied to the task of question generation. In one approach using BiLSTM, attention-based sentence encoder and paragraph encoder are used. Input could include rich features such as POS and NER tags. Another research includes **pointer-softmax** to include in the question relevant words from the input. The model considers both input document and answers. It also applies **reinforcement learning** by feeding the questions to a QA system.

Jun
2018

Tang et al. explore how QG and QA models can be jointly trained. In particular, **policy gradient method** can be used to update the QG model based on QA-specific signals. They apply Generative Adversarial Network (GAN) to generate question-answer pairs. A collaboration detector (CD) determines positive versus negative training instances. In their QG model, they adapt seq2seq model to **Table2Seq**. Table headers, cells, and caption are encoded into continuous vectors using Bidirectional GRU. Decoder uses attention-based GRU with copying mechanism.

Jun
2018

Chen et al. note that Stanford Question Answering Dataset (SQuAD) and RACE datasets were collected for reading comprehension and not suited for assessing higher cognitive skills such as applying or analyzing. They address this by creating **LearningQ**, a dataset of 230K document-question pairs from online learning platforms. Dataset uses TED-Ed and Khan Academy as sources. They show that QG models that currently do well on other datasets are challenged by LearningQ.

Nov
2019

Chan and Fan use BERT for QG. Their simplest model uses only context and answer as inputs. It performs poorly because it doesn't leverage previously decoded tokens, unlike seq2seq models. They therefore propose **BERT-HLSQG** that marks the answer in context plus feeds in previously decoded answer tokens. This model achieves state-of-the-art results on SQuAD.

Question Generation


# Maximum-Entropy Markov Model

## Summary

There are many systems where there's a time or state dependency. These systems evolve in time through a sequence of states and current state is influenced by past states. For example, there's a high chance of rain today if it had rained yesterday. Other examples include stock prices, DNA sequencing, human speech or words in a sentence.

One problem is that we may have observations but not the states. For example, we know the sequence of words but not the corresponding part-of-speech tags. We therefore model the tags as states and use the observed words to predict the most probable sequence of tags. This is exactly what Maximum-Entropy Markov Model (MEMM) can do.

MEMM is a model that makes use of state-time dependencies. It uses predictions of the past and the current observation to make current prediction.

## Discussion

### Which are the building blocks of Maximum-Entropy Markov Model?

Let's consider the task of email spam detection. Using **logistic regression**, we can obtain the probability that an email is spam. It's essentially binary classification. We can extend this to a multiclass problem. For example, in image analysis, we're required to classify the object into one of many classes. We estimate the probability for each class. Rather than take a hard decision on one of the outcomes, it's better to output probabilities, which will benefit downstream tasks.

Multinomial logistic regression is also called softmax regression or **Maximum Entropy (MaxEnt)** classifier. Entropy's related to "disorder". Higher the disorder, less predictable the outcomes, and hence more information. For example, an unbiased coin has more information (and entropy) than a coin that mostly lands up heads. MaxEnt is therefore about picking a probability distribution that maximizes entropy.

Then, there's **Markov Chain**. It models a system as a set of states with probabilities assigned to state transitions. While MaxEnt computes probabilities for each input independently, Markov chain recognizes that there's dependency from one state to the next.

Thus, MEMM is about maximizing entropy plus using state dependencies (Markov Model).

### Could you explain MEMM with an example?

Consider the problem of POS tagging. Given a sequence of words, we wish to find the most probable sequence of tags. MEMM predicts the tag sequence by modelling tags as states of the Markov chain. To predict a tag, MEMM uses the current word and the tag assigned to the previous word.

In reality, MEMM doesn't take a hard decision on what tag to select. Using MaxEnt, it calculates the probability of each tag. Probabilities of past tags are used to take a soft decision so that the word sequence as a whole get the best possible tag sequence. The algorithm that can do this efficiently is the **Viterbi algorithm**.

The formula in the figure assumes a sequence of L words. Z is a normalizing term so that probabilities are in range [0, 1] and they all add up to 1.

### What are some applications where MEMM is used?

MEMM has been used in many NLP tasks including POS tagging and semantic role modelling. It's been used in computational biology for protein secondary structure prediction.

One study used MEMM for recognizing human facial expressions. Human expressions are modelled as states. Observations are from video sensors.

### How is MEMM different from Hidden Markov Model (HMM)?

HMM is a **generative model** because words as modelled as observations generated from hidden states. Even the formulation of probabilities uses likelihood P(W|T) and prior P(T). On the other hand, MEMM is a **discriminative model**. This is because it directly uses posterior probability P(T|W); that is, probability of a tag sequence given a word sequence. Thus, it discriminates among the possible tag sequences.

It can also be said that HMM uses **joint probability** for maximizing the probability of the word sequence. MEMM uses **conditional probability**, conditioned on previous tag and current word.

In HMM, for the tag sequence decoding problem, probabilities are obtained by training on a text corpus. In MEMM, we build a distribution by adding features, which can be hand crafted or picked out by training. The idea is to select the maximum entropy distribution given the constraints specified by the features. MEMM is more flexible because we can add features such as capitalization, hyphens or word endings, which are hard to consider in HMM. MEMM allows for diverse non-independent features.

### What are some shortcomings of MEMM?

MEMM suffers from what's called the **label bias problem**. Once we're in a state or label, the next observation will select one of many transitions leaving that state. However, the model as a whole would have many more transitions. If a state has only one outgoing transition, the observation has no influence. Simply put, transition scores are normalized on a per-state basis.

Consider a character-level input sequence "rib". We have an MEMM to select between "rib" and "rob". When we receive 'r', both paths 0-1 and 0-4 get equal probability mass. When 'i' arrives, we have nowhere to go from state-4 except to state-5 and pass on the entire probability mass along this path. When 'b' arrives, at state-3, both paths are equally likely.

The label bias problem was first noted by L. Bottou in 1991. **Conditional Random Fields (CRFs)** is an alternative model that solves this.

## Milestones

1996

Berger et al. at the IBM Watson Research Center note that the concept of maximum entropy is not new but were not common in real-world applications due to high computational requirements. This is changing due to more powerful computers. They apply MaxEnt to some NLP tasks such as sentence segmentation and word reordering. They show that MaxEnt and maximum likelihood give the same result,

> The model with the greatest entropy consistent with the constraints is the same as the exponential model which best predicts the sample of data.

1996

Adwait Ratnaparkhi at the University of Pennsylvania applies MaxEnt model along with Markov model to the task of part-of-speech tagging. He simply calls it *Maximum Entropy Model*. The model is able to use rich contextual features. It achieves state-of-the-art accuracy of 96.6%. This work leads to his PhD in 1998. The model considers past two POS tags, and features of past two and next two words. It combines them into a single exponential model.

1998

Saul and Rahim make use of **Markov Processes on Curves (MPCs)** for automatic speech recognition. Acoustic feature trajectory is a continuous variable that evolves jointly with phonetic transcriptions modelled as discrete states. This work later inspires MEMM by McCallum et al. who apply it to discrete time.

2000

McCallum et al. coin the term **Maximum-Entropy Markov Model (MEMM)**. They apply it to information extraction and segmentation. They note that the three well-known problems solved by HMM can also be solved by MEMM. Exponential models of MEMM can be training using **Generalized Iterative Scaling (GIS)**, which is similar to Expectation-Maximization (EM) algorithm used in HMM.

Jun
2001

Lafferty et al. propose an alternative discriminative model called **Conditional Random Fields (CRFs)** to overcome the label bias problem of MEMM. MEMM can be biased towards states with fewer successor states. While MEMM uses per-state exponential models for the conditional probabilities, CRF uses a single exponential model for the joint probability of the entire label sequence given the observation sequence. In other words, CRF uses global normalization instead of per-state normalization. However, CRF is slower to train that MEMM.

Maximum-Entropy Markov Model


# Static Site Generators

## Summary

A static site generator is an application that uses plain text files rather than a database as the content repository and builds HTML files by applying themes, layouts and templates to them.

The generated static site files are delivered to the end user exactly as they are on the server. Hence, there is no server-side language or database. The generated files are HTML, CSS and JavaScript, served with graphic files (jpeg, gif, svg, webgl) and data formats (JSON or XML).

Static sites offer several advantages in speed, security and scalability over dynamic sites (Content Management Systems).

## Discussion

### What are static sites and how are they created?

Static sites are basic websites that can be built by simply creating a few HTML pages and publishing them to a Web server.

Over a period of time, the way in which static sites were generated has progressed in the following manner:

    + 1990s: Static sites were created by directly creating HTML files.
    + 2000s: WYSIWYG (What You See is What You Get) Desktop applications were used to create and edit the website.
    + 2010s: File system compilers are used to generate static sites.

### How does a static site generator work?

A static site generator parses templates, partials, liquid code, markdown and synthesizes (combining into a coherent whole) a static site. The static site generator script itself is written in a programming language like Ruby, Go, Python, React, etc.

Most of them generate a website from a set of files:

    + The content is produced in lightweight syntax like Markdown or Asciidoc.
    + A templating engine (Liquid, Go Template, Nunjucks) is responsible for the logic while generating the HTML.
    + A converter (kramdown, commonmark, blackfriday, Asciidoctor) transforms the lightweight syntax into HTML.Most static site generators support one or more file-based data formats like JSON, YAML, and TOML to define the metadata.

### What is the JAMstack?

**JAMstack** is the abbreviation for JavaScript, APIs, and Markup stack. This describes the main attributes of a statically generated site.

Matt Billman, Co-founder & CEO, Netlify came up with the term JAMstack in 2012. The new term was intended to change the archaic connotation of "static sites". This is because the term "static site" is often misunderstood to describe the experiential characteristics of a site rather than the attributes of the site architecture which delivers it.

The JAM stack architecture enables the seamless integration of microservices. Microservices provide an easy access to functionality like form handling, authentication, real-time databases without the need for a dedicated back-end. Peter Levine of Andreesen Horowitz venture capital firm commented that,

> It’s ridiculous that the web is still clinging on to monolithic backends—with their high costs, slower speeds, and huge surface area for attacks—in an age of microservices.

### What are the advantages of using static site generators?

We note the following advantage of SSG:

**Performance**—Faster since there's no server side processing/compute involved and no database connections to be made.

**Resilience**—More resilient to sudden traffic spikes. For high-availability websites, single points of failures are reduced.

**Security**—When compared to dynamic CMS sites, static pages are easier to secure. No server-side language exploits, no database vulnerabilities and fewer attack vectors.

**Hosting**—Server setup and maintenance is simpler and cheaper as memory and CPU requirements are not much compared to dynamic sites. Upgrades and new versions of the server-side language or the database don't affect the generated static pages.

**Portability**—Since the site is not coupled to a complex hosting infrastructure, it can be moved from one place to another and redeployed in a simpler manner.

**Compliance**—The simple architecture contributes to easier compliance, especially when building something for a large company and hosting in their environment.

**Attrition Avoidance**—Static sites don't demand sustained monitoring, patching missing dependencies or infrastructure.

**Version Control**—Since the entire site is file based, a version control system like *git* can keep track of all changes to content and configuration.

### What are the challenges/limitations in using Static Site Generators?

**Complex Workflow**—Most of the mature static site generators were intended for the hobbyist programmer, which has resulted in a complex workflow for clients. Editing and previewing production changes is complicated.

**Competitive Ecosystem**—With over 450 static site generators in the open, a lot of effort is duplicated to solve the same problem of basically rendering markdown.

**Realtime changes**—Real-time changes are not possible in the current system as opposed to CMSs. Large static sites sometimes take minutes to generate accurately. They load, render, and forget on each run. For example, when a change is made to a menu, every page that uses it has to be regenerated and uploaded.

**Developer Experience**—Developers have to familiarize themselves with the file system conventions and directory structure surrounding static site generation. For example, "you have to put your markdown files here and then it turns into this URL structure". Additionally, templating languages that seem to dumb down what you can do, frustrate experienced developers.

### Could you mention some popular Static Site Generator?

Some popular static site generators include Jekyll (Ruby), Hugo (Go), Hexo (Node.js), GitBook (JS), Octopress (Ruby), Gatsby (ReactJS), Pelican (Python), Brunch (JS), Metalsmith (JS) and Middleman (Ruby). Netlify reports the ten most popular and best-supported static-site generators, based on GitHub’s top starred repositories.

A comprehensive list of all possible static site generators is maintained by Static Site Generators. Details include when it was created, the last time it was updated and the language it is built with. A similar list is maintained at StaticGen. This list offers additional details like the templating language supported by the project and a description pulled from the project repository.

### What parameters should I consider while choosing a static site generator?

**Build Time**—Time to generate the final site for a given set of input files. On large sites, this can be tens of minutes. Hugo, Gatsby and Nuxt are faster than Jekyll.

**Templates and Themes**—The availability of templates and themes with a suitable licence and the ease with which new templates or themes can be created.

**Project Activity and Adoption**—GitHub stars are a reasonable indication of good project activity. A more active project that has greater adoption is more likely to be supported over a longer period of time.

**Portability**—Are the file structure, content syntax and templating language fairly standard? With standard technologies such as Markdown, content can be more easily ported to another SSG.

**Plugins**—The plugins you require should work with the SSG. More generally, some SSG provide most features out-of-the-box (Gatsby, Next, Nuxt). Others provide minimal features that are then expanded by a vast array of plugins, often developed by community. Jekyll is in the latter camp with lots of plugins.

**Deployment**—Support for continuous integration and continuous deployment.

### What kind of sites are most suited for Static Site Generators?

Static site generators are broadly useful for building sites that focus heavily on delivering content, have a low degree of user interactivity and update infrequently. It is suited for mission-critical sites where traffic spikes and security are causes of concern.

Public administrations in the US 18F, in UK Alphagov or in France Etalab seem to embrace this workflow as the version-control-workflow makes sense for distributed teams in addition to avoiding latency, downtime and errors.

### What kinds of sites are not suited for Static Site Generators?

Static site generators may not be suited for the following:

    + Sites that are largely customised for each individual user that visits the page, such as e-commerce sites.
    + Sites that need to reflect real-time changes to content or templates.
    + Sites that require an administration interface.
    + Sites which have several content (untrusted) creators. Since static sites are flexible, anything contained within source Markdown files can be rendered as page content. Users may be able to include scripts, widgets or numerous undesired items.
    + Very large sites that could result in increased build times.

### Could you list some popular static site generator deployments?

    + Healthcare.gov went from 32 servers to 1 (and one for fall over).
    + In 2012, Obama's $250M fundraising platform was built using a static site generator. The team that built the Hillary Clinton site built the Obama site.
    + Nest and MailChimp now use static site generators for their primary websites.
    + Vox Media has built a whole publishing system around Middleman.
    + Carrot, a large New York agency and part of the Vice empire, builds websites for some of the world’s largest brands with its own open-source generator, Roots.
    + Several of Google’s sites, such as "A Year In Search" and Web Fundamentals, are static.
    + The Smashing Magazine static site was benchmarked at 10 times faster than their earlier WordPress deployment.
    + The New Dynamic Website lists some popular deployments using static site generators.

### How is content edited and managed on static sites?

Most static site generation occurs via the command line. Moving the generated files to the hosting server can also be automated with command line scripts or tools.

However, browser-based editors have been making their presence felt. While traditional CMS such as Wordpress, Joomla, Magento or Drupal generate pages dynamically, some SSG provide rich UI for editing pages and also host the site. Thus, these are hosted CMS with static content. Static content is updated when edited and saved, and not when a request comes to the server.

Among the **static CMS** are Prose, Surreal CMS and Forestry.io. Prose.io is a free service designed specifically for editing Jekyll projects stored in GitHub. There are also services like Surreal CMS and Forestry.io. Many more static CMS are listed on GitHub.

CloudCannon offers a web-based editor geared towards non-technical users. It uses Jekyll and Git workflows. It builds the site, minifies assets and publishes on its hosting.

### What are the options for hosting static sites?

Broadly, we have two options:

    + **Cloud file storage**: Content is pushed to the cloud for storage. When web requests come in, static files are directly served. Examples include Amazon S3, Google Cloud Storage, Rackspace Cloud, and Microsoft Azure.
    + **Static file hosting providers**: Content can be edited online, which then triggers static generation and update to the hosting. Examples include Netlify, Forge, GitHub, and Surge.sh.

### Can dynamic aspects be added to static sites?

Usually dynamic aspects are added using the <script> tag whose source is a JS file on a different server. The remote data is loaded using Ajax (Asynchronous JavaScript + XML).

**Comments** : Disqus, Staticman, Livewire and Facebook

**Forms** : Wufoo, Typeform, Google forms, Formspree, FormKeep.

**Events** : Webhooks, moment.js

**Search** : Google, Swiftype, AddSearch, Lunr, Algolia, Aerobatic

**Forums** : Discourse

**Shopping Carts** : Ecwid, Snipcart, GO Commerce

**Assets (CDNs)** : imgix, Cloudinary

**Backend as a Service (BaaS)** : Parse and Kinvey offer APIs for developers to pull arbitrary dynamic data into a static page.

## Milestones

Aug
1991

First static website by Tim Berners-Lee becomes publicly accessible.

Oct
1995

Vermeer Technologies launches FrontPage 1.0, a static website creation software. Microsoft acquires Vermeer in January 1996. It is branded as part of the Microsoft Office suite from 1997 to 2003.

Dec
1997

Macromedia creates Dreamweaver, a static website creation software, which is acquired by Adobe in 2005.

May
2000

Dries Buytaert launches Drupal CMS for dynamic websites.

May
2003

A joint effort between Matt Mullenweg and Mike Little to create a fork of b2/cafelog results in Wordpress CMS.

2004

John Gruber creates **Markdown**. Markdown allows content creators to focus on content rather than markup such as HTML. This may be seen as an essential piece towards the birth of static site generators, to convert Markdown to HTML that can rendered on browsers.

Aug
2005

Joomla CMS is born as a result of a fork of Mambo.

2008

Tom Preston-Werner, founder and former CEO of GitHub, creates **Jekyll**, a static site generator written in Ruby.

Aug
2010

Alexis Métaireau releases first working version of **Pelican**, a static site generator written in Python.

Apr
2011

Thomas Reynolds releases first version of **Middleman**, a static site generator written in Ruby.

2013

Johan Nordberg creates **Wintersmith**, a static site generator written in Node.js.

2013

Steve Francia creates **Hugo**, a static site generator built on the Go programming language.

Static Site Generators


# A/B Testing

## Summary

Does your manager decide based only on intuition and experience, not data? Do you like to experiment and analyze test results? Would you like data to validate your assumptions? Do you prefer to make data-driven decisions? If your answers to these questions are "YES", then A/B testing is relevant.

**A/B Testing** is a method to compare two (or more) variations of something and determine which one works better. In this method, users are randomly assigned to one of two variants. A statistical analysis is performed to determine which variation performs better for a defined business goal.

While A/B testing is applied in many areas (life sciences, social sciences, advertising, etc.) this article focuses on application to web/mobile app design and user interfaces.

## Discussion

### Why should I do A/B testing?

In the world of advertising, there's no way of telling if a campaign will be successful. Huge advertising budgets have been wasted because of bad guesses. What if we could instead test on a small sample and use that data to figure out user preferences for the entire population? This is exactly what A/B testing offers.

When applied to digital marketing, the design of web pages or mobile app screens, A/B testing gives data that can help us choose among alternatives. The goal is Conversion Rate Optimization (CRO). Conversion could mean a purchase, an email sign-up, account creation, survey completion or app download. One survey showed that 67% of participants chose A/B testing for CRO. It was also the most preferred method.

With A/B testing, you can make more out of your existing traffic. Since even small changes can lead to better conversion or lead generation, the return on investment (RoI) can be massive sometimes. The results help us to understand user behaviour better and determine what impacts user experience. Testing validates assumptions. It removes guesswork and enables data-informed decisions from "we-think" to "we-know".

### What does it mean to call it "A/B"?

The term "A/B" comes from the fact that two groups are used for the test. The **control group** is shown the base version. The **test group** is shown the variation. These two groups are also called A and B respectively.

An alternative interpretation is that A and B refer to two variations that need to be tested to ascertain which one is better. In this case, the baseline without either A or B, is the control group. We may even call this "A/B/C" testing.

In life sciences where A/B testing is used to measure the effectiveness of a drug, the test group is often called the **treatment group**.

### What's the recommended process for A/B testing?

The correct way to run a A/B testing is to follow a scientific process:

    + **Set a Goal**: Based on the business goal, a conversion metric is decided. Example: Number of visitors who sign up for free trial.
    + **Construct Hypothesis**: Observe user behaviour and brainstorm hypotheses. Prioritize what to test based on expected impact and difficulty of implementation.
    + **Create Variations**: Decide on which feature you want to vary, such as changing the colour or size of the call-to-action button.
    + **Run Experiment**: Randomly assign the users to test and control group and collect relevant data for analysis.
    + **Analyze Results**: Once the experiment is complete, you can analyze the results. Ask which variation performed better and whether there was a statistically significant difference.
    + **Repeat**: Based on the results, if we want to test another variation, repeat the same process.

### Could you share some tips when planning for A/B testing?

Use your experience to know what variations could affect user behaviour for the relevant business goal. The variation could be in text such as the title or heading. It could be in terms of design such as the choice of colour. It could be a different layout. It could be number of fields in a form or when to send an email.

How long should you run the test? The common approach is to look at 95% statistical significance, standard deviation of the conversion rate, and sample size. You need 500 users per variation for statistical significance. For search or category tests, this could be 1000 users. Online calculators are also available to know how long to test: from Convert, from Optimizely.

An article on Smashing Magazine gives a list of Do's and Don'ts. Optimizely has shared what variations are businesses testing and which ones are effective.

Some common threats to the validity of A/B tests are explained in an article on Instapage blog.

### What are some myths associated with A/B testing?

Some things about A/B testing are misunderstood. We clarify the following:

    + A/B testing is not the same as conversion rate optimization. A/B testing is part of optimization. While conversions are important, pay attention to secondary metrics. Analyze all steps of the funnel.
    + To test everything is probably a waste of resources, particularly when your user base in not as large as that of Google or Facebook. Prioritize on things you think can make a real difference.
    + While changing one variable at a time is good advice, sometimes it makes sense to include multiple changes, such as on a badly designed page. Consider also how factors interact with one another.
    + Your tests should be long enough to account for influencing factors, such as weekday vs weekend, or a big sale. Look for 95% statistical significance but don't rely on this alone to stop a test. Wait to reach a calculated sample size or a stable conversion rate.
    + If you obsess over velocity and real-time optimization, you might make mistakes and lose out on valuable insights.
    + A one-off test may give temporary results. Iterate your testing. Retest to rule out false positives.

### What do you mean by 95% statistical significance?

This is best explained with an example. Suppose the results of an A/B test show "Control: 15% (+/- 2.1%); Variation 18% (+/- 2.3%)". You might interpret that the actual conversion rate is between 15.7% and 20.3% for the variation; but this is in fact a common error among those not familiar with statistics.

The correct interpretation is that if we were to perform the tests multiple times, 95% of the ranges will include the true conversion rate. In other words, in 5% of the tests, the true conversion rate will fall outside the range.

To put it differently for the above example, we are 95% confident that the true conversion rate is between 15.7% and 20.3%.

Ultimately, results from A/B testing must be statistically validated, usually with Chi-squared test, to rule out differences due to chance. Among the tests are Fisher's Exact, Fisher's Monte Carlo and Chi-squared tests. The last is the simplest for sample sizes exceeding 10,000. Fisher's Monte Carlo, a computational feasible approximation of Fisher's Exact, is more accurate for smaller samples.

### How is A/B Testing different from Multivariate Testing?

With **A/B Testing**, you're splitting the traffic between two variations of a single variable. What if we wish to test 10 different shades of blue for a button? This is considered as **A/B/N Testing**. A visiting user is shown one of these variations.

When a page is modified in more than one way, we call it **Multivariate Testing**. For example, in the control variation there's no sign-up form or embedded video. In variations, either one of these or both are included. Tools will generate all possible combinations of variations once the variable-level changes are specified. Such variations may be important to test influence of one variable over another. For example, sign-up form might generate more sign-ups if there's also a video on the page. Because there could be many variations to test, this is suitable only for sites with high traffic.

However, use multivariate testing with caution due to the danger of *spurious correlations*, which could be nothing more than random fluctuations.

### What are some tools available for A/B testing?

TrustRadius lists a number of A/B testing tools and mentions that the top ones are Kameleoon, AB Tasty, Evergage, Qubit, Dynamic Yield, and Visual Website Optimizer. Others include Optimizely, Monetate, and Maxymiser. Another list includes Unbounce, Kissmetrics, Optimizely, Maxymiser, and Webtrends Optimize. Another list of 30 tools includes Optimizely, Unbounce, Visual Website Optimizer, Usability Hub, Maxymiser, A/Bingo, Kissmetrics, Google Analytics, AB Tasty and Adobe Target.

Tools may be for small businesses, mid-sized businesses or large enterprises. While many are paid tools or even managed services, Google Optimize is a free tool. Google Optimize 360 is a paid enterprise-level tool. These Google tools deprecate the older Analytics Content Experiments.

Basic features of a tool include splitting traffic to different variations, calculating conversions, and measuring statistical likelihood that one variation will consistently outperform another. Additional features may include easy creation of variations, customization, and multi-page campaigns.

### Are there situations where A/B testing is not suitable?

A/B testing cannot make a good design choice on its own. It can compare two or more design choices and help us decide which is better.

Incremental changes can improve design only up to a certain extent. In order to improve it radically, the designer should think more creatively and understand the user needs better. With A/B testing, we may find ourselves chasing "local maxima", the best outcome but within narrow constraints. There are therefore situations when we need to think outside the box and make radical changes. A/B testing is not the way to do it.

There are also some reasons against A/B testing. With A/B testing you're testing your own assumptions and therefore prone to individual bias. A test could take 3-4 weeks, valuable time that could be spent elsewhere. If your user base is not large, you'll not achieve statistical significance.

### When applied to websites, how does A/B testing affect SEO? Are there any best practices?

A/B testing or multivariate testing does not pose any inherent risk to the website's search rank if used properly. However, it's important to know the best practices:

    + No cloaking: Serve the same content to users and search engines regardless of user-agent.
    + Use `rel="canonical"`: In A/B testing, alternative URLs should refer to the original URL as the canonical so that search engines will not index the alternative URLs.
    + Use 302 Redirects instead of 301s: When using redirection to a variant URL, use 302 redirects since the variant will exist only for the experiment. 301 is a temporary redirect while 302 is permanent.
    + Run experiments only as long as necessary: Once you have enough data, revert to original site and remove variations.

### Could you share some examples where A/B testing has been used?

Here are a few examples:

    + **Humana's Banner Test**: Humana tested two different banners on their homepage. A simpler design plus a stronger call-to-action (CTA) led to 433% more click-throughs. In the control variation, the CTA is not clear and in the test variation the CTA is clear.
    + **Hubspot's Lead Conversion**: Using an inline form for CTA led to 71% better conversion compared to a linking to a form on a separate page.
    + **MSN Home Page Search Box**: The Overall Evaluation Criterion was the click-through rate for the search box and the popular searches links. Taller search box was compared with bigger search button. However, no remarkable differences were found.
    + **Electronic Arts**: For their SimCity 5 game, they tested conversions between direct promotional banner and no banner. Surprisingly, the latter drove 43.4% more purchases.
    + **Coderwall**: A simple title change improved Coderwall's click-through rate (CTR) by 14.8% on Google Search. Over time, this resulted in improved ranking as well.

## Milestones

1753

James Lind publishes "A Treatise of the Scurvy" in which he compares different treatments for scurvy using controlled trials. In well-planned clinical trials from the late 1740s, James Lind provides remedies to the identical diets of six pairs of men with symptoms of scurvy. He finds that a diet that includes oranges and lemons (vitamin C) cures scurvy.

1863

Austin Flint conducts a clinical trial that compares the efficacy of a dummy simulator to an active treatment. The dummy simulator, also called placebo, is really for moral effect. Flint's technique is called "Placebo and Active Treatment". Used from at least the 18th century, placebos avoid observer bias.

1920

Statistician and biologist Ronald Fisher discovers the most important principles behind A/B testing and randomized controlled experiments in general. He runs the first experiment in agriculture.

1923

In his book "Scientific Advertising", Claude Hopkins writes,

> Almost any questions can be answered, cheaply, quickly and finally, by a test campaign. And that’s the way to answer them—not by arguments around a table. Go to the court of last resort—the buyers of your product.

1948

The Medical Research Council conducts the first ever randomized clinical trial to determine the efficacy of streptomycin in the treatment of pulmonary tuberculosis. Unlike prevailing practice, subjects are *randomly* allocated to one of two test groups, thus avoiding any bias. A/B testing is increasingly applied in clinical trials from the early 1950s.

1960

In the 1960s and 1970s, marketers start using A/B testing to evaluate direct response campaigns. An example hypothesis would be, "Would a postcard or a letter to target customers result in more sales?"

1990

With the coming of the World Wide Web in the 1990s, A/B testing goes online and becomes real time. It can also scale to large number of experiments and participants. The underlying math hasn't changed though.

2000

Google engineers conduct A/B tests to find the optimal number of display results per page. The test was a failure due to a performance bias because variants that displayed more results were slower. But Google did learn many lessons about the impact of speed on user experience.

2007

Obama's presidential campaign has a new digital team advisor, Dan Siroker. He introduces A/B testing in the campaign with a goal of converting visitors to donors.

2009

A team at Google can't decide between two shades. So they test 41 different shades of blue to decide which color to use for advertisement links in Gmail. The company shows each shade of blue to 1% of users. A slightly purple shade of blue gets the maximum clicks, resulting in $200 million boost of ad revenue.

2011

Google runs 7,000 A/B test experiments on its search algorithm. Like Google, Netflix, Amazon and eBay are keen adopters of A/B testing.

A/B Testing


# 5G NR Channels

## Summary

On the air interface between a UE and a 5G base station, 5G New Radio carries information on various **physical channels**. These channels carry both User Plane (UP) or Control Plane (CP) information.

However, the 5G NR protocol stack has many layers, each layer communicating with its peer at a different level of abstraction. Higher-layers PDUs are not mapped directly to physical layer for transmission. Instead, PDUs at a layer are mapped to channel types that suits that layer's functionality and abstraction. Thus, RLC delivers its PDUs to MAC over **logical channels**; MAC PDUs to PHY are on **transport channels**.

Channels help us organize and simplify the design and development of the stack. Each channel can also be prioritized and optimized differently. Downlink channels are distinct from uplink channels, though some channel names may be the same.

## Discussion

### What are the channel types defined in 5G?

There are three types of channels:

    + **Logical Channels**: Offered by MAC to RLC. *Control channels* carry CP packets. *Traffic channels* carry UP packets. Each logical channel maps to an *RLC channel* coming from RLC layer.
    + **Transport Channels**: Offered by PHY to MAC. MAC layer multiplexes one or more logical channels to a transport channel. Whereas logical channels describe *what* is carried, transport channels describe *how* they're carried.
    + **Physical Channels**: Channels that carry information on the air interface. Transport channels map to physical channels. There are also a few standalone physical channels that don't carry higher-layer information.

### In 5G NR downlink, what's the channel mapping?

Downlink logical channels include Broadcast Control Channel (BCCH), Paging Control Channel (PCCH), Common Control Channel (CCCH), Dedicated Control Channel (DCCH) and Dedicated Traffic Channel (DTCH). Downlink transport channels include Broadcast Channel (BCH), Paging Channel (PCH) and Downlink Shared Channel (DL-SCH). Downlink physical channels include Physical Broadcast Channel (PBCH), Physical Downlink Control Channel (PDCCH) and Physical Downlink Shared Channel (PDSCH).

In terms of mapping we note the following:

    + BCCH for Master Information Blocks (MIBs) goes on BCH, which goes on PBCH. BCCH for System Information Blocks (SIBs) goes on DL-SCH, which goes on PDSCH. PBCH is used by UE for cell acquisition, selection and re-selection.
    + PCCH goes on PCH, which goes on PDSCH. This is used to page a UE whose cell location is unknown.
    + CCCH is used by a UE during initial access when there's no RRC connection yet. Once a connection is established, DCCH and DTCH become available to the UE.
    + PDSCH carries a variety of transport channels. It can be adapted to current link conditions.
    + PDCCH is used to schedule transmissions on PDSCH and PUSCH.

### In 5G NR uplink, what's the channel mapping?

Uplink logical channels include Common Control Channel (CCCH), Dedicated Control Channel (DCCH) and Dedicated Traffic Channel (DTCH). Uplink transport channels include Random Access Channel (RACH) and Uplink Shared Channel (UL-SCH). Uplink physical channels include Physical Random Access Channel (PRACH), Physical Uplink Control Channel (PUCCH), and Physical Uplink Shared Channel (PUSCH).

In terms of mapping we note the following:

    + PRACH carries RACH, which has no mapped logical channel. PRACH is used by UE for initial access to the network. Random access preambles carried on RACH help overcome message collisions due to multiple UEs transmitting at the same time.
    + PUSCH carries UL-SCH, which carries CCCH, DCCH and DTCH logical channels. Like, PDSCH in downlink, PUSCH has flexible configuration that can be adapted to link conditions.
    + PUCCH carries uplink control information although these could be sent on PUSCH as well.
    + CCCH, DCCH and DTCH are also present in the downlink. DTCH is the only one in the user plane. As an example, we note that RRC signalling messages and NAS messages (that also go through RRC) use DCCH. Application data use DTCH.

### Which are the standalone physical channels?

By standalone physical channels, we mean channels that don't map to any transport channel. Information on these channels are only for exchanging control information at the physical layer. These help in gathering feedback and quickly adjusting configuration or resource allocation to respond to varying conditions on the air interface.

In the downlink, **PDCCH** carries Slot Format Indicator (SFI) and Downlink Control Information (DCI). DCI does scheduling for PDSCH and PUSCH. It carries Transmit Power Control (TPC) commands. DCI comes in many formats and one of them carries SFI to inform the UE which slot format is to be used.

In the uplink, **PUCCH** carries Uplink Control Information (UCI), although UCI can also go on PUSCH. UCI carries channel reports, HARQ-ACK and scheduling request.

### Which are the 5G NR physical signals?

There are a few physical signals closely associated with the operation of physical channels. For downlink synchronization, there are **Primary Synchronization Signal (PSS)** and **Secondary Synchronization Signal (SSS)**. These are transmitted along with PBCH.

For acquisition and channel estimation, **Demodulation Reference Signal (DM-RS)** is used in both downlink and uplink. It's applicable for PBCH, PDCCH, PDSCH, PUCCH and PUSCH. In the uplink, **Sounding Reference Signal (SRS)** is used for channel estimation when PUCCH and PUSCH are not scheduled.

For downlink positioning, **Positioning Reference Signal (PRS)** is used. Uplink positioning is based on SRS.

For phase tracking and phase noise compensation for PDSCH and PUSCH, **Phase Tracking Reference Signal (PT-RS)** is used. This combats path delay spread and Doppler spread via fine time and frequency tracking.

For beam management to a connected UE, **Channel State Information Reference Signal (CSI-RS)** is used in the downlink.

### What are the sidelink channels?

We commonly talk about downlink (base station to UE) and uplink (UE to base station) channels. However, there are also sidelink channels that enable direct device-to-device communications. This is particular to Vehicle-to-Everything (V2X) communications.

Sidelink physical channels include Physical Sidelink Broadcast Channel (PSBCH), Physical Sidelink Control Channel (PSCCH), Physical Sidelink Shared Channel (PSSCH) and Physical Sidelink Feedback Channel (PSFCH). PSCCH and PSFCH are standalone channels. PSCCH carries a part of Sidelink Channel Information (SCI), the other part going on PSSCH. Sidelink Feedback Control Information (SFCI) goes on PSFCH. PSFCH carries HARQ feedback for PSSCH reception.

Transport channels include Sidelink Broadcast Channel (SL-BCH) and Sidelink Shared Channel (SL-SCH). Logical channels include Sidelink Broadcast Control Channel (SBCCH), Sidelink Control Channel (SCCH) and Sidelink Traffic Channel (STCH). Mapping of these are PBCCH/SL-BCH/PSBCH, SCCH/SL-SCH/PSSCH and STCH/SL-SCH/PSSCH.

Sidelink physical signals include DM-RS, CSI-RS, PT-RS, Sidelink Primary Synchronization Signal (S-PSS) and Sidelink Secondary Synchronization Signal (S-SSS). PSCCH is associated with DM-RS. PSSCH is associated with DM-RS and PT-RS.

### What are the differences between 4G/LTE and 5G NR channels?

PBCH-DMRS, PDCCH-DMRS, PUCCH-DMRS, PDSCH-PTRS and PUSCH-PTRS are new in 5G NR. CRS, EPDCCH and EPDCCH-DMRS in LTE don't exist in 5G. Some LTE transmission modes don't need PDSCH-DMRS since Cell-Specific Reference Signal (CRS) fulfils the role.

While DCI on PDCCH is present in 5G, LTE's downlink controls CFI on PCFICH and HI on PHICH are absent in 5G. These channels carry HARQ ACK/NACK for PUSCH transmissions since LTE uplink HARQ is synchronous. These are not needed in 5G since HARQ is asynchronous in both uplink and downlink, that is, HARQ process number is carried in DCI.

LTE's downlink logical channels Multicast Control Channel (MCCH) and Multicast Traffic Channel (MTCH) are absent in 5G. Their associated transport and physical channels MCH and PMCH are also absent. However, multicast is being standardized in Release 17 and some of these channels may be introduced into 5G.

## Milestones

Dec
2017

3GPP publishes Release 15 "early drop".

Jun
2018

3GPP publishes Release 15 "main drop". This includes LTE-based Cellular Vehicle-to-Everything (C-V2X) feature, first introduced in Release 14 (Q1 2017).

Dec
2018

3GPP publishes Release 15 "late drop". This includes 5G NR-based C-V2X using **sidelink channels**.

Jul
2020

3GPP publishes Release 16 specifications. This release introduces dedicated **5G positioning** signals, measurements and procedures. This includes PRS for downlink and SRS for uplink.

2022

3GPP Release 17 specifications is expected to come out in 2022. This release might introduce new channels for **multicast**. It will enhance sidelink channels include sidelink relay capability. Positioning will also be better.

5G NR Channels


# HTTP Cookie

## Summary

HTTP cookies are small bits of information exchanged between client and server over the HTTP protocol. The server sets these cookies. The client receives and saves them locally. Subsequently, whenever the client makes a request to the server, it includes these cookies in the request. The server uses these cookies to provide the desired services.

Cookies are exchanged as simple strings in HTTP headers. Each cookie is a name-value pair. Cookies are scoped to the domain and web server path. Clients (formally called user agents) are typically web browsers. Often cookies are persistent even when browsers are closed.

Cookies bring efficiency and convenience to the web browsing experience. However, cookies have been misused to track users (privacy) or hack systems (security).  These threats are being addressed by regulations and alternative technical solutions.

## Discussion

### What's the problem that an HTTP cookie solves?

HTTP protocol enables client-server communication on the web. HTTP itself is a **stateless protocol**. A server doesn't use information from previous requests to respond to the current request.

However, many scenarios require the server to maintain some state information about the client. For example, a user logs into Gmail. Subsequently, she may read emails, compose a new mail, add attachments or change her mail settings. Each of these is one or more independent HTTP requests. Without state, the server would have to authenticate the user with each request. Instead, authentication is done once, cookies are set, and cookies are exchanged in subsequent requests to identify the authenticated user.  In other words, cookies enable **stateful communication** on top of stateless HTTP.

There are plenty of other examples where stateful communication is beneficial: adding items to a shopping cart on an e-commerce site, transferring money from one bank account to another, participating in an online Q&A forum, setting the language preference on a multilingual site, or searching where search engines use past queries to determine context.

### What are the main uses of HTTP cookies?

The way Google uses cookies gives us an insight into the different uses of cookies:

    + **Preferences**: Remember a user's language preference or region. For example, this can be used to serve local weather or traffic reports. Users can also personalize font, text size or colours.
    + **Security**: Used to authenticate users and protect user data from unauthorized access.
    + **Processes**: Used to make the site work properly for the user. This includes navigation or protected access of certain resources. For example, many documents can be opened in Google Docs in the same browser.
    + **Advertising**: Serve personalized ads to user. For example, recent Google searches may be used to personalize ads.
    + **Session State**: Used to learn how user uses the site, which pages are frequently visited, or measure effectiveness of affiliate advertising.
    + **Analytics**: Collect site statistics to learn how users interact with the site, without identifying users individually.Broadly, we can group cookies into three categories: session management (login, shopping carts, game scores), personalization (preferences, themes, settings), and tracking (recording and analyzing user behaviour).

### What are the different types of cookies?

In terms of validity, we can identify session cookies and permanent cookies. **Session cookies** are deleted once the browser is closed. **Permanent cookies** persist even when the browser is closed but can be set to expire after a specified datetime. Most permanent cookies persist for one or two years. Browser will delete them if user doesn't visit those sites before the cookies expire.

In terms of domain, **first-party cookies** are cookies exchanged with the site currently being visited. **Third-party cookies** are cookies exchanged with another site that's different from the current site. This happens because the current site uses some content from the third-party site.

While not exactly HTTP cookies, there are also **Flash cookies**. These may be exchanged by sites that use Flash plugins. They're often called "supercookies" since users can't easily delete them. Related to this are **Zombie cookies** that recreate automatically even if deleted. To make the web more secure, browsers will stop supporting Flash from end of 2020.

### Could you explain first-party and third-party cookies?

Suppose a user visits the site *www.website.com*. This server will respond with the content. The content may contain links to or load resources from another server. For example, *ad.doubleclick.net* might provide ad services for the content publisher at *www.website.com*. Thus, the user sees the site's content plus ads from the ad server.

Cookies set by *www.website.com* are called first-party cookies. Such cookies are often necessary for the correct working of the site or application. Users typically don't block these cookies.

Cookies set by the third-party server are called third-party cookies. Third-party cookies can be misused since they can be used to track user behaviour across sites. For example, if the user visits another site that also uses the same third-party ad services, the ad service now knows that the user visited both the sites. Due to privacy concerns, some users block third-party cookies.

Third-party cookies come from ad services, social media plugins, live chat popups, etc. In future, such cookies may become obsolete as newer technologies replace them.

### What are some essentials on HTTP cookies?

Server sets an HTTP cookie using `Set-Cookie` response header. A single response can set multiple cookies but a single `Set-Cookie` header can set only one cookie. A cookie is basically of the form `name=value` followed by zero or more attribute name-value pairs. Pairs are separated by a semi-colon and whitespace.

When a client sends a request, applicable cookies are sent in `Cookie` header. All cookies are sent in a single `Cookie` header. Cookie name-value pairs are separated by a semi-colon and whitespace. Attribute name-value pairs are not part of the request. The order in which the cookies appear in the header should not be significant to the server.

Attributes may contain only the name and no value. Two attributes of this nature are `Secure` and `HttpOnly`.

At the client-side, clients shall ignore unrecognized attributes but not the entire cookie. When a client receives the same cookie as an already stored cookie (with same domain and path attributes), the new cookie shall replace the old one. At the server-side, semantics are application-specific.

### Could you describe the different attributes of a cookie?

Attribute names are recognized in a case-insensitive manner. These attributes can be optionally used for each HTTP cookie:

    + **Expires**: Cookie expires at the specified datetime. Servers can delete an existing cookie by specifying a past datetime.
    + **Max-Age**: Specifies in seconds when cookie expires relative to current time.
    + **Domain**: Cookie is applicable to the specified domain and its subdomains. For example, with `Domain=example.com`, cookie will be sent with requests to both `example.com` and `beta.example.com`. If absent, cookie will be sent only to the origin server and not to subdomains.
    + **Path**: Specifies the applicable document path. For example, with `Path=/dashboard`, cookie will be sent for both `/dashboard` and `/dashboard/today` but not for `/posts`. For global cookie, use `Path=/`. If omitted, path defaults to current document location.
    + **Secure**: With this attribute, cookie is sent only over HTTPS connections.
    + **HttpOnly**: This makes the cookie inaccessible using `document.cookie` JavaScript API. Only the server can edit such a cookie.
    + **SameSite**: This mitigates the risk of Cross-Site Request Forgery (CSRF) attacks. With a suitable setting, client sends the cookie only to origin server and not to third-party servers.

### Could you explain the SameSite attribute?

SameSite cookie attribute can mitigate the risk of CSRF attacks. A typical CSRF attack is when you visit *evil.com* that then makes a cross-site request from your browser to your blog *myblog.com* to modify or even delete blog posts.

SameSite cookie attribute takes one of three values:

    + **Strict**: This is the most restrictive setting. Third-party cookies are ignored. While this is safe, it can also dampen user experience. For example, assume a user is already logged into a site *myblog.com*. User visits *example.com* where she clicks a link leading to her blog. Since cookies are not sent, the login will not be recognized until the user refreshes the page on her blog in a first-party context.
    + **Lax**: This relaxes the restriction from Strict mode. When the user follows the link, the browser will include the cookies. Cookies are included only with GET requests.
    + **None**: This is the most open setting but Secure attribute is mandatory. Third-party cookies are allowed. This means that if *example.com* were to request an image from *myblog.com*, third-party cookies would be sent, which doesn't happen for Lax mode.

### What are some shortcomings of HTTP cookies?

Cookies can only contain simple strings. They're not suited to handle complex data structures. Each cookie has a size limit of 4KB. Cookies can be edited or deleted by users at the client side. Applications must be designed to handle this. Cookies are not encrypted, which means that they're not suited to handle sensitive information. Any encryption must be provided by the application. Cookies can also "leak" from one browser tab to the next, creating unintended effects on the application.

Since cookies are sent with every request, this has a performance impact, especially on slow mobile network connections.

Privacy concerns of cookies are well known. Third-party cookies are widely used by ad networks and social networking sites to track users and assemble extensive profile of users in terms of behaviour and preferences. Via social engineering, they can also be used to hack into user accounts, servers and bank accounts.

Modern solutions to avoid using cookies include Web Storage API or IndexedDB for storage; JSON Web Tokens for token-based authentication; and Google's Privacy Sandbox or Storage Access API for cross-origin requests. Fingerprinting could be used for tracking users.

### Which are the standards relevant to HTTP cookies?

The main document is IETF's RFC 6265 titled *HTTP State Management Mechanism*. This defines what's a cookie and the Set-Cookie header fields. It describes server-side and client-side requirements. It also addresses implementation, privacy and security aspects.

As on October 2020, **RFC 6265bis** was in draft form. It aims to update RFC 6265. Among its additions are cookie prefixes (`__Secure-` and `__Host-`) and SameSite attribute.

The concept of origin is essential for cookies. **Origin** is defined in RFC 6454 titled *The Web Origin Concept*.

Two relevant W3C Working Group Notes to look at are Tracking Preference Expression (DNT) and Tracking Compliance and Scope.

## Milestones

Oct
1995

Netscape Communications Corp. files a US patent titled *Persistent client state in a hypertext transfer protocol based client-server system*. It describes, "… when a server responds to an http request by returning an HTTP object to a client, the server may also send a piece of state information that the client system will store. In an embodiment of the present invention, the state information is referred to as a 'cookie'."

Feb
1997

Cookies are described in detail in an IETF document **RFC 2109** titled *HTTP State Management Mechanism*. Subsequently, this RFC is superseded by **RFC 6265** (April 2011).

2002

The European Union passes **ePrivacy Directive (EPD)**. This is amended in 2009 and again in 2017 at the direction of General Data Protection Regulation (GDPR). In time, EPD comes to be called the *Cookie Law*.

Aug
2012

Federal Trade Commission orders Google to pay a penalty of $22.5 million for placing **tracking cookies** in Apple's Safari browser. This violated a privacy agreement between FTC and Google. These cookies were placed by Google's DoubleClick advertising network and bypassed Safari's default setting of blocking third-party cookies.

Jun
2016

Details on **SameSite cookies** are published in an IETF draft that aims to update RFC 6265. This enables servers to assert that cookies should not be sent with cross-site requests. This mitigates the risk of cross-site request forgery (CSRF) attacks. This draft later becomes part of RFC 6265bis draft.

Oct
2016

Version 00 of **RFC 6265bis** is published. Version 06 of this Internet-Draft is published in April 2020. When eventually approved, this would supersede RFC 6265.

Jun
2017

**Intelligent Tracking Prevention (ITP)** is released as a new WebKit feature, where WebKit is the engine used in Safari web browser. ITP collects statistics about resource loads and user interactions. Via machine learning, it classifies if a domain is likely to do cross-site tracking. If so classified, it's cookies can't be used in third-party requests after 24 hours and are purged after 30 days. In March 2020, WebKit fully blocks third-party cookies. For cross-site integration, they propose **Storage Access API** as an alternative.

May
2018

First approved in April 2016, **General Data Protection Regulation (GDPR)** comes into effect. Any cookie that can identify individuals is considered personal data. Such cookies require a user's explicit consent. Simply using the site doesn't imply consent. Site should provide user a means to withdraw consent. User consent must be stored. Sites must clarify the purpose of each cookie and the data it collects.

Jan
2019

W3C publishes a working group note titled *Tracking Preference Expression*. Also called **Do Not Track (DNT)**, this gives users a means to express their preference with regard to tracking. When enabled, it's sent as a separate header with content `DNT:0` or `DNT:1`. If not enabled, servers can't assume users are saying yes to tracking. The earliest draft of this note can be traced to November 2011.

Aug
2019

Due to the privacy concerns surrounding third-party cookies, Google proposes a new initiative called **Privacy Sandbox**. This is a designed to protect user privacy while keeping content accessible on the web. They note that simply blocking third-party cookies is not effective since fingerprinting can still be used to identify users across websites; and unlike cookies, fingerprints can't be cleared. Moreover, blocking cookies denies publishers funding via ad revenues. Google Chrome plans to phase out third-party cookies by 2022.

Feb
2020

Google Chrome browser changes the default value for attribute SameSite from None to Lax. Cookies without the SameSite attribute will be treated as having `SameSite=Lax`. This restricts the use of third-party cookies in unsafe contexts. Mozilla brings a similar change to its Firefox browser in June (for some beta users only). In May 2019, this was proposed in an IETF draft titled *Incrementally Better Cookies*.

HTTP Cookie


# XGBoost

## Summary

 Ensemble learning is the basis for XGBoost. Ensemble learning is a method for combining the predictive abilities of numerous learners in a systematic way. The result is a single model that aggregates the results of several models.In Ensemble Learning, XGBoost stands for Extreme Gradient Boosting, is a scalable, distributed gradient-boosted decision tree (GBDT) machine learning library. It provides parallel tree boosting and the term **gradient boosting**  refers to single weak model by combining it with a number of other weak models to create a collectively strong model.

It is an extension of boosting in which an objective function is used as input and a gradient descent algorithm is used to generate weak models. Previously, only **Python and R** packages were available for XGBoost, but it has recently been expanded to include Java, Scala, Julia, and more languages[2016].

## Discussion

### How does XGBoost works?

XGBoost operates by dividing data into segments that lead to precise predictions depending on various parameters. The trees in XGBoost take into account the previous prediction value for a given data point and create a new tree that splits the existing data as best as possible to maximise the 'gain' in prediction (if tree-splits are created that don't lead to too much gain, the tree is pruned to avoid overfitting based on a hyper-parameter set threshold). Gradient Boosted Trees generate several models by taking previous trees and factoring in their predictions to create a new tree with the goal of reducing prediction error. The algorithm will combine all of the created trees' predictions to generate its final regression after it has been trained.

The training is then repeated repeatedly, adding new trees with the capacity to forecast residuals as well as prior tree mistakes, which are then combined with the previous trees to generate the final prediction.\(New prediction =Initial prediction + Learning Rate\*Residuals \)

### How does XGBoost determine features?

XGBoost automatically delivers feature relevance evaluations based on a trained predictive model. After constructing a boosting tree, it retrieves feature importance ratings for each attribute. The feature importance gives a score that reflects how useful each feature was in the model's building of the enhanced decision trees. In scikit-learn, feature importance scores can be used to pick features. This is accomplished by utilising the SelectFromModel class, which accepts a model and may turn a dataset into a subset with selected features.This class can accept a **pre-trained model**, such as one that has been trained on the whole training dataset. It can then decide which features to select by applying a threshold. This threshold is utilised when you use the **transform() method** on the SelectFromModel instance to select the same features on the training and test datasets consistently. By using scikit-permutation each feature in the model will be shuffled randomly and the difference in performance will be computed.

 Import plot\_importance from xgboost tells us how important that feature is to the model. Using this method, you can choose features not only for your XGBoost, but also for any other similar model that run on the data.

### How XGBoost handles missing values in a given dataset?

On XGBoost, it can be handled with a **sparsity-aware split finding algorithm** that can accurately handle missing values on XGBoost. The algorithm helps in the process of creating a CART on XGBoost to work out missing values directly.CART is a binary decision tree that repeatedly separates a node into two leaf nodes.The above figure illustrates that data is used to learn the optimal default directions. The key to the method to handle algorithm is to visit only those that are not missing in entries \(I\_k\).

The algorithm learns how to handle missing values by treating the non-presence as a missing value. When the non-presence corresponds to a user specified value, the algorithm can also be applied by enumerating only consistent solutions.All sparsity patterns are handled uniformly by XGBoost. This sparsity is used to make computation complexity proportional to the number of non-missing elements in the input.

### What are the XGB data pre-processing steps?

The following are the data pre-processing procedures for XGB:

    + Load the information
    + Examine the data and delete any unnecessary attributes.
    + Convert textual values to numeric values
    + If necessary, locate and replace the missing values and don't remove predictors with zero or near-zero variance like we used to do for normal models.
    + Divide the dataset into two parts: training and testing.
    + Scale the features or normalise the data.
    + Do Pricipal component analysis process which helps the performance of XGBoost. Prinicipal component analysis is a statistical process that employs an orthogonal transformation to convert a set of correlated variables to a set of uncorrelated variables.

### What does the XGBoost leaf node weight mean, and how do you figure it out?

In general, we build a tree that predicts residual values between target and base predictions. The "leaf weight" in tree building is the projected output of the model associated with each leaf (exit) node.XGBoost tries all split points given by the data values for each feature and records their gain. It then picks the feature and threshold combination with the largest gain.

Here's an example of how to compute the leaf node weights in XGBoost-Consider the following test data point: age=10, gender=female.To forecast the data point, the tree is traversed top to bottom, undergoing a series of tests. A feature is required at each intermediate node to compare against a threshold. According to the comparison's outcome, need to navigate to the left or right child node of the tree. Because "age 15" is true in the case of (10, female), the left branch should be done first, then the age 15 test. In the second test, "gender = male?" It is false and go to Leaf 2 which is constructed with a leaf weight of 0.1.

### What makes XGboost better than gradient boosting?

**Parallel Computing**: When you run XGBoost, by default it would use all the cores of your laptop/machine enabling its capacity to do parallel computation.

**Tree pruning using depth firist approach**: XGBoost uses ‘max\_depth’ parameter instead of criterion first, and starts pruning trees backward. **Tree Pruning** is a data compression technique used in machine learning and search algorithms to minimise the size of decision trees by deleting non-critical and redundant sections of the tree used to classify instances.

**Missing Values**: XGBoost is designed to handle missing values internally. The missing values are treated in such a manner that any trend in missing values (if it exists) is captured by the model.

**Regularization**: The biggest advantage of XGBoost is that it uses regularisation in its objective function which helps to controls the **overfitting** and simplicity of the model, leading to better performance.

### Is XGBoost more efficient than a random forest?

Random Forest and XGBoost are decision tree algorithms that take a distinct approach to training data. When the model fails to forecast the inconsistency for the first time in XGBoost, it gives it more preferences and weightage in subsequent iterations, enhancing its ability to predict the low-participation class; however, we cannot guarantee that **random forest** will treat the class imbalance properly.If the bulk of the trees in the forest are given comparable samples, the random forest is likely to overfit the data.

## Milestones

2016

Tianqi Chen creates XGBoost as a research project as part of the Distributed (Deep) Machine Learning Community (DMLC) group.

2019

XGBoost is also available on OpenCL for FPGAs.OpenCL (Open Computing Language) is a programming language that runs on a variety of platforms, including CPUs, GPUs, DSPs, FPGAs, and other processors and hardware accelerators.

2021

 XGBoost is integrated in Apache Hadoop, Apache Spark, Apache Flink, and Dask, which are distributed processing frameworks.

XGBoost


# Tor Network

## Summary

Using a distributed network of nodes on the Internet, Tor provides users anonymity. Your Internet Service Provider (ISP), governments or corporations can't know which sites you've been visiting. Authorities also cannot censor content or know your location.

Tor is able to do this because it hides your IP address and the addresses of sites you visit. Your packets are bounced across multiple nodes, with each node having only information about the previous and next hops along the route. Moreover, Tor nodes are run by volunteers without any centralized control. Tor is a network service, not a peer-to-peer service like BitTorrent.

The easiest way to use Tor is to use the Tor Browser. There are many other software and services based on Tor.

## Discussion

### Why should I use Tor?

As an individual, you can use Tor to prevent websites from tracking your activities online and serving you annoying ads. More importantly, your ISP or government may censor some websites. Tor helps in bypassing censorship.

In chat rooms where socially sensitive topics are discussed (rape counselling, suicide prevention, terminal illness), Tor helps users preserve their anonymity. This sort of anonymity is even more important for journalists or whistleblowers who are exposing high-level scams. Even as a normal user, you can use Tor to hide your location for your own safety.

In general, Tor protects against Internet surveillance based on **traffic analysis**. Even if your message is encrypted, there's a great deal of information in packet headers (address, size, etc.). Timing of packets can be analysed. Statistical techniques can be employed to reveal patterns. Tor overcomes traffic analysis.

### What are the building blocks of Tor?

When using Tor, traffic is routed through three Tor nodes or relays between source and destination. All Tor nodes are run by volunteers. Hence, there's no centralized control. A connection across Tor network consists of the following:

    + **Tor Client**: Either the Tor Browser or a Tor-enabled program on your computer. This adds three layers of encryption to an outgoing message (and decrypts thrice an incoming message).
    + **Guard Node**: First Tor node to receive your message. It removes one layer of encryption. It knows the next node but not the exit node or destination.
    + **Relay Node**: Intermediate node. It removes another layer of encryption to obtain who's the exit node.
    + **Exit Node**: Removes the final layer of encryption to reveal the destination. It can read the original message but doesn't know who sent it. Traffic exits this node without Tor encryption.
    + **Circuit**: The path defined from source to destination via the three Tor nodes.

### Could you explain Tor's Onion Routing?

Just as an onion has multiple layers, Tor traffic is encrypted repeatedly and then routed through multiple nodes. Let's say, the message goes through nodes A (Guard), B (Relay) and C (Exit). Tor client first encrypts the message with C's key, then B's key, and finally A's key. When A receives the message, it decrypts it with its key. This will reveal to A that the next hop is B. However, A can't know who's C or the destination.

In summary, each node along the route has only partial information. This preserves user anonymity. The focus is to hide IP addresses of source/destination and routers along the route rather than the message content itself.

Unlike traditional IP routing, Tor routing requires the sender to know in advance the route to destination, and keys of routers along the way.

In the original onion routing, public-key cryptography was used. Encryption was done with public keys and decryption with private keys. Today's Tor network uses symmetric keys for encryption and asymmetric public/private keys for authenticating Tor nodes.

### Can my ISP know that I'm using Tor and penalize me for it?

Because Tor has been used for illegal activities, some ISP might throttle or block Tor traffic. Thanks to onion routing, your ISP can't know which site you're visiting or read your traffic. However, ISP will know that you're using Tor.  This is because the IP addresses of Tor's nodes are publicly available. **Tor Bridges** provide a solution to this problem.

A bridge is also a Tor node but it's not listed on the main Tor directory. So your ISP can't easily block all bridges. But ISPs have a solution for this. They inspect packets to figure out if Tor is being used. To partly overcome this, we can use **Pluggable Transports**. These obfuscate the packets so that they don't resemble Tor traffic.

### Isn't Tor doing the same thing as a VPN or proxy?

A Virtual Private Network (VPN) routes all your Internet traffic through a VPN server that hides your IP address. The VPN provider will have many servers worldwide and one of them will be used. Because online services see requests coming from a VPN server, your location is masked. Moreover, all traffic between your device and the server will be encrypted.

Tor too routes traffic via intermediate servers. The main difference between VPN and Tor is one of *trust*. VPN provides privacy but this can be compromised if you don't trust your VPN provider. Your VPN provider may use weak encryption, may store logs about all your activities and may be forced to share them with governments. Being decentralized, there's no need to trust anyone on Tor. No logs are stored.

While VPN is about **privacy**, Tor is more about **anonymity**. We can also state that VPN emphasizes encryption while Tor emphasizes routing. Ultimately, VPN is sufficient for most online activities. Activists and journalists living under strict censorship laws, when their lives are at risk, can use Tor.

### What are the limitations or criticisms of Tor?

While Tor has its benefits, it can also be misused. Hackers use Tor to hide their identities. They can use Tor for ransomware attacks. Tor can be used for low bandwidth, slow Distributed Denial-of-Service (DDoS) attacks. Tor's hidden services feature has been used to publish suspicious or illegal content: drugs, firearms, child pornography.

One study found that a Tor user is 6-8 times more likely to perform a cyberattack. This gives bad repute to all Tor users, whose activities could be wrongly suspected. ISPs will throttle your Tor traffic or even block completely. Some websites block Tor nodes. For example, Wikipedia doesn't allow content edits from anonymous authors since logging IP addresses of Tor exit nodes makes no sense. This prevents vandals but also blocks genuine authors.

Tor has been criticized for being slow. It can't handle peer-to-peer file sharing. Setting up a Tor relay node is also non-trivial.

### How do I get started with using Tor?

The simplest way is to install Tor Browser for anonymous Internet access. If access to Tor Project website is blocked, you can send an email to `gettor@torproject.org`. Email body should contain one of these words: windows, osx or linux. You'll get a link to download Tor Browser.

If you wish to use the Tor network for purposes other than web browsing (such as FTP or SSH), read about privoxy, torsocks, and torification.

To learn more, the official Tor documentation and Tor FAQ are good places to start. FAQ is also a good place to find solutions to common problems. For community support, head to Tor StackExchange. Follow Privacy Enhancing Technologies Symposium (PETS), an annual event featuring the latest developments. For in-depth understanding, read Tor specification documents.

If you run a site, and it's getting hacked over Tor, you can block exit nodes that are accessing your server. An online service such as TorDNSEL can also help.

### What are the different projects based on or related to Tor?

For safe web browsing, use the *Tor Browser*. For Tor statistics and resource utilization, use *Nyx* and *Metrics Portal*. *Orbot* is a project that aims to bring web browsing to Android platform. *Orlib* is a library that can be used with any Android app to route traffic through Orbot/Tor. *Tails* is a live CD/USB distribution of Tor and leaves no trace on the local system.

There are also many community projects. *OnionShare* is for anonymous file sharing. *SecureDrop* is for whistleblowers to submit documents to media organizations. *GlobalLeaks* also enables whistleblowers. *Ahmia* is a search engine for content published on Tor Onion Services.

Three general methods to "torify" apps are proxification, socksification, transsockification. Torifier can be used to tunnel any software application over Tor.

For developers, *Shadow*, *Stem* and *txtorcon* might be of interest. *Orchid* can be used to torify Java and JVM applications.

## Milestones

1995

With widespread adoption of Internet in the 1990s, the US government realizes that Internet can be used by its spies for sharing intelligence. But the problem of identity and security of sources has to be solved. Hence, US Office of Naval Research starts funding research on **Onion Routing**. This research is initially carried out Paul Syverson, Michael G. Reed and David Goldschlag at the US Naval Research Laboratory.

1996

First prototype system that does onion routing is built. It runs on a single Solaris 2.5.1/2.6 machine emulating five nodes. In July, the code is approved for public distribution.

2002

Work starts on second generation onion routing. This is the beginning of **Tor**. The name is not an acronym, although it can be expanded as "The Onion Routing". Alpha release of Tor happens is September.

Oct
2003

Tor network is deployed. Tor code is open sourced under MIT license.

2004

Location hidden services are deployed. This is the beginning of **Tor Onion Services**. The Tor Project is funded by *Electronic Frontier Foundation*, whose mandate is to defend digital privacy, free speech, and innovation. In August, the Tor design paper is published at USENIX Security Symposium. By the end of the year, there are 100 Tor nodes.

2007

First major criminal activities on Tor are reported. Meanwhile, US government continues to fund Tor. About 75% of total donations for the year come from the US government. In 2013, this figure is about 93%.

2013

A leaked presentation of the National Security Agency (NSA) reveals that they tried to hack Tor and failed. Meanwhile, Edward Snowden leaks NSA documents via Tor. FBI closes Silk Road, a Dark Web marketplace for drugs, weapons and stolen data.

2015

In 2015, Tor network has over 7000 nodes worldwide. Daily, more than 2 million users connect to Tor.

2018

One study reports that Tor serves 8 million users per day. This involves 1.2 billion anonymous circuits per day carrying approximately 517 TiB of data (6.1GiB/s).

Tor Network