# Units of Information

## Summary

We wish to quantify information or data for the purpose of storage or transfer. This helps us plan the capacity of storage systems or the bandwidth of communication systems. Service providers also use these numbers to determine pricing.

The basic unit of information is called **bit**. It's a short form for *binary digit*. It takes only two values, 0 or 1. All other units of information are derived from the bit. For example, 8 bits is called a byte, which is commonly used. Since the early 2010s, we're seeing a massive growth in the amount of data being generated worldwide. This has led to the use of higher units such as petabytes and exabytes.

In the world of quantum computing, **qubit** is the basic unit of information.

## Discussion

### What's the rationale for trying to quantify information?

It's been said that a picture is worth a thousand words. In digital systems, this is not a satisfactory statement. What if there's a quantitative way to compare information regardless of the form, be it a picture, a movie, printed text or a song?

Outside the world of digital systems, different units exist. For a library, the number of books it has on its shelves is an essential metric. For a printer, the number of pages is more suitable but this too is not a uniform measure since the page size of a book is different from that of a newspaper. For a publisher or an author, the number of words is a better way to convey the amount of data.

In digital systems, a common unit makes it easier to design systems regardless of the type of data being handled. How much storage should you install for a desktop machine? How many images can you store on your SD card? How long will it take to download a 90-minute full HD movie? These questions can be answered more readily.

### What are the different units of information?

Eight bits make a **byte** or **octet**. A thousand bytes make a kilobyte. A thousand kilobytes make a megabyte. The largest unit we have (in 2019) is a **yottabyte**, which is \(10^{24}\) bytes. While higher units (brontobytes, geopbytes, etc.) have been suggested, they're not official.

We also have multiples of bits: kilobits, megabits, and so on. While storage is often quoted in bytes, communication bandwidth is often quoted in bits. For example, we may say that a file is 300KB (kilobytes) in size; or a broadband connection gives a maximum of 4Mbps (megabits per second) download speed. Web browsers, however, give download speeds in bytes per second.

Bytes are commonly used. Each letter of the English alphabet is represented by a byte. Short emails are in the order of few hundred bytes. Typed text of a few pages is a few kilobytes. An image downloaded on the web could be a few hundred kilobytes. A song encoded in MP3 is a few megabytes. A movie is hundreds of megabytes or even a few gigabytes. In production, Avatar movie took up 1 petabyte of storage.

### What units of information are of concern to a programmer?

Byte is the basic unit of memory or storage for programmers, although bit-level manipulation is possible. Since a bit can store two values, a byte can store \(2^8=256\) different values.

From the perspective of programming, four bits make a **nibble**. It's common to write a byte value in hexadecimal form of two nibbles. For example, 126 is 0x7E. These nibbles are also conveniently shown when debugging memory contents. Nibbles were useful in early computer systems that used packed BCD.

Two words make a **doubleword** but what exactly is a word? The length of a word depends on the processor architecture. In a 16-bit system, a word is defined as two bytes. In a 64-bit system, a word is eight bytes. A processor executes instructions, accesses data in memory via addresses, and processes or represents numbers in words or its multiples. Thus, programmers who write low-level code need to be aware of the word size.

In C programming language, **short** must be at least 16 bits; **long** at least 32 bits; and **long long** at least 64 bits.

### How is a kibibyte different from a kilobyte?

The prefixes kilo, mega, giga and others are based on the decimal system in which these are powers of ten. But computer systems operate on the binary system. Hence, memory and storage capacities are often quoted in powers of two. Thus, a kilobyte was 1,024 bytes, not 1,000 bytes. This led to confusion since these prefixes in SI metric system commonly referred to powers of ten.

To set right this confusion, the International Electrotechnical Commission (IEC) approved in 1998, and published in 1999, a new set of units that represented powers of two. Thus, a **kibibyte** had 1,024 bytes, a **mebibyte** had 1,024 kibibytes, and so on.

However, industry has not widely adopted the IEC units. Thus, Windows 10 continues to report 203GB disk capacity as 189GB when it should say 189GiB.

### How is information related to entropy?

The word information is often loosely taken to mean data. We assume that a file of size 1MB carries 1MB of information. However, from the perspective of information theory, data is not equal to information. In information theory, information is defined mathematically as the amount of uncertainty or entropy. A dice throw has more uncertainty than a coin toss, and therefore has more information to convey.

An uncompressed bitmap image has a lot of spatial redundancy in its pixel values. In other words, a pixel value can be used to predict values of neighbouring pixels. Image compression techniques exploit this redundancy. Thus, a compressed image is closer to the mathematical definition of information. But an MP3 song could contain repetitions of the chorus. Also, once we've heard the song and remember it well, it conveys less information the next time we hear it.

Therefore, the phrase "units of information" should be interpreted as "units of data/storage/memory".

## Milestones

1928

Hartley studies the problem of measure information. Given \(s\) symbols and \(n\) selections, he notes that information \(H=n\,log\,s\). He describes this as,

> our practical measure of information the logarithm of the number of possible symbol sequences

1948

Claude Shannon publishes his paper titled *A Mathematical Theory of Communication*. He introduces the word **bit** as a contraction of binary digit. He credits J. W. Tukey for coining this word.

Jul
1956

During the design of IBM 7030 Stretch, IBM's first transistorized supercomputer, Werner Buchholz coins the term **byte** for an ordered collection of bits. It represents the smallest amount of data that a computer can process. At this time, the byte is not defined as eight bits.

1960

During this decade, 7-bit ASCII is standardized. More importantly, IBM introduces its System/360 product line that uses 8-bit Extended Binary Coded Decimal Interchange Code (EBCDIC). This leads to the popular use of 8-bit storage systems. Thus, a byte comes to represent 8 bits.

1975

The prefixes *exa* and *peta* are standardized as part of the International System of Units (SI).

1991

The prefixes *zetta* and *yotta* are standardized as part of the International System of Units (SI).

Jan
1999

The International Electrotechnical Commission (IEC) publishes new units of information based on powers of two: kibibyte, mebibyte, gibibyte, and so on. The letters "bi" replace the last two letters of the SI prefixes. While some systems have adopted these units, many others have not.

Units of Information


# Long Short-Term Memory

## Summary

There are several variations in neural networks, each suitable for a particular kind of data input. Standard and convolutional neural networks work well on static data, such as static image where entire data is analysed all at once.

However, when data is dynamic and sequentially organised such as video frames or stock market values, a variant called Recurrent Neural Network (RNN) is employed. LSTM (Long Short-Term Memory) is a subset of RNNs.

As the name suggests, LSTM networks have ‘memory’ of previous states of the data. This memory is selectively tuned to remember only chosen parts of past data, even for a long time. In applications where predictions depend on previous values of data, LSTM finds great relevance.

Keras and TensorFlow implementations of LSTM are extensively used in sequential prediction applications such as auto-response suggestions in emails, stock value predictions and speech/writing recognition.

## Discussion

### How do LSTM networks differ from recurrent neural networks?

In conventional feed-forward NN, all data values are considered equally important, irrespective of whether a day old or months old. This works fine for reading image data sets to differentiate a cat from a dog. But while dealing with time series data where every value is time stamped, relevance of data continuously reduces with passage of time.

RNN can handle data dependencies, but within short time intervals. This is achieved by feeding the output of the hidden layer h(t−1) through a conceptual delay block back into the input of hidden layer. However, RNN aren’t fully effective with time-sensitive data because of the **vanishing gradient problem**.

LSTM networks are a type of RNN which include a 'memory cell' that maintains information in memory for long time periods. A set of **gates** is used to control when information enters the memory, when it's output, and when it's forgotten. This architecture lets them learn longer-term dependencies. To eliminate the vanishing (or exploding) gradient problem, the LSTM cell has a unique feature called **Forget Gate**. This helps in greatly reducing the multiplicative effect of small gradients.

### What is the typical node structure of LSTM?

All neural networks have a chain of repeating nodes in the hidden layers. Standard RNN nodes might have an input, output and a simple tanh function in the middle. In LSTM, the hidden layer nodes have three interacting functions or ‘gates’. These gates protect and control the ‘memory’ - data stored in the cell state.

    + **Cell-State**: Works like a conveyor belt running through the network. Value undergoes continuous changes node after node based on information added/removed by the gates.
    + **Hidden-State**: The actual output of that node for a given input. But always hidden because it only enters as input at the next time-step.
    + **Input-Gate**: Open only at time step t. Here the node decides which inputs will update the current cell state and which new candidate inputs will be added to it.
    + **Forget-Gate**: Decides what information to throw away from the cell state. It looks at the input, previous hidden state and gives a value between 0-1 for each number in the cell state. 1 – ‘Completely retain’, 0 – ‘Completely forget’.
    + **Output-Gate**: Sends out a filtered version of the cell state as output.

### How does an LSTM network selectively ‘forget’ and ‘remember’ from past data?

In LSTM, the cell state is retained as a continuous rolling value till it exits all the hidden layers and reaches the output. The 3-gate structure ensures that the cell value is controlled and protected by optionally letting information through. They are composed of a **sigmoid neural net layer** and a **tanh operation vector** for a point-wise multiplication operation.

Data is processed in batches. For a new batch, the input layer gets inputs from the hidden state (output layer) of the previous batch. Even textual values are coded and stored as numbers in the cell state for easy manipulation.

Every iteration follows these steps (at time step t):

    + Hidden state value from t-1 and input at t are sent into the node.
    + Forget gate removes unwanted information from the cell state using the inputs.
    + Input gate decides what old information to retain and what new to add to the cell state.
    + Cell state value also get stored as hidden state at t.
    + Output gate might send filtered version of cell state either out of the hidden layers or back into a recurrent loop.

### Which part of the ‘memory’ is short term and which is long?

Plain vanilla RNNs are also short-term memory networks. The output of a hidden layer node can be sent back in as input after a time lag. So every node can retain the cell value in its ‘memory’ for one time step (short term). Theoretically since the loop is recurrent, the value should remain even for long time periods. However, the error correction that happens through back propagation keeps losing significance as it reaches the initial layers. This causes the hidden layer node values to deviate from the desired output after repeated cycles and the memory is lost.

In LSTM, the three gates keep the cell state safe. So the short-term memory node manages to retain its cell state throughout the network hidden layers (long time). The 'forget gate' is the critical element which stands between different time steps within the hidden layer and ensures the control and accuracy of the cell state throughout.

### Could you explain LSTM with an example?

Take a language model trying to predict the next word based on all the previous ones - “John has lived in France for several years. So *he* speaks very good French”. An important step in prediction is deciding what pronoun to use based on the subject's gender. To predict this, the network needs to recognise the subject ‘John’.

At each time step, the cell state will keep adding/updating relevant information about the subject collected from various different sentences – name, location, gender, singular or plural, etc. The three gates will ensure only relevant information remains.

However, the next sentence may drop ‘John’ as subject and start talking about “Jenny has lived in Spain since her birth.” Now the forget gate ensures that the cell state drops the context of John and starts collecting information on Jenny. The correct pronoun to suggest for the next sentence is ‘she’.

It’s possible that the next sentences don't disturb the subject John and instead talk of other things, “France is known for its …”. Now the LSTM network will remember the context of John for a long time and suggest to use ‘he’ even 2-3 sentences later.

### What are the data preparation steps before feeding to an LSTM network?

Before fitting an LSTM model to the dataset and making a forecast, some data transformations are performed on the dataset. Let’s extend the word sequence prediction example.

    + Uni-variate input data (word sequence in example) is converted into Input -> Output format from which the model can learn. “John has lived” -> “in”, “has lived in” -> “France” and so on. We transform the time series into a supervised learning problem with stationary data. Observation at previous time-step becomes input to forecast at current time-step.
    + LSTM models understand only numeric inputs. So encode the word sequence and punctuation marks with numeric symbols. Build a dictionary of words <-> symbols.
    + Output is a one-hot vector identifying the index of the predicted symbol in the dictionary. Word2Vec is an optimal way of encoding symbols to vectors. To classify images, consider every image row as a sequence of pixels.
    + Transform observations to have a specific scale. Such as, rescaling data to values between -1 and 1 to conform to the default hyperbolic tangent activation function of the LSTM model.These transforms are inverted on forecasts to return them to their original state.

### How to train an LSTM model?

LSTM models are trained to be used for sequence prediction resulting in output as a classification (assigning categorical labels) or a regression (continuous real values).

Taking the word prediction example forward, the model must predict the next word in the sentence. This involves multiple iterations of training the model with past data, then evaluating its accuracy on test data. Finally, the prediction is applied on new production data.

The goal is to arrive at a final model that performs the best with respect to:

    + Historical data available
    + Training time spent on the model
    + Data preparation steps and chosen algorithm configurations
    + Desired prediction accuracy depending on the business caseFirst initialize the input vector, weights and biases. Define the nodes in the network (512 node LSTM network for our example). Feed inputs into the model.

Output is a multi-element vector of prediction probabilities for the next symbol normalized by the softmax function. The index of the element with highest probability is the predicted index of the symbol in the reverse dictionary.

Accuracy and loss are accumulated to monitor the progress of the training. 50,000 iterations are generally enough to achieve an acceptable accuracy.

### What the major applications of the LSTM network?

    + Sequential prediction of stock price based on historical stock price values
    + Handwriting Synthesis and recognition
    + Speech recognition using acoustic signals (phonemes) as input
    + Image Captioning and recognition of unnamed images in archives
    + Synthesised music generation from previous sequence of notes played
    + Language translation, after the entire source language input is given
    + Flood forecasting with daily discharge and rainfall as input data

### What support do TensorFlow and Keras frameworks offer to LSTM modeling?

**Keras** is an open-source Python based deep learning framework. It has a user-friendly, modular API framework covering all common neural network building blocks such as layers, nodes, optimization functions and activation functions. Keras is a high-level API wrapper to run on top of TensorFlow, CNTK, or Theano.

Keras models can be deployed on smartphones, online applications and JVMs. It also supports distributed deployment of deep learning models on GPU/TPU clusters.

Keras high-level API handles the way we make models, defining layers, or setup multiple input-output models. It supports other common utilities like dropout, batch normalization, and pooling.

**TensorFlow** is an end-to-end open source machine learning framework from Google. You install the TensorFlow module in Python and use the libraries for input-output data definition, allocating test and training data sets, model building, fitting the model, optimizing for batch and epoch sizes, training the mode and finally validating accuracy scores for prediction output.

Combined use of Keras over TensorFlow is a popular option among developers, enabling fast experimentation followed by deployment. Both frameworks have good developer community support, with source code and examples available on GitHub.

## Milestones

1991

Sepp Hochreiter develops the long short-term memory (LSTM). The first results are reported in his diploma thesis.

1997

Schmidhuber et al. publish a type of recurrent neural network called the long short-term memory (LSTM). This overcame the vanishing gradient problem.

2000

Standard LSTM architecture is introduced for the first time. This architecture later becomes popular in applications.

2005

"Vanilla LSTM" using backpropagation through time is published.

2015

LSTM trained by CTC is used in a new implementation of speech recognition in Google's software for smartphones and for Google Translate. Apple uses LSTM for the "Quicktype" function on the iPhone and for Siri. Amazon uses LSTM for Amazon Alexa.

2015

Initial version of Keras deep learning framework is released on GitHub. In 2017, Google releases stable version 1.0.0 of TensorFlow, its machine learning framework. In 2019, TensorFlow gets tighter Keras integration. Keras and TensorFlow together enable LSTM modeling.

Long Short-Term Memory


# Text Corpus for NLP

## Summary

In the domain of natural language processing (NLP), statistical NLP in particular, there's a need to train the model or algorithm with lots of data. For this purpose, researchers have assembled many text corpora. A common corpus is also useful for benchmarking models.

Typically, each text corpus is a collection of text sources. There are dozens of such corpora for a variety of NLP tasks. This article ignores speech corpora and considers only those in text form.

While English has many corpora, other natural languages too have their own corpora, though not as extensive as those for English. Using modern techniques, it's possible to apply NLP on low-resource languages, that is, languages with limited text corpora.

## Discussion

### What are the traits of a good text corpus or wordlist?

It's said that a prototypical corpus must be machine-readable in Unicode. It must be a representative sample of the language in current use, balanced, and collected in natural settings.

A good corpus or wordlist must have the following traits:

    + **Depth**: A wordlist, for instance, should include the top 60K words and not just the top 3K words.
    + **Recent**: Corpus based on outdated texts is not going to suit today's tasks.
    + **Metadata**: Metadata should indicate the sources, assumptions, limitations and what's included in the corpus.
    + **Genre**: Unless corpus has been collected for specific tasks, it should include different genres such as newspapers, magazines, blogs, academic journals, etc.
    + **Size**: A corpus of half a million words or more ensures that low frequency words are also adequately represented.
    + **Clean**: A wordlist giving word forms of the same word can be messy to process. A better corpus would include only the lemma and part of speech.

### What are the different types of text corpora for NLP?

A plain text corpus is suitable for **unsupervised** training. Machine learning models learn from the data in an unsupervised manner. However, a corpus that has the raw text plus annotations can be used for **supervised** training. It takes considerable effort to create an annotated corpus but it may produce better results.

A corpus can be assembled from a variety of sources and genres. Such a corpus can be used for **general** NLP tasks. On the other hand, a corpus might be from a single source, domain or genre. Such a corpus can be used only for a **specific** purpose.

### What are the types of annotations that we can have on a text corpus?

Part-of-speech is one of the most common annotations because of its use in many downstream NLP tasks. Annotating with lemmas (base forms), syntactic parse trees (phrase-structure or dependency tree representations) and semantic information (word sense disambiguation) are also common. For discourse or text summarization tasks, annotations aid coreference resolutions.

For instance, British Component of the International Corpus of English (ICE-GB) of 1 million words is POS tagged and syntactically parsed. Another parsed corpus in Penn Treebank. While WordNet and FrameNet are not corpora, they contain useful semantic information.

Audio/video recordings are transcribed and annotated as well. Annotations are phonetic (sounds), prosodic (variations), or interactional. Video transcripts may annotate for sign language and gesture.

Annotations could be **inline/embedded** with the text. When they appear on separate lines, it's called **multi-tiered** annotation. If they're in separate files, and linked to the text via hypertext, it's called **standalone** annotation.

### What are some NLP task-specific training corpora?

Here are some task-specific corpora:

    + **POS Tagging**: Penn Treebank's WSJ section is tagged with a 45-tag tagset. Use Ritter dataset for social media content.
    + **Named Entity Recognition**: CoNLL 2003 NER task is newswire content from Reuters RCV1 corpus. It considers four entity types. WNUT 2017 Emerging Entities task and OntoNotes 5.0 are other datasets.
    + **Constituency Parsing**: Penn Treebank's WSJ section has dataset for this purpose.
    + **Semantic role labelling**: OntoNotes v5.0 is useful due to syntactic and semantic annotations.
    + **Sentiment Analysis**: IMDb has released 50K movie reviews. Others are Amazon Customer Reviews of 130 million reviews, 6.7 million business reviews from Yelp, and Sentiment140 of 160K tweets.
    + **Text Classification/Clustering**: Reuters-21578 is a collection of news documents from 1987 indexed by categories. 20 Newsgroups is another dataset of about 20K documents from 20 newsgroups.
    + **Question Answering**: Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset with 100K questions plus 50K unanswerable questions. Jeopardy dataset of about 200K Q&A is another example.

### Could you list some NLP text corpora by genre?

**Formal** genre is typically from books and academic journals. Examples are Project Gutenberg EBooks, Google Books Ngrams, and arXiv Bulk Data Access. There are many text corpora from newswire. Examples are 20 Newsgroups and Reuters-21578.

For **informal** genre, we can include web data and emails. Corpora for these include Common Crawl, Blogger Corpus, Wikipedia Links Data, Enron Emails, and UCI's Spambase. Corpora derived from reviews include Yelp Reviews, Amazon Customer Reviews, and IMDb Movie Reviews. Even more informal are SMS and tweets, for which we have Sentiment140, Twitter US Airline Sentiment, and SMS Spam Collection.

**Spoken language** is often different from written language. 2000 HUB5 English is a dataset that's a transcription of 40 telephone conversations. **Signed language** can also be annotated and transcribed to create a corpus.

Since languages evolve, when analyzing **old text**, our models need to be trained likewise. Examples include DOE Corpus (600s-1150s), and COHA (1810s-2000s).

Another special case is of **learners** who are likely to express ideas differently. The Open Cambridge Learner Corpus contains 10K student responses of 2.9 million words.

It's also common to have **domain-specific** corpora. For example, BioCreative and GENIA are for biology.

### What are some generic training corpora for NLP?

Some of the well-known corpora are Brown Corpus, British National Corpus (BNC), Lancaster-Oslo/Beren Corpus (LOB), International Corpus of English (ICE), Corpus of Contemporary American English (COCA), Google Books Ngram Corpus, Penn Treebank-3, English Gigaword Fifth Edition, and OntoNotes Release 5.0.

Wikipedia was not made for training NLP models but it can be used. We would need to strip markup. Gensim Python package has `gensim.corpora.wikicorpus.WikiCorpus` class to process Wikipedia data.

Generic corpora are usually suited for **language modelling**, which is useful for other downstream tasks such as machine translation and speech recognition. Researchers have suggested using Project Gutenberg EBooks; Penn Treebank of about a million words pre-processed by Mikolov et al. in 2011; WikiText-2 of more than 2 million words; and WikiText-103. Google's one-billion word corpus provides a useful benchmark.

### Derived from text corpus, which datasets are useful for NLP tasks?

**Wordlists** such as list of names or stopwords are useful for NLP work. Phrases in English (PIE) is another resource to explore distribution of words and phrases. It's based on the BNC corpus.

**Tagsets** are essential for POS tagging, chunking, dependency parsing or constituency parsing. DKPro Core Tagset Reference is an excellent resource. University of Lancaster maintains a multilingual semantic tagset.

**Treebanks** go beyond just POS-tagging a corpus. A treebank is an annotated corpus in which grammatical structure is typically represented as a tree structure. Examples are Penn Treebank and CHRISTINE Corpus. Treebanks are useful for evaluating syntactic parsers or as resources for ML models to optimize linguistic analyzers.

**Word embeddings** are real-valued vectors representations of words. These have improved many NLP task including language modelling and semantic analysis. While it's possible to learn embeddings from a large corpus, it's easier to start with downloadable embeddings. Two sources for downloads are Polyglot and Nordic Language Processing Laboratory (NLPL).

Perhaps by 2020, we'll be able to download pretrained **language models** and apply it to a variety of NLP tasks.

### Which are some corpora for non-English languages?

For machine translation, it's common to have **parallel corpus**, that is, aligned text in multiple languages. We mention a few examples:

    + Aligned Hansards of the 36th Parliament of Canada containing 1.3 million pairs of aligned text segments in English and French
    + Europarl parallel corpus from 1996-2011 of 21 European languages from parliament proceedings
    + WMT 2014 EN-DE and WMT 2014 EN-FR
    + A corpus using Wikipedia across 20 languages, 36 bitexts, about 610 million tokens and 26 million sentence fragmentsAn excellent source is OPUS, the open parallel corpus. Lionbridge published a list of parallel corpora in 2019. Martin Weisser maintains a list that links to many non-English corpora.

### Are there curated lists of datasets for NLP work?

A simple web search will yield plenty of relevant results. Some include download links to the sources. We mention a few that stand out:

    + Linguistic Data Consortium has a list of corpora grouped by project
    + English Corpora hosts nine large corpora
    + The Stanford NLP Group has shared a list of corpora and treebanks
    + Registry of Open Data on AWS stores some NLP-specific datasets
    + NLP datasets at fast.ai is actually stored on Amazon S3
    + Shared by users, data.world lists 30+ NLP datasets
    + Shared by users, Kaggle list wordlists, embeddings and text corpora
    + Nicolas Iderhoff's list of NLP datasets includes collection dates and dataset sizes
    + Sebastian Ruder tracks NLP progress, organized by tasks, with links to external datasets
    + Martin Weisser maintains a list of historical and diachronic corpora
    + In NLTK Python code, call `nltk.download()` but we can download them separately as well
    + From blogs, three separate lists are from Cambridge Spark, Lionbridge and Open Data Science

### Where can I download text corpora for training NLP models?

These are the download links for some notable text corpora:

    + Brown Corpus
    + Corpus of Contemporary American English (COCA)
    + Penn Treebank-3 (paid)
    + Data dumps of English Wikipedia
    + Wikipedia Links Data
    + Project Gutenberg EBooks
    + Google Books Ngrams via Google or via Amazon S3 bucket
    + arXiv Bulk Data Access
    + Common Crawl
    + DBpedia 3.5.1 Knowledge Base
    + Amazon Customer Reviews
    + IMDb Reviews
    + Google Blogger Corpus
    + Jeopardy Question-Answer Dataset
    + Yelp Open Dataset
    + Enron Email Dataset
    + 20 Newsgroups
    + Sentiment140
    + SMS Spam Collection
    + WordNet

## Milestones

1964

W. Nelson Francis and Henry Kučera at the Department of Linguistics, Brown University, publish a computer-readable general corpus to aid linguistic research on modern English. The corpus has 1 million words (500 samples of about 2000 words each). Revised editions appear later in 1971 and 1979. Called **Brown Corpus**, it inspires many other text corpora. The corpus with annotations is included in Treebank-3 (1999).

1992

**Linguistic Data Consortium (LDC)** is formed to serve as a repository for NLP resources, including corpora. It's hosted at the University of Pennsylvania.

1994

A 100-million corpus of British English called **BNC (British National Corpus)** is assembled between 1991 and 1994. It's balanced across genres. A follow-up task called **BNC2014** is started in 2014, which can help in understanding how language evolves. Spoken BNC2014 is released in September 2017. Written BNC2014 is expected to come out in 2019.

1999

**Penn Treebank-3** is released. It's based upon the original Treebank (1992) and its revised Treebank II (1995). This work started in 1989 at the University of Pennsylvania. Treebank-3 includes tagged/parsed Brown Corpus, 1 million words of 1989 WSJ material annotated in Treebank II style, tagged sample of ATIS-3, and tagged/parsed Switchboard Corpus. Apart from POS tags, the corpus includes chunk tags, relation tags and anchor tags. The **BLLIP 1987-89 WSJ Corpus Release 1** has 30 million words and supplements the WSJ section of Treebank-3.

2008

Collected for the years 1990-2007, the **Corpus of Contemporary American English (COCA)** is released with 365 million words. By December 2017, it has 560 million words, adding 20 million each year. There's good balance of spoken, fiction, popular magazines, newspapers, and academic texts. It's been noted that COCA contains many common words that are missing in the American National Corpus (ANC), a corpus of 22 million words.

Jun
2011

**English Gigaword Fifth Edition** is released by LDC. It comes from seven English newswire services. It has 4 billion words and takes up 26 gigabytes uncompressed. The first edition appeared in 2003. In November 2012, researchers at the John Hopkins University add syntactic and discourse structure annotations to this corpus after parsing more than 183 million sentences.

Jul
2012

From digitized books, Google releases version 2 of **Google Books Ngrams**. Version 1 came out in July 2009. Only n-grams that appear more than 40 times are included. The corpus includes 1-gram to 5-grams. It includes many non-English languages as well. To experiment on small sets of phrases, researchers can try out the online Google Books Ngram Viewer.

Aug
2012

As a corpus for informal genre, **English Web Treebank (EWT)** is released by LDC. This includes content from weblogs, reviews, question-answers, newsgroups, and email. It has about 250K word-level tokens and 16K sentence-level tokens. It's annotated for POS and syntactic structure. This includes Enron Corporation emails from 1999-2002. In 2014, Silveira et al. provide annotation of syntactic dependencies for this corpus that can be used to train dependency parsers.

Sep
2019

**Common Crawl** publishes 240 TiB of uncompressed data from 2.55 billion web pages. Of these, 1 billion URLs were not present in previous crawls. Common Crawl started in 2008. In 2013, they moved from ARC to Web ARChive (WARC) file format. WAT files contain the metadata. WET file contain plaintext of the WARC files.

Text Corpus for NLP


# Ansible

## Summary

When software is delivered as a service, it's important that the server and the application running on it are initialized or configured correctly. Sometimes server modules or the application itself will need to be patched or upgraded. This task is more difficult when the service resides in the cloud and is running on multiple servers. What we need is a tool to automate these tasks. This is where Ansible is useful.

Ansible is well known as a configuration tool that can help us put a remote machine into a desired state. But Ansible can do more: deploy applications, orchestrate multiple tasks across multiple machines, run ad-hoc commands. In short, Ansible is a tool that enables Infrastructure-as-Code (IaC).

Ansible is open source and is sponsored by Red Hat.

## Discussion

### What are some typical use cases of Ansible?

The main uses of Ansible are the following:

    + **Provisioning**: Provision bare-metal servers, VMs, cloud instances, and more.
    + **Configuration Management**: Centralize configuration and push the same to all machines. This means a consistent environment for your services regardless of the machine.
    + **App Deployment**: Manage your app's lifecycle easily, from development to production.
    + **Continuous Delivery**: Create CI/CD pipelines without additional complexity.
    + **Security & Compliance**: Scan and remediate site-wide security policies.
    + **Orchestration**: Orchestrate how multiple configurations interact; manage the environment as a whole and not in silos.Consider an example of orchestration. A three-tier web app will have app servers, database servers, content servers, load balancers and a monitoring system. Ansible can do a complex cluster-wide rolling update without any downtime. In fact, Ansible enables *immutable infrastructure*, meaning that we can replace servers without service disruption.

It's interesting to note that Ansible configuration can be seen as a type of documentation and a solution for disaster recovery.

### In Ansible, what do mean by "agentless" architecture?

To configure a number of host machines, one approach is to install custom programs, called **agents**, on each of these machines. The system will also have a server or control node that contains the current configuration. Each agent will periodically poll the server to obtain the configuration. The server and agent communicate using a common pre-defined protocol and port. This is otherwise called the *pull* approach.

In Ansible, there are no agents. It uses the *push* approach, whereby the server pushes configuration or commands to the hosts. Hosts are often diverse. Installing agent programs on different OS/platform is a hassle. In Ansible, SSH is used by the server to talk to a host. OpenSSH is widely deployed. It's open source and lightweight.

What if an agent crashes? What if server and agents have mismatching versions? Some agents have reportedly used 400MB of memory. Ansible's agentless architecture avoids all these issues. We can start managing the hosts with *zero bootstrapping*.

### Could you describe some essentials Ansible terms?

From the complete Ansible Glossary, we describe a few essential terms:

    + **Node**: Can either be a *Control Node* or a *Managed Node*. Ansible is installed on control nodes but not on managed nodes. Managed nodes are also called *hosts*.
    + **Inventory**: A list of managed nodes, specified by an inventory file or *hostfile*. Hosts can be organized into *groups* for easier scaling.
    + **Task**: A unit of execution in Ansible. Each task has an associated command and does a single operation unless it has a *loop*.
    + **Playbook**: An ordered list of tasks. These are designed to be easy to read, write, share and understand. Playbooks can use *variables*.
    + **Module**: Reusable code that Ansible executes and can be invoked by a task. Ansible comes with dozens of out-of-the-box and ready-to-use modules. Modules are typically organized by their purpose: crypto, database, identity, messaging, network, storage, etc.
    + **Role**: Makes playbooks modular and reusable. These are units of organization in Ansible. A role can apply variables, tasks or handlers to hosts or groups.

### What's the Ansible architecture?

Ansible executes on a control node and connects to hosts using SSH by default. Control node can be Linux, MacOS, or even any machine that has Python 2.7 or 3.5+ installed. Nodes need to have either SSH or PowerShell (Windows); and Python 2.6+ or 3.5+ installed. File transfer is done using `sftp` or `scp`.

Hosts to be managed are listed in files called inventory. Modules offer reusable execution code. Playbooks contain step-by-step instructions.

Configuration and playbooks are stored as YAML files in a well-defined directory layout. YAML is easier to learn than bash scripting. Ansible Tower, the enterprise version of Ansible, allows configuration to be stored in configuration management databases (CMDB), using PostgreSQL or MongoDB.

It's been said that,

> If Ansible modules are the tools in your workshop, playbooks are your instruction manuals, and your inventory of hosts are your raw material.

### Could you share some best practices when using Ansible?

Keep it simple. Think declaratively. Don't write code in playbooks. Since modules abstract away complexity, always check if there's a module to accomplish what you want. Avoid using run commands (`command`, `shell`, `raw`, `script`), since error and state handling are difficult. Instead, write your own modules.

Don't ignore errors with `ignore_errors: yes` since you'll miss unexpected errors as well. Instead use `failed_when` and other alternatives. Along with roles, use `import*` and `include*` statements to logically chunk and reuse Ansible content.

In the interest of readability, use descriptive names for both plays and tasks. Variable names can collide. Use prefixes to avoid collision and also improve readability, such as, `apache_port` and `tomcat_port` rather than just `port`.

YAML's native syntax for name-value pairs is `name: value`. Use this for readability in preference of Ansible's shorthand `name=value`. The YAML syntax is easier to read and highlighted by editors and IDEs.

When using the debug module, use the `verbosity` parameter to suppress debug messages in production.

Organize your playbooks based on roles. Have separate inventory for staging and production. Define groups based on roles and geography.

Ansible-Lint and Playbook Best Practices guides can help.

### Could you point to useful resources to get started with Ansible?

From Ansible's official site, read an overview of Ansible. Learn about setting up SSH keys and running your first commands in the Getting Start guide. There are also a number of useful whitepapers on Ansible.

To be part of or contribute, read the Ansible Community Guide.

Ansible Galaxy is a hub for downloading useful roles. Official documentation lists all Ansible modules.

## Milestones

2008

While at Red Hat, Michael DeHaan starts **Cobbler**, a tool to manage Linux server environments including provisioning, managing DNS and DHCP, updating packaging, and more. Since Cobbler is unable to run ad-hoc tasks on remote hosts, **Func** is invented. The roots of Ansible are in these early tools.

Feb
2012

Ansible begins as project with system administrators and developers being early adopters. An early adopter is Fedora Infrastructure where Puppet is replaced with Ansible. Ansible embraces the "batteries included" philosophy, with folks contributing modules.

Feb
2013

Ansible 1.0 is released. We can now use `when_failed` and `when_changed` in playbooks. With v1.2.1 (July 2013), default connection is OpenSSH if it supports ControlPersists; else, fall back to Paramiko.

Aug
2014

Ansible 1.7 is released. It's the first version to support Windows hosts using native PowerShell rather than SSH. On the control node, Python module `winrm` is used. Though control node can't be Windows, Windows Subsystem for Linux (WSL) bash shell can be used.

Jan
2016

Ansible 2.0 is released.

Sep
2017

Ansible 2.4 is released. For hosts, support for Python 2.4 and 2.5 are dropped. Old `include` directives are replaced with `import` (static) and `include` (dynamic). Keyword `order` can be used to specify the order in which hosts are processed. Inventory is revamped. The release is updated in July 2018 as v2.4.7.

Mar
2018

Ansible 2.5 is released. It's recommended to use `loop` keyword instead of `with_*` style loops. The new syntax is better with filters instead of using complex `query` or `lookup`. Two top-level persistent connection types are introduced, `network_cli` and `netconf`. Where possible, prefer these over `local` connections.

2019

A survey of 786 IT professionals shows that Ansible is popular as a configuration tool for cloud infrastructure. Terraform has the largest growth in adoption.

Ansible


# MongoDB Aggregation

## Summary

Data can be processed in the application layer or in the database layer. Aggregation framework or pipeline is the manner in which MongoDB does data processing in the database layer. MongoDB's aggregation framework can do any kind of processing. Therefore it's really a Turing complete functional language.

Often just retrieving data is inadequate. We wish to perform some computations on the data or report on live data. Sometimes we don't want to copy data and process it elsewhere. These are good reasons for using MongoDB's aggregations.

While being flexible and intuitive for developers, MongoDB's aggregation is not as performant as some other databases or tools.

## Discussion

### What are the ways to do aggregation in MongoDB?

There are three ways to do aggregation in MongoDB:

    + **Aggregation Pipeline**: We can think of data as moving along a multi-stage pipeline. Each stage does specific processing on the data. Among the basic stages are *filters* that obtain a subset of matching documents and *transforms* that reshape data into a suitable output form. Other stages do sorting, grouping, string concatenation, array processing, and so on. The aggregation pipeline can operate on a sharded collection. Some stages can use indexes for better performance.
    + **Map-Reduce Function**: This legacy approach was deprecated in MongoDB 5.0. Aggregation pipeline is now the preferred approach since it offers better performance and usability.
    + **Single Purpose Aggregation Methods**: From the Collection class, the methods `estimatedDocumentCount()`, `count()` and `distinct()` aggregate from a single collection. While handy, these lack the flexibility and capability of the aggregation pipeline.

### Could you describe some aggregation pipeline operators or stages?

An aggregation pipeline is constructed with the `aggregate()` method of the `Collection` class and takes an array of stages: `db.collection.aggregate( [ { <stage> }, ... ] )`. Except for `$out`, `$merge` and `$geoNear`, other stages can appear multiple times in a pipeline.

We mention a few stages:

    + `$addFields`: Add new fields to documents. Operator `$set` is an alias.
    + `$count`: Count number of documents at this stage.
    + `$group`: Takes an identifier expression for grouping input documents. If specified, applies an accumulator expression for each group. Outputs one document per distinct group.
    + `$limit`: Passes only the first *n* documents to the next stage.
    + `$lookup`: Does a left outer join to another collection in the same database.
    + `$match`: Filters documents based on standard MongoDB queries.
    + `$merge`: Writes the output to a collection. Must be the last stage of the pipeline.
    + `$project`: Reshapes documents by adding or removing fields. Operator `$unset` is an alias for removing fields.
    + `$sample`: Randomly selects *n* documents.
    + `$skip`: Skips first *n* documents.
    + `$sort`: Reorders document stream by a specified sort key.
    + `$unwind`: Given an array field, outputs one document per array element.

### What are expression operators in MongoDB aggregation?

An expression operator has a name and takes either an array of arguments or a single argument. MongoDB has plenty of such operators. By becoming familiar with these, developers can simplify their application code by doing most of the processing in MongoDB.

There are many categories of expression operators: arithmetic, Boolean, comparison, conditional, data size, date, literal, object, set, string, text, trigonometry, type, etc. Custom expression operators include `$accumulator` and `$function`. Developers can write custom JavaScript functions (MongoDB 4.4) for these.

**Accumulators** are special expression operators that compute totals, maxima, minima and other values. They maintain their state as documents pass through the pipeline. These can be used only in some stages: `$bucket`, `$bucketAuto`, `$group` and `$setWindowFields`. However, some accumulators can be used in other stages but not as accumulators, that is, they don't maintain state. An example of this is shown in the figure: (a) averaging happens across documents within a group; (b) averaging happens over values within a document.

### How do I make use of variables in MongoDB aggregation?

Let's consider a `sales` collection with numeric fields `price` and `tax`, and Boolean field `applyDiscount`. We can use aggregation to compute the total for each sale. This is where the `$let` operator becomes useful.

The `$let` operator has two parts: `vars` block where the variables are assigned and `in` expression where the variables are used for computation. The `vars` block can access variables defined externally, including system variables. If externally defined variables are modified in `vars`, the changed values are seen only within the `in`expression. Externally, old values are retained.

System variables include `NOW`, `CLUSTER_TIME`, `ROOT`, `CURRENT`, `REMOVE`, `DESCEND`, `PRUNE` and `KEEP`.

### What's the Aggregation Pipeline Builder?

MongoDB Compass is a GUI for MongoDB. Aggregation Pipeline Builder is one of the tools within Compass. Developers can add/remove stages to the pipeline graphically. In *Auto Preview* mode, matching documents are automatically previewed. By default, first 20 documents are previewed. If *Sample Mode* is enabled, a default limit of 100,000 is applied. This affects `$group`, `$bucket`, and `$bucketAuto` stages.

When we select a pipeline operators, Compass gives the full syntax. We need to fill in the placeholders with relevant fields and expressions. If we make a mistake, useful suggestions are given to correct the same. This can be a useful feature for beginners.

It's possible to save the pipeline and reload it later within Compass. With the export feature, the pipeline can be exported in JSON or in a format compatible with a language driver. Likewise, it's possible to import into Compass a pipeline written in MongoDB Query Language.

### How's the performance of MongoDB's aggregations?

Compared to the legacy map-reduce, aggregation framework was 6x faster in a test done back in 2015. Unless you're stuck with legacy code, it makes sense to migrate to the aggregation framework.

Comparing MySQL 8.0 and MongoDB 4.0.3, it was seen that MongoDB is typically faster on more complex queries. It's faster from disk when there are no indexes, whereas MySQL is faster from RAM. BI Connector is slower for simple queries and not as fast as hand-crafted aggregation.

In a separate test, MongoDB's `insertMany()` and `updateMany()` were faster compared to their MySQL equivalents. However, a simple `find()` query was slower than MySQL's `SELECT`.

On large collections of millions of documents, MongoDB's aggregation was shown to be much worse than Elasticsearch. Performance worsens with collection size when MongoDB starts using the disk due to limited system RAM.

The `$lookup` stage used without indexes can be very slow.

While there are many powerful external compute engines (Spark, Hadoop, R, Python, Java, C), data transfer may become the bottleneck due to cost, network bandwidth, hardware limitations and security considerations. Therefore, use MongoDB aggregations when you can.

### Could you share some useful tips when building MongoDB aggregation pipelines?

The Aggregation Pipeline Quick Reference is a useful reference for developers.

When complex processing is required, aggregation pipelines help us break down the problem into smaller parts. We can think of each stage as solving a subproblem before handing over the intermediate result to the next stage. Pipelines also aid debugging or prototyping because we can comment out some stages.

The `$project` stage can be verbose and non-intuitive. Instead use `$set` and `$unset`. This makes the pipeline more maintainable when new fields are added to documents later.

While MongoDB automatically does optimization, use `db.coll.explain()` to see if further optimizations are possible.

Stages `$sort` and `$group` are blocking, that is, they can't send their output to the next stage until all input documents are processed. Blocking stages reduce concurrency and consume more memory. Some mitigation strategies include sorting on index, using `$limit` with sorting, and sorting later on a smaller subset. Group summary data only. Avoid unnecessary grouping. Use array operators instead of unwinding and regrouping. Move match filters to earlier in the pipeline.

### How do aggregation operators map to concepts in SQL?

The SQL to Aggregation Mapping Chart is a useful resource. From here, we note the following SQL-MongoDB equivalence:

    + **Filtering**: SQL uses `WHERE` and `HAVING` while MongDB uses the `$match` aggregation operator.
    + **Grouping**: SQL uses `GROUP BY` while MongDB uses the `$group` aggregation operator.
    + **Selecting**: SQL uses `SELECT` while MongDB uses the `$project` aggregation operator.
    + **Sorting**: SQL uses `ORDER BY` while MongDB uses the `$sort` aggregation operator.
    + **Limiting**: SQL uses `LIMIT` while MongDB uses the `$limit` aggregation operator.
    + **Joining**: Joining in MongoDB is a left outer join performed using the `$lookup` aggregation operator.
    + **Summing/Counting**: SQL's `SUM()` and `COUNT()` are implemented in MongoDB with the `$sum` aggregation operator.
    + **Combining**: SQL uses `UNION ALL` while MongDB uses the `$unionWith` aggregation operator.The figure gives some examples. The `orders` collection includes an array of `items` with each item containing fields `sku`, `qty` and `price`.

## Milestones

Feb
2009

**MongoDB 1.0** is released. By August, this version becomes generally available for production environments.

Dec
2009

MongoDB 1.2 is released with support for **aggregations**. This is implemented in Node.js via a Map-Reduce API. This is slow and queries are non-intuitive for developers.

Aug
2012

MongoDB 2.2 is released. This introduces the **Aggregation Framework**. This is exposed via the `aggregate` command and the `aggregate()` helper in the shell. Compared to earlier implementation of aggregations, this is far more intuitive, powerful, efficient and scalable. It soon becomes the go-to tool for developers. However, MongoDB continues to support Map-Reduce for legacy reasons.

Dec
2015

MongoDB 3.2 is released. New stages include `$sample`, `$indexStats` and `$lookup` (left outer join). Many new aggregation arithmetic operators and aggregation array operators are introduced. Results of aggregations can be routed to any shard for merging, thus avoiding an overload on primary shard.

Nov
2016

MongoDB 3.4 is released. New stages include `$graphLookup`, `$bucket`, `$bucketInfo`, `$facet`, `$sortByCount`, `$addFields`, `$replaceRoot` and `$count`. Many aggregation array/string/date operators and `$switch` aggregation control flow expression are introduced.

Nov
2017

MongoDB 3.6 is released. New stages include `$currentOp`, `$listSessions` and `$listLocalSessions`. New aggregation operators include `$arrayToObject`, `$objectToArray`, `$mergeObjects`, `$dateFromString`, `$dateFromParts` and `$dateToParts`. For date operators, time zones are supported. Aggregation variable `REMOVE` is added.

Jun
2018

MongoDB 4.0 is released. New aggregation operators for type conversion and trimming strings are added. MongoDB Compass 1.14 is released with **Aggregation Pipeline Builder**. In Compass 1.15 (Aug 2018), import/export feature is added to the builder. In Compass 1.16 (Nov 2018), collation support is added. In Compass 1.19 (Aug 2019), views can be created from pipeline results. Builder comes with new settings of sample size, number of documents to preview, and a maximum timeout. In Compass 1.20 (Dec 2019), `$set`, `$unset` and `$replaceWith` pipeline operators are added. In Compass 1.26.1 (Apr 2021), functions are allowed in pipelines.

Aug
2019

MongoDB 4.2 is released. Stage `$merge` is added. This helps create on-demand materialized views. Other new stages include `$planCacheStats`, `$replaceWith`, `$set` and `$unset`. Trigonometry expressions are added. New regex expressions include `$regexFind`, `$regexFindAll`, and `$regexMatch`. In earlier versions, only `$regex` query operator could be used in `$match` stage. New variables include `NOW` and `CLUSTER_TIME`. Aggregation pipeline can be used `findAndModify` and `update` commands.

Jul
2020

MongoDB 4.4 is released. Stage `$unionWith` can combine pipeline results from multiple collections. Operators `$accumulator` and `$function` allow custom aggregation expressions instead of using `mapReduce` and `$where`. Other new operators include `$binarySize`, `$bsonSize`, `$first` (first array item), `$last`, `$isNumber`, `$replaceOne` and `$replaceAll`. From this version, `$merge` can output to the same collection that's being aggregated; `$out` can output to a collection in a different database.

Jul
2021

MongoDB 5.0 is released. Pipeline stage `$setWindowFields` is added. Some date aggregation operators are added. Aggregation operators `$rand` and `$sampleRate` are added. For better performance, comparison operators with `$expr` use indexes. Multiple input expressions are possible for `$ifNull`. For better readability, `let` can be used to define and use variables within the pipeline. Concise correlated subqueries are supported for `$lookup`.

MongoDB Aggregation


# Yeoman

## Summary

Starting a new web app project often involves creating a basic structure of files and folders. Every web framework has its own conventions and recommended project structure. This is commonly called *scaffolding* of the project. Creating such a scaffolding is tedious. Yeoman is a Node.js module that simplifies this process.

**Generator** is the term used by Yeoman to refer to the packaging and distribution of code that creates scaffolding for a specific type of project. For example, `generator-node` creates a Node.js project while `sls` creates a serverless project.

For web apps, it automates mundane frontend tasks such as creating navigation, header or footer; or backend tasks such as creating APIs. Initially introduced for web apps, Yeoman's being used today for many types of project. Yeoman is open source, widely used, cross-platform, easy to use and based on the popular JavaScript language.

## Discussion

### Could you outline Yeoman's typical workflow?

A project will often have dependencies with other projects or packages. To bring in dependencies, we have **dependency/package managers** such as npm or Bower. These managers automate the task of downloading dependencies.

A project will also need a **build system** such as Gulp or Grunt. In web apps for example, these build system will minify CSS and JS, and do other transformations on the code.

A typical Yeoman workflow starts with setting up the scaffolding using the project's Yeoman **generator**. As part of the setup, a dependency file (`package.json` for npm) is created and the required dependencies are downloaded. Build configuration file is created and relevant build tasks are pulled into the project.

Note that the generator itself doesn't have dependent packages or build tasks, which are separately developed and delivered by different teams in the open source model. But the generator's author knows what tools are useful for the project. The author may also decide to give users options. For example, when the generator is invoked, user may be given a choice to use Gulp or Grunt.

### How is Yeoman better than native generators or CLI project tools?

Frameworks typically come with their own tools to create a sample project structure. For example, to build an Angular frontend app, `@angular/cli` provides scaffolding for a basic app with one static page. With Yeoman generators, we get many advanced options including promises, authentication, routing and a choice of desktop, mobile or PWA app.

Take the example of `ngx-rocket` generator for an Angular 7+ project. This includes a complete starter template, improved tooling, extensive base documentation, ready-to-use UI components, API proxy example setup for faster remote debugging, and generator output customization. For UI, we could choose Angular Material, Bootstrap 4 or Ionic. Mobile app uses Cordova, desktop app uses Electron, or we could use both in the same code base. Among output customization are SSO authentication, enterprise theming and service integrations. There's also a built-in development server with automatic reload when source files change.

Moreover, Yeoman enforces good coding practices since the generators themselves are written by experienced developers.

### What are Yeoman sub-generators?

A generator creates all the different parts of an application but what if you want to create only one part of the application? For example, a web app based on MVC pattern would include models, views and controllers. What if you want to create an additional view or model in an existing project? This is where Yeoman sub-generators are useful.

One example is the Joomla3 generator by Diego Castro. The generator creates a Joomla 3.5 component but it also has sub-generators to create CRUD for a component, a module, a plugin, a template, a rule, and so on. Thus, generators create the basic scaffolding and sub-generators can help us add more parts to it as required.

A sub-generator is invoked using the suffix `:subgen-name` followed by instance name. For example, to generator an Angular project we would call `yo angular` but to add an Angular service (a sub-generator), we use `yo angular:service myservice`; or an Angular filter using `yo angular:filter myfilter`.

It's also easy to pass data from generator to its sub-generator.

### Which are the core components of Yeoman?

The core system has the following components:

    + `yo`: This is the main command-line interface to use Yeoman. Users need to install this first and then install generators using this.
    + `yeoman-environment`: This is the runtime environment. It provides high-level API to discover, create and run generators.
    + `yeoman-generator`: This provides base classes that you can extend to make your own generators. These classes make it easier to create generators.All three components are open source and available in separate repositories on Yeoman's GitHub account. By studying the file `package.json` in these repositories, we can see that both `yo` and `yeoman-generator` depend on `yeoman-environment`. Thus, there's no need for developers or users to separately install the Yeoman environment.

### Is Yeoman only for JavaScript-based projects?

While Yeoman started with the idea of improving workflows of JavaScript-based projects, today it's used in many other types of projects. A generator is written as a Node.js module but otherwise it can target any project and create boilerplate code in any language.

An example is `generator-jvm` that creates a Java project. By passing the option `--type java-spring-boot` to this generator we can create a Java Spring Boot project.

Another example is `generator-joomla3`. Joomla is a Content Management System (CMS) based on PHP and yet we have a generator (and many sub-generators) to simplify the workflow for Joomla developers.

An interesting example is `generator-sls` that helps create a serverless project. This generator can create scaffolding targeting a number of languages as desired: Go, Node.js, Python, Java, C#.

### As a beginner, how can I get started with Yeoman?

Head to the official Yeoman site and read Getting Started, Tutorials, and FAQ pages.

Start by installing `yo` CLI tool using the command `npm install -g yo`. Subsequently, you can install any of the thousands of generators already out there.

Installing a specific generator is easy. To install `ngx-rocket` for example, type `npm install -g generator-ngx-rocket`. Refer to the documentation of that generator to know how to use it and create the project scaffolding. In `ngx-rocket` for example, we would start with the command `ngx new` and then respond to the prompts to customize the app.

### What's the procedure for creating my own Yeoman generator?

The official documentation explains how to create your own generator. You can start by creating a simple generator that just gives a bunch of boilerplate files. You can also give developers options to create a customized project. Any generator you create must depend on `yeoman-generator`, which in turn depends on `yeoman-environment`.

It's interesting to note that there's a generator to create a new generator. This is called `generator-generator` and this is probably the best place to start for beginners. Read the API documentation before updating the boilerplate output of this generator. The API documentation talks about modules, classes and mixins. The main class is named `Generator`.

Note that `generator-generator` depends on the package called `yeoman-generator`, which is the base from which all generators are written.

## Milestones

Sep
2012

Yeoman v0.9 is released on GitHub. A project can be scaffolded using the command `yeoman init`. Yeoman is described as "a robust and opinionated client-side stack, comprised of tools and frameworks that can help developers quickly build beautiful web applications". Yeoman is said to be inspired by Rails Generator system.

Aug
2013

Version 1.0.0 of CLI tool **yo** is released. This replaces the older commands to execute Yeoman.

Jun
2019

As of June 2019, Yeoman has 8000+ generators. Some popular ones at this point include jhipster, loopback, hyperledger-composer, feathers, code, travis, and node.

Yeoman


# Avionics Data Bus

## Summary

A bus is a data highway that links one LRU (Line Replaceable Unit) to another. In earlier avionic systems, communication cables are used to transmit the data. To transfer a bit of data at least one pair of wires is needed for each signal. These wires transmit information over a single twisted and shielded pair of wires (the databus) to all other system elements having need of that information (up to as many as 20 receivers). A data bus connects all the internal components of the computer to the CPU and main memory. When referred to as an aircraft, it is the data highway that links one computer to another aircraft like the FMC (Flight Management Computer) and the ADC (Air Data Computer).

## Discussion

### What are the types of avionics data bus?

Based on the type of connection data buses are classified into two types. They are serial bus and parallel bus.

    + **Serial Bus**: As the mentioned in the name, each bit of the data word is transferred via the same wire. Only one bit of word is sent at a time. Examples of serial data bus are PCIe, USB, ARINC, I2C, etc.
    + **Parallel Bus**: In this type, each bit of the data word is transferred via a specific wire. It requires a lot of wiring. Examples of parallel bus are conventional PCI.

### Which type of data bus is better, serial or parallel?

Parallel buses should have been faster than serial data buses. But, there are many issues in using parallel data buses. Such as, parallel buses suffer from clock skew. It means a bit of data can reach the destination before or after other bits. It causes lack of synchronization in parallel data buses. Also, the no of wires used in the parallel data bus is more than the serial data bus. Also, parallel data buses are not synchronized as fast as the serial data buses results in low data rate.

### What are the types of serial digital data transmission?

Common serial digital data transmissions are two types. They are single source-single sink and multiple source-multiple sink.

    + **Single Source-Single Sink**: It is the earliest application that contains a dedicated link. One LRU (line replaceable unit) communicates with a single LRU. Example: Sea Harrier and Tornado aircraft.
    + **Single Source-Multiple Sink**: In this system, one LRU transmits data to multiple LRUs at a time. ARINC 429 is the example for this type.
    + **Multiple Source-Multiple Sink**: In this system, multiple computers transmit data to multiple LRUs at a time. This is also known as full duplex and is widely used by military users. Example: MIL-STD-1553B and ARINC 629.

### Why avionics data buses are created in such a way?

In 1950s and 1960s, communication between two stations is done through a lot of wires without a bus. It makes very difficult for the installation and also for the modification. The parts of these systems are elctro-mechanical which are heavy and requires large space. After that digital systems are used to transmit the data. They are also heavy, slow and difficult to reprogram. Each component of the system contains its own computer and memory. They use a bus between components limiting the usage of more wires.

### What are the major avionics data buses?

Many commercial and military aircraft utilize many avionics data bus technologies. The major avionics data buses are MIL-STD-1553, ARINC 429, ARINC 629.

    + ARINC 429 defines how avionics equipments and other systems communicate on the aircraft. It is a unidirectional broadcast bus (simplex). The highest speed of data transmission is 100kbps and the least is 12 to 14.5kbps. It is a low weight and low cost with high reliability.
    + MIL-STD-1553B is a serial time division command response data bus. It has a bit rate of 1Mbps. It is a military half-duplex protocol.
    + ARINC 629 is used in Boeing 777. It is a multi-transmitter data bus. It is half-duplex with a bit rate of 2Mbps. The data is transferred using electromagnetic induction between two or more systems.

### What are the problems a data bus solves?

Data buses solve many problems. They are:

    + **ARINC 429**: It is developed to promote interoperability between devices on civil aircraft. Designed to connect a single transmitter to 10 receivers. Messages are expected to be transmitted between once and up to 50 times a second. Receivers receive their data simultaneously which accomplishes the synchronization.
    + **MIL-STD-1553B** Designed to connect up to 32 devices with the ability to transmit 16-bit words of data in a message. Simultaneous transmission by multiple devices is prevented. Synchronization is accomplished by using a Synchronization Mode Code.
    + **ARINC 629**: Published in 1990. It is intended to work in concert with ARINC 429. It is an attempt to reduce the wiring complexity inherent in the point-to-point ARINC 429 architecture. Facilitates the transfer of non-periodic data such as databases or video. Designed to connect up to 120 devices and be able to transmit strings of up to 256 sixteen-bit words with a speed of 2MHz. Simultaneous transmission by multiple devices is prevented by a set of timers in each device along with the ability to detect transmission on the bus.

## Milestones

1929

Four major airlines found ARINC to offer a single radio communications licensee and coordinator outside the government.

1970

ARINC 429 is widely used in commercial aircraft.

1970

Single Source-Single Sink used in Tornado and Sea Harrier avionics system.

1973

US air force first publishes the MIL-STD-1553B standard data bus to use in the F-16 Falcon fighter aircraft.

1975

Many improvements are done in ARINC 424 to introduce new record types and field values and revision of coding rules are done as required by new technologies and classes of equipment such as GPS and Flight Management Computers.

1978

ARINC introduces a datalink system known as ACARS (Aircraft Communications Addressing and Reporting Systems). It uploads and downloads data via an onboard Communications Management Unit (CMU).

1980

Boeing develops a digital data bus known as Digital Autonomous Terminal Access Communication (DATAC) is the digital data bus which later becomes an ARINC standard as ARINC629.

1995

The ARINC 629 computer bus introduced to use on the Boeing 777.

2013

ARINC Inc., sold a small subsidiary and is looking to sell another in the wake of its $1.4 billion purchase of the aviation engineering and communications firm.

2014

SAE International bought the ARINC Industry Activities (IA) Division. It now operates under the SAE Industry Technologies Consortia (SAE ITC).

2018

MIL-STD-1553C is the latest version of MIL-STD-1553 made in February.

Avionics Data Bus


# Penetration Testing

## Summary

Today's networked systems have a greater attack surface because computers are interconnected and can be accessed remotely. Systems and servers are also complex since they offer dozens of services and run many different software packages. Anyone gaining access to such a system without authorization can damage operations and steal data. It's therefore essential to know what security vulnerabilities are present and take corrective action early on.

Penetration testing is a proactive attempt to discover and exploit vulnerabilities in the system. The idea is not to cause damage as a malicious hacker would. Instead, it's to discover what's wrong, show how much damage can be done, or prove that existing measures are mitigating these attacks as they happen. Penetration testing can be manual or automated.

Penetration testing is also called **pen testing**.

## Discussion

### What's a typical pen testing process?

With pen testing, we typically explore or scan the system to discover vulnerabilities. Once discovered, we try to exploit them and see what happens. The behaviour is captured and reported for analysis. Scanning can be static code analysis or dynamic run-time analysis.

Pen testing requires a proper testing plan. Reporting should be consistent. Problems should be marked with levels of severity. Finally, there should be an action plan to mitigate or manage vulnerabilities.

Since attacks are often preceded by an assessment, **Vulnerability Assessment and Penetration Testing (VAPT)** is a term commonly used. It's also possible to do both these in parallel: vulnerability assessment involving scans and analysis; pen testing involving analysis and attacks. A vulnerability scan is faster than detailed pen testing but pen testing is more accurate since some reported vulnerabilities might be false positives.

### What are some types of pen testing?

Pen testing typically covers one or more of the following:

    + **Network**: Involves firewalls, routers, SSH, port scanning, and anything else that will allow an attacker to gain network access.
    + **Wireless**: Often systems involve wireless access. Pen testing should look at wireless traffic, encryption protocols, unauthorized access points, poor passwords, MAC address spoofing, etc.
    + **Web**: Any website or web app can be vulnerable at the application layer. Pen testing should check cross-site scripting, SQL injections, DoS attacks, web server misconfiguration, failure to protect sensitive data, etc. This could include vulnerabilities on the client-side of the app.
    + **Physical**: Systems have to be physically secure as well in terms of locks, motion sensors, etc.
    + **Social Engineering**: Users can be manipulated or fooled to provide unauthorized access to systems. Phishing, tailgating and eavesdropping are some techniques.
    + **Cloud**: When apps reside in the cloud, cloud pen testing may be required. You will need to know from your cloud provider what assets are off limits for pen testing.

### What are some methods or approaches to pen testing?

**External testing** is useful to mimic how an outsider might attack the system. **Internal testing** is to ascertain how much and what damage a disgruntled employee or another user with authorized access can cause.

**Targeted testing** is when multiple teams work together and everyone can see the tests as they execute. **Blind testing** is when limited information is given to the pen tester. **Double-blind testing** goes one step further: very few in the company know that pen testing is happening. This will determine if security monitoring and responses happen as expected.

**Black box testing** is an extreme case of blind testing. Pen testers are given no information. **White box testing** gives pen testers lots of useful information to speed up the process. Black box testing is more real but slow. White box testing is more thorough.

### When is the right time to do pen testing?

Pen testing should be done regularly. Previous approaches of doing this annually are seen to be outdated. The recommendation is to have pen testing as part of regular development and release processes. If this is not possible in-house, there are providers who offer pen testing as a service.

Typically, vulnerability scans could be part of CI/CD pipelines and longer attack tests could be done nightly. In addition, when there are changes made to the app/network/server/system, pen testing should be done. Such changes might include moving to a new office premise, releasing a patch, changing user policies, or adding new infrastructure.

### Could you mention some tools that help in pen testing?

It's interesting to note that tools that might be used by malicious hackers can also be useful for pen testing. The site SecTools.Org maintains a list of 100+ network security tools.

*Angry IP Scanner* is for scanning IP addresses and ports. *Ettercap* can read and inject packets into the network adaptor. For cracking passwords there's *Cain and Abel* or *John the Ripper*. To crack Wi-Fi passwords, *Aircrack-ng* is useful. For attacking web apps, we have *Burp Suite* and *OWASP ZAP*. For exploiting SQL flaws, there's *sqlmap*. *Nessus* is a well-known vulnerability scanner. *Wireshark* is a popular tool for interactively analyzing network logs. *Metasploit* is a framework to develop and reuse exploit code.  System admins can use *Responder*, *Powershell Empire*, *Hashcat*, *Arpspoof* and *Wireshark*.

In any case, you should select tools that are easy to use, are suited for automation, can re-verify previous exploits, and generate detailed logs.

Kali Linux is a Linux distribution that includes many useful tools for easy installation and use.

### Isn't pen testing the same as ethical hacking?

Often these two terms are used interchangeably but there's a subtle difference. Pen testing may be seen as a subset of ethical hacking. Pen testing is narrowly focused to discover and show that vulnerabilities can be exploited. Ethical hacking has a broader scope. An ethical hacker can use pen testing plus various other methods across the entire IT infrastructure.

While pen testing aims to give confidence about system security (usually based on known vulnerabilities), ethical hacking will try to see things from the perspective of a malicious hacker. An ethical hacker may use novel techniques of hacking. He may use social engineering, rummage through logs, examine patch installations, sniff networks, etc.

An ethical hacker may infiltrate the system, move laterally within or to other systems, and even export sensitive data. However, some pen testers may try to include these in their tests.

It's important to note that both pen testers and ethical hackers are required to get written authorization and clear understanding of the scope of work.

### Are there any standards or methodologies to guide in pen testing?

Among the more well-known standards is **Penetration Testing Execution Standard (PTES)** that was defined in 2009. Its goal is to provide guidance about tools and techniques. A standard such as PTES is particularly useful when organizations engage external testers or testing services. With PTES, it's easier to filter out low quality testers.

PTES does have its limitations. Threats are evolving rapidly and it's impossible for a static standard to be comprehensive. Thus, PTES should be seen as a minimum requirement. PTES Technical Guidelines are available online and publicly accessible.

Another standard is called **Open Source Security Testing Methodology Manual (OSSTMM)** from ISECOM. It's been around since 2001. **Payment Card Industry Data Security Standard (PCI DSS)** provides guidance for pen testing among things. There's also **Information Systems Security Assessment Framework (ISSAF)** but this is not actively updated.

## Milestones

1960

This decade sees the growing popularity of time-sharing computer systems. Users can remotely access and time-share computers over communication lines. It's recognized that if the communication lines are compromised, computer systems could be attacked.

1967

At the annual Joint Computer Conference held in the U.S., the security of computer communication lines is discussed. The term "penetration" is used for the first time. Subsequently, RAND Corporation and Advanced Research Projects Agency (ARPA) publish a detailed report, commonly called *The Willis Report*. This report becomes influential for later studies.

1972

From late 1960s and during 1970s, *tiger teams* are formed. They're sponsored by governments and companies. Their task is to attack computer systems, uncover security holes and then fix them. Pen testing as we know it today starts here. In 1972, an early pioneer of pen testing, James P. Anderson outlines steps that tiger teams can follow for pen testing.

1974

U.S. government conducts pen testing on its Multics (Multiplexed Information and Computing Service) system, an early system that precedes UNIX. The pen tests reveal many vulnerabilities despite Multics being one of the most secure systems.

1995

Dan Farmer of Sun Microsystems and Wietse Venema of the Eindhoven University of Technology release a paper titled *Improving the Security of Your Site by Breaking Into It*. They use the term "uebercracker" to refer to a system cracker who has gone beyond simple cookbook methods of breaking into systems. They also describe **SATAN (Security Analysis Tool for Auditing Networks)**, one of the earliest tools to automate pen testing. It's written in shell, perl, expect and C.

Sep
2001

**Open Web Application Security Project (OWASP)** is started. It's pen testing tool, **Zed Attack Proxy (ZAP)** is released in September 2010. OWASP ZAP is free, open source, cross-platform and well documented.

2009

**Penetration Testing Execution Standard (PTES)** is started by six information security consultants. PTES provides guidelines towards effective pen testing.

Penetration Testing


# Test Frameworks

## Summary

Engineers write test cases to ensure that software works as expected. Often test cases require common services. These services are also agnostic of each test. The purpose of test frameworks is to provide a common platform for writing, organizing and executing test cases.

A test framework is a set of guidelines or rules that enable more efficient testing. To say it differently, a test framework provides a consistent interface between your code and your tests. Test frameworks basically provide the scaffolding. Test engineers can therefore focus on writing tests and testing the core functionality of their software.

The term **Test Automation Framework** is commonly used, though test frameworks can in rare cases be used to aid manual testing.

## Discussion

### What are some benefits of using a test framework?

Some benefits include improved test efficiency, lower maintenance costs, minimal manual intervention, maximum test coverage, code reuse, handling of recovery scenarios, and easy reporting.

Obviously, test frameworks play an essential role in test automation, and test automation increases the depth and scope of testing. Lengthy or complex tests are hard to run manually. Developers can also focus on creating more tests than spend time executing tests. Essentially, better testing leads to higher quality software in a short time to market.

It's recommended that you select a framework already available in the market rather than write your own custom framework. By doing this, you can focus on writing effective test cases and test suites.

### What's the typical execution process of a test framework?

Here we describe the typical process. Further details will vary depending on features that frameworks support.

The process starts by gathering all tests for execution. These tests are commonly grouped within **test suites** that are manually specified or could be discovered by convention. For example, all method/function names starting with a certain prefix are treated executable tests.

Test executions start and end with **setup** and **teardown** code respectively. This setup/teardown code could be at test suite level, module level, test case level, or all of these.

Within each test, one or more **assertions** are invoked. These compare current runtime status with expected status. If as expected, test execution continues. If not as expected, test fails. Stack trace may be recorded for later analysis. Extra logs may be taken. In all cases, final verdict from test execution is recorded.

The final stage is **reporting**. Reports in preferred formats will contain summary, test verdicts, test-level execution metrics, and access to logs or traces.

### What features should I look for when choosing a test framework?

As in any tool, you should look at industry adoption, cost, performance, stability, scalability, (community) support, documentation, and features. Here are some features expected of a good framework:

    + **Agnostic**: Agnostic of System Under Test (SUT), automation libraries, platform or OS.
    + **Language**: What scripting languages are supported?
    + **Usage**: Easy to install, configure and use.
    + **Integration**: Integrates well with automation tools, Continuous Integration (CI) tools, issue tracking tools, etc.
    + **Reporting**: Generates detailed reports and analytics on test executions. Captures failure logs. Collects execution metrics.
    + **Customization**: Extend the framework. Group, tag, order or skip tests. Configure test environment.
    + **Modular**: Clear separation of framework layer and automation test suite. Automation infrastructure should follow coding conventions and design patterns. Promotes reuse of modules or libraries.
    + **Data**: Test scripts are separated from the data. Support for parameterized testing.
    + **Discovery**: Discover tests in project paths and automatically compose test suites for execution.
    + **Playback**: Record steps of manual tests and play them back for automated testing.

### What are some types of test frameworks?

Test frameworks differ in their approach to testing, which are briefly described as follows:

    + **Linear**: Tests are independent of one another, often created by recording manual actions and replaying them.
    + **Module Based**: The application is tested in its smaller parts (functions, modules, sub-systems). Tests can be combined to enable system-level tests as well.
    + **Library Based**: Reusable libraries are identified and written. Good for test maintenance and scalability.
    + **Data Driven**: Focus is on executing a test with different input data. Data is read from databases or spreadsheets. There's a clear separation of test scripts and input data. Hard-coding of data within scripts is avoided.
    + **Keyword Driven**: An extension of data-driven approach. Test execution is basically a sequence of steps defined by keywords. Keywords are mapped to objects and actions in a *shared object repository*. Promotes reuse.
    + **Behaviour Driven**: Tests are written in a descriptive style so that even business stakeholders can understand them. Tests are in terms of behaviours and user-focused scenarios.
    + **Hybrid**: A mix of the above.

### Could you name some test frameworks out there?

A blog post on DZone (March 2018) identified ten open source test frameworks: Selenium, Carina, EarlGrey, Cucumber, Watir, Appium, RobotFramework, Apache JMeter, Gauge and Robotium. Selenium is mainly for web apps while Appium is for mobile apps. Carina can use the automation of Selenium or Appium. Watir is based on Ruby while RobotFramework is based on Python. Watir supports data-driven testing while RobotFramework uses keyword-driven approach for acceptance testing.

Cucumber is a framework for Behaviour-Driven Development (BDD). Tests are written in **Gherkin** language and the focus is on acceptance testing.

Another list on TestBeacon includes Serenity, Cypress, RobotFramework, RedwoodHQ, Sahi, Galen, Gauge, Citrus, and Karate-DSL. Serenity uses Selenium WebDriver and BDD tools such as Cucumber and JBehave. Cypress enabled Test-Driven Development (TDD). Galen is useful for testing layouts and user experience. Gauge might suit TDD and BDD practitioners. Citrus is suitable for testing network calls such as REST or SOAP.

Specific to Android testing are Appium, Robotium, Selendroid, MonkeyTalk, Calabash, UI Automator, Magneto, Kobiton, Squish, SeeTest, KMAX, MonkeyRunner, Frank, KIF, and Testdroid.

Wikipedia has a long list of test frameworks.

### How is a test automation tool different from a test framework?

There's a difference between a test framework and a test automation tool. An automation tool provides the implementation of testing actions. For example, Selenium WebDriver implements actions that mimic user actions on user interfaces. Thus, Selenium WebDriver is an automation tool.

However, Selenium WebDriver can be called from within any framework that supports it. For example, *pytest* is a test framework in Python, and since Selenium has Python support, we can call Selenium WebDriver APIs from Python scripts that are managed by *pytest*. Tools such as Selenium IDE is a simple framework that has the automation part built into it.

### How is a test management tool different from a test framework?

A test management tool is more about managing the testing process. This includes interfacing to various tools and processes that manage requirements, plan product releases, do reporting, track defects, execute tests, etc. There are plenty of such tools in the market. A few of them include qTest, PractiTest, Zephyr, Test Collab, and TestFLO for JIRA.

A test framework is more about writing, managing and executing the tests. They are also about test automation. While frameworks may integrate with other tools and processes, their focus is more on testing rather than the bigger picture.

## Milestones

1989

At WyattSoftware, Gary Goldberg manages his test cases in spreadsheets. This is for a financial software written in Smalltalk. His colleague Ward Cunningham writes a framework to read the tests and execute them. This is probably one of the earliest test frameworks.

1994

Kent Beck writes **SUnit** for unit testing of Smalltalk code. It has only three classes (TestCase, TestSuite, TestResult) and twelve methods.

1997

Inspired by SUnit, Kent Beck and Erich Gamma create **JUnit** for Java unit testing.

2004

At ThoughtWorks, Jason Huggins creates what's now called **Selenium 1 Core** to automate web app testing. It uses scripts encoded in HTML tables. Selenium is today managed by the Software Freedom Conservancy.

2012

At the Selenium Conference 2012, Dan Cuellar gives a lightning talk about automating iOS app testing using Selenium-style syntax written in C#. Later that year, he open sources the code and adds supports for Python. Jason Huggins adds Selenium WebDriver support. At the Mobile Testing Summit in November, it's launched as **Appium**. In 2016, Appium is handed over to JS Foundation.

Test Frameworks


# Transpiler

## Summary

Suppose you've written a program in one language but wish to convert this to another language, then you would invoke what's called a **transpiler**. The programming language at the input to the transpiler may be called the source language whereas the language at the output may be called the target language. A transpiler is sometimes called a **source-to-source compiler**.

For example, converting C++ code to C code will involve a transpiler. Converting Python code to Ruby code will involve a transpiler. Let's note that in these example both source and target languages are at the same level of abstraction. But let's say we convert Java code to bytecode, or C code to assembly code, then this is not called transpilation. Transpilers don't change the level of abstraction such as translating from a high-level language to assembly code, bytecode or machine code.

## Discussion

### How is a transpiler different from a compiler?

A compiler converts high-level human-readable code to low-level instructions that can run on a computer. For example, a C compiler converts C code to machine executable code. A Java compiler converts Java code to what's called Java bytecode that's interpreted at runtime to execute on the target machine.

A transpiler on the other hand usually works at the abstraction of high-level languages. The output code is still human readable. It cannot be executed directly unless its own compiler or interpreter is invoked. For example, a transpiler can convert Java code to C++ code. Programmer will still need to invoke a C++ compiler before executing the resultant machine code.

However, some folks use these terms interchangeably. For example, VOC is stated to be transpiler from Python source code to Java bytecode when in actual fact these two are at different levels of abstraction. However, it's acceptable to call translations at the same level of abstraction as transpilation, such as from .ASM assembly code to .A86 assembly code.

ISO/IEC/IEEE 24765 defines five generations of languages that can be related to abstractions described above.

### Why would I want to use a transpiler?

Here are a few reasons for using a transpiler:

    + **Migration**: Migrate legacy code to a modern language.
    + **Compatibility**: Generate code conforming to an older version while developers benefit from features of a newer version, such as with browsers not updated to latest JavaScript standards.
    + **Coding Skills**: Codebase can be transpiled to a language in which the skills are readily available. Or it might be a matter of preference. For example, those who are from OOP background prefer TypeScript. Python coders prefer CoffeeScript instead. In both cases, code is transpiled to JavaScript.
    + **Performance**: Initial code was written for quick prototyping or project is moving to a different platform where another language is more suitable. Perhaps the target language has a better compiler that can generate more optimized code. For example, critical parts of a Python codebase can be transpiled to Fortran and then called from Python.
    + **Tooling**: Source language has better IDEs, testing and debugging tools, such as with .NET and Visual Studio.

### How does a transpiler work internally?

A transpiler parses the code and extracts the tokens, which are the basic building blocks of the language. Language keywords, variables, literals and operators are examples of tokens. This step is a combination of lexical analysis and syntax analysis. Transpiler knows how to do this because it understands the syntax rules of the input language. Given this understanding, the transpiler builds what is called *Abstract Syntax Tree (AST)*.

The next step is to transform the AST to suit the target language. This is then used to generate code in the target language.

AST Explorer is an online tool to help you view the AST given a piece of code in many popular languages.

### Could you name some transpilers out there?

JavaScript is a language that's been evolving at a regular pace but there are also lots of browsers out on the web using older versions of the language. Transpilers are therefore used commonly in the JS world to transpile code to ES5, a version that's supported by most browsers. There are lots of transpilers for JS but the popular ones are Babel, TypeScript and CoffeeScript.

ClojureScript transpiles from Clojure to JS. JSweet transpiles from Java to TypeScript or JS. GWT Web Toolkit, formerly known as Google Web Toolkit, enables Java programmers to use JS frontend for browser-based applications. GWT includes a transpiler from Java to JS. Script# transpiles from C# to JS. C2Rust and Corrode are two alternatives for transpiling C to Rust.

VOC is a transpiler from Python to Java, which means that even Android apps can be built from a Python codebase. Python itself has two incompatible versions: 2 and 3. However, tools are available to convert from one version to the other: `2to3`, `modernize`, `futurize` and `pasteurize`. These could be termed as transpilers.

## Milestones

1979

Intel publishes details of a converter called **CONV86**. It can convert from 8080/8085 assembly source code to 8086 assembly source code. Since the latter is a typed language, the converter assigns types to each symbol. Note that the term *converter* is used. The term *transpiler* possibly came later.

1980

Tim Paterson writes **TRANS.COM** to translate Z80 assembly code into .ASM assembly code for the Intel 8086 processor. The process is not fully automated and requires manual corrections.

1981

Gary Kildall writes a program called **XLT86** to transpile .ASM assembly code written for Intel 8080 processor into .A86 assembly code for the Intel 8086. The idea is to automatically port P/M-80 and MP/M-80 programs to CP/M-86 and MP/M-86 platforms.

1983

Bjarne Stroustrup, the creator of C++, creates **Cfront** to transpile C++ code to C code since C++ doesn't have its own compiler at the time. Development of Cfront is discontinued in 1993.

2010

Facebook creates **HipHop for PHP (HPHPc)** for transpiling PHP to C++ for better runtime performance. This is deprecated in 2013 by HipHop Virtual Machine (HHVM) that uses just-in-time compilation. HPHPc did not fully support PHP, its performance flattened out and its deployment across all of Facebook's servers was not trivial.

Sep
2014

Sebastian McKenzie creates a transpiler named **6to5**, which translates ES6 code to ES5 code. In February 2015, this becomes **Babel**.

Transpiler