# Tor Browser

## Summary

Tor Browser is a standalone desktop application that helps users browse the web safely and anonymously. Since many users use the Internet via a web browser, Tor Browser is a useful application for anonymity. It hides your device's IP address and its location. It prevents third parties from snooping into your online activities or even tracking you.

Tor Browser is based on the open sourced code of Firefox browser. It adds further privacy and security settings.

Underneath, Tor Browser relies on the Tor network of relay nodes. In the past, it was difficult to use Tor since one had to install a number of tools separately. Today, Tor Browser bundles all the necessary tools into a single archive of files, thus making it easier for users to adopt Tor.

## Discussion

### What are the features of the Tor Browser?

Tor Browser is cross-platform and available for x86 and x86\_64 architectures. Encryption and decryption are done automatically. It can update itself to latest version. What previously required separate installations of Tor, Firefox browser, Torbutton (Firefox add-on) and Polipo (HTTP proxy), are now conveniently bundled within the Tor Browser.

A new separate circuit is automatically created for each domain though they share a common guard node. This process is transparent to users but Tor Browser allows users to inspect the current circuit and request a new circuit if so desired. From within the browser, users can also request Tor bridges by solving a captcha. Each circuit lives for only ten minutes.

Since each website is isolated from another, this prevents tracking from third parties. Tor Browser resists browser fingerprinting. To prevent cookie-based tracking, cookies and browsing history are cleared when the browser is closed. It also prefers DuckDuckGo over Google as the search engine, since Google tracks you and logs your search queries.

### What are the variants of Tor Browser for different platforms?

Tor Browser is available for Microsoft Windows, Apple MacOS and GNU/Linux. As of March 2019, an experimental version 8.5a8 is also available for Android.

Two popular add-ons that come with Tor Browser by default are NoScript and HTTPS Everywhere.

For Android, there's also **Orbot**, a proxy that enables Tor access for any mobile app. **Orfox** is a browser for Android that was developed as part of the Guardian Project. This is likely to be discontinued when a stable version of Tor Browser for Android is released.

Third-party **Onion Browser** is the one to use for iOS.

### I already use private/incognito mode in Firefox/Chrome. Why do I need Tor Browser?

Private modes avoid saving your browsing history and cookies. Otherwise, they are vulnerable to fingerprinting and network adversaries. Shortcomings are plugins, fingerprinting, DNS leaks, SSL state leaks, autofill and site-specific zoom. This is where Tor Browser becomes useful.

On a related note, Tor provides anonymity at the routing layer. However, Tor can't protect you if your hardware is compromised, such as a key logger. Tor doesn't encrypt traffic between the exit node and final destination, for which you should use app-level encryption, such as SSL for HTTPS traffic. In fact, an add-on such as *HTTPS Everywhere* can help in enforcing security for sites that support it. Electronic Frontier Foundation has a nice animation to show how HTTPS works alongside Tor.

Tor will also not protect you from improper usage. For example, BitTorrent over Tor is not anonymous.

### What are some alternatives to the Tor Browser for anonymous web browsing?

Tor Browser is not the only way to achieve anonymity while browsing the web. *I2P* provides a peer-to-peer distributed communications layer. It's suited for hidden services. *Freenet* uses a similar P2P technology. Linux-based distributions that focus on privacy and anonymity (some of which use Tor underneath) include *Tails*, *Subgraph* and *Freepto*.

*Qubes OS* is a desktop OS focused on security. It uses virtualization to give users any OS of their choice. *Whonix* OS is integrated into Qubes. With Whonix, we can connect to Tor from inside a VM.

*Epic Browser* doesn't use any special networking architecture. However, it disables history, third-party cookies, DNS pre-fetching, and autofill in forms. These are common ways in which privacy can be compromised.

*Brave* browser integrates Tor. Users can open a private tab with Tor.

Other alternatives are listed at alternative.me.

## Milestones

Oct
2003

Tor network is deployed. Tor code is open sourced under MIT license. Tor itself is an improvement over Onion Routing that started as a research project in 1995.

Mar
2008

Version 1.0.0 of Tor Browser Bundle is released.

Jun
2017

Tor Browser 7.0 is released with multiprocess mode and content sandbox as two major features. Sandboxing is not yet available on Windows. Until this release, lack of sandboxing was a problem and was exploited by the FBI. This version is based on Firefox 52 Extended Support Release (ESR). ESR is meant to help organizations to mass deploy the browser.

Sep
2018

Tor Browser version 8 is released based on Firefox Quantum codebase that came out in November 2017. User interface is Photon UI that Firefox Quantum uses. Also in September, alpha release of **Tor Browser for Android** happens.

May
2019

Tor Browser version 8.5 becomes the first stable release for Android. Tor Browser also gets newly designed logos compatible with Firefox's Photon UI.

Tor Browser


# NumPy

## Summary

NumPy is an open source Python library that enables efficient manipulation of multi-dimensional numerical data structures. These are called **arrays** in NumPy. NumPy is an alternative to Interactive Data Language (IDL) and MATLAB.

Since it's release in 2005, NumPy has become a fundamental package for numerical and scientific computing in Python. In addition to efficient data structures and operations on them, it provides many high-level mathematical functions that aid scientific computation. Pandas, SciPy, Matplotlib, scikit-learn and scikit-image are just a few popular scientific packages that make use of NumPy.

## Discussion

### What does NumPy do differently from core Python?

Python is slower than compiled languages such as C but it's easy to learn. Python is suited for rapid prototyping and iterative development.

While Python's `list` data type can be used to construct multi-dimensional data structures (lists containing lists), NumPy is faster and provides a better API for developers. Python's lists are general purpose. They can contain data of different types. This means that types are also stored, type-dispatching code is invoked at runtime and types are checked. Lists are processed using loops or comprehensions and can't be vectorized to support elementwise operations. NumPy sacrifices some of Python's flexibility to improve performance.

Specifically, NumPy is better at these aspects:

    + **Size**: NumPy data structures take up less space. Each Python integer object takes 28 bytes whereas in NumPy an integer is just 8 bytes. A Python list of `n` items requires `64+8n+28n` bytes whereas in NumPy it's `96+8n` bytes.
    + **Performance**: NumPy code runs faster than Python code, particularly for large input data.
    + **Functionality**: NumPy provides lots of functions and methods to simplify operations. High-level operations such as linear algebra are also included.

### What are some of the main features of NumPy?

NumPy arrays are **homogeneous**, meaning that array elements are of the same type. Hence, no type checking is required at runtime. All elements of an array take up same amount of space.

The spacing between elements along an axis is also constant. This is called **striding**. This is useful when the same data in memory can be used to create a new array without copying. Different arrays are therefore different **views** into memory. Thus, it's easier to modify data subsets in memory.

Operations are **vectorized**, which means that the operation can be executed in parallel on multiple elements of the array. This speeds up computation. Developers need not write `for` loops.

NumPy provides APIs for easy manipulation of arrays. Some of these are indexing, slicing, reshaping, stacking and splitting. **Broadcasting** is a feature that allows operations between vectors and scalars, or vectors of different sizes.

NumPy **integrates easily** with C/C++ or Fortran code that may provide optimized implementations. Useful functions covering linear algebra, Fourier transform, and random numbers are provided.

### Could you share some performance numbers comparing NumPy versus Python implementations?

For a simple computation of mean and standard deviation of a million floating point numbers, NumPy was **30X faster** than a pure Python implementation. However, optimized Cython and C implementations were even faster. Another study showed that if input is small (less than 200 numbers), pure Python did better than NumPy. For inputs greater than about 15,000 numbers, NumPy outperformed C++.

One experiment in Machine Learning compared pure Python, NumPy and TensorFlow (on CPU) implementations of gradient descent. Runtimes were 18.65, 0.32 and 1.20 seconds respectively. NumPy was **50X faster** than pure Python. For more complex ML problems deployed on multiple GPUs, TensorFlow is likely to outperform NumPy.

When evaluating NumPy performance, the underlying library for vector/matrix computations matters. NumPy comes with *Default BLAS & Lapack*. Depending on the distribution, alternatives may be included: *OpenBLAS*, *Intel MKL*, *ATLAS*, etc. In general, these alternatives are faster than the default library. For example, SVD is 10X faster on Intel MKL.

Hardware platforms may provide further acceleration. For example, Intel AVX2 provides at least 20% improvement on top of OpenBLAS.

### Does NumPy automatically make use of GPU hardware?

NumPy doesn't natively support GPUs. However, there are tools and libraries to run NumPy on GPUs.

**Numba** is a Python compiler that can compile Python code to run on multicore CPUs and CUDA-enabled GPUs. Numba also understands NumPy and generates optimized compiled code. Developers specify type signatures for Python functions. Numba uses them towards just-in-time (JIT) compilation. Numba team also provides `pyculib`, which is a Python interface to CUDA libraries such as cuBLAS, cuFFT and cuRAND.

**Grumpy** has been proposed as a framework to seamlessly target multicore CPUs and GPUs. It does a mix of JIT compilation and offloading to optimized libraries such as cuBLAS or LAPACK.

**CuPy** is a Python library that implements NumPy arrays for CUDA-enabled GPUs and leverages CUDA GPU acceleration libraries. The code is mostly a drop-in replacement to NumPy code since the APIs are very similar. **PyCUDA** is a similar library from NVIDIA.

**MinPy** is similar to CuPy and is meant to be a NumPy interface above MXNet for building artificial neural networks. It includes auto differentiation in addition to transparent CPU/GPU acceleration.

### What are some essential resources to learn NumPy?

The main NumPy website is the definitive resource to consult. Beginners can start by reading their Quickstart tutorial or the absolute beginner's guide. The latter includes the basics of installing NumPy.

Rougier's book titled From Python to Numpy focuses on Python programmers who wish to learn NumPy and it's vectorization. Perhaps a classic is the PhD thesis titled Guide to NumPy, by Travis E. Oliphant who created NumPy.

MATLAB users might want to read NumPy for Matlab users. It maps MATLAB operations to NumPy equivalents.

DataCamp blog has shared a handy NumPy cheatsheet.

Those who wish to contribute to the NumPy project or study it's source code can head to NumPy's GitHub repository.

## Milestones

1995

*Numeric* is released to enable numerical computations. It's designed to provide **homogeneous numeric arrays**, that is, arrays whose elements all belong to the same data type, and therefore easier and faster to process.

2005

*NumPy* is released based on an older library named *Numeric*. It also combines features of another library named *Numarray*. NumPy is initially named *SciPy Core* but renamed to *NumPy* in January 2006.

Oct
2006

*NumPy v1.0* is released.

Apr
2009

*NumPy v1.3.0* is released. This release includes **experimental Windows 64-bit support**. Support for 64-bit OpenBLAS comes a decade later in December 2019.

Aug
2010

*NumPy v1.5.0* is released. This is the **first release to support Python 3**.

Jan
2019

GitHub publishes a study of Machine Learning (ML) projects hosted on their platform. The study spans contributions from Jan-Dec 2018. It's seen that **74%** of ML Python projects import NumPy. This is followed by SciPy and Pandas.

Jul
2019

*NumPy v1.17.0* is released. This release supports Python 3.5-3.7 but **drops support for Python 2.7**. In fact, NumPy v1.16.x is the last series to support Python 2.7 but being a long term release, v1.16.x will be maintained till 2020. NumPy v1.16.6 is released in December 2019.

Feb
2020

Following the end of life of Python 2 in January 2020, the number of downloads for older NumPy releases based on Python 2 falls sharply. By April 2020, 80% of NumPy downloads are based on Python 3.

NumPy


# Chomsky Hierarchy

## Summary

Any language is a structured medium of communication whether it is a spoken or written natural language, sign or coded language, or a formal programming language. Languages are characterised by two basic elements – syntax (grammatical rules) and semantics (meaning). In some languages, the meaning might vary depending upon a third factor called context of usage.

Depending on restrictions and complexity present in the grammar, languages find a place in the hierarchy of formal languages. **Noam Chomsky**, celebrated American linguist cum cognitive scientist, defined this hierarchy in 1956 and hence it's called **Chomsky Hierarchy**.

Although his concept is quite old, there's renewed interest because of its relevance to Natural Language Processing. Chomsky hierarchy helps us answer questions like “Can a natural language like English be described (‘parsed’, ‘compiled’) with the same methods as used for formal/artificial (programming) languages in computer science?”

## Discussion

### What are the different levels in the Chomsky hierarchy?

There are 4 levels – Type-3, Type-2, Type-1, Type-0. With every level, the grammar becomes less restrictive in rules, but more complicated to automate. Every level is also a subset of the subsequent level.

    + **Type-3: Regular Grammar** - most restrictive of the set, they generate regular languages. They must have a single non-terminal on the left-hand-side and a right-hand-side consisting of a single terminal or single terminal followed by a single non-terminal.
    + **Type-2: Context-Free Grammar** - generate context-free languages, a category of immense interest to NLP practitioners. Here all rules take the form A → β, where A is a single non-terminal symbol and β is a string of symbols.
    + **Type-1: Context-Sensitive Grammar** - the highest programmable level, they generate context-sensitive languages. They have rules of the form α A β → α γ β with A as a non-terminal and α, β, γ as strings of terminals and non-terminals. Strings α, β may be empty, but γ must be nonempty.
    + **Type-0: Recursively enumerable grammar** - are too generic and unrestricted to describe the syntax of either programming or natural languages.

### What are the common terms and definitions used while studying Chomsky Hierarchy?

    + **Symbol** - Letters, digits, single characters. Example - A,b,3
    + **String** - Finite sequence of symbols. Example - Abcd, x12
    + **Production Rules** - Set of rules for every grammar describing how to form strings from the language that are syntactically valid.
    + **Terminal** - Smallest unit of a grammar that appears in production rules, cannot be further broken down.
    + **Non-terminal** - Symbols that can be replaced by other non-terminals or terminals by successive application of production rules.
    + **Grammar** - Rules for forming well-structured sentences and the words that make up those sentences in a language. A 4-tuple **G = (V , T , P , S)** such that V = Finite non-empty set of non-terminal symbols, T = Finite set of terminal symbols, P = Finite non-empty set of production rules, S = Start symbol
    + **Language** - Set of strings conforming to a grammar. Programming languages have finite strings, most natural languages are seemingly infinite. Example – Spanish, Python, Hexadecimal code.
    + **Automaton** - Programmable version of a grammar governed by pre-defined production rules. It has clearly set computing requirements of memory and processing. Example – Regular automaton for regex.

### What are the corresponding language characteristics in each level?

Under **Type-3** grammar, we don't classify entire languages as the production rules are restrictive. However, constructs describable by **regular expressions** come under this type.

For instance, **rule for naming an identifier in a programming language** – regular expression with any combination of case-insensitive letters, some special characters and numbers, but must start with a letter.

**Context-free** languages classified as **Type-2** are capable of handling an important language construct called **nested dependencies**. English example – Recursive presence of “If <phrase> then <phrase>” – “If it rains today and if I don’t carry an umbrella, then I'd get drenched”. For programming languages, the matching parentheses of functions or loops get covered by this grammar.

In **Type-1** languages, placing the restriction on productions α → β of a phrase structure that β be at least as long as α, they become context sensitive. They permit replacement of α by β only in a ‘context’, [context] α [context] → [context] β [context].

Finally, **Type-0** languages have **no restrictions** on their grammar and may loop forever. They don’t have an algorithm enumerating all the elements.

### What are the type of Automaton that recognizes the grammar in each level?

    + **Type-3: Finite-State Automata** - To compute constructs for a regular language, the most important consideration is that there is no memory requirement. Think of a single purpose vending machine for platform tickets or a lift algorithm. The automaton knows the present state and next permissible states, but does not ‘remember’ past steps.
    + **Type-2: Push-Down Automata** - In order to match nested dependencies, this automaton requires a one-ended memory stack. For instance, to match the number of ‘if’ and ‘else’ phrases, the automaton needs to ‘remember’ the latest occurring ‘if’. Only then it can find the corresponding ‘else’.
    + **Type-1: Linear-Bounded Automata** - is a form of a restricted Turing machine which instead of being unlimited, is bounded by some computable linear function. The advantage of this automaton is that its memory requirement (RAM upper limit) is predictable even if the execution is recursive in parts.
    + **Type-0: Turing Machine** - Non-computable functions exist in Mathematics and Computer Science. The Turing machine however allows representing even such functions as a sequence of discrete steps. Control is finite even if data might be seemingly infinite.

### Can you give a quick example for each type of grammar/language?

    + Type-3: Regex to define tokens such as identifiers, language keywords in programming languages. A coin vending machine that accepts only 1-Rupee, 2-Rupee and 5-Rupee coins has a regular language with only three words – 1, 2, 5.
    + Type-2: Statement blocks in programming languages such as functions in parentheses, If-Else, for loops. In natural language, nouns and their plurals can be recognized through one NFA, verbs and their different forms can be recognized through another NFA, and then combined. Singular (The girl runs home –> Girl + Runs). Plural (The girls run home –> Girls + Run)
    + Type-1: Though most language constructs in natural language are context-free, in some situations linear matching of tokens has to be done, such as - "The square roots of 16, 9 and 4 are 4, 3 and 2, respectively." Here 16 is to be matched with 4, 9 is matched with 3, and 4 is matched with 2.
    + Type-0: A language with no restrictions is not conducive to communication or automation. Hence there are no common examples for this type. However, some mathematical seemingly unsolvable equations are expressed in this form.

### In which level of the hierarchy do formal programming languages fall?

Reading a text file containing a high-level language program and compiling it as per its syntax is done in two steps.

Finite state models associated with **Type-3 grammar** are used for performing the first step of **lexical analysis**. Raw text is aggregated into keywords, strings, numerical constants and identifiers in this step.

In the second step, to **parse the program constructs** of any high level language according to its syntax, a **Context-Free Grammar** is required. Usually these grammars are specified in *Backus-Naur Form (BNF)*.

For example, to build a grammar for IF statement, grammar would begin with a non-terminal statement S. Rules will be of the form:

S → IF-STATEMENT

IF-STATEMENT → if CONDITION then BLOCK endif

BLOCK → STATEMENT | BLOCK;

Conventionally, all high-level programming languages can be covered under the Type-2 grammar in Chomsky’s hierarchy.

Python language has a unique feature of being white-space sensitive. To make this feature fit into a conventional CFG, Python uses two additional tokens ‘INDENT’ and ‘DEDENT’ to represent the line indentations.

However, just syntactic analysis does not guarantee that the language will be entirely ‘understood’. Semantics need to match too.

### Where can we place natural languages in the hierarchy?

Natural languages are an infinite set of sentences constructed out of a finite set of characters. Words in a sentence don’t have defined upper limits either. When natural languages are reverse engineered into their component parts, they get broken down into four parts - **syntax, semantics, morphology, phonology**.

Tokenising words and identifying nested dependencies work as explained in the previous section.

**Part-of-Speech Tagging** is a challenge. “He runs 20 miles every day” and “The batsman scored 150 runs in one day” – the same word ‘runs’ becomes a noun and verb. Finite state grammars can be used for resolving such lexical ambiguity.

**Identifying cases** (subjective - I, possessive - Mine, objective - Me, etc) for nouns varies across languages: Old English (5), Modern English (3), Sanskrit and Tamil (8). Each case also has interrogative forms. Clear definition of cases enables free word order. The CFG defined for these languages take care of this.

Natural languages are believed to be at least context-free. However, Dutch and Swiss German contain grammatical constructions with **cross-serial dependencies** which make them context sensitive.

Languages having clear and singular source text of grammar are easier to classify.

### Are there any exceptional cases in natural languages that make its classification ambiguous?

NLP practitioners have successfully managed to assign a majority of natural language aspects to the regular and CFG category. However, some aspects don't easily conform to a particular grammar and require special handling.

    + **Structural ambiguity** – Example ‘I saw the man with the telescope’. A CFG can assign two or more phrase structures (“parse trees”) to one and the same sequence of terminal symbols (words or word classes).
    + **Ungrammatical speech** – Humans often talk in sentences that are incorrect grammatically. Missing words sometimes are implied in a sentence, not uttered explicitly. So decoding such sentences is a huge challenge as they don't qualify as per any defined grammar, but a native speaker can easily understand them.
    + **Sarcasm or proverb usage** – When we say something but mean something entirely different. Here the semantic analysis becomes critical. We don’t build grammars for these cases, we just prepare an exhaustive reference data set.
    + **Mixed language use** – Humans often mix words from multiple languages. So computing systems need to identify all the constituent language words present in the sentence and then assign them to their respective grammars.

### What are the important extensions to Chomsky hierarchy that find relevance in NLP?

There are two extensions to the traditional Chomsky hierarchy that have proved useful in linguistics and cognitive science:

    + **Mildly context-sensitive languages** - CFGs are not adequate (weakly or strongly) to characterize some aspects of language structure. To derive extra power beyond CFG, a grammatical formalism called Tree Adjoining Grammars (TAG) was proposed as an approximate characterization of Mildly Context-Sensitive Grammars. It is a tree generating system that factors recursion and the domain of dependencies in a novel way leading to 'localization' of dependencies, their long distance behaviour following from the operation of composition, called 'adjoining'. Another classification called Minimalist Grammars (MG) describes an even larger class of formal languages.
    + **Sub-regular languages** - A sub-regular language is a set of strings that can be described without employing the full power of finite state automata. Many aspects of human language are manifestly sub-regular, such as some ‘strictly local’ dependencies. Example – identifying recurring sub-string patterns within words is one such common application.

## Milestones

1928

Avram Noam Chomsky is born on December 7, 1928. Decades later, Chomsky is credited with the creation of the theory of generative grammar, considered to be one of the most significant contributions to the field of linguistics made in the 20th Century.

1936

Turing machines, first described by Alan Turing in 1936–7, are simple abstract computational devices intended to help investigate the extent and limitations of what can be computed.

1956

Chomsky publishes *Syntactic Structures*. He defines a classification of formal languages in terms of their generative power, to be known as the Chomsky hierarchy.

1963

John Backus and Peter Naur introduce for the first time a formal notation to describe the syntax of a given language (for ALGOL 60 programming language). This is said to be influenced from Chomsky's work. In time, this notation is called **Backus-Naur Form**.

Chomsky Hierarchy


# Decibel

## Summary

Decibel is a unit of measurement that expresses the logarithmic ratio of two physical quantities of the same dimensions. The logarithm is to base 10. This logscale definition is useful when the quantities have a wide range and losses or gains are proportional.

Decibel is dimensionless since it's a ratio. It's a relative measure. However, when a reference is defined, it can be used as an absolute measure. As a relative measure, it's represented as **dB**. As an absolute measure, often an additional suffix is appended to "dB".

Decibel originated in telephone networks of the early 20th century. Today it's commonly used in many domains including acoustics, electrical engineering, signal processing, RF power, and more.

## Discussion

### Could you explain the decibel scale with some numbers?

Decibel is defined as \(10\ log\_{10}(P\_1/P\_0)\), where power level P1 is compared against P0. When P0 and P1 are equal, we say their ratio is 0dB. Thus, 0dB doesn't imply zero power or intensity. It implies equal power.

If P1 is twice the value of P0, we say that P1 is 3dB higher than P0. If P1 is half the value of P0, we say that P1 is 3dB lower than P0, or equivalently -3dB.

For ratios P1/P0 equalling 10, 100, and 1000, the respective decibels are 10, 20, and 30. Likewise, if the ratios are 0.1, 0.01, and 0.001, the respective decibels are -10, -20, and -30. Thus, we can see that even though the power levels are changing geometrically, the equivalent decibel values change linearly. This is why the logarithmic scale of decibel is useful when quantifying values that have a large range.

### What are some examples where the decibel scale is used?

Decibel is used to quantify the intensity of sound. For sound, it's common to use decibel with a reference of 0.02 mPa (millipascals), the minimum sound that the human ear can hear. Normal speech is at 60dB. A jet engine is at 120dB, which is a million times louder than normal speech.

In electrical engineering and radio engineering, decibel is used. For example, the gain of an amplifier or the loss of signal power due to an obstruction are quantified in decibels. A directional antenna will focus its radiation in specific directions, which are specified on a chart with decibel as the unit.

In signal processing, it's common to apply filters to signals. *Bode plot* is an example that shows the magnitude response of such a filter: decibels (y-axis) vs frequency (x-axis). This tells which frequencies are allowed to pass and which ones are attenuated.

When a signal is compared to noise, called *Signal-to-Noise Ratio (SNR)*, this ratio is specified in decibels. For example, we can compare a processed image with the original and quantify the difference as SNR in decibels.

### What's the difference between decibels based on either root-power level or power level?

Decibel is commonly defined in terms of power levels. However, it can be defined in terms of signal or field level. For example, in electrical engineering, electrical signals are voltage and current. Power is proportional to square of these signals. When decibel is therefore defined in terms of voltage, the equation becomes \(20\ log\_{10}(V\_1/V\_0)\), where voltage level V1 is compared against V0.

While the term *field quantity* was previously used, ISO Standard 80000-1:2009 introduced the term *root-power quantity*.

### What are absolute and relative decibel units?

Decibel is a relative measure that compares two quantities of the same dimension. But in acoustics, it's common to say that noise was at 90dB. We don't say noise was 90dB with respect to something. This is because a reference of 0.02 mPa (millipascals) is implied since it's the quietest sound that we can hear. Thus, sound levels are in absolute decibels.

In most other applications, **dB** is a relative measure. When we say that path loss of a wireless channel is 120dB, we mean that from the transmitter to receiver, signal power drops by 120dB. However, if we take 1 milliwatt as the reference, then we can use the unit **dBm** to indicate absolute measure. If transmitter sends a 10 kilowatt signal, it's sending a 70dBm signal. If path loss is 120dB (relative measure), the receiver will get a -50dBm (absolute measure) signal.

Here are some examples of absolute measures (reference in parenthesis): dBW (1 watt), dBi (isotropic antenna), dBV (1 volt), dBµV (1 microvolt), dBµV/m or dBu (1µV/m electrical field strength), dBA ("A" weighted pressure levels), and more.

### What are some limitations of using decibels?

In image processing, Signal-to-Noise Ratio (SNR) and Peak Signal-to-Noise Ratio (PSNR) are sometimes used to compare a processed image with the original image. SNR and PSNR are expressed in decibels. The problem is that image quality is a matter of human perception. We may sometimes perceive one image as better than another even though it has a lower SNR. Admittedly, this is not a limitation of decibel itself but rather a limitation of its usage for SNR.

There have been suggestions that decibel is outdated for the modern era. This is probably due to incomplete understanding of logarithms and incorrectly mixing absolute and relative values.

## Milestones

1614

Scottish mathematician John Napier (aka Neper) publishes a book detailing his invention of the **logarithms**. This would prove useful centuries later in the definition of the decibel.

1904

In the early days of telephony, there's a need to quantify transmission losses due to links and nodes in a connection. AT&T proposes to use **Mile of Standard Cable (MSC)**, which is equivalent of a mile of dry-core cable-pair with a loop resistance of 88 ohms and a mutual capacitance of 0.054 farads. Let's note that MSC is not a measure of distance, but one of loss, useful for comparing the efficiencies of two telephone circuits.

1923

MSC unit has dependency on frequency. This was useful in earlier decades when there was a need to characterize distortion. With newer circuits having much less distortion that the standard circuit, there's a need for a distortionless unit.  Thus, AT&T invents the **Transmission Unit (TU)** to replace MSC. TU is defined based on the logarithmic scale. The logscale is useful since the losses due to two successive parts can simply be added instead of multiplied.

1924

The International Advisory Committee on Long Distance Telephony in Europe, plus representatives of the Bell System, decide to adopt two units based on TU: **bel** based on power ratio \(10^1\) and **neper** based on power ratio \(e^2\). Since TU is based on power ratio \(10^.1\), the Bell System decides to use **decibel** as the unit. One decibel is the smallest difference in sound that the human ear can detect. The word "bel" itself is honouring Alexander Graham Bell, inventor of the telephone.

1933

By this time, in the UK at least, the unit of loss measurement becomes decibel rather than msc, though some other countries continue to use neper as the unit.

Jul
1937

At the First International Acoustical Conference in Paris, decibel is adopted as an international unit for energy and pressure levels.

Apr
2003

There's talk of including the decibel within the International System of Units (SI) but this proposal is rejected. However, decibel has been recognized by IEC and ISO.

Decibel


# TLV Format

## Summary

TLV (Tag-Length-Value) is a binary format used to represent data in a structured way. TLV is commonly used in computer networking protocols, smart card applications, and other data exchange scenarios. The three parts of TLV are:

* **Tag**: Identifies uniquely the type of data. It's typically a single byte or a small sequence of bytes.
* **Length**: Length of the data field in bytes. In some protocols, the lengths of tag and length fields are also included.
* **Value**: Actual data being transmitted, which can be of any type or format.

Entities that send messages would **encode** information into TLV format. Entities that receive such messages would **decode** them to retrieve the information. Many programming languages have libraries for TLV encoding and decoding. Developers can also build their own custom encoders and decoders, perhaps optimized for their applications.

## Discussion

### Could you explain the TLV format with an example?

Developers will usually represent data in a form that's most convenient and efficient for processing. For example, this could be an associative array, a linked list, or a class with attributes. When this data needs to be stored or transmitted, it has to be **serialized**. This is where TLV format is used. A TLV encoder reads the data/message and outputs a stream of bytes. A TLV decoder does the reverse.

The figure shows a TLV example used for F-TEID, an information element (IE) used in 5G's PFCP protocol. The type value 21 indicates that this is F-TEID. The protocol defines other values for other IEs. Since type field is two bytes, it will be encoded as 0x0015.

The length field is 2 bytes. It's value indicates the number of bytes that follow. The latter are part of the 'V' in TLV. The 4-byte TEID field is mandatory. Byte 5 contains some flags (CHID, CH, V6, V4) to indicate the presence of optional fields. So if V6 is set to 1, 16 bytes of IPv6 address is present.

### Does 'T' in TLV refer to "tag" or "type"?

The TLV acronym can refer to either "Tag-Length-Value" or "Type-Length-Value." Both of these terms are used interchangeably and can be considered correct.

In some contexts, "Tag" and "Type" may be used to refer to slightly different things. "Tag" may refer to an identifier used within a particular protocol. "Type" may refer to a more general data type, such as an integer, string, or binary data. However, in most cases, the terms "Tag" and "Type" are used interchangeably to refer to the identifier of the data being transmitted.

### What are the benefits of using the TLV format?

The TLV format allows data to be structured in a **flexible** way. Data can be organized into logical groups. The length field allows for variable-length data to be represented.

The format is also **extensible**, meaning that new tags can be added to the format without requiring changes to existing code. This makes it easy to add new functionality to an existing system.

It's also an **efficient** way to store and transmit data. By using a binary format, the data can be transmitted more quickly and with less overhead than with text-based formats.

The use of length fields in the TLV format makes it easy to **detect errors** in the data. If the length of a field does not match the expected value, it is likely that an error has occurred.

### Which standards have adopted the TLV format?

The OSI (Open Systems Interconnection) reference model is a layered architecture. TLV is used in many protocols across the OSI layers. For example, at the data link layer, Ethernet frames and Wi-Fi frames use TLV. At the network layer, IP and ICMP are two examples that use TLV. At the application layer, there are plenty of protocols that use TLV: HTTP, CoAP, DNS, and MQTT are some examples.

TLV is typically not used at the physical layer, which usually deals with raw bits. TLV is typically not used at the transport, session or presentation layers.

Here are more standards that use TLV:

    + ISO 7816: This is a communication protocol between smart cards and card readers. The APDU (Application Protocol Data Unit) format is based on TLV.
    + Bluetooth: The Bluetooth Low Energy (BLE) specification uses the TLV format to encode the data for advertising and communication between BLE devices.
    + SIM/eSIM cards: In cellular systems, SIM and eSIM cards use the TLV format to store data, and exchange data with the mobile device.

### What endianness is used in TLV?

The endianness used depends on the specific protocol or application, which must specify the endianness it's using. If messages are being exchanged between devices with different endianness, proper conversion would be needed before processing those messages.

There are some protocols that mix endianness for different fields within the same TLV message. One such protocol is the Bluetooth Low Energy (BLE) protocol. The endianness of the length field and the value field may be different. Specifically, the length field in the TLV message is always encoded in little-endian byte order, while the endianness of the value field depends on the type of data being transmitted. For example, if the value field contains a 16-bit unsigned integer, it's encoded in little-endian byte order, since the length field is also in little-endian byte order. However, if the value field contains a 32-bit floating-point value, it's encoded in big-endian byte order.

### What are some best practices for designing TLV-based messages?

Use standardized tags to ensure that your messages can be easily understood and implemented by other systems. Consider using a standardized byte order or including byte order information in the message.

Define a clear message structure that defines the order and the type of fields. When possible, use fixed-length fields. These approaches make parsing and processing more efficient. Where variable-length fields are needed, use a length field to indicate the size of the data. Include error checking in the message to ensure that the message is valid and has not been corrupted during transmission.

When designing a TLV message, reserve certain tags for future use. This can ensure that new fields can be added to the message without requiring changes to existing code. Use flags to indicate which fields in the message are optional. This can help reduce the size of the message and make it easier to parse. Flags also help with backward compatibility. A version number included in a message can help older systems interwork with newer systems. Another technique is to use a a variable-length encoding scheme that can represent both older and newer data formats.

### How should developers encode/decode messages in TLV format?

Any TLV encoder/decoder must be tested for correctness. Even when invalid input is fed to them, they should fail gracefully. They can log a warning message and keep applications robust and secure.

The length value can't be set until all message fields are encoded. One approach is to increment the length value as each field is encoded.

If fields are not byte-aligned, bit manipulation is required. This involves bit shifting and bit masking to get or set specific bits from a field. Here's an example (assuming big-endian byte order):

    + Encoder: Data `dt` is of range 0-3 (2 bits). It's to be encoded into bits 5 and 4 without disturbing other bits in a 2-byte field `fld`. We can do `fld |= (dt & 0x0003) << 3`.
    + Decoder: We wish to extract bits 5 and 4 from a 2-byte field `fld`. We can do `dt = (fld & 0x0018) >> 3` or `dt = (fld >> 3) & 0x0003`.Developers can use a well-tested and widely-used TLV library rather than writing a custom implementation. Examples include libtins and tlvcpp (C++); Apache MINA and TlvParser (Java); Construct and TLV-Coding (Python); BouncyCastle and PeterO.TlvLib (.NET).

### What are some variations of the TLV format?

TLV has some variations and they offer different trade-offs between simplicity, flexibility, and efficiency:

    + **TVL**: The tag comes first, followed by the value, and then the length. This format is sometimes used in legacy systems, but it is less common than the TLV format.
    + **LV**: There's no tag field. This format is simpler than TLV, but it doesn't provide any information about the meaning or purpose of the data.
    + **TFLV**: Type-specific flags field is also included.
    + **Nested TLV**: The value field can contain nested TLV structures, allowing for the representation of more complex data. This is often used in protocols that require hierarchical or nested data structures.
    + **Extended TLV (ETLV)**: This includes an extra "extended" field in the tag that provides additional information about the tag's purpose or context. This allows for more flexibility in the use of tags, as well as better support for backward compatibility.
    + **Binary TLV (BTLV)**: BTLV is a binary variant of TLV that's designed for use in low-level system programming. It uses fixed-width fields for the tag, length, and value, and is often used in embedded systems and low-level network protocols.

### What are the alternatives to the TLV format?

TLV is a binary format. Other binary formats include Protocol Buffers, ASN.1 and MessagePack. Protocol Buffers was developed by Google. It's language- and platform-independent. ASN.1 is an older format that's widely adopted. It has different TLV-based encoding rules (BER, DER) and non-TLV-based encoding rules (PER, XER). MessagePack is designed to be fast and compact. These alternatives allow developers to define and maintain message definitions in readable syntax while encoding them into binary formats.

Sometimes an efficient binary format is not essential. Developers may prefer a textual format that's easier to read and parse. In such situations, XML and JSON formats could be used. These are widely used in web services and mobile applications. XML is also used for document formatting.

## Milestones

1980

The use of TLV encoding can be traced back to the development of the **Abstract Syntax Notation One (ASN.1)** standard in the 1980s. This happens within the telecommunications industry. ASN.1 uses TLV encoding to represent complex data structures in a compact and efficient format.

1990

In the 1990s, TLV is used in the development of the **EMV (Europay, Mastercard, and Visa)** standard for payment cards. The EMV standard uses TLV encoding to represent the data stored on a payment card, including the cardholder's name, card number, expiration date, and other information.

2000

Through the 2000s, the use of TLV encoding becomes more common in computer networking protocols, such as the Simple Network Management Protocol (SNMP) and the Link Layer Discovery Protocol (LLDP).

TLV Format


# Siri

## Summary

Siri is a voice-controlled virtual assistant for Apple devices. It can be used for multiple purposes such as getting weather reports, setting an alarm, sending a message to someone, scheduling a meeting, or locking a car. At times, Siri can also respond with wit and sarcasm. Siri Shortcuts and Siri Suggestions are additional features that enhance how users can interact with Siri.

Siri is available in many countries and in different languages. Initially, it was available only on the iPhone and later expanded to include iPod touch, iPad, Mac, AirPods, Apple Watch, Apple TV, and HomePod.

Despite being the first mover in the industry, Siri has fallen behind the competition. In fact, users may prefer to use Amazon Alexa or Google Assistant on Apple devices.

## Discussion

### What are the capabilities of Siri?

Siri is an artificial intelligence designed to help Apple users in various tasks. It can **perform actions** such as reading your last email, texting your friend, booking a hotel, or calling your parents instantly. It can tell good hotels nearby, set an alarm, give directions to reach a certain place, or describe tomorrow's weather.

By asking Siri "What can you do?", Siri will respond with a list of all things it can do. There are already apps to do these things but Siri brings a voice interface to these apps and therefore a better user experience.

It can also be trained to **give answers** to some specific questions. Siri can be taught to pronounce your name or which of your contacts are your family members.

Siri can also be funny and sarcastic sometimes. For instance, if you ask it "What's your favourite animal?", it will answer "Software doesn't usually get to choose one, but I'll say birds. What's yours?" To the question "What is the meaning of life?", it will answer "I can't answer that. Ha ha!"

### How can users launch Siri?

Each Apple device has its unique way to invoke Siri. Typically, a device button (physical or virtual) is pressed and the request is made. Users need to press and hold for longer requests. **"Hey Siri"** is a hands-free option to invoke Siri. It's available in latest versions of most devices.

On iPhone X or later, press the Side button. In other models, press Home or Top button. With AirPods Pro and AirPods (3rd generation), set the force sensor on either left or right AirPod. On AirPods (1st generation) double-tap the outside of either AirPod and wait for a chime. On recent Apples Watch models, "Hey Siri" prompt is unnecessary. Another method is to press the Digital Crown for a few seconds.

On Macs, press the Siri button on Touch Bar, menu bar or Dock (macOS Sierra or later). On vehicles that support CarPlay or Siri Eyes Free, hold down the voice-command button on the steering wheel while making a request. On HomePod, press the top of the device. For Apple TV, use Siri button on the Siri Remote.

### How can users automate tasks using Siri?

A user may want to automate a sequence of frequent tasks rather than perform them manually. For example, for the way home from work, the user may want directions, send ETA to a family member and start listening to music. Another example is to download all images on a webpage, reduce them in size and upload them to Twitter. These tasks can be automated with **Siri Shortcuts**.

*Shortcuts* is a separate app that's installed by default since iOS 13. It comes with 300+ built-in actions. Users can create or edit shortcuts. Users can browse a gallery of shortcuts and launch any shortcut. Specific shortcuts can also be launched from icons or widgets on the home screen. Or invoke and tell Siri the name of the shortcut to execute.

For third-party apps that support this feature, users can add shortcuts to automate tasks concerning those apps.

Since iOS 13, shortcuts can also be triggered automatically. For example, a shortcut can be configured to run at 11am daily. A practical example is to automatically enable Do Not Disturb when the user starts watching Netflix.

### What are Siri Suggestions?

We may view Siri as just a voice interface to apps on Apple devices but Siri is more than this. Using its AI capabilities, Siri learns how you use your devices and apps. Via a feature called **Siri Suggestions**, it gives personalized suggestions. For this purpose, Siri looks at your browsing history, emails, messages, images, notifications, contacts and information shared by third-party apps on your devices. For privacy, synchronization across devices is done using end-to-end encryption. On-device processing is used. Any information sent to Apple is anonymized.

Here are some things Siri Suggestions can do: suggests people to include in emails and calendar events based on previous emails or events; based on numbers shared in emails, guesses who may be calling even if the number is not in contacts; gives search suggestions in Safari browser; recommends news stories based on your past reading history; notifies when to leave for an appointment based on current traffic conditions.

Siri Suggestions also makes use of Shortcuts. In fact, suggestions can be seen as informing users what shortcuts to run.

### How does Siri work under the hood?

The fundamental technologies used in Siri are Automatic Speech Recognition (ASR) that converts audio waveforms to text, Natural Language Understanding (NLU) that determines user's intent, and Text-to-Speech (TTS) that enables Siri to speak out a response.

Since iOS 15, on some devices, some of these steps can be done offline without connecting to a server. However, the true utility of Siri comes from connecting to a server. Server-side logic might do a better job with ASR, taking into account regional accent, ambient noise, and linguistic information such as syntax and context. For example, the NLU engine must differentiate between "byte" and "bite" based on the context. The server may query databases to find answers. If an answer can't be found, Siri may prompt the user if it should search the web.

The "Hey Siri" hands-free prompt is based on an acoustic model implemented as a Deep Neural Network (DNN). In fact, Siri does two-pass detection: a small DNN that's always on, a larger DNN that's triggered for more accurate processing.

### What patents cover the technologies used in Siri?

Without being exhaustive, we note the following patents along with filing and publication dates:

    + US20120016678A1, Jan 2011 / Jan 2012: Intelligent automated assistant: a conversational agent that uses natural language dialog and external services to perform actions, retrieve information and solve problems rather than simply return search results.
    + US20170358301A1, Sep 2016 / Dec 2017: Digital assistant providing whispered speech: determines that user is whispering and modulates the response to whispers. A patent published April 2021, US20210097980A1, expands this idea to devices that are aware of their environments.
    + US20180329957A1, Aug 2017 / Nov 2018: Feedback analysis of a digital assistant: client device receives user inputs, processes these inputs based on instructions received from a server and sends the result to server.
    + US20180330731A1, Sep 2017 / Nov 2018: Offline personal assistant: an assistant that can receive multiple inputs, rank and score them, and decide which one to act upon. The patent focuses on on-device processing without relying on server backend.
    + US20180329677A1, Mar 2018 / Nov 2018: Multi-modal interfaces: visual interface is used to augment the voice interface.

### What developer tools are available to integrate Siri into apps?

Apple provides developers an API called **SiriKit**. Interactions are enabled using Intents and IntentsUI frameworks that are part of SiriKit. An app can have custom vocabulary and sample phrases so that Siri can interact better with the app. Via an app-specific extension, Siri can be used even when the actual app isn't running. Siri handles user interactions while the extension provides necessary information.

SiriKit is available in Swift and Objective-C programming languages. It's available for iOS 10.0+, iPadOS 10.0+, macOS12.0+, Mac Catalyst 13.0, tvOS 14.0+ and watchOS 3.2+ platforms.

Developers can read SiriKit documentation for more information.

### What are the shortcomings of Siri?

Ten years after its launch in 2011, Siri was considered inferior to its competitors Google Assistant and Amazon Alexa. This is despite Siri being the first mover in the industry. New releases came with only insignificant updates. Siri mishears even simple commands. Users become frustrated by its failures to complete basic tasks. Some of these can be blamed on poor management and lack of product focus.

Apple being a closed ecosystem, Siri doesn't integrate well with third-party apps and services, which is where Alexa trumps. Different versions of Siri on different Apple devices means that users don't get a consistent experience.

In 2019, it was revealed that recordings of real conversations were given to external analysts to improve Siri's algorithms. However, these recordings contained sensitive information.

Without an active internet connection, Siri's capabilities are limited. Back in 2019 it was reported that user interaction with maps was limited to English. Background noise, low-quality audio, spelling variations, strong accent and fast speech all caused problems for Siri. Even if the iPhone is password protected, Siri could be invoked via the Home button without unlocking the phone.

## Milestones

2010

A Norwegian company named Siri, Inc. launches an iOS app based on speech recognition technology. Siri is a virtual assistant. It enables users to interact with their iPhones via a voice interface. In April, **Apple acquires Siri**.

2011

Apple makes Siri an integral feature of **Apple iPhone 4S**. Despite Apple's best efforts to find a better name, they stick to "Siri". Later, people suggest that Siri stands for "Speech Interpretation and Recognition Interface". Competition soon follows: Samsung's S Voice (2012), Google Now (2012), Microsoft Cortana (2014), Amazon Alexa (2014), and Google Assistant (2016).

2012

Apple announces many improvements to Siri in iOS 6, mainly based on access to new sources of data such as information about sports, restaurants and movies. Siri is now capable of posting to social media, reading incoming messages and notifications, and opening apps. It's now available in 15 languages. Apple expands support of Siri to iPad and iPod touch. This encourages Apple users to upgrade their devices.

2013

Siri acquires new capabilities including switching on/off Bluetooth and Wi-Fi, using databases to answer search queries, and using more natural sounding voices. Apple introduces "iOS in the Car", an extension of Siri Hands-free designed to work with in-car systems.

2014

Hands-free interaction with Siri was first introduced in 2012. This is improved with the **"Hey Siri"** feature. Users can use Siri without needing to press buttons or touchscreens. In July, Siri gets a major upgrade with adoption of ML techniques including Deep Neural Networks (DNNs). Error rates drop by at least a factor of two.

2015

Siri is added to **Apple TV**. Standalone Siri remote for Apple TV is also released. It gives suggestions based on previous user behaviour such as recommending news stories, music, and apps. It's now 40% faster and 40% more accurate than before.

2016

Apple releases the **SiriKit API** for developers. This allows developers to integrate Siri with their apps. However, this integration is limited to some apps such as messaging, phone calls, photo search, ride booking, personal payments, workouts in apps and some in-car apps. Siri can do intelligent scheduling, integrate with the QuickType keyboard, react to text conversations and make useful suggestions based on user behaviour.

2017

Developers get support for deploying more Siri features such as to-do lists, notes and payments in SiriKit. Machine Learning becomes deeply embedded into Apple products, including Siri. Apple reveals that Siri is used by over 375 million devices each month in different countries and languages. Siri can also translate from English to Chinese, French, German, Italian, and Spanish.

2018

Introduced at WWDC 2017, Apple's intelligent speaker system called **HomePod** becomes available along with iOS 11.2.5. Users can interact with HomePod via Siri.

2019

**Siri Shortcuts**, though available earlier as a downloadable app, is now installed by default on iOS 13 and iPadOS devices. Shortcuts can also do things automatically without users invoking it explicitly.

2021

With the release of iOS 15, Siri can now be used **offline** though with limited capabilities. This is available on recent iPhones with A12 Bionic processor and iPads. On-device speech recognition is now possible but questions such as "What is the capital of Switzerland?" or "Will it rain later today?" can't be answered without connecting to the server. While offline, Siri can open apps, create timers, make calls, and adjust volume. But it can't add new calendar entries or select downloaded music albums.

Siri


# Hidden Markov Model

## Summary

Consider weather, stock prices, DNA sequence, human speech or words in a sentence. In all these cases, current state is influenced by one or more previous states. Moreover, often we can observe the effect but not the underlying cause that remains hidden from the observer. Hidden Markov Model (HMM) helps us figure out the most probable hidden state given an observation.

In practice, we use a sequence of observations to estimate the sequence of hidden states. In HMM, the next state depends only on the current state. As such, it's good for modelling time series data.

We can classify HMM as a **generative probabilistic model** since a sequence of observed variables is generated by a sequence of hidden states. HMM is also seen as a specific kind of **Bayesian network**.

## Discussion

### Could you explain HMM with an example?

Suppose Bob tells his friend Alice what he did earlier today. Based on this information Alice guesses today's weather at Bob's location. In HMM, we model weather as *states* and Bob's activity as *observations*.

To solve this problem, Alice needs to know three things:

    + **Transition Probabilities**: Probability of moving from one state to another. For example, "If today was sunny, what's the probability that it will rain tomorrow?" If there are N states, this is an NxN matrix.
    + **Emission Probabilities**: Probability of a particular output given a particular state. For example, "What's the chance that Bob is walking if it's raining?" Given a choice of M possible observation symbols, this is an NxM matrix. This is also called output or observation probabilities.
    + **Initial Probabilities**: Probability of being in a state at the start, say, yesterday or ten days ago.Unlike a typical Markov chain, we can't see the states in HMM. However, we can observe the output and then predict the state. Thus, the states are hidden, giving rise to the term "hidden" in the name HMM.

### What types of problems can be solved by HMM?

Let A, B and π denote the transition matrix, observation matrix and initial state distribution respectively. HMM can be represented as λ = (A, B, π). Let observation sequence be O and state sequence be Q.

HMM can be used to solve three types of problems:

    + **Likelihood Problem**: Given O and λ, find the likelihood P(O|λ). How likely is a particular sequence of observations? *Forward algorithm* solves this problem.
    + **Decoding Problem**: Given O and λ, find the best possible Q that explains O. Given the observation sequence, what's the best possible state sequence? *Viterbi algorithm* solves this problem.
    + **Learning Problem**: Given O and Q, learn λ, perhaps by maximizing P(O|λ). What model best maps states to observations? *Baum-Welch algorithm*, also called *forward-backward algorithm*, solves this problem. In the language of machine learning, we can say that O is training data and the number of states N is the model's hyperparameter.

### What are some applications where HMM is useful?

HMM has been applied in many areas including automatic speech recognition, handwriting recognition, gesture recognition, part-of-speech tagging, musical score following, partial discharges and bioinformatics.

In speech recognition, a spectral analysis of speech gives us suitable observations for HMM. States are modelled after phonemes or syllables, or after the average number of observations in a spoken word. Each word gets its own model.

To tag words with their parts of speech, the tags are modelled as hidden states and the words are the observations.

In computer networking, HMMs are used in intrusion detection systems. This has two flavours: anomaly detection in which normal behaviour is modelled; or misuse detection in which a predefined set of attacks is modelled.

In computer vision, HMM has been used to label human activities from skeleton output. Each activity is modelled with a HMM. By linking multiple HMMs on common states, a compound HMM is formed. The purpose is to allow robots to be aware of human activity.

### What are the different types of Hidden Markov Models?

In the typical model, called the **ergodic HMM**, the states of the HMM are fully connected so that we can transition to a state from any other state. **Left-right HMM** is a more constrained model in which state transitions are allowed only from lower indexed states to higher indexed ones. Variations and combinations of these two types are possible, such as having two parallel left-to-right state paths.

HMM started with observations of discrete symbols governed by **discrete** probabilities. If observations are continuous signals, then we would use **continuous** observation density.

There are also domain-specific variations of HMM. For example, in biological sequence analysis, there are at least three types including profile-HMMs, pair-HMMs, and context-sensitive HMMs.

### Could you explain forward algorithm and backward algorithm?

Every state sequence has a probability that it will lead to a given sequence of observations. Given T observations and N states, there are \(N^T\) possible state sequences. Thus, the complexity of calculating the probability of a given sequence of observations is \(O(N^{T}T)\). Both forward and backward algorithms bring down the complexity to \(O(N^{2}T)\) through **dynamic programming**.

In the forward algorithm, we consider the probability of being in a state at the current time step. Then we consider the transition probabilities to calculate the state probabilities for the next step. Thus, at each time step we have considered all state sequences preceding it. The algorithm is more efficient since it reuses calculations from earlier steps. Instead of keeping all path sequences, paths are folded into a forward trellis.

Backward algorithm is similar except that we start from the last time step and calculate in reverse. We're finding the probability that from a given state, the model will generate the output sequence that follows.

A combination of both algorithms, called forward-backward algorithm, is used to solve the learning problem.

### What's the algorithm for solving HMM's decoding problem?

Viterbi algorithm solves HMM's decoding problem. It's similar to the forward algorithm except that instead of summing the probabilities of all paths leading to a state, we retain only one path that gives maximum probability. Thus, at every time step or iteration, given that we have N states, we retain only N paths, the most likely path for each state. For the next iteration, we use the most likely paths of current iteration and repeat the process.

When we reach the end of the sequence, we'll have N most likely paths, each ending in a unique state. We then select the most likely end state. Once this selection is made, we backtrack to read the state sequence, that is, how we got to the end state. This state sequence is now the most likely sequence given our sequence of observations.

### How can we solve the learning problem of HMM?

In HMM's learning problem, we are required to learn the transition (A) and observation (B) probabilities when given a sequence of observations and the vocabulary of hidden states. The **forward-backward algorithm** solves this problem. It's an iterative algorithm. It starts with an initial estimate of the probabilities and improves these estimates with each iteration.

The algorithm consists of two steps:

    + **Expectation or E-step**: We compute the expected state occupancy count and the expected state transition count based on current probabilities A and B.
    + **Maximization or M-step**: We use the expected counts from the E-step to recompute A and B.While this algorithm is unsupervised, in practice, initial conditions are very important. For this reason, often extra information is given to the algorithm. For example, in speech recognition, the HMM structure is set manually and the model is trained to set the initial probabilities.

### Could you describe some tools for doing HMM?

In Python, *hmmlearn* package implements HMM. Three models are available: `hmm.GaussianHMM`, `hmm.GMMHMM` and `hmm.MultinomialHMM`. This package is also part of Scikit-learn but will be removed in v0.17. Stephen Marsland has shared Python code in NumPy and Pandas that implements many essential algorithms for HMM.

In R, *HMM* package implements HMM. It has functions for forward, backward, Viterbi and Baum-Welch algorithms. Another package *depmixS4* implements dependent mixture models that can be used to fit HMM to observed data. R-bloggers has an example use of depmixS4.

## Milestones

1913

Russian mathematician A. A. Markov recognizes that in a sequence of random variables, one variable may not be independent of the previous variable. For example, two successive coin tosses are independent but today's weather might depend on yesterday's weather. He models this as a chain of linked events with probability assigned to each link. This technique later is named **Markov Chain**.

1966

Baum and Petrie at the Institute of Defense Analyses, Princeton, introduce the **Hidden Markov Model (HMM)**, though this name is not used. They state the problem of estimating transition and emission probabilities from observations. They use maximum likelihood estimate.

1967

Andrew Viterbi publishes an algorithm to decode information at the receiver in a communication system. Later named **Viterbi algorithm**, it's directly applicable to the decoding problem in HMM. Vintsyuk first applies this algorithm to speech and language processing in 1968.

1970

The **Baum-Welch algorithm** is proposed to solve the learning problem in HMM. This algorithm is a special case of the Expectation-Maximization (EM) algorithm. However, the name HMM is not used in the paper and mathematicians refer to HMM as "probabilistic functions of Markov chains".

1975

James Baker at CMU applies HMM to speech recognition in the DRAGON speech understanding system. This is one of the earliest engineering applications of HMM. HMM is further applied to speech recognition through the 1970s and 1980s by Jelinek, Bahl and Mercer at IBM.

1989

Lawrence Rabiner publishes a tutorial on HMM covering theory, practice and applications. He notes that HMM originated in mathematics and was not widely read by engineers. Even when it was applied to speech processing in the 1970s, there were no tutorials to help translate theory into practice.

2003

HMM is typically used when the number of states is small but one research team applies it to large scale web traffic analysis. This involves hundreds of states and tens of millions of observations.

Hidden Markov Model


# Cicada Principle

## Summary

Periodical cicadas are insects of North America that mature into adulthood and emerge every 13 or 17 years. They come out together in their billions, thus making it difficult for predators to eat them all. What's interesting is their periods are **prime numbers**. This means that predators can't easily synchronize their own periods to that of the periodical cicadas. Another theory is that prime numbers prevent the emergence of hybrids from the cicadas with different periods.

Inspired by this natural phenomenon, now called Cicada Principle, web frontend designers have sought to use prime numbers to produce variations and patterns in their designs in a more efficient manner. Often CSS containing prime numbers are used to achieve this.

## Discussion

### Could you illustrate how prime numbers are useful in frontend design?

Let's take two prime numbers 3 and 5. If we shade two rows based on these primes, we'll find that the shading will not fall on the same column until the 15th column. This is because 3 and 5 don't have any common factors other than 1. In fact, for this to happen, it's sufficient that the numbers are co-primes.

In the case of the periodical cicadas, the periods are 13 and 17. This means that it's only once in 13x17 = 221 years that the two different species will come out in the same year.

Frontend designers who wish to create pseudorandom elements in their design need not actually generate random numbers or hardcode many variations. The use of large prime numbers will give them sufficient variations because patterns can be discerned only on a larger scale based on the multiples of those prime numbers.

### Where and how can I apply the Cicada Principle in CSS?

**Backgrounds** can be created without having any repeating patterns in them or showing gaps or seams between tiles. We can tile and overlap images of different widths, these widths having prime-numbered pixels. More efficiently, we can implement the same using CSS gradients. Michael Arestad has created interesting backgrounds by using prime numbers for background sizes and positions.

In general, Alex Walker proposed a "stacking order model", where many layers are stacked to create a background. The bottom layer can be small and repetitive since much of it will be obscured by higher layers. The topmost layer should have the largest dimension and also be thinly scattered (largest prime number in the group). It should also preferably not have eye-catching details.

Element **borders**, including border radius, can be varied so that each element of a group gets a different border. These borders can also be animated for a mouse hover event.

We can use the principle for CSS **animations** too by having the durations as prime numbers, scaled if necessary by a suitable factor.

### Could you illustrate an example of creating a CSS background based on Cicada Principle?

Let's create a background by applying Cicada Principle to the `background-size`. By setting, the sizes to either 17px or 37px and then applying `linear-gradient` we get a repetitive pattern, which is not very interesting.

When we combine both 17px and 37px widths, we obtain a more interesting background. However, this still shows some visual tiling. By adding another layer of width 53px, which is also a prime number, we get a background that's closer to what we want.

To create the gradients in the first place, we need to adjust the alpha values. Here too we can make use of prime numbers, with each layer getting a different alpha value for creating the gradients.

## Milestones

Apr
2011

Alex Walker reads about periodical cicadas and gets the idea to use prime numbers for generating pseudorandom background patterns. He uses three images of different prime-numbered widths: 29, 37 and 53 pixels. When these three are overlapped and tiled, the resulting background will not repeat for 29x37x53 = 56,869 pixels. The three images together take up less than 7kB in size.

Jun
2012

Eric Meyer uses CSS gradients for implementing the Cicada Principle for backgrounds. Compared to background images, this reduces requests to server and saves on download bandwidth. He calls these gradients **Cicadients**.

Jan
2015

Lea Verou produces CSS animations using the Cicada Principle.

Sep
2016

Charlotte Jackson use Cicada Principle to create pseudorandom borders around images. She does this by adjusting the CSS `border-radius` using prime numbers in CSS selectors.

Cicada Principle


# Word2vec

## Summary

Word2vec is a set of algorithms to produce word embeddings, which are nothing more than vector representations of words. The idea of word2vec, and word embeddings in general, is to use the context of surrounding words and identify semantically similar words since they're likely to be in the same neighbourhood in vector space.

Word2vec algorithms are based on shallow neural networks. Such a neural network might be optimizing for a well-defined task but the real goal is to produce word embeddings that can be used in NLP tasks.

Word2vec was invented at Google in 2013. Word2vec simplified computation compared to previous word embedding models. Since then, it has been popularly adopted by others for many NLP tasks. Airbnb, Alibaba and Spotify have used it to power recommendation engines.

## Discussion

### What's the key insight that lead to the invention of word2vec?

Before word2vec, a feedforward neural network was used to jointly learn the language model and word embeddings. This network had input, projection, hidden and output layers. The complexity is dominated by the mapping from projection to hidden layers. For N (=10) previous words, D-dimensional vectors (500-2000 dimensions), and hidden layer size H (500-1000), complexity is N x D x H.

A recurrent neural network model removes the projection layer. Hidden layer connects to itself with a time delay. Complexity is now H x H.

Word2vec does away with the non-linear hidden layer that was a bottleneck in earlier models. There's a tradeoff. We lose precise representation but training becomes more efficient. This simpler model is used to learn word vectors. The task of learning a language model is considered separately using these word vectors. Finally, not just past words but also future words are considered for context. When input words are projected, their vectors are averaged at the projection layer, unlike earlier models.

Further simplification of the softmax layer computation, enabled word2vec to be trained on 30 billion words, a scale that was not possible with earlier models.

### What are the main models that are part of word2vec?

Let's use a vocabulary of V words, a context of C words, a dense representation of N-dimensional word vector, an embedding matrix W of dimensions VxN at the input and a context matrix W' of dimensions NxV at the output.

Word2vec has two models for deriving word embeddings:

    + **Continuous Bag-of-Words (CBOW)**: We take words surrounding a given word and try to predict the latter. Each word is a one-hot coded vector. Via an embedding matrix, this is transformed into a N-dimensional vector that's the average of C word vectors. From this vector, we compute probabilities for each word in the vocabulary. Word with highest probability is the predicted word.
    + **Continuous Skip-gram**: We take one word and try to predict words that occur around it. At the output, we try to predict C different words.

### Could you describe the details of how word2vec learns word embeddings?

Word2vec uses a neural network model based on word-context pairs. With each training step, the weights are adjusted with the goal of minimizing the loss function, that is, minimize the error between predicted output and actual output. An iteration uses one word-context pair. Training on the entire input corpus may be considered one training epoch.

Consider the skip-gram model. A sliding window around the current input word is used to predict the words within the window. Once this iteration adjusts the weights, the window slides to the next word in the corpus.

Word2vec is not a deep learning technique. In fact, there are no hidden layers, although it's common to refer to the embedding layer as hidden layer, or projection layer. A typical pipeline involves selecting the vocabulary from a text corpus, sliding the window to select context, performing extra tasks to simplify softmax computation, and iterating through the neural network model.

### Why is the softmax layer of word2vec considered computationally difficult?

The softmax layer treats the problem of selecting the most probable word as a multiclass classification problem. It computes the probability of each word being the actual word. Probabilities of all words should add up to 1. For skip-gram model, it does this for each contextual word.

Consider a vocabulary of K words, and input and output vectors \(v\_w\) and \(v'\_w\) of word w. For skip-gram, softmax function is the probability of an output word given the input word, $$p(w\_O|w\_I) = \frac{e^{{v'\_{w\_O}}^T\,v\_{w\_I}}}{\sum\_{w=1}^{K} e^{{v'\_w}^T\,v\_{w\_I}}}$$

With a vocabulary of hundreds of thousands of words, computing the softmax probability for each word for each iteration is computationally expensive. **Hierarchical Softmax** solves this problem by doing computations on word parts and reusing the results. **Negative Sampling** is an alternative. It selects a few negative samples and computes softmax only for these and the actual outputs. Both these simplify computation without much loss of accuracy.

Sebastian Ruder gives a detailed explanation of different softmax approximation techniques.

### What are other improvements to word2vec?

Word2vec implementation has the ability to select a dynamic window size, uniformly sampled in range [1, k]. This has the effect of giving more weight to closer words. Smaller window sizes lead to similar *interchangeable* words. Larger window sizes lead to similar *related* words.

Word2vec can also ignore rare words. In fact, rare words are discarded before context is set. This increases the effective window size for some words. In addition, we can subsample frequent words with the insight that being frequent, they are less informative. The net effect of this is that words that are far away could be topically similar and therefore captured in the embeddings.

### What are some tips for those trying to use word2vec?

Developers can read sample TensorFlow code for the CBOW model, sample NumPy code, or sample Gensim code.

Designed by Xin Rong, **wevi** is a useful tool to visualize how word2vec learns word embeddings.

Sebastian Ruder gives a number of tips. Use Skip-Gram Negative Sampling (SGNS) as a baseline. Use many negative samples for better results. Use context distribution smoothing before selecting negative samples so that frequent words are not sampled quite so frequently. SGNS is a better technique than CBOW.

## Milestones

2005

Morin and Bengio come up with the idea of **hierarchical softmax**. A word is modelled as a composition of inner units, which are then arranged as a binary tree. Given a vocabulary of V words, probability of an output word is computed from softmax computation of inner units that lead to the word from the root of the tree. This reduces complexity from O(V) to O(log(V)). This idea becomes important later in word2vec models. In 2009, Mnih and Hinton explore different ways to construct the tree.

2012

Gutmann and Hyvarinen introduce **Noise Contrastive Estimation (NCE)** as an alternative to hierarchical softmax. The basic idea is that a good model can differentiate data from noise using logistic regression. Mnih and Teh apply NCE to language modelling. This is similar to hinge loss proposed by Collobert and Weston in 2008 to rank data above noise.

Jan
2013

At Google, Mikolov et al. develop **word2vec** although this name refers to a software implementation rather than the models. They propose two models: continuous bag-of-words and continuous skip-gram. They improve on earlier state-of-the-art models by removing the hidden layer. They also make use of hierarchical softmax, thus making this a **log-linear model**. Softmax uses Huffman binary tree to represent the vocabulary. They note that this speeds up evaluation by 2X.

Oct
2013

Mikolov et al. improve on their earlier models by proposing **negative sampling**, which is a simplification of NCE. This is possible because NCE tries to maximize the log probability of the softmax whereas we are more interested in the word embeddings. Negative sampling is simpler and faster than hierarchical softmax. For small datasets, 5-20 negative samples may be required. For large datasets, 2-5 negative samples may be enough.

Jan
2019

Inspired by word2vec, some researchers produce **code embeddings**, vector representations of snippets of software code. Called **code2vec**, this work could enable us to apply neural networks to programming tasks such as automated code reviews and API discovery. This is just one example of many advances due to word2vec. Another example is **doc2vec** from 2014.

Jun
2019

Word2vec is sequential due to strong dependencies across word-context pairs. Researchers show how word2vec can be trained on a GPU cluster by reducing dependency within a large training batch. Without loss of accuracy, they achieve 7.5 times acceleration using 16 GPUs. They also note that using Chainer framework, it's easy to implement CNN-based subword-level models.

Word2vec


# BERT (Language Model)

## Summary

NLP involves a number of distinct tasks each of which typically needs its own set of training data. Often each task has only a few thousand samples of labelled data, which is not adequate to train a good model. However, there's plenty of unlabelled data readily available online. This data can be used to train a baseline model that can be reused across NLP tasks. **Bidirectional Encoder Representations from Transformers (BERT)** is one such model.

BERT is **pre-trained** using unlabelled data on language modelling tasks. For specific NLP tasks, the pretrained model can be **fine-tuned** for that task. Pre-trained BERT models, and their variants, have been open sourced. This makes it easier for NLP researchers to fine-tune BERT and quickly advance the state of the art for their tasks.

## Discussion

### What's the typical process for using BERT?

BERT is an evolution of self-attention and transformer architecture that's becoming popular for neural network models. BERT is an encoder-only transformer. It's **deeply bidirectional**, meaning that it uses both left and right contexts in all layers.

BERT involves two stages: **unsupervised pre-training** followed by **supervised task-specific fine-tuning**. Once a BERT model is pre-trained, it can be shared. This enables downstream tasks to do further training on a much smaller dataset. Different NLP tasks can thus benefit from a single shared baseline model. In some sense, this is similar to transfer learning that's been common in computer vision.

While pre-training takes a few days on many Cloud TPUs, fine-tuning takes only 30 minutes on a single Cloud TPU.

For fine-tuning, one or more output layers are typically added to BERT. Likewise, input embeddings reflect the task. For question answering, an input sequence will contain the question and the answer while the model is trained to learn the start and end of answers. For classification, the `[CLS]` token at the output is fed into a classification layer.

### Could you describe the tasks on which BERT is pre-trained?

BERT is pre-trained on two tasks:

    + **Masked Language Model (MLM)**: Given a sequence of tokens, some of them are masked. The objective is then to predict the masked tokens. Masking allows the model to be trained using both left and right contexts. Specifically, 15% of tokens are randomly chosen for masking. Of these, 80% are masked, 10% are replaced with a random word, 10% are retained.
    + **Next Sentence Prediction (NSP)**: Given two sentences, the model predicts if the second one logically follows the first one. This task is used for capturing relationship between sentences since language modelling doesn't do this.Unlike word embeddings such as word2vec or GloVe, BERT produces contextualized embeddings. This means that BERT produces multiple embeddings of a word, each representing the context around the word. For example, word2vec embedding for the word 'bank' would not differentiate between the phrases "bank account" and "bank of the river" but BERT can tell the difference.

### Which are some possible applications of BERT?

In October 2019, Google Search started using BERT to better understand the intent behind search queries. Another application of BERT is to recommend products based on a descriptive user request. Use of BERT for question answering on SQuAD and NQ datasets is well known. BERT has also been used for document retrieval.

BERT has been used for aspect-based sentiment analysis. Xu et al. use BERT for both sentiment analysis and comprehending product reviews so that questions on those products can be answered automatically.

Among classification tasks, BERT has been used for fake news classification and sentence pair classification.

To aid teachers, BERT has been used to generate questions on grammar or vocabulary based on a news article. The model frames a question and presents some choices, only one of which is correct.

BERT is still new and many novel applications might happen in future. It's possible to use BERT for quantitative trading. BERT can be applied to specific domains but we would need domain-specific pre-trained models. SciBERT and BioBERT are two examples.

### Which are the essential parameters or technical details of BERT model?

BERT pre-trained models are available in two sizes:

    + **Base**: 12 layers, 768 hidden size, 12 self-attention heads, 110M parameters.
    + **Large**: 24 layers, 1024 hidden size, 16 self-attention heads, 340M parameters.Each of the above took 4 days to train on 4 Cloud TPUs (Base) or 16 Cloud TPUs (Large).

For pre-training, a batch size of 256 sequences was used. Each sequence contained 512 tokens, implying 128K tokens per batch. The corpus for pre-training BERT had 3.3 billion words: 800M from BooksCorpus and 2500M from Wikipedia. This resulted in 40 epochs for 1M training steps. Dropout of 0.1 was used on all layers. GELU activation was used.

For fine-tuning, batch sizes of 16 or 32 are recommended. Only 2-4 epochs are needed for fine-tuning. Learning rate is also different from what's used for pre-training. Learning rate is also task specific. Dropout used was same as in pre-training.

### How do I represent the input to BERT?

BERT input embeddings is a sum of three parts:

    + **Token**: Tokens are basically words. BERT uses a fixed vocabulary of about 30K tokens. To handle rare words or those not in token vocabulary, they're broken into sub-words and then mapped to tokens. The first token of a sequence is `[CLS]` that's useful for classification tasks. During MLM pre-training, some tokens are masked.
    + **Segment/Sentence**: An input sequence of tokens can be a single segment or two segments. A segment is a contiguous span of text, not an actual linguistic sentence. Since two segments are packed into the same sequence, each segment has its own embedding. Each segment is terminated by `[SEP]` token. For example in question answering, question is the first segment and answer is the second.
    + **Position**: This represents the token's position within the sequence.In practice, input embeddings can also contain an **input mask**. Since sequence length is fixed, the final sequence may involve padding. Input mask is used to differentiate between actual inputs and padding.

Beginners may wish to look at a visual explanation of BERT input embeddings.

### What are some variants of BERT?

BERT has inspired many variants: RoBERTa, XLNet, MT-DNN, SpanBERT, VisualBERT, K-BERT, HUBERT, and more. Some variants attempt to compress the model: TinyBERT, ALERT, DistilBERT, and more. We describe a few of the variants that outperform BERT in many tasks:

    + **RoBERTa**: Showed that the original BERT was undertrained. RoBERTa is trained longer, on more data; with bigger batches and longer sequences; without NSP; and dynamically changes the masking pattern.
    + **ALBERT**: Uses parameter reduction techniques to yield a smaller model. To utilize inter-sentence coherence, ALBERT uses Sentence-Order Prediction (SOP) instead of NSP.
    + **XLNet**: Doesn't do masking but uses permutation to capture bidirectional context. It combines the best of denoising autoencoding of BERT and autoregressive language modelling of Transformer-XL.
    + **MT-DNN**: Uses BERT with additional multi-task training on NLU tasks. Cross-task data leads to regularization and more general representations.

### Could you share some resources for developers to learn BERT?

Developers can study the TensorFlow code for BERT. This follows the main paper by Devlin et al. (2019). This is also the source for downloading BERT pre-trained models.

Google has shared TensorFlow code that fine-tunes BERT for Natural Questions.

McCormick and Ryan show how to fine-tune BERT in PyTorch. HuggingFace provides `transformers` Python package with implementations of BERT (and alternative models) in both PyTorch and TensorFlow. They also provide a script to convert a TensorFlow checkpoint to PyTorch.

IBM has shared a deployable BERT model for question answering. An online demo of BERT is available from Pragnakalp Techlabs.

## Milestones

Jun
2017

Vaswani et al. propose the **transformer** model in which they use a seq2seq model without RNN. The transformer model relies only on **self-attention**, although they're not the first to use self-attention. Self-attention is about attending to different tokens of the sequence. This would later prove to be the building block on which BERT is created.

Feb
2018

Peters et al. use many layers of bidirectional LSTM trained on a language model objective. The final embeddings are based on all the hidden layers. Thus, their embeddings are deeply contextual. They call it **Embeddings from Language Models (ELMo)**. They show that higher-level LSTM states capture semantics while lower-level states capture syntax.

Oct
2018

Devlin et al. from Google publish on arXiv a paper titled *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*.

Nov
2018

Google **open sources pre-trained BERT models**, along with TensorFlow code that does this pre-training. These models are for English. Later in the month, Google releases **multilingual BERT** that supports about 100 different languages. The multilingual model preserves case. The model for Chinese is separate. It uses character-level tokenization.

Apr
2019

NAACL announces the **best long paper award** to the BERT paper by Devlin et al. The annual NAACL conference itself is held in June.

May
2019

**Whole Word Masking** is introduced in BERT. This can be enabled with the option `--do_whole_word_mask=True` during data generation. For example, when the word 'philammon' is split into sub-tokens 'phil', '##am' and '##mon', then either all three are masked or none at all. Overall masking rate is not affected. As before, each sub-token is predicted independent of the others.

Aug
2019

Since a BERT model has 12 or 24 layers with multi-head attentions, using it in a real-time application is often a challenge. To make this practical for applications such conversational AI, NVIDIA releases **TensorRT optimizations** for BERT. In particular, the transformer layer has been optimized. Q, K and V are fused into a single tensor, thus locating them together in memory and improving model throughput. Latency is 2.2ms on T4 GPUs, well below the 10ms acceptable latency budget.

Oct
2019

**Google Search** starts using BERT for 10% of English queries. Since BERT looks at the bidirectional context of words, it helps in understanding the intent behind search queries. Particularly for conversational queries, prepositions such as "for" and "to" matter. BERT's bidirectional self-attention mechanism takes these into account. Because of the model's complexity, for the first time, Search uses Cloud TPUs.

BERT (Language Model)