# Knowledge Distillation

## Summary

Deep learning is being used in a plethora of applications ranging from Computer Vision and Digital Assistants to Healthcare and Finance. The popularity of the fields of Machine Learning and Deep Learning can be attributed to the high accuracy of the obtained results, which is largely due to the average of an ensemble of thousands of models. However, such computationally intensive models cannot be deployed on mobile devices, or FPGAs for instant use. These devices have constraints on resources like limited memory and input/output ports.

One way of mitigating this problem is to use Knowledge Distillation. We train an ensemble of models or a complex model ('teacher') on the data. We then train a lighter model ('student') with the help of the complex model. The less-intensive student model can then be deployed on FPGAs.

## Discussion

### What is a teacher-student network?

The best Machine Learning models are those that average the predictions of an ensemble of thousands of models. While deploying on hardware devices like FPGAs, however, problems ensue. FPGAs have a limited number of I/O ports, which forces developers to drastically reduce the number of inputs and outputs at each layer of their network.

To alleviate this problem, we use two networks - a teacher and a student. Essentially, we train a bulky ensemble of models (teacher) and use a smaller, lighter model (student) for testing, prediction and deployment. The student is trained to mimic the prediction capabilities of the teacher. How we go about doing this constitutes the crux of Knowledge Distillation.

In other words, the ensemble is simply a function that maps input to output. Transfer the knowledge in this function to the student network is knowledge distillation.

### What is dark knowledge and softmax temperature?

In classification problems, neural networks output *logits* that are computed for each class. A *softmax layer* "normalizes" these logits \(z\_i\) into probabilities \(q\_i\). For a softer distribution, logits are 'softened' or divided by a constant value, called the **temperature** \(T\):

$$q\_i = \frac{exp(z\_i/T)}{\sum\_j exp(z\_j/T)}$$

When the temperature is 1, the probabilities obtained are said to be unsoftened. Hinton et.al. that, in general, the temperature depends on the number of units in the hidden layer of a network. For example, when the number of units in the hidden layer was 300, temperatures above 8 worked well, whereas when the number of units was 30, temperatures in the range of 2.5-4 worked best. Higher the temperature, softer the probabilities.

Consider a classification problem with four classes, `[cow, dog, cat, car]`. If we have an image of a dog, unsoftened hard targets would be `[0, 1, 0, 0]`. This doesn't tell much about what the ensemble has learned. By softening, we may get `[0.05, 0.3, 0.2, 0.005]`. It's clear that predicting a cow is 10 times greater than a car. It's this 'dark' knowledge that needs to be distilled from the teacher network to the student.

### How could I implement this Knowledge distillation?

Buciluǎ et al. designed the first methods of model compression. Later, Hinton et.al. showed the means of distilling the knowledge from an ensemble of models into a single, lighter model.

For example, in image classification, the student would be trained on the class probabilities, or logits, output by the teacher. The logits represent a similarity metric over the classes and help in training good classifiers. Extracting this form of 'dark knowledge' from the teacher network and passing it on to the student is called **distillation**.

Kariya's Medium article provides a simple implementation of Hinton's paper. He touches upon dark knowledge and proceeds to build a simple CNN-based network on the MNIST dataset , showing how the teacher-trained student performed better than a standalone student.

Implementing knowledge distillation can be a resource-intensive task. It requires the training of the student model on the teacher's logits, in addition to training the teacher model.

While training the student, care should be taken to avoid the vanishing gradient problem, which can occur if the learning rate of the student is too high.

### How about performance?

The objective of distilling the knowledge from an ensemble of models into a single, lightweight model is to ease the processes of deployment and testing. It is of paramount importance that accuracy not be compromised in trying to achieve this objective.

In the original paper authored by Hinton et. al., the performance of the student network after knowledge distillation improved, when compared with a standalone student network. Both networks were trained on the MNIST dataset of images. The accuracies of the various models have been tabulated.

As is obvious from the table, the best results are obtained from the bulky ensemble of models and their student alternatives must be used only in case of constrained resources.

### What are the challenges with Knowledge Distillation?

KD is limited to classification tasks that use softmax layer. Sometimes the assumptions are too strict, such as in FitNets where student models may not suit constrained deployment environments. Other approaches to model compression may therefore be preferred over KD.

However, KD continues to be a promising area of research. In 2017, it was adapted for multiclass object detection. In 2018, KD was applied to construct specialized student models for visual question answering. Also in 2018, Guo et al. improved the robustness of student network so that it resists perturbations.

In some domains such as healthcare, DNNs are not preferred. Decision trees are preferred since their predictions can be more easily interpreted. KD has been used to distil DNN into decision tree and thereby provide good performance and interpretability.

## Milestones

1989

Hanson and Pratt propose **network pruning** using biased weight decay. They call their pruned networks *minimal networks*. In the early 1990s, other pruning methods such as optimal brain damage and optimal brain surgeon are proposed. These are early approaches to compress a neural network model. Knowledge distillation as an alternative is invented about two decades later.

2006

Buciluǎ et al. publish a paper titled *Model Compression*. They present a method for “compressing” large, complex ensembles into smaller, faster models, usually without significant loss in performance. They use the ensemble to label large unlabelled datasets. They then use this labelled data to train a single model that performs as well as the ensemble. Because it's not easy to obtain large sets of unlabelled data, they develop an algorithm called *MUNGE* to generate pseudo data. This work is limited to shallow networks.

2014

Ba and Caruana propose the idea of **teacher-student** learning method. They show that shallow models can perform as well as deep models. A complex teacher network (either a deep network or an ensemble) is trained. Instead of using the softmax output, the logits are used to train the shallow student network. Thus, the student network benefits from what the teacher network has learned without losing information via the softmax layer. Some call this **softened softmax**.

2014

Hinton et al. introduce the idea of passing on the 'dark' knowledge from an ensemble of models into a lighter, deployable model. In a paper published in March 2015, they explain that they're "distilling knowledge" from the complex model. The core idea is that models should generalize well to new data rather than optimize on training data. Instead of using logits, they use **distillation**, in which the softmax is used with a higher temperature, also called "soft targets". They note that using logits is a special case of distillation.

2015

**FitNet** aims to produce a student network that's thinner than teacher network while being of similar depth. In addition to the teacher's distilled knowledge of the final softmax layer, Fitnets also make use of intermediate-level hints from the hidden layers. Yim et al. propose a variation of this in 2017 by distilling knowledge from the inner product of features of two layers.

2018

Furlanello et al. show that student models parameterized similar to teacher models outperform the latter. They call these **Born-Again Networks (BANs)** where model compression is not the goal. Students are trained to predict correct labels plus match the teacher's output distribution (knowledge distillation).

2019

Researchers at the Indian Institute of Science, Bangalore, propose **Zero-Shot Knowledge Distillation (ZSKD)** in which they don't use teacher's training dataset or a transfer dataset for distillation. Instead, they synthesize pseudo data from the teacher's model parameters. They call this **Data Impressions (DI)**. This is then used as a transfer dataset to perform distillation. Another research group, with the aim of reducing training, show that just 1% of the training data can be adequate.

2019

Park et al. look at the mutual relationships among data samples and transfer this knowledge to the student network. Called **Relational Knowledge Distillation (RKD)**, this departs from the conventional approach of looking at individual samples. Liu et al. propose something similar, calling it **Instance Relationship Graph (IRG)**. Attention network is another approach to distil relationships.

Sep
2019

Yuan et al. note the conventional KD can be reversed; that is, the teacher can also learn from the student. Another observation is that a poorly-trained teacher can improve the student. They see KD not just as similarity across categories but also as a regularization of soft targets. With this understanding, they propose **Teacher-free Knowledge Distillation (Tf-KD)** in which a student model learns from itself.

Knowledge Distillation


# REST API to GraphQL Migration

## Summary

While GraphQL has its limitations, there are many advantages that make it worthwhile to migrate existing REST APIs to GraphQL. To reap these benefits, it's important to think in terms of graphs rather than endpoints.

The migration itself can be done in one go for small projects. An incremental approach is preferred for complex projects handled by large teams. For most projects, it's expected that current REST APIs will coexist with new GraphQL APIs. This is because clients may not migrate to new app versions or update their code immediately.

In general, migration will involve building the **GraphQL schema** and writing the **resolvers** for that schema.

## Discussion

### What's a possible migration path from REST to GraphQL?

Let's assume that we're using REST API and React client (a). We can start by keeping the REST endpoints but wrapping it with a **GraphQL gateway** (b). We need to define the GraphQL schema. React client will now call a single GraphQL endpoint. Resolvers will use REST endpoints for fetch the data.

A natural evolution from here is to implement the **GraphQL server** (c). We can remove the gateway. React client will directly call the new GraphQL API. This has direct access to the database layer and therefore lower latency compared to the gateway architecture. REST endpoints may coexist to support legacy clients.

A React client does imperative data fetching and might do manual client-side caching. By migrating to **Apollo Client**, we can do declarative data fetching and zero-config caching. By using Apollo Link REST, we don't even need changes at the server side. This could be an easy way to get started. The final goal is to have GraphQL API and Apollo Client (d).

### To design for GraphQL, what sort of mindset should I have?

One study found that reducing the number of API calls by migrating to GraphQL is not a trivial exercise. Suppose the current code is organized to consume small amounts of data returned by specific REST endpoints. Refactoring this into fewer endpoints that return large graph structures requires a graph-based mindset.

Developers have to stop thinking in terms of endpoints and start thinking in terms of graphs. It's important to understand this before designing the GraphQL schema. A typical endpoint-based approach is to ask, "What pages are there and what data each of them needs?" With graph-based approach, we should instead focus on the data and its relationships, and seek to expose these to clients.

Consider a blogging site. Let's think about exposing user information. Relationships from the user might include posts, comments, liked articles, groups, etc. Thus, we are building a graph around the user. Relationships are bidirectional. When focusing on a comment, we should also link it back to the user.

In short,

> Think in graphs, not endpoints. Describe the data, not the view.

### How is TypeScript relevant to GraphQL?

Unlike vanilla JavaScript, TypeScript gives type safety at the client side. GraphQL gives type safety at the server side.

Suppose we have a client that uses only JavaScript. If this client makes a query to the GraphQL server in which the types don't match. This error will be caught at the server during runtime. Instead, if the client code had been in TypeScript, such a mismatch would have been caught during the build process. We can deploy our code with lot more confidence.

A good approach is to define GraphQL schema and then use tools to auto-generate TypeScript types for the front-end. Ideally, we could extend this to the database layer so that from browser to the database, types are consistent.

In general, we should strive for type consistency systemwide. At Airbnb, they defined Thrift IDL that was then compiled into GraphQL schema. Moreover, they then automatically generated TypeScript types from the GraphQL schema.

### Should I implement caching for my GraphQL service?

A single client query can trigger multiple resolvers. GraphQL resolvers run as independent units. This has the benefit that even if some of them fail, the ones that succeed return the requested data to the client. One problem is that across resolvers there could be many duplicate calls to REST endpoints.

It's to solve this problem that a caching layer is helpful. Resolvers need not worry about optimizing their calls to REST endpoints. At Airbnb, engineers approached this problem via what they call a *data-centric schema* and a *data hydration layer*.

At another level, we can cache client request-response. Since GraphQL doesn't support HTTP-level caching, we need an alternative. It's possible to use application-level caching, such as Spring Boot caching. Database queries can also be optimized via caching.

### Could you share some case studies on migrating to GraphQL?

At Netflix, the Marketing Technology team adopted GraphQL to solve their network bandwidth bottlenecks. Their use cases were complex and building custom REST endpoints for each page was cumbersome. GraphQL matched their graph-oriented data structures. They could roll out features lot faster. A GraphQL middle layer reduced number of direct calls from the browser. Instead, server-to-server calls within the data centre gave 8x performance boost. Payload of 10MB reduced to 200KB.

At Airbnb, they reconciled legacy presentation services with GraphQL by having Thrift/GraphQL translators per service. These services were then exposed to clients via a GraphQL gateway service.

At PayPal, a user checkout involved many round trips. Reducing round trips with REST meant overfetching data. They also found that UI developers spent less time building UI. Mostly, they were trying to figure out where and how to fetch data. After moving to GraphQL, data performance and developer productivity improved.

Interested readers can head to the official GraphQL website that shares a number of GraphQL case studies.

### Are there tools that help with GraphQL migration?

GraphiQL is an in-browser IDE. Via GraphiQL, you can look at the schema, make queries and look at the responses. GraphQL Editor, GraphQL Playground and GraphQL Voyager are other useful tools.

Apollo Server and Apollo Client provide server and client-side implementations of GraphQL. Apollo Client can integrate with React, Angular or Vue frameworks. As an alternative to Apollo Client, there's Relay to act as a bridge between GraphQL and React.

There are many tools that can read your REST API files and export GraphQL schema. Apimatic's API Transformer is one such tool. Input files can be in various formats such as OpenAPI, RAML, or API Blueprint. This tool can also migrate SOAP users by translating WSDL files.

Some clients may want to use only REST. To avoid maintaining two codebases, we can have a GraphQL schema and generate REST APIs from it. Sofa is a Node.js package that does this.

### How I do map REST features to their equivalent GraphQL features?

Using REST, a client would make an endpoint call, parse the response and make further calls to get all relevant data. This is the N+1 problem. This can be solved using the nested API design but this leads to overfetching. With a well-designed GraphQL API, clients should be able to and should request only relevant data with a single API call. Moreover, we should make use of **connections**. A connection encapsulates the queried node and other *nodes* to which it's connected by *edges*. This comes directly from the underlying graph-based data abstraction.

In REST, we paginate with a query string such as `?limit=1&page=2`. **Cursor-based pagination** is the GraphQL equivalent. It's typically used with connections using arguments `first`, `last`, `offset` or `after`.

In REST, POST, PUT or DELETE requests modify data. In GraphQL, clients can instead use **mutations**. We could define mutations to create, update or delete items.

GraphQL prefers serving a versionless API and avoids breaking changes as the API evolves. Clients only request what they need and understand. In REST, the practice has been to serve different versions.

### Could you share some tips and best practices for moving to GraphQL?

GraphQL should be a thin layer. Tasks such as authentication, caching or database queries should happen below the GraphQL service layer. By doing this, application will be more resilient to changes in platform or services. Likewise, name your fields in a self-documenting manner. Once clients start using them, it becomes hard to change later.

Don't force clients to replace all their REST endpoint calls to GraphQL. Instead, allow them to adopt GraphQL incrementally. Shopify documentation shows how responses from REST endpoints can include GraphQL IDs that can be used for subsequent GraphQL queries.

Resolvers should be thin and fetch data asynchronously. Fetch data at field level and de-duplicate requests using a library such as DataLoader.

Server logs help in debugging GraphQL calls. To debug within the browser, include logs into the response payload when enabled by a flag.

With GraphQL, we often fetch partial objects that can't be directly used in methods that need full objects. Use duck typing to get around this.

## Milestones

Jul
2015

Facebook open sources GraphQL, which is also released as a draft specification. The development of GraphQL within Facebook can be traced back to 2012.

Jun
2019

Brito et al. compare the performance of GraphQL against REST using seven open source clients to call GitHub and arXiv APIs. They also make typical queries noted in recent papers presented at engineering conferences. They find that GraphQL results in 94% less number of fields fetched and 99% less number of bytes. These are median values. However, there's no significant reduction in the number of API calls.

Oct
2019

Airbnb engineer Brie Bunge shares Airbnb's migration journey to GraphQL. They adopted an incremental approach, making sure the app is always shippable and regression-free. As pre-requisites, they set up GraphQL in the backend and adopted TypeScript in the frontend. Migration itself had five stages. The approach was to first replace REST with GraphQL. Refactoring and optimizations were done later.

REST API to GraphQL Migration


# Regex Engines

## Summary

A regular expression describes a search pattern that can be applied on textual data to find matches. A regex is typically compiled to a form that can be executed efficiently on a computer. The actual search operation is performed by the regex engine, which makes use of the compiled regex.

To write good regexes, it's helpful for programmers to know how these engines work. There are a few types of engines, often implemented as a Finite Automaton. In fact, regexes are related to Automata Theory and Regular Languages.

The syntax and semantics of regexes have been standardized by IEEE as POSIX BRE and ERE. However, there are many non-standard variants. Often, the differences are subtle. Programmers who design regexes must be aware of the variant being used by the engine.

## Discussion

### Is my regex executed directly by a regex engine?

A regex engine receives two inputs: the regex pattern plus the input string. Programmers specify regexes as strings. While an engine could be designed to work with strings directly, there's a better and more efficient way.

It's been shown that for every regex there's an equivalent **Finite State Machine (FSM)** or **Finite Automaton (FA)**. In other words, the regex can be modelled as a finite set of states and transitions among these states based on inputs received at a state. Therefore, the job of a compiler is to take the original regex string and compile it into a finite automaton, which can be more easily executed by the engine.

In some implementations, a preprocessor may be invoked before the compiler. This substitutes macros or character classes, such as, replacing `\p{Alpha}` with `[a-zA-Z]`. A preprocessor also does locale-specific substitutions.

Let's note that there's no standard definition of what's a regex engine. Some may consider parsing, compiling and execution as part of the engine.

### Could you explain how automata theory is applied to regexes?

There are two types of automata:

    + **Deterministic Finite Automaton (DFA)**: Given a state and an input, there's a well-defined output state.
    + **Non-Deterministic Finite Automaton (NFA)**: Given a state and an input, there could be multiple possible output states. A variant of NFA is **ε-NFA**, where a state transition can happen even without an input.It's been proven that every DFA can be converted to an NFA, and vice versa. In the accompanying figure, all three automata are equivalent and represent the regex `abb*a` (or `ab+a`). For NFA, when 'b' is received in state q1, the automaton can remain in q1 or move to q2. Thus, it's non-deterministic. For ε-NFA, the automaton can move from q2 to q1 even without an input. It's been said,

> Regular expressions can be thought of as a user-friendly alternative to finite automata for describing patterns in text.

### What are the different types of regex engines?

Regex engines could be implemented as DFA or NFA. However, in simpler language, a regex engine can be classified as follows:

    + **Text-Directed**: Engine attempts all paths of the regex before moving to next character of input. Thus, this engine doesn't backtrack. Since all paths are attempted at once, it will return the longest match. For example, `(Set|SetValue)` on input "SetValue" will match "SetValue".
    + **Regex-directed**: If the engine fails at a position, it backtracks to attempt an alternative path. Paths are attempted in left-to-right order. Thus, it returns the leftmost match even if there's a longer match in another path later. For example, `(Set|SetValue)` on input "SetValue" will match "Set".Most modern engines are regex-directed because this is the only way to implement useful features such as lazy quantifiers and backreferences; and atomic grouping and possessive quantifiers that give extra control to backtracking.

Today's regexes are feature rich and can't always to implemented efficiently as an automaton. Lookaround assertions and backreferences are hard to implement as NFA. Most regex engines use **recursive backtracking** instead.

### How do the different regex engines compare?

With recursive backtracking, pathological regexes result in lots of backtracking and searching through alternative paths. The time complexity grows exponentially. Thompson's NFA (or its equivalent DFA) is more efficient and maintains linear-time complexity. A compromise is to use Thompson algorithm and backtrack only when needed for backreferences. GNU's awk and grep tools use DFA normally and switch to NFA when backreferences are used.

Ruby uses non-recursive backtracking but this too grows exponentially for pathological regexes. Ruby's engine is called *Oniguruma*.

DFA is more efficient than NFA since the automaton is in only one state at any given time. A traditional NFA tries every path before failing. A POSIX NFA tries every path to find the longest match. A text-directed DFA spends more time and memory analyzing and compiling the regex but this could be optimized to compile on the fly.

In terms of code size, NFA regex in *ed* (1979) was about 350 lines of C code. Henry Spencer's 1986 implementation was 1900 lines and his 1992 POSIX DFA was 4500 lines.

### What are the essential rules that a regex engine follows?

A regex engine executes the regex one character at a time in **left-to-right order**. This input string itself is parsed one character at a time, in left-to-right order. Once a character is matched, it's said to be **consumed** from the input, and the engine moves to the next input character.

The engine is by default **greedy**. When quantifiers (such as `* + ? {m,n}`) are used, it will try to match as many characters from the input string as possible. The engine is also **eager**, reporting the first match it finds.

If the regex doesn't match a character in input, it does two things. It will backtrack to an earlier greedy operation and see if a less greedy match will result in a match. Otherwise, the engine will move to the next character and attempt to match the regex all over again at this position. Either way, the engine always knows its **current position** within the regex. If the regex specifies alternatives, if one search path fails, the engine will backtrack to match the next alternative. Therefore, the engine also stores **backtracking positions**.

### Could you explain the concepts "greedy" and "eager" with examples?

Take regex `a.*o` on input "cat dog mouse". Even though "at do" is a valid match, since the engine is greedy, the match it gives is "at dog mo". To avoid this greedy behaviour, we can use a lazy or non-greedy match: `a.*?o` will match "at do". Ungreedy match in some flavours such as PCRE can be specified using 'U' flag.

A non-greedy `\w{2,3}?` on input "abc" will match "ab" rather than "abc". Suppose the regex is `\w{2,3}?$`, then the match is "abc" and not "bc", even though the regex is non-greedy. This is because the engine is eager to report the first match it finds. Thus, it first matches "ab", then sees that `$` doesn't match "c". At this point, the engine will not backtrack for position "b". It will remain at "a" and compare the third character due to `{2,3}`. Thus, it finds "abc" as match. It eagerly reports this match.

Another example is `(self)?(selfish)?` applied on input "selfish". Because of eagerness, engine will report "self" as the match. However, a text-directed engine will report "selfish".

### Could you explain backtracking with an example?

Let's take the regex `/-\d+$/g` and input string "212-244-7688". The regex engine will match `-\d+` to "-244" but when it sees `$` it declares no match. At this point, it will backtrack to the start of the regex and current position in input string will advance from "-" to "2". In this example, only one backtracking happens. Suppose we apply `/\d-\d+$/g` on the same input, we'll have five backtracks as shown in the figure.

Engine can also backtrack part of the way. Let's apply `/A\d+\D?./g` to "A1234". The engine will match `A\d+\D?` to "A1234" but when it sees `.` there's no match. It will backtrack to `\d+` and give up one character so that `A\d+` now matches only "A123". As the engine continues, it will match `.` with "4".

Another example of backtracking is `pic(ket|nic)`. If the string is "Let's picnic.", the engine will match `pic` to "pic" but will fail the next character match (k vs. n). The engine knows there's an alternative. It will backtrack to end of `pic` and process the second alternative.

## Milestones

1959

Michael Rabin and Dana Scott introduce the concept of non-determinism. They show that an NFA can be simulated by a DFA in which each DFA state corresponds to a set of NFA states.

1968

Ken Thompson shows in an ACM paper how a regex can be converted to a NFA. He presents an engine that can track alternative paths in the NFA simultaneously. This is now called **Thompson NFA**. For a regex of length m and input of length n, Thompson NFA requires \(O(mn)\) time. In comparison, the backtracking regex implementation requires \(O(2^n)\) time when there are n alternative paths. An NFA can be built from simple operations (such as concatenation, alternation, looping) on partial NFAs.

1971

Unix (First Edition) appears and *ed* text editor is one of the programs in it. The editor uses regex but it's not Thompson's NFA but recursive backtracking. Other utilities such as grep (1973) follow suit.

1979

Unix (Seventh Edition) includes *egrep*, the first utility to support full regex syntax. It pre-computes the DFA. By 1985, it's able to generate the DFA on the fly.

1985

The *regexp(3)* library of Unix (Eighth Edition) adapts Thompson's algorithm to extract submatches or capturing groups. This work is credited to Rob Pike and Dave Presotto. This goes unnoticed and is not widely used. However, a year later, Henry Spencer reimplements the library interface from scratch using recursive backtracking. This is later adopted by Perl, PCRE, Python, and others.

1999

Henry Spencer writes an engine for Tcl version 8.1. This is a hybrid engine. It's an NFA supporting lookaround and backreferences. It also return the longest-leftmost match as specified by POSIX.

2007

Russ Cox provides a 400-line C implementation of Thompson's NFA. He shows that for pathological regex, this is a lot faster than common implementations (recursive backtracking) used in many languages including Perl.

Mar
2010

Google open sources **RE2** that's based on automata theory, has linear-time complexity and uses a fixed size stack. Because Google uses regexes for customer-facing tools such as Code Search, backtracking and exponential-time complexity of earlier implementations could lead to Denial-of-Service (DoS) attacks. In addition, recursive backtracking can lead to stack overflows. Work on RE2 can be traced to the work on Code Search in 2006.

Regex Engines


# Google Cloud Authentication

## Summary

Google Cloud supports three main types of credentials by which apps can gain access to APIs and services. These are API keys, OAuth 2.0 client IDs and service accounts. This article gives an overview of these methods. It offers some guidelines on how to choose the right authentication method for an application.

Improper use of these methods or careless management of credentials can lead to a security breach. It can result in identity theft and data theft/loss. This article shares some best practices for better security.

## Discussion

### What are the different authentication methods available in Google Cloud?

Anyone with a Google account can login to Google by supplying a valid username and password. Two-Factor Authentication (2FA) enhances security by adding one more step to the authentication process. In Google, 2FA is called 2-Step Verification. For a better user experience, Google also provides Sign In with Google, One Tap and Automatic sign-in.

Often developers want their apps to have access to Google accounts or services, for the apps themselves or on behalf of end users. Google offers two ways to achieve this:

    + **User Account**: Represents a developer or administrator. Used when an application needs to access Google Cloud resources on behalf of the user. Managed by Google Accounts. Credentials used include API keys and OAuth 2.0 client credentials.
    + **Service Account**: Represents non-human users. Used when an application needs to access Google Cloud resources on its own without any user intervention. Managed by Google's Identity and Access Management (IAM). Credentials include service account keys. Unlike a user account, a service account doesn't have a user-facing login interface.Google Cloud APIs use OAuth 2.0 protocol to authenticate both user accounts and service accounts.

### How do I select a suitable Google Cloud authentication method?

Many Google Cloud APIs allow anonymous access to public data. **API keys** can be used. These identify the application, not the actual user.

If your app requires access to a user's private data, then **OAuth 2.0 client ID** can be used. In OAuth 2.0 terminology, your application is called the client. The client ID identifies the application. The user is prompted to allow Google share some of his/her data with your application. Your application can now use Google Cloud APIs on behalf of the user.

Where user authorization is not possible, use a **service account**. For apps running inside Google Cloud, it's automatically created. Otherwise, you manually create a service account. You download the credentials as a JSON file that your app can use. Your app assumes the identity of the service account so that users are not involved. Such server-to-server interactions are sometimes called *Two-Legged OAuth (2LO)*.

### Could you share some use cases for each Google Cloud authentication method?

Google Cloud has dozens of APIs that can be accessed via API keys or OAuth 2.0 client IDs. Some of these APIs are Google Drive, Google Calendar, GMail, YouTube Data, App Engine Admin, Cloud Datastore, Cloud Speech-to-Text, Cloud Vision, Firebase Management, and many more.

For example, YouTube Data API can be used read metadata of any video. It's implemented via the `videos` endpoint. Since this is public data, an API key is sufficient. The same API also allows an app to manage videos and playlists that you own. This requires OAuth 2.0 client ID.

Managing your appointments on Google Calendar or your drafts on GMail are further examples where OAuth 2.0 client ID can be used. Web/mobile apps can use OAuth 2.0 client ID for visitors to quickly create accounts on their platforms.

Consider an external application that wants to store nightly data backups to Google Drive. A service account suits this use case.

### Could you share some tips related to Google Cloud authentication?

API keys, OAuth 2.0 client IDs and service accounts are typically created via a web interface called *Google Cloud Console*. In all cases, the credentials fall within the scope of a project that has an associated billing account. Advanced users can look at the *API Keys API* to create or manage API keys via `gcloud` CLI. `gcloud` can also be used to manage service accounts.

Developers can try out APIs using the Google API Explorer. This helps in understanding the APIs before doing it programmatically from application code. While it's possible to implement OAuth 2.0 flows, it's easier to use Google Cloud Client Libraries.

When using OAuth 2.0 client IDs, code your apps to ask for minimum required permissions. This is done by setting *scopes*. Google maintains a list of available OAuth 2.0 Scopes per API.

For local development, use `gcloud auth login` and `gcloud auth application-default login` commands. Use short-lived tokens.

### What are some best practices when using API keys?

Don't embed API keys directly in code. Store them in environment variables or in files outside your application's version controlled source code. Do a code review to ensure API keys aren't there by mistake.

When not used, delete API keys to minimize security attacks. Regenerate API keys periodically.

For better security, place restrictions on the use of API keys:

    + **Source**: Restrictions on who can use the key. Restrict by HTTP referrers, IP addresses, Android apps or iOS apps. HTTP referrers can be whitelisted by subdomains, paths, protocols (HTTP or HTTPS) and even wildcards.
    + **Target**: Restrictions on what services are allowed. For example, restrict the key to only Google Drive API.API keys are vulnerable to man-in-the-middle attacks since they're part of the request. Don't use them when API calls contain user data.

Don't use API keys to identify users. An API key only identifies the project. Don't use API keys for secure authorization. API keys are useful to block anonymous traffic, control volume of API requests, identify usage patterns, or filter logs by the API key.

### What are some best practices when using service accounts?

Service accounts are both a resource and an identity. As an identity, avoid granting Owner/Editor/Viewer roles to a service account. Grant instead a predefined or custom role. In general, grant only minimum necessary permissions.

Use unique, descriptive names so that it's easier to manage multiple accounts. Delete accounts if they're unused since they present a security risk. Disable accounts before deleting them.

Service account keys are either Google Cloud-managed or user-managed. For the latter, follow processes for key storage, distribution, revocation, rotation and recovery.

Don't use service accounts to access data without user consent. Don't use service accounts during development. Instead authenticate using tools such as `gcloud`, `gsutil`, `terraform`, etc. In other words, use your personal credentials in a development environment.

You can attach applications to an existing service account. On a GKE cluster, use multiple Google Cloud and Kubernetes service accounts and map them via Workload Identity.

## Milestones

Apr
2008

Google previews **App Engine**, a platform for developing and hosting web applications in Google-managed data centres. App Engine is therefore the first cloud service offered by Google. It marks the birth of Google Cloud.

Oct
2012

IETF releases **OAuth 2.0** as RFC 6749. Although OAuth 2.0 is designed for authorization and not for user authentication, Google Cloud later uses OAuth 2.0 for both authorization and authentication.

Mar
2011

Google launches **Google API Explorer** that allows developers to experiment with Google's APIs before implementing them in their apps. By August 2021, the Explorer supports 230 APIs. API keys and OAuth 2.0 client IDs can be used with the Explorer.

Aug
2021

Google launches **Google Identity Services**, a single SDK that brings together different ways of authenticating users. It provides developers frictionless flows to onboard users to their platforms. *Sign in with Google* and *One Tap* are two such flows. These use secure tokens rather than passwords. These internally use OAuth 2.0. Developers can use client libraries provided by Google to quickly implement these flows. An alternative is *Firebase Authentication*.

Google Cloud Authentication


# Natural Language Processing

## Summary

In computer science, languages that humans use to communicate are called "natural languages". Examples include English, French, and Spanish. Early computers were designed to solve equations and process numbers. They were not meant to understand natural languages. Computers have their own programming languages (C, Java, Python) and communication protocols (TCP/IP, HTTP, MQTT).

To instruct computers to perform tasks, we traditionally use a keyboard or a mouse. Why not speak to the computer and let it respond in a natural language? This is one of the aims of Natural Language Processing (NLP). NLP is an essential component of artificial intelligence.

NLP is rooted in the theory of linguistics. Techniques from machine learning and deep neural networks have also been successfully applied to NLP problems. While many practical applications of NLP already exist, NLP has many unsolved problems.

## Discussion

### Why do computers have difficulty with NLP?

Computers have mostly been dealing with **structured data**. This is data that's organized, indexed and referenced, often in databases. In NLP, we often deal with **unstructured data**. Social media posts, news articles, emails, and product reviews are examples of text-based unstructured data. To process such text, NLP has to learn the structure and grammar of the natural language. Importantly, 80% of enterprise data is unstructured.

Human languages are quite unlike the precise and unambiguous nature of computer languages. Human languages have plenty of complexities such as ambiguous phrases, colloquialisms, metaphors, puns, or sarcasms. The same word or text can have multiple meanings depending on the context. Language evolves with time. Worse still, we communicate imperfectly (spelling, grammar or punctuation errors) but still manage to be understood. These variations, so natural to human communication, are complex for computers.

Ambiguities in natural languages can be classified as lexical, syntactic or referential.

When the source of information is speech, more challenges arise: accent, tone, loudness, background noise or context, pronunciation, emotional content, pauses, and so on.

### Could you share some examples of the complexities of English?

Consider the sentence, "One morning I shot an elephant in my pajamas". The man was in his pajamas but grammatically it's also correct to think that the elephant was wearing his pajamas. Likewise, a person may say, "Listening to loud music slowly gives me a headache". Was she listening to music slowly or does the headache develop slowly?

A more confusing example is, "The complex houses married and single soldiers and their families". Confusion arises because we may initially interpret "complex houses" as an adjective-noun combination. The sentence makes sense only when we see that "complex" is a noun and "houses" is a verb. NLP addresses this via part-of-speech tagging.

Consider this one, "John had a card for Helga, but couldn't deliver it because he was in her way". Was John was in Helga's way? In fact, "he" refers to an earlier reference to a third person. NLP calls this coreference resolution.

"The Kiwis won the match" is an example that requires context to make sense. New Zealand nationals are referred to as "Kiwis", after their national bird. Natural language is full of metaphors like this.

### What are some example problems that NLP can solve?

From the number of problems that NLP solves, we describe a few:

    + **Sentiment Analysis**: From product reviews or social media messages, the task is to figure out if the sentiment is positive, neutral or negative.  This is useful for customer support, engineering and marketing departments.
    + **Machine Translation**: Suppose original content is published only in one language, machine translation can deliver this content to a wider readership.  Tourists can use machine translation to communicate in a foreign country.
    + **Question Answering**: Given a question, an NLP engine leveraging a vast body of knowledge, can provide answers. This can help researchers and journalists. Whitepapers and reports can be written faster.
    + **Text Summarization**: NLP can be tasked to summarize a long essay or an entire book. It can provide a balanced summary of a story published at different websites with different points of view.
    + **Text Classification**: NLP can classify news stories by domain or detect email spam.
    + **Text-to-Speech**: This is an essential aspect of voice assistants. Audiobooks can be created for the visually impaired. Public announcements can be made.
    + **Speech Recognition**: Opposite of text-to-speech, this creates a textual representation of speech.

### Who's been using NLP in the real world, and for what purpose?

Facebook uses machine translation to automatically translate posts and comments. Google Translate processes 100 billion words a day. To connect sellers and buyers across language barriers, eBay is using machine translation.

Using speech recognition and text-to-speech synthesis, voice assistants such as Amazon Alexa, Apple Siri, Facebook M, Google Assistant, and Microsoft Cortana are enabling human-to-device interaction using natural speech.

Amazon Comprehend offers an NLP API to perform many common NLP tasks. This has been extended by Amazon Comprehend Medical for healthcare domain.

Uber uses NLP for better customer support. Human agents are involved but they are assisted by NLP models that suggest top three solutions. This has reduced ticket resolution time by over 10%.

Perception offers an NLP-based product to do theme clustering and sentiment analysis. This helps with performance reviews and employee retention while minimizing bias.

For aircraft maintenance, NLP is used for information retrieval, troubleshooting, writing summary reports, or even directing a mechanic via a voice interface. It's been observed that NLP can classify defects better than humans.

### What are the main approaches adopted by NLP?

Classical NLP from the 1950s took the **symbolic approach** rooted in linguistics. Given the rules of syntax and grammar, we could obtain the structure of text. Using logic, we could obtain the meaning. But rules had to be hand-crafted and were often numerous. They didn't handle colloquial text well. Rules worked well for specific use cases but couldn't be generalized.

In practice, better accuracy was achieved by using a **statistical approach** that began in the 1980s. Rules were learned and they had associated probabilities. Machine Learning (ML) models came in with support vector machines and logistic regression. More recently, Deep Learning (DL) models that employ a neural network of many layers have brought better accuracy. This success is partly due to the more efficient representations given by *word embeddings*.

NLP involves different levels or scope of analysis. **Low-level** analysis is about word tokens and structure. **Mid-level** analysis is about identifying entities, topics, and themes. **High-level** analysis leads to meaning and understanding. Alternatively, some classify text processing into two parts: **shallow parsing or chunking** and **deep parsing**.

### How is NLP related to NLU and NLG?

NLP is broadly made of two parts:

    + **Natural Language Understanding (NLU)**: This involves converting speech or text into useful representations on which analysis can be performed. The goal is to resolve ambiguities, obtain context and understand the meaning of what's being said. Some say NLP is about text parsing and syntactic processing while NLU is about semantic relationships and meaning. NLU tackles the complexities of language beyond the basic sentence structure.
    + **Natural Language Generation (NLG)**: Given an internal representation, this involves selecting the right words, forming phrases and sentences. Sentences need to ordered so that information is conveyed correctly.NLU is about analysis. NLG is about synthesis. An NLP application may involve one or both. Sentiment analysis and semantic search are examples of NLU. Captioning an image or video is mainly an NLG task since input is not textual. Text summarization and chatbot are applications that involve NLU and NLG.

There's also **Natural Language Interaction (NLI)** of which Amazon Alexa and Siri are examples.

### What's the typical data processing pipeline in NLP?

A typical NLP pipeline consists of text processing, feature extraction and decision making. All these steps could apply classical NLP techniques, machine learning or neural networks. Where ML and NN are used, we would have to train a model from sufficient volume of data before it can be used for prediction and decision making.

In text processing, the input is just text and the output is a structured representation. This is done by identifying words, phrases, parts of speech, and so on. Since words have variations (go, going, went), it's common to reduce them to a root form with techniques such as **stemming** and **lemmatization**. Common words that don't add value to analysis (the, to, and, etc.) are called *stop words* and these are removed. Punctuations are also removed to simplify analysis. **Named Entity Recognition (NER)** involves identifying entities such as places, names, objects, and so on. **Coreference resolution** tries to resolve pronouns (he, they, it, etc.) to the correct entities.

More formally, text processing involves analysis of three types: syntax (structure), semantics (meaning), pragmatics (meaning in context).

### What are some challenges that NLP needs to solve?

NLU is still an unsolved problem. Systems are as yet incapable of understanding the way humans do. Until then, progress will be limited to better pattern matching. Where NLU is lacking, it affects the success of NLG.

In the area of chatbots, there's a need to model common sense. It's also not clear if models should begin with some understanding or should everything be learned using the technique of reinforcement learning. Computing infrastructure needed to build a full-fledged agent that can learn from its environment is also tremendous.

Not much has been done for low-resource languages where the need for NLP is greater. Africa alone has about 2100 languages. We need to find a way to solve this even if training data is limited.

Current systems are unable to reason with large contexts, such as entire books or movie scripts. Supervision with large documents is scarce and expensive. Unsupervised learning has the problem of sample inefficiency.

Just measuring progress is a challenge. We need datasets and evaluation procedures tuned to concrete goals.

### Could you mention some of the tools used in NLP?

In Python, two popular NLP tools are **Natural Language Toolkit (NLTK)** and **SpaCy**. NLTK is supposedly slower and therefore not the best choice for production. TextBlob extends NLTK. Textacy is based on SpaCy and handles pre-processing and post-processing tasks. There's also PyTorch-NLP suited for prototyping and production. AllenNLP and Flair are built on top of PyTorch for developing deep learning NLP models. Intel NLP Architect is an alternative. Gensim is a library that targets topic modelling, document indexing and similarity retrieval.

There are also tools in other programming languages. In Node.js, we have Retext, Compromise, Natural and Nlp.js. In Java, we have OpenNLP, Stanford CoreNLP and CogCompNLP. The last two have Python bindings as well. There are libraries in R and Scala as well but these haven't been updated for over a year.

For execution, **Jupyter Notebook** provides an interactive environment. If you don't want to install Jupyter, it's also available as web services. Azure Notebook Service is an example. Via subscriptions, these services allow you to use powerful cloud computing resources.

## Milestones

1948

In the area of automated translation, a dictionary look-up system developed at Birkbeck College, London can be seen as the first NLP application. In the years following World War II, researchers attempt translating German text to English. Later during the era of Cold War, it's about translating Russian to English.

1957

American linguist Noam Chomsky publishes *Syntactic Structures*. Chomsky revolutionizes the theory of linguistics and goes on to influence NLP a great deal. The invention of Backus-Naur Form notation in 1963 for representing programming language syntax is influenced by Chomsky's work. Another example is the invention of Regular Expressions in 1956 for specifying text search patterns.

1966

In the U.S., the Automatic Language Processing Advisory Committee (ALPAC) Report is published. It highlights the limited success of machine translation. This results in a lack of funding right up to 1980. Nonetheless, NLP advances in some areas including case grammar and semantic representations. Much of the work till late 1960s is about syntax though some addressed semantic challenges.

1970

In this decade, NLP is influenced by AI with focus on world knowledge and meaningful representations. Thus, semantics becomes more important. SHRDLU (1973) and LUNAR (1978) are two systems of this period. Into the 1980s, these lead to the adoption of logic for knowledge representation and reasoning. **Prolog** programming language is also invented in 1970 for NLP applications.

1980

This decade sees the growing adoption of Machine Learning and thereby signalling the birth of **statistical NLP**. Annotated bodies of text called *corpora* are used to train ML models to provide the gold standard for evaluation. ML approaches to NLP become prominent through the 1990s, partly inspired by the successful application of Hidden Markov Models to speech recognition. The fact that statistics has brought more success than linguistics is echoed by Fred Jelinek,

> Every time I fire a linguist, the performance of our speech recognition system goes up.

1982

Project Jabberwacky is launched to simulate natural human conversations in the hope of passing the Turing Test. This heralds the beginning of **chatbots**. In October 2003, Jabberwacky wins third place in the Loebner Prize.

1998

The *FrameNet* project is introduced. This is related to **semantic role modelling**, a form of shallow semantic parsing that's continues to be researched even in 2018.

2001

For language modelling, the classical N-Gram Model has been used in the past. In 2001, researchers propose the use of a **feed-forward neural network** with vector inputs, now called *word embeddings*. In later years, this leads to the use of RNNs (2010) and LSTMs (2013) for language modelling.

2003

**Latent Dirichlet Allocation (LDA)** is invented and becomes widely used in machine learning. It's now the standard way to do topic modelling.

2013

Improvements to word embeddings along with an efficient implementation in *Word2vec* enable greater adoption of neural networks for NLP. RNNs and LSTMs become obvious choices since they deal with dynamic input sequences so common in NLP. CNNs from computer vision get repurposed for NLP since CNNs are more parallelizable. Recursive Neural Networks attempt to exploit the hierarchical nature of language.

Mar
2016

Microsoft launches *Tay*, a chatbot on Twitter that would interact with users and get better in conversing. However, Tay is shut down within 16 hours after it learns to talk in racist and abusive language. A few months later Microsoft launches *Zo* chatbot.

Sep
2016

Google replaces its phrase-based translation system with **Neural Machine Translation (NMT)** that uses a deep LSTM network with 8 encoder and 8 decoder layers. This reduces translation errors by 60%. This work is based on **sequence-to-sequence learning** proposed in 2014, which later becomes a preferred technique for NLG.

Natural Language Processing


# Speech Recognition

## Summary

Speech Recognition is the process by which a computer maps an acoustic speech signal to text.

Speech Recognition is also known as Automatic Speech Recognition (ASR) or Speech To Text (STT).

Speech Recognition crossed over to 'Plateau of Productivity' in the Gartner Hype Cycle as of July 2013, which indicates its widespread use and maturity in present times.

In the longer term, researchers are focusing on teaching computers not just to transcribe acoustic signals but also to understand the words. Automatic speech understanding is when a computer maps an acoustic speech signal to an abstract meaning.

It is a sub-field of computational linguistics (an interdisciplinary field concerned with the statistical or rule-based modeling of natural language) that develops methodologies and technologies that enables the recognition and translation of spoken language into text by computers.

## Discussion

### What are the steps involved in the process of speech recognition?

We can identify the following main steps:

    + Analog-to-Digital Conversion: Speech is usually recorded/available in analog format. Standard sampling techniques/devices are available to convert analog speech to digital using techniques of sampling and quantization. The digital speech is usually a one-dimension vector of speech samples, each of which is an integer.
    + Speech Pre-processing: Recorded speech usually comes with background noise and long sequences of silence. Speech pre-processing involves identification and removal of silence frames and signal processing techniques to reduce/eliminate noise. After pre-processing, the speech is broken down into frames of 20ms each for further steps of feature extraction.
    + Feature Extraction: It is the process of converting speech frames into feature vector which indicates which phoneme/syllable is being spoken.
    + Word Selection: Based on a language model/probability model, the sequence of phonemes/features are converted into the word being spoken.

### Which are the popular feature extraction methods?

While there are many feature extraction methods, we note three of them:

    + **Relative Spectral Transform-Perceptual Linear Prediction (RASTA-PLP)**: PLP is a way of warping spectra to minimize differences between speakers while preserving important speech information. RASTA applies a band-pass filter to the energy in each frequency subband in order to smooth over short-term noise variations and to remove any constant offset resulting from static spectral coloration in the speech channel e.g. from a telephone line
    + **Linear Predictive Cepstral Coefficients (LPCCs)**: A cepstrum is the result of taking the inverse Fourier transform (IFT) of the logarithm of the estimated spectrum of a signal. The power cepstrum is used in the analysis of human speech.
    + **Mel Frequency Cepstral Coefficients (MFCCs)**: These are derived from a type of cepstral representation of the audio clip (a nonlinear "spectrum-of-a-spectrum"). The difference between LPCC & mel-frequency cepstrum is that in the MFCC, the frequency bands are equally spaced on the mel scale, which approximates the human auditory system's response more closely than the linearly-spaced frequency bands used in the normal cepstrum. This frequency warping allows for better representation of sound.

### Which are the traditional Probability Mapping and Selection methods?

A **Hidden Markov Model** is a type of graphical model often used to model temporal data. Hidden Markov Models (HMMs) assume that the data observed is not the actual state of the model, but is instead generated by the underlying hidden (the H in HMM) states. While this would normally make inference difficult, the Markov Property (the first M in HMM) of HMMs makes inference efficient.

The hidden Markov model can be represented as the simplest dynamic Bayesian network. The mathematics behind the HMM were developed by L. E. Baum and coworkers.

Because of their flexibility and computational efficiency, Hidden Markov Models have found a wide application in many different fields like speech recognition, handwriting recognition, and speech synthesis.

### How is the accuracy of a speech recognition program validated?

**Word error rate (WER)** is a common metric of the performance of a speech recognition or machine translation system. Generally it is measured on **Switchboard** - a recorded corpus of conversations between humans discussing day-to-day topics. This has been used over two decades to benchmark speech recognition systems. There are other corpora like LibriSpeech (based on public domain audio books) and Mozilla’s Common Voice project.

For some languages, like Mandarin, the metric is often CER - **Character Error Rate**. There is also Utterance Error Rate.

An IEEE paper that focussed on ASR and machine translation interactions, in a speech translation system, showed that BLEU-oriented global optimisation of ASR system parameters improves translation quality by absolute 1.5% BLEU score, while sacrificing WER over the conventional WER-optimised ASR system. Therefore the choice of metrics for ASR optimisation is context and application dependent.

### How has speech recognition evolved over the years?

Starting from the 1960s, the pattern recognition based approaches started making speech recognition practical for applications with limited vocabulary with the use of LPC (Linear Predictive Coding) Coefficient and LPCC (Linear Predictive Cepstral Coefficient) based techniques. The advantage of this technique was that low resources were required to build this model and could be used for applications requiring up to about 300 words.

In the late 1970s, Paul Mermelstein found a new feature called MFCCs. This soon became the de-facto approach for feature extraction and helped to tackle applications related to multi-speaker as well as multi-language speech recognition.

In the 1990s, H. Hermansky came up with the RASTA-PLP approach of feature extraction which could be used for applications requiring very large vocabulary with multiple speakers and multiple languages with good accuracy.

### What are the AI based approaches for speech recognition?

In the 1990s and in early 2000s, Deep Learning techniques involving Recurrent Neural Networks were applied on Speech Recognition. In 2000s, the variant of RNNs using LSTM (Long Short Term Memory) to include Long Term Memory aspects into the model helped to minimise or avoid the problems of vanishing gradient or exploding gradients as part of training the RNNs.

Baidu research released DeepSpeech 2014 achieving a WER of 11.85% using RNN. They leveraged the “Connectionist Temporal Classification” loss function.

With DeepSpeech2 in 2015 they achieved a 7x increase in speed using GRUs (Gated Recurrent Units).

DeepSpeech3 was released in 2017. They perform an empirical comparison between three models — CTC which powered Deep Speech 2, attention-based Seq2Seq models which powered Listend-Attend-Spell among others, and RNN-Transducer for end-to-end speech recognition. The RNN-Transducer could be thought of as an encoder-decoder model which assumes the alignment between input and output tokens is local and monotonic. This makes the RNN-Transducer loss a better fit for speech recognition (especially when online) than attention-based Seq2Seq models by removing extra hacks applied to attentional models to encourage monotonicity.

### How is the speed of a speech recognition system measured?

Real Time Factor is a very natural measure of a speech decoding speed that expresses how much the recogniser decodes slower than the user speaks. The latency measures the time between the end of the user speech and the time when a decoder returns the hypothesis, which is the most important speed measure for ASR.

Real-time Factor (RTF): the ratio of the speech recognition response time to the utterance duration. Usually both mean RTF (average over all utterances), and 90th percentile RTF is examined in efficiency analysis.

### How is Word Error Rate calculated?

In October 2016, Microsoft announced its ASR had a WER of 5.9% against the Industry standard Switchboard speech recognition task. This was surpassed by IBM Watson in March 2017 with a WER of 5.5%. In May 2017 Google announced it reached a WER of 4.9%, however google does not benchmark against the Switchboard.

ASR systems have seen big improvements in recent years due to more efficient acoustic models that use Deep Neural Networks (DNNs) to determine how well HMM states fit the extracted acoustic features rather than statistical techniques such as Gaussian Mixture Models, which were the preferred method for several years.

### Which are the popular APIs that one can use to incorporate automatic speech recognition in an application?

Here's a selection of popular APIs: Bing Speech API, Nuance Speech Kit, Google Cloud Speech API, AWS Transcribe, IBM Watson Speech to Text, Speechmatics, Vocapia Speech to Text API, LBC Listen By Code API, Kaldi, CMU Sphinx.

### What are the applications for speech recognition?

Applications of speech recognition are diverse and we note a few:

    + Aerospace (e.g. space exploration, spacecraft, etc.) NASA's Mars Polar Lander used speech recognition technology from Sensory, Inc. in the Mars Microphone on the Lander.
    + Automatic subtitling with speech recognition
    + Automatic translation
    + Court reporting (Realtime Speech Writing)
    + eDiscovery (Legal discovery)
    + Education (assisting in learning a second language)
    + Hands-free computing: Speech recognition computer user interface
    + Home automation (Alexa, Google Home etc.)
    + Interactive voice response
    + Medical transcription
    + Mobile telephony, including mobile email
    + Multimodal interaction
    + People with disabilities
    + Pronunciation evaluation in computer-aided language learning applications
    + Robotics
    + Speech-to-text reporter (transcription of speech into text, video captioning, Court reporting )
    + Telematics (e.g. vehicle Navigation Systems)
    + User interface in telephony
    + Transcription (digital speech-to-text)
    + Video games, with Tom Clancy's EndWar and Lifeline as working examples
    + Virtual assistant (e.g. Apple's Siri)

### Which are the available open source ASR toolkits?

Here's a selection of open source ASR toolkits:

    + **Microsoft Cognitive Toolkit**, a deep learning system that Microsoft used for its ASR system is available on GitHub through an open source license.
    + The Machine Learning team at Mozilla Research has been working on an open source Automatic Speech Recognition engine modelled after the **Deep Speech** papers published by Baidu. It has a WER of 6.5 percent on LibriSpeech’s test-clean set.
    + **Kaldi** is integrated with TensorFlow. Its code is hosted on GitHub with 121 contributors. It originated at a 2009 workshop in John Hopkins University. It is designed for local installation.
    + **CMU Sphinx** is from Carnegie Mellon University. Java and C versions exist on GitHub.
    + **HTK** began in Cambridge University in 1989.
    + **Julius** has been in development since 1997 and had its last major release in September of 2016.
    + **ISIP** originated from Mississippi State. It was developed mostly from 1996 to 1999, with its last release in 2011.
    + **VoxForge** - crowdsourced repository of speech recognition data and trained models.

## Milestones

1952

Bell Labs researchers - Davis, Biddulph and Balashak - build a system for single-speaker digit (10) recognition. Their system worked by locating the formants in the power spectrum of each utterance. They were basically creating realisations of earlier analysis establishing relationships between sound classes and signal spectrum by Harvey Fletcher and Homer Dudley, both from AT&T Bell Laboratories. Speech recognition research at Bell Labs was defunded after an open letter by John Robinson Pierce that was critical of speech recognition research.

1962

IBM demonstrates its 'Shoebox' machine at the 1962 World Fair, which could understand 16 words spoken in English.

1968

Dabbala Rajagopal **"Raj" Reddy** conducts a demonstration of voice control of a robot, large vocabulary connected speech recognition, speaker independent speech recognition and unrestricted vocabulary dictation. *Hearsay-I* is one of the first systems capable of continuous speech recognition. Reddy's work lays the foundation for more than three decades of research at Carnegie Mellon University with his work in the field of **continuous speech recognition** based on dynamic tracking of phonemes. Funding by DARPA's Speech Understanding Research (SUR) program was responsible for Carnegie Melon's "Harpy" speech understanding system, that could understand 1011 words.

1968

In Russia, Professor Taras Vintsyuk proposes the use of dynamic programming methods for time aligning a pair of speech utterances (generally known as **Dynamic Time Warping (DTW)**). Velichko and Zagoruyko use Vintsyuk's work to advance use of pattern recognition ideas in speech recognition. The duo build a 200-word recogniser.  Professor Victorov develops a system that recognises 1000 words in 1980.

1974

First commercial speech recognition company - Threshold Technology

1975

James Baker starts working on **HMM-based speech recognition systems**. In 1982 James and Janet Baker (students of Raj Reddy) cofound Dragon Systems, one of the first companies to use Hidden Markov Models in speech recognition.

1991

Tony Robinson publishes work on **neural networks** in ASR  . In 2012 he founded Speechmatics, offering cloud-based speech recognition services. In 2017 the company announced a breakthrough in accelerated new language modelling. By 1994 Robinson's neural network system was in the top 10 in the world in the DARPA Continuous Speech Evaluation trial, while the other nine were HMMs. .

2007

Google begins its first effort at speech recognition came after hiring some researchers from Nuance. By around 2007 itself, LSTM trained by Connectionist Temporal Classification (CTC) started to outperform traditional speech recognition in certain applications.

Sep
2015

Google's speech recognition experiences a performance jump of 49% through CTC-trained LSTM.

Oct
2017

Baidu Research releases Deep Speech 3 which enables end-to-end training using a pre-trained language model. Deep Speech 1 was Baidu's PoC, followed by Deep Speech 2 that demonstrated how models generalise well to different languages.

Nov
2017

Mozilla open sources speech recognition model - DeepSpeech and voice dataset - Common Voice.

Speech Recognition


# SQLite

## Summary

Traditional databases are often difficult to setup, configure and manage. This includes both relational databases (MySQL, Oracle, PostgreSQL) and NoSQL databases (MongoDB). What if you needed a simple database to manage data locally within your application? What if you don't need to manage remote data across the network? What if you don't have lots of clients writing to the database at the same time? This is where SQLite offers a suitable alternative.

SQLite is an in-process library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine. It's open source. The library is compact, using only 600KiB with all features enabled. It can work even in low-memory environments at some expense of performance. It's even faster than direct filesystem I/O. The database is stored as a single file.

## Discussion

### How's SQLite different from traditional databases?

Traditional databases are based on the client-server architecture. Database files are not directly accessed by client applications but via a database server. SQLite takes a different approach. There's no server involved. The entire database–all tables, indices, triggers, views–is stored in a single file.

By eliminating the server, SQLite eliminates complexity. There's no need for multitasking or inter-process communication support from the OS. SQLite only requires read/write to some storage. Taking backup is just a file-copy operation. The file can be copied to any other platform be it 32-bit, 64-bit, little-endian or big-endian.

While SQLite doesn't adopt a client-server architecture, it's common to call applications that read/write to SQLite databases as "SQLite clients".

### What are the use cases where SQLite is a suitable database?

For embedded devices, particularly for resource-constrained IoT devices, SQLite is a good fit. It has a small code footprint, uses memory and disk space efficiently, is reliable and requires little maintenance. Because it comes with a command line interface, it can be used for analysis on large datasets. Even in enterprises, SQLite can stand in for traditional databases for testing, for prototyping or as a local cache that can make the application more resilient to network outages.

Using SQLite directly over a networked file system is not recommended. It can still be used for web applications if managed by a web server. It's suited for small or medium websites that receive less than 100K hits/day.

Applications can use SQLite instead of file access commands such as `fopen`, `fread` and `fwrite`. These commands are often used to manage various file formats such as XML, JSON or CSV, and there's a need to write parsers for these. SQLite removes this extra work. Because SQLite packs data efficiently, it's faster than these commands. It's been noted that,

> SQLite does not compete with client/server databases. SQLite competes with fopen().

### Where shouldn't I use SQLite?

SQLite is not suitable for large datasets, such as exceeding 128 TiB. It's also not suited for high-volume websites, particularly for write-intensive apps. SQLite supports concurrent reads but writes are serialized. Each write typically locks database access for a few dozen milliseconds. If more concurrency is desired due to many concurrent clients, SQLite is not suitable.

If a server-side database is desired, traditional databases such as MS SQL, Oracle or MySQL have intrinsic support for multi-core and multi-CPU architectures, user management and stored procedures.

### How's the adoption of SQLite?

It's been claimed that SQLite is the most widely deployed database in the world with over one trillion database instances worldwide. It's present in every Android device, every iOS device, every Mac device, every Windows 10 device, every Firefox/Chrome/Safari browser, every Skype/Dropbox/iTunes client, and most set-top boxes and television sets.

Going by DB-Engines Ranking that's updated monthly, in January 2019, SQLite was seen in tenth position.

Some users of SQLite include Adobe, Airbus, Apple, Bentley, Bosch, Dropbox, Facebook, General Electric, Google, Intuit, Library of Congress, McAfee, and Microsoft.

### What tools are available for working with SQLite?

SQLite comes with a command line interface (CLI) called `sqlite3` to create, modify and query an SQLite database. Another useful CLI tool is `sqlite3_analyzer` that tells about memory usage by tables and indices.

For those who prefer a graphical user interface (GUI), there are a number of utilities: SQLite Database Browser, SQLite Studio, SQLite Workbench, TablePlus, Adminer, DBeaver, DbVisualizer and FlySpeed SQL Query. Most are free and cross-platform with lots of useful features. DbVisualizer has Free and Pro editions. Navicat for SQLite is a paid product. Adminer is implemented in a single PHP file to be executed by a web server.

To embed SQLite into your own application, there are dozens of language bindings including C, C++, PHP, Python, Java, Go, MATLAB, and many more. SQLite's inventor noted that it was designed from the beginning to be used with Tcl. Tcl bindings have been present even before version 1.0 and regression tests are written in Tcl.

### What's the architecture of SQLite?

SQLite compiles SQL statements into bytecode, which runs on a virtual machine. To generate the bytecode, SQL statements go via the tokenizer (identify tokens), parser (associate meaning to tokens based on context) and finally to the code generator (generate the bytecode).

SQLite itself is implemented in C. SQL functions are implemented as callbacks to C routines. In terms of lines of code, the code generator is the biggest followed by the virtual machine.

Database itself is organized as a **B-Tree**. Each table and index has a separate tree but all trees are stored on the same file. Information from disk is accessed in default page size of 4096 bytes. Pager does the reading, writing and caching. It also does rollback, atomic commit and locking of the file.

### How's the performance of SQLite?

SQLite is 35% faster than filesystem I/O. When holding 10 KB blobs, an SQLite file uses 20% less disk space.

In one study, SQL CE 4.0 was compared against SQLite 3.6. SQLite fared slightly better for read operation but for everything else SQL CE was significantly faster. However, the hardware was a Dell Workstation with two Intel Xeons and 8 GB of RAM.

When compared against Realm, SQLite performed better with inserts but with counts and queries Realm was faster.

In Python environment, one study compared `pandas` with `sqlite`. SQLite was used in two variants: file database and in-memory database. SQLite performed better with select, filter and sort operations. Pandas did better with group-by, load and join operations. There was no great performance gain with in-memory SQLite as compared to file-based SQLite.

### What are the limits and limitations SQLite?

Limits are documented on the SQLite's official website. We mention a few of them:

    + Maximum database size is 128 TiB or 140 TB.
    + Maximum number of tables in a schema is 2147483646.
    + Maximum of 64 tables can be used in a JOIN.
    + Maximum number of rows in a table is 264.
    + Maximum number of columns in a table/index/view is 1000 but can be increased to 32767.
    + Maximum string/BLOB length is 1 billion bytes by default but this can be increased to 231-1.
    + Maximum length of an SQL statement is 1 million bytes but can be increased to 1073741824.SQLite implements SQL with some omissions. For example, RIGHT OUTER JOIN and FULL OUTER JOIN are not supported. There's partial support for ALTER TABLE. GRANT and REVOKE commands are not supported. This implies there's no user management support. BOOLEAN and DATETIME are examples of data types missing in SQLite. AUTOINCREMENT doesn't work the same way as in MySQL. SQLite has a flexible type system (string can be assigned to an integer column) but from the viewpoint of static typing this can be seen as a limitation.

## Milestones

Aug
2000

Version 1.0 of SQLite is released. The earliest public release of alpha code dates back to May 2000. D. Richard Hipp creates SQLite from a need to have a database that didn't need a database server or a database administrator. The syntax and semantics are based on PostgreSQL 6.5.

Sep
2001

Version 2.0.0 of SQLite is released. While 1.0 was based on GNU Database Manager (gdbm), 2.0.0 uses a custom B-tree implementation.

May
2003

Support for **in-memory database** is added in version 2.8.1. This is useful when performance is important and data persistence is not required. Data is lost when the database connection is closed.

Sep
2004

Version 3.0.7 of SQLite is released. An alpha release of 3.0.0 was available earlier in June 2004.

Jan
2006

In version 3.3.0, a **shared-cache** mode is introduced in which multiple connections from the same thread can share the same data and schema cache. This reduces IO access and memory requirements. In version 3.5.0 (September 2007), this is extended to an entire process rather than just a thread.

Sep
2006

Python standard library version 2.5 includes SQLite under the package name `sqlite3`.

Oct
2008

From version 3.6.4, SQLite language syntax is described as **syntax diagrams** rather than BNF notation.

Dec
2018

Version 3.26.0 of SQLite is released.

SQLite


# Decision Trees for Machine Learning

## Summary

In machine learning, we use past data to predict a future state. When data is labelled based on a desired attribute, we call it *supervised learning*. There are many algorithms facilitating such a learning. **Decision tree** is one such. Decision tree is a directed graph where nodes correspond to some test on attributes, branch represents an outcome of a test and a leaf corresponds to a class label.

## Discussion

### Could you explain with an example how a decision tree is formed?

Assuming sufficient data has been collected, suppose we want to predict if an individual has risk of high blood pressure. We have two classes, namely, high BP and normal BP. We consider two attributes that may influence BP, overweight and extent of exercise.

The first decision node (topmost in the tree) could be either weight or extent of exercise. Either way, we also decide the threshold by which data is split at the decision node. We could further split the branches based on the second attribute. The idea is to arrive at a grouping that clearly splits those at risk of high BP and those who are not. In the figure, we note that being overweight clearly separates individuals at high risk. Among those who are not overweight, exercise attribute must be used to arrive at a suitable separation.

The process of choosing the decision nodes is iterative. The leaf node is chosen when the data in the node favours one class over all other classes.

### How is information entropy related to decision trees?

Entropy measures the uncertainty present in a random variable X. For example, an unbiased coin can either show head or tail when tossed. It has maximum entropy. If the coin is biased, say towards head, then it has less entropy since we already know that head is more likely to occur. The uncertainty is quantified in the range of 0 (no randomness) and 1 (absolute randomness). If there is no randomness, the variable does not hold any information.

Consider a binary classification problem with only two classes, positive and negative class. If all samples are positive or all are negative, then entropy is zero or low. If half of the records are of positive class and half are of negative class, then entropy is one or high. Mathematically, entropy is defined as

$$H(X) = - \sum\_{i=1}^n p\_i(x) \* log \_2 p\_i(x)$$

### What is Gini Impurity?

Gini Impurity (also called *Gini Index*) is an alternative to entropy that helps us choose attributes by which we can split the data. It measures the probability of incorrectly identifying a class. Lower the Gini Index, better it is for the split. The idea is to lower the uncertainty and therefore get better in classification.

Mathematically, Gini Impurity is defined as

$$Gini(X) = 1 - \sum\_{i=1}^n p\_i(x) ^2$$

The choice of attribute between 2 classes, \(x\_1\) and \(x\_2\), with \(N\_1\) and \(N\_2\) as number of instances, and \(N=N\_1+N\_2\) is calculated from

$$Gini(X) = \frac{N\_1}{N} \* Gini(x\_1) + \frac{N\_2}{N} \* Gini(x\_2)$$

where

$$ Gini(x1)=1 - p(x\_1) ^2$$

### What is information gain?

Information gain is change in entropy, prior to split and post split, with respect to selected attribute. We select that attribute for splitting data if it gives a high information gain. Information gain is also called as **Kullback-Leibler Divergence**.

Information Gain IG(S,A) for a set S is the effective change in entropy after deciding on a particular attribute A. Mathematically,

$$IG(S,A)=H(S)-H(S,A)$$

### What are the pros and cons of decision trees?

Among the pros are,

    + Decision trees are easily interpretable since it mirrors human decision-making process and a white box model.
    + Decision trees can be used for both regression (continuous variables) and classification (categorical variables).
    + Decision trees are non-Parametric. This means we make no assumption on linearity or normality of data.Among the cons are,

    + Small changes to data may alter the tree and therefore the results. Decision Trees are therefore not robust.
    + When a tree grows large and complex, it tends to overfit. This is overcome by limiting tree depth, boosting or bagging.

## Milestones

1963

Morgan and Sonquist propose **Automatic Interaction Detection (AID)**, the first *regression tree* algorithm. This recursively splits data depending on impurity and stops splitting on reaching a certain level of impurity.

1972

Messenger and Mandell propose **Theta Automatic Interaction Detection (THAID)** , the first *classification tree* algorithm. This recursively splits data to maximize cases in modal category.

1980

Kass propose **Chi-square Automatic Interaction Detection (CHAID)** that splits data into nodes to start with and then merge nodes that have non-significant splits. This is quicker but can be inaccurate.

1984

Researchers improvise AID and THAID to arrive at **Classification and Regression Trees (CART)**. This is done to improve accuracy by pruning the tree. The CART also gives variable importance scores.

1986

Qunilan proposes **Iterative Dichotomiser (ID3)** where he uses *information entropy* to quantify impurity. This is then used to calculate gain ratio for *two-class decisions*.

1993

As an extension of ID3, Quinlan proposes **C4.5**. This can handle *multiple-class decisions*.

Decision Trees for Machine Learning


# Backward Compatibility

## Summary

Technology is rarely static. Technical specifications and products based on them evolve continually. When new versions are released, we can't simply throw away old versions and ask everyone to upgrade. There's a need for new versions to interwork with older versions. This is what we mean by **backward compatibility**.

Industry bodies that define standards must consider backward compatibility. Within companies, when technologies and products are evolved, product managers, architects and developers must consider backward compatibility. Backward compatibility ensures that customers can continue to use current products and services without being forced to upgrade. Customers can plan a phased migration towards latest versions.

Backward compatibility is not without its critics. It has the negative effect of prolonging older technologies. Industry may become reluctant in adopting latest technologies unless they offer significant benefits. There's also an additional cost for designing and maintaining backward-compatible implementations.

## Discussion

### Which technology layers need to consider backward compatibility?

Backward compatibility is applicable at both hardware and software levels. In **hardware**, it's about pin compatibility so that old peripherals can interface with newer versions of the system hardware. It can also be about clock synchronization, signal levels or timing so that different hardware components can interwork correctly.

In **software**, backward compatibility is defined for open or public interfaces of software components or products. This is essential when products are being built from many third-party components. At the **system level**, newer hardware should support older software.

Modern software often rely on web APIs for specific functionality. The **API layer** needs to be backward compatible so that calling a newer version of the API doesn't require changes in client software.  In general, any communication **protocol** (including for instance ISO, IETF or IEEE standards) must be evolved in a backward-compatible manner.

Backward compatibility applies to the **data layer** as well. Data is stored, encoded and decoded in specific formats and versions. However, as data readers and processing tools evolve, they must be able to read and understand older formats. This also applies to how **database schemas** are evolved.

### What are some examples of backward compatibility?

The Raspberry Pi hardware has evolved through many revisions. The functions of the 26 pins present in the original Model 1A were changed in a backward-compatible manner, that is, Do Not Connect (DNC) pins are mapped to 5V, 3.3V or GND; and other pins are unchanged. This means that peripherals that could connect to Model 1A via the header can also connect to Model 4B.  From a software perspective, recent releases of 32-bit Raspberry Pi OS will work on Model 1A as well.

The success of Palm Pilot is partly attributed to its ability to hot sync with Microsoft Windows and connect to PCs via USB ports. By supporting existing interfaces, Palm Pilot enabled vendors and users adopt it more easily.

Microsoft Office 2007 introduced a new format to save documents and spreadsheets: `*.docx` and `*.xlsx` that are essentially based on XML. However, Office 2007 can read from and save to `*.doc` and `*.xls` legacy formats.

In cellular telecom, CDMA2000 1xEV-DO and CDMA2000 1xEV-DV can interwork with CDMA2000 1X and cdmaOne.

### What's the difference between backward and forward compatibility?

Consider a client-server example in which different versions of clients and servers operate. If the new server version works with all clients that worked with the old server version, then it's backward compatible. If the new server version will continue to work with all clients in any future update, then it's forward compatible. In the figure, \(S\) is backward compatible with \(S^{-1}\) and forward compatible with \(S^{+1}\). We could say that,

> When a product or interface is said to be forward compatible, it is really a statement of intent that future versions of the product will be designed to be backward compatible with this one.

Forward and backward compatibility are also called upward and downward compatibility, though some sources apply the former terms to versions of the same entity and the latter terms to communicating entities.

New software that can read data in older formats is considered backward compatible. Old software that can read newer formats (by skipping unknown fields) is forward compatible.  But it's a matter of perspective. We can say that the new format itself is backward compatible.

### What are the different types backward compatibility?

Source compatibility means that software compiles even when some dependencies have been updated to a newer version. With binary compatibility, code compiles and also links to newer components. Wire compatibility is applicable to communication protocols. It means that two entities (such as client and server) can communicate even when they're running different protocol versions. Semantic compatibility implies that communicating or interfaced entities process or interpret the messages correctly.

When products, protocols or standards evolve, we talk about each version being backward compatible with earlier versions. Another kind of backward compatibility is about new technologies interworking with older ones. When CSS came out, it allowed developers to continue styling concrete elements in HTML documents.

### What are the main techniques to achieve backward compatibility?

It's okay to add new interfaces, new operations to existing interfaces or new optional parameters to existing operations. Anything else runs the risk of breaking backward compatibility. In particular, avoid changing the order of enumeration values, removing or renaming operations, and adding mandatory parameters to existing operations.  Don't change field semantics or make input validation more restrictive.

Avoid versioning. Instead prefer to add a new resource or new endpoint. Where not possible, server deployments can run multiple versions in parallel and retire older versions gracefully. Upgraded servers should check for client support before invoking new features. When clients upgrade to newer interface versions, they should support all mandatory features. Communicating entities can negotiate revision levels and ignore features that are not mutually understood. Some call this an **etiquette standard**.

Following Postel's Law, clients can be conservative with requests and tolerant of API extensions. However, against Postel's Law, servers should return useful feedback for unknown or invalid inputs.

Obey technology-specific rules. For example, message types in Protocol Buffers can be evolved in a backward-compatible manner by following specific rules.

### What's the cost of making technology backward compatible?

In the world of microservices, it's common for a service to maintain multiple versions of the API. Older versions are deprecated but clients continue to use them over a long period of time. This creates a technical debt, particularly for integration tests.

Backward compatibility can come in the way of innovation. The need to continue with older technologies and make them coexist with newer technologies becomes a challenge. The final solution would also be more expensive for both vendors and consumers. Sony's PS3 is an example. It included additional hardware for backward compatibility with PS1 and PS2. Back in the early 1990s, Europeans invented GSM without attempting to make it backward compatible with various standards. This allowed them to apply state-of-the-art technologies.

Backward compatibility can lead to less efficient solutions. Reduced codec efficiency was shown in a study of G.722 when extended for super-wideband.

## Milestones

1950

The Radio Corporation of America (RCA) introduces the first **backward-compatible colour television system**. In 1952, this is accepted as a standard for broadcast television in the U.S. by the National Television Systems Committee (NTSC). Information is composed of luminance (brightness) and chrominance (colour) components. Luminance carries the finer details and is allocated more bandwidth. Chrominance component, which consists of hue and saturation, is ignored by B&W TV receivers.

1982

**The Atari 5200** console is launched. However, it's **not backward compatible**, meaning that Atari 2600 games can't be played on Atari 5200. Later, Atari gives way to consumer pressure and creates an adaptor so that 2600 games could be played on the 5200.

1998

Circuit City introduces a new DVD rental format called **DIVX** (aka Digital Video Express). Unfortunately, the DIVX format needs a DIVX player since current DVD players don't understand this format. DIVX players themselves are backward compatible: they can play both DIVX and DVD formats. However, backward compatibility actually backfires for DIVX commercially. In 1999, DVD format continues to be more popular than DIVX.

2003

Under EU laws, Restriction of Hazardous Substances (RoHS) in electrical and electronic equipment is defined. In July 2011, the **RoHS Directive** comes into force. Among the ten substances are lead. During the early transition period, tin-lead components are assembled with lead-free paste (forward compatibility). During the late transition period, some high reliability applications continue to use conventional tin-lead paste with lead-free components (backward compatibility).

2007

**Microsoft releases Office 2007** that supports newer XML-based file formats. However, it's can read and write in older formats and hence **backward compatible**. While the earlier Office 2003 was not designed to be forward compatible, it's possible to update Office 2003 so that it can read newer formats.

2008

The first phone using Android OS is launched. Apps for this OS are built using the **Android SDK**, which is **forward compatible** but not backward compatible. Apps built with a specific `compileSdkVersion` (further qualified by `targetSdkVersion`) will run on newer Android versions as well. This is because old APIs are deprecated but never removed.

2015

Microsoft announces the **Xbox One Backward Compatibility** program. The aim is to run original Xbox and Xbox 360 games on newer Xbox One consoles. By 2019, over 600 Xbox and Xbox 360 games could be played on newer generation hardware including the use of newer hardware features. With Xbox Series X and S, backward compatibility becomes an essential requirement. This also helps in preserving older games. Moreover, legacy games benefit from the latest hardware with features such as Auto HDR and doubled framerate.

Backward Compatibility