# PyTorch

## Summary

To build an artificial neural network from scratch for machine learning models is not a trivial task. One approach is to use a library that simplifies many of the common tasks. PyTorch is a Python-based library for machine learning.

PyTorch was designed to be both user friendly and performant. Python programmers will find it easy to learn PyTorch since the programming style is *pythonic*. While PyTorch provides many ready-to-use packages and modules, developers can also customize them. It's been said that,

> Every aspect of PyTorch is a regular Python program under the full control of its user.

PyTorch is open source with an active community developing it. It's a popular choice for research work, as seen in its growing adoption in 2019.

## Discussion

### Why should I learn PyTorch and what are its benefits?

An essential Python package for numerical and scientific computing is NumPy. However, NumPy is unable to use the power of GPUs. PyTorch allows us to use GPUs while also giving a number of useful machine learning architectures and utilities out of the box.

For Python programmers, writing and reading PyTorch code is very much like that of Python code. It also integrates easily with other Python packages. PyTorch API is intuitive and easy to learn. It's easy to experiment and learn with PyTorch. There's no need to spend hours reading its documentation.

PyTorch combines the best of usability and speed. It's imperative, Pythonic and easy to debug. It offers GPU acceleration with automatic differentiation. Complexity of ML modelling is hidden behind intuitive APIs. For performance, most of it is written in C++. Via YAML metadata files, new language bindings can be quickly created.

### How does PyTorch compare with other deep learning libraries?

An alternative to PyTorch is TensorFlow, although PyTorch remains a popular choice for research. It's been said that TensorFlow is better for production. With eager execution (inspired by Chainer) and Keras integration, TensorFlow is becoming acceptable for research as well. In other words, with respect to ease of use and debugging, TensorFlow is becoming as good as PyTorch.

Some production environments may not have a Python runtime. Sometimes ML models have to be embedded in constrained devices such as smartphones. Updates to models should be seamless without any downtime. TensorFlow has addressed all these considerations. PyTorch is addressing these production considerations via a subset of Python called **TorchScript**. TorchScript captures the structure of PyTorch programs whereas a JIT compiler uses that structure to optimize. In addition, PyTorch has announced experimental support for quantization and mobile.

### What are main features of PyTorch?

We note these features of PyTorch:

    + **Usability and Speed**: With eager mode, PyTorch provides flexibility for research. Via TorchScript, models can be converted to graph mode for speed and optimization. For efficiency, most of PyTorch is implemented in C++.
    + **Automatic Differentiation**: Gradient computations are easy to perform. Via operator overloading, PyTorch builds up a representation of the computed function every time it is executed.
    + **Distributed Training**: There's native support for asynchronous execution and peer-to-peer communication.
    + **Mobile Deployment**: ML models can be deployed in mobile applications.
    + **Interoperability**: PyTorch can export models in ONNX format that can then be imported into ONNX-compatible platforms, runtimes and visualizers. Moreover, it's easy convert between PyTorch tensors and NumPy arrays.
    + **Extensibility**: It's easy to add custom behaviour. For example, automatic differentiation can be customized by deriving from `torch.autograd.Function` and implementing `forward()` and `backward()` methods.
    + **C++ Frontend**: For bare metal C++ applications, this provides an alternative to Python frontend. It enables higher performance and lower latency.
    + **Quantization**: Store and manipulate tensors at lower bit-widths instead of floating-point precision. It's done mainly to improve performance during inference.
    + **Cloud Support**: Popular cloud platforms support PyTorch including AWS, GCP and Azure.

### Which are main PyTorch packages and modules?

The main package is called `torch`. It contains `torch.Tensor`, which is the main data structure to store multi-dimensional tensors. Operations on tensors are also defined in this package.

The package `torch.nn` is useful for building neural networks. To create a NN model, we need to subclass `torch.nn.Module`. NN layers that are available include `torch.nn.Conv1d`, `torch.nn.MaxPool1d`, `torch.nn.Sigmoid`, `torch.nn.BatchNorm1d`, `torch.nn.LSTM`, `torch.nn.Linear`, `nn.Dropout`, `nn.MSELoss`, and many more. Functional equivalents of these are in `torch.nn.functional`.

Neural network weights and biases are adjusted via optimizers to minimize the loss. The package `torch.optim` provide many optimizers including `torch.optim.SGD` and `torch.optim.Adam`. Learning rates can be adjusted via the module `torch.optim.lr_scheduler`.

Another useful module is `torch.utils.data` in which `Dataset` and `DataLoader` are important classes. Among other things, they're useful for batching data and distributing them to multiple workers.

Three application-specific packages are `torchvision`, `torchaudio` and `torchtext`. For example, `torchvision.datasets` gives easy access to image datasets and `torchvision.transforms` encapsulates useful transforms.

### Could you give an example of a neural network model in PyTorch?

The example in the figure shows how to build a NN model by creating a subclass of `torch.nn.Model` class. The model is initialized with a convolutional layer and a linear layer. While PyTorch has `torch.nn.Linear`, this example shows how easy it is to build a custom linear layer. Essentially, the model is implemented as a class whose members are the model's layers.

The model itself is evaluated on an input activation by calling the `forward()` method. We specify in this method how the layers are connected. In this example, there's a non-linear ReLU activation between the two layers. There's a softmax layer at the end. The backward pass happens in `backward()` method but this is supplied automatically by PyTorch's `autograd` module.

Loss is computed by the caller based on the output of `forward()`. Gradients are calculated next during the backward pass. Finally, optimizer is invoked to adjust the model's parameters.

### How do I use GPUs with PyTorch?

To keep track of and use GPUs, `torch.cuda` is the module to use. We can check if GPUs are available using `torch.cuda.is_available()`. If so, call `torch.device('cuda')` to select a GPU.

When tensors are allocated to a GPU, operations and their results will be on the same GPU device. Cross-GPU operations are not allowed by default but this is possible when peer-to-peer memory access is enabled. Data can be copied or moved from one GPU to another. To use multiple GPUs for distributed training, consider using `nn.DataParallel`.

GPU operations are asynchronous. Operations are queued up for a particular device. CPU can continue running and queueing operations to other GPUs. All of this is transparent to the programmer. It's possible to force synchronous GPU calls, which can be useful to debug errors.

A linear sequence of execution on a particular device is called a **CUDA Stream**. Every device has a default stream but we can create new streams. Operations within a stream are executed in the order they were queued. Since streams execute concurrently, operations across streams can be in any order. It's possible to synchronize across streams using for example `synchronize()` or `wait_stream()`.

### What are some useful resources to get started with PyTorch?

The official PyTorch documentation is an essential reference. Beginners can start with step-by-step tutorials. The official PyTorch Resources page includes links to discussion forums, Slack channels. and how to get started with PyTorch on cloud platforms.

Beginners may also refer to a handy cheatsheet of PyTorch modules and how to use them.

A curated list of applications and models written in PyTorch can be useful for developers who like to learn by examples.

TechRepublic has published a useful list of books on PyTorch.

NVIDIA provides PyTorch container images that include NVIDIA CUDA, NVIDIA cuDNN, TensorBoard, TensorRT and optimized examples of well-known models.

## Milestones

Oct
2002

As a modular machine learning library, **Torch** is released under free BSD license. With new ML algorithms being proposed and presented at conferences, a tool such as Torch can help in implementing and comparing them. Torch is implemented in C++. It's modular because simple modules can be combined to create complex models. For example, a multi-layer perceptron can be realized with two linear modules (for input and output) with a non-linear hidden layer (such as `tanh`) in between.

2011

**Torch7** is released as a framework for numeric computing and machine learning. It's based on Lua, which is a fast interpreted language with a good C API. Since Lua is written in ANSI C, it can be compiled for various target platforms. Torch7 comes with 8 built-in packages. It also supports parallelization with OpenMP and CUDA.

Mar
2015

**Version 1.0 of autograd** is released on GitHub. First commit of this package is from November 2014. Maclaurin, in his doctoral thesis, describes autograd in detail. This is a Python implementation of **Automatic Differentiation (AD)**. It can compute gradients on any NumPy code. He also notes that autograd has become quite popular within the machine learning community. A 2017 survey paper credits autograd, Chainer and PyTorch for popularizing AD.

Sep
2016

A week after alpha-0, **alpha-1 release of PyTorch** appears on GitHub. This includes `torch.nn`, `torch.autograd`, `torch.optim`, `torch.load` and `torch.save`. On MNIST dataset, PyTorch runs as fast as Torch-Lua. PyTorch uses 1500MB of system memory whereas Torch-Lua uses 2300MB.

Jan
2017

**PyTorch** gets its first public beta release. It's early development is guided by Soumith Chintala, a core developer of Torch. It starts as a fork of Chainer that has dynamic graphs and interpretable development environment. Though Torch is popular and even accepted by organizations such as Facebook AI Research, Lua is not as mainstream as Python. Other reasons to move to Python are easy debugging and an imperative-style framework.

Apr
2018

Caffe2 is merged into PyTorch codebase. Caffe2 and PyTorch are both open source ML frameworks from Facebook.

Dec
2018

Version 1.0.0 of PyTorch is released. This comes with a JIT compiler and **TorchScript**, which is a subset of Python. Due to PyTorch's Intermediate Representation (IR), models can be optimized and deployed in non-Python environments. This release introduces `nn.distributed` package. This release also includes **Torch Hub**, a repository for pre-trained models.

2019

PyTorch adoption grows a lot within the research community. Among the many deep learning frameworks, only PyTorch and TensorFlow seem to matter most. This year also sees a number of regular PyTorch releases from v1.0.1 (February) to v1.3.1 (November). By now, while PyTorch is growing, Torch community is defunct.

Jan
2020

**PyTorch 1.4** is released. This allows customized builds for PyTorch Mobile. As an experimental feature, there's an RPC framework for distributed model parallel training. Also experimental are Java bindings. In February, NVIDIA releases a container image for this version of PyTorch.

Sep
2022

Meta announces the formation of **PyTorch Foundation**, under the Linux Foundation. Cloud companies (AWS, Google Cloud) and hardware vendors (AMD, NVDIA) are some of the other members of the Foundation. The Foundation will market PyTorch and ecosystem around it. The Foundation's motto is "to drive adoption of AI tooling by fostering and sustaining an ecosystem of open source, vendor-neutral projects with PyTorch."

PyTorch


# Java Modifiers

## Summary

A modifier is a programming construct in Java used to modify/refine/restrict a declaration. With modifiers, developers can restrict access, limit class instantiation to a single instance, disallow value modification, control persistent storage, configure sharing of variables across threads, and more. Modifiers can be applied to classes, class fields, class methods, class constructors and interfaces.

Most modifiers are keywords, meaning that developers can't use them as identifiers. Modifiers `sealed` and `non-sealed` are not keywords.

In some cases, default modifiers are applied when modifiers are not explicitly specified by the developer. However, it's a good practice to explicitly specify modifiers.

Annotations are considered as modifiers  but these are not covered in this article.

## Discussion

### Could you explain Java's access modifiers?

Access to a class, interface, field, method or constructor can be controlled via access modifiers. Any one of these three can be used:

    + `public`: access to everyone
    + `protected`: access to subclasses anywhere plus everyone within the package
    + `private`: access limited to the classWhen no access modifier is specified, by default access is restricted to within the package; that is, anyone within the package can access. This is also called *package-private*. The Java Language Specification (section 6.6) gives more details.

Broadly, there are two levels of access control: (a) top level: public or package-private; (b) member level: public, protected, package-private, private. Top level applies to classes and interfaces. Member level applies to members of classes and interfaces, including nested classes and interfaces.

As a best practice, members should be private unless there's a need to expose them. This hides the internal implementation. Except for constants, avoid public fields. Access to top level public classes and interfaces can be restricted to the module by not exporting them.

### Could you explain Java's static modifier?

There are use cases where we wish that all instances of a class share fields rather than have separate copies. Logging is an example where multiple instances log via the same logger. Another example is the method `public static main(String[] args)` that forms the entry point for any program. The JVM starts execution from this method even when no instance has been created. The `static` modifier enables us to do this.

Nested classes can be static. Local or anonymous classes can't be static. Nested enum classes and nested record classes are implicitly static.

A static field/method is also called a **class variable/method**. In a static context, the use of `this` and `super` are not allowed. Static fields can be initialized using field initializers or static initializers that may include one or more blocks of code.

A nested interface, member or local, is implicitly static.

Static fields/methods are preferably accessed via the class name. Static methods can't be overridden since they're resolved at compile time. Static methods can't be abstract.

### Which modifiers affect inheritance in Java?

A subclass or subinterface can extend or implement from another class or one or more interfaces. This is one way by which Java enables code reuse. However, modifiers `final`, `sealed` and `non-sealed` help developers control inheritance. These modifiers may be used towards a more maintainable code. For example, a `Vehicle` class may state that only `Car` and `Truck` subclasses can extend it. This avoids the inelegant use of `instanceof` or `if-else` statements. Thus, runtime checks are replaced with compile-time checks.

A class is declared `final` when it's desired that no subclass can extend it. It's methods can't be overridden. A final class can't be abstract.

A class/interface is declared `sealed` if it specifies all known subclasses/subinterfaces. The subclasses/subinterfaces themselves may be `sealed`; or `non-sealed` if they can be freely extended. A permitted subclass must be itself be declared as `final`, `sealed` or `non-sealed`.

### Could you explain the abstract method modifier?

Sometimes we wish to share code among closely related classes. But what's common among these classes is incomplete in the sense that their superclass can't be instantiated. An example is the `area()` method of class `Shape`. Subclasses `Circle` and `Rectangle` can and should implement this method but `Shape` itself is too abstract to implement it.

The `abstract` modifier is applicable to classes, methods and interfaces. However, since every interface is implicitly abstract (since classes are expected to implement an interface), explicit use of this modifier for interfaces is obsolete.

A class with at least one abstract method must itself be declared abstract. A class that partially implements an interface must be declared abstract. A subclass of an abstract class itself can be abstract. In all cases, abstract classes can be subclassed but can't be instantiated. An abstract class can be `sealed` but not made `final`.

### Could you explain transient and volatile field modifiers?

Sometimes there's a need to store an object in a persistent manner, such as into a database or a disk file. Java's `java.io.Serializable` class helps us to serialize the object for storage. But what if there are some fields that we don't wish to save? This is where the `transient` modifier can be used. Fields with this modifier are ignored for storage.

Java allows multiple threads to share variables. Using locks, developers can enforce mutual exclusion on those shared variables. However, the use of `volatile` modifier gives developers an alternative approach. The Java Memory Model ensures that all threads see a consistent value without extra effort from developers. A `final` variable can't also be declared `volatile`.

### Could you explain native and synchronized method modifiers?

Sometimes we wish to reuse functionality already implemented in another language. This may be legacy code, platform-dependent code or code written for better performance or low-level hardware access. The `native` modifier tells the Java compiler that the method is implemented elsewhere. No method body is supplied. Instead the method declaration ends with a semicolon. Native code is interfaced to the JVM via Java Native Interface (JNI).

The `synchronized` modifier helps developers write methods that can run concurrently in a multi-threaded environment. For example, suppose one method puts items into a queue or updates a variable, and another method executing in a concurrent thread reads from the queue or variable. With the use of `synchronized` modifier, method executions will not get pre-empted.

Under the hood, a synchronized method acquires a monitor. For a class method, the monitor is associated with the class. For an instance method, it's associated with `this`.

### Could you explain the Modifier class?

Java's `Modifier` class is part of the `java.lang.reflect` package. Reflection allows a program to get information about classes at runtime. Via static methods and constants, the `Modifier` class gives information about class and method access modifiers. Examples of constants are `ABSTRACT`, `FINAL` and `STATIC`. Examples of methods are `isAbstract(int)`, `isFinal(int)`, `isStatic(int)`, `classModifiers()` and `methodModifiers()`.

An example usage would be to call the `getModifiers()` methods on an instance of `java.lang.Class`, `java.lang.reflect.Field`, `java.lang.reflect.Method` or `java.lang.reflect.Constructor`. The returned value can be processed using the `java.lang.reflect.Modifier` class.

## Milestones

Jan
1996

Java 1.0 is released. This release includes access modifiers `public`, `protected` and `private`; class modifiers `abstract` and `final`; field modifiers `static`, `final`, `transient` and `volatile`; method modifiers `abstract`, `static`, `final`, `native` and `synchronized`; and interface modifier `abstract`. The use of `abstract` modifier for interfaces is redundant since interfaces are abstract by definition.

Dec
1998

Java 1.2 is released. This adds the modifier `strictfp` to enforce strict floating-point semantics regardless of the platform. In Java SE 17 (September 2021), this modifier is made obsolete since by then processors support SSE2 (Streaming SIMD Extensions 2) that natively supports strict floating-point semantics.

Sep
2017

Java SE 9 is released. For better modularity, the **module** concept is introduced. A module's packages are accessible to other modules only if the module exports them and modules that use them explicitly require them. This refines the semantics of the top level public access modifier and nested public and protected modifiers. Moreover, runtime-only access to packages can be achieved via `opens` and `open` directives.

Sep
2020

Java SE 15 is released. This adds modifiers `sealed` and `non-sealed` to give developers control on inheritance hierarchy. Classes and interfaces can be controlled with these modifiers. The `permits` clause specifies which classes and interfaces can extend or implement the current class or interface.

Java Modifiers


# Semantic Web

## Summary

In the original web of the 1990s information was shared as webpages or documents that could be understood by humans. Via hyperlinks, these linked to other webpages or documents anywhere on the web. Servers and desktop computers processed or displayed all this information but didn't understand them. For example, a computer could tell that a particular text is a heading or another text is in italics. It didn't know that the heading is actually the title of a blog post or that the text in italics is the author's name.

Semantic Web is an attempt to describe and link web content in a manner that's meaningful to machines. Semantic Web extends the original web. Semantic Web wants to transform the web from a "web of documents" into a "web of data".

## Discussion

### What's the motivation behind Semantic Web?

By enabling machines to understand data, we can benefit in many ways:

    + **Automation**: We can avoid doing mundane stuff such as booking tickets or rescheduling appointments. These can be efficiently handled by virtual assistants or agents.
    + **Personalization**: Content on the web is growing daily. It's impossible for us to follow everything. Agents can personalize or curate content for us.
    + **Information Retrieval**: Within enterprises or via web search engines, Semantic Web can give us more relevant answers.
    + **Data Reuse**: Because Semantic Web enables linking of data from a variety of sources, data can be reused. Data that was previously stored in isolated databases can now be shared in a standard manner.
    + **Knowledge Discovery**: By linking data across the web, new knowledge can be discovered. Semantic Web enables machines to apply logic on existing relationships and infer new ones. For example, this could be useful in discovering new drugs.

### With the progress of Machine Learning (ML), is Semantic Web relevant?

Semantic Web doesn't make machines intelligent in the sense of Artificial Intelligence or Machine Learning. Instead of asking machines to understand humans, we help machines to solve well-defined problems on well-defined data via well-defined operations.

It therefore does appear that AI/ML has gone ahead and enabled machines to see, hear and speak. The mid-2010s have seen the arrival of voice assistants, chatbots, computer vision applications, and more. This has been possible because of the availability of data to train ML algorithms. Though some ML algorithms require labelled data for training, there's no need to add semantic metadata to all data.

However, Semantic Web complements and even overlaps with AI/ML. Semantic Web can help with explainable AI. Natural Language Processing (NLP) is an application area with chatbots and intelligent assistants using semantically linked data. Semantic Web can add background knowledge to AI/ML systems, particularly in areas where data is scarce. We're also seeing AI/ML being applied to conceptualize domain knowledge for the Semantic Web. AI can fill in missing data or clean noisy data in knowledge graphs.

### What are the basic building blocks of Semantic Web?

Semantic Web builds upon the foundations of the original web. Data (old and new) must be described with metadata. This metadata will identify data, interlink data and relate data to concepts so that machines can understand them.

Data must be uniquely identified and this is done using **Uniform Resource Identifier (URI)** or **Internationalized Resource Identifier (IRI)**. **Resource Description Framework (RDF)** provides the data model. Meaning is added at a higher layer with what we call **ontologies**. In other words, RDF specifies the syntax while ontologies specify the semantics.

Just as HTML was the building block of the original web, RDF is the building block of the Semantic Web. Web content can expose their semantics by embedding RDF statements within webpages. There are many ways to do this: RDFa, RDF-XML, RDF-JSON, JSON-LD, Microdata, etc. Semantic data already processed and stored in RDF format can be queried. Just as MySQL exists to query relational databases, **SPARQL** is a language to query **RDF stores**. Given the semantics, rules can help in applying logic and reasoning.

### What is Resource Description Framework (RDF)?

As the name suggests, RDF helps us describe any resource so long as that resource has a unique identifier. In other words, RDF helps us define data about other data, that is, the metadata.

RDF has three components: subject, predicate, object. It's a statement about the relationship between the subject and the object. Thus, the fact that Villa Nellcôte is located in France can be expressed as an **RDF Triple**. All three parts of the triple are expressed as URIs, literals or blank nodes.

When we combine many such statements together, we get what is called an **RDF Graph**. Subjects and objects are nodes of the graph. Predicates form the connecting arcs. For example, we can state that France is in Europe, Paris is the capital of France, Paris has a population of 2.2 million... Each of these can be expressed as an RDF Triple. Collectively, they form an RDF Graph.

### How do we add meaning to data on the web?

RDF on its own doesn't give meaning to data. RDF is a data model, a method to express relationships. To give meaning, *vocabularies* and *ontologies* are defined. These are typically written in terms of classes, their properties and relationships to other classes.

For example, an RDF triple can express that Paris is the capital of France but for a computer this still makes no sense. A vocabulary can define that capital is a type of city, city belongs to a country, and country is a political entity. This helps the computer to get a sense of the context though it can never truly understand the way humans do.

**RDF Schema (RDFS)** is a simple vocabulary while **Web Ontology Language (OWL)** is more powerful.

### Could you explain the term ontology?

Ontology has a metaphysical meaning but in computer science it refers to a formal description of knowledge. Concepts and their relationships within a specific domain are described. Classes, attributes, and relations such as restrictions, rules and axioms are defined. These represent the knowledge of that domain.

For example, in the domain of education, we can define that a course is taught by an academic staff member; academic staff member is a subclass of staff member; the union of staff member and student is all people at the university; academic staff member is equivalent to faculty; and so on. If we state that John Smith teaches Chemistry 101, using the ontology we can infer that John Smith is a staff member.

Data coming from different sources can rely on ontologies for a common understanding. Ontologies can also help in finding logical inconsistencies, classes that cannot have instances, or different instances that share the same names. One important benefit is that logic and reasoning can be applied to discover new knowledge.

### Could you describe some real-world applications of Semantic Web?

It was reported in 2007 that the oil and gas industry is using RDF/OWL to combine data from diverse sources, and standardize data exchange, sharing and integration across partners or applications. Collaborative knowledge management also became possible.

In April 2010, Facebook launched *Open Graph* that web publishers can use to integrate their web pages into Facebook's social graph. This enables Facebook to understand what a user likes, give personalized recommendations, or connect users with similar interests. A simplified form of RDFa was adopted.

During the FIFA World Cup of 2010, BBC website used semantic web technologies to dynamically display content. SPARQL queries and OWL 2 RL reasoning were employed. With the success of this project, in January 2013, BBC committed to the development of Linked Data Platform to enable dynamic semantic publishing. BBC's Music site from 2008 was also an early example of using the semantic web.

Other examples include causality mining in pharma, semantic web mining, mining health records for insights, and fraud detection. You may also look up W3C's page titled Semantic Web Case Studies and Use Cases for more examples.

## Milestones

1997

W3C produces the first working draft of *RDF Model and Syntax*. This evolves to a 2004 W3C Recommendation titled *Resource Description Framework (RDF): Concepts and Abstract Syntax*.

1998

A working draft of *RDF Schema (RDFS)* is published by W3C. Since RDF itself lacks means to define semantics, RDFS provides a basic type system for use in RDF models. *RDF Schema 1.1* is published in February 2014 as a W3C Recommendation.

May
2001

American science magazine Scientific American publishes an article by the inventor of the web, Tim Berners-Lee. Titled *The Semantic Web*, this article talks about web content that's meaningful to computers. The article narrates futuristic interactions enabled by the semantic web. It discusses knowledge representations, ontologies and agents. Though Berners-Lee had discussed some of these as early as 1994, it's only with this article that the vision of semantic web reaches a wide audience.

2004

W3C publishes **Web Ontology Language (OWL)**. OWL enables complex relationships that are not possible with RDFS. A revision of this (often called "OWL 2") is published in 2009. A second edition of OWL 2 is published in 2012.

2006

Tim Berners-Lee points out that putting data out on the web isn't enough. Data has to be linked to other related data. He proposes what he calls *Linked Open Data (LOD)* and defines a 5-star system to grade how well people share data. He also states that,

> Linked Data is the Semantic Web done right.

Jan
2007

*DBpedia* is created by using structured data available in Wikipedia and representing them as RDF triples. By using a query language such as SPARQL, DBpedia can be used to make semantic queries. As on April 2016, DBpedia contains 9.5 billion RDF triples.

2010

This may be the year when many companies adopt semantic web technologies for commercial use. Examples include Best Buy, BBC World Cup site, Google, Facebook and Flipboard.

Jun
2011

Google, Microsoft, Yahoo and Yandex agree on **Schema.org**, a vocabulary for associating meaning to data on the web. The vocabulary is defined by a community process. Schema.org can be used with various encodings including RDFa, Microdata and JSON-LD. Google itself recommends the use of Schema.org along with JSON-LD.

Oct
2011

Apple integrates voice assistant *Siri* into some of its products. Within the next couple of years, competitive voice assistants *Microsoft Cortana*, *Google Now* and *Amazon Alexa* are released. These are largely powered by machine learning techniques but may also leverage the semantic web technologies.

Sep
2013

Google Search gets the Hummingbird update, which enables the search engine to figure out user's intent. This is beyond searching by keywords alone. This capability comes from semantics attached to data plus Google's Knowledge Graph. Knowledge Graph was announced in May 2012.

2021

Given that 2001 is considered the birth year of Semantic Web (with the publication of the Scientific American article), Semantic Web celebrates its **20th anniversary**. In an article, Hitzler writes that the grand goal of data sharing, discovery, integration and reuse hasn't been achieved. Every sub-field of Semantic Web needs further progress. Given many varied approaches, some application-oriented consolidation is needed with good interworking of tools and well-documented processes.

Semantic Web


# Firebase

## Summary

Modern web and mobile apps are typically built with a mix of backend services and frontend frameworks. Wouldn't it be nice if there was a way for developers to focus on just the frontend and user experience, and let someone else take care of all backend requirements? It's exactly in this context that **Backend-as-a-Service (BaaS)** has emerged.

Firebase started as a BaaS but after its acquisition by Google in 2014, it's now more deeply integrated to some parts of Google Cloud Platform. Since then, Firebase is described as "a comprehensive mobile development platform". The idea is to build apps faster without managing infrastructure.

## Discussion

### Why should I adopt Firebase for my next web/mobile app?

Firebase provides easy-to-use workflows common to many apps. Examples include onboarding flow, customizing the "welcome back" screen, progressively rolling out new features, following the user journey across devices, adding chat to your app, optimizing ads based on user behaviour, enabling users to share and resize photos, processing third-party payments, and more. These typically involve significant development on the server side. Instead, Firebase offers out-of-the-box and easily deployable solutions to do these.

As developers, you don't have to worry about server-side software. Mobile and web SDKs enable this. The main offering is real-time sync of data between client apps and Firebase via **Realtime Database** and **Cloud Firestore**. Using **Authentication**, user authentication feature is easy to build, even if it involves third-party providers such as Facebook or Twitter. Messaging and notifications are efficiently done using **Firebase Cloud Messaging (FCM)**. Unlike native apps updated via app stores, **Remote Config** can update Firebase apps without asking users to update their installation. Using **Crashlytics** and **Performance Monitoring** app stability and performance can be improved. There's good integration with Adwords and Admob. Even UI testing across multiple devices is simplified via **Test Lab**.

### What are the products offered by Firebase?

Firebase products can be grouped into three categories:

    + Build: Helps developers build more powerful, secure and scalable apps. Products include Cloud Firestore, ML Kit, Cloud Functions, Authentication, Hosting, Cloud Storage, Realtime Database.
    + Improve: Gives developers insights into app performance and stability. Products include Crashlytics, Performance Monitoring, Test Lab.
    + Grow: Helps your app to grow. Products include In-App Messaging, Google Analytics, Predictions, A/B Testing, Cloud Messaging, Remote Config, Dynamic Links, App Indexing.

### What exactly is Firebase Hosting?

With the rise of powerful frontend frameworks (Angular) and static site generators (Jekyll) it's become possible to develop web apps without much server-side processing. For hosting such web apps, Firebase Hosting is a suitable solution. It serves content over secure connections, delivers content quickly due to caching on solid-state devices at CDN edges, enables rapid deployment and supports one-click rollbacks. CDN edge servers ensure global availability of your app. SSL certificates are free and automatically provisioned even for custom domains.

It's also easy to deploy a sophisticated Progressive Web App (PWA). While Firebase Hosting serves static content, dynamic content can be served using Cloud Functions for Firebase or Google's Cloud Run.

### What's a suitable deployment architecture when using Firebase?

With Firebase, a web/mobile app can be deployed without running your own backend server. However, Firebase can also co-exist with an app server.

If the app requires only client-side code or doesn't require complex processing, then all the backend needs can be served by Firebase. If the app needs more complex processing or integration with third-party APIs, then Firebase can sit between clients and the app server. Clients access Firebase data and the server listens to changes to the data. Server manipulates data in Firebase, to which clients are notified. In another deployment architecture, Firebase sits logically next to the app server. Clients communicate to both. This architecture is ideal for large legacy apps where some parts (such as real-time features) are served by Firebase while older parts are served by the legacy server code.

Given that Firebase supports server-side execution (Cloud Functions since March 2017, Cloud Run since April 2019 ), it's possible that even when an architecture requires an app server, it could be offloaded to the serverless cloud infrastructure.

### Could you share more details on Firebase Cloud Messaging (FCM)?

FCM a cross-platform messaging solution that can be used for pushing new data to client apps or notifying users. Instant messaging has a payload limit of 2KB. From client to server, acknowledgments, chats, and other messages can be sent.

FCM consists of **FCM servers** managed by Google and an app server or other trusted environment such as **Cloud Functions for Firebase**. All message go via FCM servers. Developers can use raw protocols (HTTP/XMPP) or use Admin SDK. HTTP is synchronous and downstream. XMPP is asynchronous and both downstream and upstream. HTTP supports multicast as well.

In April 2018, Google deprecated its **Google Cloud Messaging (GCM)** service. GCM APIs will be available only till April 2019. Since GCM and FCM cannot coexist in an app, developers must migrate to FCM. FCM inherits GCM infrastructure and adds many new features. For more details, refer to the GCM/FCM FAQ.

### How's the pricing of Firebase products?

Firebase gives following services under free tier: A/B Testing, Analytics, App Indexing, Authentication (except Phone Auth), Cloud Messaging (FCM), Crashlytics, Dynamic Links, Invites, Performance Monitoring, Predictions, and Remote Config. With free tier, you get some free access to other products with limits on storage, bandwidth, number of reads/writes/deletes/uploads/downloads/tests, etc. Realtime Database allows only 100 simultaneous connections.

Besides the free tier, there's a fixed price per month and pay-as-you-go plans. With the latter, the pricing of Realtime Database, Cloud Firestore, Storage, Phone-Auth, Hosting, Test Lab, ML Kit, Use of BigQuery & other IaaS, is based on the use.

### What are some criticisms of Firebase?

Since 2016, developers have noted many problems with Firebase. While it's good for enabling real-time apps with client-side logic, running complex database queries is almost impossible. Firebase Realtime Database is nothing more than a large JSON file. Data is duplicated since the benefits of relationships in a relational database are absent. Filtering or paginating records are not possible.  More recently, the situation has improved with Cloud Firestore that manages data using collections and documents.

Your data is on a server that you don't control. There's no easy way to import/export data. Firebase is also not open source. If the service is discontinued, you have no choice but to rework your app. While initial development may be faster, scaling up later may prove difficult. Moving business logic from client to server won't be trivial. When you scale, the cost of using Firebase will be higher than adopting a platform-as-a-service.

Developers can't work with a local installation. Network connection is required. They're also at the mercy of Firebase's upgrade cycles.

## Milestones

Sep
2011

Firebase starts as a startup named **Envolve**, founded by James Tamplin and Andrew Lee. It provides developers an API to integrate online chat functionality into their websites.

Apr
2012

It's noticed that some apps using Envolve are doing things beyond chat. They use Envolve's capability for synchronizing app data in real time. This prompts its creators to separate chat functionality from the underlying framework that enables real-time communications. The latter could now be offered as a standalone service. Thus, Firebase as a separate company is born.

Oct
2014

Google acquires Firebase. In following years, other Google acquisitions of Fabric/Crashlytics, Divshot, and LaunchKit add value to Firebase portfolio of products.

May
2016

Google announces Firebase at Google I/O as a unified platform for mobile developers. Firebase now integrates deeply with some of Google cloud tools. The user base of Firebase is now 470,000, up from about 110,000 when Google acquired it.

Mar
2017

Firebase releases **Cloud Functions for Firebase**. This makes is possible to execute JavaScript (Node.js) on the server side, on Google's cloud infrastructure. Cloud Functions has integrations with many components of Firebase. Projects that needed to deploy servers until now, can migrate to Cloud Functions and become completely server-free.

Oct
2017

Firebase team announces a beta release of **Cloud Firestore**, a fully managed NoSQL document database. Built in close collaboration with Google Cloud Platform team, it's replicated across regions and offers strong consistency. However, for special use cases where cost and latency are important, Firebase Realtime Database may be preferred.

May
2018

As a beta release, **ML Kit** becomes available on Firebase. This allows developers to add machine learning features into their apps. Use cases supported are recognizing text, detecting faces, scanning barcodes, labelling images and recognizing landmarks. For advanced use cases, developers can also import *TensorFlow Lite* models.

Oct
2018

It's reported that over 1.5 million apps are using Firebase every month. India's *Hotstar* entertainment app is a Firebase-based app with over 150 million monthly active users.

Apr
2019

Firebase announces integration with Google's **Cloud Run** service. Whereas Cloud Functions is limited to Node.js backend, Cloud Run enables developers write server-side functionality in any language. This requires a docker image that can respond to HTTP requests on the configured port.

Sep
2019

At the fourth annual Firebase Summit, Firebase launches Firebase Extensions, Firebase App Distribution and integration with Google Analytics. **Firebase Extensions** could to be a time saver, particularly for independent developers. These extensions implement common tasks. Examples include image resizer, URL shortener and text translation. As on August 2021, there are 11 extensions and many more experimental extensions.

Firebase


# Pair Programming

## Summary

A developer typically writes code alone. Pair programming is a practice in which two developers are paired together to jointly complete a task. The task could be software design, algorithm, coding or testing. The rationale is that two minds are better than one. If done correctly, pair programming yields better software faster and at lower cost.

The pair share the same computer and possibly share a single keyboard. Each developer of the pair has a specific role but roles alternate often. Moreover, pairs are not static. Any two developers in the team could become a pair for a few hours.

Pair programming is an essential practice in Extreme Programming. *Remote pair programming* is a specialization where developers can be at different locations. *Peer programming* is a term that's sometimes used when more than two developers are involved.

## Discussion

### Why should I adopt pair programming?

Among the many benefits of pair programming are faster onboarding of employees, better collaboration within the team, faster processing of pull requests (where code changes are reviewed and accepted), fewer bugs and ultimately better quality software. Pair programming also fits nicely into modern development pipelines such as CI/CD and truck-based development.

Codebase is more consistent with coding guidelines due to collaboration. Solutions are better designed. Where problems are challenging, pair programming finds solutions faster. Due to increased social interaction, there's greater job satisfaction and higher productivity.

Ultimately, pair programming is not just about writing code. It's also about planning the work, discussing ideas, clarifying viewpoints and coming up with better solutions.

### What are the different styles of pair programming?

A few types of pairings exist:

    + **Driver-Navigator**: One developer (driver) writes the code while the other (navigator) observes, asks questions, and thinks ahead. Driver explains her actions so that the navigator understands. Navigator is essentially doing a real-time code review.
    + **Driving School**: Aka *backseat navigator* or *strong-style*. Navigator takes more control and gives specific instructions such as "create a method" or "create a new file".
    + **Tour Guide**: The tour guide (driver) providing strategic and tactical thinking while also typing. The other developer is like a tourist, listening passively. In the long term, this style is not favoured since we learn better by doing than by observing.
    + **Ping-Pong**: Suited for Test-Driven Development (TDD) methodology. One developer writes the test while the other writes the code to make the test pass. Then they switch roles.
    + **Pomodoro**: Similar to ping-pong style but based on time limits. Pairing is for 25 minutes, followed by a 5-minute break. Then developers switch roles. After four such sessions, there's a longer 20-minute break. Breaks keep the sessions productive.
    + **Tag-Team**: Switching of roles can happen anytime for any valid reason.

### How is strong-style different from driver-navigator style of pairing?

In traditional driver-navigator style of pairing, the developer who gets an idea grabs the keyboard and takes the role of the driver. If the navigator doesn't know where the driver is going (due to lack of clear communication), the navigator may become bored and disengaged.

In strong-style, the person with the idea takes the role of the navigator instead and guides the driver. Thus, both developers are actively involved. There's greater trust and knowledge sharing.

In strong-style, the navigator gives high-level commands and the driver implements them at a low level. To ensure understanding, the navigator adjusts the level of abstraction. If the driver doesn't understand, the navigator provides more detailed commands. The driver trusts the navigator. The driver may sometimes work with partial understanding so that the navigator's flow is not broken. At a suitable pause in the flow, he may asks questions. It's been said of this style that,

> For an idea to go from your head into the computer it MUST go through someone else's hands.

### What underlying mechanisms make pair programming effective?

In pair programming developers are required to discuss the code and ask questions. Asking the right questions can challenge assumptions and trigger new perspectives. Pair programming is therefore not just coding but also active learning. Vocalizing our thoughts helps us to arrive at the right understanding.

As humans, we tend to focus on general ideas than fine details. As a pair, developers are likely to spot mistakes faster. No two developers will look at code the same way. This is aided by the assignment of specific roles, driver and navigator. The driver is thinking tactically while the navigator is thinking strategically and looking at the big picture. Code reviews are immediate.

Pair programming avoids the bad practice of writing code by trial and error: if the code works, the developer may move on without checking for hidden flaws or bigger design problems. Pair programming allows developers to understand the problem fully before starting to code.

Pair programming allows each developer to gauge the expertise of the other and even oneself. It's easy to claim expertise but when developers discuss and explain things, they become better at gauging each other's expertise.

### How should novices and experts be paired together?

Pair programming is beneficial at all expertise levels because it's essentially a learned social skill. You're learning to work cooperatively with other team members. Where junior developers are involved, prioritize learning over productivity.

It's common to identify the following pairings:

    + **Novice-Novice**: Novice developers may not be confident enough to tackle a challenging problem alone but they can do so more effectively as a pair.
    + **Novice-Expert**: Useful for training novices or quick onboarding of new developers to the project. The novice must ask questions and grow in confidence as a developer. The expert must be empathetic and explain the reasons for the code. The expert should not do all the work.
    + **Expert-Expert**: Useful when each expert specializes in different areas, such as frontend versus backend. They can pair up when a feature impacts both areas of expertise. One study showed that expert-expert pairing is less effective when both experts are familiar with the coding task and its solution.

### What are some best practices for pair programming?

Developers must have an attitude to share and learn from each other. Empathy, patience and openness are important traits. One developer should move only as fast as the other can understand.

For better knowledge sharing and team collaboration, each developer must pair with different team members. However, where there's a personality conflict between two developers, such pairings must be avoided. Pairings can be changed daily, weekly or when user stories change.

Pair programming can be demanding at times. Frequent breaks can help. It's also best not to do pair programming all the time or for every task. During pair programming, avoid interruptions from others. Between sessions, it's good to give and take feedback.

To make participation easier, each developer has his own keyboard. Configure the system with key bindings and shortcuts that's comfortable to both developers. Create dedicated pairing stations uncluttered with personal items.

Those new to pair programming can start with a simple task that can be completed in a short time. Minimize distractions during the session. Management should ask developers to try pair programming rather than forcing it on them.

### What are the antipatterns in pair programming?

The navigator should avoid a few things: pointing out errors too quickly, giving too detailed low-level instructions, and perhaps not bringing a keyboard.

The driver should avoid a few things: driving too fast that the navigator can't keep up, partially blocking the screen, not taking breaks, driving without listening to the navigator, and avoiding eye contact.

Both developers should avoid a few things: distractions such as notifications or text messages, not swapping roles, not communicating with your partner, or giving up too early if you're just learning to code.

### Could you share some case studies on pair programming?

Jensen (1996) reported from a study that pairs wrote 175 lines of code per person-month compared to 75 from individuals. Error rate was three orders of magnitude lower than the organization's norm.

Nosek (1998) compared five experienced programmers working independently and ten others working in pairs. All teams outperformed the individuals, completing their tasks 40% faster and producing better code.

A survey by Williams (1999) found that 96% enjoyed their job more when pair programming.

Cockburn and Williams (2001) reported that pair programming incurs 15% extra development cost. But in the long term, it saves 15x the cost when bugs are discovered during testing or 60x the cost when bugs are experienced by customers.

Hulkko and Abrahamsson (2005) empirically found that pair programming is most useful for learning and complex tasks. It leads to higher comment ratio. It doesn't always increase productivity, decrease defect count or conform to coding guidelines.

### What are some concerns or criticisms of pair programming?

Not every developer is a fan of pair programming, particularly when it's imposed on them by the management. Developers like to be trusted and pair programming makes them feel they're being watched. However, once the benefits of pair programming are understood, developers are more likely to question the implementation rather than the practice itself.

Some problems require deep thinking that's best done in isolation. One approach is to work out the solution in advance before starting a pair programming session. In fact, humans can't multitask effectively. Pair programming requires the pair to constantly communicate while also performing their individual tasks.

There could be a personality clash. Developers are usually introverts. To them, pair programming can be overstimulating. Even for extroverts, pair programming all the time can be exhausting. Feedback between sessions is useful but developers are not naturally inclined to share feedback.

Two developers doing the job of one is sometimes seen as wasteful. Pairing two novices maybe ineffective but other types of pairings if correctly implemented bring long term rewards.

Lack of ownership and accountability can be a problem. The developer who creates the pull request can be made accountable.

## Milestones

1945

ENIAC, the first programmable, electronic digital computer is completed. In an interview in 2011, one of ENIAC's programmers, Jean Jennings Bartik, claims that she and Betty Synder were a pair and they programmed together. This anecdote shows that pair programming is certainly not an invention of the 21st century.

1950

Fred Brooks, author of *The Mythical Man-Month (1975)*, has claimed that he and fellow graduate student Bill Wright programmed together in the 1950s. He claims, "We produced 1,500 lines of defect-free code; it ran correctly first try."

1980

In the early 1980s, Larry Constantine visits P.J. Plaugher's software company, Whitesmiths, Ltd. He observes a room full of two programmers working at each computer. He calls them "dynamic duos". Their code was defect-free. He states that, "Two programmers in tandem is not redundancy; it's a direct route to greater efficiency and better quality."

1998

Nosek publishes *The Case for Collaborative Programming*, which is perhaps the first empirical study on the subject using experienced programmers. An earlier 1993 study by Wilson et al. used student programmers. Also in 1998, **Extreme Programming (XP)** as practiced at Chrysler is talked about. Pair programming is one of the core practices within XP.

2000

Some practitioners of XP introduce the **roles of driver and navigator** to explain pair programming in a better way.

2002

The book *Pair Programming Illuminated* by Williams and Kessler is the first book dedicated to pair programming. It's also perhaps a sign that pair programming is ready for mainstream adoption.

2003

The **ping-pong style** of pairing is suggested on C2 Wiki. This combines pair programming with test-driven development.

2012

On his blog, Arlo Belshee coins the term **strong-style** for the style of programming advocated by Llewellyn Falco. In July 2016, Falco and his colleague Maaret Pyhäjärvi present this style at the Agile2016 conference.

Pair Programming


# Sentiment Analysis

## Summary

Sentiment Analysis is a process which focuses on analyzing people’s opinions, feelings, and attitudes towards a specific product, organization or service.

It is not uncommon for us to consider what other people think in our decision-making process. Prior to the advent of the Internet, many of us relied on friends and families for product or service recommendations, or information when buying a product. The Internet eases our efforts to get opinions of the general population.

In a world where colossal amounts of user-generated content is produced every day, it is practically impossible for human workforce to collect all the data and determine the opinions expressed in those data. Therefore, there arises a need to develop computer algorithms to automate the classification of reviews on the basis of their polarities as: positive, negative or neutral.

## Discussion

### What kind of questions are answered by Sentiment Analysis?

Since Sentiment Analysis tools classify a sample of text as positive, negative or neutral, some of the questions which can be answered using Sentiment Analysis are as follows:

    + Is a given product review positive or negative?
    + Is a customer satisfied or dissatisfied based on his email response?
    + Based on a sample of tweets, how are people responding to a given ad campaign, product release, or news item?
    + How have bloggers' attitudes about the president changed since the election?

### What are the steps involved in Sentiment Analysis?

Sentiment analysis typically has the following steps:

    + **Data acquisition:** The collection of data is an important phase since a proper dataset needs to be defined for analyzing and classifying the text in the dataset.
    + **Text preprocessing**: After collecting the data, preprocessing allows to reduce noise in data. This is done by removing the unnecessary stop words, repeated words, stemming, removal of emoticons, removal of URLs etc.
    + **Feature selection and extraction:** Proper selection and extraction of features plays a key role in determining the accuracy of the model. Hence, the appropriate feature extraction technique must be chosen for extracting the features.
    + **Sentiment classification:** In this phase, various sentiment classification techniques are applied to classify the text. Some popular sentiment classification techniques are Naïve Bayes (NB) and Support Vector Machines(SVM).
    + **Polarity detection:** After classifying the sentiments, the polarity of the sentiment is determined. The goal of polarity detection is to decide whether a text expresses positive, negative or neutral sentiment.
    + **Validation and evaluation**: Finally, validation and evaluation of the obtained results is performed so as to determine the overall accuracy of the techniques used for sentiment analysis.

### What are the various approaches for sentiment analysis?

Broadly, there are ML-based and lexicon-based approaches, plus a hybrid approach that's a combination of these two.

**ML models** classify the text as positive, negative or neutral using classification algorithms and linguistic features.

**Lexicon models** make use of sentiment lexicons, which are collections of annotated and preprocessed sentiment terms. Sentiment values are assigned to words that describe the positive, negative and neutral attitude of the speaker. It is further classified as:

    + *Dictionary-based Method*: It uses a small set of seed words and an online dictionary. The strategy here is initial seed set of words with their known orientations are collected and then online dictionaries are searched to find their probable synonyms and antonyms. The sample is classified based on the presence of such signalling sentiment words.
    + *Corpus-based Method*: Uses corpus data to identify sentiment words. Even though it is not as effective as dictionary based scheme, it is helpful in finding the domain and context of specific sentiment words against the corpus data. The algorithm will have access not only to sentiment labels, but also to a context.

### What are the advantages and limitations of the Sentiment Analysis approaches?

*Machine Learning based*

    + **Advantage**: Unlike Lexicon-based, these models can be built for a specific purpose or context.
    + **Limitation**: Obtaining labeled data for training could be difficult or expensive.*Lexicon-based*

    + **Advantage**: No training is required.
    + **Limitation**: Accuracy depends on lexical resources. Finite number of words in lexicons and the assignment of a fixed sentiment orientation and score to words.*Hybrid:*

    + **Advantage**: It incorporates the best of Machine learning based and Lexicon based approaches.Generally, Machine Learning based models perform better than Lexicon-based models. But Machine Learning models require labeled data in huge quantities.

### What are some tools available for Sentiment Analysis at present?

Here are some developer tools for sentiment analysis:

    + Python NLTK: A python based tool for text processing, cataloging, tokenization, stopping, tagging, parsing and much more.
    + GATE, the General Architecture for Text Engineering: A Java suite of tools used for all sorts of natural language processing tasks, including information extraction in many languages.
    + LingPipe: LingPipe is tool kit for processing text using computational linguistics.
    + LIWC (Linguistic Inquiry and Word Count): A computerized text analysis tool that reads a given text and counts the percentage of words that reflect different emotions, thinking styles and social concerns.Among other tools are Opinion Finder, Grooper document classification, MonkeyLearn, IBM Watson, Lexalytics, MeaningCloud, Rosette, Repustate, Clarabridge and Aylien.

### What are the challenges involved in Sentiment Analysis?

Human language is intricate. People often express opinions in complex ways. To mention a few:

    + **Named entity recognition**: Locating and classifying named entities in text into pre-defined categories such as the names of persons, organizations, locations. Eg: Is 300 Spartans a group of Greeks or a movie?
    + **Anaphora Resolution**: It is the problem of resolving references to earlier or later items in the discourse. Eg: "We watched the movie and went to dinner. It was awful." What does "It" refer to?
    + **Parsing**: This refers to resolving a sentence into its component parts. What is the subject and object of the sentence, which one does the verb and/or adjective actually refer to?
    + **Rhetorical modes**: Typically the analysed posts contain sarcasm, irony, implication, etc, which are particularly difficult to detect.
    + **Social media website**: It is not uncommon to find reviews and opinions containing slang, abbreviations, lack of capitals and poor punctuation, which would make sentiment analysis even more challenging.
    + **Visual sentiment analysis**: Posts often contain a mixture of visual and textual information. The sentiment polarities implied by texts may contradict the sentiments of images, which poses a challenge for textual sentiment analysis.

### Where is Sentiment Analysis being used at present?

A very broad answer to this can be broken up into three categories:

    + **Brand Monitoring** - Sentiment Analysis is used to gauge how a brand, product or company has been received by the public. In fact, private companies like Unamo offer this as a service.
    + **Customer Service** - Customer service agents classify incoming mail into 'urgent' and 'non-urgent', in order to be able to serve the more frustrated customers quicker. The speech analytics platform Callminer Eureka implements AI and ML techniques to draw insight from consumer interactions, in order to offer quality customer service.
    + **Market Research and Analysis** - Opinion mining plays a crucial role in business intelligence, by helping analysts understand why a particular product was well received or not. Stock markets and hedge funds have been known to shift with the shift in sentiments on social media.Sentiment analysis has also driven forward various other initiatives. Bing Search used the concept in their newly launched Multi-Perspective Answers product.

Apart from the above, Sentiment analysis is used in the areas of political science, sociology, psychology; flame detection, identifying child-suitability of videos, bias identification in news sources are the variety of applications.

## Milestones

300
BC

The history of Sentiment Analysis begins in Ancient Greece with the concept of *Doxa*, which refers to common belief or popular opinion.

1824

The Harrisburg Pennsylvanian conducts a "straw polling" to predict the outcome of the United States presidential election. The prediction turns out to be incorrect, possibly because it didn't draw the right sample.

1997

Hatzivassiloglou and McKeown use the term **semantic orientation** in a paper titled *Predicting the Semantic Orientation of Adjectives*. Their approach is corpus-based and adapts to new domains. Thus it can tell that 'bull' and 'bear' are opposites in stock market reports. Using only adjectives, their model achieves 90% precision.

2004

Pang and Lee apply machine learning to sentiment analysis. They propose a subjectivity detector to pick out subjective sentences. Then they employ text categorization techniques on the subjective sentences. Algorithms used include Naive Bayes and SVM to find minimum cuts in a graph. They claim an accuracy of 86.4% on the NB polarity classifier.

2005

Gruhl et al. conduct one of the first studies to determine if online comments influence the sales figures of a product. They obtain the sales data of books from Amazon.com. From blog mentions and online chatter, they use automated query generation algorithms to predict the rise and fall of sales of certain books. They find that positive comments lead to increased sales.

Mar
2011

The shares of Buffet-owned Berkshire Hathaway rises by as much as 2.94% following the Oscars award ceremony. Likewise, there' correlation between Anne Hathaway's movie release dates and stock price increases of Berkshire Hathaway during the period 2008-2010. The reasoning is that automated trading programs are picking up online chatter about 'Hathaway' and applying it to the stock markets. This is an example where sentiment analysis fails to understand context.

2012

Kucuktunc et al. pioneer large-scale sentiment analysis of Yahoo! Answers. They find that answers differ according to the attributes of users, such as best-rated answers have a neutral tone to them. They also identify particular feelings evoked on reading a certain question. These findings begin to be used in advertising and recommendations.

Jun
2014

Aleksandr Kogan collects and provides a database containing information of about 87 million Facebook users to Cambridge Analytica. Cambridge Analytica subsequently uses it to make 30 million "psychographic" profiles about voters. In later years, it is alleged that this data was used to influence voter opinion on behalf of politicians who hired them.

2017

Poria et al. identify the sentiments present in online videos based on utterances. An utterance is a unit of speech bound by a pause. Each utterance is classified subjectively.

2019

Sun et al. apply **BERT**, a pre-trained neural network language model, to the task of Aspect-Based Sentiment Analysis (ABSA). Another research group also applies BERT to ABSA to show state-of-the-art results on SemEval-2015 Task 12 subtask 2 and SemEval-2016 Task 5.

Sentiment Analysis


# Artificial Intelligence

## Summary

Intelligence demonstrated by machines that mimics human decisions and actions for problem solving is called Artificial Intelligence. Intelligence accumulated in the machine is a result of an incremental learning process from data accumulated in the business environment.

Human intelligence consists of varied aspects – discriminating between objects, identifying people and places, understanding languages, recognizing sounds and music, logical analysis and mathematical computations. The list is endless. AI systems are being developed for specific goal-oriented applications covering some of these abilities.

Machine Learning, Big Data and Robotics are its key enablers. Neural networks (a study that understands how the human brain works) helps to add human-like intelligence to the computer. Examples include Google Street View, iPhone Face unlock and Walmart's automated supply chain. AI is still in a nascent stage, but is one of most rapidly growing technologies in the world.

## Discussion

### AI is not a new discovery. Why is there such a buzz around AI all of a sudden?

A popular theme in science fiction and folklore, the field of AI research was founded as an academic discipline as early as 1956. There was thriving activity for two decades, with ambitious targets set for matching human intelligence.

However, nothing substantial could be achieved by the 1970s with heavy rule-based systems having poor computational power. A bleak withdrawal period called the **AI Winter** followed when research and funding slowed down.

In the 21st century, several parallel developments renewed interest in AI. Exponential increase in computer processing speed, major drop in memory costs, upsurge of open source programming frameworks, advancement in voice and image processing techniques turned out to be game changers. The advent of big data and cloud computing enabled large scale distributed deployments feasible at company/national level.

When machine learning moved from rule-driven to data-driven algorithms, AI got the biggest boost. Historical data turned into a powerful enabler. Wireless networks, smart devices, IOT and developments in robotics have completed the circle. AI solutions have reached the maturity to be cost effective and implementable.

### What are the major components of an AI solution?

Listed below are the main components that act as building blocks in any modern day AI solution:

    + **Machine Learning** - Computers use statistical techniques to "learn" (progressively improve performance on a specific task) from data, without being explicitly programmed. ML is supported by large volumes of data called *Big Data*. Data could be historical or real-time data, structured or unstructured. Data is stored in distributed cloud computing and storage systems. ML also employs a computing system called Artificial Neural networks, inspired by how biological nervous systems and the brain process information. Eg: Image recognition, how a child identifies a cat.
    + **Natural Language Processing** - Ability to detect words and meanings from human voice in various spoken languages. Systems like voice assistants learn from past data and improve their language interpretation over time.
    + **Robotics** - The mechanical arm of AI that enables automated decisions to be implemented though robots, driverless cars, drones and other smart devices.
    + **Speech and Vision** - Enabling the computer with human-like senses of speech and vision using speech encoding and image recognition algorithms.

### What are the different approaches to AI?

AI means empowering machines to act as humans would. This intelligence can be fed into machines in several ways. These approaches can be grouped into one of the two categories listed below:

    + **Symbolic, rule-based approach – Expert Systems – Artificial General Intelligence**: Goal is to produce general human-like intelligence, not specific to a given problem at hand. Symbolic representations are used to cover responses to unplanned, uncertain situations adding weightages to social, emotional aspects as well as logic. This approach was prevalent in the early years (1960s-1990s) with the development of fuzzy logic and expert systems. IBM chess program that beat Garry Kasparov in the 1990s is a good example of this approach.
    + **Statistical, data-based approach – Machine Learning – Narrow/Weak AI**: With advent of Machine Learning, approach to AI changed. ML employs statistical modelling algorithms on historical data and derives "intelligence" from experience. This method is effective for specific business problems, hence the name *Narrow AI*. Goal is to build AI by consolidating intelligence from different scenarios. Applications in the medical field for disease detection and drug testing on patients is a prominent application of this approach.

### Which are the common open source frameworks to build AI solutions?

One of the main reasons for explosive growth in AI research and development in the recent times is the availability of reliable, efficient open source programming frameworks that support AI solutions development. Not just major IT corporations, university projects and small startups are also joining the AI bandwagon because of this. Some of the widely used programming languages and frameworks are listed below:

    + **Python and R Programming Languages** - Basis for almost all ML algorithm implementations powering AI solutions. They have a strong developer community and excellent package library support for most applications. Python is the base over which several ML/DL and NLP frameworks are built.
    + **TensorFlow Deep Learning Framework** - Python based computational framework from Google for building Deep Learning models. Widely used for building hierarchical ML applications, deep belief nets. It's being used for translation in Google's Translate, for OCR in WPS Office, fraud detection in PayPal.
    + **PyTorch** - Deep Learning framework from FaceBook. Major vendors like Microsoft and Amazon are expected to provide complete support to the framework across their cloud products.Keras, Theano, and MS CNTK are some other frameworks.

### What are the applications of AI that are currently in the market?

AI applications have pervaded all areas of work and life. From the mobile phone in hand to futuristic space research, AI is everywhere. AI applications domain-wise are listed below:

    + **Smartphone Apps and Utilities** - Google Street View, iPhone Face unlock, voice assistants.
    + **Retail** - Automated supply chains, scanning product images and shop, automated billing, customized shopping experience.
    + **Healthcare** - Early disease detection, preventive diagnostics, automated drug trials.
    + **Legal Industry** - AI is revolutionizing the legal industry by automating repetitive tasks, streamlining document review, and enhancing legal research capabilities.
    + **Financial Services** - Bots on the stock market, fraud detection, estimating customer credit worthiness.
    + **Auto Industry** - Vehicle preventive maintenance, driverless cars, delivery drones.
    + **Fashion** - Customized clothing for buyers, smart inventory management.
    + **Human Resources** - Resume elimination and talent acquisition process automation, personalised training and on-boarding for employees.

### What are the technical challenges associated with AI research and development?

    + **Computing Power Limitations** - There is exponential growth in processing speeds, distributed computing abilities and large-scale memory storage. But demands are even greater, especially with deep learning applications. Only large corporations can match demand, and this limits the scope for startups.
    + **Skilled Manpower** - AI applications require niche computer science skills such as ML, NLP, image processing, robotics, and data sciences. Since most of these fields of study are still evolving, finding high quality manpower is a challenge. Reskilling engineers from traditional programming or design skills to new-age AI skills is the need of the hour.
    + **Data Security** - Data is the biggest wealth of an organization today. With more and more private business data coming online, it becomes vulnerable to external misuse and attacks. Companies need to focus on data security and network frameworks to protect their data.

### What are the social and economic risks associated with AI technology?

However promising and revolutionary AI may sound, it comes saddled with several risks impacting society and livelihoods. Key risks are:

    + **Fear of widespread job loss** - Since AI brings a new bunch of technologies into the fore, several IT and business skills such as manual testing, customer support, traditional programming, and managerial tasks are becoming obsolete. In the wider world, impact is even more pronounced. Autonomous cars replace human drivers. Drone deliveries replace home delivery staff. AI and robotics replace several manufacturing jobs. Impact on healthcare, financial services and retail are expected to be high.
    + **Bias due to bad data** - AI systems are as good or bad as the data fed into them. If companies and governments feed biased data with malicious intent, results can be disastrous. Self-regulation is the only method in practice. Stricter legislations and tight international data policing rules are needed.
    + **Over-reliance on digital devices** - Smart devices running AI are invading organizations, homes, and government institutions. Lifestyle deficiencies such as exposure to wireless radiation, excessive time spent with gadgets, and lack of basic computational and social skills are worrying.

## Milestones

1952

Arthur Samuel writes the first computer learning program. This is a game of checkers that runs on an IBM machine. It improves its game every time it plays. Samuel is also credited for coining the term **Machine Learning (ML)**.

1957

Frank Rosenblatt designs the first neural network for computers, called the **perceptron**, which simulates the thought processes of the human brain.

1979

Students at Stanford University invent the robotic *Stanford Cart*. This can navigate obstacles in a room on its own.

1980

Through the 1970s and into the 1980s, AI sees a subdued period known as the **AI Winter**. Interest and development dwindles, people become pessimistic about its chances of success, and funding drops. Research continues but AI steps out of the limelight.

1990

In the 1990s, work on machine learning shifts from a knowledge-driven approach to a **data-driven approach**.

May
1997

In a contest of man vs machine, IBM's *Deep Blue* beats the world champion at chess. This showcases computing capability to tackle complex calculations for drug discovery, large database searches, broad financial modelling, and more. In February 2011, another IBM machine named *Watson* beats humans at another game, called Jeopardy! Computers can now process and reason about natural languages, heralding rich human-computer interactions for the future.

2006

Geoffrey Hinton coins the term **Deep Learning (DL)** as a term for artificial neural networks using many layers of neurons. With innovations happening in the Computer Vision space, DL algorithms enable computers to "see" and distinguish objects and text in images and videos.

2011

Apple includes *Siri* as a built-in digital assistant in its iPhone 4S. Siri triggers a new age of automation, personalization and AI-driven assistants.

2013

Many major automotive manufacturers, including General Motors, Ford, Mercedes Benz, Volkswagen, Audi, Nissan, Toyota, BMW, and Volvo, start testing **driverless car systems**. Tesla Motors announces its first version of AutoPilot. Model S cars equipped with this system are capable of lane control with autonomous steering, braking, automated parking and speed limit adjustment.

2014

Facebook releases *DeepFace*, a software algorithm that is able to recognize or verify individuals on photos to the same level of accuracy as humans.

2015

Over 3,000 AI and Robotics researchers, endorsed by Stephen Hawking, Elon Musk and Steve Wozniak (among many others), sign an open letter warning of the danger of autonomous weapons that can select and engage targets without human intervention.

2017

Apple CEO Tim Cook says, "We are focussing on autonomous systems". He describes Apple's secretive *Project Titan* as "the mother of all A.I. projects".

Artificial Intelligence


# Database Sharding

## Summary

In the age of Big Data, popular social media platforms and IoT sensor data, datasets are huge. If such a dataset is stored in a single database, queries become slow. Overall system performance suffers. This is when database sharding becomes useful.

A single logical dataset is **split into multiple databases** that are then **distributed across multiple machines**. When a query is made, only one or a few machines may get involved in processing the query. Sharding enables effective scaling and management of large datasets. There are many ways to split a dataset into shards.

Sharding is possible with both SQL and NoSQL databases. Some databases have out-of-the-box support for sharding. For others, tools and middleware are available to assist in sharding.

Database replication, partitioning and clustering are concepts related to sharding.

## Discussion

### When do I need to shard my database?

SQL query performance can be improved by simply **indexing** tables. But on large tables (millions of rows) searching even the indices can be slow.

One approach is to do **partitioning**. Large tables are split into smaller tables. Each partition has it's own index. Suppose tables are partitioned by rows, query is processed on a smaller index. All partitions reside on the same server.

A complementary approach is **vertical scaling**, which is a hardware upgrade: faster processor, more RAM, faster disk, etc. However, this is not scalable or cost effective in the long run.

Even with partitioning and vertical scaling, the server can get overwhelmed with too many requests. One solution is **replication**. The master database is replicated in other servers called slaves. Read query is processed by one of the slaves. This speeds up database reads but not writes, which happen only on the master server. Reads can also be improved with **caching**.

When all these have failed, use sharding as the last resort. Sharding enables **horizontal scaling**, which involves adding more servers that share the load and process requests in parallel.

### How is sharding different from partitioning?

Some use the terms partitioning and sharding interchangeably. However, there are clear differences.

All partitions of a table reside on the same server whereas sharding involves multiple servers. Therefore, sharding implies a distributed architecture whereas partitioning does not.

Partitions can be horizontal (split by rows) or vertical (by columns). Shards are usually only horizontal. In other words, all shards share the same schema but contain different records of the original table. For this reason, sharding is sometimes called *horizontal partitioning*.

Due to its distributed nature, sharding is more complex than partitioning. Many databases including older ones offer built-in support for partitioning. Sharding is supported mainly in modern databases. Where databases lack support for sharding, and no relevant middleware is used, the logic of which shard to hit resides in application code. With partitions, the logic resides at the database level.

Sharding adopts a **shared-nothing architecture**. A shard is not aware of other shards. This is often not the case with database clustering or replication.

### What are some strategies for database sharding?

A sharding strategy is necessary to determine which record goes into which shard:

    + **Key-Based**: One of the columns, called *shard key*, is put through a hash function. The result determines the shard for that row. Also called hash-based sharding or algorithmic sharding.
    + **Range-Based**: For example, given a product-price database, prices in range 0-49 go into shard 1, 50-99 into shard 2, and so on. Price column is the shard key. If the store sells lot more low-value products, this will result in unbalanced shards and hotspots.
    + **Dictionary-Based**: A lookup table is used. This is a flexible approach since there's no predetermined hash function or ranges. An example is to shard by customer's region such as UK, US, or EU.
    + **Hierarchical**: A combination of row and column is used as the shard key. Previously mentioned sharding strategies can be used on the key.
    + **Entity Groups**: To facilitate queries across multiple tables, this strategy places related tables within the same shard. This brings stronger consistency. For example, all data pertaining to a user reside in the same shard.

### How should I select a suitable shard key?

Selecting a shard key is an important decision. Once selected, it's hard to change this in future. Consider how the choice will affect the current schema, queries and query performance. The choice should support current and future business use cases. Identify main parts of the data and clustering patterns.

For efficiency, the data type of the shard key is ideally an integer.

Sharding must balance two constraints: minimize cross-partition queries and distribute the load evenly by sharding at the right granularity.

Shard key must be on a column (SQL) or field (NoSQL) that meet the following:

    + **Stability**: It almost never changes. Otherwise, we could incur expensive movement of data from one shard to another.
    + **High Cardinality**: It has many more unique values relative to the number of shards.
    + **Low Frequency**: Every value occurs with a low frequency.
    + **Non-Monotonic**: The value doesn't increase or decrease monotonically.

### Could you share some examples of shard keys?

In an eCommerce application, Inventory, Sales and Customers could be functional areas with shard keys `productID`, `orderID` and `customerID` respectively. Alternatively, `customerID` could be the shard key for Sales as well. Typically `customerID` is unique and of high cardinality. Hence, this could be the shard key rather than `country`.

In an IoT application, if all devices are storing data at similar intervals then `deviceID` is a suitable key. If some devices store lot more data than others, this would create hotspots and therefore `deviceID` may not suitable. For an application with a few status codes, using status code as shard key is not a good idea.

Writing across shards is to be avoided. As an example, an eCommerce order may insert buyer address and seller address. Though both are part of the same logical table, they could reside on different shards. Another example is when a transaction inserts an order entry and an inventory entry. These two could be on different shards. It's preferable to locate all data pertaining to a transaction on a single shard.

### What's a typical database sharding architecture?

There's no requirement that a server must host a single shard. Multiple shards can be hosted on the same server. When load increases, they can be moved to separate servers. Original database is split by a shard key. Data that have the same key is called a **logical shard**. One or more logical shards when hosted on the same server is called **physical shard**.

In a typical architecture, a cluster proxy is connected to multiple shards. The proxy has access to a configuration service (such as Zookeeper) that keeps track of shards and their indices. The cluster proxy looks at a query and the configuration to know which shard to hit. If shards are large, each shard may also have its own proxy that can do caching and monitoring. With hierarchical sharding, each shard may be sharded further.

In Oracle Sharding, Shard Catalog contains the configuration. Routing is done by Shard Director. All shards are in the same region (data center). For high availability, replicated shards are in the same region as the primary shard. For disaster recovery, replicated shards are in another region.

### How is sharding different between SQL and NoSQL databases?

SQL databases were designed with ACID transactions in mind, not scalability. NoSQL databases were designed with scalability in mind. Sharding in NoSQL databases happens almost transparent to the user. Availability is prioritized over consistency.

In SQL tables, shard key is based on one or more columns. In NoSQL records, one or more fields are used instead.

Since SQL databases combine sharding with replication, it's a master-slave architecture. In NoSQL world, shards are referred to as **nodes**. All nodes are equal, with one node connected to a few others in a graph-like structure. When a query comes in, a coordinator node is selected. If the coordinator can respond directly, it will. Otherwise, it forwards the query to another node that has the data. Nodes frequently share information about the data they store via what's called the **gossip protocol**.

### What are the pros and cons of database sharding?

The main benefits of sharding include horizontal scaling, better performance, and higher availability. Processing and storage are distributed. An outage will most likely affect only one shard.

Sharding can lower cost by avoiding high-end hardware or expensive software. Commodity hardware and open-source software can be used.

Shards are suited for cloud deployments. Any upgrade can be tested on one shard before rolling it out to the rest. Shards can be geographically distributed to meet regulatory requirements or be closer to customers.

The complexity inherent with sharding is the main drawback. Improper implementation can lead to loss of data or data corruption. Maintaining multiple shards is not trivial and teams have to be trained in new workflows. Once sharded, it's hard to return to the unsharded architecture. Databases without native support for sharding will need custom solutions.

Since data is distributed across many servers, a `JOIN` operation easily done in a single SQL database becomes difficult. Such an operation may hit many shards and require merging the responses.

Shards may become unbalanced over time. In other words, some shards grow faster than others, thus becoming **database hotspots**. Resharding is the solution.

### What resources or tools are available to facilitate database sharding?

Cassandra, HBase, HDFS, MongoDB and Redis are databases that support sharding. Sqlite, Memcached, Zookeeper, MySQL and PostgreSQL are databases that don't natively support sharding at the database layer.

For databases that don't offer built-in support, sharding logic has to reside in the application. To avoid this complexity in application code, middleware can help. Two open-source middleware are Apache ShardingSphere and Vitess. Plugins for ShardingSphere offer support for MySQL, PostgreSQL, SQLServer, and Oracle. Vitess supports MySQL and MariaDB.

MySQL itself doesn't offer sharding but MySQL NDB Cluster and MySQL Fabric can be used to achieve sharding. For sharding PostgreSQL, PL/Proxy, Postgres-XC/XL and Citus can be used.

On Google Cloud Platform, Cloud SQL and ProxySQL services can be used to shard PostgreSQL and MySQL databases. On AWS, Amazon RDS is a service that can implement a sharded database architecture. The DB engine can be MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, and Amazon Aurora. For data analytics, Amazon Redshift can store data from all shards. Amazon RDS also facilitates resharding.

### Could you share some best practices for database sharding?

Consider sharding only when other approaches have failed. Main reasons for sharding include constraints on storage, processing, network bandwidth, regulations and geographic proximity.

Data analytics is usually performed on the entire dataset. Hence, sharding is more suited for Online Transaction Processing (OLTP) rather than for Online Analytical Processing (OLAP).

Tables with foreign key relationships can all share the same shard key. To keep primary keys unique across all shards for later OLAP, shard IDs can be suffixed to primary keys.

Sharding can be combined with replication. In fact, for high availability it's common to replicate shards. We could even duplicate some data across shards, such as duplicating chat messages in sender's and recipient's shards.

Monitor the performance of all shards in terms of CPU utilization, memory usage, and read/write performance. Consider resharding if there are hotspots.

## Milestones

1983

It's predicted that distributed databases with specialized hardware will die. Multiprocessor systems are seen to be I/O limited. However, later years prove this prediction to be false. This is due to the growth of relational databases through the 1980s. These can be easily parallelized. High-speed networking to interconnect parallel processors become available. Teradata, Tandem and other startups successfully market parallel machines.

1986

DeWitt et al. present a parallelized database they call *GAMMA*. It consists of 20 VAX 11/750 processors, each with dedicated memory. Eight of them have disk drives for database storage. The machines are connected by 80Mbps token ring. They note four ways to split data: round-robin, hashed, range (user specified), and range (uniform distribution). In a later version of GAMMA (1990), each processor has its own disk storage, giving rise to the term **shared-nothing architecture**.

Jun
1992

DeWitt and Gray apply dataflow programming paradigm to propose a distributed database architecture they call **partitioned parallelism**. Input data is partitioned and sent to multiple processors and memories. They propose four sharding strategies: round-robin, range, list and hash.

1996

The use of the term "shard" in a computing and storage context probably originates with the massively multiplayer online game called *Ultima Online*. The story is that the fictional world Sosaria was trapped in a crystal. When the crystal shattered, each **shard** contained a copy of Sosaria. Each copy of the world ran on its own server and database.

Aug
2005

In a presentation, Fitzpatrick explains how sharding helped LiveJournal scale their backend. The site launched in 1999 but grew to 7.9 million accounts by August 2005. With thousands of hits per second, sharding is adopted as the solution. The term "shard" is not used. Instead, the term **user cluster** is used. Within a cluster, replication is done.

2006

Google's *Bigtable* is a distributed storage that can handle petabytes of data across commodity servers. Web indexing, Google Earth, Google Analytics, Orkut, and Google Finance are some projects at Google using Bigtable.

2007

At Facebook, engineers start work on an NoSQL database called *Cassandra*. They're inspired by **range sharding** of Google's Bigtable and **consistent hash sharding** of Amazon's DynamoDB. They adopt linear hash sharding that is a hybrid of the two. It preserves the sort order but distribution is uneven. This is later changed to consistent hash sharding at the expense of range queries.

2010

A team in YouTube creates *Vitess* to solve scalability challenges to its MySQL data storage. Before Vitess, YouTube tried replication to speed up reads, and then application-level sharding to speed up writes. With Vitess, routing logic can be removed from application code. Vitess sits between the application and the database. In February 2018, Vitess becomes a CNCF incubation project, graduating in November 2019.

Database Sharding