# Kubernetes

## Summary

A scalable application on the cloud is typically composed of many microservices running within containers. When there are multiple containers running the workloads of multiple applications across machines, there's a need to manage the containers. This is where **container orchestration** comes in. By automating container deployment, scaling, monitoring and recovery, developers can focus on code rather than operations.

Kubernetes is an open source container orchestrator. The project is overseen by the *Cloud Native Computing Foundation*, a project of the Linux Foundation. A shortform for Kubernetes is **k8s**. Kubernetes is written in Go language.

Kubernetes can work with different container technologies such as Docker or rkt. It can manage containers on clusters of physical or virtual machines. Deployments can be on-site or across various cloud providers, and therefore there's no vendor lock-in.

## Discussion

### Why was Kubernetes invented in the first place?

Internally, Google had been using containers for more than a decade before Kubernetes itself was born. They used a cluster management system called **Borg** for managing hundreds of thousands of containers. This infrastructure was used to power Google Compute Engine but there was a problem. Customers were simply spinning up virtual machines (VMs), paying for them but using them well below capacity. In other words, resources were not optimally utilized. Engineers at Google realized that an open source variant of Borg was needed that everyone could use.

Containers themselves had been popularized by Docker but managing even dozens of containers was becoming an operational challenge. Kubernetes was therefore created in 2013 before Docker Swarm itself appeared the next year. By making it easier to manage containers, tools like Kubernetes enable more widespread adoption of containers instead of VMs.

### What are the key features of Kubernetes?

When Kubernetes first came out, its basic feature set included replication, load balancing, service discovery, basic health checking, self-healing, and scheduling. Current features of Kubernetes include the following:

    + **Service discovery and load balancing**: Uses a single DNS name for a set of containers. Jobs are load-balanced across them.
    + **Automatic binpacking**: Also called scheduling, containers are assigned to nodes based on resource requirements. Mix critical and batch workloads in order to drive up utilization.
    + **Storage orchestration**: Support for various storage options.
    + **Self-healing**: Restart failed containers. Reschedule containers when nodes dies.
    + **Automated rollouts and rollbacks**: App changes are rolled out progressively without bringing down all instances at the same time. If something goes wrong, rollback the changes.
    + **Secret and configuration management**: Update secrets without rebuilding image or exposing secrets in your stack configuration.
    + **Batch execution**: Manage batch and CI workloads.
    + **Horizontal scaling**: Also called replication, apps can be scaled up or down automatically based on CPU usage.

### Could you define some common Kubernetes terminology?

A Kubernetes glossary is available online but here are some common terms:

    + **Cluster**: A set of worker nodes and at least one master node managed by Kubernetes.
    + **Controller**: A control loop that watches and attempts to move the cluster to its desired state.
    + **Deployment**: An API object that manages a replicated application. Each replica is a pod.
    + **Label**: Key-value pairs used to identify and organize Kubernetes objects such as pods.
    + **Namespace**: An abstraction used by Kubernetes to support multiple virtual clusters on the same physical cluster.
    + **Node**: Previously called minion, this is a worker machine that can be virtual or physical. It has services (such as Docker) to run pods.
    + **Pod**: The smallest and simplest Kubernetes object, it's a set of running containers on your cluster.
    + **ReplicaSet**: Ensures a specific number of instances of a pod is always running.
    + **Service**: An API object that describes how to access applications, such as a set of Pods, and can describe ports and load-balancers.
    + **Selector**: Allows users to filter a list of resources based on labels.
    + **Volume**: A directory containing data, accessible to the containers in a pod with data preserved across container restarts.

### Could you describe the architecture of Kubernetes?

A Kubernetes cluster consists of at least one **master** plus a number of **nodes**. The master consists of API server, scheduler and controller. Configuration and management is via API calls processed by the master. Master also contains an **etcd** key-value database to store the state of the cluster.

Overall management of the cluster is with the master but the real workhorses are the nodes. Jobs execute within containers. One or more containers are combined into a single **pod**. One or more pods are scheduled to run on a node. Master schedules pods on nodes, not the containers themselves. Scheduling is based on resource requirements. When scheduled, the node pulls the required container image, invokes the container runtime on the node and launches the container. An agent called **kubelet** runs on each node to manage containers.

Kubernetes essentially does three things: resource management, scheduling and load balancing. Periodically, the controller obtains pod utilization and uses this to scale. This scaling is transparent to clients since everything is exposed as **services**.

### Why do we need pods when containers might be good enough?

While Kubernetes could have been designed to map containers directly to nodes, pods offer an extra level of abstraction to simplify container management. Because there are different container implementations—Docker, rkt, LXD, Windows Containers —pods provide a unified interface.

Pods also encapsulate containers that closely depend on one another. Each pod has an IP address and all its containers share the same port space. Inter-process communication (IPC) and sharing storage across containers in the pod is also easy.

When a pod is scheduled, all its containers are scheduled. One may argue why not bundle them into a single container. This violates the "one process per container" principle. Having multiple processes in a container makes it difficult to debug problems. By using pods, if a container fails, it will be restarted while other containers in the pod remain unaffected.

It's perfectly valid for a pod to have a single container. An example use case of a multi-container pod is a main container plus a sidecar container. The latter can be a log watcher or a data loader.

### What are some criticisms of Kubernetes?

Since Kubernetes manages the low-level details, if something is wrongly configured or things don't work as expected, it can be difficult to find the root cause. Kubernetes dashboard provides limited information though more detailed information is available via command line interface.

## Milestones

2003

Internal to Google, **Borg** is conceived and built for large scale cluster management. The idea is to use resources at high utilization while running hundreds of thousands of jobs. Borg is the system that goes on to power Google Cloud Platform and Google Compute Engine. Details of Borg are not disclosed outside of Google until much later in 2015.

2013

Within Google, **Omega** is introduced as a successor to Borg. From the experience of Borg and Omega, some Google engineers think of building an open source container management system. A prototype is built within three months.

Jun
2014

The first public commit of Kubernetes code is made in GitHub. A few days later, it's released to the world at DockerCon.

Jul
2015

Version 1.0 of Kubernetes is released at Open Source Convention (OSCON). With Kubernetes as the its first project, **Cloud Native Computing Foundation (CNCF)** is founded.

Nov
2015

KubeCon is organized in San Francisco as the first Kubernetes conference.

2016

This is the year when Kubernetes goes mainstream with more conferences, developers and features. The game Pokemon GO that becomes an international is powered by Kubernetes. Windows Server Support arrives. In May, v0.1.0 of **minikube** is released to deploy a single-node Kubernetes cluster in a VM locally.

2017

This is the year when we see greater adoption of Kubernetes among enterprises. By October, it's possible for Docker containers to seamlessly use either Docker Swarm or Kubernetes.

Feb
2018

In Kubernetes 1.9, as an alpha release, there's support for **pod priority and preemption**. High-priority workloads can thus scale faster than cluster autoscaler can add nodes. This leads to better resource utilization, lower costs and better service levels for critical applications.

May
2018

Kubernetes now supports **Containerd** containers. DigitalOcean offers Kubernetes as a hosted service. Google offers its own **Google Kubernetes Engine (GKE)**. Similar services from Microsoft and Amazon follow.

Kubernetes


# Object-Oriented Programming Concepts

## Summary

Developers who are familiar with procedural languages such as C and Pascal will understand variables, functions, and scope. When learning an object-oriented programming (OOP) language such as Java or C++, the same developers might have difficulty. This article presents an overview of OOP concepts.

Whereas the building block of procedural languages is a **function** or procedure, the building block of OOP languages is an **object**. Procedural languages have variables (data) and functions (code). In OOP, objects contain both data and code. With procedural languages, functions operate on data. With OOP, objects operate on their own data and selectively expose this data to other objects.

Different OOP languages sometimes differ in terminology. Some OOP languages may lack advanced OOP features. In general, OOP concepts are mostly similar across languages.

## Discussion

### What are objects and classes in OOP?

In the real world, just about anything can be seen as an object: car, dog, person, department, city, etc. These have **state** and **behaviour**. For example, a dog's state is its colour, breed and name; its behaviour is the way it barks, runs or wags its tail. Objects in OOP are quite similar.

In OOP, an **object** is a software construct that includes both data and code to process that data. Unlike procedural languages in which functions are independent of variables, OOP objects retain control of their state and how others can affect that state via executable code. Data specifies the state of the object whereas executable code specifies behaviour. A real-world dog could be represented in OOP as a software object.

Let's create many dogs of the same or different breeds. It's inefficient to write code for each dog. This is where the concept of **class** becomes important. A class is sort of a blueprint from which an object can be created. Thus, each dog is an *instance* of the same `Dog` class. **Instantiation** is the process of creating objects or instances from classes.

### What are attributes and methods of objects in OOP?

Attributes contain an object's state. Methods define an object's behaviour. Both are bound to the object and therefore accessed via the object. Attributes are also called properties, fields or instance variables. Each object has its own copy of attributes in memory. As for methods, only their addresses are stored within an object. Attributes and methods are also called **members**.

Consider the class `Rectangle`. It's attributes could be `length` and `breadth`. It's methods could be `getArea()` and `rotate()`.

Two special methods are worth noting: constructor and destructor. When an object is instantiated from its class, the **constructor** method is called to initialize the object state. When the object is destroyed, its **destructor** method is called so that necessary clean up is done.

Consider a database connector class. The constructor method might initialize the database server IP address, port, username and password for access. It might even open the DB connection. The destructor on the other hand might abort pending queries and close the connection. Other methods of the class might manage the connection, process DB queries and responses.

### Which are the essential concepts of OOP?

Among the many concepts of OOP, three stand out:

    + **Encapsulation**: A class and its instances bring together attributes and methods that are closely related. Encapsulation hides internal implementation and exposes public interfaces. Thus, it achieves *data hiding*.
    + **Inheritance**: Using an existing class, a more refined class can be created. The former is called a *base class* or *super class*. The latter is called a *derived class* or *subclass*. For example, `Shape` is a base class from which `Square` and `Circle` are derived classes. Base class might define common attributes such as colour and line width. Derived classes might define more specific attributes such as length or radius, and specific methods to compute area and perimeter.
    + **Polymorphism**: Different classes having the same attributes and methods, though implemented differently, can be interchanged at runtime. For example, a `Canvas` object might wish to compute the area of a shape object. It doesn't care if it's a square or a circle since both have the `getArea()` method. Canvas object calls this method on a shape object and the correct method of the current derived object is called.

### What is meant by abstraction in OOP?

Abstraction is about describing something at a conceptual level while leaving out the details. For example, we may talk about a vehicle without being explicit if it's a ship or a car. Abstraction a useful design tool since considering too many details upfront can be distracting and confusing. We start with an abstract concept and refine it later by adding more details to it. It's been said that,

> Abstraction hides details at the design level, while encapsulation hides details at the implementation level.

Consider `Shape` class with members `color` and `getArea()`. Subclasses `Rectangle` and `Triangle` implement how area is calculated. Since `Shape` doesn't know how to calculate area, we call it an **abstract class**. It can't be instantiated. Only its subclasses that provide an implementation of `getArea()` can be instantiated. Shape's `getArea()` is an **abstract method**.

Other than inheritance, an alternative to achieve abstraction is to use interfaces. **Interfaces** provide method signatures without implementing them. Classes that adopt these interfaces are required to implement them. **Mixins** are similar to interfaces except that they implement the methods themselves.

### How is composition different from inheritance in OOP?

Consider the classes `Vehicle`, `Car` and `Engine`. Car is a *type of* vehicle whereas engine is *part of* a car. The relationship between vehicle and car is best modelled as **inheritance**: `Car` is a subclass of `Vehicle`. The relationship between engine and car is best modelled as **composition**: an `Engine` instance is an attribute of either `Vehicle` or `Car` class.

While both inheritance and composition are relevant to OOP, it's important for software architects and designers to look at the relationships among classes to select the right one. Specifically, we should ask, "Is it a *is-a* or a *has-a* relation?" Inheritance is a pillar of OOP but composition is essential to every language.

Two other concepts closely related to composition are **association** and **aggregation**. Aggregation describes a *part-whole* relation. A `Course` object is part of a `DegreeProgram` object. It's also a many-to-many relation since the same course can belong to different programs and a program has many courses.

Some languages such as JavaScript, allow **prototypal inheritance**. Object prototypes form the basis of other objects. Classes don't exist.

### What are access modifiers in the context of OOP?

It's possible to restrict access to classes, methods and attributes. By the *principle of least privilege*, limit access only to classes that need it.

In Java, when no access modifier is specified, all classes within the package get access. Otherwise, we can use the following access modifiers:

    + `public`: Access is allowed for all classes.
    + `protected`: Access is allowed for subclasses (in any package) and classes within the same package. Thus, access is allowed only to closely related classes.
    + `private`: Access is allowed only within the class to which the members belong. This is typically used to hide data within the class. Any access to private data is provided via methods. Read and write access are via **getters** and **setters** respectively. These are also called **accessors** and **mutators**.In Kotlin, the term *visibility modifier* is used. Modifier `internal` gives access to all classes within the same module. Modifier `protected` gives access to subclasses.

In C++, access modifiers can also be applied to inheritance. For example, private inheritance would make all methods and attributes of the base class private and therefore inaccessible to the subclass.

### What are static classes and methods?

We know that objects are at the centre of OOP. However, there are use cases that don't need to store state. Class methods operate only on the input arguments they receive. In this case, we use a **static class**. In .NET, `System.Math` is an example of a static class.

A static class contains only static members. It can't be instantiated into objects. It's accessed by its class name, and not by instance variable name as is customary with objects.

Sometimes we wish to share an attribute across all objects of a class. For example, `numberOfBicycles` is an attribute that should be shared across all `Bicycle` objects. Java's `static` modifier can be used to make this a **static variable**. Likewise, a method to access this static attribute is called a **static method**. Static methods can access only static members. The class itself need not be static.

### What are some additional concepts in OOP?

Some OOP concepts are not common to all languages or were introduced later as languages evolved. We mention a few of these:

    + **Multiple Inheritance**: This is when a subclass inherits from multiple base classes. C++ and Python support this. In Java and C#, a class can implement multiple interfaces but not inherit from multiple classes.
    + **Generics**: Classes, methods and interfaces that can be parameterized are called generics. For example, a class might implement a set of objects with parameterized type. Thus, the same class definition can be used to instantiate a set of `Shape` objects or `Car` objects.
    + **Templates**: Whereas types in generics are substituted at runtime, types in templates are specialized at compile time. Each template specialization has its own compiled code. C++ and D support templates.
    + **Delegates**: We can call a method via its delegate that has a compatible method signature. With delegates, methods can be passed as arguments to other methods.
    + **Reflection**: Not exclusive to OOP, reflection allows a class to introspect and modify its own definition. In Java, reflection is possible on classes, interfaces, fields and methods at runtime. This is often useful for testing, data serialization and metaprogramming.

## Milestones

1967

**Simula 67** is born, officially the first object-oriented programming language. It's initially used in computer simulations. Classes in this language are called *processes* since object lifecycle depends on the process executing the statements.

1985

In the mid-1980s, many systems and application programmers adopt C++ for OOP. It takes another decade (1990s) for OOP to become more widely adopted.

Oct
1994

Object-oriented programming (OOP) and object-oriented design (OOD) are quite complex. To give software architects and developers guidelines to get these right, principles and design patterns have evolved over the years. At ACM's annual OOPSLA conference, four authors announce their book titled *Design Patterns: Elements of Reusable Object-Oriented Software*. This book describes 23 **software design patterns**. In time, this book becomes an essential read for OOP and OOD.

Dec
1997

Object Management Group (OMG) standardizes **Unified Modelling Language (UML), v1.1**. It's roots are in OOP. It's evolution can be traced to the early 1990s. UML establishes a common language among stakeholders involved in the design of object-oriented software and systems. Subsequently, UML undergoes many revisions, with v2.5.1 coming out in December 2017.

2002

Robert Martin publishes a book on Agile where he describes five principles of OOD. Sometime later, Michael Feathers coins the term **SOLID** as a useful way to remember the principles.

Object-Oriented Programming Concepts


# CAPTCHA

## Summary

CAPTCHA is an acronym for *Completely Automated Public Turing Test To Tell Computers and Humans Apart*. CAPTCHA may be described as a cyber security tool often used on websites. Websites that offer a service or collect data require the user to give inputs. This process can be thwarted with the use of bots. Bots can present themselves as humans with fake identities.

CAPTCHAs are designed in many ways but all of them have the same goal: to let humans into the system and deny admission to bots. They attempt to do this by presenting problems that are hard for bots to solve but relatively easy for humans to solve. With the coming of Artificial Intelligence (AI), computers have become smarter at solving CAPTCHAs. Therefore CAPTCHAs themselves have evolved to counteract intelligent bots.

## Discussion

### Where do we need CAPTCHAs?

Automated programs or bots are fast compared to humans. This gives bots an unfair advantage. For example, bots can book tickets in bulk and then sell them later at a higher price in the black market. Likewise, bots can generate bulk votes and thereby skew the results of an online voting system. Bots can also be used to spam, troll or trigger DDoS attacks. Here are some use cases where CAPTCHAs are useful:

    + Booking of tickets.
    + Online voting system.
    + Creating a new online/email account.
    + Preventing dictionary attack in password systems.
    + Promoting products as a comment to a blog post.
    + Liking or sharing a web page via social networking sites.
    + Protecting site contents from scrapers or search engine bots.
    + Stealing personal information at chat rooms.

### Why is a CAPTCHA sometimes called a reverse Turing test?

Turing test was conceived in 1950 by Alan M. Turing as a way to check if machines were as intelligent as humans. If a human interrogator interacting electronically with a machine and another human cannot tell which of them is the machine, then the machine passes the Turing test.

With CAPTCHA, the interrogator is not a human but a computer. Hence the term reverse Turing test is sometimes used.

### What are the different types of CAPTCHAs?

The following is a broad classification:

    + Text-based: Users have to type the characters from a distorted text.
    + Image-based: Users are asked to select images belonging to a certain group, such as those containing cats.
    + Video-based: Users are shown short video clips and asked to answer a question based on it. NuCAPTCHA is an example.
    + Audio-based: Users listen to an audio clip and type what they hear.
    + Action-based: Users are asked to perform some action. reCAPTCHA V2 asks users to click a checkbox. MotionCAPTCHA asks users to trace a shape on the canvas. Dynamic Cognitive Game (DCG) is another example.
    + Question-based: Users are presented a question that they have to answer correctly. Examples include "What is 1 + six?" or "Flower, resting, lawyer, campsite: the word starting with 'c' is?"
    + Fun-based: Users are asked to solve a puzzle or play a game. Bongo is an example. PlayThru from Are You A Human is another example.
    + Invisible: User verification is done in the background without requiring specific user inputs. Google introduced this with its Invisible reCAPTCHA.
    + Hybrid: Combinations of the above.

### How are CAPTCHAs related to AI?

Computers are getting faster and having access to more memory than before. The algorithms that power them are getting better. So tasks that were once difficult for computers are becoming easier. This means CAPTCHAs that were previously unsolvable are becoming solvable by bots.

Advances in optical character recognition (OCR) have enabled bots to solve text-based CAPTCHAs. Image-based CAPTCHAs can be cracked due to advances in image processing, pattern recognition and object recognition. Question-based CAPTCHAs rely on advances in natural language processing (NLP). Audio-based CAPTCHAs too can be solved due to advances in speech-to-text processing. With reCAPTCHA, humans assist machines in solving difficult problems. All these contribute to machines getting smarter everyday.

Ultimately, we have to come up with better CAPTCHAs to defeat smarter bots. A CAPTCHA that's secure today may not be so tomorrow. reCAPTCHA's creator Luis von Ahn commented in 2012 that CAPTCHAs could become useless in another ten years.

### What techniques are typically used to solve CAPTCHAs?

Let's note that text-based are presented as images. To solve text-based CAPTCHAs, segmenting the text into individual characters is the first step. One way to enhance OCR is to remove noise. Image transformation such as rotating, shifting, mirroring, warping can lead to better OCR. With audio, waveform analysis can help in solving the CAPTCHA. Machine learning that solves segmentation and the recognition problems simultaneously have been shown to give better results.

Some sites don't implement CAPTCHA in a secure manner. They can be vulnerable due to reuse of session ID of a known CAPTCHA image.

Web services have come up to solve CAPTCHAs: Death by CAPTCHA, 2Captcha. Where the economics make sense, some of these use humans to solve the CAPTCHAs. Some web services forward the CAPTCHA to pornographic sites. Visitors need to solve the CAPTCHA before they can view pornographic content. It's been argued that this is not really economical.

### Are there guidelines to making good CAPTCHAs?

Some guidelines are worth noting:

    + Accessibility: Give users options. Audio CAPTCHAs are suited for the visually impaired. Allow users to request another CAPTCHA if a particularly hard one comes up. With touchscreen interfaces, text-based CAPTCHAs are less suited compared to CAPTCHAs that require clicks or drag-and-drop actions.
    + Dynamic generation: Avoid serving CAPTCHAs from a fixed database. CAPTCHAs should be generated dynamically. Avoid using trivial distortions. Avoid reuse of CAPTCHAs.
    + Security: Avoid sending the solutions to the client along with the challenge.

### What are reCAPTCHAs?

reCAPTCHA was invented by researchers at Carnegie Mellon University in 2007. It used distorted characters of text. Rather than using just one word, a pair of words was presented to the user. One of these words is known to the CAPTCHA system but the other one is unknown. Such unknown words, rather than being randomly generated, were picked up from old books or articles that were being digitized. Since state-of-the-art OCR systems had difficulty deciphering these words, why not crowdsource this task to humans via CAPTCHA? It was with this idea that reCAPTCHA was born.

Thus reCAPTCHA in its original form helped in digitizing scanned text. In 2008, it was claimed that the system had transcribed 440 million words via 40,000 sites.

reCAPTCHA was acquired by Google in 2009. It evolved into No CAPTCHA reCAPTCHA (aka reCAPTCHA V2) in 2014. In 2017, Google introduced Invisible CAPTCHA. With these newer versions, the CAPTCHA system tracks user behaviour to determine if it's coming from a bot. This includes mouse movement, scrolling of the page, time taken to submit a form, and many more.

### How effective are CAPTCHAs in stopping bots?

One of the early CAPTCHAs, Gimpy, was used by Yahoo. In October 2002, a program was able to solve Gimpy CAPTCHAs. Back in 2005, W3C stated that many CAPTCHA systems could be solved with 88-100% accuracy. At PARC, one researcher was able to crack Assira CAPTCHA with 7.5% probability. Blogger Jeff Atwood reports that CAPTCHAs of Yahoo, Hotmail and Google were broken in early 2008.

A Stanford team showed in 2010 that their Decaptcha tool could bypass many text-based CAPTCHAs. At a conference in 2012, a program was able to solve Google's audio CAPTCHA 99.1% of the time. In 2013, Vicarious AI claimed to be able to solve CAPTCHAs 90% of the time. Google itself reported that distorted text can be solved by AI with 99.8% accuracy. Using machine learning, researchers in 2014 were able to crack reCAPTCHA with 33% success rate and Baidu at 39%.

This does not mean that CAPTCHAs are useless. It just means that current CAPTCHAs have to be better than what AI is capable of solving.

### Are there alternatives to using CAPTCHAs?

WCAG Working Group of W3C has a list of alternatives to using CAPTCHAs. Another list is by Karl Groves. It's important to note that alternatives may not work all the time since smart bots may find a way, now or in the future, to bypass them.

Honeypots and timestamp analysis are alternatives that have proven effective. Another option is to use an anti-spam service such as Akismet, Mollom and SBlam. It's possible to enforce user verification via emails or text messages sent to their mobiles. While this may defeat bots, it reduces usability for humans. Game-based CAPTCHAs are still CAPTCHAs but they are less annoying and more fun to users. It has been claimed that PlayThru takes on average 10-12 seconds compared to 12 seconds of text-based CAPTCHA.

### What are the accessibility issues surrounding CAPTCHAs?

In response to smarter bots, CAPTCHAs have gotten more sophisticated. Unfortunately, this makes it harder for humans as well. A study from 2009 found the while CAPTCHAs reduced spam, they also reduced conversion rates. Another study with Animoto web app showed that conversion rates were better by 33% without using CAPTCHAs. A survey from 2010 showed that audio CAPTCHAs for non-native speakers of English are hard. Even with text-based CAPTCHAs, some text can be hard to solve. In 2000, success rate was 97% but this dropped to 92% in 2012.

It has been claimed that CAPTCHAs ignore issues that senior citizens and visually impaired face. A study from 2009 with visually impaired showed that with audio CAPTCHAs success rates was only 45% and it took users 65 seconds to solve one.

Harry Brignull has even questioned the approach, "Using a CAPTCHA is a way of announcing to the world that you’ve got a spam problem, that you don’t know how to deal with it, and that you’ve decided to offload the frustration of the problem onto your user-base."

### Can you name some providers of CAPTCHAs?

Google's reCAPTCHA is the well known. As of March 2017, more than million websites are using it. Also, 11.2% of the top 10k sites are using it. NoMoreCaptchas is a possible alternative to Invisible reCAPTCHA since it does not require explicit input from users.

PICATCHA is an image-based CAPTCHA that also uses it as an advertising medium. Microsoft had a research project named Asirra that used image-based CAPTCHA. Confident CAPTCHA claims 96% success rate and usage of 50 million verifications per month. Other examples are Ironclad CAPTCHA, PlayThru, NuCAPTCHA and Solve Media. Site captchas.net is free service. BotDetect CAPTCHA uses image and sound. JCAPTCHA, implemented in Java, can be downloaded and integrated into websites. Dice Captcha shows pictures of dice.

### In my automated tests, how can I test form submissions that have CAPTCHAs?

One approach is to disable CAPTCHAs during testing. Depending on the system, this could be in code or a flag in the database. Attempting to automatically solve CAPTCHAs is a fundamentally flawed approach. If test automation can solve, it really suggests that the CAPTCHA is not strong enough to prevent bots from passing themselves off as humans.

## Milestones

1996

DEC uses a pixelated image of the US flag as CAPTCHA for gathering opinion polls before the US Presidential Election 1996. This method doesn't work very well. Simple programs could click the flag correctly.

1997

Scientists at DEC led by Andrei Broder design a noise-induced text-based CAPTCHA. This is used by AltaVista to prevent bots from adding URLs to the search engine's platform.

2000

Luis von Ahn, Manuel Blum, Nicholas Hopper and John Langford of Carnegie Mellon University come up with a better text-based CAPTCHA. They coin the term **CAPTCHA**.

2007

**reCAPTCHA** is invented at Carnegie Mellon University. Later, as part of Google, Google deprecates this version in May 2016, and shuts it down in March 2018.

Dec
2008

Inventors of reCAPTCHA introduce the audio version of the same.

Sep
2009

Google acquires reCAPTCHA since this can also assist with digitization as part of Google Books and Google News Archive Search. Also known as **reCAPTCHA v1**, it's shutdown in March 2018.

Dec
2014

Google releases reCAPTCHA v2, also known as **No CAPTCHA reCAPTCHA**. Users merely have to click a checkbox that says "I'm not a robot".

Mar
2017

Google launches a variant of reCAPTCHA v2 and calls it **Invisible reCAPTCHA**.  This does not require explicit user inputs.

Oct
2018

Google launches **reCAPTCHA v3**. This doesn't require users to solve any CAPTCHA and gives greater control to website owners. reCAPTCHA v3 returns a score (1.0 for good interaction, 0.0 for a likely bot). Based on this score, website owners can take appropriate action. In March 2019, it's reported that reCAPTCHA v3 has been partially tricked by an AI program using Reinforcement Learning.

CAPTCHA


# MongoDB View

## Summary

MongoDB's aggregation pipeline processes documents of a collection through one or more stages. Views are read-only results of such aggregations. With views, consumers can simply read those results without needing to execute the aggregation pipeline again.

MongoDB's views are **non-materialized** by nature, that is, the views are created and updated on the fly. Results are not persisted on disks. Since MongoDB 4.2, **on-demand materialized views** are also possible with the `$merge` pipeline stage. Such views are stored in an output collection that are updated every time the pipeline is executed.

Views are useful for many use cases. They bring benefits in terms of data privacy, performance tuning, code reuse, and analytics. Non-materialized views take up very little storage: view data is not stored, only view definition is stored. Views add a layer of abstraction between the application and complex aggregation pipelines.

## Discussion

### What are some use cases for MongoDB views?

An organization's employee data can have sensitive information. When exposing this data to applications, personal and private information can be excluded in a view. An alternative to exclusion, is to pseudonymize fields, which is necessary for regulatory compliance such as for GDPR. Applications have access only to the view, not the underlying collection.

In IoT, applications may not require every single measurement provided by the sensors. In many cases, they require only aggregates: counts, min/max values, averages, etc. A view stores these aggregates and keeps them updated as new data arrives. A specific example is a collection that contains daily rainfall. A view stores monthly averages, which can then be more easily queried for driest and wettest months.

When analysis requires data from multiple collections, views can offer applications a simpler interface. Applications need not be burdened with the underlying complex pipelines. For example, a view can bring together data from Inventory and Orders collection in an e-commerce application.

### Could you describe MongoDB view with an example?

Consider Game of Thrones data. Documents of the `characters` collection has names plus status. Status is optional and its value is either dead or alive. Status field can be considered as a spoiler. We wish to hide it for some consumers of this data.

We project the status field thus, `"status" : { "$cond": [{ "$eq": [ "$status", undefined ] }, $status = "Not Specified", $status = "*****" ] }`. If undefined, we set it to `Not Specified`. Otherwise, we mask it with `*****`.

The above projection can be used in a pipeline on the `characters` collection and the result stored in a view named `charactersNoSpoil`: `db.createView("charactersNoSpoil", "characters", <pipeline>)`. In addition, we could also include only some fields in the view via the projection.

Once views are setup this way, we can restrict access using `db.createRole()` and `db.createUser()`. It's also possible to enable auditing and analyse the logs to verify that access privileges are working as expected.

### Can I create materialized views in MongoDB?

Since MongoDB 4.2, on-demand materialized views are possible. The output can be stored in a collection in the same or different database. The output collection could be a sharded one. If collection doesn't exist, it's created. Results can go into an existing collection with the ability to merge/replace/keep existing documents, fail the operation or process documents with an custom update pipeline. On-demand materialized views are implemented via the `$merge` pipeline stage.

The figure shows an example of `monthlybakesales` materialized view created from `bakesales` collection. The pipeline preceding the view groups data by year and month, and stores sales quantity and amount. When new sales figures are obtained for January 2019 and later, the view is updated by calling `updateMonthlySales(new ISODate("2019-01-01"));`.

In the above example, the input and output are in the same database. If we plan to store the view in a different database named `reporting`, we can use the following syntax within `$merge`: `into: { db: "reporting", coll: "monthlybakesales" }`. For merging, documents are identified using `_id` field by default but `on` field can be used to customize this.

### How do I deal with indexes for MongoDB views?

Views use indexes of the underlying collection. This also means that we can't create, drop or rebuild indexes directly on views.

For on-demand materialized views, it's possible to create indexes. Indexes have to be unique as mandated by the `$merge` stage. An example of creating a unique index is `db.metricsByWeek.createIndex({week:1}, {unique:true})`. This can be used for merging with the field `on: "week"` or `on: ["week"]`.

## Milestones

Feb
2009

**MongoDB 1.0** is released. By August, this version becomes generally available for production environments.

Nov
2016

MongoDB 3.4 is released with support for **creating read-only views** from existing collections or other views. Options `viewOn` and `pipeline` are added to the existing `create` command. On the `mongo` shell, the corresponding helper is `db.createView()`.

Aug
2019

MongoDB 4.2 is released. With the newly added stage `$merge`, we can create **on-demand materialized views**. Prior to MongoDB 4.2, `db.createView()` obtained exclusive lock on the parent database. With this release, exclusive lock is obtained only for the input collection or view. MongoDB Compass 1.19 is released. There's now support for **views in Compass**.

Jul
2020

MongoDB 4.4 is released. When running `find` operation on a view, `$natural` sort is now possible. This is the order in which the database refers to documents on disk.

MongoDB View


# 5G NR Measurement Gaps

## Summary

Measurement gaps are opportunities given to the UE to perform measurements on downlink signals. A UE can't perform inter-frequency or inter-RAT measurements while also transmitting or receiving. Even for intra-frequency measurements, a 5G UE may require measurement gaps if such measurements are to be performed outside the UE's currently active Bandwidth Part (BWP).

The network configures a UE with measurement gaps via RRC signalling. The network configures these gaps such that they don't coincide with UE transmissions or receptions. It's possible to start with a few gaps and later reconfigure the UE with more gaps to gather neighbour cell measurements, for instance if a handover looks likely.

Measurement gaps are periodic. A UE may be configured with multiple measurement gaps. UE RRC informs Layer 1 of these gaps. Layer 1 obeys these gaps for making measurements. Collected measurements are reported to the network either at Layer 1 or RRC.

## Discussion

### What are the different types of 5G NR measurement gaps?

There are three types of gap patterns: gapUE, gapFR1, and gapFR2.  More commonly, these are called **per-UE** or **per-FR** gap types.

If the UE is capable, the network may signal separate gap configurations for FR1 and FR2. Otherwise, gapUE is used, that is, these gaps can be used for measurements in both FR1 and FR2. When either gapFR1 or gapFR2 is configured, gapUE can't be configured for the UE.

In NGEN-DC and EN-DC, signalling is over E-UTRA, not 5G NR. Hence, gapFR1 and gapUE are set up by LTE RRC.

In NE-DC and NR-DC, signalling is over 5G NR. Hence, gapFR1 and gapUE are set up by NR RRC. gapFR2 is always setup via NR RRC signalling. In NR-DC, the gaps must be associated with MCG.

### What constitutes a measurement gap configuration?

A measurement gap configuration has the following elements:

    + **Measurement Gap Repetition Period (MGRP)**: Specifies the gap period. Values include 20, 40, 80, 160ms. For example, for 40ms the gap repeats every four frames.
    + **Gap Offset**: Specifies the starting subframe when the gap starts. Being relative to period, it's range is 0 to mgrp-1.
    + **Measurement Gap Length (MGL)**: Specifies the duration of the gap in milliseconds. Values include 1.5, 3, 3.5, 4, 5.5, and 6ms. For positioning measurements, 10 and 20ms are applicable.
    + **Measurement Gap Timing Advance (MGTA)**: UE starts measurements in advance of the subframe when the gap starts. This could be 0, 0.25 or 0.5ms. For FR2, 0 and 0.25ms are applicable.
    + **Reference Serving Cell Indicator**: Applicable for NE-DC and NR-DC, this indicates which cell's SFN and subframe numbering to use for gap calculation.The 5G system has what's called a System Frame Number (SFN), a frame being 10ms long. This is divided into 10 subframes, each of 1ms duration. A measurement gap is configured by the following equations:

$$SFN\;mod\;(mgrp/10)\;=\;FLOOR(gapOffset/10)\\subframe\;=\;gapOffset\;mod\;10$$

Applying this to the figure, we get `SFN mod 4 = 2` and `subframe = 4`.

### What are measurement gap pattern configurations?

The standard specifies 26 different **gap pattern configurations**. Each pattern is a combination of MGL and MGRP.

Not all patterns are always applicable. The standard defines applicable patterns with respect to gap type, serving cell type, and measurement purpose. A UE can indicate if it support per-FR type. Otherwise, it's expected to support all gap pattern configurations that are applicable.

### In 5G NR, how does a measurement gap relate to SMTC?

To make neighbour cell measurements, the network informs the UE the timing of neighbour cell SSBs via what's called SSB Measurement Timing Configuration (SMTC). UE will measure all SSBs that fall within a configured SMTC window. This window is contained with a measurement gap. Network configures measurement gap length and SMTC window based on SSB burst periodicity.

SSB burst periodicity can have values 5/10/20/40/80/160 milliseconds. However, a UE in connected mode need not measure so often, particularly under good channel conditions. In such cases, SMTC window periodicity can be longer.

Two examples are shown in the figure. We also note that the entire measurement gap is usually not used for measurements. A UE requires some time for retuning its RF transceivers. Hence, actual measurement duration is smaller than the measurement gap length.

### Why does a 5G UE require the Measurement Gap Timing Advance (MGTA)?

In some cases, measurement gap starts at the same time as SMTC window. Given the time needed to retune RF, this implies that UE may miss some SSBs that it's supposed to measure. This is solved by informing the UE to advance the start of the configured measurement gap. This advance is 0.5ms for FR1 and 0.25ms for FR2.

### What is measurement gap sharing in 5G NR?

A UE may need measurement gaps for both intra-frequency and inter-frequency measurements; or in either case, the SMTC window fully overlaps with the configured measurement gap. In such cases, configured measurement gaps may need to be shared between intra-frequency and inter-frequency measurements. The gaps could be per-UE or per-FR.

The sharing is determined by RRC signalling of \(X\) that can take four values: equal split, 25, 50, 75. It's applied as follows:

$$K\_{intra} = 1 / X \* 100 \\ K\_{inter} = 1 / (100 – X) \* 100$$

Details of measurement gap sharing are defined in TS 38.133 for four scenarios: EN-DC, SA, NE-DC and NR-DC.

Sharing affects the calculation of Carrier-Specific Scaling Factor (CSSF) that scales the measurement delay requirements defined in the standard.

### What are autonomous gaps in 5G NR?

As part of reporting configuration, the network may ask the UE to report Cell Group Identity (CGI). This report is not exactly a measurement but it requires the UE to acquire system information from neighbour cells. The relevant messages to decode are MIB and SIB1. For doing this, IE `useAutonomousGaps` may be configured.

Autonomous gaps is when the network doesn't configure measurement gaps for the UE. UE autonomously selects suitable gaps to receive system information of neighbour cells.

## Milestones

Dec
2017

3GPP publishes Release 15 "early drop". In this release, measurement gaps and gap patterns are partially defined. Some details are marked for further study.

Jul
2020

3GPP publishes Release 16 specifications. SSB-based inter-frequency measurements can be done without measurement gaps if SSB is within active DL BWP of UE. UE reports some measurements to assist the network configure measurement gaps for location-related measurements.

Jan
2021

For positioning measurements, MGL of 10 and 20ms are introduced.

5G NR Measurement Gaps


# Software as a Service

## Summary

Software-as-a-Service (SaaS) is a software delivery model that allows multiple users to use a software hosted on a cloud server. It has replaced the traditional model of software installation on on-premise servers and desktop computers.

SaaS leads to rapid deployment, better user adoption, reduced support needs and less maintenance overheads. Customer support is also faster and more efficient than traditional on-premise software. Salesforce, Dropbox, Cisco and Webex are some famous examples of SaaS providers.

While adoption of SaaS solutions has been exponential, there are still some organizations reluctant to adopt SaaS mainly due to the risk of data security and limited customization.

## Discussion

### What's the reason of exponential growth of SAAS?

SAAS has replaced the traditional on-premises software which needed to be installed at organizations. On-premise software have many hidden costs in terms installation, maintenance, personnel training or even downtime. SAAS has eliminated the hardware installation & maintenance cost of software. It allows the users to only focus on their dedicated services without thinking about availability, speed or maintenance of the software. Also, it is very easier to use than traditional on-premises software.

In SaaS deployment models, software vendors are responsible for the reliability of the software delivery platform, IT infrastructure, software development, hardware maintenance, security, user support and data backup. The consumer can be assured of the reliability of the hosted software service through a contract called a service level agreement (SLA).

SaaS proved to be a profitable model for both consumers & software vendors. While consumers could avoid the overhead of maintenance & in-house IT resources, vendors could have more economic benefits in software production and distribution and a shorter product development lifecycle.

Due to these competitive advantages over traditional software SaaS grew exponentially over the years.

### What are some famous examples of SaaS?

There are multiple examples of SaaS based companies, that not only affect our day-to-day lives but also generate millions of dollars in revenue. Some famous examples of SaaS are as following-

**Salesforce**: It is a cloud-based CRM (Customer Relationship Management) used to boost the sales of organizations by managing all the leads and prospects at one place.It is one of the oldest & most esteemed examples of SaaS. It was launched in the year 1999. It generates a revenue of 13.28 billion USD.

**Dropbox**: It is a cloud-based file storage platform used to store and access files worldwide. It's one of the reputed startups of USA. It was launched in the year 2007 & generates a revenue of 1.6 billion USD.

**Google Applications (G suite)**: Google provides a bunch of SaaS applications inside G suite such as Gmail, Google Docs, Google Sheets, Google Drive & many more which makes our daily professional life very easy & convenient. It was launched in the year 2006 & generates a revenue of 2.6 billion USD.

### What are different types of SaaS architectures?

Different types of SaaS architectures as following -

**Monolithic Architecture**: The monolithic SaaS software is an indivisible module consisting a big database & a solution that is built around a server-side and client-side interface. All the functions are managed and served at one location.

**Microservices Architecture**: In this architecture, each function of the SaaS product is an independent module that can be deployed separately as required. Application Programming Interfaces (APIs) enable modules to communicate with each other and sync their independent processes to work as one single entity. Each service can be upgraded, updated, and scaled separately.

**Single-tenant Architecture**: This architecture serves a dedicated client. Dedicated software instance, infrastructure & database is given to the client.

**Multi Tenant Architecture**: Unlike single tenant architecture, multi-tenant architecture allows sharing resources. Each tenant's data is secured and protected.

### What does it take to build a SaaS product?

Below are the steps to create a SaaS product-

**Analyze the market**: The prior step is to analyze the market. Questions such as who is the target customer?, who are the competitors?, what is the problem that the product will solve? should be asked.

**Develop a business & marketing plan**: Plan for making the SaaS product unique, profitable & successful in solving the targeted problem.

**Define SaaS requirements** There are many common characteristics of SaaS solutions such as multi-tenancy, optimized solution and data security . Decide the features & functionalities of the product.

**Choose the technical stack**: The technology stack is list of programming languages, frameworks and tools used to develop the software. Decide the preferred technologies for front-end, back-end & database of the SaaS product.

**Choose the SaaS hosting provider**: There are many SaaS hosting providers such as Amazon (AWS), Google, Microsoft, Heroku and many more. Choose any of them to host the software.

**Create the team**: For the success of the SaaS product, a team of experts in different fields such as marketing & designing should be made. One can also hire a software development company for developing the software.

### What are some important characteristics of SaaS?

Following are some important characteristics of a SaaS product-

**Total cost of ownership (TCO)**: There are no hidden costs in SaaS.

**Data Security**: Data security is an important aspect of any SaaS product. Due to multiple users, it's important to encrypt the data of each user.

**Scalability**: Resources are allocated/released dynamically to serve immediate performance requirements. SaaS can scale vertically (more powerful resources) or horizontally (more resource instances).

**Multi-tenant**: Multi-tenant is a method in which a single software hosted on a remote server, serves multiple clients(tenants). Muti-tenancy allows faster updates deployment.

**Personalised Customization**: Unlike the on-premises software, users can customize the software accordingly at both UI level & software business logic level.

**Optimized utilization** Customers can purchase exactly the requirements(data storage, bandwidth, software features) & can extend it whenever they want.

**Subscription fee**: The consumers pay a monthly/annually fee to use the software. Hence, they save the costs of software installation, maintenance & upgradation.

**Reliability**: The requirements of businesses to run 24/7 is fulfilled efficiently with SaaS. Many vendors mention the reliability promise as an integral part of their contract, guarranting an uptime of upto 99.5 per cent.

### What is Open SaaS?

Open SaaS is a form of SaaS that is built on open-source code. Such software are initially developed by an individual or organization but later used & improved by many developers. OpenStack, OpenNebula, Cloudify, Wordpress, OpenShift and CloudStack are good examples of Open SaaS.

Major upgrades & product updates are controlled by the central provider whereas the community of users also contribute to the development of the software.

Whereas Wordpress is developed by the company Automattic, it is completely open source, in terms of both licensing & practical usage. Automattic never charged a penny to the thousands of Wordpress-based website owners while many developers who contributed in making continous improvements in WordPress code base and plug-ins never received a dollar for it.

### What's the difference between vertical & horizontal SAAS?

Horizontal SaaS is a type of SaaS that targets a wider network of businesses, regardless of their industry. Examples of Horizontal SaaS are QuickBooks (accounting) and Salesforce (CRM). Horizontal SaaS is more familiar than Vertical SaaS. Due to targeting a wider market, Horizontal SaaS business owners manage to acquire the customers at a lower cost.

Whereas, Vertical SaaS software are made for a specific niche industry. BioIQ(MedTech) & Health Assurance Plan(dental software) are examples of Vertical SaaS software. Due to a industry specific solution, the size of potential market for Vertical SaaS owners is lower. Vertical SaaS solutions are often made by the experts of a certain industry.

### What can be the challenges of adopting SaaS?

Consumers can face some challenges in adopting SaaS. Adopting SaaS model involves risks in terms of data recovery and migration, high switching costs, lack of software customization features and integration with software. There is a risk of losing important data for consumers while migrating the data from one SaaS vendor to another. Service suspension due to lack or delay of payment can also cause data loss. Customers may find it hard to customize features with SaaS. The critical & important data of enterprises should also be kept hidden to other users of the service. Availability of the services is also a big concern. The disruption of the Amazon cloud service in the year 2011, took down a number of websites including Reddit, Foursquare, and Quora. SaaS works on a muti-tenancy model in which data of different users reside at the same location. Intrusion of data of one user by another becomes possible in this environment.

Weak cloud standards can also resist enterprises from adopting SAAS. SAS 70 Audit is a common cloud standard. ISO 27001 is a better cloud standard than SAS 70 Audit.

## Milestones

1999

Salesforce, a company based on the complete SaaS software is released. It serves the purpose of Customer Relationship Management (CRM).

2006

Amazon introduces Amazon Web Services(AWS), providing on-demand cloud computing platforms and APIs to individuals, companies, and governments. Now, developing & hosting the SaaS software becomes easy for SaaS developers.

2020

The market size of SaaS reaches to 157 billion US dollars.

Software as a Service


# Qubit

## Summary

In classical computing and digital communications, a bit is the most basic unit of information. Similarly, a qubit or quantum bit is the basic unit of quantum information in quantum computing. A qubit is the quantum version of the classic binary bit physically perceived by a two-state device.

A quantum computer can utilise a variety of basic particles such as electrons or photons. In fact, ions accomplish success through their charging or separation, which serves as a symbol of 0 and/or 1. A qubit is a unit of measurement for each of these particles. The character and conduct of these particles, as demonstrated in quantum theory form the basis of quantum computing. The two most important aspects of quantum physics are the principles of Superposition and Entanglement.

## Discussion

### What is the physics of Qubit?

**Qubit is a two-dimensional quantum-mechanical system (or two-dimensional) system**, one of the simplest quantum systems that reflects the rarity of quantum mechanics. Examples include electron rotation where two levels can be considered as spin up and spin down; or polarization of a single photon in which two regions can be considered as direct polarization and horizontal polarization.

In the classical system, the bit will have to be in 0 or 1. However, quantum mechanics allows qubit to occupy a high degree of coherence for both regions simultaneously, a basic position in quantum mechanics and quantum computing.

To create a qubit, an object capable of achieving quantum superposition between two states is required. One type of qubit is an atomic nucleus. The orientation of its magnetic moment, that is, its "spin", can point in different directions in relation to a magnetic field, such as up or down. The difficulty is in locating and then dealing with that solitary atom.

### How is Qubit represented in the form of an equation?

The two basis states (or vectors) are commonly expressed as 0 and 1 (pronounced 'ket 0' and 'ket 1'), as this corresponds to the standard bra-ket nomenclature for quantum states. As a result, a qubit can be viewed as a quantum mechanical counterpart of a traditional data bit. A linear quantum superposition of those two states is a pure qubit state. **Each qubit can be represented as a linear combination of 0 and 1 in this way:**

**where α and are β the amplitudes of complex probability.**

**α and β are constrained by the equation:**

**\(∣α∣^2 + ∣β∣^2 = 1\).**

The probability amplitudes, α and β, encode more than just the probabilities of the outcomes of a measurement; the relative phase between α and β is for example responsible for quantum interference, as seen in the two-slit experiment.

### How is a Qubit different from a Bit?

Qubits are counted in the same way that bits are, using the binary system of 0 and 1. A bit's state can only be 0 or 1, whereas a qubit's state can be both 0 and 1 at the same time.

According to quantum mechanics, a qubit's general state can be a coherent superposition of both. The duration of qubit coherence, is used to compare the quality of qubits as it indicates how long a qubit keeps its information and, as a result, determines its lifetime.

A measurement of a qubit, would disrupt its coherence and irreversibly damage the superposition state, although a measurement of a classical bit would not. One bit can be entirely encoded in a single qubit. A qubit can store more data, such as up to two bits utilizing superdense coding. Superdense coding, often known as dense coding, is a quantum communication protocol in quantum information theory. It allows a sender and receiver to communicate a number of classical bits of information using only a few qubits, given that the sender and receiver had previously shared an entangled resource.

### What technologies help in keeping Qubits in their coherent state?

**Superconducting Qubits:**

The most sophisticated qubit technology is currently superconducting qubits. Most existing quantum computers, including the one that "surpassed" the world's fastest supercomputer, use superconducting qubits. They use Josephson junctions, which are metal-insulator-metal sandwiches. Scientists chill these materials to incredibly low temperatures to make them into superconductors - materials that transmit electricity without loss. Pairs of electrons, for example, pass through the material as if they were single particles. The quantum states are more long-lived as a result of this movement than in conventional materials.

**Defects and Qubits:**

The movements of electrons in materials are altered by the gaps formed by deficiencies in a material's structure. These gaps confine electrons in particular quantum materials, allowing researchers to access and regulate their spins. Unlike superconductors, these qubits don't have to be kept at extremely low temperatures always. They have the potential for lengthy coherence times and could be mass-produced.

Quantum computing can also use three-state or multilayer states, which can be decoupled to conduct operations in qubits, while two-state systems are the most common. The storage is always consistent, which helps with data storage and the development of complex systems.

### What does it mean for two Qubits to be entangled?

**Entanglement** is another counter-intuitive phenomenon in quantum physics. When each particle's quantum state cannot be characterised independently of the quantum state of the other, a pair or group of particles is said to be entangled. Although the parts of the system are not in a distinct state, the quantum state of the system as a whole can be stated.

There is a particular link between two qubits when they are entangled. The results of the measurements will reveal the entanglement. The measurements on the various qubits could result in a 0 or a 1. However, the result of one qubit's measurement will always be correlated with the result of the other qubit's measurement. Even when the particles are separated by a considerable distance, this is always the case. The Bell states are an example of such states.

### What is Qudit?

**The superpositions that qubits can allow them to assist in the execution of two calculations at the same time.** When two qubits are quantum-mechanically coupled, or entangled, they can aid in the simultaneous execution of four calculations; three qubits, eight calculations; and so on. As a result, a quantum computer with 300 qubits might execute more computations in an instant than the known universe's atoms, solving certain problems significantly faster than classical computers. However, superpositions are extremely fragile, making working with several qubits problematic.

**Qudit is a multi-level computational unit that surpasses the two-level qubit.** Qudit, as compared to qubit, has a broader state space in which to store and process information, allowing for a reduction in circuit complexity, simplification of the experimental setup, and improved algorithm performance.

## Milestones

1959

Richard Feynman states the possibility of using quantum effects for computation.

1973

Alexander Holevo publishes a paper showing that n qubits can carry more than n classical bits of information, but at most n classical bits are accessible (a result known as "Holevo's theorem" or "Holevo's bound"). Charles H. Bennett shows that computation can be done reversibly.

1976

Polish mathematical physicist Roman Stanisław Ingarden publishes a seminal paper entitled "Quantum Information Theory" in Reports on Mathematical Physics, vol. 10, 43–72. It is one of the first attempts at creating a quantum information theory, showing that Shannon information theory cannot directly be generalized to the quantum case, but rather that it is possible to construct a quantum information theory, which is a generalization of Shannon's theory, within the formalism of a generalized quantum mechanics of open systems and a generalized concept of observables (the so-called semi-observables).

May
2016

IBM Research announces that for the first time ever it is making quantum computing available to members of the public via the cloud.

Oct
2017

IBM Research scientists successfully "break the 49-qubit simulation barrier".

2017

IBM, Intel, and Google each report testing quantum processors containing 50, 49, and 72 qubits.

Jun
2018

Intel begins testing a silicon-based spin-qubit processor.

Mar
2012

Caltech physicist John Preskill describes the moment when "well-controlled quantum systems can perform tasks surpassing what can be done in the classical world" as the arrival of "quantum supremacy."

Oct
2019

Google announces it has achieved quantum supremacy - marking a huge milestone in the advancement of practical quantum computing.

Qubit


# JavaScript to TypeScript Migration

## Summary

Large projects can benefit from migrating their JavaScript code to TypeScript. Due to TypeScript's support for static types and good tooling, type-related problems can be caught during development and at compile time. However, migrating a large codebase can be a daunting task but this is easier than imagined in TypeScript.

Since TypeScript is a superset of JavaScript, the TypeScript compiler can deal with JavaScript files as well, leaving them untouched and skipping type checks. This means that migration can happen incrementally, file by file. Tools are available to refactor code during the migration process.

It's also been observed that the workflow for TypeScript projects is lot easier now than what it used to be. This means that TypeScript can be adopted for even small projects.

## Discussion

### Could you share some case studies of JS-to-TS migration?

Lucidchart migrated their entire codebase of 600k lines across 2840 files in just three days, resulting in 500k lines of TypeScript code. About 80% of the conversion was aided by tools. They made a dependency graph in a spreadsheet, started converting from the leaves and moved up file by file. At each step, they checked that TS files compiled successfully. After pushing to production, the team expected issues to arise but none were reported.

One developer reported that his code size (involving 180+ files) increased 30% when moving from JavaScript to TypeScript. In general, this may be acceptable in large projects where the benefits of type checking are far greater.

The fact that all JavaScript is valid in TypeScript was seen in a Dojo-AMD project. AMD module imports were converted to ES6-style understood by TypeScript. Loading of plugins and exporting of modules were updated. Otherwise, no additional changes were required.

### What's the recommended procedure for migrating my JavaScript codebase to TypeScript?

The simplest approach is to create a new TypeScript project and add all legacy JavaScript code into it. TS compiler will understand JS source code, although without the type checking. This is possible due to `allowJs` option. Any new files that you create should be in TypeScript.

You can start adding type checking to existing JS files using type declaration files, of extension `*.d.ts`. Use the `types` option when invoking the TS compiler. At a later stage, you can refactor your JS files into TS files, at which point type declaration files become redundant.

Gradually, legacy code can be migrated. Simply rename `*.js` files to `*.ts` files. Then start introducing type declarations in the legacy files, although TypeScript's type inference can resolve types in most cases. Linting can throw up a lot of issues when JS files are renamed to TS files. One approach is to disable linting by adding `// tslint:disable` at the start of all legacy files. Then, enable linting file by file when you start working on them.

### Do you have any tips for refactoring JS code to TS?

Since TypeScript is object-oriented, convert function constructors to class constructors. Give member variables explicit access modifiers. Use `public` but you can restrict access using `protected` or `private`. Replace instance methods with arrow methods, which will use `this` reference to the object. Replace prototype methods with methods.

To pass TS compilation, you may be tempted to use `any` type: avoid doing this. However, since migration is often incremental, allow TypeScript to automatically infer the `any` type if type is not specified. You can suppress this behaviour later using the `noImplicitAny` option.

Legacy code might contain runtime type checks and related exception handling. Because now type checking is done at compile time, these extra bits of code can be removed. This can even improve performance.

When migrating security code, it's recommended that critical code be migrated to TS first. App and third-party code can be migrated later.

### What extra configuration should I do in a TypeScript project?

JavaScript executes directly within a JS runtime. This is not possible with TypeScript. TypeScript files have to converted to JavaScript (called *transpilation*) before they can be executed. Hence a TypeScript project must be set up to automate this conversion.

**Webpack** can be set up with a config file `webpack.config.js`. This can make use of `ts-loader` to load TS files and `json-loader` for JSON files. Type declarations for third-party JS modules can be installed from **@types**. For example, to install "prompt-sync" you can do `npm install --save-dev @types/prompt-sync`. Another possible loader to use is `awesome-typescript-loader`. This can be combined with `source-map-loader` to assist source level debugging.

Note that it's common for JavaScript projects to use a transpiler to convert between different versions of JavaScript. This is particularly true on the web where code has to run correctly on old browsers. If your JS project is already setup to do this, migrating to TS is even easier. For example, a React project might be using Webpack and Babel. This can be reconfigured so that Webpack reads the output of TypeScript compiler rather than the original JS code.

### As a beginner, should I learn JavaScript before moving on to TypeScript?

TypeScript is a superset of JavaScript. It doesn't hide JavaScript syntax. For this reason, beginners are encouraged (but not required) to learn JavaScript. In fact, TypeScript shouldn't be used to bring non-JS developers to the frontend.

Before migrating to TypeScript, it's good to ask what's the background of those who will be working on it. In one instance, a team of C# developers could quickly learn TypeScript for frontend work. In fact, developers with a background in object-oriented programming will be able to pick up TypeScript easily.

### Are there tools to automate JS-to-TS migration?

Back in 2015, JetBrains described how easy it is to use **ReSharper 9** tool to perform conversions. Some of the limitations of the tool are possibly removed in later updates. The tool allows us to convert a file, an entire folder, a project or a solution.

Npm package js-to-ts-converter may also help. Airbnb's ts-migrate may also help to automate the migration. However, a manual follow-up is needed to improve type safety.

If you're using Closure-annotated JS, **gents** (open sourced by Google) can be used to convert to TypeScript. Related tools are clutz and tsickle.

## Milestones

Feb
2016

TypeScript 1.8 starts supporting compiler option `allowJs`. This makes it easier to directly use JS files within a TypeScript project. TypeScript compiler will do basic sanity checks on JS files but otherwise pass them to the output.

Jul
2016

Type definitions are now available via NPM at `@types`. This deprecates earlier approaches: DefinitelyTyped, TSD, Typings. As on April 2019, more than 6,300 packages have type definitions.

Dec
2018

**Babel 7** is released with support for TypeScript. This means that TypeScript can be transpiled to JavaScript using Babel. Thus, TypeScript-based projects can continue to benefit from Babel's rich ecosystem and plugins. TypeScript compiler can still be used for type checking.

JavaScript to TypeScript Migration


# Aspect-Based Opinion Mining

## Summary

Aspect-Based Opinion Mining (ABOM) involves extracting aspects or features of an entity and figuring out opinions about those aspects. It's a method of text classification that has evolved from sentiment analysis and named entity extraction (NER). ABOM is thus a combination of aspect extraction and opinion mining. While opinions about entities are useful, opinions about aspects of those entities are more granular and insightful.

The ABOM workflow constitutes initial text pre-processing, POS tagging, splitting sentences to extract aspects and classifying them into various dimensions/buckets.

A 2018 survey states that 90% of the total data in world has been generated in last two years in the form of tweets, text, images and video. ABOM is therefore important in analyzing this unstructured data.

## Discussion

### What is opinion mining and its importance?

Suppose a written or spoken content expresses an opinion about some subject. Extraction of these opinions is called **opinion mining**. For example, "Adam was very satisfied with the flavour of black tea at Starbucks". Here we extract a positive opinion about black tea (entity) whose aspect is flavour.

With increased use of online transactions and service-based industries, there has been immense growth of consumer reviews and opinions through voice or text. Opinion mining finds out sentiments from these reviews and estimates customer satisfaction.

Sentiment analysis and opinion are related. Sentiment analysis is first done to extract positive, negative or neutral sentiments. This leads to opinion mining using various text classifiers.

### What do you mean by aspects and why are they relevant?

Consider the example of "ice-cream" as the entity. Aspects or features of this entity include flavour, temperature, taste, presentation, etc. A person may express that he disliked the ice-cream but this is expressing opinion on the overall entity. If we analyze this deeper, we may find that the person liked the flavour and the presentation but didn't like the taste. Thus, figuring out opinions on specific aspects gives more useful information that can aid decision making.

With "restaurant" as the entity, its aspects could include food, service, price, ambience, etc. Often user reviews or ratings of the restaurant as a whole is not actionable. However, if we know that the food was great but the music was terrible, the restaurant owner can take actions to improve on specific aspects.

### What's the workflow of ABOM?

ABOM involves data collection, data cleaning, extraction of aspects and entities, classification of aspects and entities, sentiment scoring of aspects, evaluation and validation.

Text is typically split into sentences and parsed according to Named Entity Recognition (NER), which extracts the entities. With aspect mining, we find out features or characteristics of these entities.  There are some automated aspect extractor APIs. The output from such APIs can be validated and improved manually.

Data pre-processing is time consuming. Selecting and training aspects of entities are complex tasks. Some approaches include aggregate score of opinion words, SentiWordNet, aspect table, dependency relations, emotion analysis using lexicon and semantic representation.

Opinion mining is usually a rule-based approach where frame certain rules to identify most used words. These are further analyzed and processed. A dictionary-based approach is to curate common words of aspect terms for further classification according to various domains.

### Could you describe the classification of opinions in ABOM?

Classification of opinion completely depends on the dataset and business problem. Opinion classification could be trend based, aspect based, sentence based, etc. from tweets, reviews, blogs and others. The most versatile way of classification holds positive and negative analysis.

Similarly, classification can be done on a multi-aspect basis using co-occurrence of aspects and sentiments, aspect-sentiment hierarchy and polarity classification of sentiments. These are different approaches of classification where aspects are filled in different buckets according to polarity from sentiment scores or design/occurrence of aspects with POS tags.

There are several researchers who work on automatic aspect generation and classification after effective cleaning of unwanted and irrelevant sentences. Insignificant words or sentences make the data noisy and degrade the classification accuracy. To get rid of this, automatic aspect extraction methods like fuzzy aspect based opinion classification can be used. This method covers the demerits of infrequent and coreferential aspects and gives a good classification accuracy.

### What are some techniques for extracting aspects?

Among the **supervised** techniques are Hidden Markov Model (HMM), Conditional Random Fields (CRFs) and dependency tree kernels. Among the **unsupervised** ones are frequency or statistical methods, rule-based methods, and Pointwise Mutual Information (PMI). Latent Dirichlet Allocation (LDA) from topic modelling has also been adapted and applied.

PMI is a score that indicates how often a candidate aspect co-occurs with an entity. Low PMI implies it's probably not an aspect. PMI has been used along with Term Frequency-Inverse Document Frequency (TF-IDF).

Some techniques are able to extract both explicit and implicit aspects. Association rule mining and clustering have been adapted for extracting implicit aspects. Dependency rules and lexicons such as WordNet and SenticNet have been used to identify implicit aspects.

Some techniques jointly model both aspects and opinions. One well-known technique is called **double propagation** that exploits syntactic relations of opinion words and aspects. From known opinion words, aspects can be extracted. From known aspects, new opinion words can be identified; and so on.

Neural network approaches such as CNN, LSTM, and attention mechanisms have been applied as well.

### Could you share some tips and best practices for doing ABOM?

When extracting aspect, we must always know which entity it belongs to. Consider for example "The picture quality of this camera is amazing" versus "I love this camera". In the former sentence, "picture quality" is the aspect. In the latter sentence, camera entity is evaluated as a whole and hence the aspect is "general".

It's essential to consider the domain of application. The phrase "please go and read the book" is a positive opinion for a book review but a negative one for a movie review. This implies that we should train our models with domain-specific data.

Sometimes opinion words on their own are inadequate. They need to be analyzed along with the aspect. For example, even within the same domain of digital cameras, "long" can either be positive or negative depending on the context, such as "long battery life" versus "takes long time to focus".

### For a developer, what are some resources and projects to learn ABOM?

One can start an ABOM project from online reviews. There are many e-commerce and social websites containing thousands of reviews on any topic, person or product. Twitter, Facebook, Amazon, MouthShut, and Yelp are example sources.

These reviews can be scraped using any API or libraries like BeautifulSoup or Scrapy in Python. Some sample projects can be aspect-based sentiment analysis for e-commerce websites or restaurants. In latent aspect rating analysis, ratings were predicted using the latent opinions from review text data. This work can be replicated in different industries such as hospitality, automotive, and education.

### What are some industrial applications of ABOM ?

ABOM applications is spread out across various industries wherever there is customer feedback. Examples include travel, automotive, hotel, tourism, ecommerce and many more.

Usually, before any hotel booking, people read reviews. We can distribute these into several buckets like service, ambience or hygiene. This makes it easier for customers to compare different hotels. Similarly, there are several websites for travel like Yelp, Trip Advisor or MakeMyTrip that categorize the service into relevant aspects and shows it in the form of ratings.

In every industry, reviews are broken down into aspects and rated it according to domain knowledge. Good domain knowledge is essential for better decision making. Before any drug trials or testing, there needs to be a lot of expert opinion from doctors and domain-specific experts. This work takes very long to come to a conclusion. ABOM simplifies it by rating several aspects like adverse reactions, efficacy of a drug, symptoms and conditions of patients.

## Milestones

2000

The early 2000s, sees the birth of social networking sites. Blogs, online reviews, and other forms of unstructured content online become common. This leads to **opinion classification** from web documents, which are assessed for positive or negative sentiments.

2004

Work on opinion mining starts with the realization that importance of opinions on various aspects is more insightful rather than summary of reviews. At this time, ABOM is named **feature-based opinion mining**. Researchers start working on text summarization and **opinion mining** due to large scale availability of customer reviews and an increasing demand for customer satisfaction.

2007

Aspect evaluation relations and aspect-of relations are combined together. Contextual and statistical combination started to be used in text classification based on opinions.

2011

Researchers start to rank aspects according to various web-based opinions after training them. New **opinion search engine** is proposed. This searches for the item along with its summary of sentiments across features. This gives a start to **automatic aspect extraction**.

2013

Aspect based opinion mining is now popular in many industries. This enhances consumer satisfaction and drives **decision making**. Techniques begin to recognize explicit versus implicit aspect expressions. Precision and recall scores from experiments exceed 90%.

2017

Due to millions of online reviews, a user could get confused about the features of product. By scraping online reviews and content, ABOM can now be used as a **personal recommender**. People are given recommendations according to their choices and preferences.

2018

As a **neural network approach** to ABOM, Long Short-Term Memory (LSTM) is used for extracting opinion target expressions (OTEs) and aspect sentiment polarities. Using various types of neural networks is an art now for researchers and we hope to see better accuracy and automation in ABOM across various industries.

Aspect-Based Opinion Mining


# GraphQL

## Summary

Often when client apps wish to get data, they have to work with available REST APIs and live with their limitations. To obtain data, they make multiple API calls. In the process, they end up overfetching data and using only some of it. On slow connections, this dampens user experience.

GraphQL is an alternative to traditional REST APIs but its not meant to replace REST.  You can describe your data with a well-defined GraphQL schema. Clients have the flexibility to get as much data as needed. While API calls are similar to querying databases, GraphQL is not a database language. GraphQL also interfaces nicely with frontend JavaScript frameworks such as React.

GraphQL was created at Facebook and then open sourced. It's now managed by GraphQL Foundation.

## Discussion

### What exactly is GraphQL?

GraphQL is described as "a query language for APIs and a runtime for fulfilling those queries with your existing data". You describe your data using a schema. Clients then ask for exactly what they want and nothing more. Thus, control is with the client rather than the server. With SOAP or REST, clients had to work with well-defined endpoints, often making multiple calls to obtain a complete set of data. With GraphQL, a single API call is sufficient to do the same.

GraphQL is not a database language. GraphQL is also not an implementation. It's an open source specification that can be implemented in any language. A GraphQL service sits between your app and the data. It provides a layer of abstraction. Based on a schema, it does validation on the queries and responses. It's APIs are based on types and fields, not endpoints. Clients no longer need to parse responses, since they get data in a form they want it.

### Why do we need GraphQL when there's already REST API?

It's typical in REST to think of data as a resource. Each resource is exposed via an API endpoint. For example, data about a library would consists of books, authors, branch locations, members, staff, and so on. Each of these is typically exposed via its own endpoint. With GraphQL, the entire data is considered as a single interconnected graph of data points. What does this mean?

Suppose you wish to know about books written by an author and the branches where these are available. With REST, we would need to make multiple queries since the information might come from multiple endpoints. With GraphQL, a single request is sufficient since relationships among books, authors and branches are all part of the same data graph.

A book could be identified by a unique integer. GraphQL can validate this. Errors due to type mismatches are caught early by the GraphQL service instead of allowing applications to fail in unpredictable ways. GraphQL also has a feature called Subscriptions that makes it easier to update applications with real-time data.

### Why does GraphQL treat my data as a graph?

GraphQL provides a layer of abstraction between the application and the data storage. In this abstraction, GraphQL models data as **nodes**. Connections among these nodes are **edges**. Thus, data is a graph of interconnected objects rather than resources accessed via multiple endpoints. We can call this the *application data graph*.

Consider a library's catalogue of books. Books and authors are the nodes and they are interconnected. An author may have written multiple books. A book may have co-authors. If we query for a specific book and all its authors, GraphQL is basically extracting a subtree from the data graph. Let's note that a tree is also a graph but without any loops.

A tree has a hierarchical structure. The beauty of GraphQL is that the query to obtain a subtree from the graph also has a similar tree structure. We can say that the query and its response have same data shape. This makes it easier to write queries and process the responses.

### What are the advantages of using GraphQL?

GraphQL is a specification. Implementations of the GraphQL server in various languages are available. Client implementations are also available to ease the developer's job.

GraphQL does not impose how or where data should be stored. Data may come from multiple databases or REST APIs. There's no need to discard your existing REST API endpoints. GraphQL can work with them. In any case, GraphQL presents a single source of truth. GraphQL eliminates the need to build separate API endpoints for each type of client (web vs mobile).

GraphQL eliminates the "chattiness" or the N+1 problem that's inherent in REST APIs. Another point of efficiency is that data is not overfetched.

A UI component can obtain all the data is needs without worrying about how this maps to databases or multiple endpoints. Data fetching is declarative and UI-driven, which is also why many frontend frameworks (React, Angular, Vue) interface nicely with GraphQL.

GraphQL is strongly typed. With **introspection**, we can retrieve GraphQL schema. Versioning like done in REST APIs is not needed. GraphQL API can be updated at field level.

### What are the phases of a query in GraphQL?

A GraphQL has to be understood by a GraphQL service before it can respond with the correct data. A GraphQL service tokenizes the query string and then parses the tokens to build the Abstract Syntax Tree (AST). It then validates the query against the schema. If invalid, an error response is returned. If valid, the AST may be reduced to a form that's simpler to execute. If there are any defined query analyzers, they will be called. During execution, resolvers are called to obtain the actual data pertaining to each field.

The GraphQL client is not part of the GraphQL specifications but clients simplify processing. Different clients handle the queries and responses differently. For example, Apollo Client normalizes the response and caches it. It also minimizes the original query for efficiency.

### Could you briefly introduce common GraphQL terms?

Apollo Docs has published a handy GraphQL glossary. Another source is GraphQL's learning page. Here we describe some essential terms:

    + **Query**: A read-only operation to get data from a GraphQL service.
    + **Mutation**: While a query could be designed to do data writes, this is not recommended. An explicit mutation is recommended.
    + **Field**: The basic unit of data that we can obtain. GraphQL is in fact about selecting fields on objects.
    + **Fragment**: A set of fields that can be reused across multiple queries.
    + **Argument**: Every field and nested object can have an argument, thus enabling us to filter or customize the results.
    + **Alias**: To avoid naming conflicts in the results, aliases are useful. For instance, we can query the same object with different arguments and get the results in different aliases.
    + **Directive**: This can be attached to a field or fragment to dynamically affect the shape of data. Mandatory directives are `@include` and `@skip` to either include or skip a field based on a condition.

### What are some criticisms of GraphQL?

It's been said that REST can do what GraphQL does. Using JSON API, we can implement data-specific queries. Query parameters can filter data. JSON Schema can be used for type validation. OData can be used as a query language. These alternatives are better for small applications where GraphQL can be an overkill. GraphQL comes with the need to write resolvers.

Error handling is more complex since the response has to be parsed. An error will be returned as a HTTP 200 OK status. This also means that we can't use monitoring tools that mostly ping an endpoint without a request body and look at only status codes.

Unlike POST/PUT requests of REST, GraphQL can't do file uploading.

REST calls conform to HTTP semantics and web caching is easier due to multiple endpoints. With GraphQL, given a single endpoint, web caching is harder to implement. Tools such as Relay or Dataloader can help. Apollo client-server implemented persisted GraphQL queries.

Performance can be an issue since clients can construct complex queries. This is more so for data coming from third parties. With REST, an endpoint can be designed and fine tuned for a specific query.

## Milestones

2012

Facebook's iOS and Android apps become native apps. These need an API to retrieve Facebook's News Feed. RESTful APIs and Facebook Query Language (FQL) are available but developers see a gap between what apps wanted and the queries that servers exposed. Developers had to write lot of code on server side to prepare data and on client side to consume it. It's to solve this problem that Nick Schrock of Facebook creates a first prototype called **SuperGraph**. This is renamed to **GraphQL**.

2015

Facebook open sources GraphQL, which is also released as a draft specification. This move is partly influenced by Facebook's React that was open sourced in 2013, Facebook's React Native that's also open sourced in 2015 and Facebook's Relay that provides a framework for building mobile apps based on React, React Native and GraphQL.

Jun
2018

GraphQL specifications exits "Draft RFC" status. It defines *Type System Definition Language* and delivery of live data over GraphQL subscriptions.

Nov
2018

**GraphQL Foundation** is formed, to be hosted by the Linux Foundation. This paves the way for a vendor-neutral development of GraphQL. By now, GraphQL is being used by Airbnb, Audi, GitHub, Netflix, Shopify, Twitter, NY Times and many more. At Facebook alone, it powers billions of API calls every day. In March 2019, GraphQL Foundation announces collaboration with the Joint Development Foundation.

GraphQL