# SOLID Design Principles

## Summary

In the world of object-oriented programming (OOP), there are many design guidelines, patterns or principles. Five of these principles are usually grouped together and are known by the acronym **SOLID**. While each of these five principles describes something specific, they overlap as well such that adopting one of them implies or leads to adopting another.

Programming in strict conformance to SOLID is generally not expected. However, programmers must be aware of SOLID and use them depending on the context.

In general, SOLID helps us manage code complexity. It leads to more maintainable and extensible code. Even with big change requests, it's easier to update the code.

## Discussion

### What problems are solved or avoided by applying SOLID?

Software may start with a clean and elegant design but over time they become hard to maintain, often requiring costly redesigns. Robert Martin, who's credited with writing down the SOLID principles, points out some symptoms of rotting design due to improperly managed dependencies across modules:

    + **Rigidity**: Implementing even a small change is difficult since it's likely to translate into a cascade of changes.
    + **Fragility**: Any change tends to break the software in many places, even in areas not conceptually related to the change.
    + **Immobility**: We're unable to reuse modules from other projects or within the same project because those modules have lots of dependencies.
    + **Viscosity**: When code changes are needed, developers will prefer the easier route even if they break existing design.Antipatterns and improper understanding of design principles can lead to *STUPID* code: Singleton, Tight Coupling, Untestability, Premature Optimization, Indescriptive Naming, and Duplication. SOLID can help developers stay clear of these.

The essence of SOLID is managing dependencies. This is done via interfaces and abstractions. Modules and classes should not be tightly coupled.

### As a beginner, what are some things to keep in mind when applying SOLID?

SOLID is really a guideline and not a rule. They work in many cases but they're not guaranteed to always work.

Just knowing the principles will not turn a bad programmer into a good one. This means that programmers need to understand why these principles make sense. They need to apply them with judgement. If they see code that violates these principles, they must try to see if the violations can be justified. If not, they can apply one or more principles to improve the code. Practice is the key.

### Could you explain the Single Responsibility Principle?

A class or a module must have a specific responsibility and nothing more. Put it another way, it should change for only one reason. We can say that the responsibility is encapsulated within the class and there's stronger cohesion within the class.

For example, an Automobile class can start or stop itself but the task of washing it belongs to the CarWash class. In another example, a Book class has properties to store its own name, author and text. But the task of printing the book must belong to the BookPrinter class. The BookPrinter class might print to console or another medium but such dependencies are removed from the Book class.

When SRP is followed, testing is easier. With a single responsibility, the class will have fewer test cases. Less functionality also means less dependencies to other modules or classes. It leads to better code organization since smaller and well-purposed classes are easier to search.

### Could you explain the Open-Closed Principle?

It should be possible to extend a module with additional behaviour without modifying it. This means that functions or base class methods should not get polluted with details of subclasses. A function that checks object types is a violation of OCP. This is because when a new object type is added, this function has to be modified to handle the new type. **Abstraction** is the key to getting OCP right. Two possible ways of doing this is polymorphism or templates/generics.

OCP is important since classes may come to us via third-party libraries. We should be able to extend those classes without worrying if those base classes can support our extensions. But inheritance can lead to subclasses depending on base class implementation. To avoid this, use of **interfaces** is recommended. This additional abstraction leads to loose coupling.

For example, a Car class might have methods AccelerateAudi, AccelerateBMW, and so on. This is a violation of OCP. Instead, we should have an ICar interface with method Accelerate. Each car subclass can implement this interface.

### Could you explain the Liskov Substitution Principle?

This principle states that we can substitute a subclass for its base class without affecting behaviour. This avoids misusing inheritance. It helps us conform to the "is-a" relationship. We can also say that subclasses must fulfil a contract defined by the base class. In this sense, it's related to **Design by Contract** that was first described by Bertrand Meyer. For example, it's tempting to say that a circle is a type of ellipse but circles don't have two foci or major/minor axes.

In another example, TRubberDucky is not exactly a duck because it can swim or quack but can't fly. So if a wildlife simulator instantiates a rubber duck and tries to make it fly, it will encounter an error. Thus, TRubberDucky as a subclass of TDuck is a violation of LSP.

LSP is also related to Duck Typing. In fact, duck typing doesn't even require a type hierarchy via inheritance.

### Could you explain the Interface Segregation Principle?

Segregate the interfaces so that subclasses are not required to use all of them. In other words, many client-specific interfaces are better than one general-purpose interface.

For example, a single logging interface for writing and reading logs is useful for a database but not for a console. Reading logs make no sense for a console logger.

Another example is an IMatrixOperations interface that has two methods for inverse and transpose operations. Problem is that for a matrix that's not regular, inverse operation is invalid. Such a matrix based on this interface would throw an exception for an inverse operation. Applying ISP, we would break this interface into separate ones, IRegularMatrixOperations (with inverse operation) and IMatrixOperations (that adds transpose operation). IMatrixOperations derives from IRegularMatrixOperations.

### Could you explain the Dependency Inversion Principle (DIP)?

DIP says that modules should depend upon interfaces or abstract classes, not concrete classes. It's an inversion because implementations depend upon abstractions and not the other way round. Instead of high-level modules depending on low-level modules, let's decouple them and make use of abstractions.

Let's say we instantiate a Windows98Machine object that's automatically created with a StandardKeyboard. Problem is that now we've coupled this keyboard with the machine. It's difficult to instantiate another machine with a different keyboard. This tight coupling also makes it difficult to test the Windows98Machine class. Instead, keyboard must be an interface. Any keyboard variant that uses this interface can be passed into the machine when the latter is constructed.

### What are some common criticisms of SOLID?

While some criticisms are valid, often they arise due to misunderstanding of SOLID. People have called the principles vague. They lead to complex and unintelligible code since we end up with many interfaces and many small classes. DIP depends on dependency injection frameworks. LSP requires implementation inheritance. Most of these are not really true. SOLID principles are condensed software wisdom. Just knowing them is not a substitute for experience and understanding.

SOLID focuses too much on dependencies. It encourages use of abstractions resulting in codebase littered with interfaces. To manage this problem, we end up introducing IoC containers and mocking frameworks, making the code more difficult to understand. SOLID produces testable code with low coupling but code is unintelligible.

It's been said that OCP doesn't work in practice, particularly for domain entities or business classes that by nature change frequently. Following OCP would lead to long inheritance chains and also a violation of another design principle, "favour composition over inheritance". People have different interpretations about what should be open and what should be closed. This leads to inconsistencies.

## Milestones

Oct
1987

Barbara Liskov of MIT presents at a conference a paper titled *Data Abstraction and Hierarchy*. She uses the term "substitution property" and explains,

> Data abstractions provide the same benefits as procedures, but for data. Recall that the main idea is to separate what an abstraction is from how it is implemented so that implementations of the same abstraction can be substituted freely.

1988

Bertrand Meyer, the creator Eiffel programming language, publishes a book titled *Object-Oriented Software Construction*. Meyer is credited with introducing the Open-Closed Principle.

1990

With growing codebases and complexity, object-oriented software design becomes popular to better manage software. But this change of programming paradigm from procedural to object-oriented does not automatically lead to clean code. Developers write large classes and methods. Code is duplicated. Old habits continue. There arises a need to guide developers design object-oriented software the right way.

1994

Barbara Liskov and Jeannette Wing present the case that anyone making use of supertype objects should not see any difference in behaviour even if using a subtype object instead. This later becomes a principle of SOLID. This principle is also related to what Bertrand Meyer calls **Design by Contract**.

Mar
1995

On com.object group, Robert Martin mentions a number of commandments for OOP. Three of these would later become part of SOLID: Open-Closed, Liskov Substitution, Dependency Inversion.

2000

Robert Martin writes in detail about four of the SOLID principles. The Single Responsibility Principle doesn't appear on this list.

2002

Robert Martin publishes the book *Agile Software Development: Principles, Patterns, and Practices*. He explains in detail all the five SOLID principles under a section named "Agile Design". Thus, SOLID becomes an essential aspect of Agile methodology. Sometime later Michael Feathers coins the term SOLID as a useful way to remember the principles.

2017

InterVenture publish extensions of SOLID. They identify six more principles. These are by no means universal in the industry but are worth studying.

Sep
2020

Paulo Merson notes in a blog article that SOLID was designed OOP. It may not exactly fit microservices. For microservices, he therefore proposes **IDEALS**: interface segregation, deployability (is on you), event-driven, availability over consistency, loose-coupling, and single responsibility.

SOLID Design Principles


# Linux

## Summary

Linux is a kernel used with GNU operating system. It is a Unix-like, open source and community-developed system for computers, servers, mobile devices and embedded devices. It communicates between hardware and software devices of the system and enables all the applications.

In use cases and applications Linux is similar to other operating systems such as Windows or MacOS. It has word processors, photo editors, video editors and many other applications. It is open source which allows users to view, edit or even contribute to the Linux.

Individuals and companies use it mostly for operating servers as it is more secure and flexible but it is also used in making Android phones, tablets, Chromebooks, digital storage devices, personal video recorders, cameras and many more devices.

## Discussion

### What are components of a Linux system?

Linux operating system comprises different components:

    + **Bootloader**: It is code that manages the boot process of computer. GRUB is the most common boot loader in Linux.
    + **Kernel**: It is the main component of Linux operating system. It manages the CPU, memory, and other peripheral devices. It communicates between a computer's hardware and its processes.
    + **Init System**: It is the first process that run once the kernel is loaded. It acts as the parent process to all other processes running on the system.
    + **Daemons**: These are programs that run in background and handle requests for a service such as printing, sound or scheduling. Httpd daemon is used to listen for web server requests on Linux web servers.
    + **Graphical Server**: It controls how graphics are displayed on the computer screen. Servers and even low-end embedded devices don't need graphical servers or desktop environments.
    + **Desktop Environment**: It is a collection of software and UI controls that users interact with. Each desktop environment comes with built-in applications such as file managers, configuration tools, web browsers, and games.

### What are some basic Linux terms?

Below are some basic terms used in Linux:

    + **Command**: These are printable characters that user type into a program called the ‘command line' to give instructions to the operating system.
    + **Distro**: Linux distribution or distro is a specific version of Linux OS packaged with other components including installation programs, management tools and other software.
    + **Root**: Linux systems have predefined set of rules for users. Different users have different levels of permissions. For instance, a guest user won't be able to modify the OS's main files while a root user (aka *administrator*) has full access to every command and file in the system.
    + **Terminal**: It invokes applications whose execution is scheduled by OS transparent to the user. The UI of terminal is very simple and easy to use.
    + **Shell**: It is Linux command line interpreter. It executes the command given by user.
    + **Device**: There are various special files under the directory `/dev`. These files are called device files. The most common types of device files are block devices and character devices. Character devices communicate by sending and receiving single characters. Block devices communicate by sending blocks of data.

### What are some basic services/utilities provided in Linux?

Below are some basic services/utilities provided in Linux:

    + **User management**: User management activities such as adding user, deleting user, updating or giving certain permissions, creating and removing groups require administrator privileges. Admin permissions can be gained either by changing to the root user with the `su` command or using `sudo`.
    + **File system**: Linux file system has a hierarchal file structure as it contains a root directory and its subdirectories.
    + **Scheduler**: The scheduler also called as process scheduler is the component of the kernel that schedules the processes to run in a systematic manner.
    + **Process management**: Each process starts with a unique five-digit number called process id or PID. We can track the process using this PID.
    + **Package management**: It helps to install, update, remove, and keep track of software updates in the Linux system.
    + **Memory management**: Linux memory management subsystem manages the implementation of virtual memory and demand paging, memory allocation both for kernel internal structures and user space programs, mapping of files into processes address space and many other things.

### How does Linux differ from other operating systems?

The Linux is totally free of cost whereas Windows is a costly OS. Unlike in Windows, users can access the source code of Linux as it is open source. Linux is more secure compared to Windows and MacOs. File names are case sensitive in Linux which is not the case in Windows. Linux uses the monolithic kernel while Windows utilizes the micro-kernel. Linux is more efficient than Windows in performing operations while Windows has a better user friendly UI and navigation. Hackers use Linux based systems as Windows is not a very efficient OS for hacking purposes.

### Which industries use Linux?

Linux is the core of many industries while Windows OS dominates in market share. Linux is the most favourite operating system of Microsoft Azure's customers. It is used to develop several coding languages and build embedded systems. Microsoft's operating system named Azure Sphere is also built on Linux kernel.

There are broadly two types of categories that use Linux:

    + **Consumer devices**: Google's video streaming platform Chromecast also runs on Linux. Panasonic, Samsung, Phillips and many more use Linux-based operating systems for their devices. Lightweight laptops such as Chromebook use Chrome OS, which is built on Linux. Instagram, Facebook, YouTube and Twitter all run on Linux.
    + **Non-consumer devices**: In US, The Library of Congress, House of Representatives, Senate, and White House all use Linux. The New York Stock Exchange uses Linux. The Indian state Kerala has made Linux mandatory for all the high schools computers. NASA's Pleiades supercomputer runs on Linux. The International Space Station switched from Windows to Linux in 2013 due to its reliability. Linux is the leading operating system used for servers.

### How can I get started using Linux?

The prerequisites for learning Linux includes basic computer skills and a basic computer system with single-core CPU, 128 MB of memory and few GBs of hard disk.

First step is to decide the Linux distro version to use. There are many beginner friendly and free-to-use Linux distros such as Ubuntu, Debian GNU, openSUSE and CentOS. Each of them has its own advantages and features. After selection, one needs to install the Linux distro on the system.

In the learning phase users have a lot of problems and challenges which they can tackle using communities such as StackOverflow and GitHub as they are helping open-source communities. Below are some courses and cheatsheets to learn Linux command line:

    + Learn The Linux Command Line: Basic Commands
    + Linux Tutorials and Projects
    + Learn Linux on a MAC
    + Linux Commands Cheat Sheet by Linux training academy
    + Linux Command Line Cheat Sheet by DaveChild
    + Linux Command Line by Das
    + Books: *The Linux Command Line: A Complete Introduction* by William Shotts and *How Linux Works* by Brian Ward are two books for beginners.

### What are the pros and cons of using Linux?

As it has a small user base compared to other OS, hackers avoid targeting it. Every Linux distribution is tested many times against bugs before releasing a stable version which makes it a stable option. Linux is released under the GNU GPL license which makes it free to use. Unlike Windows and MacOS, users are free to customize Linux distributions according to their needs. Linux systems can run on even 500 MB ram or at 256 MB RAM, as it uses resources very efficiently. Users can install software using a few commands on terminal.

However, Linux has some disadvantages. It is less user-friendly compared to Windows and MacOS. Troubleshooting in Linux is difficult for beginners. It is not a preferred option for games. There are no drivers in Linux which can be a significant issue at times.

## Milestones

Sep
1991

Linus Torvalds releases **Linux kernel**.

1992

Orest Zborowski ports the X Window System to Linux. As a result Linux starts gaining more importance and user base. It allows Linux to support **Graphical User Interface (GUI)**. Linux Kernel is released under GPL licence.

1993

**Slackware** Linux version 1.0 is released. Even 25 later, it remains the oldest Linux distribution that's actively maintained.

1994

**Linux 1.0.0** is released with 176,250 lines of code. In June 1996, Torvalds releases Linux version 2.0.0. that contains 41 releases. Support for symmetric multiprocessing (SMP) and support for more types of processors are added in this version. On 04 January 2001, Linux version 2.4.0 is released. It consists support for ISA Plug and Play, USB, and PC Cards. On 17 December 2003, Linux version 2.6.0 is released. Support for files of other OS such as JFS, XFS, Solaris, Windows and MS-DOS is added. Torvalds releases Linux version 3.0 on 22 July 2011.

1996

A project for desktop Linux called as **KDE** is announced. A year later, an alternative desktop environment called **GNOME** is announced. It's supposed to provide free and complete set of user friendly applications and desktop tools. In 1999, GNOME 1.0 is released at Linux Expo.

2003

An attempt is made to insert a backdoor in the Linux kernel source. The backdoor is designed to obtain root privileges under specific conditions. Linux Kernel maintainers acknowledge it before it could access Linux kernel and do any harm.

2005

The free version control system used for the Linux Kernel development called BitKeeper becomes paid. Linus Torvalds decides to work on his own version control system and creates **Git**.

2008

A mobile operating system based on Linux Kernel called as **Android** is released. Android would go on to become the leading OS in the world of mobile operating systems.

2011

Microsoft which was at war with Linux now partners with LinuxTag for their event. In 2012, Microsoft sponsors LinuxTag event. It also starts hosting Linux virtual machines in the Azure cloud computing service.

Linux


# Progressive Web Applications

## Summary

A Progressive Web App (PWA) is a web app that uses the latest advances in web technologies and browser support to deliver users an app-like experience. Like any traditional mobile app, a PWA can be launched from an icon installed on the mobile home screen, data can be cached for offline use and can receive push notifications even when the browser is closed. PWAs don't have a navigation bar or URL bar usually seen in web browsers. Thus, they can deliver the same fullscreen experience as mobile apps.

Compared to mobile web, PWAs have better performance and thereby improve user retention and engagement. At the same time, unlike native apps, they don't need long downloads and installs. They're also easier to deploy and maintain since developers don't have to worry about different versions of the app out there.

## Discussion

### Why do we need PWA?

The next billion users to come online will be from the developing world and they are likely to do so from their smartphones rather than desktops. They are likely to consume rich multimedia and do voice-based searches. They're also likely to use low-end devices with limited memory, storage and processing.

53% of users leave a page that takes longer than 3 seconds. An average mobile web page takes 19 seconds to load on a 3G connection. Among Indian mobile users, 33% run out of storage every day. 79% of shoppers won't return to slow sites. 20% users are lost at every step of native app installation process.

Hybrid apps tried to bring app-like experience by wrapping web apps into installable mobile apps but this approach may give way to PWA. Many others tried to deliver app-like experiences but sacrificed web linkability in the process. What we need is a way to improve user experience even on low-end devices with limited memory and slow or no connections. PWA aims to deliver this.

### What's so "progressive" about PWA?

PWA are web apps that "progressively change with use and user consent to give the user a more native-app-like experience." Unlike native apps, PWA don't ask users upfront permissions to install the app or access private data. PWA involves minimum friction for users. If the user is happy with the website, she can choose to add it to the homescreen. The site earns the right to become an app. It becomes progressively an app.

If new features are introduced into web standards (example, Payment Request API, Credential Management API or WebVR), and when browsers support them, and when apps start using them, the entire experience becomes richer and more progressive for users. The experience is seamless. It doesn't involve new installs. Users can continue using older versions of browsers until they are ready. We call this *progressive enhancement*. This means that even on browsers that don't support service workers, progressive enhancement allows those browsers to display content. It's been said that progressive enhancement is the opposite of graceful degradation.

Let's also note that publishers can deploy a PWA without the approval process that native apps go through to get listed on an app store.

### What are the key enablers of PWA?

We can identify three key enablers of PWA:

    + **Service Workers**: These cache data, serve data from cache in offline mode or when downloading fresh content in the background, and enable push notifications. While service workers do caching for offline mode, they also bring reliability in the face of varying network conditions. We can view service workers as proxies implemented in JavaScript. These are proxies listening for network activity, intercepting them and handling them programmatically.
    + **Application Shell Architecture**: The app's layout is already with the service worker and hence this can be shown to user while content is being downloaded in the background. Thus, the "app shell" is almost instantly shown to the user without any waiting time. We can think of this shell as minimal HTML, CSS and JS without the dynamic content. App shell is not a Web API but really a design approach that works nicely with the capabilities of service workers.
    + **Application Manifest**: This tells the browser that the mobile website can be installed as an app.

### What are the characteristics of a good PWA?

In general, Google states that a delightful web experience needs to be fast, integrated, reliable and engaging. To quality as PWA, an app should have the following characteristics:

    + **Progressive**: Works across browsers and enabled with progressive enhancement.
    + **Responsive**: Works on a variety of devices of different form factors.
    + **Connectivity-independent**: Service workers make this possible by serving from cache.
    + **App-like**: Browser tabs and navigation are replaced with app shell architecture and app style navigation.
    + **Fresh**: Service workers keep dynamic content up to date.
    + **Safe**: All communication is over HTTPS.
    + **Discoverable**: Manifest file allows them to be identified as PWA. Search engines can find them.
    + **Re-engageable**: Push notifications keep user engagement and retention high.
    + **Installable**: Add to homescreen without requiring download and install via an app store.
    + **Linkable**: Deep link to specific pages within the app. Power of web linking is not sacrificed.

### Is PWA a new web standard?

No. Making a PWA is simply using existing web standards and adopting certain best practices to improve user experience. PWAs combine the best of the web and native apps.

Having said that, browsers and platforms can recognize an app as PWA and give it special handling. For example, they may make it easier to add a PWA to homescreen, list it among installed native apps or even add it to their app stores.

### What tools and techniques are available for building a PWA?

Since PWA uses an app shell, it's important to keep the shell minimal. The app infrastructure and UI should be separated from the data. The former can be cached and only data needs to be fetched when changes occur.

The use of PRPL pattern is common when building a PWA. Polymer templates can be used for this. HTTP/2 server push and/or the use of `<link rel="preload">` can improve performance. Code splitting and lazy loading have been suggested. Feature detection and CSS fallbacks can help in delivering progressive enhancements. Consider using Knockout for small projects.

The Application tab of Chrome Developer Tools is useful during development. It also comes with Lighthouse, which can give a PWA conformance score for your app. HN PWA is a useful reference that has Lighthouse scores of different frameworks. The Service Worker Cookbook, PWABuilder, Workbox, sw-precache and sw-toolbox are useful resources. In Chrome or Opera, `chrome://serviceworker-internals` can be useful. Osmani and Gaunt mention other useful tips and tricks.

### Can I use PWA along with Accelerated Mobile Pages (AMP)?

PWA loads an app shell first while service workers download and cache the dynamic content in the background. Cache helps when users revisit the page but PWA does not cut down on load time for the first access. This is the problem that Accelerated Mobile Pages (AMP) aims to solve.

AMP gives the experience of "instant" loading of content at first access. They achieve this by statically laying out content and avoiding JS execution. In addition, AMP caches content for subsequent requests. AMP is suitable for static content but when dynamic functionality is required (push notifications, web payments, etc.) PWA is more suitable.

In fact, AMP can work nicely with PWA. Different models have been proposed: **AMP as PWA**, **AMP to PWA**, **AMP in PWA**. Thus, AMP and PWA should be seen as complementary technologies. As an example, AMP HTML can be the canonical data format that's downloaded and cached by PWA service workers. Alex Russell says of AMP and PWA,

> AMP gets content in front of users fast. PWA deliver reliable performance for re-visits.

### How does PWA relate to Single Page Applications (SPAs)?

SPAs are heavy on JavaScript and regardless of the framework of choice they use the App Shell Architecture. Shell is served first to the user while JavaScript downloads and displays dynamic content to complete the view. SPAs can now benefit by adopting PWA to precache content, sync to dynamic content in the background and reuse content from cache. This means that SPAs are fundamentally unchanged but they are powered by PWA best practices to give better user experience.

In fact, SPAs usually load only data and render them on the client side. With PWA, server-side rendering is preferred. Client-side rendering should be kept minimal to improve user experience.

Some claim that you need an SPA to build a PWA. In other words, a site that already has an app-like interface will be easy to transition to PWA. However, Condé Nast makes the point that it's perfectly possible to build a PWA without it being an SPA.

### How is browser and platform support for PWA?

Since PWA is based on web standards, developers need to consider support for those standards. This would include parsing a manifest file, adding to homescreen, supporting service workers and so on. As of November 2017, Google Chrome and Mozilla Firefox support it but Safari doesn't. Jake Archibald maintains current support for service workers in browsers.

There's no consensus on menu items such as "Add to Homescreen", which may be phrased differently in each browser.

Where cross-browser support is lacking, a native mobile app may be better than a PWA. It's also been suggested that having an app as PWA plus native or hybrid variants may give users more choice.

React, Preact, Angular and Vue.js are some examples of platforms that have support for service workers and app manifest file generation.

### Are there good examples of PWA?

A curated list of ten PWA apps includes Ali Express, Flipkart Lite, Wego, Trivago, Flipboard and Telegram. Flipkart Lite is an e-commerce app from India. Air Berlin that allows for online check-in and displays cached boarding cards in offline mode. Twitter, Lyft, Uber and Starbucks are some big names that have taken the PWA route. Other PWA examples include Wired, The Washington Post, The Guardian, The Weather Company, Forbes and Financial Times. AirHorner showcases PWA capabilities well.

### Has PWA proven itself in the real world against web apps and native mobile apps?

Here are some success stories:

    + Twitter Lite, built as PWA with React and Node.js, has achieved 75% increase in tweets and reduced data usage by as much as 70%.
    + Google PWA install banners have 5-6x more conversions than native install banners.
    + The Weather Channel achieved 80% faster loading with PWA.
    + Lancôme's PWA version of its e-commerce site gave 84% decrease in time to interaction and 17% increase in conversion.
    + India's MakeMyTrip saw conversion rate triple and a 160% increase in shopper sessions.
    + Alibaba's PWA has brought 76% rise in conversions and four times higher interaction rate.
    + Forbes has seen a 100% increase in session duration and more 43% more sessions per user.

### Is PWA only for the mobile world?

No. PWA can be used to enhance desktop experience as well. Samsung DeX is an example of this. While Chrome for Android supports PWA, desktop version of Chrome will also start supporting PWA in 2018. If we consider VR experiences within a browser using WebVR, PWA can enable these on mobile as well as desktop.

## Milestones

2011

It becomes possible to bookmark web apps to the homescreen.

Oct
2014

Web app manifest is now a JSON file allowing us to specify app name, icon and how the app should be launched. The implies that we no longer need to give configuration in the `head` part of the HTML page.

Dec
2014

Chrome starts supporting service workers.

Jun
2015

Google engineer Alex Russell explains PWA.

May
2016

Google announces PWA.

Oct
2017

Mozilla announces that will embrace PWA, starting first with Android. Firefox has supported service workers and push notifications since 2016.

Progressive Web Applications


# Structured vs Unstructured Data

## Summary

Data is available in many forms, shapes and formats. Broadly, data can be either structured or unstructured. Data that's properly organized, with well-defined constraints and relationships among its different parts, can be considered as **structured**.

There's no precise definition of structured data. In the hazy boundary between structured and unstructured data, some have identified semi-structured data. Others have argued that all data has some structure: it's just that some are more difficult to store or analyze.

The general viewpoint is that all data can add value to data mining and analytics. Technology that evolved for structured data have been adapted, and new ones invented, to handle unstructured data as well. Unstructured data has become increasingly important due to its volume, velocity, variety and value.

## Discussion

### What do you mean by "structure" with respect to data?

Data that's highly organized such as in a database can be considered as structured data. These often use a Relational Database Management System (RDBMS). Such data has schema that defines attributes and their types, constraints on values, and relations with other data tables and attributes. Data must conform to this schema. This makes data easy to query using Structured Query Language (SQL) and thereby facilitates analysis. Data in spreadsheets may be considered structured.

Unstructured data is not organized in this manner, that is, in RDBMS. Consider an audio stream, which has a well-defined format. Otherwise, it couldn't be decoded and played. Consider newspaper text. A linguist would say there's structure. But from the perspective of business analysts, this data is unstructured simply because it's harder to analyze and obtain insights.

Some say the term unstructured data is a misnomer. If it's truly lacking structure, it's useless to store it or analyze it. It would be better to categorize data as fixed or variable structure; repetitive or hierarchical; textual or non-textual.

### Could you give examples of unstructured and semi-structured data?

Rich media such as image, video or audio are unstructured. Social media generates lots of unstructured content. Websites host unstructured content that are commonly textual in nature. Information in documents such as MS Word files or PDF files are also seen as unstructured. Machine-generated content such as satellite images, IoT sensor data, or CCTV video feed are considered unstructured.

Many of the same data sources mentioned above could have attributes that make them **semi-structured**. For instance, Tweets, Facebook posts, blog articles, and news stories published online often have number of likes, retweets/shares, and comments, including names of readers who did these. Email text may be unstructured but the header contains names of sender/receiver, date and subject that give some structure.

IoT data may be seen as semi-structured when it's in JSON or XML formats. Data about data, called **metadata**, such as author name and publication date make data semi-structured. In fact, rich semantic markup on webpages gives them lot more structure that what HTML alone does. Most unstructured data can be considered as semi-structured because of metadata.

### What are some use cases of analysis on unstructured data?

In a manufacturing plant, operation logs and reports are unstructured. Text analysis to pick out frequent words or sentiments can help the plant manager make a maintenance plan or quickly understand the nature of a particular line.

Koorong Books in Australia uses text analysis to identify duplicate postings or suggest similar books. This is an example of content-based profiling.

In banking, customer may often give feedback or complaint on social media rather than via web forms. If banks wish to be customer centric, they need to act on this unstructured data. In fact, this could apply to any industry that needs to listen to its customers. Companies can use chatbots with NLP capability to automate customer support functions.

Deep learning techniques are being used to analyze images and sounds. Images can be automatically labelled. Mammograms can be analyzed for cancer. The sound of a motor can inform in advance if it's going to fail. This is of importance in automobile and aviation sectors.

### What are some myths about unstructured data?

An early myth was that unstructured data can't be quantitatively analyzed. Perhaps true in the past, but with recent advances in computer vision, speech processing, and natural language processing, algorithms are able solve many complex problems in these domains.

Some might believe that unstructured data replaces structured data. In reality, many companies have not fully exploited the potential of structured data. Good old predictive models and analytical capabilities should continue to be used. Unstructured data will give access to new insights not otherwise available in structured data. In conclusion, both structured and unstructured data are valuable.

Another myth says that all big data is unstructured data. Telecom companies, smart energy meters, smartphones, and cars fitted with sensors are all generating big data that's structured.

Some businesses just store the data with a vague idea of using them later. In fact, data and insights depreciate over time. Real-time dashboards are probably the best opportunity to act on data in a timely manner. When collecting or acting on data, tie it to a business vision.

### How should I store unstructured data?

Big Data technologies such Hadoop and NoSQL databases have come about to address the needs of storing and managing unstructured data. Data warehouses and data lakes are places where big data is stored. Unstructured data is often used alongside structured data. Many technologies cater for both.

Hadoop enables distributed storage and computing on big data. It's a good engine for handling unstructured data, though it's a myth to think that unstructured data can't be stored or analyzed without Hadoop.

NoSQL databases are highly scalable and support flexible schema. Their storage is distributed. They're non-relational. They're good at storing multimedia, social media or textual data. Among the different types are document stores, column stores, key-value stores and graph data stores. A mix of these is often used, each suited to a particular data. This approach is called **polyglot persistence**.

As cost of flash memory drops, flash becomes a faster alternative to disk storage. However, file services such as data protection, backup, and search need to be in place.

For a minimalistic storage system, try the open source MinIO. Alternatives include Ceph, Scality and Cleversafe.

### What are some techniques to analyze unstructured data?

The general perception is that unstructured data is hard to analyze. This is changing due to advances in machine learning models.

The simplest way to get started is perhaps to call APIs that others have published. For example, cognitive APIs from IBM such as Watson Tradeoff Analytics can help in decision making. Geneea is an NLP API. For speech recognition, we can use AT&T Speech API. Google's Cloud Vision API is useful for many image-specific tasks.

## Milestones

1960

In the 1960s, businesses start using computers. Storing and managing data becomes important to them. Because memory is expensive, there's a need to store data efficiently. For these reasons, databases are invented. IBM's IMS is an example. These early databases store only structured data.

1970

The 1970s sees the arrival of **relational databases**. In 1974, IBM invents **SQL** as a language to query such databases.

Sep
1991

Andy Rehn, VP of marketing for Data Base Architects, states that "as much as 90 percent of the information business uses is non-numerical, freeform data." This is one of the earliest published reports that quantifies the prevalence of unstructured data.

1995

The early and mid-1990s is when text mining starts entering real-world applications. Document-management systems emerge, later rebranded as Enterprise Content Management (ECM) systems. The World Wide Web also starts generating lots of unstructured data. Business Intelligence (BI) that had grown up on structured data starts mining text for useful insights.

1998

A rule of thumb is that 80% of all data is unstructured or semi-structured at best. This 80% figure is mentioned in a Merrill-Lynch report but it's not due to primary research.

2007

From a survey involving data warehousing and business intelligence search, TDWI Research finds that structured/unstructured ratio is not 20/80 as often claimed. Structured data is at 47%, unstructured data at 31% and the rest being semi-structured. However, the report recognizes that unstructured data is on the rise.

Mar
2009

OASIS approves *Unstructured Information Management Architecture (UIMA)*, version 1.0. Apache UIMA is an open source implementation of this standard. V1.0.0 of this software is released in January 2014. In August 2019, V3.1.0 is released.

Mar
2019

Analysis of unstructured data is still new in some domains. A case in point is the healthcare industry where doctors' handwritten notes, images, histories, formulae and genetics have not been fully analyzed.

Structured vs Unstructured Data


# ROC Curve

## Summary

In many applications, there's a need to decide between two alternatives. In the military, radar operators look at approaching objects and decide if it's a threat. Doctors look at an image and decide if it's a tumour. For facial recognition, an algorithm has to decide if it's a match. In Machine Learning, we call this **binary classification** while in radar we call it **signal detection**.

The decision depends on a threshold. **Receiver Operating Characteristic (ROC) Curve** is a graphical plot that helps us see the performance of a binary classifier or diagnostic test when the threshold is varied. Using the ROC Curve, we can select a threshold that best suits our application. The idea is to maximize correct classification or detection while minimizing false positives. ROC Curve is also useful when comparing alternative classifiers or diagnostic tests.

## Discussion

### How do we define or plot the ROC Curve?

Let's take a binary classification problem that has two distributions: one for positives and one for negatives. To classify subjects into one of these two classes, we select a threshold. Anything above the threshold is classified as positive. The accuracy of the classifier depends directly on the threshold we use. ROC Curve is plotted by varying the thresholds and recording the classifier's performance at each threshold.

ROC curve plots **True Positive Rate (TPR)** versus **False Positive Rate (FPR)**. TPR is also called recall or sensitivity. TPR is the probability that we detect a signal when it's present. FPR is the complement of specificity: (1-specificity). FPR is the probability that we detect a signal when it's not present. Being based on only recall and specificity, ROC curve is independent of prevalence, that is, how common is the condition in the population.

An ideal classifier will have an ROC curve that rises sharply from origin until FPR rises when TPR is already high. Each point on the ROC curve represents the performance of the classifier at one threshold value.

### Which application domains are using ROC Curves?

ROC started in radar applications. It was later applied in many other domains including psychology, medicine, radiology, biometrics, and meteorology. More recently, it's being used in machine learning and data mining.

In medical practice, it's used for assessing diagnostic biomarkers, imaging tests or even risk assessment. It's been used to analyse information processing in the brain during sensory difference testing.

In bioinformatics and computational genomics, ROC analysis is being applied. In particular, it's used to classify biological sequences and protein structures.

ROC has been used to describe the performance of instruments built to detect explosives. In engineering, it's been used to evaluate the accuracy of pipeline reliability analysis and predict the failure threshold value.

### What is AUC and its significance?

After plotting the ROC Curve, the area under it is called **Area Under the ROC Curve (AUC)**, **Area Under the Curve (AUC)**, or **AUROC**. It's been said that "ROC is a probability curve and AUC represents degree or measure of separability". In other words, AUC is a single metric that can be used to quantify how well two classes are separated by a binary classifier. It's also useful when comparing different classifiers.

AUC has some useful properties. It's **scale-invariant**. This means it tells how well predictions are ranked rather than their absolute values. AUC is also **classification-threshold-invariant**. We can objectively compare prediction models irrespective of classification thresholds used. However, these properties are not desirable for some applications.

AUC is also **prevalence-invariant**. Suppose a health condition is prevalent in only 1% of the population. A simple classifier can achieve 99% accuracy by predicting negative always. AUC however gives a more useful value of 0.5.

### How do I interpret an AUC value?

Since both axes of the ROC Curve range [0,1], AUC also ranges [0,1]. Some researchers map AUC to **Gini Coefficient**, which is 2\*AUC-1, with range [-1,-1].

More realistically, AUC has a range [0.5,1] since the ROC curve is expected to be above the diagonal. Value 0.5 implies very poor separation and is represented by the diagonal ROC curve. Value 1 implies perfect separation, where TPR is always 1 at all values of FPR. As a thumb rule, we have an excellent classifier if AUC is >=0.9 and a good classifier when it's >= 0.8.

### Why do I need an ROC Curve when TPR and FPR may be adequate?

ROC Curve is a useful tool to compare classification methods and decide which one is better. Suppose a computer algorithm is implemented to diagnose a medical condition. Using ROC curves, we can compare its performance against a doctor's diagnosis, and against doctor's diagnosis when aided with computer-assisted detection (CAD). As shown in figure, a doctor using CAD gives best performance. The other two approaches have the same AUC but the doctor has a higher specificity (lower FPR).

In any binary classification problem, it's not possible to agree on a single threshold and consequently on values of sensitivity and specificity. Take the case of diagnostic testing as an example. Threshold would be adjusted based on the context and available information, such as patient history, presence of symptoms, or even likelihood of getting sued for a missed cancer. If we just plot two points for two classifiers, it's hard to know which one is better. Once we plot entire ROC curves, it's easy to see which one is better.

### For a binary classification problem, how to I select the optimum threshold on the ROC Curve?

There are basically two methods of determining the optimum threshold:

    + **Minimum-d**: This is the shortest distance of the curve from the top-left corner or (0,1) point.
    + **Youden index**: This is the vertical distance from the curve to the diagonal. To find the optimum point on the curve, we should maximize the Youden index.ROC Curve and AUC ignore prevalence or misclassification costs. For example, poor sensitivity means missed cancer and delayed treatment whereas poor specificity means unnecessary treatment. Likewise, a false positive on a blood test for HIV simply means a discarded blood sample but a false negative will infect the blood recipient. It's for this reason decision makers should consider financial costs, and combine ROC analysis with utility-based decision theory to find the optimum threshold.

### How do I apply ROC Curves to multiclass problems?

Given \(c\) classes, the ROC space has \(c(c-1)\) dimensions. This makes it difficult to apply ROC Curve methodology to multiclass problems. However, some attempt has been made to apply it to 3 classes where AUC concept is extended to **Volume Under the ROC Surface (VUS)**.

One approach is to reframe the problem into \(c\) one-vs-all binary classifiers. However, ROC Curve may not be suitable since FPR will be underestimated due to large number of negative data points. For this reason, Precision vs. Recall curve is more suitable.

For computing the AUC, one technique is to average pairwise comparisons. This equivalent AUC value is useful since we can ignore the costs associated with different kinds of misclassification errors.

### What are some pitfalls or drawbacks of using ROC Curve and AUC?

In practice, AUC must be presented with a confidence interval, such as 95% CI, since it's estimated from a population sample. However, one research in clinical chemistry showed that many researchers failed to include CI or constructed them incorrectly.

AUC involves loss of information. Two ROC curves crossing each other can have the same AUC but each will have a range of thresholds at which it's better.  Clinicians and patients interpret sensitivity and specificity but don't find AUC useful. They're not interested in performance across all thresholds. In ML, **cost curves** have been proposed as an alternative. Another alternative is **H-measure**.

AUC ignores the misclassification costs. A new test may be deemed worthless by using AUC alone. AUC also ignores prevalence but it's known that prevalence affects test results. While sensitivity and specificity are also independent of prevalence, prevalence can be considered during interpretation of the ROC curve.

Jorge M. Lobo et al. give many other reasons why AUC is not a suitable measure.

### What software packages are available for ROC analysis?

In R language, we can use the *pROC* package. Once we obtain the actual and predicted values, we can obtain the AUC along with confidence interval using the function `ci.auc()`. On GitHub, `sachsmc/plotROC` is an open source package for easily plotting ROC curves. It uses *ggplot2*, to which it adds handy functions for plotting: `geom_roc`, `geom_rocci` and `style_roc`.

In Python, a webpage on Scikit-learn gives code examples showing how to plot ROC curves and compute AUC for both binary and multiclass problems. It makes use of functions `roc_curve` and `auc` that are part of *sklearn.metrics* package.

## Milestones

1940

The idea of ROC starts in the 1940s with the use of radar during World War II. The task is to identify enemy aircraft while avoiding false detection of benign objects. ROC provides a suitable threshold for radar receiver operators. This also explains the origin of the term **Receiver Operating Characteristic (ROC)**.

1950

In the 1950s, psychologists start using ROC when studying the relationship between psychological experience and physical stimuli.

1953

Peterson and Birdsall explain the ROC Curve in detail in the context of signal detection theory. They plot probability of signal detection versus probability of false alarm. Curve-1 represents optimum operation, curve-3 sets the lower limit, and curve-2 is by guessing. Curve-1 is produced by varying the operating level or threshold β, which is also the slope of the curve at that point. The value on the y-intercept is the one that needs to be maximized.

1960

L.B. Lusted applies ROC methodology to compare different studies of chest film interpretations for detection of pulmonary tuberculosis. This is the first application of ROC to radiology. It subsequently inspires the use of ROC in many diagnostic imaging systems. Lusted himself publishes *Decision-making studies in patient management* in 1971.

1968

Dorfman and Alf develop a method of curve fitting and use software to automate ROC analysis. A maximum likelihood approach under binomial assumption is developed. Many other programs written in FORTRAN are developed later: ROCFIT, CORROC, ROCPWR, and LABROC.

1969

The concept of **Free-Response Receiver Operating Characteristic (FROC) Curve** is introduced in auditory domain. In free-response analysis, in addition to detection, we also need to point out the location. The term "free-response" was coined in 1961. In 1978, FROC is applied for the first time in imaging. FROC can help where ROC can fail. For example, ROC can show location-level false positive and false negative that could "cancel" each other. This gives an image-level true positive: image shows cancer but wrong location is reported.

1982

Although **Area Under the ROC Curve (AUC)** was previously used, Hanley and McNeil develop analytical techniques to bring out its statistical properties. They estimate the standard error for different underlying distributions and sample sizes.

1989

As one of the earliest application of ROC Curve to **machine learning**, K.A. Spackman uses it evaluate and compare ML algorithms.

1997

Andrew Bradley notes that ROC curve is useful for visualizing a classifier's performance but not suitable for comparing multiple classification methods. A single performance measure is more desirable. He discusses how **AUC** can be used as a measure for comparing **machine learning** algorithms. He explains why AUC is a better measure than overall accuracy.

2000

An article titled *Better decisions through science* appears in Scientific American. It brings ROC Curve to the attention of a wider audience. One example in this article talks about glaucoma diagnosis using eye fluid pressure. It defines the basic terms and shows hypothetical distribution curves, ROC Curves, and AUC. It states that AUC is a reflection of a test's accuracy.

2001

Hand and Till generalize the concept of AUC for multiclass problems. In 2007, Landgrebe and Duin approximate the problem via pairwise analysis.

ROC Curve


# ACID Transactions

## Summary

Computer systems are rarely perfect and this applies to databases as well. To ensure that data is updated in a consistent manner, a sequence of operations are often grouped together into what's called a **transaction**. We may say that a transaction is something that changes the system state.

In case something should go wrong when executing these operations, the transaction as a whole can be aborted. This means that the database can be safely rolled back to the consistent state before the transaction. In database design and implementation, a transaction must fulfil four essential properties: Atomicity, Consistency, Isolation, Durability (ACID).

Monolithic or SQL databases adopted ACID in their transactions but distributed or NoSQL databases didn't. However, distributed databases are starting to support ACID without impacting performance.

## Discussion

### What's the meaning of the acronym ACID?

ACID can be explained as follows:

    + **Atomicity**: All operations of a transaction happen as if they're a single operation. All changes are performed or none at all. If one operation in a transaction fails, the entire transaction fails and completed operations are rolled back.
    + **Consistency**: Data starts with a consistent state and ends up in a consistent state. During the transaction, it may be inconsistent, but at the end, data is left in a consistent state.
    + **Isolation**: Intermediate state of a transaction is not visible to other concurrent transactions. Concurrent transactions are effectively serialized.
    + **Durability**: When a transaction completes, changes are stored in a persistent manner. Even if there's a power failure or other system errors, the effect of the completed transaction remains.

### Could you explain ACID with an example?

Consider a money transfer transaction. At the minimum, it would involve two operations: debiting from one account, then crediting to another account. What happens if money is debited from an account, then some failure happens (disk crash or power outage), and the receiver's account is never credited? This scenario leaves the database in an inconsistent state. To guard against this, we wrap both these operations into a single transaction that's expected to have ACID properties.

So now, even if there's a failure, atomicity ensures that the transaction will rollback the debit operation. In a normal successful scenario, both accounts will be updated correctly. It's not an issue if there's a failure after the transaction, since accounts have been updated in some persistent storage.

Suppose another transaction *T2* queries the database when the money transfer transaction *T1* is in progress. It will see original balance in the sender's account since *T1* hasn't completed, even when the debit operation has happened within *T1*. This is the concept of isolation.

### How do databases meet the ACID requirements?

A transaction moves through various states. In the end, it can either succeed (committed) or fail (aborted). When committed, all changes made by the transaction become persistent. When aborted, changes made by the transaction are rolled back so that the starting state of the database is returned. This is usually implemented by making changes in a temporary space and then committing the entire transaction as a final step.

Consistency is ensured by having a strongly consistent core involving a single operation on a single row. Integrity checks are built on top of this for multiple operations across multiple rows.

For isolation, read-write locking and multiversion concurrency control are two approaches. We also use globally ordered timestamps so that transactions can be organized serially. Partial ordering is possible when sub-transactions belonging to multiple transactions interleave. This lowers latency but doesn't give full isolation. Instead, a mechanism to detect write conflicts is used.

### What are the some problems when isolation is not proper?

**Dirty read** happens when full isolation is not present across transactions. A transaction updates a value but it's not yet committed. Another transaction reads the uncommitted value but the original transaction rolls back the changes due to an error.

What happens if consecutive reads give different values? This problem is called **non-repeatable read**. This can be solved by locking the record until the current write transaction has completed.

Another problem is **phantom read** that happens when a new record is inserted, and this record matches the selection criterion issued by another concurrent transaction. The reading transaction will end up using stale data. Range locking or predicate locking is the solution to this problem.

### Are there scenarios where we can relax ACID requirements?

Database operations are either write-intensive transactional (OLTP) or read-intensive analytical (OLAP). ACID is essential for OLTP but can be relaxed for OLAP.

Likewise, ACID has been traditionally relaxed in NoSQL or distributed databases where the focus has been on scale and availability. Strong consistency is sacrificed for eventual consistency. This is because in a distributed database it's inefficient to globally to acquire and release locks. However, modern databases of the late 2010s are bringing back ACID even in a globally distributed environment. Examples include Google Cloud Spanner, CockroachDB, TiDB, YugaByte DB, FoundationDB and FaunaDB.

The point is that where ACID is not guaranteed by the database, developers should take of consistency and isolation in the application logic. This may end up being more expensive in terms of development effort. ACID may be essential in the banking and finance industry but not every application requires ACID. Developers should evaluate if eventual consistency or "ACID in practice" is adequate for their application.

### What do you mean by Distributed ACID?

The simplest transaction in a distributed database is **single row ACID** where a transaction impacts only one row. **Single shard ACID** is where the transaction impacts multiple records but they're all on the same shard, which is present on a single node. A true **Distributed ACID** is when the transaction impacts records across shards and nodes.

Achieving distributed ACID is not trivial. A transaction manager that uses Two-Phase Commit (2PC) protocol can be used for atomicity. Paxos or Raft consensus protocols can be used for consistency. To synchronize times for isolation, Network Time Protocol (NTP) can be used.

Not all distributed databases support distributed ACID. For example, since version 4.0, MongoDB supports single shard ACID but not distributed ACID.

Consider Amazon Aurora, Amazon DynamoDB or Azure Cosmos DB. These have multiple masters with data replication. These are not compliant to distributed ACID. Each master has the entire data. Write conflicts are resolved using Last-Write-Wins (LWW) or Conflict-Free Replicated Data Types (CRDT).

## Milestones

1981

Jim Gray at Tandem Computers defines a **transaction** and notes that it must have three properties: consistency, atomicity and durability. At this point, neither isolation as a property nor ACID as an acronym is mentioned. He explains how the transaction concept can be implemented as time-domain addressing or logging plus locking. Tandem itself implemented these ideas in its NonStop approach, thus making their systems highly fault-tolerant.

1983

Based on Gray's concept of a transaction, Theo Haerder and Andreas Reuter identify **isolation** property. They also coin the term **ACID**. They state that "transaction is the only unit of recovery in a database system".

1985

From the mid-1980s, as business applications get more complex, new transaction models appear: nested transactions, chained transactions, distributed transactions. Based on application semantics, a transaction is divided into **sub-transactions**. The concept of **save points** (invented in the 1970s) becomes useful to rollback to specific sub-transactions. However, ACID guarantees are only at the transaction level. Thus, ACID requirements are relaxed for these complex applications.

2000

With the growth of NoSQL and distributed databases since the early 2000s, **CAP Theorem** becomes the blueprint for designing a database system. Likewise, database system designers prefer to adopt **BASE** (Basic Availability, Soft-state, Eventual Consistency). In short, scale, resilience and availability are considered more important than immediate consistency.

2012

Google publishes a paper on **Spanner**, a globally-distributed database that gives high performance without sacrificing strong consistency. It's only in May 2017 that it becomes production ready under the name of **Cloud Spanner**. Likewise, researchers at Yale University publish details of **Calvin** for distributed transactions on partitioned database systems. These are couple of examples showing the trend towards **distributed ACID** from the mid-2010s.

ACID Transactions


# Asynchronous Programming in Python

## Summary

Asynchronous programming is a programming paradigm that enables better concurrency, that is, multiple threads running concurrently. In Python, `asyncio` module provides this capability. Multiple **tasks** can run concurrently on a single thread, which is scheduled on a single CPU core.

Although Python supports multithreading, concurrency is limited by the Global Interpreter Lock (GIL). The GIL ensured that only one thread can acquire the lock at a time. Asynchronous programming doesn't solve the GIL limitation but still enables better concurrency.

With multiprocessing, task scheduling is done by the operating system. With multithreading, the Python interpreter does the scheduling. In Python's asynchronous programming, scheduling is done by what's called **the event loop**. Developers can specify in their code when a task voluntarily gives up the CPU so that the event loop can schedule another task. For this reason, this is also called **cooperative multitasking**.

## Discussion

### What's the background for adopting asynchronous programming in Python?

Python has long supported both multiprocessing and multithreading. **Multiprocessing** can make use of multiple CPU cores but at the overhead cost of inter-process communication (IPC). Each process has its own Python interpreter and GIL. **Multithreading** avoids the IPC overhead by having all threads share the same memory space. But scheduling these threads is limited by the GIL.

For **I/O-bound** threads, if a thread waits on I/O, GIL is automatically released and given to another thread. For **CPU-bound** threads, GIL doesn't give us a mechanism to run all threads concurrently, even if the system has multiple CPU cores. In either case, asynchronous programming helps us achieve better concurrency (for the single CPU core scenario).

GIL is used in Python's default CPython implementation. CPython's use of GIL makes memory management thread safe. Other implementations such as JPython and IronPython are not limited by the GIL. They solve memory management and thread synchronization within their virtual machines (JVM/CLR). Historically, CPython's GIL made it easy to interface with C extensions in a thread-safe manner, which made Python popular.

### Which are the basic constructs that enable asynchronous programming in Python?

Executing asynchronous code requires an **event loop**. Python provides a default event loop implementation. It's also possible to use alternative implementations such as `uvloop`, which has shown at least 2x better performance.

The event loop executes **coroutines**, one at a time. A coroutine is simply a method or function defined with `async` keyword. A coroutine needs to be added to the event loop, such as using `asyncio.run()`.

When a coroutine waits for the result of another coroutine using the `await` keyword, it gets suspended. The event loop then schedules another coroutine that's ready to run. More formally, a coroutine waits on an **awaitable object**. This can be another coroutine, a Task or a Future.

A **Task** is really a wrapper on a coroutine typically created with `asyncio.create_task()`. Via the Task, the coroutine is automatically scheduled.

A **Future** is a low-level object that represents the eventual result of an asynchronous operation. When a Future is awaited, it means the coroutine will wait until the Future is resolved in some other place.

### Which are some basic API calls worth knowing for a beginner?

We can create multiple event loops in a thread but only one can be active at a time. Relevant APIs to learn are `asyncio.new_event_loop()`, `asyncio.set_event_loop()`, `asyncio.get_event_loop()` and `asyncio.get_running_loop()`. If `asyncio.get_event_loop()` is called without a prior call to `asyncio.set_event_loop()`, a new event loop is automatically created and set as the current one.

Event loop methods include `run_until_complete()`, `run_forever()`, `is_running()`, `is_closed()`, `stop()` and `close()`. Callbacks can be scheduled with `call_soon()`, `call_soon_theadsafe()`, `call_later()` and `call_at()`. Another useful method of loop is `create_task()`, which is also possible with `asyncio.create_task()`.

Loop also includes methods to manage network connections and entities: `create_connection()`, `create_datagram_endpoint()`, `create_server()`, `sendfile()`, `start_tls()`, plus low-level methods to work directly with sockets.

`asyncio.run()` runs a coroutine, in the process creating an event loop and closing it at the end. This can't be called if an event loop is already running in the current thread. To run many tasks concurrently, call `asyncio.gather()`. Other useful methods include `asyncio.wait()`, `asyncio.wait_for()`, `asyncio.current_task()` and `asyncio.all_tasks()`. To sleep asynchronously, use `asyncio.sleep()` rather than `time.sleep()`.

### What some essential tips when working with asynchronous Python code?

An event loop can run in any thread but it must run in the main thread if it has to handle signals and execute subprocesses.

To schedule callbacks from another thread, use `loop.call_soon_threadsafe()` rather than `loop.call_soon()`.

Note that calling a coroutine function doesn't actually execute the function and return the result. It only returns a coroutine object, which must be passed into `asyncio.run()`.

Avoid CPU-bound blocking calls. For example, if a CPU-intensive computation takes 1 second, all asynchronous tasks and I/O operations will be delayed by 1 second. Use `loop.run_in_executor()` along with `concurrent.futures.ThreadPoolExecutor` to execute blocking code in a different thread or even a different process.

### Which Python packages or modules enable asynchronous programming?

Module `asyncio` is the main one for asynchronous programming in Python. While `gevent` and `eventlet` achieve similar behaviour, `asyncio` is easier and more approachable even for non-experts.

Use module `threading` for I/O-bound concurrent operations. Use `multiprocessing` for CPU-bound parallel computations. Equivalently, `concurrent.futures.ThreadPoolExecutor` and `concurrent.futures.ProcessPoolExecutor` can be used as these provide a simpler API.

Among the alternatives to `asyncio` are `curio` and `trio`. Both these are somewhat compatible with `asyncio`. Trio in particular claims to focus on usability and correctness.

Timo Furrer curates a list of asynchronous Python frameworks and packages. Among the web frameworks are aiohttp, Tornado, Sanic, Vibora, Quart and FastAPI. For message queues we have aioamqp, pyzmq, and aiokafka. For testing we have aiomock, asynctest, and pytest-asyncio. For databases we have asyncpg, aioredis, and aiomysql. The popular `requests` library with asynchronous support is called `requests-async`.

To make use of multicores or even multiple machines, Apache Spark (via PySpark) should be considered. Meanwhile, work is going on to bring multicore support within the interpreter via projects Software Transactional Memory, Dask and PyParallel.

### How can I debug asynchronous Python code?

By default, `asyncio` runs in production mode. There are many ways to run it in debug mode:

    + Set `PYTHONASYNCIODEBUG` environment variable to 1
    + Use `-X dev` command line option
    + Use argument `debug=True` to `asyncio.run()` (default is `debug=False`)
    + Call `loop.set_debug(True)` (can inspect current value with `loop.get_debug()`)Adjust the logging level by calling for example `logging.getLogger("asyncio").setLevel(logging.WARNING)`. Python interpreter will emit log messages for never-awaited coroutines or never-retrieved exceptions. In debug mode, we get more information about these issues.

The third-party library aiodebug might help in debugging asyncio programs.

### How's the performance of asynchronous programming in Python?

Performance of `asyncio` falls short of the what Node.js and Go can achieve. However, when `asyncio` is used along with `uvloop` (for the event loop) and `httptools` (for HTTP), it gives best performance in terms of both throughput and response time.

While `aiohttp` offers asynchronous HTTP, it has a slow HTTP parser. This limitation is addressed by `httptools`. However, in one experiment, `aiotools` outperformed its synchronous equivalents such as Flask or Django. It was 26x faster than Flask.

In another experiment involving multiple HTTP calls, total runtime was 13.1 secs (synchronous, 1 thread), 1.7 secs (synchronous, 20 threads), and 1.3 secs (aiohttp, 1 thread). When used with Gunicorn web server, `aiohttp` was 2x faster than Flask when a single Gunicorn worker was used.

Another experiment noted that it's unfair to compare synchronous and asynchronous web frameworks without adjusting the **number of workers**. Synchronous frameworks must be given more workers since the entire worker blocks on I/O. With more workers, they can utilize all the CPU cores. This experiment showed that synchronous frameworks did better.

## Milestones

Oct
2002

First version of **Twisted** is released. At a time when the Python standard doesn't offer asynchronous support out of the box, Twisted supports it. It's a Python-based event-driven framework useful for building networked clients and servers. It supports multiple protocols and interfaces.

2009

Ryan Dahl and others at Joyent develop **Node.js**. Asynchronous programming is at the heart of Node.js. With subsequent widespread adoption of Node.js, it becomes important for Python to start looking into asynchronous programming. What Node.js calls promises, Python calls them **awaitables**.

Dec
2012

Work towards standardizing asynchronous support in Python begins as part of **PEP 3156**, titled *Asynchronous IO Support Rebooted: the "asyncio" Module*. This supersedes an earlier proposal (PEP 3153). It also acknowledges earlier work of Twisted, Tornado, and ZeroMQ that have supported asynchronous calls in their packages.

Oct
2013

Version 0.1.1 of `asyncio` is released on PyPI. Version 0.4.1 comes out in April 2014.

Mar
2014

**Python 3.4** is released. Module `asyncio` is formally introduced as part of this release.

Nov
2015

Python 3.5 is released with support for **awaitable objects**, **coroutine functions**, **asynchronous iteration**, and **asynchronous context managers**. This includes syntaxes `async def`, `async for`, `async with` and `await`. Any object that has the `__await()__` method can be awaited. These changes are expected to make asynchronous programming a lot easier. The async/await syntax of this release is inspired by `yield from` of Python 3.3 and `asyncio` module of Python 3.4.

Dec
2016

Python 3.6 is released. **Asynchronous generators** and **asynchronous comprehensions** are now supported. The relevant proposals for these are PEP 525 and PEP 530 respectively.

Jun
2018

Python 3.7 is released in which `async` and `await` are now **keywords**. This release improves the `asyncio` module in terms of **usability** and **performance**. For example, assuming `main()` is our coroutine, scheduling a coroutine to the event loop is as simple as calling `asyncio.run(main())`. This simplifies the earlier syntax of `loop = asyncio.get_event_loop()` and `loop.run_until_complete(main())`.

Oct
2019

Python 3.8 is released with asynchronous support for unit testing. Specifically, `AsyncMock` is added along with suitable assert functions. In the `unittest` module, coroutines can be used as test cases with `unittest.IsolatedAsyncioTestCase`.

Asynchronous Programming in Python


# Artificial Neural Network

## Summary

Artificial Neural Network (ANN) belongs to the field of Machine Learning. It consists of computational models inspired from the human brain and biological neural networks. The goal is to simulate human intelligence, reasoning and memory to solve forecasting, pattern recognition and classification problems.

ANN is effective in scenarios where traditional ML methods such as regression, time series analysis or PCA cannot perform or forecast accurately. This could be because of data bias, mix of continuous and categorical data, unclean or uncertain data.

Complex networks with multiple layers, nodes and neurons are possible today, thanks to dramatic increase in computing power, super-efficient GPUs and Big Data. A modern approach to ANN known as Deep Learning which processes and transforms data cascading through hierarchical layers, has gained immense prominence. Computer vision, character and image recognition, speech detection and NLP are the popular applications of ANN.

## Discussion

### In what ways do ANN replicate the functioning of the human neural network?

In a biological neural network, nerve cells (neurons) are interconnected by signal transmitters (synapses) which pass on electrical/chemical signals to a target neuron. By summation of potentials, either a signal to excite (positive) or inhibit (negative) is transmitted. Human brain contains about 86 billion neurons on average.

When stimulated by an electrical pulse, neurotransmitters are released. They cross into the synaptic gap between neurons and bind to chemical receptors in the receiving neuron. This affects the potential charge of the receiving neuron, and starts up a new electrical signal in the receiving neuron. The whole process takes less than 1/500th of a second. As a message moves from one neuron to another, it is converted from an electrical signal to a chemical signal and back in an ongoing chain of events which is the basis of all brain activity.

To draw parallels in ANN, a network of data elements called artificial neurons receive input and change their internal state (activation) based on that input. They then produce a result depending on the input value and the activation function.

### What is the structure and function of an ANN?

The basic structure of an ANN involves a network of artificial neurons arranged in layers - one input layer, one output layer and one or more hidden layers in between.

Each neuron with the exception of those in the input layer, receives and processes stimuli (inputs) from other neurons. The processed information is available at the output end of the neuron.

Every connection has a weight (positive or negative) attached to it. Positive weights activate the neuron while negative weights inhibit it. The figure shows a network structure with inputs (x1, x2, … xm) being connected to a neuron with weights (w1, w2, … wm) on each connection. The neuron sums all the signals it receives, with each signal being multiplied by its associated weights on the connection.

This output is then passed through a transfer (activation) function g(y) that is normally non-linear to give the final output y. By comparing this output to actual value we determine the error. The process is repeated over several iterations until the error is within acceptable limits.

### What are some key terms used in describing ANN?

We note the following terms:

    + **Perceptron** - A linear binary classifier used in supervised learning. It acts as a single-node neural network. A neural network consists of multi-layer perceptrons (MLPs). Perceptrons consist of 4 parts – inputs, weights & bias, weighted sum and activation function. All inputs x are multiplied with their weights w, then added to get a weighted sum. This sum is applied to the activation function, to get output y of the perceptron.
    + **Activation Function** - A transfer function used to determine output of a neural network. It maps the resulting values into ranges (0 to 1), (-1 to 1) etc. They may be linear or non-linear, the sigmoid function being one of them.
    + **Weights and Biases** - Weights assigned to an input indicate its strength. Higher the weight, greater the influence of that input on the outcome. Bias value allows you to shift the activation function curve up or down.
    + **Loss Function** - A way to evaluate "goodness" of predictions. It quantifies the gap between predicted and actual values. *Sum Of Squares Error* is an example loss function.

### What are the most common types of Artificial Neural Networks used in ML?

We note the following types of ANNs:

    + **Feed Forward Networks** - Simplest form of ANN where data travels in one direction through a network of perceptrons, from input layer towards output layer. Error in prediction is fed back to update weights and biases using backpropagation. Over multiple iterations, weights and bias values are tuned so that the predicted values converge on the actuals. Activation functions commonly used are sigmoid, tanh or RELU.
    + **Recurrent Neural Networks** - Works on the principle of saving the output of a layer and feeding this back to the input to help in predicting the outcome of the layer. From one time-step to the next, each neuron remembers some information it had in the previous time-step. This makes each neuron act like a memory cell in performing computations (LSTM).
    + **Radial Basis Function** - Functions that have a distance criterion with respect to a center. It has an input layer, radial basis hidden layer and output layer. Commonly used activation functions are Gaussian and multi-quadratic. Applied in Power Restoration Systems when restoration happens from core priority areas to the periphery.

### How does an ANN learn?

The ANN learns through an iterative training routine where weights and biases are continually adjusted to improve strength of the prediction. Steps are:

    + Identify input values, assign random initial values as weights to these inputs.
    + For supervised learning, segregate training and test data sets.
    + Finalise the hidden layers and nodes in them.
    + Identify number of epochs and batch sizes. Training data set is generally sliced into **batches**. An **epoch** is one complete pass through the whole training set. Training typically requires several epochs.
    + Pass the training set through the layers. Determine predicted outcome at the end of the iteration.
    + Study extent of miscalculation, now adjust the weights (W = W + ΔW) through backpropagation. Amount of change in W per iteration is called **Learning Rate**. Large values of learning rate would train the network faster, but may result in overshooting the optimal solution. We need a balance, determined by the gradient descent method of optimising the loss/cost function.
    + After several iterations, network achieves acceptable reliability in predicted outcome. Verify by applying the test data set, check for model accuracy.Now the ANN is set to have ‘learned’ and is ready for deployment.

### What are the key differentiators between traditional ML techniques and ANN?

ANN and traditional ML techniques like logistic regression are algorithms meant to do the same thing - classification of data. However, while logistic regression is a statistical method, ANN is a heuristic method modelled on the human brain.

In many cases, simple neural network configurations yield the same solution as many traditional statistical applications. For example, a single-layer, feed forward neural network with linear activation for its output perceptron is equivalent to a general linear regression fit. When you use sigmoid activation function in ANN, it behaves like logistic regression.

However, one of the unique aspects of an ANN is the presence of its hidden layers. Since movement of data between these layers is automatic, these steps cannot be statistically expressed. Hence debugging or tracing data values through these intermediate steps isn't possible. Whereas in regular ML algorithms, the input to output transition can be traced entirely.

The ability of data to reiterate through hidden layers allows hierarchical processing. This is the reason ANN forms the basis of Deep Learning, while other ML techniques are unsuitable.

### When to use and not to use Artificial Neural Networks?

When data is well structured, free of inconsistencies and somewhat linear in nature, traditional ML models such as linear regression, classification or PCA work remarkably well.

But in applications such as text validation or speech recognition, data tends to be non-linear and incomplete. It is also subject to human error and variations of language, dialect or handwriting. In such applications, ANN works with good accuracy.

For large and complex data where a small amount of time series data is available or where large amounts of noise exist, standard ML approaches can become difficult, even impossible. They may require weeks of a statistician’s time to build, as the data clean up and pre-processing effort is huge.

Neural networks accommodate circumstances where the existing data has useful information to offer, but it might be clouded by factors mentioned above. Neural networks can also account for mixtures of continuous and categorical data.

To build a predictive model for a complex system, ANN does not require a statistician and domain expert to screen through every possible combination of variables. Thus, the neural network approach can dramatically reduce the time required to build a model.

## Milestones

1943

Neurophysiologist Warren McCulloch and mathematician Walter Pitts write a paper on how neurons might work. In order to describe how neurons in the brain might work, they model a simple neural network using electrical circuits.

1948

Donald Hebb in his book *The Organization of Behaviour* proposes a model of learning based on neural plasticity. Later called **Hebbian Learning** it's often summarized by the phrase "cells that fire together, wire together".

1958

Frank Rosenblatt, a psychologist at Cornell proposes the idea of a **Perceptron**, modeled on the McCulloch-Pitts neuron.

1960

Widrow and Hoff develop a learning procedure that examines weight values and determine output of perceptrons accordingly.

1975

The first multilayered network is developed. It's an unsupervised network.

1975

A key trigger for renewed interest in neural networks and learning is Werbos's **backpropagation** algorithm that made the training of multi-layer networks feasible and efficient.

1985

American Institute of Physics, establishes a *Neural Networks in Computing* annual meeting.

1987

The first International Conference on Neural Networks is organized by the Institute of Electrical and Electronics Engineers (IEEE).

2009

**Recurrent neural networks** and **deep feedforward neural networks** developed in Schmidhuber's research group win eight international competitions in pattern recognition and machine learning.

2010

Backpropagation training through **max-pooling** is accelerated by GPUs and shown to perform better than other pooling variants.

2012

Ng and Dean create a network that learns to recognize higher level concepts such as cats, only from watching unlabeled images taken from YouTube videos.

Artificial Neural Network