# Single Page Application

## Summary

A web application broadly consists of two things: data (content) and control (structure, styling, behaviour). In traditional applications, these are spread across multiple pages of HTML, CSS and JS files. Each page is served in HTML with links to suitable CSS/JS files. A Single Page Application (SPA) brings a new programming paradigm for the web.

With SPA, we have a single HTML page for the entire application. This page along with necessary CSS and JS for the site are loaded when the page is first requested. Subsequently, as the user navigates the app, only relevant data is requested from the server. Other files are already available with the client. The page doesn't reload but the view and HTML DOM are updated.

SPA (along with PWA) is the modern way to build web applications. SPA enhances user experience. There are frameworks that simplify building SPAs.

## Discussion

### Could you explain the single page application for a beginner?

In a typical multi-page application, each page is generated as HTML on the server and served to the client browser. Each page has its own URL that's used by the client to request that page.

When a user navigates from one page to another, the entire page loads. However, it's common for all pages to share many UI components: sidebar, header, footer, navigation menu, login/logout UI, and more. It's therefore wasteful to download these common elements with every page request. In terms of user experience, moving from one page to another might be annoying. Current page might lose UI interaction as user waits for another page to load.

In SPA, there's a single URL. When a link is clicked, relevant content is downloaded and specific UI components are updated to render that content. User experience improves because user stays with and can interact with the current page while the new content is fetched from the server. When an update happens, there's no transition to another page. Parts of the current page are updated with new content.

### How does the lifecycle of an SPA request/response compare against a traditional multi-page app?

In multi-page apps, each request is for a specific page or document. Server looks at the URL and serves the corresponding page or document. The entire app is really a collection of pages.

In SPA, the first client request loads the app and all its relevant assets. These could be HTML plus JS/CSS files. If the app is complex, this initial bundle of files could be large. Therefore, the first view of the app can take some time to appear. During this phase, a loader image may be shown to the user.

Subsequently, when the user navigates within the SPA, an API is called to fetch new data. The server responds with only the data, typically in JSON format. The browser receives this data and updates the app view. User sees this new information without a page reload. The app stays in the same page. Only the view changes by updating some components of the page.

SPAs are well-suited when we wish to build rich interactive UI with lots of client-side behaviour.

### Which are the different SPA architectures?

Application content might be stored in files or databases. It can be dynamic (news sites) or contextual (user specific). Therefore, the application has to transform this content into HTML so that users can read them in a web browser. This transformation process is called *rendering*. From this perspective, we note the following SPA architectures:

    + **Client-Side Rendering**: When browser requests the site, server responds quickly with a basic HTML page. This is linked to CSS/JS files. While these files are loading, user sees a loader image. Once data loads, JavaScript on the browser executes to complete the view and DOM. Slow client devices can spoil user experience.
    + **Server-Side Rendering**: HTML page is generated on the fly at the server. Users therefore see the content quickly without any loader image. At the browser, once events are attached to the DOM, app is ready for user interaction.
    + **Static Site Generators**: HTML pages are pre-generated and stored at the server. This means that the server can respond immediately. Better still, the page can be served by a CDN. This is the fastest approach. This approach is not suitable for dynamic content.

### What are the benefits of an SPA?

With SPA, applications load faster and use less bandwidth. User experience is seamless, similar to a native app. Users don't have to watch slow page reloads. Developers can build feature-rich applications such as content-editing apps. On mobile devices, the experience is richer: clicks can be replaced with scrolling and amazing transitions. With browsers providing many developer tools, SPAs are also easy to debug on the client side.

SPA optimizes bandwidth usage. Main resources (HTML/CSS/JS) are downloaded only once and reused. Subsequently, only data is downloaded. In addition, SPAs can cache data, thereby saving bandwidth. Caching also enables the application to work offline.

### What are some criticisms or disadvantages of an SPA?

Among the disadvantages of SPA is **SEO**. SPA has a single URL and all routing happens via JavaScript. More recently, Google is able to crawl and index JS files. In general, use multi-page apps if SEO is important. Adopt SPA for SaaS platforms, social networks or closed communities where SEO doesn't matter.

SPA breaks **browser navigation**. Browser's back button will go to previous page rather than previous app view. This can be overcome with the *HTML5 History API*.

SPA could lead to **security issues**. Cross-site scripting attacks are possible. If developers are not careful, sensitive data could be part of the initial data download. Since all this data is not necessarily displayed on the UI, it can give developers a false sense of security. Developers could also unknowingly provide access to privileged functionality at the client side.

SPA needs **client-side processing** and therefore may not work well on old browsers or slow devices. It won't work if users turn off JavaScript in their browsers. SPAs can be hard to maintain due to reliance on many third-party libraries.

It's worth reading Adam Silver's article on the many disadvantages of SPAs.

### What are some best practices when converting a traditional app to SPA?

An SPA has to implement many things that come by default in traditional apps: browsing history, routing, deep linking to particular views. Therefore, **select a framework** that facilitates these. Select a framework with a good ecosystem and a modular structure. It must be flexible and performant for even complex UI designs.

After the initial page loads, subsequent data is loaded by making API calls. Building an SPA implies a **well-defined API**. Involve both frontend and backend engineers while creating this API. In one approach, serve static files separately from the data that's handled by API endpoints.

Define clearly which parts of the UI are dynamic. This helps to organize project modules. Structure the project to enable **reusable components**.

Due to its high reliance on JavaScript, invest in **build tools** for better dependency management. Webpack is a good choice. A build process can do code compilation (via Babel), file bundling and minification.

When converting to an SPA, don't take an all-out approach. **Migrate incrementally**, perhaps one page at a time.

### How do I test and measure performance of an SPA?

Testing tools Selenium, Cypress and Puppeteer can also be used to measure app performance. WebPageTest is an online tool that's easier to use. Compared to multi-page apps, there's more effort to fill forms or navigate across views.

Application performance on the client side can be monitored via Navigation Timing API and Resource Timing API. But these fail to capture JavaScript execution times. To address this, User Timing API can be used. LinkedIn took this approach and improved the performance of their SPA by 20%. Among the techniques they used are lazy rendering (defer rendering outside viewport) and lazy data fetching.

At Holiday Extras, their app took 23 seconds to load on a good 3G connection. To reduce this, they adopted code splitting to defer loading of non-critical libraries. CSS was also split into three parts loaded at different stages: critical, body, onload. They moved from JS rendering to HTML rendering, and then started serving static HTML from Cloudfront CDN. They did real user monitoring (RUM). Among the tools they used were React, EJS, Webpack, and Speed Curve.

### Could you mention some popular websites or web apps that are SPAs?

Facebook, Google Maps, Gmail, Twitter, Google Drive, and GitHub are some examples of websites built as SPAs.

For example, in Gmail we can read mails, delete mails, compose and send mails without leaving the page. It's the same with Google Maps in which new locations are loaded and displayed in a seamless manner. In Grammarly, writers get suggestions and corrections as they compose their content. All this is powered by HTML5 and AJAX to build responsive apps.

Trello is another example of SPA. The card layout, overlays, and user interactions are all done without any page reloads.

### Which are some tools and frameworks to help me create an SPA?

The three main frameworks for building SPAs are React, Angular and Vue on the client side, and Node.js on the server side. All these are based on JavaScript. Other JavaScript frameworks include Meteor, Backbone, Ember, Polymer, Knockout and Aurelia.

Developers can choose the right framework by comparing how each implements or supports UI, routing, components, data binding, usability, scalability, performance, and testability. For example, while Ember comes with routing, React doesn't; but many modules for React support routing. React supports reusable components. React supports one-way data binding whereas Angular supports two-way data binding. Ember and Meteor are opinionated whereas React and Angular are less so and more flexible.

.NET/C# developers can consider using Blazor. Blazor can work both at client side and server side. It runs in a web browser due to WebAssembly.

Design tools support traditional multi-page sites. Adobe Experience Manager Sites is a tool that allows designers to create or edit SPAs. It supports drag-and-drop editing, out-of-the-box components and responsive web design.

### How does an SPA differ from PWA?

PWA uses standard web technologies to deliver mobile native app-like experience. They were meant to make responsive web apps feel more native on mobile platforms. PWA enables the app to work offline, push notifications and access device hardware. Unlike SPA, PWA use service workers, web app manifest and HTTPS.

PWA load almost instantly since service workers run in a separate thread from the UI. SPAs need to pre-fetch assets at the start and therefore there's always an initial loading screen. SPAs can also use service workers but PWA do it better. In terms of accessibility, PWA are better than SPAs. SPAs might be suited for data-intensive sites that are not necessarily visually stunning.

But PWA are not so different from SPA. Both offer app-like user experience. Many PWA are built with the same frameworks that are used to build SPA. In fact, an app might initially be developed as an SPA. Later, additional features such as caching, manifest icons and loading screens could be added. These make an SPA more like a PWA.

## Milestones

1995

In the mid-1990s, rich interactions on web browsers become possible due to two different technologies: **Java Applets** and **Macromedia Flash**. Browsers are merely proxies for these technologies that have to be explicitly installed as browser plugins. With these technologies, all content is either loaded upfront or loaded on demand as the view changes. No page reloads are necessary. In this sense, these are ancestors of modern SPAs.

2005

Jesse James Garrett publishes a paper titled *Ajax: A New Approach to Web Applications*. This describes a novel way to design web applications. AJAX, that expands to **Asynchronous Javascript + XML**, makes asynchronous requests in the background while the user continues to interact with the UI in the foreground. Once the server responds with XML (or JSON or any other format) data, the browser updates the view. AJAX uses the `XMLHTTPRequest` API. While this was around since the early 2000s, Garrett's paper popularizes this approach.

2008

With the launch of GitHub, many JavaScript libraries and frameworks are invented and shared via GitHub. These become the building blocks on which true SPAs would later be built.

Sep
2010

Twitter releases a new version of its app with client-side rendering using JavaScript. Initial page load becomes slow. Due to diversity of client devices and browsers, user experience becomes inconsistent. In 2012, Twitter updates the app towards server-side rendering and defers all JS execution until the content is rendered on browser. They also organize the code as CommonJS modules and do lazy loading. These changes reduce the initial page load to a fifth.

May
2016

Google builds an app for its Google I/O event. Google engineers call this both an SPA and a PWA. With an App Engine backend, the app uses web components, Web Animations API, material design, Polymer and Firebase. During the event the app brings more user engagement than the native app. We might say that the app started as a SPA to create a PWA. In general, it's better to plan for a PWA from the outset rather than re-engineer an SPA at a later point.

Feb
2019

Google engineers compare different SPA architectures in terms of performance. One of these is called **rehydration** which combines both server-side and client-side renderings. This has the drawback that content loads quickly but not immediately interactive, thus frustrating the user.

May
2019

With the rise of edge computing, Section describes in a blog post how a Nuxt.js app (based on Vue.js) can be deployed at the edge. The app is housed within a Node.js module deployed at the edge. This SPA uses server-side rendering.

Single Page Application


# Document Object Model

## Summary

Document Object Model (DOM) is the object-oriented representation of an HTML or XML document. It defines a platform-neutral programming interface for accessing various components of a webpage, so that JavaScript programs can change document structure, style, and content programmatically.

It generates a hierarchical model of the HTML or XML document in memory. Programmers can access/manipulate tags, IDs, classes, attributes and elements using commands or methods provided by the document object. It's a logical structure because DOM doesn't specify any relationship between objects.

Typically you use DOM API when documents can fit into memory. For very large documents, streaming APIs such as Simple API for XML (SAX) may be used.

The W3C DOM and WHATWG DOM are standards implemented in most modern browsers. However, many browsers extend these standards. Web applications must keep in view the DOM standard used for maintaining interoperability across browsers.

## Discussion

### What are the different components of a DOM?

Purpose of DOM is to mirror HTML/XML documents as an in-memory representation. It's composed of:

    + Set of objects/elements
    + Hierarchical structure to combine objects
    + An interface to access/modify objectsDOM lists the required interface objects, with supported methods and fields. DOM-compliant browsers are responsible to supply concrete implementation in a particular language (mostly JavaScript).

Some HTML DOM objects, functions & attributes:

    + **Node** - Each tree node is a Node object. Different types of nodes inherit from the basic `Node` interface.
    + **Document** - Root of the DOM tree is the HTMLDocument node. Usually available directly from JavaScript as document or window. Gives access to properties associated with a webpage such as URL, stylesheets, title, or characterSet. The field `document.documentElement` represents the child node of type `HTMLElement` and corresponds to `<html>` element.
    + **Attr** – An attribute in an `HTMLElement` object providing the ability to access and set an attribute. Has name and value fields.
    + **Text** — A leaf node containing text inside a markup element. If there is no markup inside, text is contained in a single `Text` object (only child of the element).

### Can you show with an example how a web page gets converted into its DOM?

The simplest way to see the DOM generated for any webpage is using "Inspect" option within your browser menu. DOM element navigation window that opens allows you to scroll through the element tree on the page. You can also alter some element values and styles – text, font, colours. Event listeners associated with each elements are also listed.

The document is the root node of the DOM tree and offers many useful properties and methods. `document.getElementById(str)` gives you the element with `str` as id (or name). It returns a reference to the DOM tree node representing the desired element. Referring to the figure, `document.getElementById('div1')` will return the first "div" child node of the "body" node.

We can also see that "html" node has two direct children, "head" and "body". This example also shows three leaf nodes containing only text. These are one "title" and two "p" tags.

Corresponding CSS and JavaScript files referenced from HTML code can also be accessed through DOM objects.

### How is JavaScript used to manipulate the DOM of a web page?

The ability to manipulate webpages dynamically using client-side programming is the basic purpose behind defining a DOM. This is achieved using DHTML. DHTML is not a markup language but a technique to make dynamic web pages using client-side programming. For uniform cross-browser support of webpages, DHTML involves three aspects:

    + **JavaScript** - for scripting cross-browser compatible code
    + **CSS** - for controlling the style and presentation
    + **DOM** - for a uniform programming interface to access and manipulate the web page as a documentGoogle Chrome, Microsoft Edge, Mozilla Firefox and other browsers support DOM through standard JavaScript. JavaScript programming can be used to manipulate the HTML page rendering, the underlying DOM and the supporting CSS. List of some important DOM related JavaScript functionalities:

    + Select, Create, Update and Delete DOM Elements (reference by ID/Name)
    + Style setting of DOM Elements – color, font, size, etc
    + Get/set attributes of Elements
    + Navigating between DOM elements – child, parent, sibling nodes
    + Manipulating the BOM (Browser Object Model) to interact with the browser
    + Event listeners and propagation based on action triggers on DOM elements

### Can DOM be applied to documents other than HTML or XML?

By definition, DOM is a language-neutral object interface. W3 clearly defines it as an API for valid HTML and well-formed XML documents. Therefore, a DOM can be defined for any XML compliant markup language. The WHATWG community manages the HTML DOM interface. Some Microsoft specific XML extensions define their own DOM.

**Scalable Vector Graphics (SVG)** is an XML-based markup language for describing two-dimensional vector graphics. It defines its own DOM API.

**XAML** is a declarative markup language promoted by Microsoft, used in UI creation of .NET Core apps. When represented as text, XAML files are XML files with `.xaml` extension. By treating XAML as a XAML node stream, XAML readers communicate with XAML writers and enable a program to view/alter the contents of a XAML node stream similar to the XML Document Object Model (DOM) and the `XmlReader` and `XmlWriter` classes.

**Standard Generalized Markup Language (SGML)** is a standard for how to specify a document markup language or tag set. The DOM support for SGML documents is limited to parallel support for XML. While working with SGML documents, the DOM will ignore `IGNORE` marked sections and `RCDATA` sections.

### What are the disadvantages of using DOM?

The biggest problem with DOM is that it is **memory intensive**. While using the DOM interface, the entire HTML/XML is parsed and a DOM tree (of all nodes) is generated and returned. Once parsed, the user can navigate the tree to access data in the document nodes. DOM interface is easy and flexible to use but has an overhead of parsing the entire HTML/XML before you can start using it. So when the document size is large, the memory requirement is high and initial document loading time is also high. For small devices with limited on board memory, DOM parsing might be an overhead.

SAX (Simple API for XML) is another document parsing technique where the parser doesn’t read in the entire document. Events are triggered when the XML is being parsed. When it encounters a tag start (e.g. `<sometag>`), then it triggers the tagStarted event. When the end of the tag is seen (`</sometag>`), it triggers tagEnded. So it's better in terms of memory efficiency for heavy applications.

In earlier days, the DOM standard was not uniformly adopted by various browsers, but that incompatibility issue doesn’t exist anymore.

### What sort of DOM support is offered by React, Node.js and other JavaScript-based platforms?

Everything in DOM is a node – document/element/attribute nodes, etc. But if you have a list of 10 items on your webpage and after some user interaction, need to update one of them, the entire DOM will be re-rendered. This is especially troublesome in Single Page Applications (SPAs).

React WebUI framework solves this by creating a **virtual DOM** which is an in-memory data-structure cache for selective rendering. Differences in the node rendering are computed, browser's displayed DOM is updated efficiently, "reconciled" by the algorithm.

NodeJS runtime environment has its own implementation for DOM interface, used when we need to work with HTML on server side for some reason. The DOM `Node` interface is an abstract base class upon which many other DOM API objects are based, thus letting those object types to be used similarly and often interchangeably. `jsdom` is a pure JavaScript implementation of WHATWG DOM and HTML Standards for use with Node.js.

In AngularJS scripting framework, there are directives for binding application data to attributes of HTML DOM elements. Ex. `ng-disabled` directive binds AngularJS application data to the disabled attribute of HTML elements.

## Milestones

1995

Brendan Eich and Netscape design and release JavaScript, first supported in Netscape Navigator. In subsequent years, JavaScript becomes one of the core technologies of the World Wide Web, alongside HTML and CSS. All major web browsers have a dedicated JavaScript engine to execute it. In 1997, it's standardized as ECMAScript.

1996

JScript is introduced as the Microsoft dialect of the ECMAScript standard. Limited support for user-generated events and modifying HTML documents in the first generation of JavaScript & JScript is called "DOM Level 0" or **Legacy DOM**. No independent standard is developed for DOM Level 0, but it's partly described in the specifications for HTML 4.

1997

Netscape and Microsoft release version 4.0 of Netscape Navigator and Internet Explorer respectively. DHTML support is added to enable changes to a loaded HTML document. DHTML requires extensions to Legacy DOM implementations but both browsers developed them in parallel and remain incompatible. These versions of the DOM later become known as the **Intermediate DOM**.

1998

W3C DOM Working Group drafts a standard DOM specification, known as **DOM Level 1** that becomes the W3C Recommendation in 1998. This is after the standardization of ECMAScript.

2001

Microsoft Internet Explorer version 6 comes out with support for W3C DOM.

2004

Mozilla comes out with its *Design Principles for Web Application Technologies*, the consensus opinion of the Mozilla Foundation and Opera Software in the context of standards for Web Applications and Compound Documents. This defines browser code compatibility with HTML, CSS, DOM, and JavaScript.

2005

Large parts of W3C DOM are well-supported by all the common ECMAScript-enabled browsers including Safari and Gecko-based browsers (like Mozilla, Firefox, SeaMonkey and Camino).

2020

The HTML DOM living standard is a constantly updated standard maintained by WHATWG.org, with latest updates happening continuously.

Document Object Model


# Open Data

## Summary

The idea of open data is to share data freely with others. Openness also allows others to modify, reuse or redistribute the data. Openness has two facets: legal and technical. **Legal openness** is about applying a suitable open license to the data. **Technical openness** is about removing technical barriers and making it easy to access, read, store or process the data.

By opening up data, others can unlock value in the form of information and knowledge. For example, in the travel sector, locations, images, prices and reviews are data that can help us plan a holiday. Information is data within a given context. Knowledge personalizes information and helps us make decisions.

Many organizations worldwide promote open data. Open licenses, datasets and tools are available. Governments are releasing open data that citizens can use.

## Discussion

### Could you describe open data?

Database is really about structure and organization of data, also known as the database model. This is generally covered by copyright. In the context of open data, we're more concerned with the contents of the database, which we simply call data.

Data can mean a single item or an entire collection. Particularly for factual databases, a collection can be protected but not individual items. For example, a protected collection may be about the melting point of various substances but no one can be prevented from stating a particular item, such as element E melts at temperature T.

To understand the meaning of openness, we can refer to the *Open Definition* that states, "Open means anyone can freely access, use, modify, and share for any purpose (subject, at most, to requirements that preserve provenance and openness)."

Open data should be accessible at a reasonable cost if not free. It should be available in bulk. There shouldn't be restrictions on who or for what purpose they wish to use the data. Tools to use the data should be freely available and not proprietary. Data shouldn't be locked up behind passwords or firewalls.

### Where could open data be useful?

Open data can make governments more transparent. Citizens will have confidence that their money is being spent as budgeted or in implementing the right policies. For example, one activist noted that Canadian citizens used open data to save their government $3.2bn in fraudulent charitable donations. In Brazil, DataViva provides millions of interactive visualizations based on open government data.

New business opportunities are possible. For example, once Transport for London opened their data, developers used it to build apps. Likewise, Thomson Reuters uses open data to provide better services to its customers. OpenStreetMap and Copernicus are examples that enable new GIS applications.

In research, open data is a part of what is called Open Science. It leads to reproducible research and faster advancements. Open data also enables researchers to revalidate their own findings.

Open data can be used to protect the environment. Some apps that do this include mWater, Save the Rain and Ecofacts.

### Could you mention some sources of open data?

There are many curated lists on the web for open data. From some of these, we mention a few useful sources of open data by category:

    + **General**: DBpedia, Datasets Subreddit, Kaggle, FiveThirtyEight, Microsoft Marco
    + **Government**: Data.gov, Data.gov.uk, European Union Open Data Portal
    + **Economy**: World Bank Open Data, Global Financial Data, International Monetary Fund
    + **Business**: OpenCorporates, Yellowpages, EU-Startups, Glassdoor
    + **Health & Science**: World Health Organization, HealthData.gov, NHS Digital, Open Science Data Cloud, NASA Earth Data, LondonAir
    + **Research**: Google Scholar, Pew Research Center, OpenLibrary Data Dumps, CERN Open Data
    + **Environment**: Climate Data Online, IEA Atlas of Energy

### Which are some organizations working with or for open data?

We mention a few organizations:

    + Open Data Institute: Works with companies and governments to build an open, trustworthy data ecosystem, where people can make better decisions using data and manage any harmful impacts.
    + Open Data Commons: Provides a set of legal tools to publish and use open data. They've published open licenses applicable to data.
    + Open Knowledge Foundation: A worldwide network of people passionate about openness, using advocacy, technology and training to unlock information and enable people to work with it to create and share knowledge. It was briefly called Open Knowledge International. The Open Definition is one of their projects.
    + Open Data Charter: A collaboration between governments and organisations working to open up data based on a shared set of principles.

### Why and what aspects of open data should we standardize?

Data is more valuable if we can combine two different datasets to obtain new insights. **Interoperability** is the key. Given diverse systems, tools and data formats, interoperability can be almost impossible.

Without standards, it becomes more difficult for us to publish, access or share data effectively. Standards also make it easier to repeat processes, compare results and reach a shared understanding. Moreover, we need **open standards** that are available to public and are defined through collaboration and consensus.

Open standards should define a common data model. The data pipeline should be streamlined. It should be easy to combine data. It should promote common understanding.

Open Data Institute's page on standards is a useful resource to learn more. A checklist for selecting a suitable standard looks at the licensing, project needs, maintenance of the standard, and guidance on usage.

### What sort of licensing should I adopt when opening my data?

Releasing your data without a license creates uncertainty. In some jurisdictions, data lacking explicit permission may be protected by intellectual property rights. It's therefore better to attach a license.

Aspects that define a license include public domain, attribution, share-alike, non-commercial, database only and no derivatives.

There are a number of licenses that conform to Open Definition. *Creative Commons CC0* is an example. It releases the data into public domain. Anyone can copy, modify and distribute, even for commercial purposes, without asking permission. A similar license is *Open Data Commons Public Domain Dedication and Licence (PDDL)*. Other conformant but less reusable licenses are data licenses from governments (Germany, UK, Canada, Taiwan). UK's Open Government Licence is an example.

Two other licenses worth looking at come from the Linux Foundation: CDLA–Sharing-1.0 and CDLA-Permissive-1.0, where CDLA refers to Community Data License Agreement.

*Open Data Commons Open Database License (ODC-ODbL)* is seen as a "viral license". Any changes you make to the dataset, you're required to release the same under this license.

### What are some challenges with open data?

Publishers continue to use customized licenses. This makes it hard to reuse data. It makes licenses incompatible across datasets. Instead, they should use standardized open licenses. Ambivalent or redundant clauses cause confusion. Licenses often are not clear about the data to which they apply. Data is often not linked to legal terms.

Data is hard to find. Sometimes their locations change. A platform such as CKAN might help.

Data could be misinterpreted, which could result in a wrong decision. This creates a fear of accountability and prevents producers from opening their datasets.

Quality of data is another concern. Data should be machine-readable and in raw form. Publishing data in HTML or PDF is not research friendly. For context and interpretation, metadata should be shared.

Raw data is good but also requires advanced technical skills and domain knowledge. We therefore need to build better data literacy. AI algorithms are being used to analyse data but many are black-box models. They also promote data centralization and control, which are at odds with the open data movement.

Data infrastructure must of consistent quality. Data can also be biased by gender or against minority groups.

## Milestones

1942

The concept of open data starts with Robert King Merton, one of the fathers of the sociology of science. He explains how freely sharing scientific research and results can stimulate growth and innovation.

1994

In the US, the Government Printing Office (GPO) goes online and opens a few government-related documents. This is an early example of **open government data** at a time when the Internet was becoming popular. Historically and legally, the idea can be traced back to the the Freedom of Information Act of 1966.

2005

The Open Knowledge Foundation creates the **Open Definition**. This is based on established principles from the open source movement for software. This definition is later translated into more than 30 languages. In November 2015, Open Definition 2.1 is released.

Feb
2006

At a TED talk, Hans Rosling presents compelling visuals of global trends in health and economics. Using publicly available datasets from different sources, he debunks myths about the developing world. He makes a case for governments and organizations to open up their data. Data must be enabled using design tools. His mantra is to animate and liberate data from closed databases. Data must also be searchable.

Dec
2007

Thirty individuals meet at Sebastopol, California to discuss open public data. Many of them come from the culture of free and open source software movement. This event can be seen as a convergence of many past US and European efforts in open data. They identify **eight principles**: complete, primary, timely, accessible, machine processable, non-discriminatory, non-proprietary, and license-free.

Feb
2009

At a TED talk, Tim Berners-Lee gets people to chant "Raw data, now." He makes reference to Rosling's talk of 2006 and adds that it's important to link data from different sources. He calls this **Linked Data**. He mentions *DBpedia*, which takes data from Wikipedia and connects them up.

May
2009

The US government launches **Data.gov** with 47 datasets. About five years later, it has about 100,000 datasets.

2010

Many governments attempt to open their data to public but there are concerns about privacy. It's only in 2015 that privacy principles become an essential part of discussions on open data. It's become clear that providing raw data is not always possible. Governments must balance potential benefits to public against privacy rights of individuals.

2012

Guillermo Moncecchi of Open Knowledge International writes that beyond transparency, open data is also about building a **public data infrastructure**. While the focus during 2007-2009 was on data transparency, during 2010-2016 focus shifts to public infrastructure. Consider data about street light locations. Citizens can use this data to solve problems on their own. Metrics change from number of published datasets to APIs and reuse.

Jun
2017

Open Knowledge Foundation publishes the **Global Open Data Index (GODI)**. This shows that only 38% of government data is really open. This is based on Open Definition 2.1. A later update available online shows that only 166 of 1410 datasets (11%) are open. The report gives a number of recommendations to governments and policy makers to make their data more open.

Jan
2018

Open Data Charter replaces their earlier ambitious call of "open by default" with a more practical "publish with purpose". Rather than getting governments to open up as much data as possible quickly, the intent is to get them to take small steps in that direction. Governments can open datasets with a clear view of tangible benefits to citizens.

Feb
2018

Using open data exposed by Here.com via an API, Tjukanov uses visualization to show average traffic congestion in many of the world's largest cities. Interesting patterns emerge, plotted within a radius of 50km from the city center. He uses QGIS, an open source GIS platform.  This is an example of the value that open data plus open source can unlock.

Jan
2020

Using open data collated by the Institute for Government from data.gov.uk, Peter Cook produces interesting **organograms**, which are visualizations of organization structures. He does this for many UK government departments. Clicking on individual data points shows name, job title, salary range and other details. This sort of data has been available (though patchy) since March 2011.

Open Data


# Remote Pair Programming

## Summary

Pair programming is a practice in which developers work in pairs. When the pair is not sitting next to each other, we call it **Remote Pair Programming (RPP)**. Remote pair programming has the same benefits as pair programming: higher quality software, fewer defects, knowledge sharing, team cohesion, faster delivery, and more.

The nature of remote working demands the use of suitable tools. Selecting the right set of tools can enable teams get the best out of pairing. There are plenty of tools in the market today (Sep 2021) for RPP.

Since the coming of COVID-19 in 2020 and an increased adoption of work-from-home culture, RPP has become all the more essential.

RPP is also called *Distributed Pair Programming (DPP)*, a term that appears to be popular among researchers.

## Discussion

### How is remote pair programming different from pair programming?

In pair programming, the pair can point to code with their fingers. In RPP, this is done using the keyboard cursor or the mouse pointer. In some tools, both developers may be able to move their own cursors or mouse pointers. In other tools, there's only one cursor and mouse pointer. The person controlling it becomes the driver.

In pair programming, both developers are looking at the same monitor. RPP allows more flexibility. The navigator could be checking out other parts of the code while the driver is typing.

In pair programming, it's possible to share physical artefacts such as printed documents or a whiteboard. In RPP, everything must happen online. It's therefore essential to have tools to perform these activities online.

RPP can be used for diverse use cases beyond coding: mentoring, hiring, live tutorials, etc. These are use cases where participants are likely to be at remote locations.

We should clarify that sharing code for reviews, making pull requests or using version control isn't RPP. These are asynchronous workflows. RPP happens only when both developers are participating concurrently within the same workspace.

### What should I look for when selecting a tool for RPP?

Here are some factors to consider:

    + **Installation**: Often called *Cloud IDEs*, these are tools hosted in the cloud. No local installation is required other than a web browser. Other tools require local installation. Better still are *plugins* that extend popular editors/IDEs with collaborative editing capability.
    + **Cross-Platform**: Some tools may run on Windows but not on Mac or Linux. Plugins are better in this regard. They extend familiar software already available for various platforms.
    + **Editing**: Simultaneous editing by both developers. Copy/paste between systems. One developer can navigate to other parts of codebase or other applications while the other is editing. Editor/IDE agnostic.
    + **Multimodal**: Bidirectional live audio/video streaming. Chat window. Integrated with editor/IDE.
    + **Usability**: Uncluttered layouts. Awareness about what's shared and the current editing context. Automatic turn-off of notifications, thus providing a distraction-free environment.
    + **Performance**: Minimal lag. Supports high video resolutions. Fall back to lower resolutions on low-bandwidth connections. Visibility into performance metrics.
    + **Integration**: Connect to code repositories (GitHub, GitLab, Bitbucket) and other tools (Jira, Trello).
    + **Others**: Security, cost and customer support are also important. Open source may be important for some teams.

### What tools are available for RPP?

There are dozens of tools out there. One way to classify tools is as follows:

    + **Screen Show**: Only screensharing. Before switching roles, we need to push code changes to a shared repository. Videoconferencing tools such as Skype, Google Hangouts, Google Meet and Slack Calls are examples.
    + **Screen Control/Share**: Temporary remote control of your partner's system. Interactions can lag. Zoom, VNC, Join.me, CoScreen, Tuple, TeamViewer and tmux are examples.
    + **Application Share**: True collaborative editing and hence most preferred. Each environment can be personalized. Developers can use different editors or IDEs. A developer can navigate within the codebase without interrupting the partner. Developers can even edit different parts of the code in parallel. Live Share (with Visual Studio and VS Code), CodeTogether, GitLive, Floobits, Drovio, Atom Teletype, and AWS Cloud9 are examples.Among the Cloud IDEs are AWS Cloud9, Codenvy, and Replit. For privacy, Codenvy has a self-hosted option.

Among the plugins are Live Share, Remote Collab, Atom Teletype, CodeTogether, GitLive and Floobits. CodeTogether supports VS Code, IntelliJ and Eclipse. Guests can join from IDE or browser. GitLive supports VS Code, IntelliJ and Android Studio. Floobits supports Emacs, Sublime Text, Neovim, IntelliJ and Atom.

### How do I select the right tool for RPP?

Teletype for Atom suits pair programming's driver-navigator style. Live Share allows more open-ended collaboration, which perhaps is not what you want. However, Live Share might suit ping-pong style of pairing. In strong-style pairing, we want the navigator to guide the driver step by step. This is best done with only screensharing.

In Linux, tmux and tmate are popular. These work even on low-bandwidth connections. However, this may not be the best choice for beginners who find it hard to learn the commands.

Cloud IDEs may not be optimal for all-day coding. It's also too dependent on net connection. Plugins such as CodeTogether track changes within IDEs and are therefore not demanding on bandwidth (unlike screensharing). CodeTogether team also found that allowing multiple developers to edit code is hard to follow. Their design enforces a master controller who can give/take temporary control to others.

### Could you share some tips for RPP?

RPP is easiest if the developers have met before in person. If not, have icebreakers at the start of the project. Informal chats or online games can also help in building rapport. Allow for frequent breaks since remote pairs get tired more easily. When remote pairing across time zones, select a time convenient for both.

Start every session with a clear agenda. Tackle one task at a time. Some companies such as GitLab have an internal app to help developers pair up.

Before pairing for long hours, get the basics right. Use a good headset that mitigates external noise and echo. Use a large monitor or even two monitors. Use comfortable desk and chair.

Make use of non-verbal cues. Lean forward when you wish to speak. Gesture to draw attention.

Frequently check with your partner about audio/video quality and quickly take corrective action. An audio splitter with two connected headsets allows another colleague to easily join in on any conversation. Always have the audio and video on, even during breaks. This gives the feeling of being connected to the "office vibe".

## Milestones

1998

**Extreme Programming (XP)** as practiced at Chrysler is talked about. Pair programming is one of the core practices within XP.

2001

In the *Agile Manifesto*, Beck et al. point out that face-to-face interactions are most effective. In this light, remote pairing is guaranteed to fail. This motivates research into adapting XP for distributed teams. Schümmer and Schümmer present *TUKAN* as a "synchronous distributed team programming environment". TUKAN includes chat, audio, multi-user code editing and integration with version management. They use the term **distributed pair programming**.

2004

Hanks publishes results of an empirical study on RPP. He uses a screensharing application called Virtual Network Computing (VNC). He modifies VNC to support a second cursor that the navigator can use as a pointer. Unlike in earlier tools, the second cursor appears only when required.

Jul
2015

In a literature survey, da Silva Estácio and Prikladnicki find that most studies have been from a teaching perspective. Only a few studies talk about RPP in a real-world software development setting. They also survey RPP tools. They make recommendations about tool features: shared code repository, support for specific pairing roles, role switching, gesturing, etc.

Oct
2015

Tsompanoudi et al. modify a previously available Eclipse plugin and use it in an educational setting to help students learn programming collaboratively. Tasks are streamlined using **collaboration scripts** that were studied by other researchers as early as 2007.

Jan
2019

**Tuple** launches alpha release. It claims to be the "best pair programming tool on macOS". The focus of Tuple is performance: low CPU usage, low latency, high resolution video and no distracting UI components (sometimes called UI chrome). It also exposes performance graphs to the user so that they can take corrective action.

2020

The COVID-19 pandemic forces teams to work remotely. Teams used to pair programming are now required to do the same remotely. The situation forces teams to better understand the dynamics of pairing remotely.  It's also expected that the arrival of 5G will make RPP more reliable and sophisticated.

Mar
2021

Packt Publishing Limited publishes Bolboacă's book titled *Practical Remote Pair Programming*.

Remote Pair Programming


# Continuous Integration

## Summary

Continuous Integration (CI) is the practice of routinely integrating code changes into the main branch of a repository, and testing the changes, as early and often as possible. Ideally, developers will integrate their code daily, if not multiple times a day.

Martin Fowler, Chief Scientist at ThoughtWorks, has stated that,

> Continuous Integration doesn't get rid of bugs, but it does make them dramatically easier to find and remove.

## Discussion

### Why do we need Continuous Integration?

In the past, developers on a team might work in isolation for an extended period of time and only merge their changes to the master branch once their work was completed. This made merging code changes difficult and time consuming, and also resulted in bugs accumulating for a long time without correction. These factors made it harder to deliver updates to customers quickly.

With CI, each code change can potentially trigger a build-and-test process. Testing becomes an essential part of the build process. Bugs, if any, are highlighted early before they get a chance to grow or become hard to trace. Essentially, CI breaks down the development process into smaller pieces while also employing a repeatable process of build and test.

### What are the benefits of Continuous Integration?

Among the many benefits of CI are the following:

    + Shorter integration cycles
    + Better visibility of what others are doing leading to greater communication
    + Issues are caught and resolved early
    + Better use of time for development rather than debugging
    + Early knowledge that your code works along with others' changes
    + Ultimately, enabling the team to release software faster

### How does Continuous Integration work?

With continuous integration, developers frequently commit to a shared repository using a version control system such as Git. Prior to each commit, developers may choose to run local unit tests on their code as an extra verification layer before integrating. A continuous integration service automatically builds and runs unit tests on the new code changes to immediately surface any errors.

Continuous integration refers to the build and unit testing stages of the software release process. Every revision that is committed triggers an automated build and test.

### What are some CI tools and to choose among them?

There are many solutions out there. Some of them include Codeship, TravisCI, SemaphoreCI, CircleCI, Jenkins, Bamboo, and Teamcity.

Some factors to consider when selecting a tool include price (commercial or free), features, ease of use, integration (with other tools and frameworks), support (commercial or community) and more.

### What are the challenges to Continuous Integration?

To improve and perfect your CI, you need to overcome 3 major challenges:

    + **No Standalone Fresh Checkout**: The single biggest hurdle to a smooth CI build is ensuring that your application's tests can be run from a fresh code checkout (e.g. a git clone). This means that all of your app's dependencies are either included in the checkout, or they're specified and can be pulled in by a script in the checkout.
    + **Unreliable Tests**: Now that your app sets up with a single command, you've built a foundation for effective CI. The next challenge is to ensure that your test results are repeatable and reliable. Intermittent or "expected" failures that persist for too long are pernicious. Once the habit of treating failures as intermittent takes hold, legitimate errors often get ignored.
    + **Obscure Build Results**: Once you've produced a reliable test suite, the next challenge is to get results quickly, take appropriate action on them, and distribute information to the people who matter.

### How are Continuous Integration, Continuous Delivery, and Continuous Deployment practices related to one another?

Continuous integration leads to both continuous delivery and continuous deployment. Continuous deployment is like continuous delivery, except that releases happen automatically.

More specifically, continuous integration requires automated testing to ensure that nothing is broken when new commits are made. Continuous delivery takes this to the next step by automating the release process so that your customers get regular fixes and upgrades.

Continuous delivery still requires manual intervention to initiate the final deployment to production. Continuous deployment automates this last step too. There's no "Release Day" as such. Customers see a steady stream of improvements and this enables early feedback. Since releases are small, they're less risky and easier to fix. Jeff Humble, author of the book *Continuous Delivery*, says this about Continuous Deployment,

> Essentially, it is the practice of releasing every good build to users.

## Milestones

1991

Grady Booch first proposes the term **Continuous Integration (CI)**. In 1994, he uses the term in his book *Object-Oriented Analysis and Design with Applications*.

1997

Kent Beck and Ron Jeffries invent **Extreme Programming (XP)** while on the Chrysler Comprehensive Compensation System project. Beck publishes about continuous integration in 1998. Extreme Programming embraces the practice of CI.

2001

**CruiseControl**, one of the first open-source CI tools, is released.

Continuous Integration


# Grammar and Spell Checker

## Summary

A well-written article with correct grammar, punctuation and spelling along with an appropriate tone and style to match the needs of the intended reader or community is always important. Software tools offer algorithm-based solutions for grammar and spell checking and correction.

Classical rule-based approaches employ a dictionary of words along with a set of rules. Recent neural network-based approaches learn from millions of published articles and offer suggestions for appropriate choice of words and way to phrase parts of sentences to adjust the tone, style and semantics of the sentence. They can alter suggestions based on the publication domain of the article like academic, news, etc.

Grammar and spelling correction are tasks that belong to a more general NLP process called **lexical disambiguation**.

## Discussion

### What is a software grammar and spell checker, its general tasks and uses?

A grammar and spell checker is a software tool that checks a written text for grammatical mistakes, appropriate punctuation, misspellings, and issues related to sentence structure. More recently, neural network-based tools also evaluate tone, style, and semantics to ensure that the writing is flawless.

Often such tools offer a visual indication by highlighting or underlining spelling and grammar errors in different colors (often red for spelling and blue for grammar). Upon hovering or clicking on the highlighted parts, they offer appropriately ranked suggestions to correct those errors. Certain tools offer a suggestive corrected version by displaying correction as strikeout in an appropriate color.

Such tools are used to improve writing, produce engaging content, and for assessment and training purposes. Several tools also offer style correction to adapt the article for specific domains like academic publications, marketing, and advertising, legal, news reporting, etc.

However, till today, no tool is a perfect alternative to an expert human evaluator.

### What are some important terms relevant to a grammar and spell checker?

The following NLP terms and approaches are relevant to grammar and spell checker:

    + **Part-of-Speech (PoS)** tagging marks words as noun, verb, adverb, etc. based on definition and context.
    + **Named Entity Recognition (NER)** is labeling a sequence of text into predefined categories such as name, location, etc. Labels help determine the context of words around them.
    + **Confusion Set** is a set of probable words that can appear in a certain context, e.g. set of articles before a noun.
    + **N-Gram** is a sub-sequence of n words or tokens. For example, "The sun is bright" has these 2-grams: {"the sun", "sun is", "is bright"}.
    + **Parallel Corpus** is a collection of text placed alongside its translation, e.g. text with errors and its corresponding corrected version(s).
    + **Language Model (LM)** determines the probability distribution over a sequence of words. It says how likely is a particular sequence of words.
    + **Machine Translation (MT)** is a software approach to translate one sequence of text into another. In grammar checking, this refers to translating erroneous text into correct text.

### What are the various types of grammar and spelling errors?

We describe the following types:

    + **Sentence Structure**: Parts of speech are organized incorrectly. For example, "she began to singing" shows misplaced 'to' or '-ing'. Dependent clause without the main clause, run-on sentence due to missing conjunction, or missing subject are some structural errors.
    + **Syntax Error**: Violation of rules of grammar. These can be in relation to subject-verb agreement, wrong/missing article or preposition, verb tense or verb form error, or a noun number error.
    + **Punctuation Error**: Punctuation marks like comma, semi-colon, period, exclamation, question mark, etc. are missing, unnecessary, or wrongly placed.
    + **Spelling Error**: Word is not known in the dictionary.
    + **Semantic Error**: Grammar rules are followed but the sentence doesn't make sense, often due to a wrong choice of words. "I am going to the library to buy a book" is an example where 'bookstore' should replace 'library'. Rule-based approaches typically can't handle semantic errors. They require statistical or machine learning approaches, which can also flag other types of errors. Often a combination of approaches leads to a good solution.

### What are classical methods for implementing grammar and spell checkers?

Classical methods of spelling correction match words against a given dictionary, an approach alluded by critiques to be unreliable as it can't detect incorrect use of correctly spelled words; or correct words not in the dictionary, like technical words, acronyms, etc.

Grammar checkers use hand-coded grammar rules on PoS tagged text for correct or incorrect sentences. For instance, the rule `I + Verb (3rd person, singular form)` corresponds to the incorrect verb form usage, as in the phrase "I has a dog." These methods provide detailed explanations of flagged errors making it helpful for learning. However, rule maintenance is tedious and devoid of context.

Statistical approaches validate parts of a sentence (n-grams) against their presence in a corpus. These approaches can flag words used out of context. However, it's challenging to provide detailed explanations. Their efficiency is limited to the choice of corpora.

**Noisy channel model** is one statistical approach. A LM based on trigrams and bigrams gives better results than just unigrams. Where rare words are wrongly corrected, using a blacklist of words or a probability threshold can help.

### What are Machine Learning-based methods for implementing grammar and spell checkers?

ML-based approaches are either Classification (discriminative) or Machine Translation (generative).

**Classification** approaches work with well-defined errors. Each error type (article, preposition, etc.) requires training a separate multi-class classifier. For example, a proposition error classifier takes n-grams associated with propositions in a sentence and outputs a score for every candidate proposition in the confusion set. Contextual corrections also consider features like PoS and NER. A model can be a linear classifier like a Support Vector Machine (SVM), an n-gram LM-based or Naïve Bayes classifier, or even a DNN-based classifier.

**Machine Translation** approaches can be Statistical Machine Translation (SMT) or Neural Machine Translation (NMT). Both these use parallel corpora to train a sequence-to-sequence model, where text with errors translates to corrected text. NMT uses encoder-decoder architecture, where an encoder determines a latent vector for a sentence based upon the input word embeddings. The decoder then generates target tokens from the latent vector and relevant surrounding input and output tokens (attention). These benefit from transfer learning and advancements in transformer-based architecture. Editor models reduce training time by outputting edits to input tokens from a reduced confusion set instead of generating target tokens.

### How can I train an NMT model for grammar and spell checking?

In general, NMT requires training an **encoder-decoder model** using cross-entropy as the loss function by comparing maximum likelihood output to the gold standard correct output. To train a good model requires a large number of parallel corpora and compute capacity. Transformers are attention-based deep seq2seq architectures. Pre-trained language models generated by transformer architectures like BERT provide contextual embeddings to find the most likely token given the surrounding tokens, making it useful to flag contextual errors in an n-gram.

**Transfer learning** via fine tuning weights of a transformer using the parallel corpus of incorrect to correct examples makes it suitable for GEC use. Pre-processing or pre-training with synthetic data improves the performance and accuracy. Further enhancements can be to use separate heads for different types of errors.

**Editor models** are better as they output edit sequences instead of corrected versions. Training and testing of editor models require the generation of edit sequences from source-target parallel texts.

### What datasets are available for training and evaluation of grammar and spell check models?

MT or classification models need datasets with annotated errors. NMT requires a large amount of data.

*Lang 8*, the largest available parallel corpora, has 100,051 English entries. *Corpus of Linguistic Acceptability (CoLA)* is a dataset of sentences labeled as either grammatically correct or incorrect. It can be used, for example, to fine tune a pre-trained model. *GitHub Typo Corpus* is harvested from GitHub and contains errors and their corrections.

Benchmarking data in Standard Generalized Markup Language (SGML) format is available. Sebastian Ruder offers a detailed list of available benchmarking test datasets along with the various models (publications and source code).

**Noise models** use transducers to produce erroneous sentences from correct ones with a specified probability. They induce various error types to generate a larger dataset from a smaller one, like replacing a word from its confusion set, misplace or remove punctuations, induce spelling, tense, noun number, or verb form mistakes, etc. **Round-trip MT**, such as English-German-English translation, can also generate parallel corpora. **Wikipedia edit sequences** offer millions of consecutive snapshots to serve as source-target pairs.  However, only a tiny fraction of those edits are language related.

### How do I annotate or evaluate the performance of grammar and spell checkers?

ERRor ANnotation Toolkit (ERRANT) enabled suggestions with explanation. It automatically annotates parallel English sentences with error type information, thereby standardizing parallel datasets and facilitating detailed error type evaluation.

Training and evaluation require comparing the output to the target gold standard and giving a numerical measure of effectiveness or loss. Editor models have an advantage as the sequence length of input and output is the same. Unequal sequences need alignment with the insertion of empty tokens.

Max-Match (\(M^2\)) scorer determine the smallest edit sequence out of the multiple possible ways to arrive at the gold standard using the notion of Levenshtein distance. The evaluation happens by computing precision, recall, and F1 measure between the set of system edits and the set of gold edits for all sentences after aligning the sequences to the same length.

Dynamic programming can also align multiple sequences to the gold standard when there is more than one possible correct outcome.

### Could you mention some tools or libraries that implement grammar and spell checking?

*GNU Aspell* is a standard utility used in GNU OS and other UNIX-like OS. *Hunspell* is a spell checker that's part of popular software such as LibreOffice, OpenOffice.org, Mozilla Firefox 3 & Thunderbird, Google Chrome, and more. Hunspell itself is based on MySpell. Hunspell can use one or more dictionaries, stemming, morphological analysis, and Unicode text.

Python packages for spell checking include `pyspellchecker`, `textblob` and `autocorrect`.

A search for "grammar spell" on GitHub brings up useful dictionaries or code implemented in various languages. There's a converter from British to American English. Spellcheckr is a JavaScript implementation for web frontends.

Deep learning models include Textly-DRF-API and GECwBERT.

Many online services or offline software also exist: WhiteSmoke from 2002, LanguageTool from 2005, Grammarly from 2009, Ginger from 2011, Reverso from 2013, and Trinka from 2020. Trinka focuses on an academic style of writing. Grammarly focuses on suggestions in terms of writing style, clarity, engagement, delivery, etc.

## Milestones

1960

Blair implements a simple spelling corrector using heuristics and a dictionary of correct words. Incorrect spellings are associated with the corrected ones via abbreviations that indicate similarity between the two. Blair notes that this is in some sense a form of pattern recognition. In one experiment, the program successfully corrects 89 of 117 misspelled words. In general, research interest in spell checking and correction begins in the 1960s.

1971

R. E. Gorin writes **Ispell** in PDP-10 assembly. Ispell becomes the main spell-checking program for UNIX. Ispell is also credited with introducing the generalized affix description system. Much later, Geoff Kuenning implements a C++ version with support for many European languages. This is called **International Ispell**. GNU Aspell, MySpell and Hunspell are other software inspired by Ispell.

1980

In the 1980s, GEC systems are **syntax-based systems**, such as EPISTLE. They determine the syntactic structure of each sentence and the grammatical functions fulfilled by various phrases. They detect several classes of grammatical errors, such as disagreement in number between the subject and the verb.

1990

This decade focuses on simple **linear classifiers** to flag incorrect choice of articles or statistical methods to identify and flag use of commonly confused words. Confusion can be due to identical sounding words, typos etc.

2000

Rule-based methods evolve in the 2000s. Rule generation is based on parse trees, designed heuristically or based on linguistic knowledge or statistical analysis of erratic texts. These methods don't generalize to new types of errors. New rules need to be constantly added.

2005

The mid-2000s sees methods to record and create aligned corpora of pre- and post-editing ESL (English as a Second Language) writing samples. SMTs offer improvement in identifying and correcting writing errors. GEC sees the use of semantic and syntactic features including PoS tags and NER information for determining the applicable correction. Support Vector Machines (SVMs), n-gram LM-based and Naïve Bayes classifiers are used to predict the potential correction.

2010

**DNN-based classifier** approaches are proposed in 2000s and early 2010s. However, a specific set of error types have to be defined. Typically only well-defined errors can be addressed with these approaches. SMT models learn mappings from source text to target text using a noisy channel model. SMT-based GEC models use parallel corpora of erratic text and grammatically correct version of the same text in the same language. Open-source SMT engines are available online and include Moses, Joshua and cdec.

2016

**Neural Machine Translation (NMT)** shows better prospects by capturing some learner errors missed by SMT models. This is because NMT can encode structural patterns from training data and is more likely to capture an unseen error.

2018

With the advent of **attention-based transformer architecture** in 2017, its application to GEC gives promising results.

2019

Methods to improve the training data by text augmentation of various types, including cyclic machine translation, emerge. These improve the performance of GEC tools significantly and enable better flagging of style or context-based errors or suggestions. Predicting edits instead of tokens allows the model to pick the output from a smaller confusion set. Thus, editor models lead to faster training and inference of GEC models.

Grammar and Spell Checker


# Question Answering

## Summary

Search engines, and information retrieval systems in general, help us obtain relevant documents to any search query. In reality, people want answers. Question Answering (QA) is about giving a direct answer in the form of a grammatically correct sentence.

QA is a subfield of Computer Science. It's predominantly based on Information Retrieval and Natural Language Processing. Both questions and answers are in natural language.

QA is also related to an NLP subfield called *text summarization*. Where answers are long and descriptive, they're probably summarized from different sources. In this case, QA is also called **focused summarization** or **query-based summarization**.

There are lots of datasets to train and evaluate QA models. By late 2010s, neural network models have brought state-of-the-art results.

## Discussion

### Which are the broad categories of questions answered by QA systems?

**Factoid** questions are the simplest. An example of this is "What is the population of the Bahamas?" Answers are short and factual, often identified by named entities. Variations of factoid questions include single answer, list of answers (such as "Which are the official languages of Singapore?"), or yes/no. Questions typically ask what, where, when, which, who, or is.

QA research started with factoid questions. Later, research progressed to questions that sought **descriptive** answers. "Why is the sky blue?" requires an explanation. "What is global warming?" requires a definition. Questions typically ask why, how or what.

**Closed-domain** questions are about a specific domain such as medicine, environment, baseball, algebra, etc. **Open-domain** questions are regardless of the domain. Open-domain QA systems use large collections of documents or knowledge bases covering diverse domains.

When the system is given a single document to answer a question, we call it **reading comprehension**. If information has to be searched in multiple documents across domains, the term **open-context open-domain QA** has been used.

### What are the main approaches or techniques used in question answering?

QA systems rely on external sources from where answers can be determined. Broad approaches are the following:

    + **Information Retrieval-based**: Extends traditional IR pipeline. *Reading comprehension* is applied on each retrieved document to select a suitable named entity, sentence or paragraph. This has also been called *open domain QA*. The web (or CommonCrawl), PubMed and Wikipedia are possible sources.
    + **Knowledge-based**: Facts are stored in knowledge bases. Questions are converted (by semantic parsers) into semantic representations, which are then used to query the knowledge bases. Knowledge could be stored in relational databases or as RDF triples. This has also been called *semantic parsing-based QA*. DBpedia and Freebase are possible knowledge sources.
    + **Hybrid**: IBM's DeepQA is an example that combines both IR and knowledge approaches.

### What are some variations of question answering systems?

We note the following variations or specializations of QA systems:

    + **Visual QA (VQA)**: Input is an image (or video) rather than text. VQA is at the intersection of computer vision and NLP.
    + **Conversational QA**: In dialogue systems, there's a continuity of context. The current question may be incomplete or ambiguous but it can be resolved by looking at past interactions. CoQA and QuAC are two datasets for this purpose.
    + **Compositional QA**: Complex questions are decomposed into smaller parts, each answered individually, and then the final answers is composed. This technique is used in VQA as well.
    + **Domain-Specific QA**: Biomedical QA is a specialized field where both domain patterns and knowledge can be exploited. AQuA is a dataset specific to algebra.
    + **Context-Specific QA**: Social media texts are informal. Models that do well on newswire QA have been shown to do poorly on tweets. Community forums (Quora, StackOverflow) provide multi-sentence questions with often long answers that are upvoted or downvoted.

### What are the key challenges faced by question answering systems?

QA systems face two challenges: **question complexity (depth)** and **domain size (breadth)**. Systems are good at either of these but not both. An example of depth is "What's the cheapest bus to Chichen Itza leaving tomorrow?" A much simpler question is "Where is Chichen Itza?"

**Common sense reasoning** is challenging. For example, 'longest river' requires reverse sorting by length; 'by a margin of' involves some sort of comparison; 'at least' implies a lower cut-off. Temporal or spatial questions require reasoning about time or space relations.

**Lexical gap** means that a concept can be expressed using different words. For example, we're looking for a 'city' but the question asks about a 'venue'. Approaches to solving this include string normalization, query expansion, and entailment.

**Ambiguity** occurs when a word or phrase can have multiple meanings, only one of which is intended in a given context. The correct meaning can be obtained via corpus-based methods (distributional hypothesis) or resource-based methods.

Sometimes the answer is **distributed** across different sources. QA systems need to align different knowledge ontologies. An alternative is to decompose the question into simpler queries and combine the answers later.

### What are the steps in a typical question answering pipeline?

In IR-based factoid QA, tokens from the question or the question itself forms the query to the IR system. Sometimes stopwords may be removed, the query rephrased or expanded. From the retrieved documents, relevant sentences or passages are extracted. Named entities, n-gram overlap, question keywords, and keyword proximity are some techniques at this stage. Finally, a suitable answer is picked. We can train classifiers to extract an answer. Features include answer type, matching pattern, number of matching keywords, keyword distance, punctuation location, etc. Neural network models are also common for answer selection.

For knowledge-based QA, the first step is to invoke a semantic parser to obtain a logical form for querying. Such a parser could be rule-based to extract common relations, or it could be learned via supervised machine learning. More commonly, semi-supervised or unsupervised methods are used based on web content. Such methods help us discover new knowledge relations in unstructured text. Relevant techniques include distant supervision, open information extraction and entity linking.

### How are neural networks being used in question answering?

Widespread use of neural networks for NLP started with **distributed representation** for words. A feedforward model learned the representation as it was being trained on a language modelling task. In these representations, semantically similar words will be close to one another. The next development was towards **compositional distributional semantics**, where sentence-level representations are composed from word representations. These were more useful for question answering.

Iyyer et al. reduced dependency parse trees to vector representations that were used to train an RNN. Yu et al. used a CNN for answer selection. A common approach to answer selection is to look at the similarity between question and answer in the semantic space. Later models added an attention layer between the question and its candidate answers. Tan et al. evaluated BiLSTMs with attention and CNN. Dynamic Coattention Network (DCN) is also based on attention. Facebook researchers combined a seq2seq model with multitasking.

Transformer architecture has been applied for QA. In fact, QA was one of the tasks to which BERT was fine-tuned (on SQuAD) and evaluated. BERTserini used fine-tuned BERT along with information retrieval from Wikipedia.

### What are some useful datasets for training or evaluating question answering models?

Datasets are used for training and evaluating QA systems. Based on the design and makeup, each dataset might evaluate different aspects of the system better.

Among the well-known datasets are Stanford Question Answering Dataset (SQuAD), Natural Question (NQ), Question Answering in Context (QuAC) and HotpotQA. All four are based on Wikipedia content. Conversational Question Answering (CoQA) is a dataset that's based on Wikipedia plus other sources. Wikipedia often presents data in tables. WikiTableQuestions is a dataset in which answers are in tables rather than freeform text. TyDi QA is a multilingual dataset. TweetQA takes its data from Twitter.

Question Answering over Linked Data (QALD) is a series of datasets created from knowledge bases such as DBpedia, MusicBrainz, Drugbank and LinkedSpending.

Other datasets to note are ELI5, ShARC, MS MARCO, NewsQA, CMU Wikipedia Factoid QA, CNN/DailyMail QA, Microsoft WikiQA, Quora Question Pairs, CuratedTREC, WebQuestions, WikiMovies, GeoQuery and ATIS.

Papers With Code lists dozens of datasets along with their respective state-of-the-art models.

## Milestones

1961

MIT researchers implement a program named *Baseball*. It reads a question from a punched card. It references a dictionary of words and idioms to generate a "specification list", which is a canonical expression of what the question is asking. Content analysis involves syntactic phrase structures.

1963

Bertram Raphael at MIT publishes a memo titled *Operation of a Semantic Question-Answering System*. He describes a QA model that accepts a restricted form of English. Factual information comes from a relational model. Program is written in LISP. Raphael credits LISP's list-processing capability for making the implementation a lot easier.

Dec
1993

Developed at MIT, *START* goes online. This is probably the world's first web-based QA system. It can answer questions on places, people, movies, dictionary definitions, etc.

Jun
1997

With the growth of web, *AskJeeves* is launched as an online QA system. However, it basically does pattern matching against a knowledge base of questions and returns curated answers. If there's no match, it falls back to a web search. In February 2006, the system is rebranded as *Ask*.

Nov
1999

At the 8th Text REtrieval Conference (TREC-8), a Question Answering track is introduced. This is to foster research in QA. TREC-8 focuses on only open-domain closed-class questions (fact-based short answers). At future TREC events, the QA track continues to produce datasets for training and evaluation.

2002

It's helpful to identify the type of question being asked. Li and Roth propose a machine learning approach to **question classification**. Such a classification imposes constraints on potential answers. Due to ambiguity, their model allows for multiple classes for a single question. For example, "What do bats eat?" could belong to three class: food, plant, animal. The features used for learning include words, POS tags, chunks, head chunks, named entities, semantically related words, n-grams, and relations.

2010

After about three years of effort, IBM Watson competes at human expert levels in terms precision, confidence and speed at the *Jeopardy!* quiz show. It's *DeepQA* architecture integrates many content sources and NLP techniques. Answer candidates come with confidence measures. They're then scored using supporting evidence. Watson wins *Jeopardy!* in February 2011.

Dec
2014

Yu et al. look at the specific task of answer selection. Using **distributed representations**, they look for answers that are semantically similar to the question. This is a departure from a classification approach that uses hand-crafted syntactic and semantic features. They use a bigram model with a convolutional layer and a average pooling layer. These capture syntactic structures and long-term dependencies without relying external parse trees.

Jul
2017

Chen et al. use Wikipedia as the knowledge source for open-domain QA. Answers are predicted as text spans. Earlier research typically consider a short piece of already identified text. Since the present approach searches over multiple large documents, they call it "machine reading at scale". Called *DrQA*, this system integrates document retrieval and document reading. Bigram features and bag-of-words weighted with TF-IDF are used for retrieval. The reader uses BiLSTM each for the question and passages, with attention between the two.

Oct
2018

Researchers at Google release **BERT** that's trained on 3.3 billion words of unlabelled text. BERT is a pre-trained language model. As a sample task, they fine-tune BERT for question answering. SQuAD v1.1 and v2.0 datasets are used. Question and text containing the answer are concatenated to form the input sequence. Start and end tokens of the answer are predicted using softmax. For questions without answers, start/end tokens point to `[CLS]` token.

Jan
2019

Google release *Natural Questions (NQ)* dataset. It has 300K pairs plus 16K questions with answers from five different annotators. Answer comes from a Wikipedia page and the model is required to read the entire page. The questions themselves are based on real, anonymized, aggregated queries from Google Search. Answers can be yes/no, long, long and short, or no answer.

2019

On SQuAD 2.0 dataset, many implementations start surpassing human performance. Many of these are based on the **transformer neural network architecture** including BERT, RoBERTa, XLNet, and ALBERT. Let's note that SQuAD 2.0 combines 100K questions from SQuAD 1.1 plus 50K unanswerable questions. When there's no answer, models are required to abstain from answering.

Jun
2019

Since datasets are available only for some domains and languages, Lewis et al. propose a method to synthesize questions to train QA models. Passages are randomly selected from documents. Random noun phrases or named entities are picked as answers. "Fill-in-the-blanks" questions are generated. Using neural machine translation (NMT), these are converted into natural questions.

Feb
2020

Google Research releases *TyDi QA*, a typologically diverse multilingual dataset. It has 200K question-answer pairs from 11 languages. To avoid shared words in a pair, a human was asked to frame a question when they didn't know the answer. Google Search identified a suitable Wikipedia article to answer the question. The person then marked the answer. Researchers expect their model to generalize well to many languages.

Question Answering


# Inter-Service Communication for Microservices

## Summary

In a monolithic application, all parts of the app access a shared database. Each part can easily invoke the functionality of another part. In a microservices architecture, an app is composed of many microservices, each potentially managing its own database. What happens if one service requires data or processing from another service? This is not as trivial or efficient as in a monolithic application.

Inter-service communication (ISC) is an important consideration when designing a microservices-based application. A badly designed app can result in a lot of communication among the different services, resulting in a *chatty app* or *chatty I/O*. Communication can be reduced by having fewer microservices but this replaces the monolith with smaller monoliths. The goal is therefore to achieve a balance by following good design principles and patterns.

## Discussion

### Why do microservices need to communicate with one another?

In the traditional approach to building monolithic applications, lot of communication was internal to the app. Such communication was often local, fast and easily manageable. When designing a microservices architecture, we break up the monolithic into independent parts, each of which has a well-defined role. While each microservice can be deployed and scaled independently, none of them deliver the full value of the application. A microservice will often require data managed by another, or require the services of another.

For example, consider a taxi booking application. Trip management and passenger management are separate microservices but a trip cannot be initiated without some knowledge or authentication of the passenger. Hence these two independent microservices, each doing its specific roles, will still need to communicate.

While microservices architecture has brought benefits to build large-scale applications, it has also exposed the communication across microservices. Complexity that was previously hidden is now visible. Dozens or even hundreds of microservices that make up an app must be "wired together" properly to make the whole thing work.

### What are the different types of inter-service communication for microservices?

Broadly, there are two types:

    + **Synchronous**: Client sends a request and waits for a response. Client code execution itself may prefer to receive the response via a callback (thread is not blocked) or wait (thread is blocked). Either way, the communication to the external world is synchronous.
    + **Asynchronous**: Client sends a request and doesn't wait for a response.With synchronous communication protocols, the receiver has to be available to send back the response. From application perspective, synchronous implies a less responsive user experience since we have to wait for the response. If one of the services in a chain of synchronous requests delays its response, the entire call flow gets delayed.

With asynchronous communication protocols, the request (often called message) is typically sent to a message queue. Even if the receiver is not available, the message will remain in the queue and can be processed at a later time. Even if a service fails to respond, the original asynchronous call is not affected since it's not waiting for a response.

### What protocols and data formats are suitable for inter-service communication for microservices?

**HTTP/HTTPS** is a synchronous protocol. Often service APIs are exposed as REST endpoints. **AMQP** and **MQTT** are examples of asynchronous protocols. To manage the queue, we can use RabbitMQ **message broker**. Instead of a message queue, we can also use an **event bus** for updating data asynchronously.

Synchronous protocols are usually limited to one-to-one interactions. Asynchronous protocols have more options: **one-to-one** (notifications), **one-to-many** (publish/subscribe), or even allow for responses coming back asynchronously. For example, a user sends a tweet on her Twitter account that has many followers. This is an example of one-to-many publish/subscribe model.

The data that these protocols carry must be formatted in a manner understood by all. **Text-based formats** include JSON and XML. XML in particular is very verbose. Therefore, some implementations may prefer **binary formats**: MessagePack, Thrift, ProtoBuf, Avro. Note that these are well-known and popular formats and using them enables easier integration of microservices. It's also possible (but not preferred) to use a proprietary non-standard format internal to an application.

### What design patterns are available for inter-service communication for microservices?

Here are a few patterns to note:

    + **Saga Pattern**: A sequence of transactions, each one local to its database. A microservice triggers another via an event or message. If something fails, reverse operations undo the changes.
    + **API Composition**: Since table joins are not possible across databases, a dedicated service (or API gateway) coordinates the "joins", which are now at application layer rather than within the database.
    + **Command Query Responsibility Segregation (CQRS)**: Services keep materialized views based on data from multiple services. These views are updated via subscribed events. CQRS separates writes (commands) from the reads (queries).
    + **Event Sourcing**: Rather than store state, we store events. State may be computed from these events as desired. This is often used with CQRS: write events but derive states by replaying the events.
    + **Orchestration**: A central controller or orchestrator coordinates interactions across microservices. API Composition is a form of orchestration.
    + **Choreography**: Each microservice knows what to do when an event occurs, which are posted on an event stream/bus.
    + **Service Mesh**: Push application networking functions down to the infrastructure and not mix them with business logic.

### Could you compare orchestration and choreography?

Orchestration is a centralized approach. Calls are often synchronous: orchestrator calls service A, waits for response, then calls service B, and so on. This is good if service B depends on data from service A. However, if service A is down, service B can't be called. By coupling B with A, we've created a dependency. The orchestrator also becomes a single point of failure.

Choreography enables peer-to-peer interactions without a centralized controller. It's more flexible and scalable than the orchestration approach. It's event-driven architecture applied to microservices. The logic of handling an event is built into the microservice. Choreography is asynchronous and non-blocking. The patterns CQRS and Event Sourcing are applicable to choreography.

There are also hybrid approaches where a service orchestrates a few services while it interacts with others via an event stream. In another approach, an orchestrator emits events for other services and consumes response events asynchronously from the event stream for further processing.

To conclude,

> Orchestration is about control whereas choreography is about interactions.

### How do we handle service API calls that fail?

The simplest solution is to **retry** after a specified timeout. A maximum number of retries can be attempted. However, if the operation is not idempotent (that is, it changes application state), then retry is not a safe recovery method.

The other approach is to use a **circuit breaker**. Many failed requests can result in a bottleneck. There's no point sending further requests. This is where we "open the circuit" to prevent further requests to a service that's not responding.

We can also proactively reduce chances of failure by **load balancing** requests. A request must be processed by a service instance and we can select an instance that has less load. Container orchestrators (Kubernetes) or service meshes (Istio) enable this.

### Are there any best practices for defining a service API?

Microservices must be designed to be independent of one another. One approach is to use **Domain-Driven Design**. This talks about understanding the problem space, using design patterns, and refactoring continuously. API should model the domain. It shouldn't leak internal implementations.

APIs must have well-defined semantics and versioning schemes. A microservice can support multiple versions of an API or you could have a service for each version. Public APIs are usually REST over HTTP. Internal APIs can adopt RPC-style, where remote calls can look like local calls. However, they should be designed right to avoid chattiness. Consider the trade-off between making many I/O calls and retrieving too much data.

Since application state is now distributed across microservices, design for and manage **Eventual Consistency**.

While REST calls may use JSON, RPC calls can be more efficient with binary formats enabled by RPC frameworks such as gRPC, Apache Avro and Apache Thrift. To simplify API design and development, use an **Interface Definition Language (IDL)**. This will generate client code, serialization code and API documentation.

### What are some anti-patterns to avoid when microservices need to communicate?

Sharing a database across many microservices is an anti-pattern since this introduces tighter coupling. A single data model for all microservices is another. Using synchronous protocols across many microservices increases latencies and makes your app brittle to failures. If microservices are not properly defined, this may result in chatty I/O that affects performance and responsiveness.

An application may depend on hundreds of shared libraries. In the spirit of code reuse, all microservices may be relying on these libraries. This results in another anti-pattern called **distributed monolith**. Reusing code within a domain or service boundary is fine but anything beyond that is coupling. This form of coupling is worse than code duplication. Shared libraries can be considered but not made mandatory in some areas: logging, tracing, routing.

It's not required that every event should contain full data, particularly when consumers are going to use only some of it. Consider sending essential data and URLs pointing to additional data. To communicate, consider REST plus its alternatives such as messaging and event sourcing.

### What tools can I use to implement inter-service communication for microservices?

Among the message brokers are RabbitMQ, Kafka, ActiveMQ, and Kestrel. Cloud providers offer their own messaging systems such as Amazon SQS, Google Cloud Pub/Sub, and Firebase Cloud Messaging (FCM). Microsoft Azure offers Event Grid, Event Hubs and Service Bus for messaging. NATS, a CNCF project, is an open source messaging system.

Istio enables service mesh technology. Azure Service Fabric Mesh is an alternative. These rely on Envoy as the networking proxy. Similar proxies include Linkerd and Cilium. Conduit is a service mesh designed for Kubernetes.

Netflix's Conductor can help with orchestration. For logging and monitoring, we have Retrace, Logstash, Graylog, and Jaeger. OpenTracing is an API that enables distributed tracing. Circuit breaker pattern can be implemented with Netflix's Hystrix.

To define service APIs, Swagger can be used. In Java, REST-based microservices can be created with Spring Boot.

## Milestones

2004

Eric Evans publishes *Domain-Driven Design: Tackling Complexity in the Heart of Software*. The book relates directly to object-oriented programming. Only later, it's relevance to microservices would be understood.

2007

Michael Nygard explains the **circuit breaker** pattern in his book *Release It!: Design and Deploy Production-Ready Software*. This helps us build fault tolerant systems. In 2011, Netflix invents the Hystrix framework that includes the circuit breaker pattern.

2009

Netflix embraces API-driven architecture that affects both development and operations. This is today seen as the birth of microservices.

2010

At the start of this decade, the three-tier model (web tier, app tier, database tier) is found to break under heavy load. Microservices come to the rescue but they also bring problems relating to inter-service communication. Companies introduce libraries that are built into microservices to handle the networking aspects: Google's Stubby, Netflix's Hystrix, Twitter's Finagle. This however introduces coupling. A few years later these evolve to networking proxy and **service mesh**.

Jan
2016

Version 0.0.7 of **Linkerd** is open sourced on GitHub. It's based on Twitter's Finagle and Netty. This is one of the early beginnings of a service mesh. Likewise, **Istio** version 0.1.0 is released as an alpha version in May 2017.