# Technical Debt

## Summary

To write good software, developers have to follow best practices: architecture, design patterns, code structure, naming convention, coding guidelines, test coverage, documentation, etc. In practice, business needs may push developers to release a functioning product as quickly as possible. Developers may violate the best practices, hoping to make improvements at a later time. When this happens we have **technical debt**.

The goal is not to eliminate technical debt. We accept technical debt in the short term and manage it in the long term. If it's ignored, problems will accumulate until demoralized developers, missed deadlines, unhappy customers and increasing costs drive the product out of the market.

The problems indicated by technical debt are related to many other terms: software maintenance, software evolution, software aging, code decay, code reengineering, design smells, code refactoring, deficit programming, and technical inflation.

## Discussion

### Why do we call technical debt a "debt"?

The debt metaphor comes from finance. It's common for companies to borrow money to grow faster and capture the market quickly. However, this debt has to be eventually repaid with interest. The longer the company delays the repayment, the more it pays in terms of interest. Ward Cunningham applied this debt metaphor to technology, more specifically to software development.

Due to various reasons, developers might not adopt the best approach to build their product. They might release the product even when they don't understand some parts of it. This is the debt they acquire in the hope that they will fix these issues in a future release. Bad code and poor understanding of that code lead to more time and effort to add new features. This is the interest paid on the debt. An hour saved today would require more than an hour tomorrow to fix the problem.

Code refactoring leads to better design that's easy to understand and maintain. However, the longer they postpone this refactoring, the more difficult it becomes to refactor, just as interest payments continue to grow in financial debt.

### What's principal and interest with respect to technical debt?

In finance, when we borrow money, the principal has to be paid along with interest. The longer we delay the payment of principal, the more interest we pay.

In technical debt, suppose we keep working with poor code, design or documentation. **Interest** is the extra effort we pay to maintain the software. As the software gets more complex with each release, interest keeps going up.

Instead, we could refactor the code incrementally and make it better for future maintenance. **Principal** is therefore the cost of refactoring. Paying the principal today reduces future interest payments (extra effort).

### How does technical debt arise in a project?

Technical debt could arise due to "business pressure, incorrect design decisions, postponing refactoring indefinitely, updating dependencies or simply the lack of experience of the developer". Bad practices can lead to technical debt: starting development without proper design, lack of testing, poor documentation, or poor collaboration.

A software product may start with a clean design but incur technical debt as requirements change. Sometimes a software component could undergo a number of incremental changes made by multiple developers. These developers may not fully understand that component or its original purpose. Some call this "bit rot technical debt". Coding by copy-paste is a symptom.

Sometimes a third-party software is upgraded in an incompatible manner, such as Magento 1.x to 2.0. All websites using Magento 1.x now have a technical debt since there's no simple upgrade path.

Consider a web application. Though Ajax might be the right approach, it would take longer to develop. So developers use frames instead. This is a conscious decision that incurs technical debt. Developers must still implement a clean solution that can be migrated to Ajax easily in a future release.

### What are the different types of technical debt?

Back in 2009, Martin Fowler identified four types of technical debt:

    + **Reckless & Deliberate**: Developers are aware of good design practices but deliberately choose to ignore them and produce messy code. This leads to excessive interest payments and long time to payback the principal.
    + **Reckless & Inadvertent**: Developers are clueless about good design practices and produce messy code.
    + **Prudent & Deliberate**: Developers willing incur debt because they estimate that interest payments are small (rarely touched code). They do a cost-benefit analysis and accept the debt if it can be overcome.
    + **Prudent & Inadvertent**: This is often retrospective in nature. Developers release clean code but later realize that they could have done it differently. This is inadvertent debt. Developers are constantly learning and trying to improve their code. As systems evolve and requirements change, developers may find better designs.From the perspective of interest payment, we could classify technical debt as **eventually repaid** (refactoring on a daily basis, debt tends to zero), **sustainable** (refactoring regularly, debt is constant) or **compound growth** (adding new features but not refactoring, exponential growth of debt).

### Could you share some case studies of technical debt?

According to a study by Stripe, developers spend nearly half their time fighting with technical debt. They estimated that this amounts to $85 billion in annual cost.

Another study of large software organizations showed that 25% of development time is the cost of technical debt. Only some used backlogs and static analyzers to manage technical debt. Very few had a systematic process for addressing technical debt.

Twitter's platform was built on Ruby on Rails that was hard to optimize for search performance. Twitter solved this by eventually switching to a Java server, thus paying off its technical debt.

A Canadian company successfully released a product locally. When they expanded to the rest of Canada, they had to cater to 20% of French-speaking Canadians. They quickly solved this by using a global flag and lots of if-else statements in code. Later they got an order from Japan. Had they made their software multilingual earlier, they could have easily updated their software for Japanese or any other language.

### What are the consequences of technical debt?

If not managed, technical debt has long term effects on the product. It becomes difficult to add features or improve the product. More time is spent on paying off the interest. Any change implies higher costs. As product quality deteriorates, system outages or security breaches can lead to lost sales or fines. The cost of a quick release might be poor design, more bugs, volatile performance, and insufficient testing.

Technical debt has a human cost too. Developers become unhappy. Adding new features becomes a pain. Even when new developers are hired, it takes time to explain what the code does or why it's so messy. New developers could start blaming older developers for the technical debt. Teamwork suffers.

Crippling technical debt might force the team to postpone big changes. They might not adopt latest technologies or upgrade to the latest versions of third-party libraries. Developers might get stuck with outdated frameworks and have no opportunity to upgrade their skills. They may even leave for better opportunities elsewhere.

Ultimately, these problems can be related to business risk, cost, sales, and employee retention. Technical debt gives developers a language to communicate clearly with business folks.

### How can I manage technical debt within my project?

Technical debt when addressed early requires only some code refactoring. If this is postponed, a more expensive rewrite may be needed.

As in financial debt, it's better to pay off the principal to save on future interest payments. In other words, developers must **continuously refactor code**. If a new feature takes three days, a developer could take an extra day to refactor. This might simplify the current feature and make the code easier to work with for the future.

Once code is shipped to customers, developers are reluctant to change it, for the fear of breaking system behaviour. The solution is to add **more tests**. In fact, technical debt implies a more disciplined approach towards refactoring and testing.

Developers should follow **good design practices**: reuse rather than duplicate code; design highly cohesive and loosely coupled modules; name variable, classes and methods to reveal intention; document the code; organize code reviews.

Unlike bugs, technical debt is often invisible. **Tools** such as *Designite* can help identify and track debt. Prioritize high-interest debts for refactoring. Motivate and reward developers for refactoring.

## Milestones

1992

Ward Cunningham coins the term **Technical Debt** while working on WyCASH+, a financial software written in Smalltalk and employing object-oriented programming. He notes that it's important to revise and rewrite code towards better understanding. He also notes,

> A little debt speeds development so long as it is paid back promptly with a rewrite. … The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt.

2001

**The Agile Manifesto** is signed. In the following years, as the Agile software movement gains adoption, it also brings more visibility to technical debt. Technical debt becomes an essential concept of software engineering. Agile is about faster development and responding quickly to customer needs. This focus on short-term deliveries should not compromise long-term goals. It's in this context that technical debt becomes relevant in Agile.

Sep
2009

Robert C. Martin (commonly called Uncle Bob) makes the point that technical debt is incurred due to real-world constraints. It's risky but could be beneficial. It requires discipline and clean coding. **Messy code** is not technical debt. Messy code is due to laziness and unprofessionalism.

Oct
2009

Martin Fowler expands on Uncle Bob's explanation of technical debt. In the process, he identifies the **four quadrants** of technical debt. He also notes that technical debt is a useful metaphor when communicating with non-technical people.

2016

At the IEEE 8th International Workshop on Managing Technical Debt, researchers present their findings on **technical debt in embedded systems**. They find that test, architecture and code debts are more common. In embedded systems, runtime aspects are prioritized higher than design aspects. When the expected lifetime of a component is more than ten years, its maintainability is more seriously considered.

Technical Debt


# Document Development Life Cycle

## Summary

Documentation is considered a means of communication among software developers and software users. It includes written specifications for software, what it does, how it fulfils the specified details, and how to use it.

**Document Development Life Cycle (DDLC)** is a methodical procedure. It enables the creation of documents in a structured order. It's important while creating a document to improve its precision and understandability for end consumers.

Each member of the content management system must be aware of DDLC. It includes technical writers and other content producers. The content developers can improve the accuracy and quality of the complete documentation. For this, they must follow a systemic process. They need to deliver it on schedule.

DDLC can be divided in the following phases: Analysis and planning, Design, Content development, Proofreading and editing, Publishing, and Maintenance.

## Discussion

### Why is the document development life cycle important?

The objective of the workflow for user information/ documentation, is to:

    + Transform software use cases into designs for online and print information. This includes navigation and classification.
    + Implement the documentation design as an in-depth, accurate, and consistent set of information.
    + Make sure that the data is enough for people to use the product.
    + Generate or distribute the data in a timely and effective manner.It helps extract textual information on software specifications. This includes what it does, how it completes the required tasks, and even user instructions.

### What are the stages of DDLC?

To begin a software product's documentation process, the first stage is researching the target audience's needs and analyzing their pain points.

The next stage is to design the document and break it down into smaller segments. Then, structure it with the appropriate format and style.

Technical writers handle the content development stage. This stage requires domain knowledge and technical information. Reviewing and editing are also crucial. Technical writing tools ease collaboration and a transparent review process.

After completing the first draft, technical and editorial experts review the document to ensure accuracy and generate feedback. The writers must integrate the suggestions to maintain quality.

Technical writers take a printout of the document to ensure structure and address any minor issues. The content is published online with added links.

A clear plan for frequent updates is necessary. It requires monitoring with authoring tools. Be responsive and provide users with instant information.

### What tools are used for documentation development?

Microsoft Word will work for a small company with one/two technical writers. This tool won't be enough for a larger company. They'll need to look at other authoring tools like ClickHelp to handle the development life cycle (DDLC).

Technical writers can use translation tools to write. They can also translate their documentation. This helps to speed up the documentation process.

Tools are required for taking screenshots. Pictures in software documentation help achieve better outcomes. They provide more basic yet thorough understanding of the product.

Sometimes, the best option might be to use diagrams to present information. The major benefit comes from embedding a diagram from a cloud-based diagramming tool into a topic. All the diagrams across all the documents update automatically on updating the source.

Inserting videos in online documentation is a recent trend. Documentation becomes clearer and enjoyable for the readers. Hence, it produces excellent results.

### Describe the Spiral model of DDLC and the Waterfall model of DDLC.

The Waterfall Model is a linear sequential approach to software development, where each stage of the development process must be completed before the next one can begin. It has six steps: Analyse, Design, Write, Revise, Distribute and Maintain. It has an advantage. Once the design is finalised, you don't need to return to the Design and Analyse phases for further iteration. Process optimization is achieved here.

The Spiral Model is a more flexible and iterative approach that incorporates elements of the Waterfall model and prototyping. It allows for feedback and revision at each phase of development and is based on the idea of continuous improvement. It is divided into four parts: Analyse, Design, Write and Evaluate. The disadvantage is that, it is time-consuming. All four stages must be followed, every time the document needs to be changed/updated. There are similarities in both process models. Hence, there are challenges for the documentation team in both.

### What are the challenges associated with DDLC?

Document development can be a time-consuming process. And, deadlines may be tight, especially in fast-paced industries.

The document's content might be complex or technical. This would make it challenging to present the information in a way that is easy to understand for the target audience.

Keeping track of document revisions. Ensuring that everyone is working on the most up-to-date version can be challenging. This is particularly when multiple stakeholders are involved.

Document approval processes may involve many stakeholders with differing opinions. It can result in delays and conflicts.

It can be challenging to ensure that all documents are consistent in terms of formatting, style, and tone, particularly when multiple authors are involved.

Ensuring that the documents are accessible to all users, including those with disabilities, may require additional effort.

The document might be needed to be translated into multiple languages. This can add additional complexity and time to the development process.

### What are some practical tips for technical writers?

Understand the target audience and their knowledge level. It helps customize the writing style and language accordingly.

Avoid using technical and complex terms as much as possible. Use simple and clear language that is easy to understand.

Organize your content in a logical and simple manner. Use headings, subheadings, and bullet points to break up the content.

Use visuals such as diagrams, charts, and illustrations. This would help make your content more engaging and easier to understand.

Always review and edit your work. It helps ensure accuracy and clarity. Get feedback from your peers or subject matter experts to improve your writing.

Keep up with the latest industry trends and technologies. This ensures the writing is relevant and informative.

Use templates and style guides to ensure consistency in your writing style and formatting.

Be open to feedback. Be willing to adapt your writing style to meet the needs of your audience.

Continuously improve your writing skills by seeking feedback and learning from others. Stay updated with the latest trends and technologies in your field.

## Milestones

1986

The American National Standards Institute (ANSI) releases the Standard Generalized Markup Language (SGML). It becomes the basis of several subset markup languages including HTML.

1987

Early desktop publishing and page layout software begin appearing on writers’ desktops. This includes products like Ventura Publisher, Interleaf, FrameMaker, and Aldus PageMaker.

1992

ProEdit is founded in Atlanta, GA.

1999

Writers begin using XML, an “eXtensible Markup Language”. It's is evolving from HTML.

2002

The Sarbanes-Oxley Act of 2002 creates new opportunities for technical writers documenting policies, procedures, and internal controls.

Document Development Life Cycle


# Dotdot

## Summary

Dotdot is the universal language of the Internet of Things (IoT), making it possible for devices to work together on any network. Dotdot makes devices interoperable, regardless of vendor or connectivity technologies. Consumers can buy any appliance and expect it to talk to other appliance, so long as both Dotdot certified.

Dotdot was developed by the Zigbee Alliance, an open, global, non-profit organization. Dotdot itself was made possible by the Zigbee Cluster Library (ZCL) defined for the Zigbee stack. Dotdot is more universal. It sits at the application layer but lower layers can be Zigbee, Thread, Wi-Fi, Bluetooth, and more. Zigbee devices could already interoperate among themselves seamlessly. Dotdot extends this to non-Zigbee devices.

Other organizations or communities (Bluetooth, OMG, OCF, GS1, Haystack, Schema.org) are also developing specifications for IoT interoperability at the application layer. These could complement or compete against Dotdot.

## Discussion

### What's the need for Dotdot?

IoT devices connect with various wired or wireless technologies and networking protocols. That's fine so long as devices can understand one another at the application layer.

Back in 2017, a toaster couldn't communicate with a coffee machine. A smart hub probably couldn't control an off-the-shelf smart lock or thermostat. A light switch couldn't turn off or dim a lamp. This is because devices come from different vendors. Their interfaces are either closed, limited or proprietary. For example, devices from Apple, Google, Amazon or Samsung probably need different mobile apps to control them.

This is a difficult scenario for any systems integrator who has to understand and interface many different technologies. Even at runtime, devices will sacrifice some processing to protocol translation. What we need is a common language at the application layer that all devices can understand. This is exactly what Dotdot provides.

It's possible to build a cloud platform that understands all devices. The cloud then becomes the convergence point for devices to talk to one another. But for many applications, relying on the cloud is unacceptable due to reliability, complexity and latency.

### What are the benefits of using Dotdot?

Dotdot has the following benefits:

    + Single solution for all markets - A single application layer that works over many networks means a single choice for developers and consumers. This ends market fragmentation across segments: home, building, industrial, retail, health, energy, and more.
    + Easy - All necessary documents, references and tools are available in a single location and a single certification mark on every certified product and package.
    + Secure - Unique IDs, DTLS sessions, operational certificates and Access Control Lists (ACLs).
    + Global - Built on the open standards and global membership of 400 Zigbee Alliance members. Uses 2.4 GHz and 800-900 MHz bands that are globally available.
    + Proven - Based on the Zigbee Cluster Library (ZCL) that's deployed in over 300 million products for over 10 years.
    + Reliable and robust - Dotdot brings a rich catalog of device interaction models to IP networks. This enables devices to interoperate natively, while being able to interact with similar/complimentary devices on a Zigbee network.
    + Interoperable - Certification ensures device-to-device interoperability. Enables and connects multi-vendor ecosystems.

### Where does Dotdot fit within the protocol stack?

Dotdot is meant for the application layer. Zigbee Alliance also has a certification program to certify devices that are compliant to the Dotdot specification. At the lower layers, Dotdot can interwork with any network or connectivity protocol. This includes Zigbee, Thread, Wi-Fi, Bluetooth, and more.

With Dotdot, we can have direct device-to-device communications within a Zigbee network or within a Thread network. When a Zigbee device needs to talk to a Thread device, there will be a gateway to translate between the two network. However, the devices can understand each other at the application layer, thanks to Dotdot. With Dotdot over Thread, there's no need for a gateway to connect the device to the cloud. Since Thread is IP based, such devices can directly talk to the cloud.

### How is Dotdot related to the Zigbee Cluster Library (ZCL)?

Zigbee Alliance standardized ZCL so that different devices can interoperate at the application layer. ZCL contains 100+ device types, 2400+ certified products, and has matured over 15 years. Only problem with ZCL is that it works only with the Zigbee stack. This is where Dotdot fits in.

Dotdot can be seen as a universal application layer that can work with any underlying stack. As an example, Dotdot interfacing to Thread stack was shown at CES 2017. In this case, ZCL maps to CoAP Resources and HTTP verbs such as GET, PUT, POST, and DELETE.

Dotdot enables Zigbee devices to get connected to the Internet. In fact, Dotdot is considered an alias for **ZCL over IP (ZCLIP)**. It's been said that,

> Dotdot is a standard that allows you to put ZCL on any “rails” other than Zigbee – WiFi, Thread, and so on.

ZCL is optimized for constrained devices. Messages are compact, most fitting within 127-byte 802.15.4 packet. Zigbee Alliance gets a head start by reusing the work done on ZCL for IP networks.

### Which are the documents relevant to Dotdot development?

For mapping ZCL to IP and RESTful interfaces, the following IETF documents are relevant:

    + RFC 6690: Constrained RESTful Environments (CoRE) Link Format
    + RFC 7252: Constrained Application Protocol (CoAP)
    + RFC 7049: Concise Binary Object Representation (CBOR)ZCL Specification is an essential reference. It's also worth reading NXP's ZCL User Guide.

### What exactly is the Dotdot Commissioning App?

The Dotdot Commissioning App is an app that's mostly based on the Thread Commissioning App developed for commissioning Thread-enabled devices. This app facilitates management and expansion of a Dotdot network.

The app first discovers Dotdot-compliant devices in a Thread network. It interrogates each device to discover services, clusters and endpoints. It discovers commands supported by each device. Once this is done, the app can send commands and change attributes on devices.

Zigbee Alliance members get access to this app. This saves operators the trouble of developing their own apps.

### What are the alternatives to Dotdot?

IoTivity is a reference implementation of Open Connectivity Foundation (OCF) specifications. It uses RESTful interfaces.

IPSO Alliance is also involved in defining specifications.

Object Management Group (OMG) defined the Data Distribution Service (DDS) for it's Industrial Internet Consortium (IIC). OMG has defined standards for healthcare and retail.

Bluetooth Special Interest Group (SIG) concerns itself with application layer interoperability among constrained devices. It does this via Generic Attributes (GATT) profiles and assigned numbers.

GS1 is involved in supply chain standards for data exchange across Retail, Healthcare, and Transport & Logistics industries. It's standards include EPC Information Service (EPCIS), Core Business Vocabulary (CBV), Global Product Classification (GPC), and more.

IETF's Extensible Provisioning Protocol (EPP) is an application-layer, client-server protocol for the provisioning and management of objects stored in a shared central repository. Developed for Internet domains and hosts, it can be extended to IoT.

Project Haystack is an open community looking at semantic data models and web services.

Schema.org manages a common set of semantic schemas. It's current ontology can be extended to IoT.

## Milestones

2007

Zigbee Alliance publishes the first release of the **Zigbee Cluster Library (ZCL)**. Revision 6 of this document appears in January 2016. A cluster is a related collection of commands and attributes, which together define an interface to specific functionality. Example clusters include HVAC, lighting, security and safety, and measurement and sensing. ZCL later becomes a starting point for Dotdot.

2013

At a meeting in Boston, a stack vendor and two door lock manufacturers discuss how best to interface their products. Over a few days, they agree on what eventually becomes the **Door Lock Cluster** within ZCL. This cluster defines how doors can be locked/unlocked or their pin codes changed. This is just one example about how clusters in ZCL are organized around specific device types.

2017

Though the IoT world has seen a number of protocols at the connectivity layers the last few years, this is the year when there's serious talk about interoperability at the application layer. Since various devices have to understand one another in terms of types, capabilities and interfaces, it's also called **semantic interoperability**. Many specifications and guidelines are available by the end of the year.

Jan
2017

At CES, Zigbee Alliance demonstrates how Dotdot can enable devices across Zigbee and IP networks talk to each other. In particular, they show Thread-based devices that have Dotdot at the application layer.

Feb
2017

Thread Group releases version 1.1 of the **Thread specification** and a certification program.

Dec
2017

The Zigbee Alliance and Thread Group announce the availability of the **Dotdot over Thread Specification**. The specification is available to Zigbee Alliance members. By mid-2018, they expect to release Dotdot Commissioning Application and launch a certification program.

Jan
2019

Dotdot over Thread certification program is launched. Support for more networks other than Thread is added later in the year.

Dotdot


# Bidirectional RNN

## Summary

Many applications are sequential in nature. One input follows another in time. Dependencies among these give us important clues as to how they should be processed. Since Recurrent Neural Networks (RNNs) model the flow of time, they're suited for these applications.

RNN has the limitation that it processes inputs in strict temporal order. This means current input has context of previous inputs but not the future. Bidirectional RNN (BRNN) duplicates the RNN processing chain so that inputs are processed in both forward and reverse time order. This allows a BRNN to look at future context as well.

Two common variants of RNN include GRU and LSTM. LSTM does better than RNN in capturing long-term dependencies. **Bidirectional LSTM (BiLSTM)** in particular is a popular choice in NLP. These variants are also within the scope of this article.

## Discussion

### Could you explain Bidirectional RNN with an example?

Consider the phrase, 'He said, "Teddy \_\_\_". From these three opening words it's difficult to conclude if the sentence is about Teddy bears or Teddy Roosevelt. This is because the context that clarifies Teddy comes later. RNNs (including GRUs and LSTMs) are able to obtain the context only in one direction, from the preceding words. They're unable to look ahead into future words.

Bidirectional RNNs solve this problem by processing the sequence in both directions. Typically, two separate RNNs are used: one for forward direction and one for reverse direction. This results in a hidden state from each RNN, which are usually concatenated to form a single hidden state.

The final hidden state goes to a decoder, such as a fully connected network followed by softmax. Depending on the design of the neural network, the output from a BRNN can either be the complete sequence of hidden states or the state from the last time step. If a single hidden state is given to the decoder, it comes from the last states of each RNN.

### What are some applications of Bidirectional RNN?

BiLSTM has become a popular architecture for many NLP tasks. An early application of BiLSTM was in the domain of speech recognition. Other applications include sentence classification, sentiment analysis, review generation, or even medical event detection in electronic health records.

BiLSTM has been used for POS tagging and Word Sense Disambiguation (WSD). For Named Entity Recognition (NER), Lample et al. used word representations that captured both character-level characteristics and word-level context. These were fed into a BiLSTM encoder layer. The sequence of hidden states was decoded by a CRF layer.

For lemmatization, one study used two-layer bidirectional GRUs for the encoder. The decoder was a conditional GRU plus another GRU layer. Another study used a two-layer BiLSTM encoder and a one-layer LSTM decoder. A stack of four BiLSTMs has been used for Semantic Role Labelling (SRL).

In general, the paradigm of **embed-encode-attend-predict** has become popular in NLP work. The encode part benefits from BiLSTM, which has been shown to capture position-sensitive features.

Beyond NLP, BiLSTM has been applied to image processing applications such as OCR.

### What are merge modes in Bidirectional RNN?

Merge mode is about how forward and backward hidden states should be combined before being passed on to the next layer. In Keras package, supported modes are summation, multiplication, concatenation and averaging. The default mode is concatenation and this is what most research papers use.

In MathWorks, as on December 2019, only concatenation was supported.

### What are some limitations of Bidirectional RNN?

One limitation with BRNN is that the entire sequence must be available before we can make predictions. For some applications such as real-time speech recognition, the entire utterance may not be available and BRNN may not be adequate.

In the case of language models, the task is to predict the next word given preceding words. BRNN is clearly not suitable since it expects future words as well. Applying BRNN in this application will give poor accuracy. Moreover, BRNN is slower than RNN since results of the forward pass must be available for the backward pass to proceed. Gradients will therefore have a long dependency chain.

LSTMs capture long-term dependencies better than RNN and also solve the exploding/vanishing gradient problem. However, stacking many layers of BiLSTM creates the vanishing gradient problem. Deep neural networks so successful with CNNs are not so successful with BiLSTMs.

## Milestones

Nov
1997

Schuster and Paliwal propose **Bidirectional Recurrent Neural Network (BRNN)** as an extension of the standard RNN. Since the forward and backward RNNs don't interact, they can be trained similar to the standard RNN. On regression and classification experiments they observe better results with BRNN.

2005

For phoneme classification in speech recognition, Graves and Schmidhuber use **Bidirectional LSTM** and obtain good results. It's based on the insight that humans often understand sounds and words only after hearing the future context. In particular, often we don't require an output immediately upon receiving an input. We can afford to wait for a sequence of inputs and then work on the output. They also show that BRNN takes eight times longer to converge than BiLSTM.

Sep
2016

Google replaces its phrase-based translation system with **Neural Machine Translation (NMT)**. It uses a deep LSTM network with 8 encoder and 8 decoder layers. The first layer of encoder is BiLSTM while all others are LSTM.

Apr
2017

Pre-trained word embeddings are commonly used in neural networks for NLP. However, they don't capture context. Supervised learning for capturing context uses limited labelled data. To overcome this limitation, Peters et al. use BiLSTM to learn a language model (LM) and initialize a neural network for sequence tagging. This network uses **two-layer bidirectional GRUs**. They experiment with NER and chunking. They find best result when LM embeddings are used at the output of the first layer.

Jul
2017

For the task of Semantic Role Labelling (SRL), He et al. use an eight-layer network consisting of four BiLSTMs. Their network includes highway connections and transform gates that control inter-layer information flow. Output prediction is done by a softmax layer.

Feb
2018

When BRNNs are stacked, they suffer from vanishing gradients and overfitting. Ding et al. propose a **Densely Connected BiLSTM (DC-BiLSTM)** as a solution. This essentially means that a layer's hidden state includes the hidden states of all preceding layers. They show that the proposed architecture can handle up to 20 layers while improving performance over BiLSTM.

Jun
2018

Peters et al. publish details of a language model called **Embeddings from Language Models (ELMo)**. ELMo representations are deep, that is, they're a linear combination of the states of all LSTM layers rather than using only the top layer representation. They show that higher layers capture context-dependent semantics whereas lower layers capture syntax. While their model uses both forward and backward LSTMs, forward LSTM stack is independent of the backward LSTM stack. Representations at each layer of the two stacks are concatenated. For this reason, they use the term **Bidirectional Language Model (BiLM)**.

Bidirectional RNN


# Word Sense Disambiguation

## Summary

Many words have multiple meanings or senses. For example, the word *bass* has at least eight different senses. The correct sense can be established by looking at the context of use. This is easy for humans because we know from experience how the world works. **Word Sense Disambiguation (WSD)** is about enabling computers to do the same.

WSD involves the use of syntax, semantics and word meanings in context. It's therefore a part of **computational lexical semantics**. WSD is considered an AI-complete problem, which means that it's as hard as the most difficult problems in AI.

Both supervised and unsupervised algorithms are available. The term **Word Sense Induction (WSI)** is sometimes used for unsupervised algorithms. In the 2010s, word embeddings became popular. Such embeddings used with neural network models represent the current state-of-the-art models for WSD.

## Discussion

### Could you explain Word Sense Disambiguation with an example?

Consider sentences "I can hear bass sounds" and "They liked grilled bass". The meaning or sense of the word 'bass' is low frequency tones or a type of fish respectively. The word alone is not sufficient to determine the correct sense. When we consider the word in the context of surrounding words, the sense becomes clear. Using WSD, the second sentence can be **sense-tagged** as "They like/ENJOY grilled/COOKED bass/FISH".

Context comes from the words 'sounds' or 'grilled'. It's also helpful to know that these collocated words are noun and adjective respectively. One comes after 'bass' and the other comes before 'bass'. These syntactic relations give additional information. In general, pre-processing steps such as POS tagging and parsing help WSD.

A difficult example is "the astronomer married the star".

There's no universally agreed senses for a word. Senses can also vary with domains. WSD also relies on knowledge. Without knowledge, it's impossible to determine the correct sense. It's expensive to build knowledge resources. Back in the 1990s, this was seen as the *knowledge acquisition bottleneck*.

### Could you mention some applications of WSD?

WSD is usually seen as an "intermediate task", as a means to an end. Obtaining the correct word sense is helpful in many NLP applications. Exactly how WSD is used is application specific.

Here are some examples:

    + **Machine Translation**: An English translation of the French word 'grille' can be railings, bar, grid, scale, schedule, etc. Correct word sense disambiguation is therefore necessary.
    + **Information Retrieval**: When searching for judicial references with the word 'court', we wish to avoid matches pertaining to royalty.
    + **Thematic Analysis**: Themes are identified based on word distribution but we include only words of the relevant sense.
    + **Grammatical Analysis**: In POS tagging or syntactic analysis, WSD is useful. In the French sentence "L'étagère plie sous les livres", livres refers to 'books' and not 'pounds'.
    + **Speech Processing**: WSD helps in obtaining the correct phonetization in speech synthesis.
    + **Text Processing**: For inserting diacritics, WSD helps in correcting the French word 'comte' to 'comté'. For case changes, WSD corrects 'HE READ THE TIMES' to 'He read the Times'. To "Wikify" online documents, WSD helps.

### What are some essential terms to know about word senses?

Consider the word 'bank'. This can refer to a financial institution or a sloping mound. These two senses of the same word are unrelated but they look and sound the same. We call them **homonyms**. The sense relation is called **homonymy**. Typically, homonyms have different origins and different dictionary entries.

A bank can also refer to the building that houses the financial institution. These senses are semantically related. The sense relation is called **polysemy**.

**Synonyms** are different words with same or nearly same meaning. **Antonyms** are words with opposite meaning.

Consider two words in which one is a subclass of the other, a type-of relation, such as mango and fruit. Mango is a **hyponym** of fruit. Fruit is a **hypernym** of mango.

Consider two words that form a part-whole relation, such as wheel and car. Wheel is a **meronym** of car. Car is a **holonym** of wheel.

### Which are the essential elements for doing WSD?

WSD requires two main sources of information: **context** and **knowledge**. Context is established from neighbouring words and the domain of discourse. Sense-tagged corpora provide knowledge, leading to data-driven or corpus-based WSD. Use of lexicons or encyclopaedia lead to knowledge-driven WSD.

We also need to know possible word senses. These can be **enumerative**, with WordNet being an example. A **generative** model underspecifies senses until context is considered. Rules generate senses. These rules capture regularities in the creation of senses.

In **lexical sample** task, WSD is applied for a sample of pre-selected words. Supervised ML approach is possible based on hand-labelled corpus. In **all-words** task, WSD is applied for all words, for which supervised ML approach is not practical. Dictionary-based approaches or bootstrapping techniques are more suitable.

A **bag-of-words** approach can be used for context. To preserve word ordering, **collocation** can be used when forming feature vectors. Such a vector might include the word's root form and its POS. Syntactic relations, distance from target and selectional preferences are other approaches.

### Could you describe some algorithms for WSD?

A simple supervised approach is to use a **naive Bayes classifier**. We maximize the probability of word sense given a feature vector. The problem is simplified by using Bayes' Rule and assuming features are independent of one another. Another approach is **decision list classifier**.

The **Corpus Lesk** algorithm uses a sense-tagged corpus. We also use the definition or gloss of each sense from a dictionary. Examples from the corpus and the gloss become the signature of the sense. Then we compute the number of overlapping words between the signature and the context. Inverse Document Frequency (IDF) weighting is applied to discount function words (the, of, etc.). The simplified Lesk algorithm uses only the gloss for signature and doesn't use weights.

For evaluation, **most frequent sense** is used as a baseline. Frequencies can be taken from a sense-tagged corpus such as SemCor. Lesk algorithm is also a suitable baseline. *Senseval* and *SemEval* have standardized sense evaluation.

### How are word embeddings relevant to the problem of WSD?

Schütze (1998) proposed the use of word vectors and context vectors. These are large dimensional vectors and often sparse. To make them practical for computation, Singular Value Decomposition (SVD) reduces the number of dimensions. Within such a vector space model, Latent Semantic Analysis (LSA) helps to determine semantic relations. Word embeddings is a modern alternative.

Word embeddings were proposed by Bengio et al. (2003). These are low-dimensional dense vectors that capture semantic information. However, words with multiple senses are reduced to a single vector. This is not directly useful for WSD. To overcome this, Trask et al. (2015) proposed **sense2vec**, where representations are of senses, not words. Sense2vec improved the accuracy of other NLP tasks such as named entity recognition and neural dependency parsing.

Iacobacci et al. (2016) explored the direct use of word embeddings. Using *It Makes Sense (IMS)* framework along with Word2vec, they improved F1 scores on various WSD datasets. Word embeddings of target word and its surrounding words are converted into "sense vectors" using various methods: concatenation, average, fractional decay, exponential decay.

### What are some neural network approaches to WSD?

Neural network approaches to WSD have become popular in the 2010s. Wiriyathammabhum et al. (2012) applied **Deep Belief Networks (DBN)**. They pre-trained the hidden layers using various knowledge sources, layer by layer. They then used a separate fine tuning step for better discrimination.

We lose sequential and syntactic information when averaging word vectors. Instead, Yuan et al. (2016) proposed a semi-supervised **LSTM** model with label propagation. To capture contexts from both sides, Kågebäck and Salomonsson (2016) applied **Bidirectional LSTM**.

Many models consider only context. Knowledge sources such as WordNet are ignored. **Gloss-Augmented WSD (GAS)** considers both context and glosses (sense definitions) and uses BiLSTM.

One attention-based approach is a **encoder-decoder model** with multiple attentions on different linguistic features. Another is **GlossBERT**. It uses BERT, encodes context-gloss pairs of all possible senses, and treats WSD as a sentence-pair classification problem.

### What are some resources to do WSD?

A number of datasets and sense-annotated corpora are available to train WSD models: Senseval and SemEval tasks (all-words, lexical sample, WSI), AIDA CoNLL-YAGO, MASC, SemCor, and WebCAGe. Likewise, word sense inventories include WordNet, TWSI, Wiktionary, Wikipedia, FrameNet, OmegaWiki, VerbNet, and more. These are supported by the modular Java framework **DKPro WSD**.

**UKB** is an open-source toolkit that can be used for knowledge-based WSD. It should be used with optimal default parameters.

In January 2017, Google released word sense annotations on MASC and SemCor datasets. Senses are from New Oxford American Dictionary (NOAD). NOAD senses are also mapped to WordNet.

ACLWiki has curated a list of useful WSD resources. This includes papers, inventories, annotated corpora and software.

Ruder captures the current state of the art in WSD with links to relevant papers. Papers With Code also maintains a list of recent papers on WSD.

## Milestones

Jul
1949

Warren Weaver considers the task of using computers to **translate** text from one language to another. He recognizes the importance of context and meaning. He makes references to cryptography and statistical semantic studies as possible approaches to obtaining the correct meaning of a word. Weaver also notes that a word mostly has only one meaning within a particular domain.

1953

Oswald and Lawson propose **microglossaries** for machine translation. These are glossaries assembled for a particular domain. Through the 1950s, researchers produce many such domain-specific glossaries to aid machine translation. For example, a microglossary for mathematics would define 'triangle' as a geometric shape and not as a musical instrument.

1955

Erwin Reifler defines what he calls *semantic coincidences* between a word and its context. He also notes that **syntactic relations** can be used to disambiguate. For example, the word 'kept' can have an object that's gerund (He kept eating), adjectival phrase (He kept calm), or noun phrase (He kept a record).

1957

Masterman makes use of synonyms, near synonyms and associated words from Roget's Thesaurus. She gives the example of "flowering plant". Using the **thesaurus**, we can determine that 'vegetable' is the only common sense for the words 'flowering' and 'plant'. This is therefore the correct sense of the word 'plant' in this context.

1965

Madhu and Lytle propose the use of what they call **Figure of Merit**. This is a probabilistic measure that's useful when grammatical structure alone is unable to disambiguate. They focus on scientific and engineering literature and identify ten groups. The group or context is determined using words with single meaning. Then the most probable meaning of words with multiple meanings is selected given the context. Paradoxically, this is also the time when interest in machine translation declines.

1970

In the 1970s, some notable approaches include **Semantic Networks** of Quillian and Simmons; **Preferential Semantics** of Wilks; word-based understanding of Riesbeck. Early semantic networks can be traced to the late 1950s.

1986

Lesk uses a **machine-readable dictionary (MRD)** for WSD. In general, the 1980s sees large-scale lexical resources (such as WordNet) for automated knowledge extraction. This is also when focus shifts from linguistic theories to empirical methods.

1990

The use of neural networks had been suggested in the early 1980s but was limited to a few words and hand-coded. Véronis and Ide extend this idea by using a machine-readable Collins English Dictionary. Network is formed using dictionary entries and words used to define them. Word nodes activate sense nodes. Feedback allows competing senses to inhibit one another.

1991

Brown et al. show that it's possible to disambiguate by aligning sentences in two languages. A word in one language might translate into different words in another language, each with a unique sense. In 1992, Gale et al. extend this idea by using Canadian Hansards (parliamentary debates) that's available in more than one language. This avoids expensive hand-labelled corpus.

1995

Supervised WSD algorithms have the problem of requiring sense-annotated corpora, which is expensive and laborious to create. Yarowsky proposes an **unsupervised** WSD algorithm. The algorithm uses two useful constraints: one sense per collocation, one sense per discourse. It's a bootstrapping procedure that seeds a small number of sense annotations. The algorithm then determines and iteratively improves on the senses for other occurrences of the word. An example is to disambiguate 'plant', which can be about plant life or a manufacturing plant.

1998

Schütze proposes a vector space approach to WSD via **clustering**. Senses are seen as clusters of similar contexts. A sense of a particular word is the cluster to which it's closest. Since the technique is unsupervised, senses are induced from a corpus. Word vectors are calculated using cooccurrences. Word vectors are sparse but context vectors are dense. The dimensions of both vectors are reduced using Singular Value Decomposition (SVD).

1999

Mihalcea and Moldovan make use of **WordNet** for WSD. They rank different senses using WordNet's semantic density for a word-pair and **web mining** for word pair cooccurrences. In 2004, Peter Turney also employs web mining to calculate cooccurrence probabilities that are used to generate semantic features for WSD.

2010

This decade sees an increasing use of word embeddings and neural network models for WSD. Some of these include Gloss-Augmented WSD (GAS), GlossBERT, and use of BiLSTM.

Mar
2018

Ramiro et al. study the evolution of word senses. They note that cognitive efficiency drives this evolution through a process called **nearest-neighbour chaining**. For new word senses, reuse of existing words (polysemy) is more common than new word forms.

Word Sense Disambiguation


# Confusion Matrix

## Summary

In statistical classification, we create algorithms or models to predict or classify data into a finite set of classes. Since models are not perfect, some data points will be classified incorrectly. Confusion matrix is basically a tabular summary showing how well the model is performing.

In one dimension, the matrix takes the actual values. The matrix then maps these to the predicted values in the other dimension. In reality, the matrix is like a histogram. The entries in the matrix are counts. For example, it records how many data points were predicted as "true" when they were actually "false".

Confusion matrix is useful in both binary classification as well as multiclass classification problems. There are many performance metrics that can be computed from the matrix. Learning these metrics is handy for a statistician or data scientist.

## Discussion

### What are the elements and terminology used in Confusion Matrix?

Let's also consider a concrete example of a pregnancy test. Based on a urine test, we predict if a person is pregnant or not. We assume that the ground truth (pregnant or not) is available to us. We therefore have four possibilities:

    + **True Positive (TP)**: We predict a pregnant person is pregnant. This is a good prediction.
    + **True Negative (TN)**: We predict a non-pregnant person is not pregnant. This is a good prediction.
    + **False Positive (FP)**: We predict a non-pregnant person is pregnant. This type of error is also called *Type I Error*.
    + **False Negative (FN)**: We predict a pregnant person is not pregnant. This type of error is also called *Type II Error*.When these are arranged in matrix form, it will be apparent that correct predictions are represented along the main diagonal. Incorrect predictions are in the non-diagonal cells. This makes it easy to see where predictions have gone wrong. We may also say that the matrix represents the model's inability to classify correctly, and hence the "confusion" in the model.

### What metrics are used for evaluating the performance of a prediction model?

Performance metrics from a confusion matrix are represented in the following equations:

$$Recall\ or\ Sensitivity=TP/(TP+FN)=TP/AllPositives\\Specificity=TN/(TN+FP)=TN/AllNegatives\\Precision=TP/(TP+FP)=TP/PredictedPositives\\Prevalence=TP+FN/Total=AllPositives /Total\\Accuracy=(TP+TN)/Total\\Error\ Rate=(FP+FN)/Total$$

It's important to understand the significance of these metrics. Accuracy is an overall measure of correct prediction, regardless of the class (positive or negative). The complement of accuracy is error rate or misclassification rate.

High recall implies that very few positives are misclassified as negatives. High precision implies very few negatives are misclassified as positives. There's a trade-off here. If model is partial towards positives, we'll end up with high recall but low precision. It model favours negatives, we'll end up with low recall and high precision.

High specificity, like high precision, implies that very few negatives are misclassified as positives. If positive represents some disease, specificity is the model's confidence in clearing a person as disease-free. Selectivity is the model's confidence in diagnosing a person as diseased.

Ideally, recall, specificity, precision and accuracy should all be close to 1. FNR, FPR and error rate should be close to 0.

### Could you give a numerical example showing calculations of performance measures of a prediction model?

This example has 165 samples. We show the following calculations:

    + Recall or True Positive Rate (TPR): TP/(TP+FN) = 100/(100+5) = 0.95
    + False Negative Rate (FNR): 1 - TPR = 0.05
    + Specificity or True Negative Rate (TNR): TN/(TN+FP) = 50/(50+10) = 0.17
    + False Positive Rate (FPR): 1 - TNR = 0.83
    + Precision: TP/(TP+FP) = 100/(100+10) = 0.91
    + Prevalence: (TP+FN)/Total = (100+5)/165 = 0.64
    + Accuracy: (TP+TN)/Total = (100+50)/165 = 0.91
    + Error Rate: (FP+FN)/Total = (10+5)/165 = 0.09

### Why do we need so many performance measures when accuracy can be sufficient?

If the dataset has 90% positives, then achieving 90% accuracy is easy by predicting only positives. Thus, accuracy is not a sufficient measure when dataset is imbalanced. Accuracy also doesn't differentiate between Type I (False Positive) and Type II (False Negative) errors. This is where the confusion matrix gives us more useful measures with FPR and FNR; or their complementary measures, Recall and Specificity respectively.

Consider the multiclass problem of iris classification that has three classes: setosa, versicolor and virginica. This has an accuracy of 84% (32/38) but it doesn't tell us where the errors are happening. With the confusion matrix, it's easy to see that only versicolor is wrongly classified. The matrix also shows that versicolor is misclassified as virginica and never as setosa. We can also see that Recall is 62% (10/16) for versicolor.

In fact, when classes are not evenly represented in the data, confusion matrix by itself doesn't give an adequate visual representation. For this reason, we use a **normalized confusion matrix** that takes care of class imbalance.

### What are other performance metrics for a classification/prediction problem?

**F-measure** takes a harmonic mean of Recall and Precision, (2\*Recall\*Precision)/(Recall+Precision). It's a value closer to the smaller of the two. Applying this to our earlier example, we get F-measure = (2\*0.95\*0.91)/(0.95+0.91) = 0.92

A commonly used graphical measure is the **ROC Curve**. It's generated by plotting the True Positive Rate (y-axis) against the False Positive Rate (x-axis) as we vary the threshold for assigning observations to a given class.

How often will we be wrong if we always predict the majority class? **Null Error Rate** gives us a measure for this. It's a useful baseline when evaluating a model. In our example, null error rate would be 60/165 = 0.36. If the model always predicted positive, it would be wrong 36% of the time.

**Cohen's Kappa** can be applied to know how well a classifier is performing as opposed to classifying simply by chance. A high Kappa score implies accuracy differs a lot from null error rate.

### What's the procedure to make or use a Confusion Matrix?

We certainly need both the actual values and the predicted values. We can arrange the actual values by rows and the predicted values by columns, although some may swap the two. It's therefore important read the arrangement of the matrix correctly. For each actual value, count the number of predicted values for each class. Fill these counts into the matrix.

There's no threshold for good accuracy, sensitivity or other measures. They should be interpreted in the context of problem, domain and business.

### Could you mention some tools and techniques in relation to the Confusion Matrix?

In R, package *caret: Classification and Regression Training* can be used to get confusion matrix with all relevant statistical information. The function is `confusionMatrix(data=predicted, reference=expected)`. This plots actuals (called reference) by columns and predictions by rows.

In Python, package *sklearn.metrics* has an equivalent function, `confusion_matrix(actual, predicted)`.  This plots actuals by rows and predictions by columns. Other related and useful functions are `accuracy_score(actual, predicted`) and `classification_report(actual, predicted)`.

## Milestones

1904

Mathematician Karl Pearson publishes a paper titled *On the theory of contingency and its relation to association and normal correlation*. Contingency and correlation between two variables can be seen as the genesis of confusion matrix.

1971

James Townsend publishes a paper titled *Theoretical analysis of an alphabetic confusion matrix*. Uppercase English alphabets are shown to human participants who try to identify them. Alphabets are presented with or without introduced noise. The resulting confusion matrix is of size 26x26. With noise, Townsend finds that 'W' is misidentified as 'V' 37% of the time; 32% of 'Q' are misidentified as 'O'; 'H' is identified correctly only 19% of the time.

1998

The term **Confusion Matrix** becomes popular in the ML community when it appears in a glossary featured in *Special Issue on Applications of Machine Learning and the Knowledge Discovery Process* by Ron Kohavi and Foster Provost.

2011

In a paper titled *Comparing Multi-class Classifiers: On the Similarity of Confusion Matrices for Predictive Toxicology Applications*, researchers show how to compare predictive models based on their confusion matrices. For lower FNR, they propose regrouping performance measures of multiclass classifiers into a binary classification problem.

Confusion Matrix


# Neural Networks for NLP

## Summary

The use of statistics in NLP started in the 1980s and heralded the birth of what we called **Statistical NLP** or **Computational Linguistics**. Since then, many machine learning techniques have been applied to NLP. These include naïve Bayes, k-nearest neighbours, hidden Markov models, conditional random fields, decision trees, random forests, and support vector machines.

The use of neutral networks for NLP did not start until the early 2000s. But by the end of 2010s, neural networks transformed NLP, enhancing or even replacing earlier techniques. This has been made possible because we now have more data to train neural network models and more powerful computing systems to do so.

In traditional NLP, features were often hand-crafted, incomplete, and time consuming to create. Neural networks can learn multilevel features automatically. They also give better results.

## Discussion

### Which are the main innovations in the application of NN to NLP?

Two main innovations have enabled the use of neural networks in NLP:

    + **Word Embeddings**: This enabled us to represent words as real-valued vectors. Instead of having a sparse representation, word embeddings allowed us to represent words in a much smaller dimensional space. We could identify similar words due to their closeness in this vector space, or use analogies to exploit semantic relationships between words.
    + **NN Architectures**: These had evolved in other domains such as computer vision and were adapted to NLP. This started in language modelling, and later applied to morphology, POS tagging, coreference resolution, parsing, and semantics. From these core areas, neural networks were applied to applications: sentiment analysis, speech recognition, information retrieval/extraction, text classification/generation, summarization, question answering, and machine translation. These architectures are usually not as deep (many hidden layers) as found in computer vision.

### Which are the NN architectures that have been used for NLP?

Early language models used a feedforward NN or convolutional NN architectures but these didn't capture context very well. Context is how one word occurs in relation to surrounding words in the sentence. To capture context, recurrent NNs were applied. **LSTM**, a variant of RNN, was then used to capture long-distance context. **Bidirectional LSTM (BiLSTM)** improves upon LSTM by looking at word sequences in forward and backward directions.

Typically, the dimensionality of input and output must be known and fixed. This is problematic for machine translation. For example, the best translation of a 10-word English sentence might be a 12-word French sentence. This problem is solved by a **sequence-to-sequence** model that's based on **encoder-decoder** architecture. The essence of the encoder is to encode an entire input sequence into a large fixed-dimensional vector, called the **context vector**. The decoder implements a language model conditioned on the input sequence.

To encode contextual information in a single context vector is difficult. This gave rise to the idea of **attention** where more information is given to decoder. From here, the **transformer** model evolved.

### What's been the general trend in NLP research with neural networks?

Language modelling has been essential for the progress of NLP. Because of the ready availability of text, it's been easy to train complex models in an unsupervised manner on lots of training data. The intent is to train the model to learn about words and the contexts in which they occur. For example, the model should learn a vector representation of "bank" and also discriminate between a river bank and a financial institution.

A **pretrained language model**, first proposed in 2015, can save us expensive training on vast amounts of data. However, such a pretrained model may need some amount of training on domain-specific data. Then the model can be applied to many downstream NLP tasks. This approach is similar to pretrained word embeddings that didn't capture context.

The use of a pretrained language model in another downstream task is called **transfer learning**, a concept that's also common in computer vision.

It's expected that transformer model will dominate over RNN. Pretrained models will get better. It'll be easier to fine tune models. Transfer learning will become more important.

### Could you share some real-world examples of NN in NLP?

In 2018, Google introduced Smart Compose in Gmail. A seq2seq model using email subject and previous email body gave good results but failed to meet latency constraints. They finally settled on a hybrid of bag-of-words (BoW) and RNN-LM. Average embeddings are fed to RNN-LM.

At Amazon, they've used a lightweight version of ELMo to augment Alexa functions. While ELMo uses a stack of BiLSTM, they use a single layer since Alexa transactions are linguistically more uniform. They trained the embeddings in an unsupervised manner, and then trained on two tasks (intent classification and slot tagging) in a supervised manner while only slowly adjusting the embeddings. They also did transfer learning on new tasks.

Uber has used NLP to filter tickets related to map data. Using word2vec, they trained word embeddings on one million tickets. This had the limitation that all words are treated equally. They then experimented with WordCNN and LSTM networks. They got best results with word2vec trained on customer tickets and used it with WordCNN. For future work, they suggested character-level (CharCNN) embeddings that are more resilient to typos.

## Milestones

2001

Bengio et al. point out the **curse of dimensionality** where the large vocabulary size of natural languages makes computations difficult. They propose a **feedforward neural network** that jointly learns the language model and vector representations of words. They refine their methods in a follow-up paper from 2003.

2008

Collobert and Weston train a language model in an unsupervised manner from Wikipedia data. They use supervised training for both syntactic tasks (POS tagging, chunking, parsing) and semantic tasks (named entity recognition, semantic role labelling, word sense disambiguation). To model long-distance dependencies, they use a **Time-Delay Neural Network (TNN)** inspired from CNN. They use multiple layers to move from local features to global features. Moreover, two models can share word embeddings, an approach called **multitask learning**.

2010

Mikolov et al. use a **recurrent neural network (RNN)** for language modelling and apply this for speech recognition. They show better results than traditional n-gram models.

2012

Dahl et al. combine deep neural network with hidden Markov model (HMM) for large vocabulary speech recognition.

2013

Going beyond just word embeddings, Kalchbrenner and Blunsom map an entire input sentence to a vector. They use this for machine translation without relying on alignments or phrasal translation units. In another research, **LSTM** is found to capture long-range context and therefore suitable for generating sequences. In general, 2013 is the year when there's research focus on using CNN, RNN/LSTM and recursive NN for NLP.

2014

Using pretrained word2vec embeddings, Yoon Kim uses CNN for sentence classification. Also in 2014, Sutskever et al. at Google apply **sequence-to-sequence** model to the task of machine translation. They use separate 4-layered LSTMs for encoder and decoder. Reversing the order of source sentences allows LSTM to exploit short-term dependencies and therefore do well on long sentences. Seq2seq models are suited for NLG tasks such as captioning images or describing source code changes.

Sep
2014

Bahdanau et al. apply the concept of **attention** to the seq2seq model used in machine translation. This helps the decoder to "pay attention" to important parts of the source sentence. It doesn't force the encoder to pack all information into a single context vector. Effectively, the model does a soft alignment of input to output words.

Nov
2015

Dai and Le propose a two-step procedure of unsupervised pre-training followed by supervised training for text classification. This **semi-supervised** approach works well. Pre-training helps to initialize the model for supervised training and generalization. More unlabelled data during pre-training is seen to improve supervised learning. In later years, this approach becomes important.

Sep
2016

Google replaces its phrase-based translation system with **Neural Machine Translation (NMT)**. This reduces translation errors by 60%. It uses a deep LSTM network with 8 encoder and 8 decoder layers. The first layer of encoder is BiLSTM. The model also uses residual connections among the LSTM layers.

May
2017

Recurrent architectures can't be parallelized due to their sequential nature. Gehring et al. therefore propose using CNNs for seq2seq modelling since CNNs can be parallelized and make best use of GPU hardware. The model uses gated linear units, residual connection and attention in each decoder layer.

Dec
2017

Vaswani et al. propose the **transformer** model in which they use a seq2seq model without using RNN. The transformer model relies only on **self-attention**. By 2018, the transformer leads to state-of-the-art models such as **OpenAI GPT** and **BERT**.

2018

Researchers at the Allen Institute for Artificial Intelligence introduce **ELMo (Embeddings from Language Models)**. While earlier work derived contextualized word vectors, this was limited to the top LSTM layer. ELMo's word representations use all layers of a bidirectional language model. This allows ELMo to model syntax, semantics and polysemy. Such a language model can be pretrained on a large scale and then used for a number of downstream tasks.

Feb
2019

**OpenAI GPT-2** shows its power in natural language generation. Trained on 8 million websites, it has 1.5 billion parameters. The model is initially not released to the public due to concerns of misuse (such as fake news generation). However, in November 2019, GPT-2 is released.

Neural Networks for NLP


# Probabilistic Neural Network

## Summary

A Probabilistic Neural Network (PNN) is a feed-forward neural network in which connections between nodes don't form a cycle. It's a classifier that can estimate the probability density function of a given set of data. PNN estimates the probability of a sample being part of a learned category. Machine learning engineers use PNN for classification and pattern recognition tasks. A PNN is designed to solve classification problems by using a statistical memory-based approach that can be supervised or unsupervised.

The probabilistic neural net is based on the idea of conventional probability theory, such as Bayesian classification and other estimators for probability density functions, to construct a neural net for classification. The widespread use of PNN originated from the usage of kernel functions for discriminant analysis and pattern recognition.

## Discussion

### What is the architecture of a PNN?

The PNN architecture has four layers:

    + **Input Layer**: \(p\) neurons represent the input vector and distribute it to the next layer. \(p\) equals the number of input features.
    + **Pattern Layer**: This layer applies the kernel to the input. It organizes the learning set by representing each training vector by a hidden neuron that records the features of this vector. During inference, each neuron calculates the Euclidean distance between the input test vector and the training sample, then applies the radial basis kernel function. In this way, it encodes the PDF centered on each training sample or pattern. For class \(j\), we have \(n\_j\) neurons, for \(j\) in \([1,m]\).
    + **Summation Layer**: This layer computes the average of the output of the pattern units for each class. There's one neuron for each class. Each class neuron is connected to all neurons in the pattern layer of that class.
    + **Output Layer**: This layer selects the maximum value from the summation layer, and the associated class label is determined accordingly.

### Could you explain PNN with a simple example?

Consider the task of classifying the letters O, X, and I. The characters can be in uppercase or lowercase. We consider two features: length and area of each character. Consequently, the training set will have 6 letters `(O,o,X,x,I,i)`. Each training data point will be identified with a `(length, area)` value. For example, `O(0.5,0.7)`, `o(0.2,0.5)`, `X(0.8,0.8)`, `x(0.4,0.5)`, `I(0.6,0.3)` and `i(0.3,0.2)`.

The input layer of the PNN will have two neurons, one for each feature, that is, one node for length and one for area.

We have three classes. Each class has two patterns in the pattern layer, one for uppercase and one for lowercase. For example, for class O there are two subtypes (O,o). In total, the pattern layer has six neurons.

The summation layer will calculate the average value for each pattern type of the pattern layer and output layer will pick the maximum value, thereby determining the suitable class O, X, I.

An advantage of PNN is that there is no back-propagation training. New pattern units can be added without additional time overhead, since no training is needed; it is automatic.

### What are the concepts from which PNN was derived?

The **Parzen window** density estimation, or the Kernel Density Estimation (KDE), is a non-parametric density estimation technique. It's used to derive a density function \(f(x)\). When we have a new training sample \(x\) and there's a need to compute the value of the likelihoods, \(f(x)\) is used. \(f(x)\) takes the sample input data value and returns the density estimate of the given data sample. This doesn't require any knowledge about the underlying distribution and is also used for classification.

Parzen windows are seen as a generalization of **k-Nearest Neighbour (KNN)** techniques. Rather than choosing k nearest neighbours of a test point and labelling the test point with the weighted majority of its neighbours' votes, one can consider all points in the voting scheme and assign their weights by using kernel function.

KNN is a non-parametric algorithm based on supervised learning. This is used for classification and regression. The KNN algorithm assumes that similar things exist in close proximity. It considers k nearest neighbours (data points) to predict the class or continuous value for the new data point.

### How does PNN work?

The input layer transmits the characteristics of the sample to the network, specifically the pattern layer. The number of input neurons are the same as dimensions of the sample.

For the pattern layer, the Euclidean distance between the feature vector of the training sample \(X\) and radial center \(x\_{ij}\) realizes matching between the input feature vector and various types in training set. Here, \(X=[x\_1, x\_2, … , x\_n] \cdot T\), for \(n\) in \([1 .. l]\), and \(l\) represents all types of training, \(d\) is the dimension of eigenvector, \(x\_{ij}\) is the j-th center of the i-th training sample, and σ is a smoothing factor. The pattern layer also shows \(m\) different classes. Among the \(l\) neurons, each one belongs to exactly one class.

\(v\_i\) is the output for class \(i\) in the summation layer. \(L\) is the number of class \(i\) neurons. The type corresponding to maximum output in the summation layer is the output type of the output layer, given by\(Type(v\_i) = arg max(v\_i)\).

### How do I select the right smoothing parameter for a PNN?

Particularly when the training dataset is limited, performance of a PNN depends on the right selection of the smoothing parameter σ. Small σ creates a multimodal distribution. Larger σ leads to interpolation between points. Very large σ approaches Gaussian PDF. Intuitively, σ should depend on the density of the samples.

The simplest technique is to use the standard deviation of training samples for each dimension or feature. Cross-validation (training vs validation datasets) gives better generalization. Clustering is another technique. In **gap-based estimation**, Zhong et al. improved on these techniques by modelling the distances between a training sample and its neighbours. They estimated σ per input feature, noting that estimating σ per feature per class is not as good.

**Genetic algorithms** have been used to estimate σ.  In R language, `pnn` package uses a genetic algorithm from `rgenoud` package to estimate σ.

Kusy and Zajdel studied three techniques from **reinforcement learning**: Q(0)-learning, Q(λ)-learning, and stateless Q-learning. Results were similar to state-of-the-art performance of alternative approaches.

### What are some of the variations of the traditional PNN?

**Enhanced PNN (EPNN)** uses Local Decision Circles (LDCs) that enable incorporation of local information and non-homogeneity existing in the training population. The circle has a radius that limits the contribution of the local decision. The two Bayesian rules used by EPNN are: (a) A global rule that estimates the conditional probability of each class, given an input vector of data considering all training data and using the spread parameter. (b) A local rule that estimates the conditional probability of each class, given an input vector of data existing within a decision circle, considering only the training data.

**Competitive PNN (CPNN)** adds novel competitive features to EPNN to utilize data most critical to the classification process. A competitive layer ranks kernels for each class and an optimum fraction of kernels are selected to estimate the class-conditional probability.

**Supervised Learning PNN (SLPNN)** has three kinds of network parameters that are adjusted through training: variable weights representing the importance of input variables, reciprocal of kernel radius representing the effective range of data, and data weights representing the data reliability.

### What are some specializations of PNN?

Often researchers attempt to change the PNN's structure so that it becomes more practical to implement the pattern layer even with large number of training samples. Type of data is also a motivation to specialize. We note a few of these.

Zaknich et al. applied PNN for **time series analysis**. Their architecture was modified so that current output depends on current input, preceding five inputs and following five inputs. Thus, their input vector had 11 coefficients. Tested on sinusoidal signals, attenuated and then corrupted with noise, the PNN network produced a smoothened output. Performance improved when the model contained more classes.

**Interval PNN** is a PNN that classifies interval data. As is common in practical applications, the training dataset may be accurate but test data contains less precise measurements. This imprecision is handled by the model using intervals.

### What are the applications based on PNN?

PNN's main applications include classifying labelled stationary data patterns or patterns with time-varying PDF. In signal processing, PNN considers waveforms as patterns and thereby recognizes specific events and their severity. In one example, PNN was used to recognize 11 types of disturbances in power quality using waveforms of voltage magnitude, frequency, and phase.

PNN is applied to pattern recognition problems such as character/object/face recognition. PNN brings flexibility, straightforward design and minimal training time.

For text-independent speaker identification, PNN provides good results in matching speaker for each input vector. To increase the success rate, multiple input vectors from each sample are needed.

The figure shows PNN applied to ship identification. Even with a noisy image has been used as input of neural network, PNN performs well. Covariance matrix of discrete wavelet transform of ship image is used as input.

PNN is used for overcoming the computational complexity involved in performing sensor configuration management in a wireless ad-hoc network.

### What are the pros and cons of PNN?

In a PNN, there's no extensive training computation time associated with networks that use back-propagation. Instead, each data pattern is represented with a unit that measures the similarity of the input pattern to the data patterns. PNN learns from the training data instantaneously. With this speed of learning, PNN has the capability to adapt its learning in real time, deleting or adding training data as new conditions arise. Additionally, PNNs are relatively insensitive to outliers and approach Bayes optimal classification as the number of training samples increases. PNNs are guaranteed to converge to an optimal classifier as the size of the representative training set increases.

However, PNN has its limitations. Because there's one hidden node for each training instance, more computational resources (storage and time) during inference.  Additionally, the performance of the system usually decreases in terms of the classification accuracy and speed with a very big hidden layer.

### Which are the main research approaches to improve PNN performance?

To reduce expensive computational times and storage requirements of a full Parzen window classifier, a weighted-Parzen window classifier is an option. A **clustering** procedure is used to find a set of reference vectors and weights that are used to approximate the Parzen window (kernel estimator) classifier. For clustering, even k-means algorithm can be applied. The basis is that not all the patterns contain original, independent, and discriminating information. Thus, clustering can reduce the number of neurons in the pattern layer.

When input samples have too many features, **Principal Component Analysis (PCA)** can be used to reduce the number of features.

For optimization and alteration of spread parameters in PNNs, in **heteroscedastic PNN** (hetero - different, skedasis - dispersion), the kernels of each class are allowed to have their own spread parameter matrix.

The issue of data heterogeneity and noisy datasets are addressed by **EPNN**, by implementing Local Decision Circles (LDCs) to modify the spread parameter of each training vector and bi-level optimization to find the optimal value of spread parameter and radius of LDCs.

## Milestones

1962

Parzen discusses the problem of estimation of a PDF and the problem of determining the mode of a PDF. He also relates the similarity of the problem of estimating a PDF to the problem of estimating the spectral density function of a stationary time series. While the problem of estimating the mode of a PDF is almost similar to the problem of maximum likelihood estimation of a parameter.

1966

In an attempt to perform classification for pattern recognition, Specht uses a Bayes strategy to merely transform the problem to one of estimating PDFs for each of the possible categories on the basis of training samples available. This is accomplished with an estimator which a) is shown to be consistent (tends to be identical with the true density in the limit as the number of training samples is increased to infinity and b} can be expressed in terms of a polynomial, the coefficients of which can be computed on a one-pattern-at-a-time basis.

1990

Specht introduces the term Probabilistic Neural Network, for a neural network that replaces the sigmoid activation function often used in neural networks with an exponential function. A PNN can compute nonlinear decision boundaries which approach the Bayes optimal. Architecturally, the neural network is designed to have four layers that can map any input pattern to any number of classifications. This technique offers a tremendous speed advantage for problems in which the incremental adaptation time of back propagation is a significant fraction of the total computation time.

1991

In order to compensate the flaw of PNN of not being robust with respect to affine transformations of feature space, leading to poor performance on certain data, a weighted PNN (WPNN) is derived. This allows anisotropic Gaussians, i.e. Gaussians whose covariance is not a multiple of identity matrix.

2000

Mao et al. propose two improvements to the PNN: select a suitable smoothing parameter using a genetic algorithm and then select a representative set of pattern layer neurons from the training samples using Forward Regression Orthogonal Algorithm. A similar research was published independently by Chen et al. in September 1999.

2007

A modified PNN for brain tissue segmentation with MRI is proposed. Here, covariance matrices are used to replace the singular smoothing factor in the PNN's kernel function, and weighting factors are added in the pattern of summation layer. This weighted PNN (WPNN) classifier can account for partial volume effects that exist commonly in MRI, not only in the final result stage, but also in the modelling process.

2010

An enhanced and generalized PNN (EPNN) is proposed using local decision circles (LDCs) to overcome the shortcoming of PNN wherein it doesn't consider probable local densities or heterogeneity in training data. Also, EPNN improves PNN's robustness to noise in data.

2011

A Supervised Learning PNN (SLPNN) is proposed with three kinds of network parameters that can be adjusted through training. The SLPNN is slightly more accurate than MLP and much more accurate than PNN.

2016

Considering that spread has a great influence on PNN's performance, a self-adaptive PNN (SaPNN) is proposed. In this, spread can be self-adaptively adjusted and selected and then the best selected spread is used to guide the SaPNN train and test. This SaPNN has a more accurate prediction and better generalization performance as compared to basic PNN.

2016

A modified PNN (MPNN) is introduced which is an extension of PNN with the weight coefficients introduced between pattern and summation layer of the model. These weights are calculated by using the sensitivity analysis procedure. MPNN improves the prediction ability of the PNN classifier.

2017

A Competitive PNN (CPNN) is presented wherein a competitive layer ranks kernels for each class and an optimum fraction of kernels are selected to estimate the class-conditional probability. Performance percentage of CPNN is found to be greater than or equivalent to that of traditional PNN.

Probabilistic Neural Network


# Richardson Maturity Model

## Summary

Consider remote web servers as huge repositories of data. Client applications can access them via APIs. It could be public data such as weather forecasts, stock prices, or sports updates. Data could be private such as company-specific business information that's accessible to employees, vendors or partners.

REST (REpresentational State Transfer) is a popular architectural style for designing web services to fetch or alter remote data. APIs conforming to the REST framework are considered more mature because they offer ease, flexibility and interoperability. **Richardson Maturity Model (RMM)** is a four-level scale that indicates extent of API conformity to the REST framework.

The maturity of a service is based on three factors in this model: URI, HTTP Methods and HATEOAS (Hypermedia). If a service employs these technologies, it’s considered more mature. This model covers only API architectural style, not data modeling or other design factors.

## Discussion

### What are the different levels in the Richardson Maturity Model?

RMM has four levels of maturity, from the lowest to highest:

    + **Level-0: Swamp of POX**: Least conforming to REST architecture style. Usually exposes just one URI for the entire application. Uses HTTP POST for all actions, even for data fetch. SOAP or XML-RPC-based applications come under this level. POX stands for *Plain Old XML*.
    + **Level-1: Resource-Based Address/URI**: These services employ multiple URIs, unlike in Level 0. However, they use only HTTP POST for all operations. Every resource is identifiable by its own unique URI, including nested resources. This resource-based addressing can be considered the starting point for APIs being RESTful.
    + **Level-2: HTTP Verbs**: APIs at this level fully utilize all HTTP commands or verbs such as GET, POST, PUT, and DELETE. The request body doesn’t carry the operation information at this level. The return codes are also properly used so that clients can check for errors.
    + **Level-3: HyperMedia/HATEOAS**: Most mature level that uses *HATEOAS (Hypertext As The Engine Of Application State)*. It's also known as HyperMedia, which basically consists of resource links and forms. Establishing connections between resources becomes easy as they don’t require human intervention and aids client-driven automation.

### Could you illustrate the RMM levels with an example?

Consider the example of creating, querying, and updating employee data, which is included in the Sample Code section.

At Level 0, an employee's last name would be used to get employee details. Though just a query, HTTP GET is not used instead of HTTP POST. Response body would need to be searched to obtain the employee details. If there's more than one employee with the same last name, multiple records are returned in proprietary format.

At Level 1, specificity is improved by querying with employee ID rather than with last name. Each employee is uniquely identifiable. This implies that each employee has a unique URI, and therefore is a uniquely addressable resource in REST.

While Level 1 uses only POST, Level 2 improves upon this design by using all the HTTP verbs. Query operations use GET, update operations use PUT and add operations use POST. This also ensures sanctity of employee data. Appropriate error codes are returned at HTTP protocol layer. This implies that we don't need dig into response body to figure out errors.

At Level 3, response about an employee includes unique hyperlinks to access that employee's records.

### What are the characteristics of RMM Level 0?

API at this level usually consists of functions passing the server object and the name of the interface to the server object. This is sent to the server in XML format using the HTTP post method. The server analyses the request and sends the result to the client, also in XML. If the operation fails, some sort of error message will be in the reply body.

At Level 0, you can do an actual object transfer between the client and server, not just its representation state (as in REST). Object data can be in standard formats (JSON, YAML, etc.) or in custom formats. If it’s SOAP, the specifications are in an accompanying WSDL file.

HTTP is used in a very primitive manner, just as a tunnelling mechanism for the client’s own remote operations. For that matter, other application layer protocols such as FTP or SMTP can also be used.

### What are the characteristics of RMM Level 1?

Level 1 employs several URIs, each of which leads to a specific resource and acts as an entry point to the server side. This aspect is common with Levels 2 and 3.

Requests are delivered to the specific resource in question, rather than simply exchanging data between each individual service endpoint. While this seems like a small difference, in terms of function it’s a critical change. The identity of specific objects is established and invoked. Only arguments related to its function and form are passed to each object/resource.

From this level onwards, it is always HTTP as the application layer protocol. The key non-conformity to REST in this level is disregarding the semantics of HTTP. Only HTTP POST is used for all CRUD (Create, Read, Update, Delete) operations. The return status and data retrieved are all in the body of the HTTP response.

### What are the characteristics of RMM Level 2?

The emphasis in this level is on the correct usage of the various HTTP verbs, especially the use of GET and POST commands. By restricting all fetch operations to GET, an application can guarantee safety and sanctity of the data on the server side. Caching of data at client side for quicker access becomes possible too.

The model calls this level *HTTP Verbs*. It implies that only HTTP can be used. Actually, REST architectural pattern doesn't mandate the use of HTTP. It's entirely protocol neutral.

At the server side, the 200 series of HTTP response codes should be used properly to indicate success, and the 400 series to represent different types of error.

Taken together, levels 1 and 2 clarify at a high level (such as in server logs) what resources are being accessed, for what purpose, and what happened.

The non-conformity here is quite subtle. The API at this level lacks the transparency for the client to navigate the application state without some external knowledge, generally present in supplied documentation.

### What are the characteristics of RMM Level 3?

In order to ensure complete conformance to RESTful architecture, the final hurdle is to provide smooth navigation for the client throughout the application. This is supported through Hypermedia controls.

The HTTP responses include one or more logical links for the resources that the API accesses. That way, the client doesn’t have to rely on external documentation to interpret the content. Because the responses are self-explanatory, it encourages easy discoverability. This is done using **HATEOAS (Hypertext as the Engine of Application State)**, which forms a dynamic interface between the server and client.

One obvious benefit is that it allows the server to change its URI scheme without breaking clients. As long as the client-facing URI is maintained intact, the server can juggle around its resources and contents internally without impacting the client. Service updates in the application become seamless without service disruption.

It also allows the server team to inform the client of new features by putting new links in the responses that the client can discover.

### What sort of applications can still be designed with Level 0 or Level 1 APIs?

Plenty of online services belong to RMM Level 0 or 1 such as Google Search Service (now deprecated) and Flash-based websites. However, these are slowly being phased out.

If you are designing a monolithic service that performs only one major function, then probably a Level 0 API would suffice as there is no need for multiple URIs. Level 0 is adequate for a service of short-term purpose that's not meant to be extended or upgraded. Think of an exam result declaration website as an example. Its use is restricted to a few days when the results are out. After that, it would just become inactive and irrelevant. There is just one function that the web service performs: it returns an individual student’s pass/fail state. If a few resources are to be accessed, then Level 1 can be employed.

In cases where HTTP is not desired as the transport mechanism, then Level 0 APIs written using SOAP can work even with alternatives like FTP or SMTP.

### When does it become necessary to have APIs designed at Level 2 or Level 3?

Web products and solutions are increasingly being deployed using the SaaS distribution model. To support this, design and manufacturing models are also service-oriented (SOA). They are exposed as microservices, a loosely coupled collection of services.

Popular cloud-based subscriptions such as mobile services, streaming web content, office automation products are all deployed in this fashion. Such services are essentially designed with APIs conforming to the Levels 2 and 3 of the model.

Applications choose Level 2 or 3 due to some additional factors – (1) Need to work even with limited bandwidth and computing resources (2) Rely on caching for greater performance (3) Stateless operations

Lower level APIs like SOAP can be compared to an envelope. It has content inside, address written outside. Content is obscured from public view. However, higher levels that conform to REST are like a postcard. Entire content is directly visible. Though this aspect brings a lot of transparency and discoverability, there is a security threat perception associated with the data. Applications take care of this aspect using security features such as in the Spring framework.

### What are the other models used to judge the design standards of remote application interfaces?

Richardson Maturity Model classifies applications with varying levels of API maturity based on conformance to REST framework.

There is another model called the **Amundsen Maturity Model**, which classifies APIs based on their data model abstraction. At higher levels of this model, the API is more decoupled from the internal models or implementation details.

## Milestones

Mar
1998

Dave Winer introduces **XML-RPC**, a lightweight rival of SOAP. SOAP development is held up due to several disagreements. Hence, XML-RPC comes out first. About this time (February 1998), W3C releases **XML Version 1.0**.

May
2000

**Simple Object Access Protocol (SOAP)**, originally developed at Microsoft, becomes a W3C recommendation. It marks the beginning for web services between clients and remote web servers. SOAP is widely adopted by most emerging companies like Salesforce, Ebay and Yahoo.

2000

Roy Fielding, unhappy with SOAP implementation comes out with a protocol neutral architectural style called **REST**. This is published as his PhD dissertation at UC Irvine. REST is just a set of design principles at this stage.

2003

Amazon S3, a file hosting service, releases APIs with support for URI and HTTP verbs. But the design doesn't provide resource links. Instead, resources are marked by unique key-value pairs.

2004

Adobe, Netflix and some other major service providers adopt web services design, supporting hypermedia links in their responses. This open API design is greatly appreciated.

2008

After analysing hundreds of web service designs, Leonard Richardson comes up with a model that helps to distinguish between good and bad designs. His yardstick is purely based on API maturity. This model is popularly referred to as **Richardson Maturity Model (RMM)**.

2015

News aggregators and media companies like The Guardian and Wall Street Journal upgrade their web services for better REST conformity. Now their services are compliant with Level 3 of the Richardson Maturity Model, with hypermedia controls.

2016

Rodríguez et al. publish results of a study on HTTP traffic from an Italian mobile internet provider's network. They note most APIs conform to RMM Level 2, implying a focus on providing CRUD access to individual resources. Even when some APIs are at higher levels of maturity, there's wide variation in the adoption of best practices. For example, use of trailing forward slash in URI is not recommended. Use of lowercase and hyphens (rather than underscores) in URI is recommended.

2018

Adobe releases Hypermedia driven UI framework for remote web services called Granite UI. Granite UI is an umbrella name for UI projects to build a LEGO® architecture for UI, so that one can build complex web applications with much less code.

Richardson Maturity Model