# Postel's Law

## Summary

Postel's Law was formulated by Jon Postel, an early pioneer of the Internet. The Law was really a guideline for creators of software protocols. Computers used protocols to communicate with one another on the Internet. The idea was that different implementations of the protocol should interoperate. The law is today paraphrased as follows:

> Be liberal in what you accept, and conservative in what you send.

Although first stated with reference to TCP/IP, the Law has been applied in other areas, from the parsing of HTML to the acceptance of user inputs. The growth and success of the Internet has been attributed in part to this Law. However, its use in modern systems has been questioned. It's been suggested that following the Law is harmful from the perspective of maintainability, compatibility and security.

Postel's Law is also known as the *Robustness Principle*.

## Discussion

### Could you explain Postel's Law?

The spirit of Postel's Law is to make different implementations interoperate. There are two parts to the Law and these are briefly explained in RFC 760:

    + Conservative in sending: Protocol should be careful to send well-formed datagrams.
    + Liberal in receiving: Protocol should accept any datagram that it can interpret. In other words, if the semantics are clear then errors in syntax can be overlooked.Herb Bowie calls it, "a general social code for software: be strict in your own behavior, but tolerant of harmless quirks in others."

In a more general sense, the Law is about reducing variations in input and giving a more well-defined and easily understood output. Robustness comes because chaos at the input is managed to result in a cleaner output.

### Can you give examples of Postel's Law in action?

Here are some examples where the Law allows the system to continue functioning rather than break down with a fatal error:

    + If an incorrect type is received, cast it to the correct type.
    + If users enter whitespaces, trim them.
    + If arguments are invalid and your function is supposed to return an array, return an empty array.
    + If a closing HTML tag is expected but it's missing, close it yourself.Let's take a protocol example in which if option A is present, field X has a value; otherwise, X should be zero filled. A conservative sender will be sure to zero fill X if A is absent. Suppose the sender doesn't bother doing this. A liberal receiver will accept the packet even if X is not zero filled.

Suppose in version 2 of the protocol, option B is introduced. B is applicable when A is absent. B also sets the value of X. Due to Robustness Principle, version 1 receivers will continue to accept packets sent by version 2 senders (A absent, B present with X) although they won't support B.

### Why was Postel's Law so important for the early Internet?

The Internet grew without centralized control. When a protocol was defined, it was implemented by multiple individuals and teams. For reliable communication, these different implementations needed to understand one another. However, standards are not always unambiguous. Sometimes different interpretations arise. In such cases, strict adherence to the standard will result in protocol errors. In a distributed system, it would be hard to iron out these differences. If there's tight control, Postel's Law is not needed. But the Internet is many pieces loosely joined. This is exactly why the Law made sense.

In early Internet days, implementations were liberal in accepting non-standard inputs provided they could make sense of them and decide how to handle them. Without such a liberal approach, the growth of the Internet might have been slower than what it was. Without Postel's Law, standardization and their strict implementations would have taken longer. Implementations that interwork with one another was more important. This is partly reflected in the ethos of IETF that aims for "rough consensus and running code."

A similar thing happened with browser and HTML. The Law made sense for rapid growth of the WWW.

### In what aspects of a software system has Postel's Law been applied?

Networking protocols commonly used on the Internet (such as TCP/IP, SNMP, FTP) adhere to the Law to achieve interoperability.

When HTTP became popular in the 1990s, browsers applied the Law liberally. From a user experience perspective, it was better to parse and display as much of the content as possible rather than inform the user than the HTML page was not conforming to specifications. XML parsers on the other hand are more strict about accepting malformed or invalid XML documents. Nonetheless, some XML feed readers and aggregators from the 2000s were liberal.

The Law's been applied to design systems. When parsing user inputs, the Law's been applied to deliver better user experience. It's been used in UNIX command line utilities. APIs and microservices have also applied the Law.

### How does one apply Postel's Law to APIs and microservices ?

Different services could be running different versions and maintaining strict compatibility is a challenge. However, the use of versioning for APIs can help achieve backward compatibility and avoid the situation of being too liberal in accepting invalid requests. The Law is equally relevant for consumers of API, who can be liberal and process only those fields of the response they are interested in and ignore extra fields.

Where client implementations are not liberal with responses, the server may apply counter measures. In other words, the server responds in a less conservative manner so that client code does not break when the API is changed or upgraded later. This is an example of applying the Law selectively.

### Are there problems with or exceptions to Postel's Law?

It's been claimed that the Law has no exceptions, even when it comes to XML parsing. Others argue that XML documents have to be well formed and valid.

While the Law is good for interoperability, it leads to incompatibilities. There's no agreement about how liberal an implementation needs to be. Thus, HTML5 is a stricter specification than its predecessors.

RFC 2360 clarifies, "The sender accuses the receiver of failing to be liberal enough, and the receiver accuses the sender of not being conservative enough." One solution is to define standards in greater detail, particularly for the receiving end. It should be clear when to be liberal.

There could be such a thing as being too robust. RFC 3117 states that robustness is at odds with efficiency. Robustness Principle can create deployment problems because errors in new implementations will go undetected until they meet a not-so-liberal implementation. Since errors are tolerated, over time they will become entrenched and make implementations more complex. Thomson calls this the *Protocol Decay Hypothesis*. It's therefore better to expose these errors early on and let implementations fix them.

### Can Robustness Principle be called Resilience Principle?

Robustness is quite different from resilience. A robust system attempts to avoid failure and thus incurs design complexity. A resilient system attempts to discover problems early and focuses on recovery. Equivalently, a robust system will continue to function without changing its essential nature. A resilient system will continue to function by adapting itself.

Thus, it's proper to call Postel's Law as Robustness Principle.

## Milestones

1974

Vinton Cerf and Robert Kahn publish details of Transmission Control Program (TCP) for use in packet-switched networks. This protocol is the genesis of what later became the popular TCP/IP.

1980

IETF releases two documents, RFC 760 (Internet Protocol) and RFC 761 (Transmission Control Protocol). This is the beginning of TCP/IP. Jon Postel, editor of both RFCs, mentions in RFC 760, "an implementation should be conservative in its sending behavior, and liberal in its receiving behavior." In RFC 761, he says in a section named Robustness Principle, "be conservative in what you do, be liberal in what you accept from others."

2004

Tim Bray argues why applying the Law to the parsing of XML documents can be harmful, particularly when the document has financial data.

2011

Eric Allman publishes an article titled *The Robustness Principle Reconsidered*. He explains that the Law made sense in the early days of the Internet when folks cooperated. Today, the world is more hostile and security is a concern. The Law is still relevant but a line must be drawn between being conservative and being liberal. For example, we can be liberal and accept a packet even when the byte allocated for an optional field is not set to zero. On the other hand, we shouldn't be liberal in accepting wrongly encoded digital signatures.

2015

M. Thomson proposes an alternative to Postel's Law. He states this as follows,

> Protocol designs and implementations should be maximally strict.

Postel's Law


# 5G Spectrum

## Summary

5G spectrum spans a wide range from 410 MHz to 52600 MHz. The high end of the spectrum falls in the millimeter range, which is novel to 5G. With its high bandwidth, it enables high-throughput, ultra-low-latency applications.

Not all frequencies in this range are used by 5G. 5G defines a number of operating bands within which 5G services can be offered. These frequencies have been selected based on suitability and availability. The bands should not overlap with non-5G systems such as military, maritime, and satellite communication systems.

Both licensed and unlicensed spectrum can be used in 5G. 5G can share 4G's licensed spectrum and Wi-Fi's unlicensed spectrum.

Successful worldwide deployment of 5G requires that countries agree on a common set of bands. Good agreement was reached in November 2019 at WRC-19.

## Discussion

### Which are the main spectrum bands allocated for 5G?

The standard defines two **frequency ranges**: FR1 (410-7125 MHz) and FR2 (24250-52600 MHz).

NR **operating bands** are defined within each range. While FR1 bands are either in FDD, TDD, SDL or SUL, FR2 bands can operate only in TDD. Supplementary Downlink (SDL) and Supplementary Uplink (SUL) are modes that allow only downlink or uplink in those bands. SDL and SUL are meant to provide additional capacity.

In practice, industry looks at 5G spectrum in terms of low-band (600-700 kHz), mid-band (3-5 GHz) and high-band (26-100 GHz).  Others simply refer to FR1 as "sub-6 GHz" band and FR2 as mmWave band.

Specifically, **n78 (3300-3800 MHz) TDD band** is globally harmonized and will be the primary 5G band. It's a subset of n77 (3300-4200 MHz).

### How are the 5G bands and frequencies named or numbered?

Operating bands are named with prefix "n" to signify New Radio. In Release 16, FR1 has 49 different bands from n1 to n96. FR2 has 5 bands n257-n261.

In each band, the standard gives identifying numbers to frequencies. There are two sets of numbers:

    + **NR Absolute Radio Frequency Channel Number (NR-ARFCN)**: Used in signalling to identify reference frequencies. *Channel raster* is a subset of reference frequencies used to identify RF channel position in uplink and downlink.
    + **Global Synchronization Channel Number (GSCN)**: Used for synchronization. *Synchronization raster* is specified by GSCN and indicates frequency positions of the Synchronization Signal Block (SSB). GSCN has a coarser granularity than NR-ARFCN and therefore should enable a UE do a faster cell search.

### Could you describe the 5G NR channel bandwidth?

Each operating band may have one or more **BS channel bandwidths**. Each BS channel bandwidth supports a single RF carrier in UL or DL. Multiple **UE channel bandwidths** may be supported within the same BS channel bandwidth.

One or more resource blocks (RBs) form a UE channel bandwidth or transmission bandwidth. Each RB has 12 subcarriers. Unlike 4G where sub-carrier spacing (SCS) is fixed to 15 kHz, 5G NR allows **flexible SCS** of 15, 30 or 60 kHz (FR1); and 60 or 120 kHz (FR2).

Guardbands exist at the edges of the BS channel bandwidth. Standard specifies the minimum guardband for each valid combination of SCS and BS channel bandwidth.

BS channel bandwidth can take values 5, 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90 and 100 MHz (FR1); and 50, 100, 200 and 400 MHz (FR2). However, not all values as valid for all bands. For example, 100MHz in FR1 is valid only for bands n40, n41, n77, n78, n79 and n90.

### Could you compare FR1 and FR2 operating bands?

FR1 defines bands in the sub-6 GHz spectrum (although 7125 MHz is the maximum) and FR2 defines bands in the mmWave spectrum. Because of the higher carrier frequencies in FR2, it has a higher maximum bandwidth. Bandwidths include 5-100 MHz (FR1) and 50/100/200/400 MHz (FR2). Correspondingly, valid subcarrier spacing values are also different: 15/30/60 kHz (FR1) and 60/120 kHz (FR2). In FR2, 240 kHz subcarrier spacing is allowed for only SS/PBCH with valid bandwidths 100/200/400 MHz.

Downlink MIMO differs: 8x8 (FR1) and 2x2 (FR2). FR1 caters for macro cells, high mobility and many users. FR1 bands also experience multipath effects. Higher-order MIMO therefore enable **spatial multiplexing** and Multi-User MIMO (MU-MIMO). On the other hand, FR2 is for small cells, low mobility and a few users. Multipath effects are less pronounced than in FR1. Lower-order MIMO therefore enables **beamforming**. These differences imply that spectral efficiency is higher in FR1 than in FR2.

### What's the suitability of low, mid and high-bands for 5G services?

Lower frequencies have better range but offer lower data rates. At mmWave spectrum, we get high data rates but waves can't get through walls, trees or even glass. Thus, there's a tradeoff between coverage and speed. It's therefore good that 5G spans a wide spectrum to suit many different use cases and deployment scenarios.

Low-band spectrum can offer rural coverage. Strong indoor signals could help connect to IoT devices in smart buildings and underground parking lots.

High-band spectrum could be ideal for short-range communications in dense urban areas and within buildings. It can cater to many high-value services in retail, healthcare, entertainment, and more. They could involve AR/VR applications. Serving thousands of users in a stadium, critical IoT applications or applications that need ultra low latency are more use cases.

Mid-band spectrum is a compromise of both low-band and high-band spectra. It could cover large neighbourhoods and even an entire city but probably can't serve rural areas. Smart city services can benefit from it, such as fleet management, transit services and vehicle communications.

### What are some features that enhance the use of 5G spectrum?

It's possible for 5G to share 4G spectrum. Called **Dynamic Spectrum Sharing (DSS)**, this is an attractive option for operators who don't have licensed 5G spectrum yet. Instead of statically re-farming the spectrum, DSS allows operators to dynamically migrate more spectrum from 4G to 5G as more 5G users come into the system. DSS requires only a software upgrade.

Supported since LTE Release 10, **Carrier Aggregation (CA)** is enhanced in 5G. Multiple carriers within the same band or across bands can be combined to serve a single UE. CA allows operators to achieve better capacity, throughput and coverage in mid-band and high-band spectra. For example, low band FDD with DSS can be used in uplink for better coverage and a high band TDD in downlink for higher throughput and capacity.

Supported since LTE Release 12, **Dual Connectivity (DC)** is enhanced in 5G. DC allows a UE to connect to two different cells possibly served by different gNBs. It may be an LTE or 5G cell. Whereas user plane split happens at MAC for CA, it happens at PDCP for DC. It's also possible to combine CA and DC.

### What's the relevance of unlicensed bands for 5G?

Cellular use of unlicensed spectrum started in LTE. It's variants include LTE Unlicensed (LTE-U), Licensed Assisted Access (LAA) and MulteFire.

In Release 16, **5G NR-U** allowed the use of unlicensed spectrum. In particular, NR-U supports both license-assisted and standalone use of unlicensed spectrum. It supports 5 GHz unlicensed band used by Wi-Fi and LAA. It also opens up unlicensed spectrum in the 6 GHz band. Unlicensed spectrum in the mmWave band is being studied for Release 17.

Since licensed spectrum is limited, for very high bandwidth applications, aggregating with unlicensed spectrum is an attractive approach. Shared/unlicensed spectrum can enable local/private networks and Industrial IoT applications.

With **Anchored NR-U**, operators can employ carrier aggregation with 5G NR or dual connectivity with LTE. In urban hotspots, campuses and malls, this can deliver consistent user experience. With **Standalone NR-U**, operators will find it easier to deploy private networks.

## Milestones

Nov
2015

At the World Radiocommunication Conference 2015 (WRC-15), delegates agree on additional spectrum for International Mobile Telecommunications (IMT). This includes **C-band** that later becomes a globally harmonized spectrum for 5G.

2016

Szydelko and Dryjanski publish an evolution of spectrum from LTE to 5G. Spectrum below 6 GHz and mmWave spectrum are being considered for 5G. For CA, 700 MHz is being considered for SDL. This is standardized in Release 16 (July 2020).

Mar
2017

In the US, FCC auctions 600 MHz spectrum currently used by TV stations. This spectrum will be used for LTE and 5G, with the transition expected to take 39 months. In July 2020, it's reported that 99% of the migration is complete.

Sep
2019

Ericsson claims the world's first 5G data call using **Dynamic Spectrum Sharing (DSS)** in low band FDD. In February 2020, with vendors Ericsson, Huawei and Qualcomm, Vodafone proves Dynamic Spectrum Sharing (DSS) in the low bands 700 MHz and 800 MHz on a non-standalone device. In June 2020, AT&T deploys DSS. Meanwhile, Nokia has claimed that DSS is nothing new to it. Years ago, it achieved 2G-4G DSS in which low-band GSM spectrum was used to expand LTE service.

Nov
2019

At the World Radiocommunication Conference 2019 (WRC-19), delegates meet and agree towards **harmonization of 5G spectrum**. In particular, 24.25-27.5, 37-43.5, 45.5-47, 47.2-48.2 and 66-71 GHz are identified. This translates to 17.25 GHz of spectrum compared to only 1.9 GHz available before this conference. Of this, 14.75 GHz of spectrum has been harmonized worldwide, reaching 85% of global harmonization. The next meeting will be WRC-23 in 2023.

Jun
2020

Global mobile Suppliers Association (GSA) reports that 97 operators in 17 countries have public licenses to operate 5G service in mmWave spectrum. Of these, 22 operators are already operational with 24.25–29.5 GHz being the most common band.

Jul
2020

3GPP finalizes **Release 16** specifications. This release adds many more bands to FR1. In FR2, n259 (39500-43500 MHz) is added.  This release also adds **NR-U** for operation in unlicensed spectrum.

Oct
2020

GSA reports that operators are showing most interest in n77 and n78 (mid-band); n257, n258 and n260 (mmWave); and n28 (low-band). GSA website is also the place to track the latest in 5G spectrum news.

Dec
2020

In the US, FCC puts up for **auction** 280 MHz of spectrum in the range 3.7-3.98 GHz, known as C-band. A successful bidder will likely purchase 100 MHz of contiguous mid-band spectrum. Compared to mmWave, C-band has better propagation properties, making this an important auction for 5G operators.

5G Spectrum


# Semantic Versioning

## Summary

Software is almost always versioned to coordinate installation, upgrades and correct interfacing with other software. While a flat linear numbering such as *123, 124, 125…* might do the job, **Semantic Versioning (SemVer)** presents a better numbering scheme. It contains more useful information. Each part of the version number has a well-defined meaning. This helps software professionals and automation tools to more easily update software versions for their applications.

Traditionally, every software vendor designed their own numbering scheme, sometimes with semantics.  From the late 1990s, software became more open sourced. As applications were increasingly built from many open source components, it became essential to know and use the correct version of each component. SemVer provides a common definition.

Without a proper versioning scheme, it becomes difficult to update software and its dependencies, leading to what's often called *dependency hell*.

## Discussion

### Could you describe the main parts of a semantic version number?

A semantic version number has three core parts, written in the form of `x.y.z`, where `x`, `y` and `z` are numbers:

    + **Major**: Incremented when incompatible API changes are made. Applications and other software that use the affected APIs will break. Hence, their code have to be updated.
    + **Minor**: Incremented when new functionality is added in a backward compatible manner. It's safe to update to a new minor version without requiring code changes. Code changes are needed only to make use of the new features.
    + **Patch**: Incremented when backward compatible bug fixes are made. No new features are added. Some call this *micro*.As an example, consider the base version number `1.0.0`. A bug fix to an internal implementation is a patch, resulting in `1.0.1`. Suppose a new argument with default value is added to a public method, and the default value is consistent with the base version. This is a minor update, resulting in `1.1.0`. If the new argument has no default value, this is an incompatible API change, resulting in `2.0.0`.

### Could you describe the SemVer parts about pre-release and build?

Sometimes we wish to release software informally to some users for testing purposes. These are called pre-releases. In SemVer, this is specified by a **hyphen** followed by one or more dot-separated **pre-release identifiers**. Identifiers can include hyphen and alphanumeric characters. For example, in `1.2.4-alpha.1`, `-alpha.1` is a pre-release part that has two pre-release identifiers. In pre-release, software may not be stable and might not satisfy the compatibility requirements implied by the normal version `1.2.4`.

Sometimes we wish to record build metadata such as who made the build, build machine, time of build, or checksum. In SemVer, this is specified by a **plus sign** followed by one or more dot-separated **build identifiers**. Identifiers can include hyphen and alphanumeric characters. For example, in `1.0.0-beta+exp.sha.5114f85`, `+exp.sha.5114f85` is the build part that has three build identifiers. `1.0.0+20130313144700` is an example with build information but not a pre-release.

Both pre-release and build parts are optional. Pre-release part comes before the build part.

### What's the precedence of version numbers in SemVer?

When upgrading from a lower to a higher version, it's important to define how version numbers should be compared. Comparison is left to right. Thus, `1.0.0 < 2.0.0 < 2.1.0 < 2.1.1`.

Two versions that differ only by the build part, have the same precedence. A pre-release version has lower precedence than its normal version. Numeric identifiers always have lower precedence than non-numeric identifiers. Thus, `1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0`.

Precedence is used when updating project dependencies via SemVer **ranges**. For example, in NPM projects, dependency version can be specified as:

    + `1.0` or `1.0.x` to allow all patches to `1.0`; `~1.0.4` to allow patches to `1.0.4`
    + `1` or `1.x` to allow all patches and minor updates to `1`; `^1.0.4` to allow patches and minor updates to `1.0.4`
    + `*` or `x` to allow any update

### What are the benefits of using semantic versioning?

When an application uses and depends on many software packages, SemVer helps in using the most suitable versions of these packages. Suppose a Node.js application currently uses `2.0.0` of a certain package. Suppose more recent versions `2.1.0` and `3.0.0` are available. By specifying `^2.0.0` in the configuration file, NPM package manager will update the package to `2.1.0` but not `3.0.0` thereby preserving backward compatibility.

By using semantic version ranges, such as `^2.0.0`, `<3.0.0` or `~2.2.1`, the application benefits from package updates without any update to the configuration file.

Package developers can use SemVer to keep track of how the package has evolved and manage its lifecycle. Semantic versioning helps them to know what releases are compatible with others. SemVer also avoids dependency hell, which can happen when different versions of the same package are used by other packages in the same application.

### Which are some well-known projects or communities using semantic versioning?

We note a few projects as examples. Node.js uses SemVer. Versioned dependencies are specified in a configuration file, such as `bower.json` (for Bower) or `package.json` (for NPM). NPM also has a useful SemVer calculator.

Python uses SemVer but it allows for a more flexible use beyond the SemVer specification. For example, Python PEP 440 allows a date-based versioning scheme, such as `2012.4` or `2012.7`. It allows for alpha, beta and release candidate versions, such as `1.3a2`, `1.3b1`, and `1.3rc4`. Post-releases are allowed, such as `1.3post2` or `1.3rc4.post2`. Development releases are allowed, such as `1.3dev2`. Public version identifier is separated from local version identifier with a plus sign, such as `1.0+ubuntu.1`.

Composer, a PHP dependency manager, adopts SemVer. It includes stability constraints such as `-dev` or `-stable.`. Versioned dependencies are specified in file `composer.json`.

### Could you mention some best practices when using semantic versioning?

The version core is formed of digits 0-9 but must not contain leading zeros. Thus, `2.1.0` is fine but `2.01.0` is not. Version `0.x.y` can be used during initial development with an unstable public API. This can start with `0.1.0`. Once software enters production, it should start with `1.0.0`.

Sometimes we see a prefix `v`, such as in `v2.1.3`. This is often seen as **tag names** in version control systems. This is acceptable although the version number is just `2.1.3`.

Package owners/developers must attempt to avoid frequent incompatible changes. The cost of major upgrades is significant. It's important to evaluate the cost/benefit ratio before introducing incompatible changes. Some even recommend creating a new library/package name rather than making incompatible changes to an existing one.

If an incompatible change goes into a minor version by mistake, release a new minor version to restore compatibility. The incompatible changes can go into the next major release. Avoid modifying an existing release.

Deprecating a public API should happen in at least one minor release before that API is removed in a subsequent major release.

### What are some criticisms of semantic versioning?

While the specifications are clear, putting them into practice is more difficult. It's not always clear if something is a bug fix or a new feature. For example, there's no consensus about the next version number if a new warning is added to a package.

There's no fully automated way of knowing if a change is compatible. Code reviews, golden files and automated tests can address this. For example, a Java API package has two types of users: consumers who use the API and providers who implement the API. An API change can be compatible for some users and incompatible for others. Spring Boot uses version numbers to indicate the extent of change. Adopting SemVer would lead to too many major releases.

SemVer is rigid in the sense that a minor change that "breaks" for 1% of users will still be considered as a major change. This may be okay for robots but not for humans. Version number must give us a sense of how much of the code has changed. Ultimately, SemVer is not going to protect us completely from breaking changes. We should instead take some time and responsibly update dependencies.

## Milestones

1979

Oracle releases version 2.3 of its SQL-based RDBMS. This is just an example to show that some form of semantics is part of version numbers even in the 1970s or possibly earlier. Subsequently, Oracle evolves this numbering to the form `major.maintenance.applicationserver.component.platform`.

1994

Linux kernel version 1.0.0 is released. Subsequently, this project adopts a numbering system of the form `major.minor.revision.stable`. In later years, `major.minor` together represent a major stable release. Revisions to this include bug fixes, new drivers and even features. The optional `stable` part indicates stable (even numbers) or development (odd numbers) releases.  Sometimes `major` is incremented if `minor` "gets too large".

May
2010

OSGi Alliance publishes a whitepaper titled *Semantic Versioning*. This versioning scheme is targeted at Java packages. A version number has four parts in this form: `major.minor.patch.qualifier`. Examples of this are `0.9.0`, `1.11.2.20100202-R32`, and `2.0.0.beta-20100202-R32`. We should note that this is not exactly same as the SemVer specifications that comes out in 2011.

Jun
2011

Tom Preston-Werner publishes on GitHub version 1.0.0-beta of **Semantic Versioning Specification**. Version 1.0.0 appears in September.

Jun
2013

Version 2.0.0 of Semantic Versioning Specification is published.

Nov
2015

Drupal 8.0.0 is released. Starting with this version, Drupal adopts semantic versioning. In earlier versions, for example, a contributed module of version 1.0.0 might refer to 8.x-1.0 or 7.x-1.0 depending on the Drupal core version. The project's `info.yml` file resolved the ambiguity. Adopting semantic versioning simplifies this. Drupal is a PHP-based CMS. It's first version was open sourced in 2001.

Semantic Versioning


# Machine Learning

## Summary

**Machine Learning** is a data-oriented technique that enables computers to learn from experience. Human experience comes from our interaction with the environment. For computers, experience is indirect. It's based on data collected from the world, data about the world.

What we call data is really vast amounts of historical circumstances and reactions in a machine readable format. The machine (computer) learns permutations and combinations of a given circumstance and reacts appropriately. There is **uncertainty** in circumstance and hence reaction. This is the **unknown** that the machine has to learn. Machines aim to maximize the desired outcome.

Machine learning due to its holistic approach can solve a broad set of problems such as object detection, voice recognition, computational biology, price forecasting, and more.

## Discussion

### How do machines learn?

Traditionally, *intelligence* was introduced into a system **explicitly** using rules. Rules took the form of "if this happens while in this state, do that". These *rules* are derived from a *knowledge base* that's particular to that domain or application. However, such a rule-based system has limitations. To characterize the system completely, there could be potentially hundreds of rules. Moreover, rules come with exceptions that need to be considered as well. This is clearly not manageable for complex systems.

Machine Learning takes a different approach. Instead of working on pre-defined rules, machines look at large amounts of data. For each data point, they take note of the associated response. They do this for sufficient amount of data and thereby **implicitly** learn the rules. These implicit rules can be described in terms of *features* and *outcomes*.

For machines to learn properly, relevant and wide-ranging data should be made available. Data should cover all possible scenarios. Data is typically split into *training dataset* and *testing dataset*. Machines learn from the former set. The latter is used exclusively to validate the model. The learning process is not linear. It's self-correcting and iterative.

### What are the different Machine Learning types?

Learning takes place based on what worked through historical events (*asynchronous learning*) and on what is accepted in contemporary events (*synchronous learning*).

When machine is trained using historical data, the learnings can be classified as **Supervised** and **Unsupervised** learning:

    + **Supervised Learning** uses a self-correcting feedback loop. The expectation is labelled. For instance, Temperature, Moisture and Humidity (called *features*) can be used to predict the chance of rain in the next 24 hours. Historic data that include Temperature, Moisture and Humidity are recorded and labelled as 'Rain' or 'Not Rain' depending on whether it rained or not rained in the following 24 hours. This is called *Classification* problem. The system can also be designed to learn the amount of rain. This is called *Regression* problem.
    + **Unsupervised Learning** becomes useful when labelled datasets are not available. Without explicit instructions, model attempts to find structure in the data. Clustering, anomaly detection, association, autoencoders are different ways to organized data.AI systems use synchronous learning to reward/penalise right/wrong decisions and prevent future mishaps. **Reinforcement Learning** is concurrently applied in the decision process as a result of series of actions.

### How does ML differ from regression modelling?

By purpose, ML is for predictions whereas regression is to infer relationships between variables. But this is oversimplifying the truth. Regression can also be used for predictions. In general, statistical models are easier to interpret. They use all of the data towards building the model, whereas ML partitions the dataset into training and test sets.

Linear regression comes from statistics and may be considered "too simple" to be treated as an ML approach. However, ridge or lasso regression are derived from linear regression, and these are commonly used by ML researchers. It's therefore sensible to include regression in an ML toolbox. In fact, understanding data should be the main goal and both statistical and ML models should be seen as tools.

Statistical modelling is highly contextual and with assumptions. For instance, the math behind linear regression and logistic regression are very different. Machine learning generalizes them under supervised learning and optimizes for minimum error. All machines use numerical math techniques to iteratively solve for unknown parameters.

### What is feature engineering and why is it important?

**Feature engineering** comes after data is cleaned and transformed. Feature engineering is arriving at relevant variables that relate to solving the problem at hand. Feature engineering is done by **domain experts** who understand what each variable means, how to interpret it and how it relates to other variables.

A dataset will typically contain one or more variables or features. Some of these may influence the outcome. For example, temperature and humidity may be features that influence the chance of rain in the next six hours. The data may also contain the time of day or day of the week but these are features that probably don't influence the chance of rain.

The job of an ML engineer is therefore to identify the right features for the problem. The selected features add up to the outcome of model. The accuracy of the ML model directly depends on features the ML engineer has chosen. Good features make modelling a simpler task. Bad features would result in a complicated model.

### How does ML add value to Big Data?

A Forrester report from 2016 showed that 60-73% of all big data within enterprises are not used for analytics. In 2018, a report by DataRPM on the Industrial Internet-of-Things (IIoT) showed that 85% of sensor data collected from trillions of data points are unused. Manual analysis is impossible. The use of ML can unlock the value in this data.

Today's data coming from the mobile and web include video, audio, image and text. In addition, lots of data come from sensors in automotive and healthcare verticals. Historic data in financial services or retail help ML algorithms discover patterns. Problems that rely on such data can be modelled better with the help of ML.

In fact, ML likes large volumes and variety of data. In an ideal case, ML has access to lots of data collected in diverse scenarios. This enables efficient and holistic learning.

### What kind of problems can be solved with ML?

Broadly, the following problems are solved with ML:

    + **Regression**: This is the task of predicting a continuous quantity. Here, predictions are often made for quantities, such as amounts and sizes. For example, a house may be predicted to sell for a specific dollar value, perhaps in the range of $100,000 to $200,000.
    + **Classification**: This is the task of predicting a discrete class label. For example, 1. an email of text can be classified as belonging to one of two classes: 'spam' and 'not spam', 2. image classification problems where there could be thousands classes (cat, dog, fish, car, etc.).

### What's the approach to solving ML Problems?

In a typical ML pipeline we would classify the problem, gather data, process data, model the problem, execute the models, validate results, and deploy the solution.

Once you have defined the problem and outlined the features, you then need to split the data in a way that's easy to test. You split this data in a **70 (train) : 30 (test)** ratio. 70% with which machines learns and 30% where it tests learning. The training data is modelled for validation. This model needs to be validated with testing dataset and evaluated against multiple models to find the best model.

The idea of splitting data into training and test datasets can be traced to the *Common Task Framework*.

It's important to have an acceptable accuracy percentage (say, 60%+) across both training and testing datasets. If the accuracy rate isn't high enough or not consistent across the two datasets, then the ML process should be repeated with different or modified features.

### What is overfitting in the context of ML?

Often we read too much into past. We're surprised to see that history didn't repeat itself. This could happen either because response is specific to one particular circumstance or there's too little data. When this happens in ML, we call it *overfitting*.

The possibility of overfitting exists because the criterion used for selecting the model may not be the same as the criterion used to judge the suitability of a model. For example, a model might be selected by maximizing its performance on some set of training data, and yet its suitability must be determined by its ability to perform well on unseen data. We can state that overfitting occurs when a model has **memorized** training data rather than **learned** to generalize from a trend.

### Could you compare or contrast Machine Learning (ML), Deep Learning (DL) and Artificial Intelligence (AI)?

This is better explained through an example. ML is about learning a task. For instance, a self-driving car learns many tasks: to brake or not to brake, speed up or slow down, turn the steering wheel, indicator functions, etc. While ML learns all these tasks separately, AI executes them in a coordinated manner, rewards good decisions, and penalises wrong decisions. AI also accounts for contextual information that may not be part of ML. Thus, ML is an approach to realizing AI.

DL is special case of ML and it's inspired by how neurons in the human brain process information. While ML learns once, DL does it in multiple stages. ML expects features to be provided at the input. DL discovers features on its own. When problems are complex, DL does better than ML. For example, recognizing a human may involve identifying basic features (eyes, ears, hands, legs, etc.) at stage 1; and identifying higher order features (face, upper body, lower body, etc.) in stage 2; and finally calling it out as 'Human' in stage 3.

### How to improve the accuracy of ML models?

    + **Ensemble** methods are techniques that create multiple models and then combine them to produce better results. For example, a candidate goes through multiple rounds of job interviews. Although a single interviewer might not be able to test the candidate for each required skill and trait, the combined feedback of multiple interviewers usually helps in better assessment of the candidate.
    + **Bagging (Bootstrap Aggregating)** is an ensemble method. First, we create random samples of the training dataset (subsets of the training dataset). We build a classifier for each sample. Finally, results of these multiple classifiers are combined using averaging or majority voting. Bagging helps to reduce the variance error.
    + **Boosting**: The first predictor starts by classifying original dataset with equal weights to each observation. If classes are predicted incorrectly using the first learner, then it gives higher weight to the wrongly classified observation for the successive learner. Being an iterative process, it continues to add classifier learner until a limit is reached in the number of models or accuracy. Boosting has shown better predictive accuracy than bagging, but tends to overfit the training data.

### In what scenarios are ML not applicable or has failed?

ML is applied in diverse fields where plenty of data is available. There are scenarios where ML has challenges, but constant endeavor ensures improved accuracy and increased acceptability. In the accompanying figure we can see how **ImageNet** ML algorithms have evolved for better accuracy.

Failure of ML can be attributed to **incorrect problem formulation**, **wrong choice of features** or **inappropriate algorithms**.

ML algorithms can become **biased** due to various reasons. For example, an algorithm that sees only men writing code and only women in kitchen in its training data will naturally become biased. In a real-world case of bias, Google Allo once responded with a turban emoji when shown a gun emoji. Google Translate showed gender bias in Turkish-English translations. Amazon's AI-based recruiting tool was found to be favour male candidates.

Other failures of AI/ML happened in 2018. Uber's self-driving car killed a pedestrian. IBM's Watson AI Health has failed to impress doctors.

## Milestones

1950

Alan Turing creates the *Turing Test* in which a computer must attempt to pass itself of as a human to other humans. In June 2014, a robot named Eugene passes this test by convincing 33% of the judges. A more difficult variant called *Loebner Prize* requires that more than 50% of the judges be convinced after a 25-min conversation. As of March 2018, no robot has won the prize.

1952

Arthur Samuel writes the first learning program. Applied to the game of checkers, the program is able to learn from mistakes and improve its gameplay with each new game. By mid-1970s, the program beats humans at checkers. Board games are useful in developing ML because they are understandable and complex.

1957

Just as the human brain is composed of interconnected neurons, Frank Rosenblatt designs the first artificial neural network called the *perceptron*. The idea is to solve complex problems through a series of simple decisions. Rosenblatt applies it for doing image recognition.

1967

The *Nearest Neighbour* algorithm is created and applied to map routing. This starts the field of pattern recognition.

1979

Invented by researchers at Stanford University, a robot now named the *Stanford Cart* is able to navigate obstacles in a room on its own.

1981

Gerald Dejong invents *Explanation Based Learning*. Computer uses data to train itself and create a rule to achieve a given goal. It discards information irrelevant to the problem. This is a type of supervised learning. In general, the 1980s is the decade of expert systems that are based on rules.

1990

The 1990s is the decade when approach to ML shifts from being knowledge driven to data driven. This is supported through the next two decades with greater availability of data, cloud computing and big data technologies.

2006

Geoffrey Hinton coins the term *Deep Learning (DL)* to describe new architectures of neural networks. This approach is applied to image recognition.

2012

The Google Brain project uses DL to detect visual patterns. Google X project applies Google Brain to YouTube videos to identity frames that contain cats. Geoffrey Hinton leads a team and wins ImageNet's computer vision contest by a large margin. This popularizes DL. In the coming years, DL becomes an important technique to create models with much better accuracy. This is the decade when DL becomes feasible.

2015

Google's AlphaGo uses ML to beat professional player Lee Sedol in a challenging board game called Go.

2017

Google Brain chief Jeff Dean states that DL starts to work with at least 100,000 data points. This underscores the importance of data availability for DL.

Machine Learning


# IoT Security

## Summary

The Internet of Things brings together not just computers but also sensors, cameras and controllers that are connected to vehicles, medical devices, power grids, nuclear plants, and many more critical infrastructure. Securing IoT is therefore a big concern since an attack can be life threatening. An insecure IoT system leads to lower safety, privacy and availability.

Since the mid-2010s, researchers have analyzed IoT systems to identify potential vulnerabilities and possible solutions for the same. Given IoT's large attack surface, better security can be achieved only by securing all parts of the system: from devices to apps, from APIs to network endpoints.

In 2018, Gartner reported that 40% of smart home appliances are in use for botnet attacks. In 2021, 84% of surveyed companies reported an IoT security breach. Clearly, there's much work to be done.

## Discussion

### What are some potential attacks in various IoT verticals?

IoT systems can be hacked regardless of the vertical in which they operate. Consumer appliances,  industrial facilities, connected cars, drones, video surveillance systems, power grids and utilities, smart buildings, city infrastructure and transportation systems, medical devices and hospitals, and retail stores are some examples of what can be attacked. We describe a few of these.

In a connected home, an attacker can see sensitive data sent from a wearable, perform illegal video surveillance, profile individuals, gain control over door locks and sensors, or render appliances inoperable.

With connected and semi-autonomous cars, hackers can remotely steal vehicles. They can also gain control of a vehicle when it's on a freeway.

In industrial systems, PLCs and SCADA systems could be targeted. This can halt critical operations, cause accidents or install spyware. Attacks on critical infrastructure such as nuclear power plants can be catastrophic. In one experiment, a software-enabled gun was configured to prevent firing or miss its target.

In hospitals, patient data could be stolen. Interfering with the operation of a pacemaker or changing dosage can cause death.

### Could you mention some real-world attacks on IoT devices and systems?

Researchers have studied vulnerabilities and possible exploits on IoT devices and systems. Some of these include St. Jude Medical's cardiac implants, Owlet Wi-Fi baby heart monitors, TRENDnet webcams, and Jeep SUVs. Ignoring these research findings, we focus on real-world attacks instead.

In October 2016, Dyn servers experienced a DDoS attack by the Mirai botnet, affecting many commercial websites. Mirai made use of digital cameras and DVR players. The attack involved 100,000 malicious endpoints resulting in an attack strength of 1.2Tbps.

Stuxnet (botnet) attacked industrial systems back in 2010. . In 2015, 230,000 people were without power in Western Ukraine when an electrical power grid was attacked. In 2021, an attack on the water supply to Oldsmar, Florida changed the level of sodium hydroxide. Fortunately, operators spotted the change and took corrective action.

Ring doorbells come with a mobile app. In January 2020, the app was reported to leak personally identifiable information to third parties. In December 2020, we came to know of attacks on Ring smart cameras. Hackers could speak to their victims "screaming obscenities, demanding ransoms, and threatening murder and sexual assault."

### What are the common security vulnerabilities in IoT?

We note the following vulnerabilities:

    + **Weak, guessable, or hardcoded passwords**: Dictionary and brute force attacks can exploit this vulnerability. Devices many also have unchangeable credentials and firmware backdoors.
    + **Insecure network services**: Connected devices may have ports open or services running that aren't required.
    + **Insecure ecosystem interfaces**: Outside the device, there's an ecosystem of interfaces, APIs, gateways, cloud services and mobile apps. Any of these can be compromised.
    + **Lack of secure update mechanism**: Updates are risky if done without firmware validation, payload is unencrypted, there's no rollback or notifications. In many cases, vendors don't patch the firmware even for known vulnerabilities.
    + **Use of insecure or outdated components**: Third-party components are commonly used in today's systems. Software and hardware supply chains could be compromised.
    + **Insufficient privacy protection**: Personal information is stored insecurely or used improperly.
    + **Insecure data transfer and storage**: Data is not encrypted during storage, transmission or processing.
    + **Lack of device management**: This concerns asset management, update management, secure decommissioning, monitoring and response.
    + **Insecure default settings**: Factory settings are often insecure or even disallow changes to more secure settings.
    + **Lack of physical hardening**: Devices are hacked in close proximity, thereby enabling remote hacking.

### Could you elaborate on the different types of IoT security attacks?

IoT attacks can be broadly classified by the layer in which they occur:

    + **Physical Attacks**: Concern hardware elements and happen in close proximity to the device. Device tampering, physical damage, or draining battery via sleep deprivation are examples.
    + **Software Attacks**: Buffer overflows, code injection and cross-site scripting are typical techniques. A hacker typically scans for vulnerable devices, infects them, and then steals data or makes devices inoperable.
    + **Network Attacks**: Can attack remotely via network connections. Sinkhole attack discard packets rather than forwarding them. Man-in-the-middle attack is often employed to attack Wi-Fi routers. Spoofing is another technique in which a malicious devices presents itself as an authenticated one.
    + **Encryption Attacks**: Even when data is encrypted, side-channel and cryptanalysis attacks are techniques to figure out encryption keys.Ultimately, IoT's attack surface is large. An attack can happen on an end device such as a connected thermostat, on its wireless network, on the network's access point, on the ISP gateway or an app that's communicating with the thermostat.

### Could you share some best practices for better IoT security?

Many of the best practices can be directly derived from the common IoT security vulnerabilities. Security is an **end-to-end concern** and devices, network, APIs, cloud services and apps — all need to be secured.

Network endpoints and interfaces should be secured with firewalls, anti-malware, and intrusion prevention and detection systems. For authenticating users and devices, use two-factor authentication, biometrics, Public Key Infrastructure (PKI) and X.509 digital certificates. Data should be encrypted along with mature key lifecycle management. Analytics must be specialized for IoT security with a plan to recover from attacks.  Minimize device bandwidth.

**Security by Design** is an approach to factor in security at the start of an IoT project. Device IDs and credentials can be embedded during manufacturing. This may be called the Root-of-Trust (RoT) from which other keys and credentials are generated. Micron's Authenta is an example of securing flash memory, thus "hardening" the device against attacks.

Systems should be designed with countermeasures against Simple Power Analysis (SPA) and Differential Power Analysis (DPA) that exploit emissions from semiconductor devices.

### What are some challenges in securing IoT systems?

One challenge with securing IoT is the sheer number and variety of devices, in their billions. The IoT ecosystem is complex and consists of sensor devices, network nodes, many communication protocols, and regulations. If an attack should occur, it's also not clear who should be held accountable: equipment vendor, system integrator, service provider, or the user.

Businesses are rushing to implement or adopt IoT-enabled data collection and analytics. They're doing this without considering security. When new ventures fail, their devices are abandoned without support or security updates. Even otherwise, many consumers have never done updates on their devices. Worse still, many devices don't even have a user interface to perform updates.

Many IoT devices are designed to be cheap and disposable, wearables in particular. IoT devices have constraints on processing, storage, battery and communication. As a result, vendors prioritize cost and efficiency over security.  One way to mitigate risks is for communication service providers (CSPs) to "harden" their networks in a device-independent manner.

## Milestones

Jun
2010

The **Stuxnet** worm is discovered. It targeted Microsoft Windows systems and a Windows-based Siemens software used in industrial control systems. One of attacked sites was an uranium enrichment plant in Iran. In subsequent years, more related worms and spyware such as Duqu, Flame, and Gauss are discovered. Given their sophistication, security researchers believe that these are state-sponsored.

Sep
2014

Security expert David Jacoby of Kaspersky Lab tries to hack into his own home's electronics devices and consumer appliances. He discovers a number of vulnerabilities. He's able to easily get into his smart TV, game console and storage devices. At the Security Analyst Summit (Feb 2015), other researchers explain how they could hack into police surveillance systems, car washes and fitness bands.

Oct
2016

**Mirai** botnet launches DDoS attack on Dyn servers, affecting many commercial websites. Digital cameras and DVR players are used for this attack. It becomes the largest such attack to date. The process involves scanning for vulnerable devices, reporting them to the ScanListen server, installing Mirai, registering the malware with the Command and Control (C&C) server, and executing attack commands initiated by a botmaster.

2017

Forrester Research looks at a number of critical technologies that contribute to more secure IoT systems. All of them are still in the survival or growth stages. Some of these relate to network security, API security, authentication, device hardening, threat detection and device user privacy controls.

Dec
2018

OWASP releases a list of top 10 IoT vulnerabilities. This is an initiative of *The OWASP Internet of Things Project* that was started in 2014. The research community subsequently uses this list to evaluate the security of IoT systems.

Nov
2019

Recognizing the threat to IoT systems, the European Union Agency for Cybersecurity (ENISA) publishes a report describing best practices for IoT security with an emphasis of Software Development Life Cycle (SDLC). This report is meant for IoT developers, integrators and system engineers.

Aug
2020

Under the ioXt Alliance Certification Program, about a dozen IoT devices become the **first devices to be certified from a security standpoint**. Certified devices include Google Pixel 4, Silicon Labs xG22 for Bluetooth connectivity, and T-Mobile home internet gateway. The Alliance launched the certification program in April 2020 and it aims to establish a global standard for IoT security. The certification program is based on *ioXt Pledge* but includes device-specific profiles such as Android-enabled devices and smart speakers.

IoT Security


# User Experience

## Summary

User Experience (UX) is how a person feels while engaging with a product or service. For example, when an e-commerce website is easy to use and free from ads, it's a better UX. Conversely, if it's complex to navigate and hustling with ads, it gives a bad user experience.

A good UX design delights the user. It touches upon the psychological aspects of user interaction. Human empathy is an important factor. A delightful UX design is useful, usable, valuable, desirable, accessible, credible and findable.

Good UX designs uplift failing businesses. Conversely, a bad UX can cause losses to a firm. Statistics say every dollar invested in UX brings a hundred dollars in return. About 70% of customers give up purchases due to bad UX.  A good UX improves a product's visibility that translates into higher sales, revenue and customer loyalty.

## Discussion

### Isn't UX the same as the UI (User interface)?

UX and UI are often mistaken for synonyms. UX deals with how we experience a product. UI is more about its look and appearance.

Structurally, UX is comparable to the skeleton of a product. It takes a lot of design thinking and research. UI is akin to the body or the external form. We associate the product's colours, fonts, images, navigation and other visual elements with UI.

UI is tangible, whereas UX is intangible. The first stage of product development is UX. UI comes next, and the process is recursive as the product evolves.

UX covers all types of human experiences and interactions. It doesn't restrict just to the device interface. On the other hand, UI is all about the device interface of the product or service. For instance, the UX of a food delivery app takes into account the speed of delivery. Displaying delivery time on the interface so that it's useful for the customers is a UX question. The UI bothers about putting it up on the app attractively.

### Isn't UX the same as the interaction design?

A user's interaction with a product largely shapes his/her user experience with that product, both on physical and emotional levels. Form, function and technology play important roles in this interaction. While there's considerable overlap between UX and interaction design, they're not the same.

UX is interaction design and more. It's a more holistic user-centric process that includes other aspects when the user is not interacting with the product: information architecture, content strategy, visual design, sound design, industrial design, usability, and user research. For example, interaction design concerns with user flows so that users can exercise product functionality easily and effectively. Information design concerns with presenting information for better understanding. Visual design concerns with the product "look-and-feel".

### What are some basic principles of UX design?

**Usability**: A good UX design is easy to use and learn. It is accurate and engaging. It helps a user complete tasks effectively. Its error tolerance is high. This means it understands the user's intentions and rectifies errors automatically or with minimum effort.

**Accessibility**: UX should make the product or service easily accessible to a maximum of users across the spectrum. As per a report, long-term stock prices of brands catering better to customers with disability beat their competitors.

**Hierarchy**: The flow of functionality and information is the hierarchy. UX design shall ensure that the flow is natural and easy. A well-structured hierarchy provides a great user experience.

**Consistency**: Keeping the look and feel of a product or service uniform across all platforms and interfaces is vital. Fulfilling user expectations every single time is also equally important.

**Context**: A UX designer should know the purpose of the product or service. An understanding of the user's emotional state should be a prime consideration. Different devices that the user may interface with warrant different design approaches.

### Why do we need user flows in UX design?

Users go through a certain path in their interaction with a digital interface. The series of interactions from the entry point to the final stage is a user flow. Designers plan and create new user flows or alter an existing user flow using user research and feedback. It's an iterative process.

Designers need user flows to create smooth interaction for users to interact intuitively. They make communication with the stakeholders easy by visually breaking down the design components. They help at picking flaws in the existing user flows and offer a blueprint to assess any need for changes.

There are three basic types of User flows:

    + **Task Flows** are suitable to describe the flow of a specific task. Users will follow a single path without branching out.
    + **Wire Flows** convey the layout and design on a single page along with the flow. They can't accommodate dynamic interfaces. They fit best where designs pose screen limitations and demand relative simplicity.
    + **Screen flows** are used to design the aspects of *how* a user shall interact with the product. They accommodate multiple scenarios and complex designs.

### How can we leverage mental models in UX design?

A mental model represents how we perceive an interaction or an experience. Our reactions to a stimulus stem from our past experiences and mental impressions.

Let's take the example of a digital cart on an e-commerce website. We drop items into a cart, add on to them or remove them, checkout and then pay. This is a mental model that matches a real-world shopping experience.

A mismatch between the users' mental model and the design model may occur. It is more pronounced when users don't understand the interface. Designers immersed in design projects can get stuck in their individual mental models. It's called a *designer bubble*. Designers need to keep the average user in mind to break this bubble.

UX research helps us identify the gaps in the design. There are many methods such as card sorting to identify users' mental models. Reworking the designs to suit the users' mental models can minimize such mental model mismatches.

One way of addressing mental model mismatches is by making the UX and UI clearer and more transparent.

### What's the difference between a user testing and a usability testing?

Usability testing is a research methodology. A small group of real customers or users are asked to use a product, feature, or prototype. A moderator or a computer application observes their behaviour and takes feedback. It focuses on usability. It's done as a product or feature or prototype evolves.

User testing is performed to determine whether a product is needed or not. Researchers use user testing to validate product ideas. It focuses on the target audience. It's important at the beginning stages of the design.

However, both user testing and usability testing take a user-centric approach.

### What are some useful UX tips for designers?

Designers should focus beyond what users want. Steve Jobs said it's not the customer's job to know what they want. Giving beyond users' expectations is as important as matching their mental models.

Designer's shouldn't think UX design is only about aesthetics. The main purpose of a UX design is to solve problems. It should address the entire product or service cycle. Visual appeal alone cannot make an interface usable.

Designers must not treat UX as the last item in the product design's timeline. UX design is a work in progress. It depends on contexts and conditions that change over time. Integrating UX design later becomes costly and aesthetically difficult.

Designer should give due importance to small details. ShopClues moved the 'Wholesale' section on its website towards the left side. It replaced the word with 'super saver' etc. This minor change increased the visits-to-order (percentage of visitors placing orders) conversion by 26%.

Designer's shouldn't assume that people read every word on a digital interface. Most people scan or skim web pages. Merely adding more words won't enrich the UX.  Using a concise choice of effective words bears maximum fruit.

### What are some techniques used in UX design?

**Heuristic evaluation**: It is useful to figure out the issues in an existing product. It assesses the usability, accessibility and effectiveness of the user experience.

**Value Proposition**: This method aims to match the user's needs with a proposed product. It works from the standpoint of saleability. Unique Selling Proposition (USP) and target customers are two important considerations in the technique.

**Card Sorting**: Participants organize topics into categories, make groups, and label them. This is useful to build the structure of the digital interface. The technique is cheap and easy to extract input from users.

**Brainstorming**: Participants throw fresh and creative ideas among a closed group. It can be random or structured. Depending on the method, a moderator records, classifies and drafts outcomes from the session.

**A/B testing**: It tests alternative versions of a product by offering them to different users. Comparing the outcomes gives a clear picture of what works best.

**UX roadmaps**: These communicate the firm's UX future and the problems to solve. It is not restricted just to the UX team. It reflects the UX vision of the entire organization.

### Could you discuss a couple of UX case studies?

New York Times app was fast losing its user base. The reasons were poor coverage, changing priorities, lacking regularity and irrelevant content. Johny Veno performed a case study. His team proposed minor changes to the landing page and called it *Timely*. *Timely* would send notifications at opportune moments in a packed day. It would send short reads at times around breakfast, commute, or a coffee break. These customized feeds would glue the readers onto the app.

Stacey Wang identified four pain points in the usability of the Fitbit app:

    + Users had issues tracking and logging exercises on the app.
    + Users couldn't fix a start time or set a duration for the exercises.
    + Users found it hard to identify the right exercise type.
    + The app didn't have an option for the user to challenge a friend.She suggested the following fixes:

    + Make the exercise page easily discoverable and the choice to track and log more conspicuous.
    + Add a button to browse all the exercise types.
    + Add access buttons to edit start time and duration.
    + Add a challenge button on a friend's page.

## Milestones

1960

J. C. R. Licklider introduces the concept of man-computer symbiosis. It talks about a cooperative interaction between men and electronic machines. It is one of the early stages in the evolution of **interaction design** that later becomes an integral part of UX designs.

1973

Xerox Palo Research Center(PARC) creates Xerox Alto personal workstation. It employ's the world's first Graphical User Interface. **Interface design** and UX are interdependent.

1976

Richard Saul Wurman coins the term **Information Design**. The goal of an information design is to work around structure, context, data and information to communicate effectively. Effective communication is an essential part of UX design.

1979

John Bennett publishes *The commercial impact of usability in interactive systems*. It is the first scientific publication with **usability** in its title, denoting what UX means later on.

1984

Apple introduces Macintosh computer that allows the users themselves to become creators. It incorporates *skeumorphism*, a **visual design** technique that mimics real-life objects on the interface. Mac design becomes a trend setter for future UX.

1986

John Brooke invents a questionnaire to assess Digital Equipment Corporation's *System Usability Scale (SUS)*. It becomes a widely used tool to assess the usability of systems. It is a simple, ten-statement questionnaire. Researchers can use it to assess whether a system needs an update. They can also evaluate the effectiveness of system improvements and the need to invest further on UX.

1988

John Whiteside and John Bennette publish a handful of articles in *Usability Engineering*. The articles emphasize initial goal setting, prototyping and interactive evaluation. The three practices become the starting points for most UX methodologies.

1995

Donald Norman et al. coin the term **User Experience (UX)**. It encompasses everything that refers to what the world terms user experience.

1997

Netscape Communication Corporation introduces *Netscape Composer*. It is a webpage editor that gives a **WYSIWYG** (What You See Is What You Get) environment. It permits easy web formatting, editing and spellcheck.

2000

Garrett observes that the web was conceived as an information space. But as the technology evolved, the web is now a remote software interface. Via the web interface, users can perform complex operations or workflows. This duality has led to confusion on the terms used to describe the web. Garrett clarifies some of these terms including visual design, navigation design, information design, interaction design, and more.

2004

Peter Morville introduces the **UX Honeycomb**. It's a framework to evaluate a product or service through seven facets of UX. It's a widely used tool to measure the quality of user experience.

2007

Steve Jobs introduces the iPhone. He calls it a breakthrough product that revolutionizes the history of UX. It replaces the traditional keyboard with a finger-touch keyboard.

2007

Thüring & Mahlke propose the *Components of User Experience (CUE) Model*. It's a comprehensive model that schematizes the core elements of UX. It's built from studying the research findings on smartphones and audio player usage. It categorizes the core UX components into perceived Instrumental qualities, emotional reactions and perceived non-instrumental qualities. Usefulness and ease of use are examples of perceived instrumental qualities. Perceived non-instrumental qualities could include aesthetic aspects, symbolisms and motivational aspects.

2008

Tom Tullis and Bill Albert publish the first book on usability measurement. It is titled *Measuring the User Experience*.

2012

Usability Professionals Association (UPA) changes its name to User Experience Professionals Association (UXPA). It supports people who research and design the UX of products and services.

User Experience


# OWASP ZAP

## Summary

Software security testing is the process of assessing and testing software to discover security risks and vulnerabilities. Such testing could be a **passive scan** to look for vulnerabilities. Or it could be an **active penetration test** (aka pen test) that simulates malicious users attempting to attack the system.

In complex systems, it's difficult to manually determine all possible vulnerabilities. The **Zed Attack Proxy (ZAP)** is an open source tool to automatically find vulnerabilities in web applications. It's part of the *Open Web Application Security Project (OWASP)*.

ZAP can be used as a man-in-the-middle between browser and app server. It can also be used as a standalone application, or as a daemon process without UI. ZAP is suitable for experienced security professionals as well as web developers and functional testers.

## Discussion

### Why should I use ZAP?

ZAP is free and open source. ZAP is for experts as well as beginners. Based on Java, it's cross-platform and hence it can be used on Windows, MAC or Linux. It's also easy to install and use. It's fully documented and there are plenty of community resources to help those who are new to ZAP. It's internationalized with translated versions in many languages. We can also use it with other tools that enable CI/CD workflows. Thus, it's flexible and extensible.

### Could you describe some important ZAP terminology for beginners?

The following may be a starting point for beginners:

    + **Contexts**: Typically, a context will correspond to a web application. It's a way of grouping together a set of URLs.
    + **Scope**: Defined by contexts, it's the set of URLs to test.
    + **Modes**: Each mode allows for certain types of attacks. This gives flexibility while testing. Selecting the mode affects the scope.
    + **Alerts**: An alert is a potential vulnerability. It's associated with a request. A request can have multiple alerts. An alert is flagged with a risk level: High, Medium, Low, Informational, False Positive.
    + **Tags**: A short text associated with a request. A request can have multiple tags. Passive scanning can do automatic tagging based on preset rules.
    + **Notes**: You can associate text with a request. These are for your reference or later action.
    + **Add-ons**: Add extra functionality to the ZAP core. They can be installed from the online Add-on Marketplace. Examples include Ajax Spider, Diff, Forced Browse, Fuzzer, etc.
    + **Replacer**: This is an add-on to replace strings in requests and responses.

### What are the modes in which I can use ZAP?

As of version 2.5.0, ZAP can be used in one of four modes: **Safe**, **Protected**, **Standard** and **ATTACK**. As the names suggest, Safe mode will avoid anything potentially dangerous while ATTACK mode will aggressively try to attack new URLs as soon as they are discovered. You can configure the ATTACK mode behaviour using *Scan Policy*. When pen testing is desired on sites you have permission to test, Protected mode can be used. Standard mode allows for all types of attacks.

In both Safe mode and Protected mode (for URLs out of scope), ZAP will not perform certain types of attacks. These include spider crawling, active scanning, fuzzing, force browsing, breaking (intercepting) and resending requests.

### How can I use ZAP as a proxy?

ZAP can be used as **intercepting proxy**. It stands between the tester's browser and the web application so that it can intercept and inspect messages sent across, and then forward them to the destination. In passive scan, message contents are not modified. In active scan, they are modified to simulate attacks.

To use ZAP as a proxy, you must update configuration in ZAP as well as the browser you intend to use for the tests. Configuring Proxies page on ZAP Wiki gives the details. Once you have configured ZAP as your browser's proxy, connect to the web app under test. ZAP should now start showing one or more entries in the Sites and History tabs. These are requests and responses that ZAP has intercepted for analysis.

### How will ZAP determine what to test in my web application?

ZAP uses a mix of techniques some of which may require a test engineer's help. One of the simplest is to use the **Quick Start** add-on. You enter a URL, typically the home or sitemap URL of the web app. ZAP will crawl it using its Spider add-on. This will give ZAP a list of URLs to test. It will then do an active scan of these.

For a more in-depth test, you should put ZAP in proxy setup. Then, manually explore your application using your browser. Alternatively, perform automated regression using Selenium or similar tool. ZAP will capture all the requests and responses. It can then use them later to do an attack.

### What is Spider?

Web spiders and crawlers are commonly used by search engines to discover Internet content. In the context of ZAP, **Spider** is an add-on. It's used to automatically discover new resources (URLs) on a particular site. It begins with a list of URLs to visit, called *seeds*, which depends on how the Spider is started. The Spider then visits these URLs, identifies all the hyperlinks in the page, and adds them to the list of URLs to visit. This process continues recursively until new resources are not found.

Spider can discover URLs that are not visible as links on pages.

Sites that are Ajax heavy cannot be effective crawled by Spider. An alternative add-on created by Guifre Ruiz, called **Ajax Spider**, should be used. For good coverage, both spiders should be used.

### What are Passive Scan and Active Scan?

Apart from using Spider, there are broadly two different ways in which ZAP looks for vulnerabilities:

    + **Passive Scan**: ZAP by default passively scans all HTTP messages (requests and responses) sent to the web application. Passive scanning does not change the requests and responses in any way, and is therefore safe to use.
    + **Active Scan**: Attempts to find potential vulnerabilities by using known attacks against the selected targets. You must perform active scan only if you have permission to test the application. *Fuzzing* is a technique that can be used as part of active scanning. With fuzzing, invalid or unexpected data is submitted to find vulnerabilities.Rules used for passive and active scans are well documented.

Logical vulnerabilities, such as broken access control, will not be found by any active or automated vulnerability scanning. Manual penetration testing should always be performed in addition to active scanning to find all types of vulnerabilities.

### Can ZAP be used as part of a CI/CD pipeline?

Automated pen testing is possible with ZAP and this is an important part of continuous integration. It helps to uncover new vulnerabilities as well as regressions of previous vulnerabilities in an environment that is changing quickly, and for which the development may be highly collaborative and distributed. In fact, ZAP is available as a plugin for Jenkins.

ZAP provides a Rest Application Programming Interface (API) that allows other tools to interact with ZAP programmatically. Other tools can make use of this API to trigger attacks. The ZAP API is available in JSON, HTML and XML formats. The ZAP API is particularly useful for Security Regression Tests.

Beginners can use a simple web UI to explore and use the API: at `http://zap/` when you're proxying via ZAP; or via the host/port ZAP is listening on, such as `http://localhost:8080/`.

As part of cloud-based workflows, in one example, Microsoft has explained how ZAP can be used in Azure. For passive scans, it can be part of CI/CD pipelines. For longer active scans, a nightly pipeline is preferred.

### My site runs on HTTPS. How does ZAP handle SSL certificates?

Since ZAP is set up to act as a proxy between your browser and the web application, using SSL (HTTPS) will cause the certificate validation to fail and the connection to be terminated. This is because ZAP encrypts and decrypts traffic sent to the web application using the original web application certificate. This is done so that ZAP can access the plain text in the requests and responses.

To prevent this failure from happening, ZAP automatically creates an SSL certificate for each host you access, signed by ZAP's own Certificate Authority (CA) certificate. To have your browser trust these SSL certificates, you need to first import and trust the ZAP Root CA certificate. Once it is trusted, the other ZAP SSL certificates signed by it will be trusted as well.

ZAP Root CA certificate is required only for active scans. For passive scans, since content is not modified, the web app's original certificate can be used.

### Could you share some user stories and case studies of ZAP?

OWASP ZAP is used by countless organizations across the globe for validating their web application security postures, from governments agencies and educational institutions to large enterprises. Some of these include Mozilla, Microsoft, Ernst & Young, Accenture, and Google.

It's been commented that the alert levels flagged by ZAP don't always correspond to reality. A minor risk may be flagged as High and vice versa. Reporting is in HTML and this could be improved.

### What are the alternatives to using ZAP?

There are plenty of alternatives to ZAP offering similar or subset of features. Some of these include Burp Suite, Nikto, Acunetix, w3af, Arachni, Andiparos, HTTP Analyzer, etc.

When comparing alternatives, we need to consider many aspects: features, ease of installation/upgrade, ease of use, learning curve, cost, support (commercial or community), release rate (how often tool is updated), API and extensibility, available third-party integrations, etc. UpGuard's **CSTAR Score** is one way to quantitatively compare the alternatives, where CSTAR stands for *Cybersecurity Threat Assessment Report*. For example, Arachni scores only 399 whereas ZAP scores 788.

CSTAR Score is also a quick and objective way for business stakeholders to assess security compliance without looking into the details. For example, an audit of the healthcare sector in 2016 revealed a low CSTAR score of 420 (danger zone).

## Milestones

2005

Based on Java 1.4.2, **Paros Proxy** is released as a tool to test security vulnerabilities in web apps.

2012

Simon Bennetts forks Paros Proxy and experiments with it as a way to learn about security tools. ZAP takes its birth from here. In December, ZAP 2.0.0 is released. This release contains contributions from three GSoC (Google Summer of Code) Projects that enhance the capability of ZAP. Meanwhile, Guifre Ruiz creates **Crawljax** (aka Ajax Spider), an open-source Java spider for AJAX web apps.

2014

Add-on **SOAP Scanner** is released. It comes out of GSoC2014. Scanned and detected WSDL files are now read and SOAP petitions now follow the correct format.

May
2015

Not meant for desktop environments, **ZAP as a Service (ZaaS)** is proposed. The idea is to run in 'server' mode: "long running, highly scalable, distributed service accessed by multiple users with different roles".

2016

The **ZAP Sonar Plugin** is available for reporting into SonarQube v6.3 or higher. The official **ZAP Jenkins Plugin** is released. This extends the functionality of the ZAP security tool into a CI environment.

2017

Introducing the **JxBrowser** add-on for ZAP. With **ZAP Browser Launch** we can now launch a browser from within ZAP. In November, ZAP 2.7.0 is released.

OWASP ZAP


# Shift Left

## Summary

A typical software development process is sequential (1970s-1990s): define requirements, analyse, design, code, test and deploy. In this process, testing happens towards the end. Problems uncovered by testing at such a late stage can cause costly redesign and delays. The idea of Shift Left is to involve testing teams earlier in the process and to think about testing at all stages of the process.

When the software development process is viewed as a left-to-right sequential process, doing testing earlier may be seen as "shifting it to the left". The term itself can be seen as a misnomer since we are not simply "shifting" but doing tests at all stages of the process.

Shift Left as a principle started with testing but it's beginning to be employed in other disciplines such as security and deployment.

## Discussion

### What exactly do we mean by the term "Shift Left"?

The principle of Shift Left is to take a task that's traditionally done at a later stage of the process and perform that task at earlier stages. An example of this is testing. With the traditional Waterfall model, testing is done just before releasing the product into production. This means that serious problems uncovered so late can cause major redesign and long delays. **Shift Left Testing** addresses this by involving testing teams early in the process. Issues, be it in design or code, can be solved early on before they become major. In fact, shift-left is less about problem detection and more about problem prevention.

Shift Left doesn't mean "shifting" the position of a task within a process flow. It also doesn't imply that no testing is done just before a release. It should be seen as "spreading" the task and its concerns to all stages of the process flow. It's about continuous involvement and feedback. In addition to the process changes, Shift Left is also about the people. For example, testing teams must now work more closely with development teams.

### What are the key benefits of Shift Left?

Shift Left identifies potential roadblocks and bottlenecks early on when there's still scope to change and improve design. Problems are addressed long before release without giving into pressure of an imminent deadline. There's obvious cost savings for the project since problems identified earlier are cheaper to fix. Shift Left reduces risk since many issues are addressed long before the release. Releases can be made faster with better quality.

Automation is essential. This reduces human errors, increases test coverage and lets testers focus on more inspiring tasks.

Since teams work closely, some level of whitebox testing can be done. It's also easier to estimate effort and plan for resources. Often debugging problems in production is hard and shift-left ensures most problems are caught much earlier, where they are easier to debug and fix.

### In what disciplines of software development is Shift Left applied?

Shift Left is being used in the following disciplines:

    + **Testing**: It's in testing that Shift Left started. It aligns with Agile practices such as *Test-Driven Development (TDD)* and *Behaviour-Driven Development (BDD)*. For example, by doing integration testing earlier, design can be changed more easily.
    + **Security**: Security is addressed right from the development stage. When combined DevOps, we get what we call *DevSecOps*. Security standards must be in place. Security folks must use the same tools as DevOps and be empowered to fix code on their own. Shift-lefting for security is particularly important for open source code.
    + **Deployment**: When apps are in the cloud, continuous deployment has become common practice to quickly deliver latest fixes and features. The use of templates, patterns and automation streamlines this. Building, provisioning and deploying of software is automated. In fact, continuous deployment enables continuous testing.
    + **Design**: Poor requirements can lead to a product not aligned with business needs. Going beyond requirements, the entire team can adopt a design thinking mindset. Everyone would have a shared vision of the product.

### Are there different types of Shift Left Testing?

Shift Left has been compared to a production line with raw materials on the left moving towards finished goods on the right. It's also been compared to kanban boards with completed tasks moving from left to right. In all cases, Shift Left is about getting involved at an early stage, which happens "to the left".

While shift-left testing is really about continuous testing, four types have been identified: traditional shift left, incremental shift left, Agile/DevOps shift left and model-based shift left. With **model-based shift left**, testing can begin even before a software implementation is ready. Tests are against executable requirements, architecture, and design models.

### How can I get started with Shift Left?

With regard to testing, developers can take up some testing tasks (unit tests for example) while testers need to learn to code. This will help them collaborate better and automate tests. They also need to use the same tools. By adopting TDD or BDD, a project will start with testability in mind. At the least, static testing plans, automation scripts and API tests can be written even before development starts. Test managers must move to new roles as shift-left testing could make them redundant.

Small iterative changes along with collaboration across teams, code reviews, automation and monitoring are all part of applying Shift Left. Workflows should be defined and followed. Configurations should be immutable and scripted, avoiding ad hoc changes. Give tools and dashboards to developers for insights into production. Developers should be able to see failures at all stages. It's important to be proactive rather than reactive.

Deployment procedures should be standardized so that the development and production environments are as close as possible. Use patterns to get consistent environments. Development and operations should work closely rather than in silos.

### What are some myths about Shift Left?

One of the myths is that testing is done by developers and hence QA teams will become redundant. In reality, QA teams will work more closely with development teams. Developers will be aware of testing needs. Testers will get insights into what's being developed. A related myth is that Shift Left is not aligned to Agile practices. This myth is also busted since greater collaboration between QA and development teams means that they can iterate faster.

Another myth is that the return on investment is not proven. In fact, studies have shown that fixing problems at a later stage is very costly.

Another myth is that there are no tools to aid in shifting left. In reality, Shift Left is about processes, people and tools. Automation tools play an important role. Same tools that can be used by multiple teams, including testing tools within a development environment, ease the implementation of Shift Left. Behaviour-Driven Development (BDD), Continuous Integration and Continuous Deployment rely on automation and help accelerate Shift Left.

### I've heard of Shift Right. Does it replace Shift Left?

No. Shift Right complements Shift Left. As of 2018, Shift Right is commonly used for testing.

Shift-left testing is about improving quality, testing often, shortening test cycles and automation. It's about checking for expected software behaviour. What about undefined, unknown or unexpected behaviours? What about performance, user experience or fault tolerance? Production environments with live traffic and real users are different from typical test environments. This is where shift-right testing comes in.

With **shift-right testing**, testing is done in production where you may find unexpected behaviour. It involves deploying small changes to production, monitoring closely and reacting quickly if failures happen. Software deployed for shift-right testing is not done and dusted. It has to be monitored and rolled back if needed. With shift-right testing, you can gather user feedback and improve test coverage.

Among the approaches to shift-right testing are duplicating live traffic for test purpose, A/B testing and canary deployments. **Chaos engineering**, implemented with tools such as Chaos Monkey, can be considered as a form of shift-right testing.

## Milestones

1950

In the 1950s, software is often developed and tested by the same people. It's therefore common practice to test continuously. There are no separate testing or QA teams, which come later.

1970

Winston Royce describes the different steps in building large software systems. In his description, testing happens towards the end of the development cycle. Royce himself clams that this is "risky and invites failure". Unfortunately, software teams adopted this approach while ignoring Royce's recommendations towards an iterative approach. This top-down unidirectional flow is later named the **Waterfall Model**.

1990

The 1990s is the decade when teams realize that the Waterfall model is not working out. The principle of Shift Left begins in this decade.

2001

The use of the term *Shift-Left Testing* appears in an article published at Dr.Dobb's. The author Larry Smith writes,

> Shift-left testing is how I refer to a better way of integrating the quality assurance (QA) and development parts of a software project.

Shift Left