# Programming Concepts for Beginners

## Summary

Computers operate at a low level of bits and bytes that are difficult for humans to understand. To program a computer, we use higher level languages that are closer to how humans think and reason. These programming languages make it easier to us to instruct what a computer should do.

While there are dozens of programming languages, all of them share a number of basic constructs. All languages must have constructs to do basic things such as initializing data, adding numbers, storing data, checking for certain conditions, repeating some tasks, and so on. This article introduces these basic concepts to someone new to programming.

The actual syntax of each language may be different. There are also different programming paradigms. These are not covered in this article.

## Discussion

### What's the difference between hardware and software?

Hardware refers to a physical device. For example, a computer and its peripherals such as monitor, keyboard, and mouse are all part of the hardware. Software is a set of instructions that are executed step by step to perform a specific task. Software is otherwise called *code* or *program*.

Although we think of desktops and laptops as computing hardware, any device that executes a program can be considered a computing device. Thus, digital cameras and smartphones contain both hardware and software. Some devices such as Wi-Fi routers and firewalls are called networking devices because of their specialized functions, but these too contain both hardware and software.

We may say that hardware is tangible and software is not. Some say that hardware is about *atoms* and software is about *bits*. This is because software is made of ones and zeros, which are the values that a bit can take.

Computing hardware without software does nothing and is therefore useless. Software needs hardware to do its job. Thus, both are needed for delivering a useful application.

### Why do we need programming languages?

Computers have been designed to work with ones and zeroes. They can't understand human languages. However, writing a program with ones and zeros is extremely hard for human programmers. Programming languages have therefore been invented for this reason.

Programs are written in **high-level languages** such as C, Java or Python. Such languages are independent of the hardware architecture on which they are meant to execute. This has the added advantage that the same program can execute on different hardware. There are tools called *compilers* or *interpreters* that translate high-level code into lower level instructions.

It's possible to write programs using hardware-specific instructions called **assembly-level languages**. In fact, compilers output assembly code. *Assemblers* translate them into machine code of ones and zeroes. Hand-crafted assembly code may be more optimized code (in terms of memory or speed). The trade-off is that such code can't be reused across different hardware architectures.

### What are the basic constructs of a programming language?

Here are some basic constructs of any language:

    + **Variables**: These are names for easy storage and access of data. A program is essentially about manipulating and storing data.
    + **Expressions**: Data needs to be manipulated and expressions involving mathematical operations are used to achieve this.
    + **Assignments**: The output of an expression, if needs to be stored into a variable, is saved using an assignment.
    + **Conditionals**: Program execution sometimes involves branching based on conditions. If some condition is true, do one thing; if not true, do something else.
    + **Loops**: When something has to be repeated a number of times, loops are useful. Loops typically have well-defined exit conditions.For example, `balance` is a variable to refer to a person's bank account balance. The expression `newbalance = balance - withdrawal` calculates new balance from two variables and assigns the result to another variable. The condition `if (balance > 0)` checks if the balance is positive. To calculate the sum of all withdrawals, stored in `all_withdrawals`, we would use a loop such as `for i in all_withdrawals: total_withdrawals += i`, assuming that `total_withdrawals` is initialized to zero.

### How is data stored and manipulated during program execution?

Content stored in memory is of two types: **instructions** that are executed by the computer; and **data** that are accessed and manipulated by instructions. Using memory addresses, instructions point to relevant data that need to be processed. Variables are simply names that translate to memory addresses where their values are stored.

Consider the statement, `a = a + b`, where "a" and "b" are two variables. In algebra, this equation implies that "b" must be zero. In computing, this is not an equation but an assignment statement. The right side is evaluated and the result is assigned to the left side. For example, if "a" and "b" start with values 3 and 4, after this execution, "a" will contain the value 7 while "b" will remain unchanged. In this example, we're are not concerned with preserving the original value of "a" and hence we overwrite it with the recent result. To preserve the original value, we can introduce another variable (memory location), `c = a + b`.

### What are the common types of data?

Numeric data is mostly either an **integer** or a **decimal**. Programmers can specify how much memory they wish to allocate to number variables using type names such as short, long, float, or double. More memory implies a bigger range of values.

Among the non-numeric types is the **character** type. A sequence of characters form a **string**. Thus, "Hello World" is a string of characters, enclosed in quotes. To store the result of conditionals, we have **boolean** types that have only two valid values: true or false.

It's common to deal with a sequence of data values. For example, we wish to store the names of all students in a single variable. We can do this by creating a **list** or **array** data type. Each item of the list can be accessed using an index. In computing, indices often (but not always) start with zero.

Since arrays accept only integer indices, we can create an **associative array** (also called **map** or **dictionary**) that accept strings as indices. This is useful when storing key-value pairs.

### What are conditional expressions?

Any expression that yields a boolean result of either true or false can be considered a conditional expression. For example, `balance > 0` is a numerical comparison that has a boolean result; `lastname == 'Smith'` is a string comparison that has a boolean result.

The purpose of such expressions is to do branching in code. For example, if account balance is positive, allow withdrawal; else, show customer an error message.

Conditional expressions can be combined using **logical operators**. The most common of these are the "and" and "or" operators. For example, `firstname == 'John' and lastname == 'Smith'` can be true only for John Smith. Another example is `balance <= 0 or accountDisabled` where `accountDisabled` is of boolean type. If either condition is true, we disallow account withdrawal. A value can be inverted using "not" operator.

When combining multiple boolean expressions, **Boolean logic** is used to determine the final result.

### What are the different ways of implementing loops?

There are three main looping constructs:

    + **do-while**: One or more statements are executed, then a condition is checked. If condition remains true, the next iteration of the loop is entered. If condition is false, loop is exited.
    + **while**: Similar to do-while, but the condition if checked first. This implies that statements within the loop may not be executed even once if the condition starts as false. This is unlike do-while where the statements execute at least once.
    + **for**: This starts with an initialization, such as initializing a loop counter. Then the condition if checked, and if true, loop statements are executed. Next, the loop counter is incremented before attempting the next iteration.Statements within the loop can be any valid programming construct. They can contain their own loops, resulting in *nested loops*. They can contain conditions, which if false, can help us exit the loop completely or skip following statements and move to the next iteration. These are usually called `break` and `continue` loop statements respectively.

### What are comments in the context of programming?

A programmer may write excellent code but he or she is most likely not the only person reading that code. Other members of the development team and managers will read the code. Code will be maintained and possibly improved by others. To make it easier for others to understand the code, it's good practice to write comments. The purpose of comments is to explain what the code does, particularly those parts that may not be obvious.

Comments are meant for humans and are ignored by compilers and interpreters. In other words, comments don't affect program execution. However, some tools process comments with the goal of automatically generating software documentation. This documentation is again useful for only humans. Some tools use comments to give hints and warnings to programmers while they write code. This enables programmers to catch problems early.

### What's the purpose of functions, classes and methods?

Modern applications are often complex, running into thousands or even millions of lines of code. The solution is to break up a large program into small manageable units, each of which connects with other units via well-defined interfaces.

A **function**, also called a *subroutine* or a *procedure*, is a block of code that encapsulates a specific functionality. It takes in input parameters, executes and returns an output. Thus, functions can be called from other parts of code with different inputs. They promote code reuse. For example, `float add(float a, float b)` is a function that takes two decimal numbers, adds them up and return the result as a decimal number. Calling `add(2.3, 1.2)` should return 3.5 as result.

A **class** encapsulates not only code but also data and expose well-defined interfaces. Functions encapsulated within classes are called **methods**. Consider `Rectangle` as a class with two data attributes `length` and `breadth`. One of its methods could be `float area()` that returns the rectangle's area as a decimal number.

### What tools does a programmer need to program a computer?

To write code, the programmer needs at least a text editor. A better approach is to use an **Integrated Development Environment (IDE)**. IDEs provide a useful set of features out of the box. These include language syntax highlighting, flagging programming errors, powerful code search, code formatting, and many more.

Code must be compiled or interpreted into their lower level instructions. Thus, programmers need to install **compilers** or **interpreters**. For example, C and C++ are compiled whereas JavaScript and Python are interpreted.

What if the program doesn't work as expected during execution? Where could be the problem? This is where a **debugger** becomes a useful. A debugger can help us inspect the value of a variable step by step during program execution. We can also ask the program to break at a specific line of code.

Code undergoes changes. To keep track of who has changed what, a **Version Control System (VCS)** such as Git is essential. In coding projects, it's common to use many other tools for planning, scheduling, tracking progress, or recording issues.

## Milestones

1843

Ada Lovelace translates a French article written by Luigi Menabrea on the *Analytical Engine* invented by Charles Babbage. She adds extensive notes to the translation detailing how the Engine might be instructed by a sequence of operations to solve problems. She is often regarded as the "world's first programmer".

1847

George Boole publishes *The Mathematical Analysis of Logic*. This shows the rules for manipulating expressions containing True and False values. This later influences the design of digital circuits and programming with boolean data types.

1943

At the Moore School of Electrical Engineering of the University of Pennsylvania, work begins on *ENIAC*. Unlike earlier electromechanical computers, ENIAC is electronic (no moving parts) and therefore more than thousand times faster. Possibly inspired by telephone switchboards, it's programmed using panel wiring and switches. It uses 18,000 vacuum tubes and weighs 30 tons.

1947

Herman H. Goldstine and John von Neumann invent a graphical method to represent the execution flow of a program. This is called a **flow diagram**, later named as *flowchart*.

1952

Mathematician Grace Hopper completes **A-0 compiler** that allows the use of English-like words instead of numbers to program a computer. This marks the beginning of the move towards high-level programming languages and compilers.

1968

Edgser Dijkstra publishes an article titled *Go To Statement Considered Harmful*. Goto is a programming construct that allows execution to jump to another part of the code. Dijkstra recognizes that excessive use of goto can result in confusing code. The alternative is to adopt **structured or procedural programming**. In modern programming languages, goto is not quite as bad since jumps are restricted.

1985

Bjarne Stroustrup publishes *The C++ Programming Language*. This makes C++ the dominant **object-oriented language** to tackle complex applications.

1990

This decade sees the invention of a number of high-level languages: Python (1991), Java (1994), and JavaScript (1995). Python and JavaScript are interpreted languages. Java is compiled to bytecode, which is then executed by the Java Virtual Machine.

Programming Concepts for Beginners


# Application Programming Interface

## Summary

The Application Programming Interface (API) is a technology that allows two software components to interact with one other without knowing how they're implemented.  The term "application" in API refers to any software that performs a certain activity. The agreement between two such software to interact is known as the "interface".

In any API, one entity provides the API via an endpoint and another entity consumes the API by making an API call. An API's documentation or specification will describe how to use it, how it is organised, and how it interacts with other APIs.

For instance, suppose each airline operator exposes its data via an API. An online ticketing portal gathers data from various airlines via these APIs and then shows aggregated data to end users. Users can compare prices, view schedules and book tickets.

## Discussion

### Why do we need APIs?

APIs connect different applications. It can connect two or more components from separate applications to perform some functions and share data. APIs are useful for building strong and robust applications since they allow developers to reuse capabilities while also reducing code size, which enhances user experience and application security.

Many businesses use API to provide a smooth experience of using backend architecture, allowing users to interact with them more easily. It's often shared with third-party developers so that they can integrate it with other services. It also reduces the issues of incompatibility and eliminates the need for recreating it for various platforms conversions which is expensive.

APIs provide flexibility to businesses by allowing them to easily connect with their new partners or provide new services from existing products. They can easily switch to create new products, which could potentially generate large profits.

Ultimately, APIs are designed for machines, not humans. API is not directly related to UI/UX. In the world of IoT and automation, APIs are becoming critical.  Siemens has patented the concept of human programming interfaces for future human-machine interaction in an automation environment.

### What is an API-first approach?

The concept of API-first started with the adoption of microservices in which applications are built as independent components and coupled together via API. Major companies such as Netflix, Amazon, and PayPal have moved to microservices architecture for their applications with great success.

API-first refers to giving APIs first priority in the development of software products. There's no widely accepted definition of the term API-first. This API-first strategy improves developer experience by introducing consistency, as they employ the same format and styles throughout the development. A style guide may be utilised by teams throughout since it documents the status code, error handling techniques, versioning strategies, and other API specifics. It ensures the consistency and reusability of all APIs.

The OpenAPI Specification is the most extensively used API specification language (also known as the Swagger Specification). While alternative API definition languages, such as RAML and API Blueprint, exist OpenAPI has become the standard practice.

### What are the characteristics of a good API?

We note three characteristics:

    + **Effective Documentation**: API documentation is a written reference for effectively utilising API functionalities. It should be simple for developers to grasp. A solid API documentation should be created for users who are unfamiliar with that specific API. It should also include examples describing usage of various functionality in an API created through documentation software.
    + **Error Handling**: Precise usage of error codes is must in an API. Error codes in responses help clients to understand the exact cause of the problem. Appropriate HTTP status code will also provide a positive user experience. Besides alerting them, these assist clients in fixing the problem. Generally, error codes 400-499 imply client-side errors, while 500-599 imply server-side errors.
    + **Authorization and Authentication**: Authorization is a process of evaluating access privileges to an endpoint or a method of an API, whereas authentication is the process of validating credentials like user session, id, or password provided through the request.

### What is meant by implementation hiding in API design?

In the restaurant example above, a customer interacts only with the waiter and not the cook. The waiter informs the cook the order, picks up the order when it's ready and serves it to customer. The customer has no idea what happened in the middle or how the meal was prepared by the cook.

Similarly in software, an ideal API will hide its implementation details. It will provide a generic solution without exposing implementation. This is called information hiding. API users don't care about inner workings. All they expect is simplicity of use. Furthermore, by concealing implementation details the API provider can subsequently alter or improve inner workings without impacting client applications that consume the API. This also builds trust with clients.

Sometimes an API fails to properly hide underlying implementation details. We call this a leaky abstraction, which is something to be avoided.

### What are the different types of API?

Categorised by use cases, we note the following:

    + **Operating System APIs**: These APIs ensure tasks run inside operating systems smoothly. It interacts with its own resources and services. These APIs will be specific to each operating system. Linux employs kernel-user space API, while Microsoft uses Windows API.
    + **Remote APIs**: Provider and consumer of the API are on different machines. They communicate over a local network and/or the internet. These APIs are typically developed in web standards. Java Database Connectivity API is a remote API in Java. Even though not all remote APIs are web APIs, they can be thought of as remote APIs.
    + **Web APIs**: These primarily use HTTP to receive requests/responses in the form of XML or JSON.  These are often synchronous and stateless.
    + **Database APIs**: These APIs are used as interface between database management systems and applications in order to query, read, write, or modify data on a remote database.  In general, backend APIs are asynchronous and stateful. They facilitate complex integrations.
    + **Language APIs**: Programming languages provide built-in APIs such as classes, methods, macros and even extensive libraries, such as Collections in Java and Standard Template Library (STL) in C++.

### What are the different types of release policies for an API?

Here are some API types organised by release policy rather than design style:

    + **External or Public API**: It's extensively used by third-party developers or users external to the organization. It's built with most robust technologies keeping high consumption in mind. Google Maps API is one example of Public API which is widely used by delivery apps and ride-sharing companies for directions and real-time location tracking.
    + **Internal or Private API**: It's for internal use typically within a team or organisation. It's a common type of API that is simple to develop as it doesn't require robust infrastructure for small consumer base. Because of recent practices like DevOps and microservice architecture, this type of API is essential.
    + **Partner API**: This is neither entirely private nor completely public. These APIs can be accessed by people outside the organisation or team but with special authorization. Examples include Twitter or AirBnB, where access is restricted to authorised users only.
    + **Composite API**: Composite APIs merge many API data requests into one. It saves data usage in applications by reducing the amount of API requests.

## Milestones

1960

Developers start building libraries to share procedural language capabilities with other developers. Because API has no formal definition, this is a conceptual application. These languages have subroutines, which are chunks of code that can be invoked and collaborated on by other programmers. It can be used without exposing the internal code of the libraries.

Dec
1968

In the paper titled *Data structures and techniques for remote computer graphics*, Cotton and Greatorex use the term **Application Program Interface** (without -ing suffix). This paper describes independent hardware for computer graphics system.

May
1981

In a dissertation, Nelson introduces a new method of API interaction called **Remote Procedure Call (RPC)**. It can communicate with a piece of code on another remote server, just like a local computer.

1988

**Portable Operating System Interface (POSIX)** standard is introduced to solve the portability issues. It defines an interface between hardware and operating system, specifying application programming interfaces (APIs) at the source level.

1990

Malamud, in his book *Analyzing Novell Networks* defines API as "a set of services available to a programmer for performing certain tasks."

Oct
1991

Object Management Group (OMG) defines the Common Object Request Broker Architecture (CORBA) as a standard. In CORBA, software components are treated as objects running on multiple systems and able to communicate with one another. An object could be written in any programming language. Objects are specified using an Interface Definition Language (IDL) syntax. The version 1.0 release supports C Language mapping.  CORBA uses Object Request Broker (ORB) to communicate between client and server applications.

Sep
1996

Microsoft introduces Distributed Component Object Model (DCOM), an improved version of the older Component Object Model (COM) for Windows 95. This enables communication between components like as ActiveX controls, scripts, and Java applets that exist on several machines on a LAN, WAN, or on the Internet through the use of Remote Procedure Calls (RPCs), allowing distributed communication across multiple networks.

Nov
1999

Microsoft, UserLand Software and DevelopMentor jointly release **Simple Object Access Protocol (SOAP)** which uses Extensible Markup Language (XML) syntax to send RPC messages over HTTP. SOAP links COM and DCOM objects and runs natively in Windows 95, 98 and NT. It can also connect with Internet Explorer and Java.

2000

Roy Thomas Fielding, a Ph.D scholar, introduces the concept of Representational State Transfer (REST) in his dissertation. It's a network-based architectural style that later popularizes the usage of web APIs.

Feb
2000

Salesforce launches its web API at the IDG Demo conference. It's a enterprise-class, web-based Salesforce automation offered as a "Internet as a Service." In December, eBay launches their API to a selected number of developers and partners.

Jul
2002

Amazon Inc launches Amazon.com Web Services API. This enables developers or third-party websites to display Amazon products in their websites in XML format.

Mar
2008

Apple releases iPhone 2.0 software. This includes the iPhone Software Development Kit (SDK). The SDK comprises a set of APIs and tools for developing iPhone and iPod applications. It also includes Core OS, Core Services, Media, and Cocoa Touch interfaces.

Feb
2015

Google open sources **gRPC**, a framework for handling remote procedure calls. The underlying transport protocol is HTTP 2.0. gRPC differs from HTTP in that it is based on the Remote Procedure Call (RPC) model, in which addressable "entities" of a collection are called procedures and the data is hidden behind the procedures. In contrast, in HTTP, addressable entities are called "data entities" and their behaviour is hidden behind data. Because gRPC is founded on HTTP 2.0 standards, it has many more capabilities that help speed up services while conserving resources such as energy and data.

Sep
2015

When Facebook developers attempt to build native Facebook implementations for iOS and Android instead of using the mobile-webview, which had some restrictions, they start to run into a lot of problems. They tried to use RESTful architecture, but it didn't meet their needs because they wanted the data to look like a graph of objects and models. They needed an API to display the newsfeed, which had previously been displayed as HTML. As a result, they create **GraphQL**, a new project that defines data query language in a new way to assist product designers.

Jun
2019

According to Programmable Web, an API research website, its API directory surpassed 22,000 APIs. Every month, an average of 220 APIs are introduced, representing a 30% increase over the previous four years. API adoption has been growing since 2005. API developers have begun to offer APIs in alternative architectures such as GraphQL and gRPC in addition to traditional REST, resulting in a larger directory.

Application Programming Interface


# Docker

## Summary

A container is a standard unit of software that packages an application with all of its dependencies and libraries. This makes it easy to deploy applications within containers regardless of hardware configuration or operating system. Unlike virtual machines, containers use the host OS rather than bundling an OS with the application.

Docker is a container platform. It provides a complete ecosystem of tools to build container images, pull these images from known registries, deploy containers, and manage or orchestrate running containers across a cluster of machines. Docker has popularized container adoption. Even the phrase "dockerize my app" has become common.

The word "Docker" usually refers to its engine. The commercial offering is called *Docker Enterprise*. There's also the free community edition that's simply called *Docker Engine*. Docker was initially available on Linux but later become available on MacOS and Windows as well.

## Discussion

### What are the benefits of using Docker?

Docker provides a consistent runtime across all phases of a product cycle: development, testing, and deployment. For example, if development team has upgraded one dependency, other teams must also do the same. If they don't, app may work during development but fail in deployment or work with unexpected side effects. Docker overcomes this complexity by providing a consistent environment for your app. Hence, it's become essential for DevOps practice.

Docker containers are smaller in size and boot up faster compared to VMs. They're also more cost efficient since many more containers than VMs can run on a machine.

Docker is open source. There's freedom of choice since any type of application (legacy, cloud native, monolithic, 12-factor) can run in a Docker container. Security is built into the Docker Engine by default. It's powered by some of the best components such as the *containerd*. There's also powerful CLI and API to manage containers. Via certified plugins, we can extend the capabilities of the Docker Engine.

### What do you mean by the term "Docker image"?

A Docker image is a read-only template from which containers are created. Here's a useful analogy: just as objects are instantiated from classes in object-oriented languages, Docker containers are instantiated from Docker images.

For example, your application may require an OS and runtimes such as Apache, Java or ElasticSearch. All these can be bundled into a Docker image. Thus, your app can have this exact runtime environment wherever it runs.

An image is built in multiple read-only layers. At the bottom, we might have bootfs and OS base image such as Debian. Higher layers could have custom software or libraries such as Emacs or Apache. This layering mechanism makes it easy to build new images on top of existing images. When an image gets instantiated into a running container, a thin writable layer is added on top, called **Container Layer**. This means that all changes go into the topmost layer. In fact, the underlying file system doesn't make a copy of the lower layers since they are read-only. This helps us bring up containers quickly.

### How do developers share Docker images?

A remote location where Docker images are stored is called the **Docker Registry**. Images are pulled from and new images are pushed to the registry. Without registries, it would be difficult to share and reuse images across projects. There are many registries online and **Docker Hub** is the default one. Just as developers share code via GitHub, Docker images are shared via Docker Hub.

Besides Docker Hub, here are some other registries: Gitlab, Quay, Harbor, and Portus. Registries from cloud providers include Amazon Elastic Container Registry, Azure Container Registry, and Google Container Registry.

A collection of images with the same name but different tags is called **Docker Repository**. A tag is an identifier for an image. For example, "Python" is name of the repository but when we pull the image "Python:3.7-slim", we refer to the image tagged with "3.7-slim". In fact, it's possible to pull by mentioning only the repository name. A default tag (such as "latest") will be used and only that image will be pulled. Thus, these two commands are equivalent: `docker pull rhel7` or `docker pull registry.access.redhat.com/rhel7:latest` .

### Which are the essential components of the Docker ecosystem?

The **Docker Engine** is a client-server app of two parts: the **Docker Client** and the **Docker Daemon**. Docker commands are invoked using the client on the user's local machine. These commands are sent to daemon, which is typically running on a remote host machine. The daemon acts on these commands to manage images, containers and volumes.

Using **Docker Networking** we can connect Docker containers even if they're running on different machines. What if your app involves multiple containers? This is where **Docker Compose** is useful. This can start, stop or monitor all services of the app. What if you need to orchestrate containers across many host machines? **Docker Swarm** allows us to do this, basically manage a cluster of Docker Engines.

**Docker Machine** is a CLI tool that simplifies creation of virtual hosts and install Docker on them.  **Docker Desktop** is an application that simplifies Docker usage on MacOS and Windows.

Among the commercial offerings are **Docker Cloud**, **Docker Data Center** and **Docker Enterprise Edition**.

### Which are the command-line interfaces (CLIs) that Docker provides?

Since Docker has many components, there are also multiple CLIs:

    + **Docker CLI**: This is the basic CLI used by Docker clients. For example, `docker pull` is part of this CLI with "pull" being the child command. These commands are invoked by user using the Docker Client. Commands are translated to Docker API calls that are sent to the Docker Daemon.
    + **Docker Daemon CLI**: The Docker Daemon has its own CLI, which is invoked using the `dockerd` command. For example, the command `$ sudo dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock &` asks the daemon to listen on both TCP and a Unix socket.
    + **Docker Machine CLI**: This is invoked with the command `docker-machine`.
    + **Docker Compose CLI**: This is invoked with the command `docker-compose`. This uses Docker CLI under the hood.
    + **DTR CLI**: Invoked with `docker/dtr`, this is the CLI for Docker Trusted Registry (DTR).
    + **UCP CLI**: Invoked with `docker/dtr`, this is the CLI for installing and managing Docker Universal Control Plane (UCP) on a Docker Engine.

### What are Volumes in Docker and where are they useful?

Containers are meant to be temporary. Any changes made to it at runtime are usually not saved. Sometimes we may save the changes into a new image. Otherwise, changes are discarded. However, there's merit in saving data or state so that other containers can use them. Volumes are therefore used for **persistent storage**.

A volume is a directory mounted inside a container. It points to a filesystem outside the container. We can therefore exit containers without losing app data. Newer containers that come up can access the same data using volumes. The important thing is to implement locks or something equivalent for concurrent write access. Volumes can be created via Dockerfile or Docker CLI tool.

Apart from sharing data or state across containers, volumes are useful for sharing data (such as code) between containers and host machines. They're also useful for handling large files (logs or databases) because writing to volumes is faster than writing to Docker's Union File System (UFS) that uses IO expensive copy-on-write (CoW).

### What's the purpose of a Dockerfile?

Dockerfile is nothing more than a text file containing instructions to build a Docker image. Each instruction creates one read-only layer of the image. When the container runs, a new writable layer is created on top to capture runtime changes.

Let's take the example of a Node.js application. The instruction `FROM node:9.3.0-alpine` specifies the base image of Node.js version 9.3.0 running on Alpine Linux. `ADD` instruction can be used to add files. `RUN` can be used for framework installation or application build. To expose ports from the container, use `EXPOSE`. To finally launch the app within the container, use the `CMD`.

For more details, read Dockerfile Reference and the best practices for writing Dockerfiles.

### What are some basic Docker commands that a beginner should know?

We describe some Docker commands listed in the official Docker documentation:

To build a new image from a Dockerfile use the `build` command. We can then push this to a registry using `push`. Commands `search` and `pull` can be used to find and download an image from registry to our local system. To create a new image from a running container's changes, we can use `commit`. To list images, use `images`. A downloaded image can be removed using `rmi`.

Once we have an image, we can create and start a container using `create` and `start`. Containers can be stopped using `stop` or `kill`. A running container can be restarted using `restart`. We can use `rm` to remove containers. The command `ps` will list all running containers. To list stopped containers as well, use "-all" option.

Commands that deal with processes inside containers include `run`, `exec`, `pause`, `unpause` and `top`. Commands that deal with container filesystem include `cp`, `diff`, `export` and `import`.

## Milestones

Mar
2013

Docker is debuted publicly and open sourced at PyCon, Santa Clara. Earlier work on Docker can be traced to 2010.

Sep
2013

Red Hat and Docker announce a collaboration around Fedora, Red Hat Enterprise Linux, and OpenShift.

Mar
2014

Docker replaces LXC with *libcontainer* as the runtime. The latter is written in Golang and developed by the Docker team.

Jun
2015

Docker, CoreOS and others form the **Open Container Initiative (OCI)**. At the same time, Docker's libcontainer becomes a standalone OCI-compliant runtime called *runc*.  In December 2015, Docker announces *containerd*, a daemon to manage runc.

2016

By this time, many companies including Cisco, IBM, Google, Microsoft, Huawei and Red Hat are main contributors to Docker. In June, Docker starts to support container orchestration with **swarm mode**.

Jan
2017

A YAML compose file `docker-compose.yml` can be used to deploy swarm mode services.

Mar
2017

Docker releases **Docker Enterprise Edition**. It renames the free open source product to **Docker Community Edition**.

Jul
2017

Version 1.0 of OCI runtime and image format specifications are released.

Mar
2018

Docker celebrates its fifth birthday. By now, there's an estimated 37 billion container downloads, 3.5 million dockerized apps and 450+ customers using Docker Enterprise Edition.

Docker


# IoT Security Model

## Summary

Since the Internet of Things (IoT) is not a standard, there's no single standardized approach to security. There are multiple IoT reference models defined by various stakeholders including ITU-T, Cisco, Intel, IBM, Microsoft, Symantec, and others. Security is often considered in these reference models. This article looks at some of these security models.

An IoT security model can be seen in two perspectives: (a) In a layered architecture, there's a security layer that spans the entire stack, from the connectivity layer at the bottom to the application layer at the top. (b) In an end-to-end solution, security is implemented at all points, from end devices to network to cloud.

Models provide a formal framework to implement security and evaluate the maturity of those implementations.

## Discussion

### How is security addressed within the IoT Reference Model?

For illustration, let's consider Cisco's reference model for security. It's a layered architecture with the lowest layers being device centric and the highest layers being more cloud centric. For control, information flows top to bottom. For monitoring, the flow is opposite. Security is considered at each layer. The lower layers focus on giving secure access to physical hardware while also providing a trusted environment for code execution. At higher layers, the focus is on identity management, authentication, analytics, and so on.

IBM's own model emphasizes the sensor gateway and IoT gateway where security is important. However, it too recognizes that security concerns the entire system. It identifies data security, device security, user security and application security.

Microsoft Azure divides the system into things (IoT devices), insights (data processing in the cloud), and action (business integration and machine learning). Security is a cross-cutting concerns across all three parts. We need secure provisioning of devices, secure connectivity, data protection during processing and storage, and secure user management.

### How can I apply threat modelling to secure IoT systems?

Threat modelling is an approach used in Microsoft Azure. In particular, they classify threats according to STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, and Elevation of privilege). A threat could fall under multiple categories at the same time.

When applied to IoT, the process involves modelling the application, enumerating threads, mitigating threats and validating the mitigations. Core elements of the threat model are processes, data stores, data flows, and external entities.

They also recommend partitioning the system into **zones**, each defined with its own data, authentication and authorization requirements. Typical zones include devices, field gateways, cloud gateways and services. Trust boundaries between zones are where STRIDE threats can occur. Focus must be on application features that are relevant to security and those that touch trust boundaries.

For instance, data going in an out of Azure IoT Hub or Event Hub must be protected at protocol level (eg. HTTP/MQTT/CoAP). At device level, read-only OS partition, signed OS image, strong authentication, memory encryption and Trusted Platform Module (TPM) are needed.

### What is meant shared security model for IoT?

The shared security model looks at IoT security from the perspective of stakeholders and their responsibilities. An IoT system has many components and interfaces. It can be secured only when everyone plays their part.

PTC ThingWorx, which is an IoT cloud platform, has identified the following:

    + **Product Vendor**: Securely test, release, document and support software. Alert customers on vulnerabilities. Train employees on best practices.
    + **Technology Partners & Systems Integrators**: Securely test, release, document and support extensions to the basic product.
    + **Public Cloud Providers**: Provide all elements of cloud security including hardware, software, networking, and access control.
    + **Administrators**: Manage identities and user privileges following the principle of least privilege. Scan networks for malware. Set script timeouts. Configure system in line with security policies.
    + **Users**: Use strong passwords. Regularly update local clients and devices. Avoid navigating to suspicious sites.
    + **IT Organization**: Meet regulatory requirements. Vet third-party software vendors. Train employees on product usage and best practices.

### I've heard of the IoT trust model. What is it?

The IoT trust model is inspired by how trust works in human relationships. When we trust someone, we're confident about their integrity, ability or character. The model asks if we can similarly trust IoT devices, services and data. The challenge with IoT is that a service may depend on a dozen others. Quantifying trust is not trivial, though Trust Network Analysis-Subjective Logic (TNA-SL) is one proposed method.

A trust model allows us to ask questions such as, "Can I rely on devices to defend against viruses? Can devices update themselves? If a device is compromised, do I take it offline or is there a service to do this? How are third parties using my data?"

Trust attributes become useful. A service may be marked as "child-safe". A piece of data may contain "GPS Location" or "Anonymized" attribute.

There's also a **human-centric trust model** that considers average users, not just system administrators. Users must find it easy to delegate access to devices and data, or determine who can be trusted and for what purposes.

Symantec takes the view that trust is established using cryptography, digital certificates and Certificate Authorities (CAs).

### How can I evaluate the security maturity level of IoT systems?

A maturity model can help determine if current implementations are secure enough. For example, for some IoT devices checking the diagnostic logs may be adequate. For others such as HVAC systems, real-time monitoring and control, and protection against malware are necessary. A maturity model provides a systematic approach to what should be protected, what measures are suitable and to what extent they should be applied.

The **Security Maturity Model (SMM)** proposed by the Industrial Internet Consortium (IIC) identifies five levels of maturity: Level0 (None), Level1 (Minimum), Level2 (Ad hoc), Level3 (Consistent), and Level4 (Formalized). It also looks at three scope levels: Level1 (General), Level2 (Industry specific), and Level3 (System specific).

At Level1 maturity, at least the goals are defined and basic measures are implemented. At Level2, use cases are identified. At Level3, best practices, standards, regulations and tools are included. At Level4, the entire process is defined. It's useful to note that,

> Maturity is about effectiveness, not the arbitrary use of mechanisms.

## Milestones

2011

Køien publishes a paper titled *Reflections on Trust in Devices: An Informal Survey of Human Trust in an Internet-of-Things Context*. He notes that IoT devices are varied and often operate in hostile environments. Neither software nor hardware can be trusted completely. We therefore need security measures to mitigate risks.

2013

Riahi et al. propose a systematic approach to IoT since traditional methods don't consider aspects unique to IoT. Their model considers four entities: persons, processes, intelligent objects, and the technological ecosystem. When these entities interact, there's tension among them. They consider the following tensions: identification/authentication, privacy, trust, safety, responsibility, reliability, and auto-immunity.

2017

NXP proposes a **secure distributed manufacturing model** that looks at how NXP, OEMs and ODMs can play their part in securing the supply chain. For example, OEMs generate their own salt that even NXP doesn't know but NXP processors will protect this salt. The model aims to streamline security at the chip and device level. The model brings transparency and trustworthiness.

Dec
2018

The ioXt Alliance crafts the **ioXt Security Pledge** consisting of eight principles. The Alliance was previously known as Internet of Secure Things. It brings together wireless carriers, manufacturers, standard groups, compliance labs and government organizations.

Jun
2019

ITU-T publishes the Y.4460 Recommendation, which details an IoT reference model. In fact, it describes three models based on device capability: low processing and low/high connectivity, and high processing and high connectivity. Security is not given due consideration and it's covered in a small paragraph.

May
2020

The Industrial Internet Consortium (IIC) publishes two white papers describing what it calls the **IoT Security Maturity Model (SMM)**. This is partly based on the Industrial Internet Security Framework (IISF) published by IIC in 2016. The SMM process is iterative and is based on Plan-Do-Check-Act (PDCA) cycle. The process starts with identifying a maturity target, assessing current level of maturity and working towards the target.

IoT Security Model


# Python Descriptor

## Summary

Python objects have attributes that can be accessed, modified or deleted. Python provides a default manner in which these operations are performed but what if we wish to customize these operations? This is where descriptors are helpful.

Descriptors can be used to manage access to an attribute, validate attribute values, customize error messages, track data corruption bugs, or perform dynamic computations on attribute lookup. Descriptors are commonly defined at class level but when defined at module level they can be used to display deprecation warnings or implement lazy loading.

It's been said that a good understanding of descriptors can improve a developer's coding skills. Descriptors are commonly used by library developers.

## Discussion

### What are some use cases of Python descriptors?

Suppose there's a `Directory` class that has a `size` attribute to reflect the number of files in that directory. Naturally, its value can change dynamically as files are added or removed. Without descriptors, we might implement this as a method `get_size()`. The method will query the file system and obtain the current size.

Descriptors allow us to get the size dynamically while also accessing the size as a data attribute. Thus, we can say `d.size` rather than `d.get_size()`. Moreover, the behaviour of querying the file system was previously within `Directory.get_size()`. With descriptors, the behaviour is nicely encapsulated within the descriptor class `DirectorySize`.

Another use case of descriptors is to manage data access. For example, public attribute `age` is exposed in a `Person` class but managed privately as `_age` within a descriptor class. The descriptor class could also perform validations. For example, when assigning a value to `age` it can check if the value is a non-negative integer.

A real-world example of a descriptor is `GenericForeignKey` in Django.

### How can I implement Python's descriptor protocol?

Assume one or more instances of TextWrapper class are instantiated as members of Email class. TextWrapper is the **descriptor class** and Email is the **owner class**. To support the descriptor protocol, TextWrapper implements the following methods (but not necessarily all):

    + `__get__(self, instance, owner=None)`: Called when the attribute of the owner class or its instance is accessed. Eg. `print(e.sender)`.
    + `__set__(self, instance, value)`: Called to set the attribute on an instance of the owner class. Eg. `e.sender = 'foo'`.
    + `__delete__(self, instance)`: Called to delete the attribute on an instance of the owner class. Eg. `del(e.sender)`.
    + `__set_name__(self, owner, name)`: Called when the owner class is created. This allows descriptor instances to access their own name as defined in the owner class.An object that has defined one or more of `__get__`, `__set__` or `__delete__` is a descriptor. Descriptors work only as class variables of the owner class.

The statement `e.sender += 'bar'` will automatically call both `__get__` and `__set__`.

### What's the difference between data and non-data descriptors?

A descriptor that defines `__set__` and/or `__delete` is a **data descriptor**. A descriptor that defines only `__get__` is a **non-data descriptor**.

Data descriptors override redefinitions in instance dictionary whereas instances can override non-data descriptors.

For a descriptor that doesn't define `__get__`, attribute access will return the value from the instance dictionary. If this doesn't exist, the descriptor object is returned.

Non-data descriptors are typically used for methods. In fact, Python methods are implemented as non-data descriptors, including those decorated with `@staticmethod` and `@classmethod`. Instances can override these methods. On the other hand, `property()` function is implemented as a data descriptor, meaning that instances can't override its behaviour.

### Why do we need the descriptor protocol when we have __getattr__, __setattr__ and __delattr__?

Methods `__getattr__`, `__setattr__` and `__delattr__` allow us to customize the behaviour of attribute access. These methods are defined on the owner class. If the owner class has many attributes to manage, these methods could become complex to maintain. Method implementations may have ugly `if-else` constructs to manage the different attributes.

Descriptors offer a more modular, maintainable and extensible approach. Instead of the owner class controlling how attributes should be managed, the control is now with the class of the attribute being managed. In other words, descriptors have a say on how their instances should be managed.

In the figure, we see how any value assigned to a name attribute is capitalized. Method `__setattr__` of Person class can be used for this but requires the attribute be named "name". The descriptor protocol offers a better way. Class CapitalizedValue is a data descriptor that does the capitalization. Person class defines an attribute of name "name" but any other name could have been given.

### How do Python descriptors work under the hood?

How Python descriptors work under the hood is basically about how the language does attribute access. Python applies a bunch of rules in a well-defined order of precedence. This is shown in the figure for both get and set/delete operations. These rules check whether class overrides the access methods `__getattribute__` and `__getattr__`, whether class or instance access is performed, and whether attribute is a descriptor object and what parts of the descriptor protocol are implemented.

We can see that for a get operation, data descriptor has higher precedence than instance attribute, which in turn has higher precedence than non-data descriptor.

We can summarize the lookups for instance and class attribute accesses:

    + **Get**: `obj.x` ⇒ `Cls.__dict__['x'].__get__(obj, Cls)`and `Cls.x` ⇒ `Cls.__dict__['x'].__get__(None, Cls)`
    + **Set**: `obj.x = v` ⇒ `Cls.__dict__['x'].__set__(obj, val)` and `Cls.x = v` ⇒ Regular override
    + **Delete**: `del obj.x` ⇒ `Cls.__dict__['x'].__delete__(obj)` and `del Cls.x` ⇒ Regular deletion

### What other features of Python are implemented by descriptors?

Built-in functions `classmethod()`, `staticmethod()`, `property()`, and `functools.cached_property()` are all implemented as descriptors.

A **property attribute** is one that triggers method calls when accessed. Property is implemented as a data descriptor.  It has an easier interface than a descriptor and a different abstraction. The methods are typically defined in the same class in which the attribute resides. There are two ways to link methods to a property: (i) using the `property(fget=None, fset=None, fdel=None, doc=None)` built-in function; (ii) using decorators `@property`, `@x.setter` and `@x.deleter` where `x` is the name of the property and also the name of the methods so decorated.

Another Python feature is **slots**, available as a class variable `__slots__`. This variable can be assigned all allowed attributes of the class and its instances. Thus, instances can't add new attributes. Instances also don't contain `__dict__` and `__weakref__` unless explicitly included in `__slots__`. Every item in `__slots__` is implemented as a descriptor.

### Could you share some tips for working with Python descriptors?

For most use cases, or at least for beginners, property is a simpler interface than defining a descriptor from scratch. In fact, property has been called a "high-level descriptor builder" or a "descriptor factory".

A read-only data descriptor can be constructed by defining both `__get__` and `__set__` but the latter is coded to raise an `AttributeError` exception.

We can use a descriptor to implement a one-time initialization. This is done by setting an internal flag in `__set__`.

Descriptors enable many other use cases elegantly. Logging all accesses of an attribute can be implemented by a descriptor. When an attribute changes, we can update all dependent attributes, such as, updating area when circle radius changes. Lazy evaluation is another use case in which behaviour is invoked only when necessary.

## Milestones

Jan
1994

Python 1.0 is released. Descriptors don't exist in this initial release.

Dec
2001

Python 2.2 is released. This introduces **descriptors** into the language via PEP 252. Descriptors themselves have the following attributes: `__name__`, `__doc__`, `__get__()`, `__set__()` and `__delete__()`. Descriptors are now used to support static methods and class methods. New features slots and properties are also new kinds of descriptors.

Jun
2011

Python 2.7.2 is released. In the accompanying documentation, Hettinger states in the *Descriptor HowTo Guide* that,

> Learning about descriptors not only provides access to a larger toolset, it creates a deeper understanding of how Python works and an appreciation for the elegance of its design.

Feb
2015

In PEP 487, Teichmann proposes the use of `__set_name__()` initializer for descriptors. While a descriptor knows its owner when `__get__()` is called, it doesn't know its own name, that is, the instance name that the owner has defined. Method `__set_name__()` solves this problem. This feature is accepted for Python 3.6 (December 2016).

Sep
2017

In PEP 549, Hastings proposes **instance descriptors**, since currently Python supports descriptors only as members of the type of an object. This proposal is rejected in preference to a simpler proposal by Levkivskyi in PEP 562. The idea is to support `__getattr__()` and `__dir__()` at module level. These allow customization of module attribute access. Two use cases for this are deprecation warnings and lazy loading. This feature is accepted for Python 3.7 (June 2018).

Aug
2018

At PyCon Australia, Egan reveals his analysis of the use of the descriptor protocol in public GitHub repositories. Method `__get__()` is implemented 1.2 million times. Method `_set_name__()` is implemented the least since it's been around only since Python 3.6.

Python Descriptor


# LoRa

## Summary

For the Internet of Things (IoT), many devices are likely to be constrained in terms of cost and power. For many use cases, devices require only a low data rate but long range. Cellular technologies, Wi-Fi or Bluetooth don't cater well to these use cases. This is where **LoRa (Long Range)** becomes relevant.

LoRa devices are low power, long range devices. They transmit and receive data over unlicensed frequency spectrum. A typical use case is to transmit low-rate sensor data. There are plenty of LoRa deployments worldwide, offered as free or paid services.

LoRa is one particular technology in the domain of Low-Power Wide-Area Network (LPWAN). Alternatives to LoRa include Sigfox and NB-IoT.

## Discussion

### What led to the wider deployment of LoRa?

Cellular networks such as 2G and GPRS were used for machine-to-machine (M2M) data communication, particularly for sending data from remote locations. Such networks used relatively less power compared to 3G or LTE. But by the start of 2017, network operators such as AT&T (US) announced the termination of 2G and GPRS.

3GPP's own alternatives for M2M communications were LTE-M and NB-IoT. However, these were not expected to be ready until early or mid-2018. This created a void following the sunsetting of 2G. There was a need for a protocol that catered to low power, long range and bi-directional communication devices. This led to the wider deployment of LoRa network.

LoRa doesn't need costly spectrum licensing fees typical of cellular networks. By 2013, the fundamental technologies powering LoRa were patented and owned by Semtech. LoRa Alliance was formed in early 2015 to address the market need, while also standardizing and promoting LoRaWAN that was built on top of LoRa.

### What are the advantages of LoRa?

LoRa has the following advantages:

    + Long range: Many miles on line-of-sight links.
    + Low power: Can run on battery for years.
    + Low cost: LoRa modules are pocket-friendly. It uses constant envelope modulation that brings lower cost and higher efficiency to the power amplifier.
    + Universal: Uses unlicensed bands that are globally available.
    + Bi-directional: Can send and receive data.
    + Network scalability: Easy to scale as it supports millions of messages per station. A single gateway can support thousands of end devices.
    + Easy commissioning: Easy to deploy in an existing network.

### What are the typical use cases of LoRa?

LoRa is suitable for rural use due to its long range. In urban areas, where signals have to penetrate floors and walls, LoRa is suitable. Among its application areas are smart cities, smart homes and buildings, smart agriculture, smart metering, and smart supply chain and logistics.

A typical use case is **smart metering**. For example, LoRa-enabled meters will increase efficiency and optimize processes. For water management, sensors will monitor water pressure, water level or detect leaks.

In **smart buildings**, the typical use of LoRa is for smoke detection, asset and vehicle tracking, room usage and more. Asset tracking is possible because LoRa works well even when devices are in motion.

Since LoRa is bidirectional, applications can benefit from **command-and-control** functionality. Uplink can be used for continuous monitoring and downlink can be used for controlling the device.

LoRa when combined with Wi-Fi can optimize a number of IoT use cases. For example, asset tracking, location services and on-demand streaming can be improved.

### Could you share some technical details of LoRa?

LoRa operates in different bands in different parts of the world. In the USA it operates at 902-928 MHz. In Europe, it operates at 863-870 MHz. In China, the spectrum is 470-510 MHz and 779-787 MHz. In India, it operates at 3 channels: 865.0625 MHz, 865.4025 MHz, 865.9850 MHz. In many Asian countries, the spectrum is 920-923 MHz or 923-925 MHz. Australia uses 915-928 MHz. Typically, downlink channels are higher than uplink.

Data is spread with a chip sequence, thus spreading signal energy over a wider bandwidth. LoRa has six spreading factors, SF7-SF12. Larger the spreading, longer the range and smaller the bit rate. Spreading factors are orthogonal, which means that signals with different spreading factors can coexist. In North America, there are 64 125 kHz uplink channels, 8 500 kHz uplink channels, and 8 500 kHz downlink channels. For same SF, lower bandwidth implies better sensitivity.

Semtech's SX1276/77/78/79 transceivers achieve 168 dB maximum link budget. Sensitivity is -148 dBm. Bit rate can be programmed to up to 300 kbps. They consume receive current of 9.9 mA and 200 nA for register retention.

### What is the range of LoRa?

LoRa has a typical range of 2-8 km with the higher range achieved when the spreading factor (SF) is larger. Larger spreading factor also implies a lower bit rate. Moreover, this range can also vary based on the terrain.

In an urban environment with an outdoor gateway 2-3 km is the typical range. In rural areas, this can be 5-7 km.

In an experiment using a weather balloon, a range of 702 km was achieved. A LoRa device was attached to a helium-filled balloon that rose to an altitude of 38.772 km. Packet sent from the node was received by 148 different gateways connected to The Things Network. One of the gateways at a height of 30 meters was located in Wrocław, Poland. By that time, the balloon was flying over Osterwald, Germany. The distance was 702.676 km. Transmit power was 25mW (14dBm), which is about 40 times smaller than what a mobile phone uses.

### Could you share some details about how LoRa works?

LoRa uses **Chirp Spread Spectrum (CSS)** that spreads the original signal to a larger bandwidth by continuously varying the frequency. Data signal is spread or chipped to a higher rate. This then modulates the chirp carrier signal. Due to this method, LoRa doesn't require a highly accurate reference clock, thus dropping the cost on receiver design. The signal also tolerates Doppler frequency shifts, making LoRa suitable for mobile devices. Due to spreading, processing gain is achieved, making LoRa resilient to multipath and fading.

A LoRa PHY frame has repeated upchirps for the preamble, two downchirps for start of frame delimiter (SFD), and choppy upchirps of varying lengths that signify data. Instantaneous frequency changes are due to data being modulated onto the chirps. A LoRa receiver will identify the preamble and SFD. To extract symbols, it will de-chirp with locally generated signal and take FFT of de-chirped signals. FFT length corresponds to number of possible symbols, which is twice the spreading factor.

### How is LoRa different from LoRaWAN?

LoRa is proprietary and patented by Semtech Corporation. LoRaWAN is built on top of LoRa and is standardized by LoRa Alliance.

LoRa defines the PHY layer for low-cost, low-power, and long-range communication. LoRaWAN defines the MAC layer for bidirectional communication, mobility and localization services. LoRaWAN addresses the network architecture: how devices connect to gateways, how gateways process the packets, and how packets make their way to network servers and application servers. LoRaWAN also defines three different device types (class A/B/C) to suit different power needs.

While it's typical to use LoRaWAN with LoRa, there are alternatives to LoRaWAN. For example, Link Labs' Symphony Link is an alternative. They claim that LoRaWAN is suitable for public networks but for private networks serving industrial applications Symphony Link is better.

### Where has LoRa been deployed?

LoRa is widely deployed in Europe, the US and Asia. By mid-2018, LoRaWAN was deployed in 95 countries.

Back in 2015, National Narrowband Network and Telstra did independent LoRa trials in Australia. In November 2015, KPN's LoRa network went live in Rotterdam and The Hague in the Netherlands. In July 2016, it achieved nationwide coverage, making Netherlands the first country in the world to have this capability.

In the US, MachineQ, a Comcast company, is providing LoRaWAN-based access network for IoT applications. In May 2018, it connected 10 US cities to the network.

In June 2019, Kerlink and Tata Communications Transformation Services announced a partnership to deploy LoRaWAN globally. In fact, they had already deployed LoRaWAN in 40 Indian communities and cities. The network was reliable across three monsoon seasons. In December 2019, SenRa announced LoRaWAN coverage in 60 Indian cities and plans to expand to 100 cities by end of 2020.

Taiwanese company Kiwi Technologies has enabled LoRaWAN deployments in China, Taiwan, Indonesia, Thailand, and other countries.

### Who supplies LoRa chipsets and modules?

The first LoRa commercial chipsets were manufactured by Semtech. These are SX12XX series chips.

Now there are many semiconductor companies such as Microchip and Murata who are manufacturing the chips compatible with their microcontrollers. As an example, Murata's CMWX1ZZABZ LoRa module integrates Semtech's SX1276 transceiver with STM32L0 Series MCU from STMicroelectronics. ST's LoRaWAN SDK can be used to program the MCU.

LoRa Alliance maintains a list of products that are certified for LoRaWAN. These products include system-in-packages, sensors, and modules.

### What are some useful resources to learn about or use LoRa?

LoRa Alliance's Resource Hub is the place to download the LoRaWAN specification. There are also case studies, FAQ, presentations and white papers.

LoRa Developer Portal from Semtech is a useful site to visit. It has useful case studies, technical documents, datasheets, user guides, and more.

Bastille Research has open sourced gr-lora that's a GNU Radio out-of-tree (OOT) module implementing LoRa PHY. Although LoRa is proprietary, Matt Knight did blind signal analysis that led to gr-lora.

There are many tools available online for operating LoRa-based networks. LoRatools provides tools such as Key generator, Thingpark import helper, Hex helper and Airtime calculator. For data visualisation, open source visualisers such as Kibana can be used.

### What are some criticisms of LoRa?

LoRa is not open source. The technology is patented by Semtech Corporation, who release the chipsets. Although the patents describes how LoRa works, independent analysis has shown that real-world implementation differs in many ways from what's disclosed in the patents. In particular, symbol gray indexing, data whitening and interleaving differ.

The security of LoRa at PHY layer is probably not mature. It's possible to inject and mask malicious traffic that higher layers don't detect. Many of the exploits known to IEEE 802.15.4 can be used here.

Accurate LoRa localization is hard to implement, at least with time difference of arrival (TDOA). Unless there's near line-of-sight, it's hard to detect the direct path. Since LoRa bandwidth is only 125 kHz, a receiver can separate different multipaths only if they differ by at least 2.4 km.

It may be said that compared to NB-IoT, LoRa has lower data rate and higher latency. However, LoRa also gives longer battery life than NB-IoT. Therefore, this is more of a trade-off that may benefit many IoT applications.

There are also limitations of LoRaWAN that are not discussed here.

### Could you name some alternatives to LoRa?

LoRa has the following alternatives:

    + **Sigfox**: Proprietary technology from a French company. Low cost and long range. Meant mainly for uplink. Downlink is limited. Many deployments in Europe but only few in the U.S. An indoor tracker with Sigfox can last for 5+ years on a single AA cell.
    + **Narrowband IoT (NB-IoT)**: Cellular-based technology optimised for low power and long range. Compared to LoRa, NB-IoT has the advantage of being an open standard in an already mature ecosystem for mobile networks with support from telecom equipment vendors. Operates in licensed spectrum. Related cellular standards for LPWAN include EC-GSM-IoT and LTE Cat-M1.
    + **Random Phase Multiple Access (RPMA)**: Proprietary technology developed by Ingenu, previously called On-Ramp Wireless. Superior uplink and downlink capacity. Better link budget than LoRa or Sigfox. Operates in 2.4 GHz.
    + **Weightless**: Open standard in sub-GHz unlicensed spectrum. Three variants: Weightless-W, Weightless-N and Weightless-P.

## Milestones

1899

Guglielmo Marconi experiments with frequency-selective reception in an attempt to reduce interference.

1942

Actress Hedy Lamarr and composer George Antheil are awarded U.S. Patent 2292387 for frequency hopping. Their patent application is titled *Secret Communications System*. During World War II, U.S. military uses frequency hopping methods. Years later, this technology becomes important for commercial wireless communication.

2008

First patent of LoRa is filed by French company Cycleo SAS. This uses a technology called **Chirp Modulation**. The U.S. Patent US7791415 is titled *Fractional-N Synthesized Chirp Generator*. The figure shows an example where down-chirp rate is twice the up-chirp rate. Output frequency \(f\_o(t)\) is determined by frequency ramp control word \(R(t)\) and frequency control word \(M(t)\).

Mar
2012

Semtech acquires Cycleo SAS for $5M. Semtech is a company involved in manufacturing of analog and mixed signal semiconductors for various industries.

Feb
2013

Second patent of LoRa is filed by Semtech. Patent EP2763321 titled *Low power long range transmitter* describes the use of chirp modulation to transmit low-power signals over long distances. A typical data frame has preamble and data. Preamble has unmodulated chirps (411), symbols for frame synchronization (412), symbols for frequency synchronization (413), silence (420) and further unmodulated symbols (414). Data has header (415) and payload (416).

Jan
2015

LoRa Alliance releases version 1.0 of **LoRaWAN™ Specification**. The authors are from Semtech, IBM and Actility. Version 1.1 of the specification comes out in October 2017.

Mar
2015

**LoRa Alliance** becomes formally operational. It's mandate is to standardize LPWAN around LoRaWAN specification, and oversee a certification and compliance program for better interoperability. By mid-2020, the alliance has more than 500 members. These include technology leaders (IBM, Cisco, HP, Foxconn, Semtech, Sagemcom), product companies (Schneider, Bosch, Diehl, Mueller), SMEs and startups.

Jul
2016

Netherlands becomes the first country in the world to have nationwide LoRaWAN coverage.

May
2018

Google Cloud joins LoRa Alliance. Given the recent growth of LoRa and LoRaWAN, Google Cloud hopes to "simplify the process of developing, deploying, and managing IoT solutions".

Jun
2018

Semtech Corporation demonstrates LoRa technology at Mobile World Congress Shanghai. It claims that LoRa deployments have reduced water leaks by 25% in commercial buildings, reduced energy costs by 30% in smart homes, and saved 20% costs in gas metering.

LoRa


# Google Kubernetes Engine

## Summary

Google Kubernetes Engine (GKE) is a managed environment for deploying, managing and scaling containerized applications using the Google Cloud Platform infrastructure. The environment that Google Kubernetes Engine provides consists of multiple machines, specifically Google Compute Engine instances, which are grouped together to form a cluster. Google Kubernetes Engine draws on the same reliable infrastructure and design principles that run popular Google services and provides the same benefits like automatic management, monitoring and liveness probes for application containers, automatic scaling, rolling updates, and more.

## Discussion

### Why should I use Kubernetes?

Before managing applications with Kubernetes and GKE, you should know how containers work and what advantages they provide. If you were to build an online retail application with user sign-in, inventory management, billing and shipping, you could break up your application into smaller modules called **microservices**. These are isolated and elastic, for high-availability and scalability.

**Containers** provided the environment to deploy microservices. You can run them on the same or even different machines, start and stop them quickly, but you can't specify how many machines or containers to keep running. What to do if containers fail? How to connect a container to other containers and enable persistent storage? For these aspects, you need a container orchestration system like Kubernetes.

**Kubernetes** is an open source orchestrator for a container environment. It provides the ability to define how many machines to use, how many containers to deploy, how to scale them, where the persistent disks reside, and how to deploy a group of containers as a single unit.

### Since Kubernetes is already open source, what's the need for GKE?

When you use the GKE to setup a cluster, you also gain the benefit of advanced cluster management features that Google Cloud Platform provides. These include:

    + Leverage CI/CD tools in GCP to help you build and serve application containers
    + Use Google Cloud Build to build container images from various source code repositories
    + Use Google Container Registry to store and serve your container images
    + Load balancing for Google Compute Engine instances
    + Node pools to designate subsets of nodes within a cluster for additional flexibility
    + Automatic scaling of your cluster's node instance count
    + Automatic upgrades for your cluster's node software
    + Node auto-repair to maintain node health and availability
    + Logging and monitoring with Operations (formerly Stackdriver) for visibility into your cluster

### How does GKE work?

A **cluster** is the foundation of GKE. The Kubernetes objects that represent containerized applications all run within the cluster. A cluster consists of at least one *cluster master* and multiple worker machines called *nodes*, which are the worker machines that run containerized applications and other workloads. The worker machines are Google Compute Engine (GCE) instances that GKE creates on your behalf when you create a cluster.

These master and node machines run the Kubernetes cluster orchestration system. The cluster master runs the Kubernetes control plane processes, including the Kubernetes API server, scheduler, and core resource controllers. The API server process is the hub for all communication for the cluster. All internal cluster processes (such as the cluster nodes, system and components, application controllers) all act as clients of the API server; the API server is the single "source of truth" for the entire cluster.

### What are some criticisms of GKE?

GKE is still considered new. Documentation and support could be improved. It's also not well integrated into the rest of Google Cloud Platform.

On a more technical note, some complain that the configurability of GKE is limited, including limited control on authentication, authorization or admission control. It's expected that future versions of GKE will give users more control, such as via `ConfigMaps`.

## Milestones

Jun
2014

The first public commit of **Kubernetes** code is made in GitHub. A few days later, it's released to the world at DockerCon.

Jul
2015

Version 1.0 of Kubernetes is released at Open Source Convention (OSCON). With Kubernetes as the its first project, **Cloud Native Computing Foundation (CNCF)** is founded.

May
2018

Google offers its own **Google Kubernetes Engine (GKE)** as a service for cluster management. Similar services from Microsoft and Amazon follow.

Jul
2018

As an alpha release, Google releases **GKE On-Prem** so that enterprises can deploy GKE in their own data centers. This allows Google to offer a hybrid cloud platform.

Oct
2018

It's now possible to do container-native load balancing for GKE-based applications. Previously, load balancers worked at the granularity of VMs. These VMs then used IPTable rules to route workloads to the right pods. This was suboptimal and resulted in traffic hops between nodes. With **Network Endpoint Groups (NEGs)**, load balancing can now be done based on the availability and health of pods rather than nodes.

Google Kubernetes Engine


# Cloud Computing

## Summary

Cloud computing is a model in which resources are remotely accessed via the internet, shared by many users, and automatically allocated or released to meet current requirements. Resources could be compute, storage, memory or networking. Consumers subscribe to a service rather than purchase a dedicated resource. Thus, service abstractions are more important than physical resources or implementations.

Those who provide cloud computing platforms are called *cloud providers*. Those who avail them are called *cloud consumers*. Cloud providers simplify the work of consumers by providing and managing essential resources. Consumers can therefore avoid upfront capital expenditure and focus on building their business applications. Providers also offer different service models and cloud configurations to suit different consumer needs.

Cloud computing brings many benefits not possible with traditional models of computing. This is prompting businesses to move to the cloud.

## Discussion

### Why do we need cloud computing when there are data centres and web servers?

Data centres and web servers are components of internet computing. A client device could access a remote web server via the internet and avail its services. Data centres brought economies of scale by bringing together many such servers, storage devices, firewalls, routers, and switches. These advances made it possible for enterprises to share data and applications more effectively. However, applications were deployed on a single server or sometimes a cluster of servers. The deployment was fixed. To handle higher loads, either servers had to be upgraded or more servers had to be added manually.

Cloud computing changes all this. Configurations are not fixed. A cluster of servers share the load. Servers are added or removed automatically to serve the current load. Thus, cloud computing exposes the abstraction of a service and hides the underlying physical implementations. Failure of a single server will not bring down the service.

In short, cloud computing brings better flexibility, scalability and reliability that was limited with internet computing.

### What are the characteristics of cloud computing?

The National Institute of Standards and Technology (NIST) identified in 2011 the following characteristics of cloud computing:

    + **On-Demand Self-Service**: Without help from the cloud provider, consumers can provision and monitor resources on their own.
    + **Broad Network Access**: Any device from mobile phones to powerful workstations can access cloud capabilities over the network.
    + **Resource Pooling**: Resources are shared among many consumers, making it a multi-tenant model. Consumers have no knowledge of the exact location of these resources but may be able to select the country, state or data centre.
    + **Rapid Elasticity**: Resources can be scaled on demand. Scaling can be vertical or horizontal. Scaling can be manual but often semi-automated or fully automated.
    + **Measured Service**: Provider monitors the use of resources. This is used for billing and optimal use of resources.Gong et al. identified the following additional characteristics: service-oriented model that provides the right abstractions; loose coupling between client and server, and among applications running on the same platform; virtualization that enables loose coupling; fault tolerance; pay-per-use business model; and ease-of-use via web interfaces.

### What are some myths about cloud computing?

Some myths have persisted since the early days:

    + **Security**: On-premises data centres are perceived to be more secure. In reality, cloud platforms implement best-in-class security principles and are maintained by experts. However, security is really a shared responsibility between the cloud vendor and the user.
    + **Cost**: The cloud model is not necessarily costlier. There are "pay-as-you-go" options. Moving to the cloud isn't only about cost savings. Cloud computing opens up other benefits such as business agility.
    + **Control**: Cloud doesn't result in loss of control or vendor lock-in. Different service and deployment models are available.
    + **Migration**: Moving to the cloud might have caused service downtimes in the past. Today, cloud providers offer seamless migration with minimal downtime.
    + **Value**: Cloud isn't just for big companies. Smaller companies may benefit more by saving on infrastructure and its operations. Another myth is that cloud enables only storage, analytics and IT services. Cloud can streamline operations across all business functions including customer service, sales, marketing, manufacturing, and more.
    + **Strategy**: Moving to the cloud doesn't equal digital transformation. It's only the first step. Applications may need to be refactored to leverage the cloud. Strategy should relate to business goals.

### Which are the different cloud deployment models?

Cloud computing offers three main deployment options:

    + **Public**: The dominant one. When a user or an enterprise moves to the cloud, their complete infrastructure is fully managed by the cloud provider. Consumers need not purchase or maintain their own infrastructure. It's called "public" because anyone can create an account with the provider. Resources are shared with other users.
    + **Private**: Infrastructure that's dedicated to one consumer. It may be infrastructure belonging to the consumer or rented/leased from a public cloud provider for exclusive use. Existing data centres could become private clouds if operated on the principles of cloud computing, such as virtualization and efficient resource management.
    + **Hybrid**: Combination of both public and private clouds. Consumers might retain critical functions in a private cloud while migrating others to a public cloud. Regulatory compliance and data protection needs make hybrid or private clouds a necessity.Variations of the above include community cloud (a community of users sharing a private cloud), multicloud (using multiple clouds for redundancy),  and polycloud (distributing workloads across multiple clouds).

### What are the different service models in cloud computing?

Cloud computing three main service models:

    + **Infrastructure-as-a-Service (IaaS)**: Cloud vendor provides shared or dedicated resources for compute, storage and networking. Operating system, middleware, applications, and many others are under the control and management of the consumer. Thus IaaS gives consumers more flexibility but at the expense of more work to configure and operate. This is somewhat similar to how traditional IT departments manage enterprise infrastructure.
    + **Platform-as-a-Service (PaaS)**: Developers are the main beneficiaries of this model. The cloud provider manages the hardware and virtualization as in IaaS but also provides OS, middleware and application platforms. This simplifies the work for developers who can focus on building and managing their apps.
    + **Software-as-a-Service (SaaS)**: For end users, the entire application is delivered from the cloud. The cloud provider manages all parts of the service from the hardware to the application. Users can access the application via a web browser without installing any dedicated software on their systems. Gmail is an example of SaaS.

### What some specialized service models in cloud computing?

The early service models of IaaS, PaaS and SaaS are very much relevant today. However, these have also been specialized to create niche cloud offerings. Nowadays, many services are being offered remotely on the cloud without customers having to manage them in-house. We briefly describe a few of these:

    + **Backend-as-a-Service (BaaS)**: Cloud providers take care of storage, database management, user authentication, notifications, hosting, and more. With server-side application logic, BaaS offers more that what PaaS offers. Developers can focus on the frontend user interface and client-side application logic. When applied for mobile applications, it's called Mobile Backend-as-a-Service (MBaas).
    + **Containers-as-a-Service (CaaS)**: A subset of IaaS for managing containers and virtualized applications. A further specialization is Kubernetes-as-a-Service (KaaS) if containers are managed using Kubernetes.
    + **Function-as-a-Service (FaaS)**: Enables execution of modular units of code (called functions) in response to events. Resources are allocated and billed for only when used, unlike IaaS or PaaS.
    + **Machine Learning-as-a-Service (MLaaS)**: Provides data scientists and developers ready-to-use ML models and tools to manage or execute those models.Others include Database-as-a-Service (DBaaS), Security-as-a-Service (SECaaS), Identity-as-a-Service (IDaaS), Storage-as-a-Service (STaaS), Test Environment-as-a-Service (TEaaS), Desktop-as-a-Service (DaaS), and many more.

### Who are the main vendors of cloud computing platforms?

Well-known vendors of public cloud include Alibaba Cloud, Amazon Web Services (AWS), Google Cloud, IBM Cloud, and Microsoft Azure.  Canalys estimated that in Q3 2021, 61% of total cloud spending went to just three vendors: AWS, Microsoft Azure and Google Cloud.

Many of the public cloud vendors also offer private/hybrid clouds and software stacks that enable the implementation of private/hybrid clouds. Examples include Red Hat OpenStack, Rackspace, IBM Bluemix Private Cloud, Microsoft Azure Stack, VMware Private Cloud, Amazon Virtual Private Cloud (VPC), Cisco Quickstart Private Cloud, Google VPC, HPE GreenLake, IBM Cloud Satellite and Dell.

Examples of PaaS are IBM SmartCloud Application Services, Google App Engine, AWS Elastic Beanstalk, Apache Stratos, and Magento Commerce Cloud.

There are also plenty of vendors offering specialized service models. For example, FaaS offerings include AWS Lambda, Google Functions, Azure Functions, IBM Cloud Functions, Alibaba Functions, Oracle Functions, and Cloudflare Workers. MLaaS offerings include Amazon (Polly, Sagemaker, Rekognition, etc.), Google Cloud AutoML, Azure Automated ML, IBM Watson Machine Learning, and more. BaaS offerings include AWS Amplify, Google Firebase, Back4App, Parse, Backendless, Pubnub, and more.

### What are the typical uses cases of cloud computing?

With IaaS, cloud is useful in offloading the responsibility of installing and operating infrastructure. With PaaS, developers can more quickly build, deploy and monitor their applications from components provided by the cloud provider. With SaaS, users always get the latest versions of the software without any installation from their end. Applications are also becoming cloud-native to get the most out of the cloud.

Streaming video and audio, storing social media content, storing important documents, running banking applications, CRMs and ERPs, and managing logistics are just some examples where cloud can be used.

Test and development environments can be set up in the cloud. This saves capital expenditure, manpower and time. Cloud collaboration and testing also enables faster development cycles.

Cloud is used for data storage. Cloud can also get the best out of both structured and unstructured data via big data analytics. With it's ability to scale dynamically, cloud can perform real-time analytics even when data comes in at varying rates.

Disaster recovery and data backups are further use cases where cloud saves time and cost.

### What are some criticisms of cloud computing?

Some of the criticisms are related to the myths surrounding cloud computing. With better understanding, some of these myths can be dispelled. However, there are some valid criticisms.

In the past, users could run applications on standalone systems even without an internet connection. This is not so for cloud-based applications. Good **connectivity** is essential.

**Security** is another concern. While cloud providers are continually making operations more secure, the fact that applications are accessible online make them less secure than applications or systems that run offline in isolated environments.

**Data mobility** can be problem. Getting large volumes of data in and out of cloud systems can be expensive and slower than local access. It's also not easy to move data from one cloud provider to another. In general, there's no easy way to migrate an application from one cloud provider to another due to proprietary interfaces and deployments. Hence, there's a danger of **vendor lock-in**.

## Milestones

1950

At a time when computers are expensive, many users share the use of mainframes from their dumb terminals.

1969

The first packet is sent on the **ARPANET** (Advanced Research Projects Agency Networks), a network in which a few computers at different geographic locations are interconnected. ARPANET becomes a testing bed for the development of many networking technologies. It would eventually evolve into what would be called the Internet.

1972

Developed from research work from the late 1960s, IBM releases VM/370. This is a system that can emulate different operating systems and virtual runtime environments to serve the needs of different applications and users. This is the beginning of **Virtual Machines (VMs)**.

1990

The idea of **grid computing** is proposed in the early 1990s. This is really a metaphor for making computing as accessible as the electric power grid. Grid computing is essentially distributed computing that brings together resources for tasks that would be hard to execute on a single machine alone. As such, it offers an alternate to powerful and specialized supercomputers.

1996

The term **cloud computing** appears in an internal document at Compaq. This is probably the earliest use of the term. The idea is to enable ISPs to implement and bill for cloud-computing enabled apps. The prediction is that cloud apps would become more important than enterprise apps. The term is borrowed from the word "cloud" that engineers often used for telecom networks. In fact, by early 1990s, patent diagrams had depicted the internet as a cloud.

1999

*Salesforce* replaces its traditional CRM software with a cloud-based software, thus becoming a pioneer in cloud computing. It's application is the first large-scale example of SaaS.

Jul
2002

Amazon creates a subsidiary named *Amazon Web Services (AWS)*. In 2006, AWS launches Simple Storage Service (S3) and Elastic Compute Cloud (EC2) as elements of its IaaS offerings.

Aug
2006

At a conference, Google CEO Eric Schmidt introduces the term "cloud computing". By the following year, Amazon, Microsoft and IBM start using the term, signifying that cloud computing has gone mainstream.

Apr
2008

As a PaaS, Google releases the beta version of *Google App Engine*. This includes dynamic content serving, persistent storage, automatic scaling, load balancing, use of Google APIs for user authentication and emails, and local development environment. By 2022, Google App Engine supports Python, Java, Node.js, Go, Ruby, PHP, and .NET.

2009

Vaquero et al. attempt to find a suitable definition of cloud computing. They analyse publications from 2008 and identify as many as 22 different definitions. This shows that not everyone agrees what the cloud represents or provides. They propose a definition that includes scalability, virtualization and pay-per-use utility model.

Feb
2010

Microsoft makes its cloud platform called *Windows Azure* generally available. This is initially positioned as a PaaS offering. In subsequent years, this is rearchitected into a IaaS offering and renamed to *Microsoft Azure*. In 2014, Microsoft embraces open source software, and by 2017 40% of VMs on Azure run on Linux.

Jul
2010

Rackspace Hosting and NASA jointly launch an open-source cloud-software initiative known as *OpenStack*. OpenStack can be used on commodity hardware to create an IaaS offering.

2011

The National Institute of Standards and Technology (NIST) publishes a reference model of cloud computing. In a separate document it defines cloud computing as "a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction."

Dec
2019

Amazon announces *AWS Outposts* to bring to its customers a better hybrid cloud offering. AWS Outposts extends AWS infrastructure and services to on-premise deployments and customer data centres.

Cloud Computing