# Netlify

## Summary

Netlify is a platform for web developers to host their sites in the cloud without managing any servers in the back-end where application logic and database works. Updates to the Web applications can be automated by integrating any git based Version Control System supported by Netlify.

 It can have dynamic functionalities which change constantly. Netlify builds its own kind of file storage and management system to push updates both to Git providers and Netlify simultaneously getting connected to the repository.

Every update/changes made to the content is distributed across the servers spread over in the content delivery network (CDN) and pre-built as static pages with file optimization before being delivered to the users.  Visitors of a website in Netlify get the pre-loaded version of website from a geographically nearest server reducing the loading time.

## Discussion

### Why should I use Netlify?

Netlify is the first one to offer Jamstack architecture with modern tools. .The Term "Jamstack" was coined by Netlify's Co-founder Mathias Billman which means a development stack made up of JavaScript, APIs, and templated Markup. .Netlify boosts web developer productivity since it is serverless and eliminates the need for developers to manage servers, which Netlify performs automatically. It provides many interesting features even in it's free plan. Web development sector is constantly moving to decoupled environment provided by Netlify in which the frontend,database and backend works independent of each other helping developers to easily to scale, manage and increase the performance of their websites.

### What are some useful features of Netlify?

    + **Rollbacks**: Previous versions of a web project can be selected and reverted using the Rollback option. This can be helpful if the present deployment has any issues in it. Netlify uses the atomic deploy concept in which changes are never in an inconsistent state and file uploads made sure for completed for a successful deployment.
    + **Development in Local Machine**: A local development environment with all the netlify features can be created using Netlify Dev.
    + **Redirects API** : Powerful feature to configure redirects, rewrites etc from the distributed content delivery network (CDN) of netlify. It can be done using adding a `redirects` file to the root of site folder.
    + **Serverless functions** : Adding serverless function requires only a single line of code addition to the `netlify.toml` file. It is built and deployed along with the netlify site for running server-side code without a dedicated server
    + **Deploy Previews** : Changes made to the site can be previewed even before deployment. Netlify generates previews by default for all pull and merge requests in Github and Gitlab.
    + **Playground** : An application where users can test configuration rules before setting those in the Netlify site in realtime.

### What's the Netlify architecture?

Netlify sites are prebuilt and deployed on the Global Edge Network, a CDN. This helps speed up the website loading by placing copies of files in the servers geographically closer to the website visitors. Netlify users are not required to manage any web servers. This is possible with the introduction of a new architecture known as the Jamstack.

    + **Frontend**: Netlify is a infrastructure provider of Jamstack which decouples frontend from the backend and database. Netlify's Jamstack application delivers a precompiled and optimized static frontend which is deployed globally to the Edge CDN. The Frontend uses Javascript and APIs to communicate with backend services.
    + **Backend** : Backend Services are built as Serverless functions and distributed as various microservices making the load less. Functions built on Netlify are deployed to AWS Lambda and made as an API endpoint. Credentials required for the frontend to connect with the backend services are stored securely.
    + **Database** :Netlify does not provide database services directly. Users of Netlify can utilise any backend provider they like, but preferred partners are: DataStax, Fauna, Nimbella, and StepZen.

### What's the Netlify Command Line Interface (CLI)?

Netlify CLI is used to configure and deploy a website straight from the Development Server on a local machine which can be shared with others to run a local build, plugins and deploy a site. Netlify CLI runs on top of Node JS. CLI was redesigned and introduced as Netlify Dev. There are many commands available in Netlify to work with the CLI version.

All Features of Netlify is brought to local machine. Netlify's production routing engine is installed on a local development server, which makes all redirects, proxy rules, function routes, and add-on routes available locally and environment variables are set from the site environment. Live collaboration with multiple people is also possible with the CLI version of Netlify by running netlify dev `--live` command.

### What's static routing in Netlify?

Static Routing helps to manage the traffic on a Netlify site. There are many powerful static routing features like redirects, rewrites, proxies which can be enabled using `redirects` file or in `netlify.toml` file. Edge Nodes in the Netlify CDN analyzes the rules written in these files and points the website visitors to various different paths or routes accordingly. Paths can be based on certain conditions like language or the location of the website visitor. Proxies written in the file can redirect the incoming request to another website or can execute a serverless function.

Custom HTTP headers can also be sent as a response to the site requests. They can be managed in `headers` file or in the `netlify.toml` configuration file.

### What are edge handlers in Netlify?

Edge handlers allow us to do programmable routing by giving a personalized experience to the website visitors. It is a redeveloped version of Netlify's Edge Network built with multiple cloud providers to handle complex tasks and reduce workload for developers.

It can alter both incoming and outgoing traffic also it has capabilities far beyond static routing as it can filter and manipulate HTTP requests and responses.

It can intercept incoming HTTP requests and change the route of traffic depending on the request. Also,it has lower latency than Netlify functions useful for making quick routing decisions to provide features such as Localized content or personalized ads.

During deployment of the website Netlify's bot checks for the netlify/edge-handlers directory in the base directory of the repository.

Edge handlers are configured as two files in a repository.

    + Handler file: It is written in Javascript. Handlers should export a function named onRequest which receives a Javascript event object containing the information about the handler request.
    + Handler declaration: This is declared in In netlify.toml file by adding `[[edge_handlers]]` with the path and handler to execute the requests coming in that path.

### What are Netlify functions used for?

Netlify functions are used to perform complicated javascript functions that would be impossible to do by running javascript on the client side. The Functions tab in the web interface provides access to functions. Bundlers are used to optimise functions in order to make them smaller and faster.

Functions are written as scripts that can be deployed on netlify directly. It is hidden from public view and works as API endpoints.updates to the function on a production branch will not affect the version deployed under a branch or in a deploy preview as it is immutable.

Fetching live data from an API, pulling dynamic graphics, sending an automatic email, and validating an input submitted by a user in the field are a few examples of Netlify functions.

### What are Netlify's background functions?

Traditional Functions are synchronous by default with an execution time of 10seconds which makes it difficult to manage long running process but it can be made Asynchronous by adding `--background` to the end of the filename with runtime longer than 10 seconds and up to 15minutes with a default memory of 1024mb. For tasks such as Batch processing, web scraping etc...which need the API to execute longer asynchronous Background functions are helpful.

### Is Netlify good for SEO?

SEO is a secondary priority for JamStack Technology since, while it is constantly growing, it is not particularly SEO friendly. By, taking care of a few things SEO can be improved, as indicated by Netlify.

Making Sitemaps and Maintaining a Clean URL Structure: The click through rate (CTR) and average time spent on the page have an impact on search engine ranking, which leads to a higher search engine ranking and a better user experience. Site maps are important to get higher presence in search engines by crawling pages available in a website. Also the structure of URL plays a key role in users clicking the link. Static URL is more preferred and SEO friendly.

Optimizing Assets: Netlify by default has Asset Optimization which bundles and compresses the HTML, CSS,JS files used in the project.

## Milestones

Oct
2013

Mathias Billmann launches Bit Ballon for hosting HTML5 static website and apps with easy drag and drop feature under Makerloop Inc.

Apr
2015

Mathias Billmann and Christian Bach launches it as Netlify.com under the company Makerloop Inc. to host static sites and Apps with continuous deployment workflow with github support. Netlify will run the build command and deploy whenever an update is pushed to github. .

Oct
2015

Smarter Redirects is introduced to avoid a chain of redirects. A single efficient rule for redirect is implemented.

Jan
2016

Netlify starts to offer Free SSL Certificate with automatic renewal and provisioning feature to the users in a single click in collaboration with Let's Encrypt. A new system is developed by netlify for transferring SSL certificates and associated keys over a secure encrypted connections to avoid storing sensitive keys on Netlify's CDN edge nodes.

Jul
2016

**Gitlab integration is added to deploy repositories in Netlify due to the increasing popularity and requests from users**. . Deploy previews help users to see changes in Production environment without actually deploying it.

Mar
2017

New system design is implemented. Changes made to the user interface and new header 'Hero Card' is designed. It is the first element for most of the pages showing a quick summary of page giving access to commonly used actions and information.

Jun
2017

Two New Features were introduced based on Edge Logic Requests (ELRs) in which the Netlify's CDN will handle requests that are normally done through server or client-side in Javascript to overcome traditional way of static files getting served from CDN. Branch Based Split testing feature is introduced where pull requests are deployed into a unique subdomain to see the changes even before merging. JSON Web tokens is introduced for assigning authentication and user management to any other services that can generate JSON Web Tokens to allow granular role-based permissions.

Dec
2017

Makerloop Inc under the brand which Netlify was running is renamed to Netlify Inc.

Mar
2018

**Netlify functions is launched making the deployment of serverless AWS Lambda functions easy as adding a file to a git repository**. Server side code will work as API endpoints without having to run it on dedicated server. Netlify Identity a JSON web token based user authentication feature with Netlify Forms which helps users to get all HTML form submissions in their Netlify dashboard is launched.

Aug
2018

Netlify drop is introduced to replace bitballon a drag and drop depolyment for hosting static sites written in HTML, CSS, and JS. Netlify hosts the site and gives us the link to access it.

Feb
2019

Netlify large media is launched, a service built on the top of Git LFS to remove heavy binary files from the repository. .Git LFS is an extension of Git which uses pointers on the repository to point the assets stored in a remote server. Using Netlify large media size of the repositories can be reduced and site build times can be increased.

May
2020

Netlify build plugins is introduced to make the development process easier by automating build tasks and customizing build process. It also has one click install to cache files, checking broken links and trigger tests etc.

Jul
2021

Four new database partners DataStax, Fauna, Nimbella, and StepZen is added to provide serverless database to the users. Integration with these services can help developers run their backend services more easier and faster.

Netlify


# API Testing Tools

## Summary

As organizations move towards DevOps, Continuous Integration (CI) and Continuous Delivery/Deployment (CD), there's a need to have quicker feedback from testing. End-to-end or UI testing are slow. API testing is one solution but it must be easy to create, execute and maintain API tests. It needs to be automated as much as possible. To enable this, many tools are available in the market.

There are both open-source and commercial API testing tools. We can't say which of these is the best. We should select a tool that best suits our project. The selection should be based on features and how well it integrates with other tools and our processes.

Since 2018, some tools are beginning to adopt AI/ML. This could become an essential feature in the years to come.

## Discussion

### What should I look for when selecting an API testing tool?

At the minimum, any tool should make API requests and validate responses. Other features to consider are: easy to install, learn and customize; good documentation; available on multiple platforms/devices; supports multiple protocols (REST, SOAP, HTTP, JMS, TCP/IP); integrates with CI/CD pipelines; can sniff exploratory tests to automatically create API tests; tests but also validates API specifications.

The **syntax** for writing tests must be simple enough for those with little coding experience.

Pick a tool that **integrates** with your automation framework. For example, REST-assured integrates seamlessly with Serenity.

For teams, the tool should enable **collaboration**. It should be possible to share test cases and results, or publish tests and allow others to run them easily.

Some tools such as SoapUI are dedicated to API testing. Others such as JMeter was designed for load testing but supports API testing. Swagger is meant for designing APIs but can also be used for testing. This is worth considering if you're looking for single unified tool.

### What are the types of API testing tools?

Broadly, we can identify the following types:

    + **Command-Line**: cURL is an example that can query an API and print out the response. It uses libcurl that can be used in any client-side program. Thus, developers can create their own CLI-based API testing tool using libcurl. Dredd is another CLI tool. On Windows, PowerShell is a good choice.
    + **Browser-Based**: DHC and Postman are examples that can run within a browser. Compared to CLI tools, they offer a user-friendly interface for testers. Postman was initially delivered as a Chrome App, then a standalone desktop app, and then a web app since September 2020.
    + **Standalone IDE**: Katalon Studio is an example that supports test case generation, CI/CD integration, and many types of testing (API, Web, Mobile, Desktop). Karate is an example that can be installed as a plugin/extension and used within IDEs/editors such as IntelliJ or Visual Studio Code. It supports API testing, mocks, performance testing and UI automation.

### Could you describe some API testing tools?

Katalon Studio claims to be "all-in-one test automation solution". It offers free and enterprise versions. Free version supports basic record and playback, REST and SOAP, OAuth for authentication, different types of testing (keyword-driven, data-driven, BDD), parallel execution, and headless execution. It can also import tests from other tools/frameworks such as Selenium, Postman, JUnit, TestNG, SoapUI and Swagger.

In Postman, tests can be organized into collections that can be shared with others. API design and maintenance can be done within Postman, which serves as the single source of truth. Multiple APIs can be version controlled and tagged. Mock servers and environments can be created. It can monitor API uptime and responsiveness. Newman is Postman's CLI tool for easy integration into CI/CD pipelines.

An alternative to Postman is Hoppscotch (previously called Postwoman). It runs within the browser and supports GraphQL. As is common, requests can be customized with authentication, headers, and pre-request script. Assertions are in JavaScript.

OpenAPI.Tools has a list of API testing tools, briefly describing each of them.

### What's the syntax that tools use for specifying API tests?

Karate uses Gherkin syntax that was popularized by Cucumber for BDD. REST-assured also supports BDD syntax.

Taurus allows tests to be written in YAML or JSON.

Parasoft SOAtest uses visual drag-and-drop. Drag-and-drop method of creating tests is also supported by SoapUI. In addition, SoapUI supports Groovy language, which has a concise expressive syntax for the Java platform.

ACCELQ is a cloud-based application for which no code needs to be written.

### Which are the API testing tools in various languages?

Some tools support many languages. Postman has code generators in many languages that developers can use. RedwoodHQ is another tool that has support for Java/Groovy, Python and C#. Fiddler supports .NET, Java and Ruby.

API testing in Java domain has been traditionally hard. REST-assured simplifies this with support for XML or JSON format in requests and responses.

In Ruby, Airborne is an option. It relies on RSpec. It works with APIs written in Ruby on Rails.

WebInject is a tool written in Perl and therefore requires a Perl interpreter at runtime.

In JavaScript, Chakram is a tool based on Node.js, Chai.js and Mocha. It uses BDD syntax. Another tool in JavaScript is Frisby that uses Node.js and Jasmine.

In Python, pyresttest is applicable. It's also a micro-benchmarking tool. Configuration is in JSON or YAML. Hence, little coding knowledge is needed.

### How is artificial intelligence enabling API testing tools?

Test engineers have always found UI testing to be easier than API testing. They may not know how all the software components work together via their APIs. API testing requires better understanding of the software. AI/ML is attempting to remove some of this complexity. One approach is to simply monitor UI tests, discover patterns and then use this knowledge to automatically create API tests.

One tool that generates API tests is Parasoft SOAtest Smart API Test Generator. It's installed as an add-on to a web browser. For example, in an e-commerce site, it would learn that a cart ID is important across API calls. A human could indicate that number of items in the shopping cart is important. AI would learn this and validate this in future API calls.

Other AI/ML testing tools include Applitools, Testim, Sealights, Test.AI, MABL, Retest, and ReportPortal. AI/ML can catch UI changes and identify them as bugs or features; analyze code changes and execute relevant tests; detect code changes and automatically update existing tests; automate result analysis and defect management.

## Milestones

2000

It's in this decade that APIs start getting popular, partly due to Salesforce, eBay and Amazon. Due to Agile practices, there's also a need to automate API testing.

2010

In this decade, focus shifts to scalability. Adoption of DevOps and CI/CD make it necessary to embrace automated API testing. However, even by the end of the decade, many organizations continue with UI testing.

Oct
2018

Parasoft SOAtest 9.10.6 is released. This includes AI/ML capability that's now part of it's Smart API Test Generator. In general, 2018 is the year when tools start including AI/ML. Focus shifts from just test automation to include test generation.

API Testing Tools


# MQTT

## Summary

MQTT is a **publish/subscribe** messaging transport protocol. It has low complexity, small code footprint and consumes low network bandwidth for messaging. The lightweight protocol and small packet size support makes it suitable for applications such as Machine to Machine (M2M) and Internet of Things (IoT).

The protocol runs over network protocols like TCP/IP that provide ordered, lossless, bidirectional connections. MQTT-SN v1.2, MQTT for Sensor Networks (formerly known as MQTT-S), is a version of the protocol targeted for embedded devices on non-TCP/IP networks, such as Zigbee. Connectionless network transports such as User Datagram Protocol (UDP) are not suitable because packets may be lost or arrive out of order. The protocol enables transmit of messages either in 1-to-1 or 1-to-n configuration. The messaging transport is agnostic to the content of the payload.

## Discussion

### What's the expansion for MQTT?

The published OASIS standard does not give any expansion of MQTT.

MQTT used to mean *MQ Telemetry Transport*, where MQ is sometimes mistaken for Message Queue. In fact, MQ referred to IBM's MQ Series of products. Today, MQTT is not an acronym and it doesn't use traditional message queues.

### How does the MQTT protocol work?

MQTT uses **Publish/Subscribe** model or pattern. Publishers publish messages. Subscribers receive messages if they have subscribed to the same. However, publishers and subscribers don't communicate to each other directly. There's an intermediate central **Server** or **Broker** that mediates in between. Broker filters all incoming messages and distributes them to subscribers accordingly. Publishers and subscribers are also called **Clients**.

Subscribers subscribe to messages that are of interest to them. Such filtering could be topic-based, content-based, or type-based. Topics could be organized in a hierarchy.

With the publish/subscribe pattern, the publisher and subscribers don't need to know each other. They are not dependent on each other for initialization and synchronisation. Message delivery is one-to-many and can scale with respect to subscribers without burdening publishers.

It will also be apparent that MQTT operates unlike message queues. Messages cannot get stuck in a queue if no one reads them. In message queues, only one consumer can read the message. Message queues are also defined explicitly but in MQTT topics can be created on the fly.

### What are the core features of MQTT?

The core features of MQTT are:

    + Simple to implement: This translates to small code and memory footprint for constrained devices.
    + Lightweight and network-bandwidth efficient: Payload must be transferred without significant overhead.
    + Handle abnormal client disconnection: Using what is called **Last Will and Testament (LWT)**, if a client is not sending anything for some time, broker will automatically publish a message. For example, this can be useful to inform subscribers that an IoT device (publisher) is not reachable.
    + Transport protocol is payload data agnostic: Data can be binary, JSON or any other format.
    + Provides a Quality-of-Service data delivery: QoS can be selected based on the needs of the application.
    + Supports continuous session awareness: Sessions can be continued even across intermittent network connections. This implies clients need not subscribe again when they reconnect to server.
    + **Retained messages**: The last published message (with retained flag set to true) is stored at the broker so that new subscribers can immediately obtain *last known good value* rather than wait for the next update from publisher.

### Why can't I use HTTP instead of MQTT?

Altough popular on the web, HTTP is not suitable for small data packets typical in IoT. HTTP is document-centric whereas MQTT is data-centric. HTTP is a complex protocol. It's textual and therefore parsing HTTP is not trivial. MQTT is binary and encoding/decoding MQTT packets is a lot easier. Because of its publish-subscribe model, data distribution is one-to-many in MQTT whereas in HTTP it's limited to one-to-one.

An MQTT PUBLISH message header is 2-4 bytes; CONNECT message header is 14 bytes; HTTP header can be up to 1KB. A C implementation of an MQTT client can be as small as 30KB.

MQTT uses less battery power, can send more messages per hour and send them more reliably than HTTPS. MQTT uses more battery power to initiate a connection but this quickly compensated by gains as the connection stays open longer.

In fact, there are alternatives to MQTT that may be more suitable candidates for comparison than HTTP: AMQP, CoAP, XMPP, DDS, STOMP.

### What was MQTT designed for and where is it popularly used today?

IBM originally designed MQTT protocol for communication between oil pipeline stations over low bandwidth satellite links. It was later used for applications such as home automation and message broadcasting to ferries. Other areas where it's used include connected cars, asset tracking, smart metering, process control in manufacturing.

MQTT gained traction in IoT space due to its lightweight protocol. One popular example of its usage is by Facebook for its Facebook Messenger application.

### What are the control packets supported in MQTT?

Control packets can be summarized in the following groups:

    + **Connection Management**: CONNECT, CONNACK, DISCONNECT, PINGREQ, PINGRESP
    + **Subscription Management**: SUBSCRIBE, SUBACK, UNSUBSCRIBE, UNSUBACK
    + **Message Delivery**: PUBLISH, PUBACK, PUBREC, PUBREL, PUBCOMP
    + **Authentication**: AUTHAUTH packet was introduced in MQTT version 5.0. PINGREQ is sent from client to server to indicate that client is alive though not sending any other control packet.

### What sort of Quality of Service (QoS) does MQTT offer?

MQTT protocol defines three levels of QoS for message delivery:

    + **QoS 0**: At most once delivery. Receiver does not sent a response. Sender does not retry and hence deletes the message locally as soon as it's sent. Message arrives at the receiver once or not at all.
    + **QoS 1**: At least once delivery. Receiver acknowledges the PUBLISH packet using the PUBACK packet.
    + **QoS 2**: Exactly once delivery. Neither loss nor packet duplication is acceptable. This QoS level adds protocol overhead and specifies many rules for both sender and receiver. Packets PUBREC, PUBREL and PUBCOMP are involved instead of PUBACK used in QoS 1.Sender will retry on unacknowledged packets only if client reconnects without a clean start and session is present.

### Does MQTT support security?

MQTT is a transport protocol that concerns message transmission. So, security features such as privacy of data, authentication and authorization of users, etc. have to be taken care by the developer. MQTT specification provides guidance about possible implementations such as TLS. Security-related aspects that need to be considered could be related to geography, industry or regulations.

TLS/SSL security and payload encryption can be used. The IANA has assigned port 8883 for MQTT over SSL, or secure MQTT over TCP or UDP. Incidentally, port 1883 is for MQTT over TCP or UDP We should note that SSL is not exactly lightweight and adds significant overhead.

Since Version 3.1 of MQTT protocol, there are two provisions:

    + Add username and password in the CONNECT packet
    + Include return codes in CONNACK packet to report security issuesIn Version 5.0, MQTT adds the AUTH control packet. This can be used to implement complex authentication methods such as SCRAM, Kerberos or OAuth.

There is a supplemental publication that focuses on MQTT protocol's integration within the NIST Framework for Improving Critical Infrastructure Cybersecurity, simply called *Cybersecurity Framework*.

### What are the limiting factors of MQTT to consider while using it?

Here are some limiting factors of MQTT:

    + The structure of the published data is not known to the subscriber. This information should be shared or known beforehand. When topic-based filtering is used, publisher and subscribers must know the right topics to use.
    + The delivery of a message is not confirmed as there is no automatic confirmation from the recipient. Moreover, the publisher also does not know if there are any subscribers.
    + MQTT uses TCP, which needs complex handshaking and is not easy on memory and power requirements. Standard states that TCP is a suitable transport and UDP is not. MQTT can also use TLS or WebSocket.
    + Broker is centralized. This is not only a single point of failure but also can limit scalability. However, real-world implementations overcome this by managing a **distributed broker cluster** that clients see as a single logical broker. Examples of such cluster implementations are HiveMQ, VerneMQ, and EMQ.
    + MQTT has no built-in security. Developers need to implement security on their own.

### What software implementations of MQTT are available?

Several MQTT client libraries are available based on different languages and for supporting varied devices. Broker or server implementations and tools are also available from different vendors as noted on MQTT GitHub Wiki.

Eclipse Mosquitto, a MQTT broker implementation, is about 120kB and requires 3MB RAM for 1000 clients connected.

Eclipse Paho client offers a number of features: automatic reconnect, offline buffering, message persistence, WebSocket or TCP support, blocking or non-blocking API, and high availability. For embedded devices of limited resources, Paho offers an ANSI standard C compliant library.

RabbitMQ server uses AMQP instead of MQTT but it offers a plugin for MQTT, making AMQP interoperable with MQTT.

## Milestones

1999

MQTT is developed by Andy Stanford-Clark (IBM) and Arlen Nipper (Arcom, later Eurotech). It's used to send sensor data gathered from an oil pipeline. Data is sent to a SCADA host system via low-bandwidth links.

Aug
2010

IBM releases MQTT V3.1 and makes it available to the public. They define MQTT as,

> A lightweight broker-based publish/subscribe messaging protocol designed to be open, simple, lightweight and easy to implement.

Aug
2011

Facebook announces in a blog post that their *Facebook Messenger* is using MQTT. Without it, they experienced long latency in the order of seconds. With MQTT, it's down to hundreds of milliseconds.

Nov
2011

IBM and Eurotech donate MQTT code to the Eclipse Foundation. A month later, Eclipse Foundation releases open source Java and C client code for MQTT. Client-side MQTT code is under the project named **Eclipse Paho**. Today client code from Paho is available in a number of languages.

Oct
2014

MQTT Version 3.1.1 is officially approved as an OASIS Standard. In June 2016, this is published by ISO as ISO/IEC 20922:2016.

Feb
2015

As MQTT broker implementation, open source **Eclipse Mosquitto 1.4** is released, the first release by the Eclipse Foundation. Version 1.0 of Mosquitto was available as early as 2012. The growth of MQTT can be attributed to the open source nature of Eclipse Paho and Mosquitto.

Apr
2016

Created four years earlier by Feng Lee, version 1.0 of **EMQ (Erlang MQTT Broker)** is released. Written in Erlang/OTP, EMQ is open source, distributed, and massively scalable. Messages are routed through clusters that apply *Topic Trie Match* and *Routing Table Lookup*.

May
2018

OASIS releases **MQTT Version 5.0**. It supercedes the previous standard. Eclipse Paho 1.4.0 is expected to support this by June 2018 in C and Java. V5.0 support from Eclipse Mosquitto is expected by August 2018. It's expected that the latest version will go into production systems by Q4 2018. Main changes due to V5.0 are summarized by a HiveMQ blog post.

MQTT


# 3GPP

## Summary

The Third Generation Partnership Project (3GPP) was founded in 1998 to develop and maintain the 3G Mobile System, which is an evolution of GSM. Since then, its scope of work has expanded to technologies that came after 3G, that is, 4G/LTE and 5G. 3GPP also maintains the older GSM specifications. 3GPP aims to address new use cases as they emerge while ensuring compatibility and interoperability.

Partners or members of 3GPP can participate in the specification process. 3GPP is funded by Organizational Partners, which are national or regional Standards Development Organizations (SDOs). Other associated groups also provide funding.

3GPP publishes technical reports and specifications, which are published as standards by SDOs. The documents they publish are freely available to the public.

## Discussion

### What are the types of memberships at 3GPP?

3GPP has the following types of partnerships or memberships:

    + **Organizational Partners**: These have the authority to define and publish standards regionally or nationally. They bring to 3GPP regional or national requirements, IPR policies and resources to help run 3GPP. Documents produced by 3GPP have no legal standing until they're published by Organizational Partners into corresponding regional or national standards. However, Organizational Partners jointly own the copyright to the technical specifications and reports.
    + **Market Representation Partners**: These promote 3GPP in their markets. They bring in market requirements. They encourage their members to coordinate activities and contribute towards 3GPP.
    + **Individual Members**: These can participate at meetings and contribute actively.  To become an individual member, they must first be a member of an Organizational Partner. Members are typically government bodies, regulators, educational/research institutes, network operators, and software/hardware vendors.
    + **Observers**: These can become Organizational Partners in future. They can attend meetings but can't discuss, decide or lead.
    + **Guests**: These can become Individual Members in future. Like observers, they've limited privileges.

### Who are the partners and members of 3GPP?

As on October 2022, 3GPP has 7 Organizational Partners: ARIB, ATIS, CCSA, ETSI, TSDSI, TTA, TTC. In addition, there are 26 Market Representation Partners. Some of these are 5G Americas, GCF, GSA, GSMA, Small Cell Forum, Wireless Broadband Alliance, etc. There are two Observers: Communications Alliance (Australia) and TIA (USA).

3GPP has 800+ Individual Members. Some of these are Airbus, AT&T, Ericsson, Huawei, IIT Bombay, Intel, KDDI, MediaTek, NEC, Ofcom UK, Qualcomm, Rakuten, etc.

### How is 3GPP organized?

3GPP consists of one Project Coordination Group (PCG) and a few Technical Specifications Groups (TSGs). Each TSG can establish one or more Working Groups (WGs) under it.

PCG is composed of Organizational Partners and Market Representative Partners. ITU representatives and Observers may also attend their meetings. PCG meets at least twice a year. PCG manages the overall work of 3GPP. It allocates funds and resources. It maintains the registry of Individual Members and relevant IPR declarations. It authorizes requests from TSGs for external liaisons.

TSGs are composed of Individual Members (who are really representatives of Organizational Partners), Market Representation Partners, Observers and Guests. TSGs perform technical coordination including allocating resources, managing work items and tracking progress.

### How does 3GPP work?

3GPP meetings happen quarterly. Each TSG has a plenary meeting in which all of its WGs get together. Plenary formally approves all changes proposed at the WGs. Plenary also coordinates activities across WGs. CT, RAN and SA plenary meetings are co-located at the same venue. They happen in March, June, September and December. WG meetings and related ad hoc discussions happen between two plenary meetings.

During COVID-19, in-person meetings were replaced with virtual meetings. For example, *RAN5#92-e* (August 2021) was a virtual meeting whereas *RAN5#93* (November 2021) was an in-person meeting.

Following the TSG plenary, the 3GPP Mobile Competence Centre (MCC) makes public the updated specifications. Specifications are available per TSG meeting and by series.

Two documents to understand 3GPP better are 3GPP Working Procedures and TR 21.900: TSG Working Methods.

### What types of documents are produced by 3GPP?

3GPP produces Technical Reports (TRs) and Technical Specifications (TSs). Reports contains the findings of feasibility studies. When any new feature is to be introduced, 3GPP commissions a study involving perhaps many WGs across TSGs. Work items are scoped and scheduled in Work Item Description (WID) documents. The study eventually results in new TR/TS or changes existing TR/TS via Change Requests (CRs). WIDs and CRs are just two types of Temporary Documents (TDocs).

Depending on the purpose, CRs fall into categories A-F. CRs introduce a new feature, clarify or enhance an existing feature, or correct an error in frozen specifications. CRs can be withdrawn, rejected, revised or postponed. Multiples CRs can be merged. Each CR is given a identity. For example, *CR 31.102-0095* implies a CR on 31.102 and assigned the number 0095.

Each TR or TS belongs to a specific series and has a version number. For example, *TS 38.101-4 V17.6.0* is a TS belonging to 38-series. The suffix "-4" indicates part 4 (parts and subparts are used only for long documents). Version "17.x.x" indicates Release 17.

## Milestones

Dec
1998

The 3GPP Partnership Project Agreement is signed by **five Organizational Partners**: ARIB (Japan), ETSI (Europe), ATIS Committee T1 (USA), TTA (South Korea) and TTC (Japan). At this time, **four TSGs** are proposed: Radio Access Network, Core Network, Terminals, Service and System Aspects. UMTS Forum in inducted into 3GPP as the **first Market Representation Partner**.

Jan
1999

**3GPP2** is launched with four Organizational Partners: ARIB (Japan), TIA (USA), TTA (Korea), and TTC (Japan). This is not to be confused with 3GPP. 3GPP standardizes 3G/UMTS as an evolution of GSM. 3GPP2 standardizes IS-2000 (aka CDMA2000) as an evolution of IS-95 (aka cdmaOne). 3GPP2 will specify an evolved core network from ANSI-41.

May
1999

China Wireless Telecommunication Standard (CWTS) group is inducted into 3GPP as an Organizational Partner. In May 2003 (and formally signed in April 2004), China Communications Standards Association (CCSA) becomes as an Organizational Partner instead, since CWTS is merged into CCSA.

Jul
2000

The scope of 3GPP is modified to include development and maintenance of legacy technologies **GSM, GPRS and EDGE**.

Oct
2004

PCG approves the creation of **TSG CT** by merging TSG CN and TSG T. To aid this decision, it observes that the terminal and the network deal with two ends of the same protocols. The merger is also expected to bring economies of scale.

Dec
2007

3GPP Organizational Partners agree to extend the scope of the organization from just 3G to "evolved Third Generation and beyond". The agreement notes that they will continue to develop and maintain GSM, GPRS and EDGE.

Jan
2015

From 1st January, the Telecommunications Standards Development Society of India (TSDSI) becomes a full Organizational Partner in 3GPP. The agreement is formally signed in April.

Jun
2016

TSG GERAN (GSM/EDGE Radio Access Network) is finally closed, its work being transferred to TSG RAN WG5 and WG6.  TSG RAN WG6 oversees GERAN and UTRAN radio and protocol work but is subsequently closed in July 2020.

3GPP


# IoT Cloud Platforms

## Summary

IoT cloud platforms bring together capabilities of IoT devices and cloud computing delivered as a end-to-end service. They are also referred by other terms such as *Cloud Service IoT Platform*. In this age, where billions of devices are connected to the Internet, we see increasing potential of tapping big data acquired from these devices and processing them efficiently through various applications.

IoT devices are devices with multiple sensors connected to the cloud, typically via gateways. There are several IoT Cloud Platforms in the market today provided by different service providers that host wide ranging applications. These can also be extended to services that use advanced machine learning algorithms for predictive analysis, especially in disaster prevention and recovering planning using data from the edge devices.

## Discussion

### What are the key features of an IoT cloud platform?

An IoT cloud platform may be built on top of generic clouds such as those from Microsoft, Amazon, Google or IBM. Network operators such as AT&T, Vodafone and Verizon may offer their own IoT platforms with stronger focus on network connectivity. Platforms could be vertically integrated for specific industries such as oil and gas, logistics and transportation, etc. Device manufacturers such as Samsung (ARTIK Cloud) are also offering their own IoT cloud platforms.

In most cases, typical features include connectivity and network management, device management, data acquisition, processing analysis and visualization, application enablement, integration and storage.

Cloud for IoT can be employed in three ways: Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) or Software-as-a-Service (SaaS). Examples of PaaS include GE's Predix, Honeywell's Sentience, Siemens's MindSphere, Cumulocity, Bosch IoT, and Carriots. Developers can deploy, configure and control their apps on PaaS. Prefix is built on top of Microsoft Azure (PaaS). Likewise, MindSphere is built on top of SAP Cloud (PaaS). Siemens's Industrial Machinery Catalyst on the Cloud is an example of SaaS which is a ready-to-use app within minimal maintenance.

### Where does cloud fit in with the overall architecture of IoT?

In general, there are two kinds of IoT software architectures:

    + **Cloud-centric**: Data from IoT devices such as sensors are streamed to a data centre where all the applications that do the analytics and decision making are executed, using real-time and past data from one or more sources. Servers in the cloud control the edge devices too.
    + **Device-centric**: All the data is processed in the device (sensor nodes, mobile devices, edge gateways), with only some minimal interactions with the cloud for firmware updates or provisioning. Terms such *Edge Computing* and *Fog Computing* are used in this case.Today, for IoT Cloud Platforms, the goal is to stretch the analytics and data processing across Cloud and Device, leveraging the resources at each end seamlessly. In general, we are beginning to see a shift towards leveraging the compute and service capabilities of the cloud to manage IoT devices better. This is also quite evident from a snapshot of the Google Trends showing increasing interest in Cloud compute compared to just IoT.

### How is an IoT cloud platform different from traditional cloud infrastructure?

The traditional cloud infrastructure focuses on a model of cloud computing where a shared pool of hardware and software resources are made available for on-demand access in such a way that they can be easily and rapidly provisioned and released with minimal effort. IoT Cloud Platform extends this capability to resources that are more user-centric, which increases the count and scale of data and devices. The cloud platform services can not only process big data from a wider set of IoT devices, but also provides a smart way to provision and manage each of them in an efficient manner. This also includes fine-grained control, configuration and management of IoT devices.

One of the IoT Cloud platform differentiators is the ability of the engine to massively scale to handle real-time event processing of large volumes of data generated by various devices and applications. The providers of IoT Cloud Platforms typically work with multiple parties such as hardware vendors (both for cloud services and IoT devices), telecommunication providers, software service providers and system integrators to build the platform.

### What exactly is meant by Application Enablement Platform (AEP)?

The world of IoT is one of variety: many hardware platforms, many communication technologies, many data formats, many verticals, and so on. AEP is a platform that caters to this variety by providing basic capabilities from which developers can build complete end-to-end IoT solutions. For example, AEP might offer location-tracking feature rather than a more restrictive fleet tracking feature. The former is more generic and therefore can be used in a number of use cases.

AEP gives faster time to market without sacrificing on customization and product differentiation. The disadvantage is that users of AEP must have the skill sets to develop the solution. The solution might also suffer from vendor lock-in.

With AEP, app developers need to worry about scaling up. AEP will take care of communication, data storage, management, application building and enablement, user interface security, and analytics. When selecting an AEP, developers should consider developer usability including good documentation and modular architecture, flexible and scalable deployment, good operational capability, and a mature partnership strategy and ecosystem.

Examples of AEP include ThingWorx Foundation Core, Aeris' AerCloud, and deviceWISE IoT Platform.

### Could you list some IoT cloud platforms out there in the industry today?

With the advent of IoT with billions of devices getting connected to the internet which not only does compute, storage and run applications, there is also a needed to handle large amounts of data coming into the system via the various interfaces such as sensors and user inputs.

Here are some IoT cloud platforms:

    + Amazon Web Services IoT
    + IBM Watson IoT Platform
    + Microsoft Azure IoT Hub
    + Google Cloud IoT
    + Oracle Integrated Cloud for IoT
    + SAP Cloud Platform for the Internet of Things
    + Cisco Jasper Control Center
    + PTC ThingWorx Industrial IoT Platform
    + Salesforce IoT
    + Xively
    + Carriots

### What factors should I consider when comparing different IoT cloud platforms?

Comparison across different platforms depends on both business and technical factors: Scalability, Reliability, Customization, Operations, Protocols, Hardware agnostic, Cloud agnostic, Support, Architecture and Technology Stack, Security and Cost. For example, a comparison of AWS IoT (serverless) and open-source IoT deployed on AWS showed that the former reduces time to market but is expensive at scale.

The end-to-end requirements and cost-benefit analysis between commercial and open-source solutions need to be considered while choosing the right platform. One way to compare is to look at the best fit to various sectors, viz. Management of various Device, System, Heterogeneity, Data, Deployment, Monitoring and fields of Analytics, Research and Visualization.

Each of these sectors has its own performance criteria such as real time data capture capability, data visualization, cloud model type, data analytics, device configuration, API protocols, and usage cost. Data analytics performance and outcome also depends on factors such as device ingress and device egress, intermediate connectivity network latencies and speeds and support for optimized protocol translations. Visualization of data, filtering of large masses of data and configurability of the millions of devices using smart application tools is another differentiating factor.

### What are some challenges of adopting IoT cloud platforms?

Security and privacy are the main concerns delaying the adoption of IoT Cloud Platforms. Cloud providers typically will not own the data and are only authorized to do the analytics and control of systems as permitted by the owner of the data. Any breach of data access either during transit or from storage is a concern from privacy and security perspective. Also, since the value of IoT data is immense, proper legal agreements and mechanisms must be in place to ensure the data or outcome of data analysis is only used for the intended purpose by the authorised personnel.

Existing IoT cloud platforms may not always conform to standards, thereby causing interoperability issues. They may also not support heterogeneous modules or communication technologies. When there's too much data, context awareness can help, including decisions of what needs to be done at the edge. Vertical silos continue to exist and this prevents horizontal flow of information. Middleware can solve this problem. Many system continue to use IPv4 and this could be a problem as devices run out of unique IP addresses.

### Could you compare IoT components from Amazon, Microsoft and Google?

Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP) are generic cloud platforms that have IoT-specific components. These can be compared across various metrics.

Azure offers both PaaS and SaaS options. Its Azure IoT Edge helps with data analysis at the edge. Azure's SDKs are either for running on device or as a service on the cloud. Azure has support for a number of messaging protocols. All platforms provide device SDKs to enable devices to easily authenticate and connect to the cloud.

Amazon FreeRTOS enables sensor nodes to easily connect to cloud while Amazon Greengrass brings cloud capability to edge devices. AWS offers **Device Shadow**, which is a persistent virtual equivalent of the actual device. This is useful if the device goes offline or suffers from intermittent network connectivity. Rules Engine can route messages a variety of AWS endpoints.

Google's Cloud IoT Core supports gRPC for efficient binary messaging. It also has hardware partners for easy device integration.

## Milestones

Oct
1969

As a U.S. defense project, the ARPANET is launched. By late 1980s, this evolves into a publicly accessible packet-switched network called the Internet.

1989

While at CERN, Tim Berners-Lee creates the World Wide Web (WWW) that's uses Internet as its backbone. A year later, he implements the first client and server communication over a network.

1999

Kevin Ashton coins the term **Internet-of-Things (IoT)** while making a presentation to Procter & Gamble. He notes that technologies such as sensors and RFIDs are better at tracking things in the real world than humans.

2002

To enable others to benefit from the technologies at Amazon, Amazon launches **Amazon Web Services (AWS)**, one of the first commercial Cloud Services Platform. With the infrastructure of AWS, businesses could build and manage their websites in a lot easier way. AWS is relaunched in 2006 with the release of Amazon Elastic Compute Cloud (EC2).

2013

IoT cloud platforms have been on the rise since late 2000s. The year 2013 sees a peak in new startups in this space.

Jul
2016

A market study reveals that there are 450+ companies offering IoT platforms, up from 360 companies in 2015. Not all of these are cloud platforms. A third of them cater to industrial IoT. Niche platforms focus on vertical use cases rather than horizontal cross-industry use cases.

IoT Cloud Platforms


# XML

## Summary

eXtensible Markup Language (XML) is used to store, transmit, and reconstruct data. It's not a programming language. It's a markup language used to represent data using tags. W3C's XML specifications and other related recommendations help in implementing XML in an interoperable manner.

By design, XML focuses on generality and usability across the Internet. It supports Unicode that allows communication of information in any written human language. It supports common computer science data structures such as records, lists, and trees. It allows validation using schema languages such as XSD and Schematron. Several schema systems facilitate the definition of XML-based languages.

Verbosity, complexity and redundancy are the major disadvantages of XML.

## Discussion

### What are the essential XML terms?

Some key terms used in XML programming are:

    + **XML Declaration**: XML documents mostly begin with a declaration that gives information about them. An example is `<?xml version="1.0" encoding="UTF-8"?>`.
    + **Tag**: Tags are basic markups used in XML. They define structure of the document. Each tag begin with character `<` and ends with character `>`. There are three types of tags namely start tag, end tag and empty-element tag.
    + **Attribute**: It's a name-value pair that exists within a start tag or empty-element tag. Example, in `boy="rahul"` 'boy' is name and the 'rahul' is value.
    + **Element**: Everything within and including the start tag and the end tag. Everything between these tags is called *element content*.
    + **Markup & Content**: An XML document is made up of markup and content. Strings that begin and end with characters `<` and `>` are markup. Non-markup strings are content.
    + **Processor**: It analyzes markup and passes structured information to an application.

### What are some applications of XML?

XML is widely used for interchange of data over the internet. It's also used for formatting documents, web searching and storing configuration data. Document formats such as SVG, RSS, Atom and XHTML have been developed using XML. It serves as the base language for communication protocols such as SOAP and XMPP. It's the message exchange format for the Asynchronous JavaScript and XML (AJAX) programming technique.

Microsoft Office 2007 introduced XML-based file formats for documents, spreadsheets, presentations, their templates, and more. XML led to smaller files and easier data recovery. Open XML format allowed other tools to interoperate for tasks such as comparing files or removing sensitive information.

In the cloud, XML is commonly used. Google Cloud Storage exposes an XML API. Azure has transformation policies to convert API requests and responses between JSON and XML. Application builds and deployments can be configured in XML. Oracle XML DB (on Amazon RDS) natively supports XML.

XML is the base for many industry data standards such as Health Level 7, OpenTravel Alliance, FpML, MISMO, and National Information Exchange Model.

### What's a valid well-formed XML document?

XML documents that conform to syntactic rules defined in the W3C Recommendation are said to be **well-formed**. For a document to be **valid**, it should also conform to a Document Type Definition (DTD) or XML Schema. A valid document is necessarily well-formed but the reverse is not true. A well-formed document may be invalid or even without a DTD or schema.

Since the W3C Recommendation defines the syntactic rules, no additional inputs are necessary to check for well-formedness. But to check for validity we need to specify a DTD file in the XML document, such as, `<!DOCTYPE advert SYSTEM "http://www.foo.org/ad.dtd">`, where `advert` is the root element. Alternatively, XML Schema may be specified within the document or passed to the parser separately.

### What are some important syntactic rules in XML?

Each XML element must have a **closing tag**. Even for an empty-element tag (such as `<author name="J Smith" />`), characters `/>` signify the closing part of the element. In HTML, empty-element tags are used for images, breaks, horizontal lines, inputs, metadata, etc.

XML tags are **case-sensitive**. For example, `<Body>` and `<body>` are two different tags. Hence `</body>` can't be the end tag for `<Body>`. Keywords such as `DOCTYPE` and `ENTITY` must be in uppercase.

All XML elements must be properly **nested** and not overlap. For example, `<b><i>Hello world!</b></i>` is incorrect since `</i>` must come before `</b>`. A **root element** is a must in all XML documents. This implies that elements can't be siblings at the top level. They must be enclosed with a parent root element. Each element can have only one parent.

Attribute values must always be quoted. For example, `<note data=05/05/05>` should be corrected as `<note data="05/05/05">`. Whitespaces are significant in XML. Tag names must not contain spaces. They should not start with a number or punctuation but can start with an underscore.

### What are some other XML constructs worth knowing?

**Comments** in XML start with `<!--` and end with `-->`. Occurrence of `--` inside comments is invalid. For example, `<!-- declarations for <head> -->` is a well-formed comment and `<head>` is not interpreted as a start tag.

**Processing Instructions (PIs)** pass useful information to applications. They start with `<?` and end with `?>`. For example, `<?xml-stylesheet href="person.css" type="text/css"?>` tells a web browser to apply the specified stylesheet.

**Character Data (CDATA)** is a section of element content marked for the parser to interpret as character data. In other words, CDATA content is not interpreted as markup. Hence, any character can be included in CDATA content. It can be used to embed an XML document within another. It's one method to ensure that an XML document is well-formed. CDATA section starts with the string `<![CDATA[` and ends with the `]]>`.

### What's a typical XML processing pipeline?

The processor takes an input XML pipeline document and execute pipeline processes. An XML pipeline document specifies the processes to be executed in a declarative manner. One can use multistage processing pipeline in which XML components are processed sequentially or in parallel. The output of one stage of processing is the input of another stage of processing. One can write a pipeline document that defines the inputs and outputs of the processes. The figure shows a possible pipeline sequence.

A typical XML processing pipeline parses the input XML documents, then validates those documents and then transforms the documents.

### What are the differences between XML and HTML?

HyperText Markup Language (HTML) is a static markup language used to create web pages and web applications. Like XML, HTML has a hierarchy of nested elements and attributes. However, HTML's main concern is how documents are displayed on a web browser. XML's main concern is how information is organized.

XML has syntax but no semantics. HTML has both. For example, `<h1>` tells web browsers to render the content as a header, perhaps in a larger font size and bolder weight. Similar semantics are missing in XML. Semantics are imposed by applications and not by XML parsers.

HTML tags are all well-defined and limited to the web. XML can be extended and used in any application. Data architects and developers can create tag names. Businesses can use XML to integrate information flows across systems.

A web browser parsing HTML is more tolerant of errors. It may ignore overlapping tags, case mismatch of tag names, missing end tags or missing quotes around attribute values. XML parses are generally not tolerant of errors. They expect well-formed documents.

### What are the shortcomings of XML?

XML is a **verbose** language. Developers need to type opening and closing tags while writing XML documents. It doesn't support array data types. Repetition is a major disadvantage. Every element/attribute name has to be explicitly written for every element/attribute instance. Many web developers choose JSON over XML due to its lack of terseness.

Due to its verbosity, XML impacts **performance**: more disk storage, more bandwidth to transfer XML documents, and more memory consumption and longer time to process. The choice of character encoding can increase the size. For example, integer value 15383 needs only 2 bytes in binary but might need 5-20 bytes in text form. Unicode UCS-4 encoding requires 4 bytes for each character.

Since XML documents can refer to external files, this presents a **security risk** apart from the performance impact due to slow network connections.

XML has **no semantics**. Each particular XML tag has to define what its elements and attributes mean. XML doesn't predefine anything for them. One needs to write application code (Java, Python, etc.) and their metadata (in `*.xml`) separately. It diminishes the readability of application code.

### What are other W3C technologies closely related to XML?

W3C technologies closely related to XML include:

    + **XML Schema Definition (XSD)**: It specifies how to formally define the elements in an XML document.
    + **Extensible Stylesheet Language Transformations (XSLT)**: It's used to convert XML into other document formats such as HTML for web pages, plain text or XSL-FO, which further may be converted into other formats such as PDF, PostScript and PNG.
    + **XML Path Language (XPath)**: It's used to address parts of an XML document. It can be used by both XSLT and XPointer.
    + **XML Query (XQuery)**: It's a standardized language for combining documents, databases and web pages. It can replace complex Java or C++ programs with a few lines of code.
    + **XML Pointer (XPointer)**: It's a W3C Recommendation that allows links to point to specific parts of an XML document. It uses XPath expressions to navigate.
    + **XSL Formatting Objects (XSL-FO)**: It's used for the transformation and formatting of XML data. It's most often used to generate PDF files.

## Milestones

1995

Some **SGML (Standard Generalized Markup Language)** practitioners experienced with the new World Wide Web (WWW) believe that SGML can offer solutions to some of the upcoming problems on the Web. Dan Connolly joins W3C and adds SGML to the list of W3C's activities.

1998

**XML 1.0** becomes a World Wide Web Consortium (W3C) recommendation. XML is a profile of SGML (ISO 8879). It adopts many aspects of SGML: separation of logical and physical structures, separation of data and metadata, separation of representation from processing instructions, mixed content, DTD-based validation, etc.

Jan
2000

**Extensible HyperText Markup Language (XHTML) 1.0** becomes a World Wide Web Consortium (W3C) Recommendation. Essentially, XHTML reformulates HTML as XML so that standard XML parsers can parse web content. This enables web developers and content providers to define new elements and attributes. Content can be transformed to suit its context. User agents would become more interoperable. In reality, later years show that XHTML doesn't achieve wide adoption.

2006

**XML 1.1** becomes a W3C Recommendation. Many improvements are standardized in this edition, such as support of Unicode line separator character `#x2028` and release of a set of constraints called "full normalization" on XML documents.

XML


# Text Classification

## Summary

Abundant textual data accumulates in any eco-system, unstructured and in diverse formats. To extract trends or meaningful insights from it, we need to sort data into different categories. Text classification is a simple, powerful analysis technique to sort the text repository under various tags, each representing specific meaning. Typical classification examples include categorizing customer feedback as positive or negative, or news as sports or politics.

Machine Learning is used to extract keywords from text and classify them into categories. Text classification can be implemented using supervised algorithms, Naïve Bayes, SVM and Deep Learning being common choices.

Text classification finds wide application in NLP for detecting spam, sentiment analysis, subject labelling or analysing intent. Automating mundane tasks makes search, analysis and decision making faster and easier. Text classification is very effective with historical data. It can also be used for real-time textual input analysis.

## Discussion

### What's the scope of text classification?

The scope of text classification is at document level, paragraph level, sentence level, or even sub-sentence level. In some applications we may assign one or more classes to an entire document. In other applications, we may assign a class to each paragraph or sentence in the document.

An algorithm may be designed to accept syntax and semantic information at a sentence level. This is then used to classify the document. While sentence-level analysis is more granular, it's limitation is that often sentence-level context can be determined only from sentences surrounding it.

### What are the different methods to perform text classification?

Some methods of text classification include the following:

    + Manual classification: We could do the sorting manually like secretaries did in the olden days. Assign categories to every incoming text. Then group documents as per common categories. This method will have the best accuracy, but possible only for small quantities.
    + Hand-crafted rules for automated classification: A domain expert defines the rules for categorization. For example, text with words "shares, profits, equity, liabilities" is classified into Business category. Text with words "Messi, kick, Ronaldo, goal" is of Football category.
    + Machine Learning Algorithms: Supervised techniques chosen based on dataset size and number of categories.
    + Hybrid techniques: Start with manual classification and assign categories to a small portion of the training dataset. Then apply ML algorithms to extend to the rest of the dataset.

### How does text classification work?

Let’s work with a supervised classifier example to classify customer feedback into one of these categories: COMPLAINT, SUGGESTION, APPRECIATION. Text may not explicitly include these words, so the categorisation has to happen based on interpretation of words present.

For the classifier to predict the correct label, it needs to ‘understand’ the essence of the text. So we train the classifier with millions of similar customer feedback texts which already have one of these labels assigned. After training, we allow the classifier to test its predictions. Once desired accuracy levels are attained, the classifier can now be used to categorise new customer feedback.

With text classification, the algorithm doesn’t care whether the user wrote standard English, an emoji, or an indirect reference (poetry, sarcasm, movie quotes). It only cares about the frequency of occurrence of particular text resulting in getting assigned to a category. Let’s say the words “Out of this world” occurs very frequently whenever a very appreciative customer feedback is present in the test dataset. The chance of a new prediction getting classified as “APPRECIATION” becomes very high now, if this phrase is encountered.

### How do ML algorithms automate text classification?

    + **Naïve Bayes**: Its common application is in spam filtering, to classify incoming emails/SMS as spam or not-spam. We apply conditional probability model of Baye’s theorem to answer “What is the probability the message is spam given occurrence of word X?”. We assume that samples are independent and identically distributed. Though this is often untrue, the prediction accuracy is reasonably good, especially for small sample sizes.
    + **Support Vector Machines**: For the same spam filtering, SVM offers better accuracy in results than Naïve Bayes since it uses an optimization technique. However, it requires more computational resources. SVM builds an optimal separating hyperplane which maximises the margin separating the categories (Spam/Not Spam in our case). Unlike Naive Bayes, SVM is a non-probabilistic algorithm.
    + **Deep Learning**: Works well when datasets are huge in size and continuously growing. For spam filtering on a large scale (huge dataset, large number of categories), RNN (LSTM in particular) would be highly effective as it gives weightage to the sequence of words appearing in the text. Convolutional Neural Networks are a good choice for hierarchical document classification to recognize patterns in the text sequence.

### How should I prepare the data for text classification?

Data preparation for ML-based text classification involves the following:

    + Tokenization: Identifying words, symbols, emojis, hyperlinks, based on known delimiters and format rules.
    + Word normalization: Reduce derived words into their root form (developmental becomes develop, encouragement becomes encourage).
    + Text and feature encoding: ML models require numeric features and labels to provide a prediction. So we create a data dictionary to map each word/feature in the document and each category label to a numerical ID. (Example category codes: COMPLAINT -> 0, SUGGESTION -> 1, APPRECIATION -> 2).
    + Feature representation: Every feature (category) is represented as a Word Count Vector (giving frequency count of each feature) or a TF-IDF vector (Term Frequency/Inverse Document Frequency) representing relative importance of a term in a document. As an example, Microsoft Azure has *Feature Hashing* module to convert features to integers for input into model.
    + Word/Document embedding: Every row in the dataset is an entire document, represented as a dense vector. The word position within the vector is learned from text and based on the surrounding words. Word embeddings can be trained using the input corpus but pre-trained embeddings (Glove, FastText, and Word2Vec) are available.

### How should I select features for text classification?

Feature selection involves picking a small but optimal subset of terms from the training dataset to use as features (categories) during text classification. Since vocabulary size reduces, computational efficiency during training improves that's critical for algorithms other than Naïve-Bayes. Feature selection also helps minimise noise features, which increase classification error when included.

Feature selection is essentially a process of dimensionality reduction. Contrary to belief that more categories means better classification, weaker models (fewer but well-differentiated features) deliver better accuracies when training data is limited. Feature selection methods are classified into 4 types:

    + Filter: A pre-processing step suitable for any ML algorithm. Statistical tests are done for features and their correlation with the outcome variable. Features are chosen based on test scores. Feature/Response: continuous: Pearson’s Correlation.
    + Wrapper: Based on greedy search algorithms, they choose optimal features for a specific ML algorithm. Example: Step forward/backward feature selection (Add / remove features 1 by 1 in round-robin fashion from feature set and evaluate performance)
    + Embedded: Feature selection is part of ML algorithm training phase (LASSO, Ridge Regression to reduce overfitting).
    + Hybrid: Mix of filter and wrapper methods.

### What are the methods to evaluate the effectiveness of text classification?

Cross-validation and hold-out tests are common evaluation methods to deduce how often a prediction was right (true positives, true negatives) and when it made a mistake (false positives, false negatives).

    + N-fold cross-validation: Split dataset into N folds. Runs test N times. At a time, use one fold of data as test set, remaining N - 1 folds of data as training sets. Classification accuracy is average of results in N runs.
    + Hold-out test: Divide dataset into training and test subsets. Varied splits will result in varied results and accuracy, especially for small datasets. Paired t-test can be used to measure significance in accuracy differences.Well-known performance metrics used in assessment include accuracy, precision, recall and F1 score. Classification error (1 - Accuracy) is a sufficient metric if the percentage of documents in the class is high (10-20% or higher). However for small classes, always saying ‘NO’ will achieve high accuracy, but make the classifier irrelevant. So precision, recall and F1 are better measures.

Other evaluation metrics include Matthews Correlation Coefficient (MCC), ROC and AUC.

### What are the factors behind the choice of an ML algorithm?

There are some common pointers that can be followed to determine whether to choose an supervised or unsupervised algorithm. The choice depends on the dataset size, accuracy required, training time or resources available.

For smaller datasets with humanly identifiable or limited categories, choose a Supervised algorithm. For impossibly huge, ever growing datasets, or too many categories, choose Unsupervised algorithm or ANN model. While text classification is usually considered a supervised task, sometimes the term has been used for unsupervised tasks. Text clustering is an example that identifies the categories before assigning them to documents.

Calculate the ratio number of samples/number of words per sample. If this ratio is less than 1500, tokenize the text as n-grams and use a simple multi-layer perceptron (MLP) model to classify them. If the ratio is greater than 1500, tokenize the text as sequences and use a CNN model to classify them.

### How to adapt binary text classification techniques to multi-class problems?

More classes means higher model complexity. For a fixed dataset, binary classification gives better accuracy than multi-class because of clear decision boundaries. It’s a common practice to reduce multi-class instances into binary classifiers before running ML algorithms.

Let’s take a training set with 5 classes: "Class-A", "Class-B", "Class-C", "Class-D", "Others". We relabel data under "Class-ABCD" accumulating labels "Class-A", "Class-B", "Class-C" and "Class-D". Now it’s a binary-classification problem with just 2 classes “Class-ABCD”, “Others”.

A popular problem reduction technique called OAA (One-Against-All) is employed for this:

    + For each of the 5 classes in our dataset, we create 5 new datasets D1…D5 (one per class)
    + For each dataset Di, we mark training data of the corresponding class i as positive and all others as negatives (D2 : Class-B -> Positive, All others -> Negative)
    + We train 5 binary classifiers, each on their own dataset Di
    + We get predictions from all binary classifiers. Then choose the class with positive classifier output, breaking ties randomly.An undesirable consequence is class imbalance. As negative examples far outweigh positives, learning skews towards the negative class. This is prevented by introducing relative weightages for each dataset.

### What support is available in various programming languages and libraries?

Briefly, we list some available resources:

    + Python: Scikit-Learn: sample datasets, feature encoding, Ppediction and evaluation
    + TensorFlow: TF-Hub: training, prediction, evaluators
    + Java: Stanford CoreNLP, MALLET, WEKA, Lingpipe libraries
    + R: OpenNLP, RwekaRandolphVI maintains a list of useful neural network Python implementations for Multi-Label Text Classification. This includes CNNs, RNNs, and attention networks.

## Milestones

1960

Since the 1960s, the concept of classifying documents into categories exists as a part of library sciences. Several automated library management tools include this feature.

1990

In the 1990s, text classification gains importance as an important NLP function. With the availability of larger datasets and more categories, researchers apply ML-based classification. For text classification, Support Vector Machines (SVM) and Maximum Entropy Models (MEMs) are applied in the late 1990s. By the end of the decade, text classification is seen a combination of information retrieval and machine learning. It's also seen as an application of **text mining**.

2000

In the early part of 2000s, researchers look at multi-label text classification. Unlike topic modelling where topic candidates are ranked, text classification requires a definite set of topics. Approaches include EM algorithm, Parametric Mixture Models (PMM) and multi-labelled MEMs.

2004

Pang and Lee publish a paper on sentiment classification based on Machine Learning techniques. They propose a 'subjectivity detector' that would filter out the sentences labelled 'subjective' in a document and employ text categorization techniques on the resulting data. They implement Naive Bayes and SVM to find minimum cuts in a graph. They claim an accuracy of 86.4% on the NB polarity classifier.

Oct
2014

CNNs are common for image processing but not in NLP. Yoon Kim presents the idea of using a CNN to classify text in a paper titled *Convolutional Neural Networks for Sentence Classification*.

Text Classification


# Typosquatting

## Summary

Typosquatting is the malicious practice of registering domain names that closely resemble popular brands and businesses. Attackers do this in the hope of deceiving users. A user might mistype the web address and land up on a malicious site. The site may show harmless ads. More seriously, it might look like the genuine site. The user may then perform transactions and thereby disclose sensitive information.

Typosquatting can also be done on software packages. Developers might unknowingly download and install malicious packages. This compromises the security of their applications.

Typosquatting could be combined with other cyberattacks such as phishing. Domain owners, developers and users have to take precautions to protect themselves against typosquatting.

## Discussion

### What aspects of software can be attacked by typosquatting?

Typosquatting can occur on **domain names**. This is sometimes called *DNS Typosquatting* or *URL hijacking*. Suppose a user wishes to visit `live.com` but mistypes this as `livve.com`, `live.cm` or `liv.ecom`. This is not exactly harmful if such domains don't exist. However, attackers can register these domains and thereby redirect requests meant for `live.com` to their own servers. This compromises the intended client-server interaction.

**Software supply chain** is about how software packages are distributed by developers and downloaded by other developers for use in their applications. A developer wishing to use a package may mistype the name and thereby install the wrong package. The wrong package has been shared by the attacker. Thus, instead of installing `atlas_client`, the developer may install `atlas-client` that contains malicious code.

Thus, broadly we have domain typosquatting or package typosquatting. In either case, typosquatting makes sense for highly popular domains or packages. Even if a small proportion of users make a typo error, the attackers stand to benefit a great deal.

### Which are some variations or techniques in typosquatting?

Let's assume that the correct domain name is "cisecurity.org". Here are some typosquatting variations, some of which can be easily mistyped by users:

    + **Omission**: "csecurity.org" where the first "i" is omitted.
    + **Addition**: "cissecurity.org" where an extra "s" is added.
    + **Substitution**: "cisecurlty.org" where "l" fills in for a "i". Another example is "cisecurity.com".
    + **Transposition**: "csiecurity.org" where "i" and "s" positions are swapped.
    + **Hyphenation**: "ci-security.org" where there's an extra hyphen.These variations can be described differently. Assuming that the domain is "google.com", we can have misspelling (goggle.com), extra period (goo.gle.com), confusion between numbers and letters (g00gle.com), phrasal change (googles.com), or additional words (googlesearch.com).

### Which are some advanced typosquatting techniques?

Here are some advanced typosquatting techniques:

    + **Homograph Attacks**: This relies of visual similarity. For example, uppercase "i" in sans-serif font looks similar to "l". Another example is "у", which is actually the Cyrillic U that looks like Latin "y". However, this sort of an attack is not that common.
    + **Bitsquatting**: This relies on computer hardware to make errors at bit level. For example, "microsoft.com" could be typosquatted as "mic2osoft.com", with one bit in error (0x32 for "2" and 0x72 for "r").
    + **Soundsquatting**: When two words sound similar but are spelled differently, users can end up picking up the incorrect one. For example, "ate" may be mistaken for "eight".
    + **Typosquatting Cross-Site Scripting (TXSS)**: This relates to software developers who use the wrong domain name in their code. For example, they might mistype the URL of a JavaScript library that they wish to use in a HTML page. This is a series problem since every user visiting the page gets exposed to the typosquatting domain.

### What harm can domain typosquatting cause to systems and businesses?

Popular brands lose traffic as their users get redirected to typosquatting domains. In fact, traffic can get redirected to competitor sites.

Data theft can erode brand value. In 2011, variations of YouTube.com appeared and these redirected to a survey form that collected mobile numbers from unsuspecting users. The "survey" lured users to an SMS subscription that was charged to their phone bills. In another example, omitting a dot in an email addresses resulted in loss of 120,000 emails from Fortune 500 companies over six months. These emails contained trade secrets and invoices.

Typosquatters can abuse affiliate programs and collect referral fees on behalf of genuine site owners.

Users can be tricked into downloading malware. When typosquatting sites look exactly like their genuine counterparts, users may be tricked to enter login credentials and disclose sensitive information. Some typosquatting sites are less harmful. For example, a company offering vacation packages may typosquat on the domains of major airlines and show ads for vacation packages.

It's been found that financial services, retail and critical infrastructure are industries most affected by typosquatting.

### Which are some best practices to protect against domain typosquatting?

If you have a domain, one defensive technique is to register common variations and typo errors of that domain.

Implement two-factor authentication (2FA) for the safety of your customers. Always use SSL certificates so that customers can verify site ownership. Educate your users so that they're aware of common attacks. Having a trademark gives you the option to file a case against typosquatters. Improve SEO on your site so that search engines rank it better than the typosquatted ones.

As a user, enable 2FA for better protection of your accounts. When sensitive information or money transfer is involved, it's better to verify by call before performing the online transaction. Pay attention to how domains are spelled, in URLs and email addresses. Check that site is served on HTTPS. It's equally important to check the domain name that browsers display before the URL.

Avoid typing out URLs. Instead, bookmark the sites you often visit and use these bookmarks instead. In addition, you can install suitable browser plugins that warn potential typosquatting domains on mistyped URLs.

### What tools or algorithms are available to counter domain typosquatting?

DNS Twist can "detect typosquatters, phishing attacks, fraud, and brand impersonation." By running *dnstwist* on your domain, you can find if it's being typosquatted.

First released in 2006, Microsoft provides Strider URL Tracer to investigate third-party domains. This also generates typosquatted variations and scans them.

Some typos are more likely than others. A probabilistic analysis helps in ranking them. Typing test tools and human-based computation games can help estimate the probabilities. For example, adjacent characters on the keyboard are more likely to be swapped.

Feature engineering is an essential task when building your own detection algorithm. Features could be based on domain average TTL value, domain-IP address association, common ASNs shared by IP addresses, hit-count of domain, n-gram distribution of characters in domain name, and many more. Lot of information can be obtained from DNS records, WHOIS lookup, and even crawling the site.

Algorithms could be knowledge-based where experts identify and apply features. With sophisticated attacks, machine learning-based algorithms are more effective. They're data-driven. ML for domain typosquatting is possible due to easily available data that's collected by ISPs and network administrators.

### What are some AI/ML approaches to detecting typosquatting domains?

Among the ML approaches are:

    + **Supervised**: Relies on labelled datasets. *DomainProfiler* is an example that used 55 features along with random forest algorithm. The limitation is that labelled datasets are expensive to create.
    + **Semi-Supervised**: Uses both labelled and unlabelled data. Graph-based inference methods such as belief propagation are used in which it's assumed that malicious hosts are most likely to communicate with malicious domains. Another approach is clustering based on co-occurrence patterns such as domain names often seen together in DNS resolver logs.
    + **Unsupervised**: Clustering is done to identify at least two clusters of malicious or benign domains. This is then extended to identify sub-clusters based on the type of malicious behaviour.In practice, hybrid approaches are adopted to achieve better results. For example, one approach used three different feature selection methods, reduced them to a subset of features, and trained a bagging ensemble model. Two ensemble models were explored: decision tree and k-nearest neighbour.

For evaluation of these algorithms, accuracy, precision, recall, F1-score and AUC are commonly reported.

### What harm can package typosquatting cause to applications and their users?

In software package management, typosquatting can be used to steal cryptocurrency. Malicious packages can attempt to redirect cryptocurrency payments to a wallet address of their choice. This has been seen in Ruby (via RubyGems) and JavaScript (via npm).

When a typosquatting package is installed, it gets access to environment variables. Such packages then attempt to steal user credentials and other secrets that are usually part of environment. A well-known example of this (from August 2017) is `crossenv` typosquatting on `cross-env` at npm. In PyPI, two typosquatting packages attempted to steal GPG and SSH keys from developers.

The typical exploit is to run arbitrary code during installation. Worse still is when such an execution is done with administrative privileges. Some packages may even attempt to inject malicious code into the application's source code.

A less harmful attack is to replicate the original functionality and thereby divert attribution from the original package. On npm, typosquatting package `asimplemde` does this for `simplemde`.

### What can I do to mitigate package typosquatting?

Developers must be more vigilant when copying and pasting package names. Use safe options when installing dependencies. For example, use the `--ignore-scripts` option when installing using npm.

Developers should use digital signatures (using `gpg`) and checksums (using `sha256sum`) to deliver their software packages in a secure way. This prevents hackers from modifying the content to include typosquatted names.

Package repositories for their part can have checks and rules that minimize the attack surface. For example, npm requires that new packages can't match existing package names with the punctuation removed. Given `react-native`, names `react_native` and `re.a_ct-native` are not allowed.

Repositories can also maintain a list of possible names with small Levenshtein distances. Administrators can be notified if there's an attempt to install one of these. In fact, server logs would contain 404 errors for non-existing packages that users typed by mistake. These can be added to the list to proactively prevent typosquatting.

**SpellBound** is a tool that can be used to detect typosquatting names. It adds a runtime overhead of less than 2.5%.

## Milestones

1998

R.C. Cumbow in The New York Law Journal uses the term **typosquatter**, probably the first published use of the term.

1999

Seeing the growing threat of domain typosquatting, the Internet Corporation for Assigned Names and Numbers (ICANN) introduces the **Uniform Domain Name Dispute Resolution Policy (UDRP)**. In the U.S., there's also the Anti-cybersquatting Consumer Protection Act (ACPA).

Jan
2003

What is possibly one of the first large-scale studies, Benjamin Edelman identifies 8,800 typosquatted domains. Many of these have invalid WHOIS data, provide explicit sexual content and even prevent users from exiting the site. Many of these sites are registered to John Zuccarini, who is eventually arrested in September.

Mar
2003

As part of RFC 3490, the IETF defines Internationalized Domain Name (IDN). This allows domain names to contain alphabet-specific characters. This unfortunately opens the doors to **homograph attacks**. English domain names could be typosquatted by substituting some letters with non-Latin letters that have identical glyphs (same visual appearance).

2006

An analysis of top 10,000 domains shows that 30% of all missing-dot typos come from just six domain parking services. These six account for 2995 typosquatted domains. In all, 5094 typosquatted domains are found to be active.

2008

Banerjee et al. observe that about 99% of all typosquatted domains use **one-character modification**. More formally, such a domain differs from the real domain by a Damerau-Levenshtein distance of one. They also propose ways to automate typosquatted domain name generation: 1-mod-inplace, 1-mod-deflate, 1-mod-inflate.

2011

At the BlackHat Technical Security Conference, Artem Dinaburg reveals **bitsquatting** as a technique towards typosquatting. Random bit errors in domain names can cause computers to fetch content from bitsquatted domains. With 30 bitsquatted domains, he finds 52,317 requests from 12,949 unique IP addresses over a period of eight months. An example of this is "www.mic2osoft.com". Typical use is to serve ads, abuse affiliate programs or install malware.

2014

Szurdi et al. propose a framework to detect typosquatting domains. They start with passive data sources from which they generate possible typosquatting domain names. To obtain more information, they actively crawl WHOIS database, collect DNS data and obtain web page content. They cluster domains based on various features. Their approach is called *Yet Another Typosquatting Tool (YATT)*.

Feb
2018

One researcher investigates package names on npm in terms of Levenshtein distance. He finds that 46% of names are at a distance of two or less. Longer the name, less likely it has a small distance to another name. Distance of one can indicate typosquatting. However, npm's preference for short names makes this hard. There are many genuine packages that have small distances: (preact, react), (bocha, mocha), and (class-names, classnames).

Mar
2018

A network of malicious sites ending in ".cm" is discovered. These sites were typosquatting on many popular ".com" sites and attempting to trick users with fake security alerts. Four years of access logs from this network is also discovered and analyzed by security experts. They find that within the first three months of 2018, these sites got about about 12 million visits, almost 50 million visits per year.

Apr
2020

Packages for Ruby (called gems) are distributed via RubyGems repository. An analysis by ReversingLabs finds more than 700 malicious gems due to typosquatting. They describe the example of `atlas_client` typosquatting as `atlas-client`. It contains `aaa.png` that's in fact a portable executable. It contains a Base64 encoded VBScript. It's designed to steal cryptocurrency by modifying wallet addresses.

Typosquatting