# Joule Thief Circuit

## Summary

The Joule Thief Circuit is a voltage booster circuit which converts a constant low voltage input into a periodic output of a higher voltage. This circuit can be most often seen lighting an LED with an almost dead AA battery. The peaks in voltage occur rapidly, causing the LED to flash at a very fast rate. However, the LED appears to be constantly lit to the human eye due to the persistence effect.

## Discussion

### How does the Joule Thief circuit work?

The circuit is an arrangement of a power source, a resistor, a transistor and a ferrite toroid core wrapped with two wires coming from the positive terminal of the power source, one through a resistor.

A magnetic field is created around the ferrite toroid because of the current that passes through the wires. The extra current causes the transistor to switch off and power to the ferrite toroid is cut off. As a result, the magnetic field is converted into electrical energy which is given as output. Once the magnetic field no longer exists(the pulse ends), the transistor is switched on again and conducts electricity to create the magnetic field again. This process occurs rapidly enough to provide a somewhat constant power output. The frequency of voltage spikes generated by the Joule Thief circuit is over 5KHz . The video explains the working of the Joule Thief circuit very well.

### How can a Joule Thief circuit be made?

The components required are a NPN transistor, a 1kΩ resistor, a ferrite toroid (that can be salvaged from an old CFL bulb), wiring, a 3Volt LED and an AA battery which doesn't light up a LED by itself. Use the AA battery and connect two wires from the positive terminal and take one through the 1kΩ resistor and wrap the wires around the ferrite toroid. The wire that goes through the resistor, after it comes out from the ferrite toroid, is connected to the base terminal of the transistor. The other wire from the ferrite toroid goes to the collector terminal of the transistor which is also connected to the positive terminal of the LED. The negative terminal of the LED is connected to the emitter terminal of the transistor.

More information is available at an Instructables tutorial.

### How can the Joule Thief circuit be optimized or modified?

The **inductance** in the Joule Thief circuit is determined by the loops around the ferrite toroid. The more the number of loops, the greater the inductance. An increase in inductance generally decreases the current flowing through the circuit and an increase in the duty cycle(on time%). This suggests that an increase in inductance increases the efficiency of the circuit. However, too much inductance is not good either. Trial and error can help find the sweet spot of the circuit where the least current is drawn. Increasing or decreasing one loop at a time will help find the number of loops a certain circuit requires for maximum efficiency.

### What are the applications of this circuit?

Lighting an LED with a dead battery is not the only application of the Joule Thief circuit. The circuit helps utilize almost all the energy that is stored in a battery. For example, a battery that has come out from a toy can easily light up a torch that makes use of the Joule thief circuit for hours or even days. The circuit can also be used in battery chargers, wall clocks, solar cell chargers where a low voltage input has to be increased for its intended application.

The principle behind the joule thief can be used on any low voltage source, even if it's not a "dead" battery. For example, aluminium cans filled with water and wood ash, and simple electrodes can be used with a joule thief circuit for a battery charging application.

### What are the disadvantages of using a Joule Thief circuit?

A major disadvantage is a drop in output current because the current required to carry a given power decrease when voltage is increased because power is the product of current and voltage.

Another disadvantage is Without significant improvements to the circuit, it is hard for the circuit to power more than simple led because heavy demand is put on the transistor for high current at very low voltage

### What are some alternatives to the Joule Thief circuit?

    + **Supercharged Joule Thief circuit**: Has an efficiency of above 80% while conventional Joule Thief circuits have efficiencies between 40% and 60%
    + **Buck-boost converters**: Can be used for applications which require more power. The output voltage is always reversed in polarity with respect to the input
    + **Voltage multiplier**: Converts AC electrical power of a lower voltage to DC electrical power of a higher voltage
    + **Split-pi topology**: DC-DC converter that uses MOSFETs making it bidirectional and good for applications revolving around regenerative braking

### How does the supercharged Joule Thief circuit have such a high efficiency?

All that is required in addition to the components for a conventional Joule Thief circuit is a 680 pF capacitor. Apart from this, the feedback wire which is connected between ground and the wire from the ferris toroid which is also connected to a 1.5kΩ resistor and a diode, according to the circuit diagram which shows a circuit that can be used to switch between the conventional Joule Thief circuit and the Supercharged Joule Thief circuit.

## Milestones

Feb
1930

Harold C Weber files a patent for a mechanism to control electron streams in electronic devices.

Mar
1988

Honeywell Inc files a patent for a self energized circuit that steps up ultra low voltages (as low as 0.1 volts) to a stepped-up alternating current.

Nov
1999

An article titled *One-Volt L.E.D — A Bright Light* by Z. Kaparnik, and published in Everyday Practical Electronics, features a circuit that uses the same principles as the initial Joule Thief circuit in order to create high voltage pulses.

Aug
2002

The name Joule Thief is coined by Clive Mitchell and a tutorial is published on his website. Clive's variation includes a smaller resistance than the original circuit published in the EPE magazine.

Joule Thief Circuit


# 5G NR MAC

## Summary

The MAC sublayer of 5G NR protocol stack interfaces to RLC sublayer from above and PHY layer from below. It maps information between logical and transport channels. Logical channels are about the type of information carried whereas transport channels are about how such information is carried.

Data on a transport channel are packaged into **Transport Blocks (TBs)** whose configuration are determined by **Transport Formats (TFs)**. A transport block has a dynamic size. Except in some scenarios, only one transport block is transmitted by MAC in an interval called **Transmission Time Interval (TTI)**.

MAC is configured by RRC layer. In the simplest case, a UE has a single MAC entity. However, there are some scenarios when a UE has multiple MAC entities. MAC sublayer is specified from a UE perspective in TS 38.321.

## Discussion

### What logical/transport channels are relevant to 5G NR MAC?

Transport channels are the Service Access Points (SAPs) between MAC and PHY. Likewise, logical channels are the SAPs between RLC and MAC. In other words, MAC sublayer provides data transfer services on logical channels based on type of information carried, and uses transport channels to interface to PHY.

MAC handles the following channels, mentioned here as "transport channel (logical channels)":

    + **Uplink**: RACH (-), UL-SCH (CCCH, DCCH, DTCH)
    + **Downlink**: BCH (BCCH), PCH (PCCH), DL-SCH (BCCH, CCCH, DCCH, DTCH)
    + **Sidelink**: SL-BCH (SBCCH), SL-SCH (SCCH, STCH)RACH has no mapping to a logical channel since its information originates/terminates at MAC. In fact, RACH doesn't carry any transport blocks. BCCH is a logical channel that maps to either BCH or DL-SCH. Traffic channels include DTCH and STCH, all others being control channels.

### Which are the main functions of 5G NR MAC?

RLC need not worry about when and how data is transmitted on the air interface. Data transfer and radio resource allocation is the job of MAC.

In the transmit direction, MAC sublayer maps logical channels coming from RLC sublayer to transport channels going to PHY layer. Multiple logical channels can be multiplexed within a transport block. In the receive direction, MAC does demultiplexing.

Other than multiplexing/demultiplexing MAC SDUs, MAC also has in-band signalling. These are called MAC Control Elements (CEs). In fact, MAC CEs and MAC SDUs can be multiplexed in a single MAC PDU.

In the downlink, a UE is informed on PDCCH about an imminent scheduled transmission on DL-SCH. In the uplink, UE MAC must first make a Scheduling Request (SR) on PUCCH. UE typically gets a grant on PDCCH or Random Access Response. Semi-persistent grant via RRC signalling is also possible.

MAC also does scheduling information reporting, error correction through HARQ, logical channel prioritisation, dynamic scheduling across UEs (DL only), and priority handling between overlapping resources of one UE.

### Which are the main procedures of 5G NR MAC?

When a UE is making its initial access to the network, **Random Access** procedure is essential. This procedure is also invoked in many other scenarios. It's of type 4-step or 2-step. Each type can be Contention-Based Random Access (CBRA) or Contention-Free Random Access (CFRA).

Random access procedure establishes timing advance, so that uplink transmission is properly time-aligned when it reaches gNB. Maintenance of this **uplink time alignment** is another procedure.

**Data transfer** in downlink and uplink channels involve DL assignments, UL scheduling requests and UL grants. Data transfer involves multiplexing/demultiplexing of data, prioritization, and retransmission and error correction due to HARQ. Similar procedures exist for sidelink channels.

MAC CEs enable **signalling**. Examples include activation/deactivation of SCells, activation/deactivation of PDCP duplication, recommending bit rates to the UE, and many more.

UE also **reports** buffer status and power headroom to help gNB make resourcing decisions.

Discontinuous Reception (DRX), Bandwidth Part (BWP) switching, reset/reconfiguration, beam failure detection and recovery, handling measurement gaps, data inactivity monitoring are some more MAC procedures.

### When can a 5G UE have multiple MAC entities?

If a 5G UE is connected to the Master Cell Group (MCG) and the Secondary Cell Group (SCG), then there's a MAC entity for each cell group. Each entity operates independently, that is, their serving cell, timers, parameters, radio bearers, logical channels and HARQ entities are all independent.

Dual Connectivity (DC) is a feature in which a UE can be connected to both MCG and SCG. For example, MCG could be an 4G/LTE cell and SCG could be a 5G NR cell. Another scenario is Dual Active Protocol Stack (DAPS) handover in which there's one MAC entity for the source cell and one for the target cell.

A single MAC entity can support multiple numerologies, transmission timings and cells. However, RRC can restrict the mapping of a logical channel to a subset of cells, numerologies or other configurations. This is one way to reduce latency to serve URLLC services.

### What's the role of 5G NR MAC in Carrier Aggregation?

MAC plays an important role in Carrier Aggregation (CA). It distributes MAC PDUs and CEs across different Component Carriers (CCs) and generates one TB per TTI per CC. Moreover, each CC has its own HARQ entity within MAC.

Though CA involves multiple carriers or cells, they all come under the same cell group. A single MAC entity serves all cells involved in CA within the cell group. Aggregation happens over one Primary Cell (PCell) and one or more Secondary Cells (SCells). Within a single MAC entity there can be multiple instances of transport channels: one DL-SCH, UL-SCH and RACH for Special Cell (SpCell); one DL-SCH per SCell; and optionally one UL-SCH and RACH per SCell. SpCell is the PCell of either MCG or SCG. This implies that CA can be combined with DC.

### In 5G NR, what do you mean by High-MAC and Low-MAC?

In practical deployments, a 5G base station or gNB is not a monolith located at the cell site. It's often disaggregated into Radio Unit (RU), Distributed Unit (DU) and Centralized Unit (CU). 5G allows multiple ways in which gNB functions can be split across RU, DU and CU.

One way is to split MAC sublayer into two parts: High-MAC and Low-MAC. Such a split is called **Intra-MAC Split (Option 5)**. With this split, High-MAC, RLC, PDCP and RRC will be in the CU. Low-MAC will be in the DU. The DU-CU interface is commonly called **midhaul**.

Multiplexing/demultiplexing are done in High-MAC. High-level scheduling decisions are part of High-MAC. Inter-cell interference coordination as needed in CoMP can be done in High-MAC in a more centralized manner. Time-critical processing such as HARQ, random access control, scheduling-related information processing and reporting are in Low-MAC.

## Milestones

Apr
2017

3GPP publishes version 0.0.1 of MAC specification TS 38.321. By September, this evolves to version 1.0.0.

Dec
2017

3GPP publishes Release 15 "early drop". MAC specification TS 38.321 is upgraded to version 15.0.0.

Jun
2018

3GPP publishes Release 15 "main drop". MAC changes part of this release include beam failure recovery timer, prioritized random access, and PDCP duplication.

Dec
2018

In version 15.4.0 of MAC specification, **data inactivity timer** is introduced.

Jul
2020

3GPP publishes Release 16 specifications. Main MAC enhancements or additions in this release include Integrated Access and Backhaul (IAB), dormant BWP operation, eMIMO, 2-step RACH procedure, New Radio Unlicensed (NR-U), eURLLC, Industrial IoT, UE power saving, V2X with NR Sidelink, NR positioning, and a new MAC subheader for MAC CEs.

5G NR MAC


# Scratch (Language)

## Summary

Scratch is a visual block-based programming language designed for children aged 8-16. Children can create games, animations and stories in a fun-filled manner while also learning to reason and think creatively. Scratch is also an online community where creators can share their projects and be inspired by other projects.

Scratch is available in more than 150 countries in more than 60 languages. It's usage is license free. Scratch itself is open sourced. It's developed and overseen by the Scratch Foundation.

Scratch has been called "the YouTube of interactive media." Scratch has inspired other visual programming languages such as ScratchJr for ages 5-7, Snap!, mBlock, Stencyl and MIT App Inventor.

## Discussion

### Given dozens of programming languages, why do I need Scratch?

Many popular programming languages are text-based. Programmers have to type the program code. They need to learn and remember the language syntax. For beginners, making syntax errors is a common problem. A text-based interface is less accessible and less fun for children.

On the contrary, Scratch is designed for children aged 8-16. It can be taught at schools for students across all disciplines, including math, computer science, language arts, and social studies. This is important for today's economy where learning to code is part of computer literacy.

Because Scratch is visual, it's less intimating than text-based programming languages. Programs in Scratch are created by drag-and-drop actions on colourful blocks. The use of shapes and colours act as visual cues to help programmers create, edit or understand Scratch programs more easily. In fact, shapes fit together like a jigsaw puzzle. It's designers claim that,

> Scratch is more tinkerable, more meaningful, and more social than other programming environments.

Research has shown that Scratch enables students grasp computational thinking concepts such as parallelism, synchronization, flow control, user interactivity, data representation, abstraction and problem decomposition.

### What types of projects can I create with Scratch?

The main or popular project types in Scratch are:

    + **Games**: Games are the most common type, giving their creators large fan following. Even classics such as Pacman and Mario have been recreated in Scratch.
    + **Animations**: Using costume changes and movements, animations can be easily created.
    + **Music**: MIDI sound bank allows programmers to play up to 128 instruments. Volume and tempo can be adjusted. It's also possible to import a song and play it.
    + **Art**: Interactive art is one of the purposes for which Scratch was designed. More recently, non-interactive art is becoming common though less programming may be involved in creating them.
    + **Stories**: Not very common since many can be considered as animations. Stories could be adventures or feature many costumes and backdrops.
    + **Simulations**: Not that common but high quality projects involving physics, weather, gravity and 3D simulations have been created. Operating systems and engines are two common themes.Other types include tutorials, advertisements, sprite packs, slideshows, petitions, 100% pen, interviews, etc.

### What are some essential Scratch programming terms?

From a complete glossary for Scratch, we highlight a few essential terms:

    + **Stage**: Area where the project is displayed when active.
    + **Backdrop**: Background of the stage.
    + **Block**: Programming command that can be dragged and dropped into the code area.
    + **Code Area**: Area where scripts are edited.
    + **Script**: A stack of blocks make up a script. Determines how a sprite interacts with other sprites and the backdrop.
    + **Sprite**: Character or object on the stage that performs actions controlled by one or more scripts.
    + **Clone**: A copy of a sprite.
    + **Costume**: Appearance of a sprite. Often subtle variations of a sprite can be used to create animations.
    + **Bubble**: A speech bubble or a thought bubble signifies what a sprite is speaking or thinking.
    + **Scrolling**: Action of sliding a sprite across the stage.
    + **Broadcast**: A message sent through the Scratch program. Allows sprites to communicate with one another.
    + **Event**: Key press or mouse button clicks are example events. Can be used to trigger scripts.
    + **Studio**: A place to group and organize multiple projects.
    + **Pen**: Allows us to draw on the stage.

### What's the anatomy of the Scratch IDE?

The main areas of the IDE include the Stage, the Block Palette and the Code Area. The Stage is where the currently selected sprite appears. When the program runs, the results are displayed in the Stage. Stage can be maximized to full screen.

Just below the Stage is the list of sprites that one can choose. When a sprite is selected, the scripts associated with it appear in the Code Area. These scripts can be created or edited by drag-and-drop actions of blocks from the Block Palette.

The Costumes tab allow programmers to change the look of a sprite and thereby create visual effects and animations. The Sounds tab enables attaching sounds and music to the sprite.

### Which are the different shapes and categories of blocks in Scratch?

Blocks come in different shapes and categories. Each category comes in a different colour.

**Block shape** represents a certain context of usage. For example, Hat Blocks occur at the start of a script whereas Cap Blocks occur at the end of a script. Their unique shapes mean that they can't be mistakenly used elsewhere within the script. Stack Blocks perform the main actions. Boolean Blocks return true or false. Reporter Blocks report fixed numbers, strings or variables. C Blocks (also called Wrap Blocks) wrap other blocks and are of Control category. Some C Blocks are capped at the bottom.

**Block category** represents a certain type of functionality. Categories include Motion Blocks, Looks Blocks, Sound Blocks, Events Blocks, Control Blocks, Sensing Blocks, Operators Blocks, Variables Blocks, My Blocks, and Extensions.

With My Blocks, programmers can define a custom script and call it with inputs from other scripts. Use of My Blocks reduces overall project size.

### How do I get started with programming in Scratch?

Create an account on Scratch website and start creating projects online. Perhaps the easiest way to work with Scratch is to create and edit projects via a web browser. No installation is required.

Those who wish to work offline (without Internet connection), can download Scratch. Installations are available for Windows, macOS, ChromeOS and Android (tablets only). Scratch projects created from these installed apps can't be directly shared with the online community. You can however export a project, and then upload and share online.

Official documentation is on the Scratch Wiki.

You can explore Scratch projects shared by others. Participate in the Scratch discussion forum. The Scratch Ed channel on YouTube has many videos of tutorials, examples, events and workshops.

Two useful books for beginners are Scratch 3 Programming Playground by Al Sweigart and Scratch Programming in Easy Steps by Sean McManus. The latter book doesn't cover Scratch 3.0 but its examples are available online.

Advanced programmers who wish to contribute to Scratch's open source codebase can find the code on GitHub.

## Milestones

1970

The idea of using programming as a skill and a learning tool probably begins with Seymour Papert in the 1970s. He creates the **LOGO** programming language. Language commands control movements of a digital robot. This helps children grasp geometry. A pen is used to draw shapes on the screen. LOGO and its turtle graphics are subsequently widely adopted in UK schools.

2003

Mitch Resnick and John Maeda of MIT, and Yasmin Kafai of UCLA propose the idea of Scratch to National Science Foundation (NSF). Resnick is part of Lifelong Kindergarten Group of MIT Media Labs. With an NSF grant, they create **Scratch 0.1** by October. This version doesn't have a Block Palette and blocks are more square shaped.

Jan
2004

First **Scratch website** goes live. Elsewhere, it's reported that Scratch website was publicly launched in May 2007.

Jan
2007

**Scratch 1.0** is released. It's also the first version released to public. The Scratch layout in this version resembles closely with a version from January 2005. Thus, we may state that the application has reached good level of maturity.

Jul
2009

**Scratch 1.4** is released. Earlier versions include v1.1 (May 2007), v1.2 (Dec 2007), and v1.3 (Sep 2008).

May
2013

**Scratch 2.0** is released. It gets a new user interface. Among its new features are procedures, cloning, cloud data, vector graphics, show/hide lists, and sound editor. The offline editor of this version continues to be available even in April 2021.

2015

Since Scratch 2.0 was based on Flash, and Flash is no longer preferred on the web, Google proposes a partnership with MIT Media Lab to redesign Scratch. By 2015, many other visual programming languages (Code.org, App Inventor, MakeCode) are already using a JavaScript library called **Blockly**. Scratch adopts Blockly. The redesigned UI and blocks are released as part of Scratch 3.0 (Jan 2019).

Jan
2019

**Scratch 3.0** is released. Scratch 2.0 projects are compatible with the new version. Usability of the app is improved. Pen Blocks and Music Blocks are moved to Extensions. Stage is on the right, unlike Scratch 2.0 that had it on the left.

Apr
2021

Scratch community has close to 75 million shared projects, 70 million registered users, 448 million comments, and 29 million studios. In March, the site received 613 million pageviews and 29 million unique visitors. In TIOBE Index, Scratch ranks at position 22 in terms of popular programming languages. In April 2020, it was within the top 20.

Scratch (Language)


# Arduino

## Summary

Working with electronics and programming microcontroller-based systems is generally a difficult task that's reserved for engineers. What if we could simplify the process so that more people can participate, including designers and artists? This is why Arduino was invented back in 2005.

Arduino is an open source platform of both software and hardware. It uses a simplified programming syntax based on C/C++ languages. Arduino boards can be easily programmed by anyone with a computer and a USB connection.

Arduino boards come in many variants and developers get to choose what suits their application. The ecosystem around Arduino is rich with many third-party libraries, derivative boards, modules and shields. Indeed, Arduino has been a catalyst to the Maker Movement.

## Discussion

### What can I build with an Arduino?

Arduino is a simple low-cost platform to get started with DIY projects involving basic electronics and software. It's can be used to prototype applications involving sensors and motors for the Internet-of-Things (IoT). This means an Arduino can not only interface with real-world objects but also collect data and send them via the Internet to cloud applications.

You can use Arduino to automate routine tasks in your home such as watering plants or opening doors for your pets. You can build simple gadgets such as an LED matrix clock or a digital lock box. For learning purpose, you can build a traffic light system. More useful applications include fall detection for the elderly and ultrasonic glasses for the blind. You can also use Arduino as a substitute for more expensive engineering tools such as an audio meter or an oscilloscope. Among the simplest projects is to monitor and record temperature, humidity or pollution.

### What are the licensing terms for Arduino?

Arduino adopts the principles of open source for both hardware and software. For the hardware, the original design files (Eagle CAD) are available under CC BY-SA licensing. This allows for derivatives and commercial use but requires that attribution be given to Arduino. Derived works must also be released as CC BY-SA.

There's also an IDE to program the hardware, which uses GPL license. Code is in Java and available on GitHub. C/C++ microcontroller libraries use Lesser GPL license.

You can use the Arduino logo in tutorials, websites and community channels where the content is Arduino specific. However, if you release your own design, you should not use either the name or the logo. You can however say "XXX for Arduino" or "XXX (Arduino-Compatible)".

Despite these licensing terms, there are *counterfeits* that violate Arduino trademarks. There are also *clones* that are replicas of original designs but are branded differently. Finally, there are *derivatives* that innovate on the original design and therefore add value to the Arduino ecosystem. Examples include Teensy and Flora.

### How do I power up an Arduino?

There's more than one way to power an Arduino Uno: USB port, DC power jack, or header pins. Give power via VIN header pin for standalone battery-powered applications. To power peripherals, 5V and 3.3V outputs are available.

Most Arduino variants run on 5V. Due, M0 and Yun Mini are some boards that use 3.3V. Voltage input range is 9-12V for the Uno via the DC jack; 7-12V for Leonardo, Micro and Yun Mini; 5-12V for Mega 2560 R3; 5-7V for Mini. Note that smaller boards (Nano, Micro, Mini) lack DC power jack.

USB is also the interface to program the board. While Uno has USB A-B type, Nano has mini-B USB. Leonardo, Micro, and Due are some variants that use micro USB.

### Could you explain the different memories on an Arduino?

The following are the different Arduino memories:

    + **Flash**: Used to store the program. Arduino programs are called *sketches*. They're saved as `*.ino` files during development.
    + **SRAM**: Temporary storage for variables during program execution. Contents are lost when board is powered off.
    + **EEPROM**: Used for long-term storage. Most boards have EEPROM but Due and M0 don't have this.
    + **ROM**: Read-only memory for storing bootloader. Programmed in factory. Available only on some boards such as Due.

### What hardware interfaces are available on an Arduino?

Arduino has many interfaces, often shared on the same physical pin. A pin can be used for only one function at a time. Among the interfaces are:

    + **General Purpose Input Output (GPIO)**: For digital IO. Useful for interfacing to button switches, LEDs, etc.
    + **Pulse Width Modulation (PWM)**: Mimics analogue output using digital pulses. Useful for controlling motors, fading LEDs, etc.
    + **Analogue Input**: Uses a 10-bit *Analogue-to-Digital Converter (ADC)* to bring analogue signal values into the digital world. Used to read from an analogue sensor.
    + **USART**: For serial communications. Used to send/receive data to/from laptop, Bluetooth device, etc. LEDs Tx and Rx help us see serial activity.
    + **Digital Interfaces**: Other interfaces include *I2C* and *SPI*. Used for interfacing to digital sensors or other peripherals such a memory, display, etc.

### What are the different Arduino hardware variants?

The official Arduino product page lists all the variants. Arduino hardware comes in three main types:

    + **Boards**: These offer the main functionality. Among the entry level **boards** are Uno, Leonardo, Micro, and Nano. Boards with more features or better performance include Mega 2560, Zero, Due, and M0 Pro. For the Internet of Things (IoT), we need connectivity to the internet. Some boards that offer this include Yun, Ethernet and Industrial 101. For wearable applications, the Lilypad series offer small form factor boards that can sewn to cloth.
    + **Modules**: These are small form factor boards. Modules giving IoT connectivity include Fox 1200, WAN 1300, GSM 1400, NB 1500, and WiFi 1010.
    + **Shields**: These extend the hardware features provided by boards. They can't be used on their own. They connect to boards via well-defined interfaces. There are shields for interfacing via USB, CAN, RS485, and more.

### How do I select an Arduino hardware for my application?

You should start by listing application requirements in terms of performance, memory, connectivity, and interfaces.

The most popular one is the Arduino Uno. Many shields are made compatible to the pin header of the Uno. To fit into tight spaces, Arduino Nano or Mini are cheaper and better options; but shields can't be used with them.

For USB apps, prefer Leonardo or Micro that are based on ATmega32U4.

Arduino Due has an ARM Cortex-M3 running at faster clock, with more memory and interfaces than the Uno. It runs on 3.3V, not 5V like the Uno. Standard Arduino shields can be used. Arduino Mega has similar form factor to the Due but uses ATmega2560, not an ARM processor. It runs on 5V and therefore friendly to hobbyists.

Any project needing lots of GPIO can use Due or Mega. If fast processing is needed, Due is a better choice. Due is also good for analogue, with 12 inputs and 2 outputs.

More information is available at Core Electronics and Hackster.

## Milestones

2001

At the MIT Media Lab, Casey Reas and Ben Fry invent **Processing** based on Java. Processing is a graphical library and an IDE. It simplifies programming syntax. Later, Processing goes on to inspire other projects such as Wiring, Arduino and Fritzing.

2003

Hernando Barragán creates **Wiring** platform so that designers and artists can approach electronics and programming more easily. Wiring itself is based on Processing. It's part of Barragán's Master's thesis project at the Interaction Design Institute Ivrea (IDII) in Italy. In March 2004, 25 Wiring PCBs are manufactured and hand-soldered.

2005

While the Wiring boards used ATmega128, researchers at IDII fork Wiring code and add support for the cheaper ATmega8. They name the fork **Arduino**. The first batch of Arduino boards are also made the same year.

Sep
2010

**Arduino Uno** is released based on ATmega328P. Clock is at 16MHz. Memory includes 32KB Flash, 2KB RAM and 1KB EEPROM. Unlike the first Arduino of 2005 that used RS232, Uno uses USB host interface and FTDI chip.

2012

**Arduino Due** is released. This is the first 32-bit Arduino.

Mar
2014

The first **Arduino Day** happens on 29th March. This day now marks an annual event worldwide for people to share knowledge and exhibit their Arduino projects.

May
2015

At the Maker Faire Bay Area, Massimo Banzi announces the creation of **Genuino**. Due to trademark issues and legal disputes, Arduino trademark will be limited to the U.S., for which Adafruit will be the manufacturer. In other markets, Genuino trademark will be used with manufacturing outside the U.S. In October 2016, differences are put aside and the companies behind Arduino and Genuino announce a merger.

2016

To enable code editing and programming boards from a web browser, **Arduino Create** is released.

2017

In January, to take care of manufacturing, **Arduino AG** is incorporated. Later the same year, there's talk of forming **Arduino Foundation** for maintaining the IDE and other code infrastructure in an open source community-driven manner. Subsequently, not much information has been released about the Foundation.

Apr
2018

To cater for IoT applications, Arduino announces the **MKR Family** of boards. Most of these boards come with built-in wireless connectivity; or they're of smaller form factor. Connectivity could be GSM, SigFox, or Wi-Fi. In May, new boards supporting NB-IoT and Bluetooth are introduced.

Arduino


# CSS Flexbox

## Summary

In web design, the traditional way to layout elements on a page was to use CSS properties such as `float`, `clear`, and `display`. In particular, `display` property has values `inline`, `block`, `inline-block`, and `table` that help with layout. CSS Flexbox is an easier method to do layouts.

With flexbox, items are laid out within a parent container. Flexbox makes it easy to distribute or fill extra spaces within the container. It's also possible to shrink the items dynamically to prevent overflows. We can also wrap items within the container to obtain a multiline flow. This "flexing" of items is an important aspect of flexbox that makes the design responsive to device dimensions.

Site header, side bar, centered prompt, tabs, and form footer are some examples where flexbox can be used.

## Discussion

### What are the essential terms pertaining to CSS Flexbox?

Flexbox layout involves a parent element that defines how to lay out its child elements. The parent is called **flex container** and its children are called **flex items**. CSS declarations `display: flex` or `display: inline-flex` specify the use of flexbox.

**Flex direction** determines if flex items are laid out horizontally or vertically. In either case, the direction in which items are laid out is called **main axis**. Its perpendicular axis is called **cross axis**.

The container's boundaries are called **main-start**, **main-end**, **cross-start** and **cross-end**. Items are placed from start to end boundaries. Normally, this is left-to-right and top-to-bottom order. However, the order gets reversed if `flex-direction` has values `row-reverse` or `column-reverse`. Also, main axis and cross axis can get swapped depending on the current writing mode.

Container dimensions are called **main size** and **cross size**. CSS properties `width`, `height` and their min/max equivalents are applicable to the container.

### Which are the main CSS properties to control layout in CSS Flexbox?

On the flex container, main CSS properties are:

    + display: flex | flex-inline
    + flex-direction: row | row-reverse | column | column-reverse
    + flex-wrap: nowrap | wrap | wrap-reverse
    + flex-flow: a shorthand for 'flex-direction || flex-wrap'
    + justify-content: flex-start | flex-end | center | space-between | space-around
    + align-items: flex-start | flex-end | center | baseline | stretch
    + align-content: flex-start | flex-end | center | space-between | space-around | stretchOn flex items, main CSS properties are:

    + order: integer
    + flex-grow: number
    + flex-shrink: number
    + flex-basis: content | 'width'
    + flex: a shorthand for 'none | [ flex-grow flex-shrink? || flex-basis ]'
    + align-self: auto | flex-start | flex-end | center | baseline | stretchAn article on CSS-Tricks visually depicts the use of these properties. It's recommended to use shorthand forms `flex-flow` and `flex`.

### What does it mean to grow or shrink flex items?

Along the main axis, flex items might not fill the container's main size. We could use `justify-content` to distribute the leftover space; or increase the size of items to fill the extra space. On the other hand, if items are too big for the container we could shrink the items to avoid overflow.

In (b), all items are grown in equal proportion to fill up the extra space. In (c), green items retain their original size while orange items grow. In (d), green items grow relative to orange items in the ratio 3:1. Interestingly, this results in orange items shrinking below their original size.

In (e) and (f), items are wider and would overflow the container; but they don't overflow since wrapping is enabled. Since container is 500px wide and each item is 120px wide, we would expect the first row to contain four items. However, the effect of padding (8 x 5px) results in only three items in the first row.

Item width (or height for column-wise layout) and its min/max values also affect how items can be resized.

### How can I align and justify items in a CSS flexbox container?

Along the main axis, `justify-content` controls how items are laid out and extra spaces are distributed. For example, `justify-content:flex-start` aligns content to main-start. Excess space, if any, is towards main-end; `justify-content:center` puts leftover space towards both main-start and main-end; `justify-content:space-between`, `justify-content:space-around`, and `justify-content:space-evenly` distribute leftover space between or around all items.

Along the cross axis, `align-items` controls the alignment. For example, `align-items:center` centers items; `align-items:stretch` stretches items to fill the container's cross size (if set) or match the longest item.

When wrapping is enabled, `align-content` becomes relevant. Like `align-items`, it affects alignment and spacing along the cross axis.

Container property `align-items` can be overridden on individual items with `align-self`. For example, we can align all items to cross-start (`align-items:flex-start`) and one particular item to cross-end (`align-self:flex-end`).

### How do I use the 'order' property of CSS Flexbox?

Flex items can be assigned to ordinal groups, with each group given an integer. Within the container, items are rendered based on this order, from lower to higher numbers. This is useful because the ordering of items need not be changed in the source document. Ordering can be controlled from CSS.

An example use case is a tabbed interface. If we wish the active tab to be the leftmost tab (main-start), then we can set `.tabs > .current { order: -1; }`. All other tabs have the default value of zero.

However, ordering items this way is only *visual ordering*. The logical ordering is still specified by the order in the source document. *Logical ordering* is what matters for accessibility, such as, navigating the items via tab keys.

### What are some tips and tricks for using CSS Flexbox?

CSS Flexbox is primarily for 1-D layouts. Via nesting or wrapping, it's possible to achieve 2-D flexbox layouts. An excellent approach is to combine both CSS Grid and CSS Flexbox, thus leveraging the best of both worlds. Properties `justify-content` and `align-items` are useful in such an approach.

Flex containers are not block elements. CSS properties `float`, `clear` and `vertical-align` have no effect on flex items. Property `overflow` is applicable to the container.

It's possible to overlap flex items using negative margins or absolute positioning.

Beau Carnes has published informative visualizations of how CSS Flexbox works. Flexbox Patterns is a useful resource where beginners can study many flexbox examples.

## Milestones

Jul
2009

W3C publishes a working draft of **CSS Flexible Box Layout** or Flexbox. After multiple revisions, first Candidate Recommendation of Flexbox is released in September 2012.

Sep
2012

W3C publishes the first Candidate Recommendation of **CSS Flexible Box Layout Module**. A flexbox layout can be specified with `display: flex`. However, the initial draft of July 2009 used `display: box` and a later draft of March 2012 used `display: flexbox`. Older browsers might not support the currently standardized CSS rule and fallback techniques may be required.

Mar
2016

W3C publishes as a Candidate Recommendation **CSS Flexible Box Layout Module Level 1**. As on December 2020, the latest version of this document is from November 2018.

May
2017

W3C publishes **CSS Grid Layout Module Level 1** as a Candidate Recommendation. Whereas CSS Flexbox is limited to one-dimensional layouts, CSS Grid can handle two-dimensional layouts. Designers view CSS Grid as not replacing but complementing CSS Flexbox. For example, we can do a flexbox layout within a grid layout or vice versa.

CSS Flexbox


# Go (Language)

## Summary

Go is an open source, compiled, concurrent, garbage-collected, statically typed language developed at Google. Google created it to solve its own problems in relation to multicore processors, networked systems and large codebases.

While languages such as C and C++ are fast, they are hard to master and development takes longer. While Python is easier to work with, it compromises on runtime efficiency. Go was designed to be efficient, scalable and productive.

## Discussion

### Why was Go invented when there were already many popular languages?

Go was invented (in 2007) at a time when multicore CPU architectures were common, largely because Moore's Law was failing to deliver. With multiple cores, code can run in parallel on all available cores; but there was no programming language that simplified the development of multithreaded applications. The responsibility of managing different threads safely and efficiently fell on developers. Thread-locking, race conditions and deadlocks are usual challenges. Go was designed with concurrent programming in mind.

Golang FAQ states that in the days before Go,

> One had to choose either efficient compilation, efficient execution, or ease of programming; all three were not available in the same mainstream language.

Go balances these different expectations of a programming language. Go was really designed to solve problems Google faced with large software systems. Hence, it was less about research into language design and more about software engineering to solve problems at hand. Some of the problems Google faced were slow builds, uncontrolled dependencies, poor code readability, challenges with automation tools and cross-language builds.

### What are the main features of Go language?

Go's main features include concurrency, scalability, error checking, fast performance, garbage collection and cross platform. It's fast because it's a compiled language and not an interpreted one like JavaScript. Garbage collection pauses have been reduced to about 100 microseconds. Go's built-in error type enables developers to spot errors early. It's designed for the cloud and can run on various platforms.

Because Go has types and methods, we can call it an object-oriented language. But there's no type inheritance. Instead, Go uses **interfaces**.

Go achieves concurrency using **goroutines**, which are then mapped to operating system threads. If a thread is blocked, goroutines can be moved to another thread. Goroutines are therefore cheap, requiring nothing more than stack memory. Built-in **channels** are used to communicate between goroutines.

Go is opinionated and focused on its design goals. Therefore, what it leaves out of the language are just as important. Go doesn't have generic types, exceptions and assertions.

Other features mentioned elsewhere are `godoc` for documentation; `gofmt` for code formatting; `golint` for linting; built-in testing and profiling framework; race condition detection.

### What sort of applications can benefit from using Go language?

If you're into systems programming with support for multithreading, Go is a suitable language. Go can be used for building any server-side app containing thousands of communicating threads.

Due to its roots in Google, Go is suited for large distributed projects involving multiple teams. It's taken off in the cloud infrastructure space, platform automation, tooling apps and DevOps pipelines. Go's static binaries without external dependencies are easier to manage. Cross-platform CLI tools can be created. Because Go compiles to native code, performance-critical apps can benefit from it. As Go's Garbage Collection (GC) matures, it can suit embedded systems.

While Go was originally for systems programming, it's been adopted for microservices. This is due to its "elegant concurrency model, simple binary deployments, low number of external dependencies, fast performance, and runtime efficiency." There's also an expectation (back in 2015) that Go will grow into a general-purpose programming language and be adopted for GUI apps, IoT apps, web/mobile apps, robotics frameworks, and more. It's suited for data science work.

Alternatively, it's been said that Go is not suitable for GUI apps, kernels, drivers or embedded systems.

### How's the adoption of Go language?

Back in 2014, it was reported that many open source projects have adopted Go: Docker, Kubernetes, Weave, Shipyard, etcd, Fleet, Deis, Flynn. Since Go is suited to distributed server apps, all of these apps are cloud related. Lime is an example of a desktop app for text editing. InfluxDB, a distributed time series database, is written in Go. Gogs (Go Git Service) helps you to run a Git service, like GitHub. Go is a suitable fit for Docker due to static compilation and ability to build for different architecture with less effort.

Among the big names using Go are Uber, BBC, Novartis, Soundcloud and Basecamp. Uber has reported better throughput, latency and uptime. BBC uses it for backend, including crawlers and web scrapers. Soundcloud's build and deployment system is in Go.

In 2016, Go was rated the most popular language in the TIOBE index. Multiple developer surveys in 2018 show interest and adoption of Go. In StackOverflow's survey, Go is in the top five most loved/wanted languages. Community contribution to Go is growing steadily.

There's also a curated list of companies worldwide using Go.

### What are some criticisms of Go language?

Go doesn't support generics, which means that code written for one data type can't be reused for another. Interfaces are implicit. Sometimes it's hard to tell if a struct type implements an interface. Library support for Go from vendors is lacking. While Go is open source, key maintainers are seen to be non-receptive to suggestions from the community. Packages are not versioned in a centralized place.

We can't define immutable data types. Complex data types are not available out of the box: hash, tree, set. Go doesn't support overloading. Compile-time checks are lacking. It's slow compared to C, C++ and Rust. It's been said that Rust succeeds in areas where Go fails. There's also a curated list of Go criticisms.

From Go's perspective, some missing features are actually by design, reflecting Go's simplicity. Generics come with a higher complexity. Without exceptions, developers are forced to do explicit error checking. Overloading makes problems harder to debug. Being minimal and opinionated means that everyone writes code the same way: there are no debates about using tabs versus spaces or for versus while.

### For a beginner, what online resources can you recommend to learn Go programming?

Golang's official FAQ page is a good place to start. Other resources include the Go official documentation and the Go Wiki pages.

A curated list of Go frameworks and software is available on GitHub. There's also a curated list of 18 Go experts to follow. For latest news, the official Go blog and Golang Weekly newsletter are useful. For conferences, there's GopherCon and the European dotGo.

## Milestones

Sep
2007

Three engineers at Google (Robert Griesemer, Rob Pike and Ken Thompson) invent **Go**. Their aim is to design a better language for Google's workflows. In particular, they notice lack of concurrency support in current languages and long build times. They want a small, compiled language with modern features.

Nov
2009

Go is made open source.

Mar
2012

A major release named **Go 1** is released. It's stated that language will be stable and compatible with future updates of Go 1.

2014

World's first Go conference, **GopherCon**, is organized in Denver, USA. About 700 gophers (Go developers) attend the event.

Jan
2017

As a dependency management tool for Go packages, **dep** is released. Developers adopt this in preference to many community-led tools. Dep uses semantic versioning plus a configuration file that specifies dependencies. Version 0.1.0 of dep becomes available on GitHub in May 2017.

Jul
2017

At GopherCon 2017, Russ Cox presents initial ideas for **Go 2** with a goal "to fix the most significant ways Go fails to scale." He outlines the community process through which Go 2 will be defined. Early draft designs of Go 2 consider error values, error handling, and generic programming. As of November 2018, Go 2 is still under development.

Aug
2018

Version Go 1.11 is released. Two major versions are supported at any point in time. This means that versions 1.11 and 1.10 will be supported until version 1.12 appears.

Go (Language)


# Low-Power Wide-Area Network

## Summary

Low-Power Wide-Area Network (LPWAN) is a type of wireless telecommunication wide area network designed to allow long range communications at a low bit rate among connected objects, such as sensors operated on battery.

On one end of the spectrum are cellular standards (3G, 4G) that offer good range and good data rates but also consume lot of power. On the other end are short-range technologies such as Wi-Fi and Bluetooth that consume less power but are limited by range. LPWAN fills the gap in between where a longer range is desired at lower power for sending small amounts of data.

## Discussion

### What's the big picture of LPWAN in IoT?

LPWAN is a key driver for Internet-of-Things (IoT) innovation. LPWAN helps connect far-flung sensors and other devices. LPWAN technologies optimize battery life by decreasing power consumption, while networking standards ensure reliable connections at low speeds to support low levels of data use. Thus, LPWAN enables energy-efficient transmission of small data volumes over long distances.

### Where does LPWAN fit?

The low power, low bit rate and intended use distinguish LPWAN from a wide area network that is designed to carry more data using more power. LPWAN data rate ranges from 0.3 kbit/s to 50 kbit/s per channel. Cellular networks based on GPRS used to offer this connectivity for carrying M2M data back since the late 1990s. However, operators across the world are shutting down their legacy 2G networks. For example, AT&T announced that its 2G network will be completely shutdown by end of 2016. It's in this context that new LPWAN technologies were invented and sometimes standardized.

LPWAN is focused on IoT connectivity for devices (endpoints) that consume low power, send/receive short messages at low speeds, and have low duty cycles. Two categories of LPWANs are:

    + Cellular (e.g. NB-IoT and LTE Category M1) WANs using licensed spectrum.
    + Wireless WANs operating in unlicensed frequency bands(e.g. LoRaWan and Sigfox).

### What are the different types of LPWAN and how do they compare?

The comparison between any of the LPWANs available is based primarily on coverage, bandwidth, data rate, and number of messages per day. Selecting the right LPWAN is application dependant but typically the goal is to obtain decent rates at minimum power consumption.

Let's compare LoRa, Sigfox and NB-IoT in greater detail:

    + **LoRa**: Unlicensed sub-GHz spectrum. Chirp Spread Spectrum (CSS) modulation. Can define packet size. Transceiver chip is available only from Semtech Corporation. LoRaWAN the Medium Access Control (MAC) layer for managing communication between LPWAN devices and gateways.
    + **Sigfox**: Proprietary. Unlicensed 868 MHz or 902 MHz bands with only a single operator per country. Coverage of 30-50 km (rural), 3-10 km (urban) and up to 1,000 km (line-of-site). Uplink 150 messages of 12 bytes per day. Downlink 4 messages of 8 bytes per day.
    + **NB-IoT**: Licensed bands including unused 200 kHz bands previously used for GSM or CDMA. Based on LTE resource blocks.

### What are the application areas or use cases of LPWAN?

LPWANs are deployed globally in many countries, some even nationwide, to enable services in the Smart metering, Facilities and Logistics management, Wearables, Smart dustbins, Smart street lighting, Industries with connected appliances, Environmental monitoring, and more.

Deployments noted above include both kind of licensed LPWANs (NB-IoT) and unlicensed LPWANs (LoRaWAN, Sigfox). The choice is based on requirement and availability.

### How do I select a suitable LPWAN for my application?

Selecting a LPWAN involves understanding and refining the key characteristics and requirements of the IoT application and a particular use case. The following parameters can be considered: Battery life, Capacity, Range, Cost, Licensed/Unlicensed spectrum, Quality of Service, Reliability, and Security.

### What LPWAN products are available for prototyping or deployment?

Microchip Technology Inc. have developed its RN2483 LoRa® module, the world’s first to pass the LoRa Alliance’s LoRaWAN™ Certification Program. Some of the operations performed on RN2483 transceiver include:

    + **radio get bt** - reads back current data
    + **radio get mod** - current mode of modulation - LoRa, FSK
    + **radio get fr** - reads back current frequency of transceiver
    + **radio get power** - reads back the current transmit output powerFor Cellular LPWANs there are various chip-set vendors available such as Alter Semiconductors, Huawei, Mediatek, etc.

### What's that status of LPWAN standardization and interoperability?

LoRaWAN is maintained by LoRa Alliance. Version 1.0 of the LoRaWAN specification was released in January 2015, followed by V1.1 in 2017.

3GPP governs and provides specification for the cellular IoT alternates as NB-IoT and LTE-M.

IEEE 802.15.4 is a technical standard which defines the operation of low-power wireless personal area networks (LP-WPANs). It specifies the physical layer and media access control for LP-WPANs, and is maintained by the IEEE 802.15 working group, which defined the standard in 2003. It is the basis for the ZigBee, ISA100.11a, WirelessHART, MiWi, SNAP, and Thread specifications, each of which further extends the standard by developing the upper layers that are not defined in IEEE 802.15.4. Alternatively, it can be used with 6LoWPAN, the technology used to deliver the IPv6 version of the Internet Protocol (IP) over WPANs, to define the upper layers.

Most LPWAN technologies use sub-GHz spectrum. However, those that come from cellular (NB-IoT, LTE-M, EC-GSM) have spectrum above 1 GHz as well.

### Is there anything in place to test and certify LPWAN products?

Testing and certification for LPWAN products in both licensed and unlicensed spectra are available worldwide with a group of accredited test houses.

Testing services for all major LPWAN variants are provided by 7layers GmbH, Allion Labs, Bureau Veritas Consumer Products Services, DEKRA Testing and Certification, IMST, and TÜV Rheinland.

## Milestones

1985

AlarmNet from the 1980s is among the early systems to wirelessly send machine data to a central server. It's built by ADEMCO to monitor alarm panels. It uses 900MHz band at low data rates. Such proprietary networks are used until the arrival of 2G networks in the late 1990s.

2013

The origins of **LoRa** is in two patents, US7791415 (2008) and EP2763321 (2013). It uses spread-spectrum radio modulation developed by Cycleo (Grenoble, France). It's acquired by Semtech in 2012.

2014

A new study item titled "Cellular System Support for Ultra Low Complexity and Low Throughput Internet of Things" is proposed in 3GPP. This kick-starts the work on **NB-IoT**. The document states that 3GPP M2M devices use legacy GPRS but there are competing technologies that provide better coverage and efficiency at lower cost. Document proposes looking into evolving GERAN plus designing a new access system.

Jan
2015

LoRa Alliance releases **LoRaWAN 1.0** specification. In Oct 2017, LoRa Alliance releases **LoRaWAN 1.1** specification.

Dec
2015

Microchip releases RN2483, a LoRa Alliance-certified wireless module for IoT applications.

Jun
2016

3GPP completes the standardization of NB-IoT, which is part of Release 13 (LTE-Advanced Pro). Further changes to the specifications can be done only in a backward-compatible manner.

Oct
2016

World's first commercial NB-IoT network goes live in Deutsche Telekom's German and Dutch networks. The first application is a smart parking system. Deutsche Telekom had launched a pre-standard NB-IoT on a commercial network back in October 2015.

Low-Power Wide-Area Network


# Text Normalization

## Summary

Text normalization reduces variations in word forms to a common form when the variations mean the same thing. For example, US and U.S.A become USA; Product, product and products become product; naïve becomes naive; $400 becomes 400 dollars; +7 (800) 123 1231 becomes 0078001231231; 25 June 2015 and 25/6/15 become 2015-06-25; and so on.

Before text data is used in training NLP models, it's pre-processed to a suitable form. Text normalization is often an essential step in text pre-processing. Text normalization simplifies the modelling process and can improve the model's performance.

There's no fixed set of tasks that are part of text normalization. Tasks depend on application requirements. Text normalization started with text-to-speech systems and later became important for processing social media text.

## Discussion

### What are the typical tasks within text normalization?

We can identify the following tasks for normalizing text:

    + **Tokenization**: Text is normally broken up into tokens. A token is usually a single word but there are exceptions, such as New York.
    + **Lemmatization**: Reduce surface forms to their root form. For example, sang, sung and sings have a common root 'sing'.
    + **Stemming**: Strip suffixes. For example, trouble, troubled and troubles are stemmed to 'troubl'. This is a simpler and faster alternative to lemmatization.
    + **Sentence Segmentation**: Break up text into sentences using characters `.`, `!`, or `?`.
    + **Phonetic Normalization**: Words spelled differently could sound the same. Likewise, variations in pronunciation would need to be normalized to the same token.
    + **Spelling Correction**: In some applications such as IR, it's useful to correct spelling errors. For example, 'infromation' is normalized to 'information'.
    + **Non-Standard Words**: This includes phone numbers, dates, currencies, addresses, acronyms, etc.
    + **Others**: Normalization may involve accents (naïve, naive), UK/US spelling (catalogue, catalog), and capital letters (Product, product).

### What are some NLP applications that benefit from text normalization?

**Information Retrieval (IR)** is a typical example. If the search query is 'U.S.A.', we may want to return results for 'U.S.A.' and 'USA'. One way to do this is via *query expansion* in which both forms are searched. A more efficient approach is to normalize to 'USA', store all documents with this normalized form and search only for 'USA'. Wrong normalization can produce irrelevant results, such 'C.A.T.' normalized to 'cat'.

**Conversational AI** involves both Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) synthesis. For example, when a user says "five p m", ASR should interpret this as "5:00PM". This is called *inverse text normalization*. On the reverse, a text input "6:30PM" should be spoken as "six thirty p m". This is text normalization. Another example is 'Dr.', which could be interpreted as 'Drive' or 'Doctor'. Hence, context of usage is important to determine the correct normal form.

Machine translation, opinion mining, spell checking, sentiment analysis, dependency parsing, and named entity recognition are further examples of NLP tasks or applications that can benefit from text normalization.

### What are some general approaches to text normalization?

Text normalization has a few different approaches:

    + **Substitution Lists**: Also called wordlist mapping, lookups or memorization. Uses a precompiled list. Doesn't generalize to variants not in the list.
    + **Rule-based Methods**: Manually crafted rules encode regularities in variants.
    + **Distance-based Methods**: Edit distance measures such as Levenshtein distance are used to determine if two word forms are similar.
    + **Spelling Correction**: Hidden Markov Models analyse word morphology and determine the correct spelling. However, corrections are done word by word without any context.
    + **Automatic Speech Recognition (ASR)**: Based on the insight that microtext (social media text, SMS messages) is closer to sound forms rather than proper spelling. Decodes word sequences within a weighted phonetic framework.
    + **Machine Translation (MT)**: Microtext is treated as a foreign language that needs to be translated. This approach captures context. Character-level Statistical Machine Translation (CSMT) maps character sequences rather than words. It's an example of the noisy channel model: a translation model followed by a language model.
    + **Neural Models**: Use of neural networks such as encoder-decoder model with LSTM.

### What are some approaches to text tokenization?

In English, whitespace is used to separate words. Hence, whitespace is often used to identify tokens. Some punctuation characters could also indicate word boundaries. In social media text, `:)` and `#nlproc` would be considered as tokens.

Contractions are often normalized to expanded forms. Examples, what're → what are, I'm → I am, isn't → is not. This sort of normalization results in two tokens from a single word. On the contrary, New York is an example of two words considered as a single token.

Tokenization of some words is far from unambiguous. Hyphens present a challenge. Should state-of-the-art become 'state of the art'? Should Hewlett-Packard become 'Hewlett Packard'? Should lower-case become 'lowercase' or 'lower case'? Some acronyms are also challenging. How should we tokenize m.p.h. and PhD?

In Japanese and Chinese, there are no spaces to separate words. A greedy algorithm that attempts to find the longest dictionary word is often used. In French, should L'ensemble be tokenized as L, L' or Le? In German, noun compounds are not segmented and their processing is deferred to the application.

Among the well-known tokenization approaches are Byte-Pair Encoding (BPE), WordPiece and SentencePiece.

### What are non-standard words that need to be normalized?

Non-Standard Words (NSWs) include numbers, abbreviations, dates, currency amounts and acronyms. Mixed-case words (WinNT, SunOS), Roman numerals, URLs, and email addresses are more categories of NSWs.

NSWs often occur in text apart from ordinary words and names. The challenge with NSWs is that they're not dictionary words and their interpretation tends to be ambiguous. Therefore, we need to normalize them. This basically means replacing them with ordinary words.

Take for example 'Pvt', which is interpreted as 'Private'. An ambiguous example is 'IV'. It could be read as four, fourth or intravenous, depending on the context. The number 1750 could refer to a year, a building number or a cardinal number. These differences are important for a TTS system that needs to determine the correct pronunciation. Should Amazon Alexa read '2/3' as 'two thirds' or 'February Third'?

Rather than employ ad hoc techniques to handle NSWs, formal modelling has been shown to give better results. Techniques could include n-gram models, decision trees, and weighted finite-state transducers.

### What are the challenges in normalizing social media text?

Social media text often don't conform to rules of spelling, grammar or punctuation. Among its challenges are:

    + **Abbreviations**: nite (night), gr8 (great), sayin (saying), lol (laugh out loud), iirc (if I remember correctly), hard2tell (hard to tell)
    + **Misspelling**: wouls (would), rediculous (ridiculous)
    + **Omitted Punctuation**: im (I'm), dont (don't)
    + **Slang**: that was well mint (that was well good)
    + **Wordplay**: that was soooooo great (that was so great)
    + **Disguised Vulgarities**: sh1t, f\*\*k
    + **Emoticons**: :) for smiling face, <3 for heart
    + **Informal Transliteration**: This concerns only multilingual text. Variations in transliteration occur due to long vowels, borrowed words, accents/dialects, double consonants, etc.Experiments have shown that normalizing these gives better performance in machine translation and spell checking. However, challenges remain. Emoticons :P and ;D are treated as spelling errors. Abbreviations 'b' for 'be' and 'c' for 'see' are not caught by spell checkers and later affect machine translation. When "I'm" is written as "im", it's misinterpreted as an abbreviation for instant messaging.

### What does it mean to normalize Unicode strings?

Consider the angstrom symbol Å that may require normalization. Its Unicode codepoint is 212B. It can be decomposed into A followed by a small top circle.

Unicode characters can contain diacritical marks, ligatures, or half-width katakana characters. Unicode has defined four normalization forms:

    + **Normalization Form D (NFD)**: Canonical Decomposition
    + **Normalization Form C (NFC)**: Canonical Decomposition, followed by Canonical Composition
    + **Normalization Form KD (NFKD)**: Compatibility Decomposition
    + **Normalization Form KC (NFKC)**: Compatibility Decomposition, followed by Canonical Composition*Canonical equivalence* means that equivalent characters or sequences of characters represent the same abstract character. They display and behave the same way.

*Compatibility equivalence* is a weaker type of equivalence. In this case, the visual appearance and behaviour may differ though they represent the same abstract character. For example, character ℌ becomes H and ¼ becomes 1/4. This difference may be acceptable in some applications. In some cases, applications may account for these differences with additional styling.

Consider 'schön'. Its normal forms are 'scho\u0308n' (NFD & NFKD) and 'schön' (NFC & NFKC). Moreover, NFC and NFKC differ only in the decomposition phase.

### What are some neural network approaches to text normalization?

Since 2016, **Recurrent Neural Networks (RNNs)** have been used for text normalization. In particular, a few layers of BiLSTM have been used to map character sequences to word tokens.

For a long time CSMT was the state of the art in text normalization. Neural models generally need much larger training datasets. To overcome this limitation, Lusetti et al. (2018) trained a character-level **encoder-decoder model** plus a word-level language model. Beam search is used during decoding.

Zhang et al. (2019) used **transformers** with good results but it's prone to unrecoverable errors. They got better results by modifying encoder-decoder model to capture context more effectively. Their multi-task architecture jointly trains the tagger and the normalizer.

**Memory augmented network** has been applied. A hybrid word-character attention-based encoder-decoder model has been used, with character-based component trained on adversarial examples. **Pointer-generator network** with transformer encoder and auto-regressive decoder has been used, with the pointer module replacing OOV output tokens.

For many NLP tasks in Chinese, word tokenization is not required. However, **Convolutional Neural Networks (CNNs)** have been used.

### Could you mention some useful developer tools for text normalization?

In Python, many NLP software libraries support text normalization, particularly tokenization, stemming and lemmatization. Some of these include NLTK, Hunspell, Gensim, SpaCy, TextBlob and Pattern. More tools are listed in an online spreadsheet.

Penn Treebank tokenization standard is applied to treebanks released by the Linguistic Data Consortium (LDC). This standard keeps hyphenated words together, breaks up contractions (doesn't → does and n't), and separates out all punctuation ($10 → $ and 10).

For Unicode normalization, the International Components for Unicode page links to many useful resources including open source software. There's also an online demo at Unicode.org and a Unicode normalization FAQ.

In R, `utf8_normalize` from `utf8` package does Unicode normalization. For other text analysis, R packages `tidytext`, `tm`, `SnowballC` and `topicmodels` are useful.

Wolfram supports many levels of text normalization: character-level, word-level, sentence-level, morphological and linguistic.

## Milestones

1987

An early example of text normalization in the context of Text-to-Speech (TTS) is in a system named *MITalk*. Normalization is achieved using **hard-coded rules** in either Fortran or C.

1996

In the Bell Labs multilingual TTS system, **Weighted Finite State Transducer (WFST)** is used for text normalization. Instead of doing this as a pre-processing step, normalization is done along with other linguistic tasks. To consider context, language model transducers are used. The method identifies many possible interpretations and selects the best path using Viterbi algorithm. As late as 2014, this approach continues to be used in practice, such as in Google's Kestrel system.

2001

Sproat et al. give a taxonomy of NSWs. They also treat text normalization as a **language modelling** problem. For TTS application, they present both **supervised** and **unsupervised** machine learning approaches, with the latter being a better choice for new domains.

2005

With the growth of **social media**, there's a need to normalize such text. From about mid-2000s, this drives interest in text normalization for social media text.

Jul
2006

Aw et al. propose the metaphor of **Machine Translation (MT)** for normalizing SMS messages. The idea is to "translate" SMS language to English language by adapting a phrase-based statistical MT model. For alignment during training, they use EM algorithm and Viterbi search. They show improved BLEU score. They also show that downstream English to Chinese translations improve.

Oct
2007

Choudhury et al. apply **Hidden Markov Model (HMM)** to the problem of normalizing SMS messages. Non-standard tokens are the emission states. They also adopt the **spell checking** metaphor and process text at character level rather than word level.

Nov
2011

Pennell and Liu introduce a **character-level MT** method. Examples of character-level mappings are 'a'→'er', '@'→'at', and '8'→'ate'. This is only the first phase where possible expansions are identified. In the second phase, a language model is used to choose the correct expansion in context.

Dec
2012

Li and Liu propose an algorithm in which input is **blocks of characters** segmented by **phonetic similarity**. They use two-step MT, translating non-standard words to phones, then phones to words. They use spell checking for simple corrections. In the example, character-level MT misaligns the second 'e' but character-block level MT gets it right.

2013

Previous work often treated text normalization as replacing out-of-vocabulary or non-standard words with dictionary words. Researchers realize that text normalization can't be a "one-size-fits-all" approach. Downstream NLP task or application matters. Zhang et al. normalize with a view on improving performance of dependency parsing rather than simply evaluate based on word error rate and BLEU score. Wang and Ng normalize social media text for better machine translation. Along with word replacement, they recover missing words and correct punctuation.

2015

Baldwin and Li normalize social media text. They evaluate the effect of normalization on three downstream applications: dependency parsing, NER and TTS. They also study the effect of each normalization edit on each of these applications. For example, only word replacements are critical for NER. For parsing, word replacements, token addition and removal edits are important. For TTS, it's critical to remove non-standard tokens while word addition is important but less so.

Oct
2016

Sproat and Jaitly present **neural models** for text normalization. In particular, they use a few layers of BiLSTM. In one architecture, they train a BiLSTM channel model to map characters to word tokens, followed by another LSTM for language modelling. In another architecture, they use 4-layer attention-based BiLSTM sequence-to-sequence model. This performs better than the first one. An FST-based filter improves results further.

Aug
2017

Van Esch and Sproat present a revised **taxonomy of NSWs**. They note that an earlier taxonomy from 2001 is inadequate due to many new categories that have come about due to social media. They present as many as 12 tables of various semiotic classes with useful examples for each. Some of these are word-like tokens, basic numbers, identifiers, dates, times, percentages, measures, geographic entities, and formulae.

Jun
2019

**Historical texts** need to be normalized. Bollmann evaluates and analyses the performance of three systems that do this: Norma (rule-based, distance-based, supervised), cSMTiser (CSMT with additional language modelling data), and Neural Machine Translattion (NMT). He considers texts from many languages, some dating back to 14th century. cSMTiser outperforms NMT in most cases. Norma could be used if there's limited training data.

Jun
2019

It's important to normalize NSWs correctly in spoken dialogue systems such as Amazon Alexa. Mansfield et al. approach this as a machine translation problem and sequence-to-sequence modelling. For better context, they use **attention mechanism on subword units** rather than words. With subwords, we reduce input size and handle OOV words better. BPE is used to create a subword inventory and SentencePiece to find its optimal size. They improve performance further by using linguistic features: POS, position, capitalization, and edit labels.

Text Normalization


# Natural Language Generation

## Summary

There's a lot of structured data that's perhaps easier to understand if described in a natural language. Highlights from a financial spreadsheet, next week's weather prediction, and short summary of a long technical report are some examples. **Natural Language Generation (NLG)** is the process of generating descriptions or narratives in natural language from structured data.

NLG is a sub-field of Natural Language Processing (NLP). NLG often works closely with Natural Language Understanding (NLU), another sub-field of NLP. Where input is unstructured text, NLU helps in producing a structured representation that can be consumed by NLG. Generating language uses more of our brain than understanding it. Likewise, computers might find NLG a more difficult task than NLU.

A mature NLG system can free humans from mundane writing, create narratives quickly, enable almost real-time reporting, and streamline operations.

## Discussion

### What are some applications of NLG?

There are plenty of practical NLG applications: analysis for business intelligence dashboards, IoT device status and maintenance reporting, individual client financial portfolio summaries, personalized customer communications, and more. NLG, along with NLU, is at the core of chatbots and voice assistants.

A familiar example might be Gmail's Smart Compose. It reads email content and suggests short responses. The Associated Press uses NLG to generate thousands of corporate earnings reports in seconds. During the December 2019 UK elections, BBC News published 689 local stories of 100K words in 10 hours. Public datasets were used at the input and articles were generated in a style and tone suited for local audiences.

For computer science domain, NLG has been used to write specifications from UML diagrams, or describe source code changes.

Forge.ai processes unstructured data using mainly supervised NLU models. However, training data is limited. They overcame this by using NLG to synthesize training data. They used human-annotated examples plus knowledge sources. Similarly, synthesized electronic health records can enable de-identified data sharing among healthcare providers and train ML models.

### What's expected of a good NLG system?

NLG must include in its response information that's most relevant to the user in the current context. This includes the level of detail that's to be included. For effective communication, information must be presented in a sensible order. Thus, organization and structure are important. One sentence must follow logically from another. Likewise, NLG must know when to group them into paragraphs or even sections.

NLG must conform to the syntax of the language. In addition, it's a good idea to use common expressions. A sophisticated system could be trainable, domain-independent, and even capable of sarcasms and idiomatic expressions.

Style matters. If the intent is to present high-level financial summary to top management, the writing style must be formal, concise and fact-based. If the intent is to convince readers of a certain point of view, the style could be argumentative with references to supporting evidence.

On the whole, NLG is about making choices and pursuing specific communicative goals.

### What's the typical pipeline in an NLG application?

A typical NLG pipeline has these basic steps:

    + **Content Determination and Text Planning**: Determine the information to be communicated. Structure the information. It's also called *Macro Planning* or *Document Planning*. Information could come from a knowledge base. Selection of information must consider goals and preferences of both writer and reader.
    + **Sentence Planning**: Decide how information must be split into sentences and paragraphs. Plan for a flowing narrative. Also called *Micro Planning*, this involves techniques such as referring expressions, aggregation, lexicalization and grammaticalization.
    + **Surface Realization**: Generate individual sentences in a grammatically correct manner. This involves syntax selection and inflection.
    + **Physical Presentation**: The output could be written or spoken text. Either way, this step does articulation, punctuation and layout.

### Could you describe the main components or tasks in NLG?

We describe a few NLG components:

    + **Aggregation**: Combining two sentences into one using a conjunction, such as "Sam has high blood pressure *and* low blood sugar." Aggregation could also be applied to paragraphs and higher-order structures.
    + **Lexicalization**: This is about word choice. Between 'depart' and 'leave', "the train departed" is more formal than "the train left".
    + **Referring Expression Generation**: Using preceding context, select words or phrases to identify domain entities. An example is the use of pronouns, such as "I just saw Mrs. Black. *She* has a high temperature."
    + **Using Discourse Markers**: The inclusion of the word 'also', makes this sentence more fluent: "If Sam goes to the hospital, he should *also* go to the store."
    + **Linguistic Realization**: Consider the sentence "There are 20 trains each day *from* Aberdeen *to* Glasgow." Following syntactic rules, NLG has added function words 'from' and 'to'. Due to morphology, the plural form of 'train' has been used. Due to orthography, the first word is capitalized and full stop added at the end.

### Which are main technical approaches to building NLG systems?

We've two approaches to NLG:

    + **Template-based**: Texts have pre-defined structure with gaps. Gaps are filled with data. These systems evolved to allow greater control via scripts and business rules. These developed further to include linguistic capability to generate grammatically correct text.
    + **Dynamic Creation**: At a micro level, sentences are created dynamically from semantic representations and a desired linguistic structure. At a macro level, sentences are organized into a logical narrative as suited for the audience and the purpose of communication.For dynamic creation of text, a **Markov Chain** model was initially applied. Given the current word and its relationships with other words, this model predicts or generates the next word. However, such models often lacked structure and context.

More recently, **Language Modelling** has been applied to NLG. Given recent words, the model predicts the next word. An *n-gram model* is a specific way to build a language model. Later, neural networks were successfully applied to language modelling.

Some classify NLG into Basic NLG, Templated NLG and Advanced NLG.

### How have neural networks been applied to build NLG models?

RNNs, LTSMs and transformer networks have been used for NLG models. RNNs exploit the sequential nature of text and "remember" previous words to predict the next word. LSTMs are better than RNNs at remembering longer word sequences. They also have a mechanism to selectively forget when context changes.

**Transformers** have become popular due to their state-of-the-art performance. They look at the relationships among words in context. All words are represented individually in the vector space instead of reducing them to a single fixed-length vector. Transformers have been used to build language models and thereby enable NLG. Two popular ones are BERT and GPT-2. However, it's been remarked that GPT-2 lacks language understanding.

Neural networks typically take an **end-to-end approach** to NLG. We could use an encoder-decoder architecture along with an attention layer. The encoder produces a hidden representation of the input text. The decoder generates the text. Compared to template-based systems, neural network models give less control but are more flexible and expressive.

### How can I evaluate NLG models?

Evaluating NLG systems is a challenge in itself and this has been addressed since the mid-1990s. Evaluations are useful to assess the underlying theory of an NLG system, compare two NLG systems, or determine if a particular is showing improvements. In **black box evaluation**, we evaluate the entire system. In **glass box evaluation**, we evaluate the system's component parts such as document structuring or aggregation.

Evaluations should look at **accuracy** (conveying the desired meaning) and **fluency** (text flows and is readable). These two factors are not necessarily correlated. Evaluations can be by humans or automated. Multiple methods and metrics are preferred.

Evaluation metrics can be word-based (TER, BLEU, ROUGE), grammar-based (readability, spelling errors, characters per word, etc.) or measure semantic similarity. Another way to categorize the metrics is n-gram overlap (BLEU, GTM, CIDEr), content overlap (MASI, SPICE) and string distance (Levenshtein, TER). BLEU has been criticized by researchers.

The E2E NLG Challenge (2017-18) used five metrics: BLEU, NIST, METEOR, ROUGE-L, and CIDEr; plus RankME. Rank-based Magnitude Estimation (RankME) enables reliable and consistent human evaluation.

### Could you describe some tools to help with NLG?

Among the commercial tools are Arria NLG, AX Semantics, Yseop, Quill, and Wordsmith. Among the open source tools are SimpleNLG and NaturalOWL. Wordsmith is from Automated Insights, who can be regarded as a pioneer in NLG. Quill, from Narrative Science, makes use of deep learning.

Some NLG platforms (such as Wordsmith) can be integrated with business intelligence platforms so that dashboards can be enhanced with descriptive explanations or highlights.

## Milestones

1950

During the 1950s and 1960s, NLG is not seen as a dedicated field of study but as a component of **machine translation** systems. It's only in the 1970s that researchers look into NLG for the purpose of communication. In the 1980s, NLG becomes a dedicated field of research. Also in the 1950s, the approach is **template-based** and therefore NLG produces the same output to all listeners in all situations.

Nov
1970

Simmons and Slocum adapt **semantic networks** to the task of NLG. Nodes are word senses. Connecting paths are relations. Essentially, the network defines the grammar as ordered sets of syntactic transformations. The grammar includes voice, form, aspect, tense, mood, and so on. This is used by the algorithm when generating text. They implement this in LISP.

1974

Davey makes the case that production (NLG) is not a reverse process of comprehension (NLU).  While language syntax may benefit comprehension, it's not sufficient for discourse-based production. From a given semantic representation, the form that the sentence would take is unpredictable.

1992

As one of the earliest commercial applications of NLG, Goldberg et al. describe a system that generates bilingual weather reports from graphical weather predictions.

Mar
1997

Reiter and Dale note that the most common architecture for NLG is a **three-stage pipeline**. They describe each stage along with essential tasks that belong to each stage. They give guidelines to developers to build practical NLG systems. They also note that NLG may not be the right approach for some use cases. Instead, present information graphically, use mail-merge systems or employ human authors.

Sep
1989

Hovy notes that beyond the literal meaning of words, we often tune our communication in relation to the listener, the situation, or interpersonal goals. Hovy therefore studies the importance of **pragmatics**. NLG has often asked two questions: "what shall I say?" and "how shall I say it?" NLG should also concern itself with "why should I say it?" Hovy also implements a program called *PAULINE*, which he acknowledges being primitive and much more research needs to be done.

1993

Miikkulainen proposes *DISCERN* as a sub-symbolic **neural network model** for NLG. It can create summaries or answer questions. The system includes parsers, at sentence level and story level. For generation, there's a story generator and a sentence generator. These are helped by a memory block and a lexicon. Distributed real-valued vectors represent words both at lexical and semantic levels. Representations are based on thematic case roles. Other early neural networks for NLG from the 1990s include *ANA* and *FIG*.

Jun
2000

The first *International Natural Language Generation Conference (INLG)* is held, as a continuation of international workshops on NLG held regularly during 1982-1998. INLG becomes an almost biennial event. Full papers of these conferences are available online.

2007

For college basketball in the US, *StatSheet* is launched to provide game previews, injury reports, and recaps. Much of this is generated automatically. Soon it becomes the top website for college basketball statistics. In 2011, StatSheet changes its name to Automated Insights and partners with NFL for automated content production.

2013

NLG enters the fields of **data analysis and business intelligence**. A particular example is Allstate Industries adopting NLG. In 2015, Gartner names NLG as an official technology space. It predicts that by 2020, NLG will be a standard feature in 90% on these platforms. By 2016, NLG gets integrated into many BI and data visualization platforms such as Tableau, MicroStrategy and Power BI.

Jun
2016

By exploiting similarities across domains, Wen et al. propose a method to train models for **multi-domain NLG**. Synthesized out-of-domain training data is created by data counterfeiting. Then the model is fine-tuned using a much smaller dataset of in-domain utterances. Semantically Conditioned Long Short-Term Memory (SC-LSTM) network is used. Reading gates control semantic input features presented to the network.

Aug
2016

Researchers show that when users are presented with uncertain data, NLG improves decision making. They use graphs of weather forecasts with likelihoods. Decisions are better by 24% with NLG, and by 44% when NLG is combined with graphs. The improvement is as high a 87% for women.

Mar
2017

Researchers initiate the *E2E NLG Challenge*. Whereas there are many distinct steps in traditional text generation method, **end-to-end generation** attempts to combine all steps into a single model. Such a model would be trained on vast amounts of annotated text. For the Challenge, a crowdsourced dataset of 50k instances in the restaurant domain is used. Training data consists of meaning representations (MRs) and corresponding reference texts. In 2018, from 62 submissions, a seq2seq-based ensemble model with re-ranker is judged the overall winner.

Natural Language Generation


# Log Level Per Request

## Summary

For applications in production, setting breakpoints is usually not a viable option. Logging is the preferred approach to troubleshoot problems in production. For runtime efficiency, storage and analysis we often want to log only the most important messages. However, when an error occurs, a more detailed log is needed to find the root cause. But turning on full debug or trace level logging can result in huge logs and slow down analysis.

The idea of log level per request is to increase the level of logging only for that request that's causing the error.  Executions triggered by other requests will continue to be logged in the default manner. This approach is particularly useful for microservices architecture where a single request can involve multiple microservices, all of which are possibly handling hundreds of requests per second.

## Discussion

### Why is log level per request important for microservices?

Complex large-scale applications are moving away from monolithic architecture to microservices architecture. But failures are hard to troubleshoot in microservices. A single HTTP request can trigger a sequence of calls and responses across many microservices. This distributed nature of microservices makes it hard to debug problems such as malformed data. It's harder to track down performance issues.

In monolithic applications, it's easy to put breakpoints and follow the sequence of function calls to understand the problem, at least in a development environment. This isn't possible for microservices, particularly in production. Log level per request therefore helps us obtain detailed logs pertaining to a particular problem. In general, logging, monitoring and measuring comes under **instrumentation**.

For example, a microservice may be called in dozens of user scenarios only one of which is causing an error. Detailed debug logging can be turned on for this single scenario. We avoid overwhelming the system with debug level logging for all other high-level user requests.

### What should I expect from a good implementation of log level per request?

To change log level on a per request basis, you shouldn't be asked to recompile the application. It should be possible to set the log level easily for different environments: development, production, user acceptance testing, non-functional testing, etc. Log level should be configurable for each application or component, and independent of event-raising code.

It's recommended that you log entry and exit of functions. Log enough to set the context, including name of function, user, or even IP address. Logs should be complete in the sense that the information should help troubleshoot a problem. Past incident reports can help you decide what, where and how much to log. In time, you can even develop logging patterns. Your development pipeline should catch code where logs are missing. Deployment may flag functions that are not logging.

In general, for microservices logging, centralize the logging from different hosts. Log structured data, such as in JSON format. Log asynchronously. Make logs searchable, for instance, by using ELK Stack.

### What are some possible techniques for implementing log level per request?

One suggested approach is to use a **Correlation ID**, passed from service to service. This ID is unique to a particular transaction or a high-level request. With this ID, we can follow the execution journey of a transaction in the logs. Moreover, we can set the desired log level for this transaction.

To change log levels without compiling code, we could have a definitive list of all events/requests and a configurable mapping between these and the log levels. Such a mapping could be in an XML file.

A correlation ID can be used whenever we wish to group together errors due to a request. Some examples of grouping are scheduled or background tasks; synchronized operations; all activities pertaining to a single request.

A correlation ID can be generated by an "edge" microservice that then triggers internal microservices. In addition, any microservice that doesn't see an ID can generate one for subsequent tracing. In fact, an API Gateway in AWS generates these IDs that are passed along in HTTP headers. Preferably, the ID is assigned as early as possible.

### Isn't log level per request same as event tracing?

They are related but not the same. Event tracing is about tracing an event and its effect across systems and services. It's supported by tools such as Sentry. Not just events or requests, even messages in a queue can carry the **trace ID**. However, traditionally event tracing was not really about dynamically setting the log level on a per request basis. It was about correlating log entries based on a trace ID.

### Could you name some tools that support log level per request?

Tracing frameworks might support this feature. Some of them may be based on the *OpenTracing* standard. Zipkin, Datadog, and Dynatrace are tools that accept OpenTracing format.

Logging frameworks are tied to languages: Log4j (Java), NLog (.NET), Node-Loggly (Node), etc. Select a framework based on code cleanliness, performance and familiarity.

If using Spring Cloud for implementing microservices, use Spring Sleuth along with Zipkin for using correlation IDs. You can use Spring Boot admin to change log levels dynamically.

In .NET framework, namespace `System.Diagnostics` has the property `CorrelationManager.ActivityId`. This is set per request. Thus, all errors thrown during one request will have the same ID.

ModSecurity, a web application firewall, supports log level per request and also allows us to change the level at runtime.

## Milestones

2010

At the University of Lisbon, researchers extend a framework called STEP for monitoring and performance analysis. STEP is a Java-based enterprise application framework created for educational purpose. They use Apache Log4j for logging and note that debug and trace levels are impractical for production environments. They propose activating such logging for a subset of request such as requests from a specific user.

2012

Matthew Skelton writes about tuning logging levels in production without recompiling code. He notes that often logging API suffer from **leaky abstraction** because application selects the log level at the point of calling. Thus, we would need to recompile to effect new logging levels. Apache log4net allows change of log levels without recompiling but it can't do this for specific event types.

Jul
2014

**Apache Log4j 2** (version 2.0) is released. Using `ThreadContext` and `DynamicThresholdFilter`, we can control logging level on a per-request basis. Therefore, we can "debug batch jobs which have a certain flag, WebSocket connections or any other thread-based processing". This is in addition to debugging HTTP requests by setting a flag such as `x-debug-enabled: true` in the HTTP header.

Dec
2016

A Rapid7 blog post notes that the use of correlation IDs is a well-known enterprise integration pattern called **Correlation Pattern**. Such an ID is a non-standard HTTP header and it's part of Java Messaging Service (JMS). An example is a travel reservation system. Once an ID is assigned to a reservation request, the same ID is passed to hotel, car rental and airline systems/services. Timestamps are not enough in logs and a correlation ID "provides the glue that is needed to create a coherent, understandable audit of events".

Jan
2017

One developer named Reik shows how to use Twitter's **Finagle** framework to dynamically set per-request log level. The idea is to use Finagle's "broadcast context to transport a log level override through an RPC graph". HTTP headers are used and Finagle filters can read them.

May
2018

ThoughtWords includes log level per request in its quarterly Technology Radar. It notes its relevance towards debugging microservices.

Log Level Per Request