# Microservices Observability

## Summary

With traditional monolithic applications, troubleshooting problems was easier compared to the more recent microservices architecture. Log files captured information for debugging and analysis. Monitoring highlighted problems via static dashboards and alerts. The application itself was generally well understood.

With microservices, there are more moving parts. The system is dynamic and heterogenous. It's parts are loosely coupled and transient. The system could fail in many different and unpredictable ways. It's necessary to monitor the application, the network and the infrastructure. It's in this context that observability (aka *o11y*)  becomes important.

Can we understand the inner workings of the system? Can we correlate the outputs with the inputs? Can we explain unexpected behaviour? Can data help us achieve business goals? A system that answers these questions effectively is said to be **observable**.

## Discussion

### How does observability differ from monitoring?

IT systems have been monitoring performance since the early 2000s. They sample and aggregate important data points such as system downtime, response time and memory usage. These are compared against expectations. Anomalies and violations are thrown up as alerts for support teams. Monitoring focuses on overall system health and business KPIs.

Observability goes deeper with the aim of capturing system behaviour in greater detail and context. An application composed of many interacting microservices can fail in unpredictable ways. So it's not sufficient to focus on specific failures or scenarios. While monitoring deals with known unknowns, observability deals with unknown unknowns. Data analysis is an essential aspect of observability. Failures are traced back to root causes. To put it simply,

> Monitoring tells you **when** something is wrong. Observability lets you ask **why**.

Observability doesn't replace monitoring. Monitoring is essential to observability. We may even view observability as a superset of monitoring.

### What are the main benefits of observability?

Observability leads to better visibility into the system dynamics. We can identify bottlenecks and optimize workflows. It creates a culture of innovation, improves operational efficiency and enables data-driven business decisions. DevSecOps teams can use the insights from observability to build more secure applications.

Because observability helps us understand the system better, engineers can more confidently update and release software. Release cycles can be shorter. Quality can be improved. Problems that you we didn't know existed in the system ("unknown unknowns") become visible. We can find and fix issues earlier in the software development phase. When combined with AIOps, issues can be solved automatically without human intervention.

For any organization moving from monoliths to microservices, observability helps in this transition. It can help them better manage their cloud-native applications.

### What are the key pillars of observability?

Observability has three key pillars:

    + **Logs**: Capture individual events. Logs are granular, timestamped and immutable. When things go wrong, logs are invaluable in determining the root cause. A good practice is to log in structured formats such as JSON. Structured logging enables easy visualization, search and analysis.
    + **Metrics**: Data aggregated over time. Monitoring solutions rely on metrics. Uptime, CPU utilization, system load, memory usage, throughput, response time and error rate are some examples of metrics.
    + **Traces**: Sequence of calls triggered by an event. In the world of microservices, a failure can be traced back to an offending microservice or an API call.**Events** and **exceptions** can be seen as special cases of logs.  Some identify **dependencies** as another pillar. This captures how components depend on one another.

Historically (in 2016), engineers at Twitter identified these four pillars: monitoring, alerting/visualization, distributed systems tracing infrastructure, and log aggregation/analytics.

### What's the typical pipeline for microservices observability?

A logger produces logs within each service instance. A centralized collector collects the logs. Data is then pre-processed and stored to ease later analysis. Pre-processing might include cleaning, formatting, sampling and aggregating (for metrics). Analysts may run ad hoc queries to explore and visualize data, and to solve problems.

Each logger might be part of the main service or deployed as a sidecar container within the same Kubernetes pod of the main container.  More generally, an agent pull logs from services or log files and then pushes them to the collector. Alternatively, log collection is agent-less, that is, services themselves push logs to the collector. The latter is easier to deploy but requires developers to integrate logging functionality into the service code.

Different types of analysis are possible: timeline analysis, service dependency analysis, aggregation analysis, root cause analysis and anomaly detection. Analysis is enabled by the use of statistics, rules and visualization.

### What design patterns are available for microservices observability?

Among the design patterns are:

    + **Application Metrics**: Gives a complete picture of the application. Includes infrastructure, application and end user metrics.
    + **Audit Logging**: Record all user or service account actions. Needed for regulatory compliance. Event sourcing is one approach to capturing audit logs. Likewise, changes to deployment ought to be logged.
    + **Distributed Tracing**: Helps with performance profiling and root cause analysis. A sequence of API calls triggered by a user request is a trace. A trace is composed of spans. Each span records a unit of work within the trace. For example, user request, API gateway processing, service processing and database access are all spans within a trace. Distributed tracing requires context or correlation IDs passed from one service to the next.
    + **Health Check**: An unhealthy service is one that's running but has problems. Health check API can assess current system health. It can trigger alerts, recovery, service restarts, etc.
    + **Exception Tracking**: Includes error messages, error codes and stack traces.
    + **Log Aggregation**: Logs from various microservices are sent to a centralized log server.

### As a developer, should I care about observability?

Observability is not just the concern of operations. Observability must be considered at the design stage itself. System should be designed to be testable. It should be incrementally deployable with support for rollback. System should collect detailed runtime data.

Logs should be designed to support data with high cardinality and high dimensionality. This makes it possible to query the log data effectively to bring out patterns and root causes. A useful query such as "list all 502 errors in the last 20 minutes from host foobar.com" becomes possible.

Developers must acknowledge that complex systems are unpredictable and failures are inevitable. They should proactively instrument their code, rather than rely on automatic instrumentation.  Even controlled testing in production may be permitted. Developers should consider the operational semantics and dependencies of their software. For example, developers should understand how services start/shutdown, their static/dynamic configurations, service discovery, concurrency, etc.

### What are the best practices when implementing observability?

Monitor essential metrics such as latency, traffic, error rate and saturation. Collect metrics along with contextual metadata. Configure and prioritize alerts correctly. For example, an alert for every 500 status code is a bad thing. Create specialized dashboard rather than overwhelming everyone with all the data.

Rules that detect incidents should be simple, predictable, reliable and actionable. Monitoring systems can become complex over time. They could end up collecting lots of metrics that are never used or complex rules that are never triggered. Data should lead to actionable insights. Data should create effective feedback loops.

Investing in observability tools matters, but tools alone will not solve problems. Teams must come with a mindset towards making their systems observable, from design to deployment.

An observability framework can serve as a reference for implementations. To be effective, observability must be guided by business goals such as Service Level Objectives (SLOs).  Look at observability from the perspective of end users (response times, failed purchases) and backend applications (slow queries, container restarts).

### What tools are available for observability?

Many frameworks (eg. ASP.NET and Spring Boot) have libraries or plugins that implement OpenTelemetry. Pick one that supports W3C Trace Context. Select tools that support automation and are easy to maintain.

Some have proposed observability stacks that combine analysis, logging and visualization: ELK (Elasticsearch, Logstash, and Kibana), EFK (Elasticsearch, Fluentd, and Kibana), or PLG (Promtail, Loki, Grafana).

For logging from microservices we have Splunk, Loggly, Sumologic, Logstash, Beats, Fluent Bit, and more.

Time-series data can be stored in Atlas, InfluxDB and Prometheus. Databases (such as Oracle) provide exporters and make data accessible to the observability stack. Prometheus helps with monitoring containers. Jaeger offers a visual representation of traces and dependencies. Dashboards can be created with Grafana.

A few others include Chronosphere, Cisco AppDynamics, Datadog, Dynatrace, Honeycomb, IBM Instana, Lumigo, New Relic, Sensu, Sentry, SolarWinds, Serverless360, and Zipkin.

Cloud providers offer managed services for observability.  For example, AWS has managed services for Grafana, Prometheus and OpenTelemetry (Java SDK + collector implementation). It also offers its proprietary AWS X-Ray and Amazon CloudWatch.

### What are some challenges with observability?

A New Relic survey from 2022 showed that observability is far from mature. Many problems are still detected manually. Organizations use too many tools. Some tools require developers to instrument their code with calls to tracing libraries. Some tools can monitor microservices but not monoliths. Some tools are SaaS only. Some tools sample data while others don't.

Large volumes of data can be overwhelming. Many tools may not scale well when data has high cardinality. Reducing the cardinality will make the system less observable. Designers must therefore consider this tradeoff until better tools become available.

Where service meshes are used, observability is harder due to increased variety of services, data volume and complex request paths. Often systems collect all metrics and later determine the relevant ones to analyze. It would be better to determine upfront the most relevant metrics and collect only those. ViperProbe can do this.

## Milestones

1960

Rudolf E. Kálmán coins the term **observability** in the domain of control theory. It's "a measure of how well internal states of a system can be inferred from knowledge of its external outputs." Five decades later, this term gets repurposed towards distributed software systems. In the latter case, observability is the ability to understand and explain any system state, no matter how unusual or unexpected it may be.

1988

At IETF, RFC 1067 titled *A Simple Network Management Protocol* is published. In subsequent years, this becomes the essential protocol for monitoring network infrastructure via **metrics**. Time-series databases and dashboards are later born out of metrics. But engineers reach for low-level tools (`strace`, `tcpdump`) when metrics fall short for debugging complex systems.

1999

Engineers at Sun Microsystems use the term "observability" as something that enables application performance monitoring and capacity planning. At this time, there are no microservices. Hence this definition of observability is rather narrow and closer to the modern use of the term "monitoring".

2011

The term **microservices** gets discussed at a workshop of software architects near Venice, to describe a common architectural style that several of them were exploring at the time. By 2015, microservices become mainstream as per a survey by NGINX.

2013

Twitter publishes on its engineering blog an article titled *Observability at Twitter*. The article describes the elements of observability as practised at Twitter as they moved from a monolithic architecture to a microservices architecture. The article uses the terms "observability team" and "observability stack". Beyond just traditional monitoring, the stack includes time-series database, dashboards and ability to run ad hoc queries. This extension of traditional monitoring has been called **whitebox monitoring**.

2015

By mid-2010s, the term "observability" becomes more common. By 2018, it's commonly used in conferences and blog posts.

2019

**OpenTelemetry** is formed by merging two earlier projects named OpenTracing and OpenCensus. It's an incubation project at CNCF. It collects logs, traces and metrics from services, and sends them to analysis tools. It integrates easily with popular frameworks and libraries.

Nov
2021

W3C publishes *Trace Context* as a W3C Recommendation. This standardizes new HTTP header names and formats that allow context information to be propagated across HTTP clients and servers in any distributed system. It's worth noting that OpenTelemetry supports W3C Trace Context format.

2023

GigaOm publishes a research finding on the current state of the market in the cloud observability space. The study considers supported deployment models: SaaS, on-premise or hybrid. It considers many features: dashboards, user interaction performance monitoring, predictive analysis, microservices detection, etc. It's likely that no single platform is the best fit for every use case.

Microservices Observability


# End-to-End Principle

## Summary

When a function has to be supported in a networked system, the designer often asks if it should be implemented at the end systems; or should it be implemented within the communication subsystem that interconnects all the end systems. The end-to-end argument or principle states that it's proper to implement the function in the end systems. The communication system itself may provide a partial implementation but only as a performance enhancement.

The architecture and growth of the Internet was shaped by the end-to-end principle. It allowed us to keep the Internet simple and add features quickly to end systems. The principle enabled innovation.

More recently, the principle has been criticized or violated to support features such as network caching, differentiated services, or deep packet inspection.

## Discussion

### Could you explain the end-to-end principle?

Suppose we need to transfer a file from computer A to computer B across a network. Let's assume that the network guarantees correct delivery of the file by way of checksums, retransmissions, and deduplication of packets. Thus, our hypothetical network is full of features but also complex. The problem is that despite such a smart network, the file transfer can still go wrong. The file could get corrupted on B during transfer from buffer to the file system. This implies that end computers still have to do the final checks even if the network has already done them.

This is the essence of the end-to-end (E2E) argument. A communication system may do some things for performance reasons but it can't achieve correctness. For reasons of efficiency and performance, the communication system may implement some features at minimal cost but should avoid trying to achieve high levels of reliability. Reliability and correctness must be left to end systems.

In addition, applications may not need features implemented in the communication system. An "open" system would give more control to end systems.

### How has the end-to-end principle benefited the Internet?

Complexity impedes scaling due to higher OPEX and CAPEX. Thus, the end-to-end principle leads to the **simplicity principle**. IP layer is simple, giving Internet it's hourglass-shaped architecture.

At the network layer we have IP as the dominant protocol. At higher layers, we have many protocols for supporting diverse applications. At lower layers, we have many protocols suited to different physical networks. IP can be said to "hide the detailed differences among these various technologies, and present a uniform service interface to the applications above".

IP itself is simple and general. It's supported by all routers within the network. Application-level functions are kept at endpoints. Application developers could therefore innovate without any special support from the network, with some calling it the **generative Internet**. The principle has been credited with making Internet a success.

Research has also shown that future architectures for the Internet are likely to evolve into the hourglass shape.

### Does the end-to-end principle prohibit the network from maintaining state?

End-to-end applications can survive partial network failures. This is because the network maintains only coarse-grained states, while endpoints maintain the main states. State can be destroyed only when the endpoint itself is destroyed, called **fate sharing**. Fate of endpoints doesn't depend on the network.

Routing, QoS guarantees, and header compression are some examples where the network may maintain state. However, this state is self-healing. It can be worked out even if network topology or activity changes. State maintained within the network must be minimal. Loss of this state should at most result in temporary loss of service.

State maintained in the network may be called *soft state* while that maintain at endpoints, and required for the proper functioning of the endpoints, may be called *hard state*.

### Besides the Internet, where else has the end-to-end principle been applied?

On the Internet, end-to-end principle has been applied to reliable delivery, deduplication, in-order delivery, reputation maintenance, security, and fault tolerance. For security, both authenticity and encryption of messages are best done at endpoints. Doing these within the network will not only compromise security but also complicate key management.

It's application to file transfer is well known. Checksum should be validated only after successful storage to disk. Another example is the EtherType field in Ethernet frame. Ethernet doesn't interpret this field. To do so would mean that all higher layer protocols would pay the price for the special few.

In computer architecture, RISC favours simple instructions over the complex ones of CISC. In CISC, a designer may include complex instructions but it's hard for her to anticipate client requirements. Clients may end up with their own specific implementations.

A case has been made to apply end-to-end principle for data commons, that is, sharing and organizing data for a discipline. Applications can decide how to import, export or analyze data. The core system would only define global identifiers, basic metadata, authenticated access and a configurable data model.

### What are the essential aspects of end-to-end connectivity?

RFC 2775 (published in 2000) mentions three aspects:

    + **E2E Argument**: This is as described by Saltzer et al. in 1981, and what is now called the end-to-end principle.
    + **E2E Performance**: This concerns both the network and the end systems. Research in this area has suggested some improvements to TCP plus optimized queuing and discard mechanisms in routers. However, this won't help other transport protocols that don't behave like TCP in response to congestion.
    + **E2E Address Transparency**: A single logical address space was deemed adequate for the early Internet of the 1970s. Packets could flow end to end unaltered and without change of source or destination addresses. RFC 2101 of 1997 analyzed this aspect and concluded that address transparency is no longer maintained in present day Internet. An example of this is Network Address Translation (NAT). Applications that assume address transparency are likely to fail unpredictably.

### Are there instances where the end-to-end principle has been violated?

Consider a client-server architecture involving a database write operation at the server. A "smart" server can return an immediate confirmation to the client even though it hasn't completed the database write operation. If the write fails, the server has to do retries, effectively taking up responsibilities of the client. It gets worse if the server itself fails, since client thinks that the database write actually happened.

Bufferbloat was an interesting problem the Internet had in the late 2010s. Because routers could afford larger buffers, they started accepting higher load. They didn't drop packets until later when their larger buffers started filling up. Therefore, endpoints didn't backoff early enough. This resulted in a slower internet.

HTTP caching, link-level encryption, SOAP 2.0, Network Address Translation (NAT) and firewalls are other counter-examples.

Examples where the network gets involved are traffic management, capacity reservation, packet segmentation/reassembly, and multicast routing. But these shouldn't be seen as violating the principle. Likewise, cloud computing doesn't violate the principle. Cloud infrastructure is not part of the communication system. It's actually an endpoint.

### How is end-to-end principle relevant to the net neutrality debate?

Net neutrality is about creating a level-playing field for everyone, big or small. It ensures that big companies can't pay for preferential treatment of their content. The network sees and treats all content alike. Without net neutrality, a few companies that own or control online platforms or communication infrastructure become all too powerful. Power therefore moves from the end consumers to the network controlled by a few. This goes against the end-to-end principle.

Tim Wu coined the termed "network neutrality" back in 2002, when he noticed that broadband providers were blocking certain types of services. Network providers might promise services such as blocking spam, viruses or even advertisements. Most users would rather do these on their end systems rather than lose control.

Deep Packet Inspection (DPI) is used for QoS, security and even surveillance. It's at odds with net neutrality. With DPI, intermediate nodes look into packet headers and payload. Yet, the end-to-end principle doesn't actually prohibit this.

### What are the common criticisms of the end-to-end principle?

One criticism is that the original paper by Saltzer et al. never properly understood the true nature of packet switching, which is stochastic. The paper also confused moving packets through the network (statistical) with non-functional aspects such as confidentiality (computational). It would have been better to model timely arrival of packets to enable successful computation.

The end-to-end principle never gave end users freedom. Network infrastructure has always been built and controlled for commercial reasons by those who had the means. Therefore, to protect user interests, discussions must involve everyone in the industry.

Back in 2001, researchers noted new applications and scenarios that end-to-end principle didn't address very well: untrusted endpoints, video streaming, ISP service differentiation, third-parties, and difficulty in configuring home network devices. All of these could benefit with some intelligence in the network.

In Service-Oriented Architecture (SOA), implementing stuff end-to-end would be too costly. A hop-by-hop approach would be better. Ultimately, the principle shouldn't be applied blindly. In the early days of the Internet, when bandwidth was scarce, HTTP caching made sense even when it violated the end-to-end principle, even when it made HTTP a considerably more complex protocol.

## Milestones

1950

In the 1950s, for reading and writing files to magnetic tapes, engineers attempt to design a reliable tape subsystem. They fail to create such a system. Ultimately, applications take care of checks and recovery. An example of this from the 1970s is the Multics file system. Although there's low-level error detection and correction, they don't replace high-level checks.

1973

Frenchman Louis Pouzin designs and develops CYCLADES, a packet switching network. It becomes the first network in which hosts are responsible for reliable delivery of packets. Networks transport datagrams without delivery guarantees. Even the term *datagram* is coined by Pouzin. CYCLADES inspires the first version TCP. Meanwhile, D.K. Branstad makes the end-to-end argument with reference to encryption.

1978

What was originally TCP, is split into two parts: TCP and IP. Thus, layered architecture is applied and the functions of each layer becomes more well defined. On January 1st, 1983, TCP/IP becomes the standard protocol for ARPAnet that by now connects 500 sites.

Apr
1981

J.H. Saltzer, D.P. Reed and D.D. Clark at the MIT Laboratory for Computer Science present a conference paper titled *End-to-end Arguments in System Design*. Given a distributed system, the paper gives guidance on where to place protocol functions. For example, end systems should perform recovery, encryption and deduplication. Low-level parts of the network could support them only as performance enhancements. Phil Karn, a well-known Internet contributor, comments years later that this is "the most important network paper ever written".

1996

The IETF publishes RFC 1958 titled *Architectural Principles of the Internet*, with reference to the end-to-end principle of Saltzer et al. In 2002, this RFC is updated by RFC 3439. Other IETF documents relevant to this discussion are RFC 2775 (2000), and RFC 3724 (2004).

May
1997

David S. Isenberg, an employee of AT&T, writes an essay titled *The Rise of the Stupid Network*. He notes that telephone networks were built on the assumption of scarce bandwidth, circuit-switching, and voice-dominated calls. This has led to the creation of **Intelligent Network (IN)**, where the network took on more features. Given the rise of the Internet, Isenberg argues that the time has come for telephone networks to become stupid and allow endpoints to do intelligent things. Tell networks, "Deliver the Bits, Stupid".

End-to-End Principle


# Redis Streams

## Summary

Redis has data types that could be used for events or message sequences but with different tradeoffs. Sorted sets are memory hungry. Clients can't block for new messages. It's also not a good choice for time series data since entries can be moved around. Lists don't offer fan-out: a message is delivered to a single client. List entries don't have fixed identifiers. For 1-to-n workloads, there's Pub/Sub but this is a "fire-and-forget" mechanism. Sometimes we wish to keep history, make range queries, or re-fetch messages after a reconnection. Pub/Sub lacks these properties.

Redis Streams addresses these limitations. Stream data type can be seen as similar to logging, except that Stream is an abstraction that's more performant due to logical offsets. It's built using radix trees and listpacks, making it space efficient while also permitting random access by IDs.

## Discussion

### What are some use cases for Redis Streams?

Redis Streams is useful for building chat systems, message brokers, queuing systems, event sourcing, etc. Any system that needs to implement unified logging can use Streams.  Queuing apps such as Celery and Sidekiq could use Streams. Slack-style chat apps with history can use Streams.

For IoT applications, Streams can run on end devices. This is essentially time-series data that Streams timestamps for sequential ordering. Each IoT device will store data temporarily and asynchronously push these to the cloud via Streams.

While we could use Pub/Sub along with lists and hashes to persist data, Stream is a better data type that's designed to be more performant. Also, if we use Pub/Sub and Redis server is restarted, then all clients have to resubscribe to the channel.

Since Streams supports blocking, clients need not poll for new data. Blocking enables real-time applications, that is, clients can act on new messages as soon as possible.

### Which are the new commands introduced by Redis Streams?

All commands of Redis Streams are documented online. We briefly mention them:

    + **Adding**: `XADD` is the only command for adding data to a stream. Each entry has a unique ID that enables ordering.
    + **Reading**: `XREAD` and `XRANGE` read items in the order determined by the IDs. `XREVRANGE` returns items in reverse order. `XREAD` can read from multiple streams and can be called in a blocking manner.
    + **Deleting**: `XDEL` and `XTRIM` can remove data from the stream.
    + **Grouping**: `XGROUP` is for managing consumer groups. `XREADROUP` is a special version of `XREAD` with support for consumer groups. `XACK`, `XCLAIM` and `XPENDING` are other commands associated with consumer groups.
    + **Information**: `XINFO` shows details of streams and consumer groups. `XLEN` gives number of entries in a stream.

### What are main features of Redis Streams?

Streams is a first-class citizen of Redis. It benefits from the usual Redis capabilities of persistency, replication and clustering. It's stored in-memory and under a single key.

The main features of Streams are:

    + **Asynchronous**: Producers and consumers need not be simultaneously connected to the stream. Consumers can subscribe to streams (push) or read periodically (pull).
    + **Blocking**: Consumers need not keep polling for new messages.
    + **Capped Streams**: Streams can be truncated, keeping only the N most recent messages.
    + **At-Least Once Delivery**: This makes the system robust.
    + **Counter**: Every pending message has a counter of delivery attempts. We can use this for dead letter queuing.
    + **Deletion**: While events and logs don't usually have deletion as a feature, Streams supports this efficiently. Deletion allows us to address privacy or regulatory concerns.
    + **Persistent**: Unlike Pub/Sub, messages are persistent. Since history is saved, a consumer can look at previous messages.
    + **Lookback Queries**: This helps consumers analyse past data, such as, obtain temperature readings in a particular 10-second window.
    + **Scale-Out Options**: Via consumer groups, we can easily scale out. Consumers can share the load of processing a fast-incoming data stream.

### Could you explain consumer groups in Redis?

A consumer group allows consumers of that group to share the task of consuming messages from a stream. Thus, a message in a stream can be consumed by only one consumer in that consumer group. This relieves the burden on a consumer to process all messages.

Command `XGROUP` creates a consumer group. A consumer is added to a group the first time it calls `XREADGROUP`. A consumer always has to identify itself with a unique consumer name.

A stream can have multiple consumer groups. Each consumer group tracks the ID of the last consumed message. This ID is shared by all consumers of the group. Once a consumer reads a message, it's ID is added to a *Pending Entries List (PEL)*. The consumer must *acknowledge* that it has processed the message, using `XACK` command. Once acknowledged, the pending list is updated. Another consumer can *claim* a pending message using `XCLAIM` command and begin processing it. This helps in recovering from failures. However, a consumer can choose to use the `NOACK` subcommand of `XREADGROUP` if high reliability is not important.

### Could you share more details about IDs in Redis Streams?

Entries within a stream are ordered using IDs. Each ID has two parts separated by hyphen: UNIX millisecond timestamp followed by sequence number to distinguish entries added at the same millisecond time. Each part is a 64-bit number. For example, `1526919030474-55` is a valid ID.

IDs are autogenerated when `XADD` command is called. However, a client can specify its own ID but it should be an ID greater than all other IDs in the stream.

Incomplete IDs are when the second part is omitted. With `XRANGE`, Redis will fill in a suitable second part for us. With `XREAD`, the second part is always `-0`.

Some IDs are special:

    + `$`: Used with `XREAD` to block for new messages, ignoring messages already in the stream.
    + `-` & `+`: Used with `XRANGE`, to specify minimum and maximum IDs possible within the stream. For example, the following command will return every entry in the stream: `XRANGE mystream - +`
    + `>`: Used with `XREADGROUP`, to get new messages (never delivered to other clients). If this command uses any other ID, it has the effect of returning pending entries of that client.

### In what technical aspects does Redis Streams differ from other Redis data types?

Unlike other Redis blocking commands that specify timeouts in seconds, commands `XREAD` and `XREADGROUP` specify timeouts in milliseconds. Another difference is that when blocking on list pop operations, the first client will be served when new data arrives. With Stream `XREAD` command, every client blocking on the stream will get the new data.

When an aggregate data type is emptied, its key is automatically destroyed. This is not the case with Stream data type. The reason for this is to preserve the state associated with consumer groups. Stream is not deleted even if there are no consumer groups but this behaviour may be changed in future versions.

### How does Redis Streams compare against Kafka?

Apache Kafka is a well-known alternative for Redis Streams. In fact, some features of Streams such as consumer groups have been inspired by Kafka.

However, Kafka is said to be difficult to configure and expensive to operate on typical public clouds. Streams is therefore a better option for small, inexpensive apps.

### Could you share some performance numbers on Redis Streams?

In one test on a two-core machine with multiple producers and consumers, messages were generated at 10K per second. With `COUNT 10000` given to `XREADGROUP` command, every iteration processed 10K messages. It was seen that 99.9% requests had a latency of less than 2 ms. Real-world performance is expected to be better than this.

When compared against traditional Pub/Sub messaging, Streams gives 100x better throughput. It's able to handle more than 1 million operations per second. If Pub/Sub messages are persisted to network storage, latency is about 5 ms. Streams has less than 1 ms latency.

### Could you share some developer tips for using Redis Streams?

There are dozens of Redis clients in various languages. Many of these have support for Streams.

Use `XREAD` for 1-to-1 or 1-to-n messaging. Use `XRANGE` for windowing-based stream processing. Within a consumer group, if a client fails temporarily, it can reread messages from a specific ID. For permanent failures, other clients can claim pending messages.

For real-time streaming analytics, one suggestion is to pair up Redis Streams with Apache Spark. The latter has the feature Structured Streaming that pairs up nicely with Streams. To scale out, multiple Spark jobs can belong to a single consumer group. Since Streams is persistent, even if a Spark job restarts, it won't miss any data since it will start consuming from where it left off.

## Milestones

Sep
2017

Salvatore Sanfilippo, creator of Redis, gives a demo of Redis Streams and explains the API. In October, he blogs about it. He explains that the idea occurred much earlier. He tinkered with implementing a generalization of sorted sets and lists but was not happy with the results. When Redis 4.0 came out with support for modules, Timothy Downs created a data type for logging transactions. Sanfilippo used this as an inspiration to create Redis Streams.

May
2018

The first release candidate RC1 of Redis 5.0 is released. This supports **Stream** data type.

Jul
2018

Beta version of Redis Enterprise Software (RS) 5.3 is released. This is based on Redis 5.0 RC3 with support for Stream data type.

Oct
2018

Redis 5.0.0 is released.

May
2019

At the Redis Conference 2019, Dan Pipe-Mazo talks about Atom, a microservices SDK powered by Redis Streams. Microservices interact with one another using Streams.

Redis Streams


# Machine Learning Model

## Summary

In traditional programming, a function or program reads a set of input data, processes them and outputs the results. Machine Learning (ML) takes a different approach. Lots of input data and corresponding outputs are given. ML employs an algorithm to learn from this dataset and outputs a "function". This function or program is what we call an **ML Model**.

Essentially, the model encapsulates a relationship or pattern that maps the input to the output. The model learns this automatically without being explicitly programmed with fixed rules or patterns. The model can then be given unseen data for which it predicts the output.

ML models come in different shapes and formats. Model metadata and evaluation metrics can help compare different models.

## Discussion

### Could you explain ML models with some examples?

Consider a function that reads Celsius value and outputs Fahrenheit value. This implements a simple mathematical formula. In ML, once the model is trained on the dataset, the formula is implicit in the model. It can read new Celsius values and give correct Fahrenheit values.

Let's say we're trying to estimate house prices based on attributes. It may be that houses with more than two bedrooms fall into a higher price bracket. Areas 8500 sq.ft. and 11500 sq.ft are important thresholds at which prices tend to jump. Rather than encode these rules into a function, we can build a ML model to learn these rules implicitly.

In another dataset, there are three species of irises. Each iris sample has four attributes: sepal length/width, petal length/width. An ML model can be trained to recognize three distinct clusters based on these attributes. All flowers belonging to a cluster are of the same species.

In all these examples, ML saves us the trouble of writing functions to predict the output. Instead, we train an ML model to implicitly learn the function.

### What are the essentials that help an ML model learn?

There are many types (aka shapes/structures/architectures) of ML models. Typically, this **structure** is not selected automatically. The data scientist pre-selects the structure. Given data, the model learns within the confines of the chosen structure. We may say that the model is fine-tuning the parameters of its structure as it sees more and more data.

The model learns in iterations. Initially, it will make poor predictions, that is, predicted output deviate from actual output. As it sees more data, it gets better. Prediction error is quantified by a **cost/loss function**. Every model needs such a function to know how well it's learning and when to stop learning.

The next essential aspect of model training is the **optimizer**. It tells the model how to adjust its parameters with each iteration. Essentially, the optimizer attempts to minimize the loss function.

If results are poor, the data scientist may modify or even select a different structure. She may pre-process the input differently or focus on certain aspects of the input, called features. These decisions could be based on experience or analysis of wrong predictions.

### What possible structures, loss functions and optimizers are available to train an ML model?

Classical ML offers many possible model structures. For example, Scikit-Learn has model structures for regression, classification and clustering problems. Some of these include linear regression, logistic regression, Support Vector Machine (SVM), Stochastic Gradient Descent (SGD), nearest neighbour, Guassian process, Naive Bayes, decision tree, ensemble methods, k-Means, and more.

For building neural networks, many architectures are possible: Feed-Forward Neural Network (FFNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Gated Recurrent Unit (GRU), Long Short Term Memory (LSTM), Autoencoder, Attention Network, and many more. In code, these can be built using building blocks such as convolution, pooling, padding, normalization, dropout, linear transforms, non-linear activations, and more.

TensorFlow supports many loss functions: BinaryCrossentropy, CategoricalCrossentropy, CosineSimilarity, KLDivergence, MeanAbsoluteError, MeanSquaredError, Poisson, SquaredHinge, and more. Among the optimizers are Adadelta, Adagrad, Adam, Adamax, Ftrl, Nadam, RMSprop, and SGD.

### What exactly is saved in an ML model?

ML frameworks typically support different ways to save the model:

    + **Only Weights**: Weights or parameters represent the model's current state. During training, we may wish to save checkpoints. A checkpoint is a snapshot of the model's current state. A checkpoint includes model weights, optimizer state, current epoch and training loss. For inference, we can create a fresh model and load the weights of a fully trained model.
    + **Only Architecture**: Specifies the model's structure. If it's a neural network, this would be details of each layer and how they're connected. Data scientists can share model architecture this way, with each one training the model to suit their needs.
    + **Complete Model**: This includes model architecture, the weights, optimizer state, and a set of losses and metrics. In PyTorch, this is less flexible since serialized data is bound to specific classes and directory structure.In Keras, when saving only weights or the complete model, `*.tf` and `*.h5` file formats are applicable. YAML or JSON can be used to save only the architecture.

### Which are the formats in which ML models are saved?

**Open Neural Network Exchange (ONNX)** is open format that enables interoperability. A model in ONNX can be used with various frameworks, tools, runtimes and compilers. ONNX also makes it easier to access hardware optimizations.

A number of ML frameworks are out there, each saving models in its own format. TensorFlow saves models as protocol buffer files with `*.pb` extension. PyTorch saves models with `*.pt` extension. Keras saves in HDF5 format with `*.h5` extension. An older XML-based format supported by Scikit-Learn is Predictive Model Markup Language (PMML). SparkML uses MLeap format and files are packaged into a `*.zip` file. Apple's Core ML framework uses `*.mlmodel` file format.

In Python, Scikit-Learn adopts pickled Python objects with `*.pkl` extension. Joblib with `*.joblib` extension is an alternative that's faster than Pickle for large NumPy arrays.  If XGBoost is used, then a model can be saved in `*.bst`, `*.joblib` or `*.pkl` formats.

With some formats, it's possible to save not just models but also pipelines composed of multiple models. Scikit-Learn is an example that can export pipelines in Joblib, Pickle, or PMML formats.

### What metadata could be useful along with an ML model?

Data scientists conduct multiple experiments to arrive at a suitable model. Without metadata and proper management of such metadata, it becomes difficult to reproduce the results and deploy the model into production. ML metadata also enables us to do auditing, compare models, understand provenance of artefacts, identify reusable steps for model building, and warn if data distribution in production deviates from training.

To facilitate these, metadata should include model type, types of features, pre-processing steps, hyperparameters, metrics, performance of training/test/validation steps, number of iterations, if early stopping was enabled, training time, and more.

A saved model (also called exported or serialized model), will need to be deserialized when doing predictions. Often, the versions of packages or even the runtime will need to be the same as those during serialization. Some recommend saving a reference to an immutable version of training data, version of source code that trained the model, versions of libraries and their dependencies, and the cross-validation score. For reproducible results across platform architectures, it's a good idea to deploy models within containers, such as Docker.

### Which are some useful tools when working with ML models?

There are tools to **visualize** an ML model. Examples include *Netron* and *VisualDL*. These display the model's computational graph. We can see data samples, histograms of tensors, precision-recall curves, ROC curves, and more. These can help us optimize the model better.

Since ONNX format aids interoperability, there are **converters** that can convert from other formats to ONNX. One such tool is *ONNXMLTools* that supports many formats. It's also a wrapper for other converters such as keras2onnx, tf2onnx and skl2onnx. ONNX GitHub code repository lists many more converters. Many formats can be converted to Apple Core ML's format using *Core ML Tools*. For Android, `tf.lite.TFLiteConverter` converts a Keras model to TFLite.

Sometimes converters are not required. For example, PyTorch can natively export to ONNX.

ONNX models themselves can be simplified and there are **optimizers** to do this. *ONNX Optimizer* is one tool. *ONNX Simplifier* is another, built using ONNX Optimizer. It basically looks at the whole graph and replaces redundant operators with their constant outputs. There's a ready-to-use online version of ONNX Simplifier.

## Milestones

1952

At IBM, Arthur Samuel writes the first learning program. Applied to the game of checkers, the program is able to learn from mistakes and improve its gameplay with each new game. In 1959, Samuel popularizes the term **Machine Learning** in a paper titled *Some Studies in Machine Learning Using the Game of Checkers*.

1986

Rumelhart et al. publish the method of **backpropagation** and show how it can be used to optimize the weights of neurons in artificial neural networks. This kindles renewed interest in neural networks. Although backpropagation was invented in the 1960s and developed by Paul Werbos in 1974, it was ignored back then due to the general lack of interest in AI.

1990

In this decade, ML shifts from a knowledge-driven to a **data-driven approach**. With the increasing use of statistics and neural networks, ML tackles practical problems rather than lofty goals of AI. Also during the 1990s, Support Vector Machine (SVM) emerges as an important ML technique.

2006

Hinton et al. publish a paper showing how a network of many layers can be trained by smartly initializing the weights. This paper is later seen as the start of **Deep Learning** movement, which is characterized by many layers, lots of training data, parallelized hardware and scalable algorithms. Subsequently, many DL frameworks are released, particularly in 2015.

Jun
2016

Vartak et al. propose *ModelDB*, a system for **ML model management**. Data scientists can use this to compare, explore or analyze models and pipelines. The system also manages metadata, quality metrics, and even training and test data. In general, from the mid-2000s we see interest in ML model management and platforms. Examples include *Data Version Control (DVC)* (2017), Kubeflow (2018), ArangoML Pipeline (2019), and *TensorFlow Extended (TFX)* (2019 public release).

Sep
2017

Microsoft and Facebook come together to announce **Open Neural Network Exchange (ONNX)**. This is proposed as a common format for ML models. With ONNX, we obtain framework interoperability (developers can move their models across frameworks) and shared optimizations (hardware vendors and others can target ONNX for optimizations).

Jul
2019

While there are tools to convert from other formats to ONNX, one ML expert notes some limitations. For example, *ATen* operators in PyTorch are not supported in ONNX. This operator is not standardized in ONNX. However, it's possible to still export to ONNX by updating PyTorch source code, which is something only advanced users are likely to do.

Mar
2020

In an image classification task, a performance comparison of ONNX format with PyTorch format shows that ONNX is faster during inference. Improvements are higher at lower batch sizes. On another task, ONNX shows as much as 600% improvement over Scikit-Learn. Further improvements could be obtained by tuning ONNX for specific hardware.

Machine Learning Model


# MIOTY

## Summary

MIOTY is a software-based LPWAN protocol that's targeted at IoT applications. In particular, many industrial IoT applications demand high reliability, scalability, power efficiency and mobility. High data rate is not really needed. MIOTY meets these requirements while also being an ETSI standard, TS 103 357. In addition, the MIOTY Alliance promotes the technology towards better interoperability among vendors of both endpoints and base stations.

MIOTY is often written as mioty®. Fraunhofer-Gesellschaft owns the registered trademark. Patents owned by Fraunhofer-Gesellschaft and Diehl Metering GmbH are licensed via Sisvel International.

## Discussion

### Given many other LPWAN protocols, why do we need MIOTY?

LPWAN protocols are many. Some operate in unlicensed spectra: LoRa, Sigfox, and ZigBee. Others (from the cellular world) operate in licensed spectra: LTE-M, EC-GSM and NB-IoT. ZigBee, LoRa and NB-IoT offer about 250kbps. LTE-M offers 1Mbps. Sigfox offers 100bps.

In license-free spectra, where many technologies coexist, interference causes packet loss. Cellular technologies have higher power consumption. ZigBee has a mesh topology. Though ZigBee's relay nodes extend coverage, they have lower power efficiency.  Some protocols are not standardized, leading to vendor lock-in or interoperability issues.

While these protocols have their niche applications, there are some applications that need low data rate, higher power efficiency and long range. Packet loss should be very low even in the face of interference. A single base station should be able to handle thousands of endpoints. A star topology, rather than a mesh topology, is more suited to meet these requirements. This is the space that MIOTY addresses.

### What are the use cases for MIOTY?

MIOTY is being seen as a "low-throughput tech for last mile industrial communications". In smart grids, gas and water meters can use MIOTY. In agriculture, soil sensors and irrigation controls can use MIOTY. In smart factories and buildings, asset tracking can benefit from MIOTY. Remote sensors need long range that MIOTY provides. MIOTY can handle assets moving up to 120kph, thus making it suitable for fleet management or vehicle-to-infrastructure communications.

MIOTY is designed to support **massive IoT**, where 100,000+ endpoints can be supported by a single base station handling 1.5 million messages per day. This requirement is common in smart metering in dense urban deployments or monitoring in smart factories.

MIOTY Alliance website shares news on recent applications. In November 2021, a project was initiated to automate and digitize RAG Austria's oil fields using Diehl Metering's MIOTY gateway. In January 2022, sensors in a MIOTY network detected in Germany the pressure wave triggered by an underwater eruption near Tonga.

### What are the main technical details of MIOTY?

MIOTY is based on a transmission technique called **Telegram Splitting Multiple Access (TSMA)**. A telegram or packet is split into smaller packets. These are sent slowly over a longer time period. For better resilience against interference, frequency hopping is used. Data is spread across 24 uplink or 18 downlink radio bursts. System uses 24 frequencies plus 1 for Sync-burst.

MIOTY operates in the sub-GHz range in license-free bands 868 MHz (Europe) and 915 MHz (US). Symbol rate is 2,380,371 symbols per second. Standard carrier spacing is 2,380,371Hz and channel bandwidth is 100kHz. Range is about 5km (non-LOS) and 20km (LOS). Even if 50% of the sub-packets are lost, the original information can be recovered.

Coherent MSK or GMSK demodulation is used. Receiver sensitivity is at -139dBm. Data rate is about 400bps. 10 bytes of application data can be sent in about 400ms. At the endpoint, duty cycle can be as low as 0.1% with a power consumption of 17.8μWh per message. This means that batteries could last for 20+ years. Messages are encrypted and integrity protected.

### Which are the different MIOTY device classes?

MIOTY defines three device classes:

    + **Class Z**: Unidirectional and uplink only. For monitoring applications. Very high energy efficiency.
    + **Class A**: Bidirectional so that endpoints can be configured via downlink unicast messages. Communication is initiated by endpoints.
    + **Class B**: Bidirectional. Suits low latency applications. Enables control of actuators at endpoints. Supports both unicast and broadcast messages. In broadcast mode, base station sends a periodic beacon signal. This indicates the timeslots when an endpoint needs to receive.For the TS-UNB protocol family, the ETSI standard mentions classes Z and A but not class B.

### What's the architecture of a typical MIOTY-based network?

The MIOTY system is one realization of a **Low Throughput Network (LTN)** standardized by ETSI. MIOTY is defined in the standard as TS-UNB and it specifies the air interface between endpoints and base station. In the standard, this interface is called *Interface A*.

One or more endpoints connect via the air interface to a base station (aka access point). The system can have multiple base stations but only one Service Center (SC). SC forwards/aggregates/deduplicates data, authenticates and configures endpoints, manages base stations, and coordinates roaming to other SCs. In practice, some of these functions are done by a separate entity called the Application Center. Application Center may use various protocols (MQTT, REST, COAP) to interface to applications. In fact, the ETSI standard identifies Registration Authority (RA) that stores identifiers and credentials of endpoints. In MIOTY, these RA functions are part of the Application Center. Finally, an IoT platform handles visualization and analytics.

### What's the current MIOTY ecosystem?

The MIOTY Alliance was formed in 2019. By 2022, it acquired 10 full members and many associated members. Members include chipset vendors, hardware manufacturers, software stack providers and application solution providers. It's goal is "to enable the most accessible, robust and efficient Massive IoT connectivity solution on the market".

MIOTY chipsets are being made by Radiocrafts, Silicon Labs, Texas Instruments, STMicroelectronics (STM32WL SoC series), etc. TI's CC1310 wireless MCU includes RF transceiver and Arm® Cortex®-M3 MCU. More powerful ones include CC1312R and CC1352R, the latter capable of BLE connectivity as well.

Gateways are available from BehrTech (called MYTHINGS), Swissphone, Deihl Metering, WEPTECH AVA and AST-X. Some of these vendors also offer endpoint hardware plus development kits. Others such as Radiocrafts and Sentinum offer only endpoints. ResIOT plans to offer a complete MIOTY network solution that includes base station, Service Center, and Application Center.

For the software stack, STMicroelectronics has partnered with Stackforce, which offers a multi-stack solution supporting MIOTY, LoRaWAN, Wireless M-Bus and Sigfox.  BehrTech and Swissphone have their own stacks.

## Milestones

Sep
2011

Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. files a German patent that introduces the concept of **telegram splitting**. A packet or telegram is split into smaller sub-packets that are sent over a much longer time period. In addition, sub-packets may be duplicated and sent on multiple frequencies (frequency hopping). The Fraunhofer Institute started this research in 2009.

Dec
2015

Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. apply for a German **trademark on MIOTY**. International and U.K. trademarks are filed in May 2016. Subsequently, these applications are approved and registered.

Jun
2018

ETSI standardizes the telegram splitting method of transmission in **TS 103 357: Protocols for radio interface A**.  It's named Telegram Splitting Ultra Narrow Band (TS-UNB). It's the protocol used in MIOTY.

Jul
2018

BehrTech claims to be the first to offer a MIOTY solution that's compliant with the ETSI standard. In March 2021, in partnership with WEPTECH, BehrTech claims first low-cost MIOTY gateway.

Nov
2019

**MIOTY Alliance** is formed. It's founding members include Fraunhofer Institute for Integrated Circuits ISS, Texas Instruments, Diehl Metering, Diehl Connectivity Solutions, ifm, Ragsol, Stackforce and Wika. In February 2020, the Alliance is announced officially at the Embedded World 2020 in Nuremberg. The Alliance aims to provide an open, standardized and interoperable ecosystem for MIOTY that can suit industrial IoT and smart city use cases. By mid-2022, the Alliance has 10 full members and 25 associate members.

Apr
2020

It's announced that Sisvel International will manage the licensing of MIOTY.

May
2021

Bernhard, Schlicht and Kilian of the Fraunhofer Institute for Integrated Circuits IIS are awarded the **Joseph von Fraunhofer Prize 2021** for their invention and development of MIOTY. In other news, **MIOTY Class B** is released to complement earlier classes Z and A.

Jul
2021

Fraunhofer IIS completes a successful test of MIOTY with a **GEO satellite** at a distance of about 38,000 km. This is in the S band at about 2 GHz. This test shows that MIOTY can be used in scenarios that are not dependent on any terrestrial infrastructure.

Oct
2021

At the Mobile Breakthrough Awards, BehrTech wins **Industrial IoT Solution of the Year** award for its MIOTY-BLE dual stack. The solution uses BLE 5.2 and CC1352R wireless microcontroller from Texas Instruments. Their dual stack was first announced in May 2021.

May
2022

At Fraunhofer IIS, researchers achieve up to **3.5 million telegrams per day per base station**. This is due to improved algorithms that can now run on ARM processors. For developers, Fraunhofer IIS also offers a MIOTY evaluation kit.

MIOTY


# CSS Selectors

## Summary

Cascading Style Sheets (CSS) are commonly used to style web content. To style a particular element on a HTML/XHTML page, we have to first select the element. This is done using CSS Selectors. A selector essentially selects or targets one or more elements for styling.

CSS3 specifications standardized by W3C defines a wide variety of selectors. A selector can target a single specific element or target multiple elements. In a good web design, CSS selectors are written to balance clarity, portability, stylesheet size, ease of update, and performance.

## Discussion

### Which are the commonly used CSS selectors?

A web page is essentially a set of HTML/XHTML tags organized into a hierarchy. It's common to mark elements with identifiers and/or class names. It's therefore common to use the following as CSS selectors:

    + **By Tag Name**: For instance, a paragraph can be selected with `p`, such as in `p { color: #444; }` .
    + **By Identifier**: An HTML element can have the `id` attribute that must be unique across the document. Thus, identifier can be used to target a specific element. For example, given the HTML snippet `<div id="footer"></div>`, the relevant selector is `#footer`.
    + **By Class Name**: The `class` attribute is also common for HTML elements. An element can have one or more classes. Multiple elements can belong to a class. By using a class-based CSS selector, we can target multiple elements. For example, given the HTML snippet `<span class="pincode"></span>`, the relevant selector is `.pincode`.The above three selector syntaxes can be combined into a single selector. Suppose we wish to target list items of class `icon` inside the footer, we can write `#footer li.icon`.

### How can HTML attributes be used in CSS selectors?

HTML elements can have attributes and these can be used in CSS selectors. The simplest syntax is to select by **presence of the attribute**. For example, `a[title]` targets all anchor elements with `title` attribute. A more refined selection is to match the **value of the attribute**. This comes in three variants (also explained in the figure):

    + `[attr=value]`: exact match: `div[class="alert"]` will match `<div class="alert"></div>` but not `<div class="alert-info"></div>`.
    + `[attr~=value]`: word match: `div[class~="alert"]` will match `<div class="alert-info alert"></div>` but not `<div class="alert-info"></div>`.
    + `[attr|=value]`: exact or with hyphen match: `div[class|="alert"]` will match `<div class="alert"></div>`, `<div class="alert-info"></div>` and `<div class="alert-info alert"></div>` but not `<div class="alertinfo"></div>` or `<div class="alert alert-info"></div>`.It's possible to match substrings of attribute values. These take the form `[attr^=value]`, `[attr$=value]` and `[attr*=value]` to match start, end and any part of the value string respectively.

HTML element names and attribute names are case insensitive but attribute values are case sensitive.  To match value strings in a case-insensitive manner, include the `i` flag. For example, `div[class="alert"]` will not match `<div class="Alert"></div>` but `div[class="alert" i]` will.

### What's the difference between pseudo-classes and pseudo-elements?

Pseudo-classes and pseudo-elements give additional capability to CSS selectors. Neither appears in the document source nor modifies the document tree.

**Pseudo-classes** are applied to elements due to their position (eg. last child) or state (eg. read only). Elements can gain or lose pseudo-classes as the user interacts with the document. The syntax is `:`, such as in `:enabled`, `:hover`, `:visited`, `:read-only`, `:last-child`, or `:nth-of-type()`.

Whereas pseudo-classes represent additional information about an element, **pseudo-elements** represent elements not explicitly present in the document tree. A pseudo-element is always bound to another element (called *originating element*) in the document. It can't exist independently. The syntax is `::`, such as in `::before`, `::after`, `::placeholder`, `::first-line`, or `::first-letter`.

In CSS1 and CSS2, both pseudo-classes and pseudo-elements used single colon syntax for `::before`, `::after`, `::first-line`, and `::first-letter`. Thus, browsers today accept either syntax for these special cases.

Where user actions are involved, the two can be combined. For example, `::first-line:hover` matches if the first line is hovered whereas `:hover::first-line` matches the first line of any originating element (such as second line of the paragraph) that's hovered.

### Which are the CSS selectors for targeting descendants of an element?

To select **descendants** use multiple selectors separated by spaces. For example, `p a` targets all links within paragraphs; links outside paragraphs are not selected. Selector `.agenda ul a` targets all links within unordered lists within elements of `agenda` class.

A more restrictive syntax selects only **immediate children** of an element, not grandchildren, great-grandchildren, etc. For example, `ul > a` is incorrect since children of `ul` are `li` elements. The correct selector would be `ul > li > a`. Another example is a list containing other lists. Selector `ul > li` selects only first level list items whereas `ul li` selects list items at every nested level.

Pseudo-classes provide further control for child selection. Some of these are `:first-child`, `:last-child`, `:only-child`, `:nth-child`, and `:nth-last-child` (nth child from the end). Children are one-indexed. To target every third list item from fourth item, use `ul > li:nth-child(3n+4)`. To target exactly the fourth item, use `ul > li:nth-child(4)`. To target the last but one item, use `ul > li:nth-last-child(2)`. To target last three items, use `ul > li:nth-child(-n+3)`

### Which are the CSS selectors for targeting siblings of an element?

Assume a DOM structure in which elements `h1 p p ul p` are all children of the same parent and therefore siblings in that order. CSS has two ways to select siblings by combining two selectors:

    + **Adjacent Sibling**: For example, `h1 + p` means that we target `p` if it's the immediate sibling following `h1`. Though `ul` is a sibling of `h1`, it's not the immediate sibling. Hence, `h1 + ul` will not select `ul`.
    + **General Sibling**: For example, `h1 ~ p` will select all three `p` elements that are all siblings of `h1`. The selector `p ~ h1` will not select `h1` because the way CSS works, we're targeting `h1` siblings that *follow* a `p` element.Combinator `+` targets only one sibling whereas `~` targets multiple siblings. With `~`, it's possible to obtain a specific sibling by using pseudo-classes such as `:first-of-type`, `:last-of-type`, `:nth-of-type()` or `:nth-last-of-type()`. For example, `h1 ~ p:nth-last-of-type(2)` will select the second last `p` sibling of `h1`.

### What are quantity selectors in CSS?

Given a number of children, it's possible to select one or more of them depending on the number of children. Quantity selectors help us do this. Essentially, they are pseudo-classes. Multiple pseudo-classes can be used together. We share a few examples.

To select a list item if it's the only item in the list, use `li:only-child`. This logic can be reversed by doing `li:not(:only-child)`.

To select the first child if there are exactly six children, use `li:nth-last-child(6):first-child`. This selects the sixth child from the end. This can be the first child only if there are exactly six children. To select children 2-6 if there are exactly six children, use the general sibling combinator, as in `li:nth-last-child(6):first-child ~ li`.

If there are six or more children, and we wish to ignore the last 5 children, use `li:nth-last-child(n+6)`. To select only the last 5 children given that there are six or more children, use `li:nth-last-child(n+6) ~ li`.

### Which are some uncommon CSS selectors?

Some CSS selectors are either not widely known or commonly used, but could be useful in some cases:

    + **Universal**: It's inefficient to select all elements by using the *universal selector* `*`. However, it could be used to select all children of an element, such as `.footer > *`.
    + **Empty**: Pseudo-class `:empty` targets elements with no content. HTML comments are not considered. Even if an element has only a single space, it's considered non-empty.
    + **Multiple Classes**: `p.last-section.section` targets paragraphs that belong to both classes `section` and `last-section`.
    + **Selector List**: Same CSS styles can be applied to multiple selectors, which are separated by commas. For example, `em, i {...}` styles `em` and `i` tags alike.
    + **Inverse Selection**: Specify a selector and target the inverse set. For example, `:not(p)` selects all elements that are not paragraphs. Another example is `.nav > div:not(:first-child)` to select all (expect the first) `div` children of `.nav`.

### Could you share some best practices for using CSS selectors?

To keep content separate from styling, avoid inline CSS styles. Put styles in stylesheets. An exception is when emailing HTML content since many mail clients ignore stylesheets.

Write legible CSS rulesets. Put each declaration on a separate line. Be careful about spaces. For example, selectors `#header.callout` (header of callout class) and `#header .callout` (callout descendants of header) target different elements.

Avoid repetitive rules. If many selectors contain the same style declaration, abstract that into a separate class-based ruleset.

Move animations to the end of stylesheets so that browsers load and render basic styling first.

CSS allows the use of `!important` in the value of a property. This gives precedence to the declaration, overriding even inline styles. **CSS specificity** determines selector precedence should multiple selectors target the same element. Since `!important` breaks this precedence order, it can make stylesheets hard to maintain and debug.

Long selectors resulting in high specificity, including use of identifiers, can break cascading and prevent efficient reuse of styles. Keep selectors to two or three levels deep. For example, replace `#header #intro h1.big a.normal` with `.big .normal`.

### What are some performance considerations when designing CSS selectors?

Selecting by identifier is fast. Selecting by class, tag, child, descendant, or attribute exact match are fast in that order. These results may vary across browsers. In fact, difference in speed between identifier and class selectors is negligible.

WebKit developer Dave Hyatt noted back in 2008 that it's best to avoid using sibling, descendant and child selectors for better performance.

The way browsers parse CSS selectors is different from the way web designers write them. Browsers read selectors in **right-to-left order**. For example, given `#social a` browsers will look at all `a` tags, move up the DOM tree and retain only those that are within `#social`. This insight can help us write more performant selectors. The rightmost selector is called the **key selector**.

The above example can be made more performant by adding a new class `social-link` to relevant anchor tags and then using the selector `#social .social-link`. In fact, this selector is *overqualified*. It can be simplified to just `.social-link`. A poor selector is `div:nth-of-type(3) ul:last-child li:nth-of-type(odd) *`. It's four levels deep and its key selector selects all elements of the DOM.

## Milestones

Aug
1999

W3C publishes *CSS3 module: W3C selectors*. In November 2018, W3C publishes this as *Selectors Level 3* W3C Recommendation.

Aug
2006

To simplify DOM tree traversal and manipulation John Resig releases a new library named **jQuery**. In JavaScript code, this allows us to select elements using CSS selectors, thus avoiding many lines of code that call `getElementById()` or `getElementsByClassName()`. jQuery itself is partly inspired by *cssQuery* (September 2005). An earlier effort to use CSS selectors in JavaScript is  `getElementsBySelector()` (March 2003).

Oct
2007

In a W3C Working Draft, new API methods `querySelector()` and `querySelectorAll()` are introduced. Browsers/clients must implement them for both `DocumentSelector` and `ElementSelector` interfaces. Traditionally, DOM elements were selected using methods `getElementById()`, `getElementsByClassName()`, `getElementsByTagName()`, `getElementsByName()`, etc. These are cumbersome. The new API methods allow direct use of CSS selectors. This becomes *Selectors API Level 1* W3C Recommendation in February 2013.

Mar
2009

In a study on CSS performance, it's noted that optimizing selectors matters only if a webpage has many thousands of DOM elements. For example, a Facebook page (in 2009) has 2882 CSS rules and 1966 DOM elements, a scale easily handled by browsers. Among a number of popular browsers, IE7 gives best performance. IE7 hits a performance limit only when a page has about 18K child/descendant rules.

Sep
2011

W3C publishes *Selectors Level 4* as a Working Draft. As on May 2020, it's still a Working Draft, with the latest version from November 2018. Among the new pseudo-classes are `:is()`, `:has()`, `:where()`, `:target-within`, `:focus-visible`, and `:focus-within`. Temporal pseudo-classes are `:current`, `:past` and `:future`. There are new pseudo-classes for input states and value checking. Grid pseudo-classes include `:nth-col()` and `:nth-last-col()`.

Jun
2014

Heydon Pickering proposes at a CSS conference a peculiar CSS selector of three characters that looks like an "owl's vacant stare". He calls it the **Lobotomized Owl**. This selector is `* + *`. It selects all elements that follow other elements. For example, this is useful when adding margin between two siblings without margin above the first element or below the last element. The alternative, `:not(:first-child):not(:root)` has high specificity and can't be easily overridden. The Lobotomized Owl has zero specificity.

CSS Selectors


# Tools to Manage Technical Debt

## Summary

In an ideal world, developers follow best practices and implement the best possible solution. In practice, this is rarely the case. Technical debt is often accepted to satisfy immediate business needs. However, this debt must be managed in the long term. Except for the most trivial projects, it's difficult to manage technical debt through manual effort. Fortunately, many tools exist for this purpose.

Engineering systems are diverse because of their choice of hardware, software programming language, development methodology, system architecture, test plan, and so on. As a result, technical debt is also rich in variety: architecture to code, documentation to testing. Tools to address them are also varied. A project might need to adopt more than one tool to manage its technical debt.

## Discussion

### What features are expected of tools that manage technical debt?

Good technical debt management tools are expected to have these traits:

    + **Polyglot**: Tools need to analyze the source code. As such, they must support popular programming languages (Java, JavaScript, C/C++, C#, Python, PHP, etc.). This support could be offered as plugins to the main product.
    + **Analysis**: Analysis must be from different perspectives including maintainability, reliability, portability and efficiency. Approaches can include static code analysis, SCM analysis, and test coverage. Tool should record historical data so that trends can be observed. Tool should show time needed to fix the debts.
    + **Reporting**: Dashboard should show a high-level summary and project status. Dashboard should be customizable since engineers and business folks are interested in different levels of detail. Visualizations should be clear.
    + **Deployment**: The tool can be hosted on-premise or in the cloud. For multiuser access, it should be a web application.
    + **Flexibility**: It should be possible to override default values and configure the tool to suit the project. For example, thresholds to detect duplicated code must be configurable.
    + **Logging**: We may wish to get insights into the analysis or troubleshoot issues. Tool should therefore collect logs for future study or audit.

### Could you mention some tools to manage technical debt?

Some tools only report code metrics, facilitate code reviews or support only a few languages. Others support multiple languages, cover multiple debt factors, perform risk analysis, visualize debt in different ways, and offer a useful dashboard summary. Among the latter are SonarQube, Squore, and Kiuwan.

Bliss, SonarQube, Checkstyle, and Closure Compiler are some tools that help with static code analysis. Designite can detect design smells, such as a class doing things that are not part of its core responsibility (poor cohesion). Jira Software and Hansoft are examples that identify but don't measure technical debt. Jacoco captures test debt.

Among other tools are CAST Application Intelligence Platform, Teamscale, SIG Software Analysis Toolkit, Google CodePro Analytix, Eclipse Metrics, Rational AppScan, CodeXpert, Redmine, Ndepend (Visual Studio plugin), DebtFlag (Eclipse plugin), CLIO, CodeVizard, and FindBugs.

In general, there are more tools for code/test debts than for design/architecture debts.

### How do tools quantify technical debt?

Tools should show metrics that are simple, clear, correct and objective. They should help in decision making. There are many metrics to quantify debt with respect to design, coding, defects, testing, documentation, and other aspects of product development. Tools should therefore measure code duplication, code complexity, test coverage, dependency cycles and coupling, lack of documentation, programming rules violations, and more.

Ideally, a few numbers that express overall product quality will be easier to understand at a high level. Expressing technical debt as in units of **person-days** points to the effort needed to remove the debt. This is a good measure but it doesn't indicate product quality. Therefore, we can also express **technical debt as a ratio**. Generally, debt that's higher than 10% needs to be addressed quickly.

Tool should also capture trends. For example, a declining trend in test coverage might suggest that with each iteration the team is adding more features that it's capable of testing.

In general, effort estimates are hard to get right. The suggested approach is to remove technical debt gradually (pay off the principal) with every release.

### What are some approaches that tools use to manage technical debt?

Given various aspects of managing technical debt, we note the following approaches:

    + **Identify**: Code analysis, dependency analysis, checklists; compare against an optimal solution.
    + **Measure**: Use mathematical models, source code metrics, manual estimation by experts; estimate based on debt type; use operational metrics; compare against an optimal solution.
    + **Prioritize**: Cost-benefit analysis; repay items with high remediation cost or interest payments; select a set of items that maximize returns (portfolio approach).
    + **Monitor**: Warn when thresholds are reached; track debt dependencies; regular measurements; monitor with respect to defined quality attributes; plot and visualize trends.
    + **Document**: Record each item in detail with identifier, type, description, principal and interest estimates; probability of interest payment; correlation with other items.
    + **Communicate**: Summarize in dashboards; record in backlog and schedule with each development cycle; list or visualize items.
    + **Prevent**: Improve processes; evaluate potential debts during architecture/design phases; create a culture that minimizes debt.
    + **Repay**: Refactor, rewrite, reengineer (new features); automate routine tasks; fix bugs; improve fault tolerance in areas of debt.

### What are some shortcomings of tools that manage technical debt?

Tools may differ in the way they quantify technical debt. The calculated metrics may be incomplete or even inaccurate. For example, a tool may fail to capture architectural debt, such as when code violates layered architecture.

There's no clear consensus on the dimensions or boundaries of technical debt. For example, some may consider known defects as technical debt. When tools report false positives, dealing with these can be tedious.

Many tools calculate the principal (cost of refactoring) but not the interest (extra cost due to technical debt). The latter is more difficult to quantify. However, it should be noted that the SQALE method has models to estimate both **remediation cost (principal)** and **non-remediation cost (interest)**.

## Milestones

1992

Ward Cunningham coins the term **Technical Debt** while working on WyCASH+, a financial software written in Smalltalk.

2001

**The Agile Manifesto** is signed. In the following years, the concept of technical debt is more widely adopted with the growth of the Agile movement.

Feb
2007

The first lines of code are written for the **Sonar platform**. In November 2008, SonarSource is founded to accelerate development and adoption of the open source Sonar platform. By mid-2010, SonarSource gets regular downloads. In April 2013, SonarSource gets a commercial edition and is renamed to SonarQube.

Sep
2009

In a white paper, Letouzey and Coq introduce the **SQALE (Software Quality Assessment Based on Lifecycle Expectations) Analysis Model**. They note that in qualimetry, any measure of software quality should be precise, objective and sensitive. The SQALE model provides normalized measures and aggregated indices. Since software is a hierarchy of artefacts, the model standardizes aggregation rules that lead to remediation indices. Since SQALE is open source and royalty free, by 2016, it's implemented by many tools.

2014

Multiple studies are published in literature that attempt to understand technical debt and relate it to software quality. A couple of these studies use SonarQube and its plugins for analysis.

Mar
2015

Li et al. bring together different studies and summarize the capabilities of 29 different technical debt management tools. They map them to various attributes: identification, measurement, prioritization, monitoring, repayment, documentation, communication and prevention. Most are able to identify or measure technical debt. In addition, they capture what sort of technical debts these tools capture: code, design (code smells or violations), testing, requirements, architecture, and documentation.

Apr
2015

In a blog post, Atlassian recognizes the importance of technical debt. The note that their **Jira** software can now list and track technical debt, including when a debt was created and when it's expected to be resolved.

Nov
2018

**DeepSource** is released. It's a source code analysis tool that can integrate with GitHub. Test coverage, documentation debt, security, and dependency debt are some things it can capture.

Oct
2019

Oswal studies the support for technical debt in three tools: SonarQube, PMD and Code Analytix. He notes that no single tool gives a holistic view and therefore he uses multiple tools for this analysis. The study looks at the Core Java 8 project plus a web application (Java/JavaScript/HTML/XML). The study addresses software reliability, maintainability and security.

Tools to Manage Technical Debt


# Apache Beam

## Summary

With the rise of Big Data, many frameworks have emerged to process that data. These are either for batch processing, stream processing or both. Examples include Apache Hadoop MapReduce, Apache Spark, Apache Storm, and Apache Flink.

The problem for developers is that once a better framework arrives, they have to rewrite their code. Mature old code gets replaced with immature new code. They may even have to maintain two codebases and pipelines, one for batch processing and another for stream processing.

Apache Beam aims to solve this problem by offering a **unified programming model** for both batch and streaming workloads that can run on any distributed processing backend execution engine. Beam offers SDKs in a few languages. It supports a number of backend execution engines.

## Discussion

### Why do I need Apache Beam when there are already so many data processing frameworks?

Having many processing frameworks is part of the problem. Developers have to write and maintain multiple pipelines to work with different frameworks. When a better framework comes along, there's significant effort involved in adopting it. Apache Beam solves this by enabling and reusing a single pipeline across multiple runtimes.

The benefit of Apache Beam is therefore both in development and operations. Developers can focus on their pipelines and less about the runtime. Pipelines become **portable**. Therefore there's no lock-in to a particular runtime. Beam SDKs allow developers to quickly integrate a pipeline into their applications.

The **Beam Model** offers powerful semantics for developers to think about data processing at higher level of abstractions. Concepts such as windowing, ordering, triggering and accumulation are part of the Beam model.

Beam has auto-scaling. It looks at current progress to dynamically reassign work to idle workers or scaling up/down the number of workers.

Since Apache Beam is open source, support for more languages (by SDK writers) or runtimes (by runner writers) can be added by the community.

### What are the use cases served by Apache Beam?

Apache Beam is suitable for any task that can be parallelized by breaking down the data into smaller parts, each part running independently. Beam supports a wide variety of use cases. The simplest ones are perhaps Extract, Transform, Load (ETL) tasks that are typically used to move data across systems or formats.

Beam supports batch as well as streaming workloads. In fact, the name *Beam* signifies a combination of "Batch" and "Stream". It therefore presents a unified model and API to define parallel processing pipelines for both types of workloads.

Applications that use multiple streaming frameworks (such as Apache Spark and Apache Flink) can adopt Beam to simplify the codebase. A single data pipeline written in Beam can address both execution runtimes.

Beam can be used for scientific computations. For example, Landsat data (satellite images) can be processed in parallel and Beam can be used for this use case.

IoT applications often require real-time stream processing where Beam can be used. Another use case is computing scores for users in a mobile gaming app.

### Which are the programming languages and runners supported by Apache Beam?

Apache Beam started with a Java SDK. By 2020, it supported Java, Go, Python2 and Python3. Scio is a Scala API for Apache Beam.

Among the main runners supported are Dataflow, Apache Flink, Apache Samza, Apache Spark and Twister2. Others include Apache Hadoop MapReduce, JStorm, IBM Streams, Apache Nemo, and Hazelcast Jet. Refer to the Beam Capability Matrix for more details.

Java SDK supports the main runners but other SDKs support only some of them. This is because the runners themselves are written in Java, which makes support for non-Java SDKs non-trivial. Beam's **portability framework** aims to improve this situation and enable full interoperability. This framework would define data structures and protocols that can match any language to any runner.

**Direct Runner** is useful during development and testing for execution on your local machine. Direct Runner checks if your pipeline conforms to the Beam model. This brings greater confidence that the pipeline will run correctly on various runners.

Beam's portability framework comes with **Universal Local Runner (ULR)**. This complements the Direct Runner.

### What are the essential programming abstractions in Apache Beam?

Beam provides the following abstractions for data processing:

    + `Pipeline`: Encapsulates the entire task including reading input data, transforming data and writing output. Pipelines are created with options using `PipelineOptionsFactory` that returns a `PipelineOptions` object. Options can specify for example location of data, runner to use or runner-specific configuration. A pipeline can be *linear* or *branching*.
    + `PCollection`: Represents the data. Every step of a pipeline inputs and outputs `PCollection` objects. The data can be *bounded* (eg. read from a file) or *unbounded* (eg. streamed from a continuous source).
    + `PTransform`: Represents an operation on the data. Inputs are one or more `PCollection` objects. Outputs are zero or more `PCollection` objects. A transform doesn't modify the input collection. I/O transforms read and write to external storage. Core Beam transforms include `ParDo`, `GroupByKey`, `CoGroupByKey`, `Combine`, `Flatten`, and `Partition`. Built-in I/O transforms can connect to files, filesystems (eg. Hadoop, Amazon S3), messaging systems (eg. Kafka, MQTT), databases (eg. Elasticsearch, MongoDb), and more.

### What's the Beam execution model?

Creating a pipeline doesn't imply immediate execution. The designated pipeline runner will construct a **workflow graph** of the pipeline. Such a graph connects collections via transforms. Then the graph is submitted to the distributed processing backend for execution. Execution happens asynchronously. However, some runners such as Dataflow support blocking execution.

### What are some essential concepts of the Beam model?

Data often has two associated times: **event time**, when the event actually occurred, and **processing time**, when the event was observed in the system. Typically, processing time lags event time. This is called *skew* and it's highly variable.

Bounded or unbounded data are grouped into **windows** by either event time or processing time. Windows themselves can be fixed, sliding or dynamic such as based on user sessions. Processing can also be time-agnostic by chopping unbounded data into a sequence of bounded data.

Data can arrive out of order and with unpredictable delays. There's no way of knowing if all data applicable to an event-time window have arrived. Beam overcomes this by tracking **watermarks**, which gives a notion of data completeness. When a watermark is reached, the results are materialized.

We can also materialize early results (before watermark is reached) or late results (data arriving after the watermark) using **triggers**. This allows us to refine results over time. Finally, **accumulation** tells how to combine multiple results of the same window.

### Could you point to useful developer resources to learn Apache Beam?

Apache Beam's official website contains quick start guides and documentation. The Overview page is a good place to start. There's an example to try out Apache Beam on Colab.

The Programming Guide is an essential read for developers who wish to use Beam SDKs and create data processing pipelines.

Visit the Learning Resources page for links to useful resources. On GitHub, there's a curated list of Beam resources and a few code samples.

Developers who wish to contribute to the Beam project should read the Contribution Guide. The code is on GitHub. The codebase also includes useful examples. For example, Python examples are in folder path `sdks/python/apache_beam/examples` and Go examples in `sdks/go/examples`.

## Milestones

Jul
2014

Jay Kreps questions the need to maintain and execute parallel pipelines, one for batch processing (eg. using Apache Hadoop MapReduce) and one for stream processing (eg. using Apache Storm). The batch pipeline gives exact results and allows data reprocessing. The streaming pipeline has low-latency and gives approximate results. This has been called **Lambda Architecture**. Kreps instead proposes, "a language or framework that abstracts over both the real-time and batch framework."

May
2015

At a Big Data conference in London, Google engineer Tyler Akidau talks about streaming systems. He introduces some of those currently used at Google: MillWheel, Google Flume, and Cloud Dataflow. In fact, Cloud Dataflow is based on Google Flume. These recent developments bring greater maturity to streaming systems, which so far have remained less mature compared to batch processing systems.

2016

In February, Google's Dataflow is accepted by Apache Software Foundation as an Incubator Project. It's named **Apache Beam**. The open-sourced code includes Dataflow Java SDK, which already supports four runners. There's plan to build a Python SDK. Google Cloud Dataflow will continue as a managed service executing on the Google Cloud Platform. Beam logo is also released in February.

Jan
2017

Apache Beam graduates from being an incubator project to a top-level Apache project. During the incubation period (2016), the code was refactored and documentation was improved in an extensible vendor-neutral manner.

May
2017

**Beam 2.0** is released. This is the first stable release of Beam under the Apache brand. It's said that Beam is at this point "truly portable, truly engine agnostic, truly ready for use."

Jun
2018

With the release of Beam 2.5.0, **Go SDK** is now supported. Go pipelines run on Dataflow runner.

Jul
2020

**Beam 2.23.0** is released. This release supports Twister2 runner and Python 3.8. It removes support for runners Gearpump and Apex.

Apache Beam