# ImageNet

## Summary

ImageNet is a large database or dataset of over 14 million images. It was designed by academics intended for computer vision research. It was the first of its kind in terms of scale. Images are organized and labelled in a hierarchy.

In Machine Learning and Deep Neural Networks, machines are trained on a vast dataset of various images. Machines are required to learn useful features from these training images. Once learned, they can use these features to classify images and perform many other tasks associated with computer vision. ImageNet gives researchers a common set of images to benchmark their models and algorithms.

It's fair to say that ImageNet has played an important role in the advancement of computer vision.

## Discussion

### Where is ImageNet useful and how has it advanced computer vision?

ImageNet is useful for many computer vision applications such as object recognition, image classification and object localization.

Prior to ImageNet, a researcher wrote one algorithm to identify dogs, another to identify cats, and so on. After training with ImageNet, the same algorithm could be used to identify different objects.

The diversity and size of ImageNet meant that a computer looked at and learned from many variations of the same object. These variations could include camera angles, lighting conditions, and so on. Models built from such extensive training were better at many computer vision tasks. ImageNet convinced researchers that large datasets were important for algorithms and models to work well. In fact, their algorithms performed better after they were trained with ImageNet dataset.

Samy Bengio, a Google research scientist, has said of ImageNet, "Its size is by far much greater than anything else available in the computer vision community, and thus helped some researchers develop algorithms they could never have produced otherwise."

### What are some technical details of ImageNet?

ImageNet consists of 14,197,122 images organized into 21,841 subcategories. These subcategories can be considered as sub-trees of 27 high-level categories. Thus, ImageNet is a well-organized hierarchy that makes it useful for supervised machine learning tasks.

On average, there are over 500 images per subcategory. The category "animal" is most widely covered with 3822 subcategories and 2799K images. The "appliance" category has on average 1164 images per subcategory, which is the most for any category. Among the categories with least number of images are "amphibian", "appliance", and "utensil".

As many as 1,034,908 images have been annotated with **bounding boxes**. For example, if an image contains a cat as its main subject, the coordinates of a rectangle that bounds the cat are also published on ImageNet. This makes it useful for computer vision tasks such as object localization and detection.

Then there's **Scale-Invariant Feature Transform (SIFT)** used in computer vision. SIFT helps in detecting local features in an image. ImageNet gives researchers 1000 subcategories with SIFT features covering about 1.2 million images.

Images vary in resolution but it's common practice to train deep learning models on sub-sampled images of 256x256 pixels.

### Could you explain how ImageNet defined the subcategories?

In fact, ImageNet did not define these subcategories on its own but derived these from WordNet. **WordNet** is a database of English words linked together by semantic relationships. Words of similar meaning are grouped together into a synonym set, simply called **synset**. Hypernyms are synsets that are more general. Thus, "organism" is a hypernym of "plant". Hyponyms are synsets that are more specific. Thus, "aquatic" is a hyponym of "plant".

This hierarchy makes it useful for computer vision tasks. If the model is not sure about a subcategory, it can simply classify the image higher up the hierarchy where the error probability is less. For example, if model is unsure that it's looking at a rabbit, it can simply classify it as a mammal.

While WordNet has 100K+ synsets, only the nouns have been considered by ImageNet.

### How were the images labelled in ImageNet?

In the early stages of the ImageNet project, a quick calculation showed that by employing a few people, they would need 19 years to label the images collected for ImageNet. But in the summer of 2008, researchers came to know about an Amazon service called **Mechanical Turk**. This meant that image labelling can be crowdsourced via this service. Humans all over the world would label the images for a small fee.

Humans make mistakes and therefore we must have checks in place to overcome them. Each human is given a task of 100 images. In each task, 6 "gold standard" images are placed with known labels. At most 2 errors are allowed on these standard images, otherwise the task has to be restarted.

In addition, the same image is labelled by three different humans. When there's disagreement, such ambiguous images are resubmitted to another human with tighter quality threshold (only one allowed error on the standard images).

### How are the images of ImageNet licensed?

Images for ImageNet were collected from various online sources. ImageNet doesn't own the copyright for any of the images. This has implication on how ImageNet shares the images to researchers.

For public access, ImageNet provides image thumbnails and URLs from where the original images were downloaded. Researchers can use these URLs to download the original images. However, those who wish to use the images for non-commercial or educational purpose, can create an account on ImageNet and request access. This will allow direct download of images from ImageNet. This is useful when the original sources of images are no longer available.

The dataset can be explored via a browser-based user interface. Alternatively, there's also an API. Researchers may want to read the API Documentation. This documentation also shares how to download image features and bounding boxes.

### What is the ImageNet Challenge and what's its connection with the dataset?

ImageNet Large Scale Visual Recognition Challenge (ILSVRC) was an annual computer vision contest held between 2010 and 2017. It's also called **ImageNet Challenge**.

For this challenge, the training data is a subset of ImageNet: 1000 synsets, 1.2 million images. Images for validation and test are not part of ImageNet and are taken from Flickr and via image search engines. There are 50K images for validation and 150K images for testing. These are hand-labeled with the presence or absence of 1000 synsets.

The Challenge included three tasks: image classification, single-object localization (since ILSVRC 2011), and object detection (since ILSVRC 2013). More difficult tasks are based upon these tasks. In particular, **image classification** is the common denominator for many other computer vision tasks. Tasks related to video processing, but not part of the main competition, were added in ILSVRC 2015. These were object detection in video and scene classification.

For more information, read the current state-of-the-art on image classification for ImageNet.

### What is meant by a pretrained ImageNet model?

A model trained on ImageNet has essentially learned to identify both low-level and high-level features in images. However, in a real-world application such as medical image analysis or handwriting recognition, models have to be trained from data drawn from those application domains. This is time consuming and sometimes impossible due to lack of sufficient annotated training data.

One solution is that a model trained on ImageNet can use it's weights as a starting point for other computer vision task. This reduces the burden of training from scratch. A much smaller annotated domain-specific training may be sufficient. By 2018, this approach was proven in a number of tasks including object detection, semantic segmentation, human pose estimation, and video recognition.

### How is Tiny ImageNet related to ImageNet?

Tiny ImageNet and its associated competition is part of Stanford University's CS231N course. It was created for students to practise their skills in creating models for image classification.

The Tiny ImageNet dataset has 100,000 images across 200 classes. Each class has 500 training images, 50 validation images, and 50 test images. Thus, the dataset has 10,000 test images. The entire dataset can be downloaded from a Stanford server.

Tiny ImageNet is a strict subset of ILSVRC2014. Labels and bounding boxes are provided for training and validation images but not for test images. All images have a resolution of 64x64. Since the average resolution of ImageNet images is 482x418 pixels, images in Tiny ImageNet might have some problems: object cropped out, too tiny, or distorted. It's been observed that with a small training dataset overfitting can occur. Data augmentation is usually done on the images to help models generalize better.

Similarly, *Imagenette* and *Imagewoof* are other subsets of ImageNet, created by fast.ai.

### What are the criticisms or shortcomings of ImageNet?

Though ImageNet has a large number of classes, most of them don't represent everyday entities. One researcher, Samy Bengio, commented that the WordNet categories don't reflect the interests of common people. He added, "Most people are more interested in Lady Gaga or the iPod Mini than in this rare kind of diplodocus".

Images are not uniformly distributed across subcategories. One research team found that by considering 200 subcategories, they found that the top 11 had 50% of the images, followed by a long tail.

When classifying people, ImageNet uses labels that are racist, misogynist and offensive. People are treated as objects. Their photos have been used without their knowledge. About 5.8% labels are wrong.

ImageNet lacks geodiversity. Most of the data represents North America and Europe. China and India are represented in only 1% and 2.1% of the images respectively. This implies that models trained on ImageNet will not work well when applied for the developing world.

Another study from 2016 found that 30% of ImageNet's image URLs are broken. This is about 4.4 million annotations lost. Copyright laws prevent caching and redistribution of these images by ImageNet itself.

## Milestones

1985

George A. Miller and his team at Princeton University start working on **WordNet**, a lexical database for the English language. It's really a combination of a dictionary and a thesaurus. This would enable applications in the area of Natural Language Processing (NLP).

2006

Fei-Fei Li at the University of Illinois Urbana-Champaign gets the idea for ImageNet. The prevailing conviction among AI researchers at this time is that algorithms are more important and data is secondary. Li instead proposes that lots of data reflecting the real world would improve accuracy. By now, WordNet itself is mature, with version 3.0 getting released in December.

2007

Fei-Fei Li meets Christiane Fellbaum of Princeton University, a WordNet researcher. Li adopts WordNet for ImageNet.

2008

In July, ImageNet has 0 images. By December, ImageNet reaches 3 million images categorized across 6000+ synsets. By April 2010, the count is 11 million images across 15,000+ synsets. This is impossible for a couple of researchers but is made possible via crowdsourcing on the Amazon's Mechanical Turk platform.

2009

ImageNet is presented for the first time at the Conference on Computer Vision and Pattern Recognition (CVPR) in Florida by researchers from the Computer Science Department, Princeton University.

May
2010

The first ever ImageNet Challenge is organized, along with the well-known image recognition competition in Europe called the *PASCAL Visual Object Classes Challenge 2010 (VOC2010)*.

2012

ImageNet becomes the world's largest academic user of Mechanical Turk. The average worker identifies 50 images per minute.
 The year 2012 also sees a big breakthrough for both Artificial Intelligence and ImageNet. **AlexNet**, a deep convolutional neural network, achieves top-5 classification error rate of 16% from the previous best of 26%. Their approach is adapted by many others leading to lower error rates in following years.

2015

The best human-level accuracy for classifying ImageNet data is 5.1% and GoogLeNet becomes the nearest neural network counterpart with 6.66%. PReLU-Net becomes the first neural network to surpass human-level of accuracy by achieving 4.94% top-5 error rate.

2017

This year witnesses the final ImageNet Competition. Top-5 classification error drops to 2.3% and the competition is now considered a solved problem. Subsequently, the competition is hosted at Kaggle.

May
2019

EfficientNet claims to have achieved top-5 classification accuracy of 97.1% and top-1 accuracy of 84.4% for ImageNet, dethroning it's predecessor GPipe (December 2018) by a meagre 0.1% in both top-1 and top-5 accuracies.

Jun
2019

ImageNet wins **Longuet-Higgins Prize** at CVPR 2019, a retrospective award that recognizes a CVPR paper for having significant impact and enduring relevancy on computer vision research over a 10-year period.

Jul
2019

**ImageNet-A** fools the best AI models 98% of the time, due to their over-reliance on colour, texture and background cues. Unlike adversarial attack in which images are modified, ImageNet-A has 7500 original images that have been handpicked from ImageNet. This shows that current AI models are not robust to new data.

ImageNet


# Domain-Driven Design

## Summary

Writing software involves software architects and programmers. They understand software concepts, tools and implementation details. But they may be disconnected from the business and hence have an incomplete understanding of the problem they're trying to solve. **Domain-Driven Design (DDD)** is an approach towards a shared understanding within the context of the domain.

Large software projects are complex. DDD manages this complexity by decomposing the domain into smaller subdomains. Then it establishes a consistent language within each subdomain so that everyone understands the problem (and the solution) without ambiguity.

DDD is object-oriented design done right. Among its many benefits are better communication, common understanding, flexible design, improved patterns, meeting deadlines, and minimizing technical debt. However, DDD requires domain experts, additional effort and hence best applied for complex applications.

## Discussion

### What do you mean by 'domain' in the context of domain-driven design?

Domain can be defined as "a sphere of knowledge, influence or activity." For example, *Accountancy* is a domain. An accountant is someone who knows this domain well. She is considered a domain expert. She's perhaps not a programmer and therefore can't build an accounting software. But she can advise developers on the intricacies and workings of the domain.

Consider the domain of air traffic. A developer might imagine that pilots decide on the route (a sequence of 3D points) to a destination. A domain expert might clarify that routes are pre-determined and each route is actually a ground projection of the air path.

To better manage complexity, a domain can be broken down into **subdomains**. In the e-commerce domain, *Payment*, *Offer*, *Customer* and *Shipping* are possible subdomains.

Domain is the business or problem to be solved. Model is the solution. Likewise, subdomains in the problem space are mapped to **bounded contexts** in the solution space.

### Could you explain the relevance of bounded contexts?

Consider the terms *Member* and *Payment* used in a country club. For some stakeholders the terms relate to club membership fees; for others, they're about tennis court booking fees. This disconnect is an indication that the domain is not really one and indivisible. There are subdomains hiding in there and they're best modelled separately. Bounded contexts are the solution.

When a model is proposed for a subdomain, it's applied only within the boundaries of the subdomain. These boundaries in the solution space define a bounded context. When teams understand the bounded contexts, it becomes clear what parts of the system have to consistent (within a bounded context) and what parts can develop independently (across bounded contexts). Bounded contexts therefore imply a clear **separation of concerns**.

Without bounded contexts, we'll end up with a single large complex model of many entities and relationships. Entities will get tightly coupled. The end result is often called a **Big Ball of Mud**.

Bounded contexts may overlap or may be neatly partitioned. Bounded contexts often relate to one another and this is captured in a **context map**.

### What is meant by the term Ubiquitous Language?

A developer might state that "a database was updated and triggered an SMTP service." A domain expert unfamiliar with such technical jargon will be left confused. **Ubiquitous Language (UL)** is an attempt to get everyone to use words well-understood within the bounded context. Using UL, the developer would now state that "the pizza was delivered and a coupon was sent to the customer."

UL must be consistent and unambiguous. It should evolve as understanding of the domain changes. A change in language implies a change to the model. In fact, the model is not just a design artifact used just to draw UML diagrams. Model is the backbone of the language. Within a bounded context, use the same language in diagrams, writing, speech and code.

To create an UL, have an open discussion, analyze existing documents, express the domain clearly, and define an agreed glossary. Glossary alone won't help. Use it consciously to arrive at a common understanding of the model.

### How should I implement Universal Language in my code?

Since documents can get outdated quickly, code is an enduring expression of Universal Language.

Adopting UL in **naming convention** leads to clean readable code. The purpose of variables, methods, classes and APIs become easier to see. We call this **intention-revealing interfaces**. Example names that follow UL are `TaskReservation`, `ReservationAttempt`, `IsFulfilled`, `BookSeats`, `Reservation`, and `Confirm`. In fact, there are tools to check if names in code follow the domain's defined vocabulary. *NDepend* is one such tool.

Without UL, a shared understanding is hard to achieve and teamwork suffers. Even among technical folks, one may refer to `coupon` in an API but call it `Discounts` in the backend. Another mismatch is when a checkout workflow is mapped to `RideCommerce` service. Perhaps, this was documented somewhere but the documentation was not read by everyone.

Any code refactoring mustn't happen without discussion using the UL. For example, discussion could involve these questions to clarify concepts: "When you say `User`, do you mean `Driver`?" or "Do you think a `Coupon` is applied to the `BookingAmount`, or is it added?"

### Could you describe some essential terms of DDD?

From a comprehensive online DDD glossary, we describe some essential terms:

    + **Entity**: An object that has attributes but primarily defined by an identity.
    + **Value Object**: An object with attributes but no identity.
    + **Aggregate**: A cluster of objects treated as a single unit. External references are restricted to only one member, called the *Aggregate Root*. A set of consistency rules applies within the aggregate's boundaries.
    + **Factory**: A mechanism to encapsulate and abstract away the details of creating a complex object. A factory ensures aggregates are initialized to a consistent state.
    + **Repository**: A mechanism to encapsulate storage, search and retrieval from a collection of objects. Its implementation is not a domain concern.
    + **Service**: A stateless functionality that renders its service via an interface, typically used when a workflow doesn't fit the current model.

### Could you explain entities and value objects?

Entities and value objects both follow principles of object-oriented design. They both encapsulate data (attributes) and behaviour (methods). The key difference is that an entity is distinguished by its **identity**, which must be unique within the system. On the other hand, value objects are descriptive with no conceptual identity.

When we say that two entities are the same, we mean that their identities match, with possibly different attributes. When we compare two value objects, we're only checking if their attributes match. Thus, entities use **identifier equality** and value objects use **structural equality**.

An entity has a **lifecycle**. Its form and content can change but not its identity. Identity is used to track the entity. Value objects are ideally **immutable**.

In a banking application, transactions are entities, each with a unique transaction number. The amount transacted is a value object. A cheque's date may differ from the date of clearing but entries are always reconciled not by date but by the cheque number. This is an example of comparing entities by identities rather than by attributes.

### Could you explain the concept of aggregates in DDD?

When a cluster of entities or value objects control a significant area of functionality, it's easier to abstract them into a single consistent unit. This is the **aggregate pattern**. One way to identify aggregates is to look at common transactions and the entities that get involved.

Since an aggregate is a cohesive unit, it's best to ensure consistency by using a single **aggregate root** for all updates. Changing a child entity independently will break consistency since the root is unaware of such direct updates.

As an example, consider two aggregates *Buyer* and *Order*. *Order* has as entities Order and OrderItem; and Address as a value object. However, all external interactions are via the Order entity, which is the root. This entity refers to the root of the *Buyer* by identity (foreign key) .

Keep an aggregate on one server and allow aggregates to be distributed among nodes. Within an aggregate, update synchronously. Across aggregate boundaries, update asynchronously. Often, NoSQL databases can manage aggregates better than relational databases.

### Could you share some tips for practising DDD?

Objects with hardly any behaviour represent poor application of object-oriented design. They're little more than procedural-style design. This anti-pattern is called **Anemic Domain Model**. Instead, put business logic in domain objects. Other DDD anti-patterns to avoid include repetitive data access objects (use repositories instead), fat service layers, and classes that frequently access other classes' data.

Adopt a **layered architecture** to avoid details of application, presentation or data persistence from creeping into the domain layer.

Analyze the problem domain before deciding if a concept should be an entity or a value object. Don't link a value object to an entity, which would require an identity for the value object. Instead, the value object can be inlined into the entity.

Difficulties in implementing an aggregate usually indicates a modelling problem. Instead, attempt to refine the model.

If an operation doesn't fit the current model, evolve the model. Only if that's not possible, consider introducing a service. Since services represent activities, use verbs rather than nouns in naming. Indeed, DDD is not for perfectionists. It's okay if the model can't handle some special cases. Use services rather than a leaky or confusing abstraction.

## Milestones

1960

Software engineers recognize from the late 1960s that procedural languages are inadequate in handling the growing complexity of software projects. When *Simula 67* is released in 1967, it becomes the first **object-oriented programming (OOP)** language. Concepts of OOP and OOD reach maturity in the early 1980s.

Sep
1997

Foote and Yoder observe that while there are many high-level software architectural patterns, what's really prevalent in industry is "haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle." They call this the **Big Ball of Mud**. Code is dictated by expediency than design. The mantra has been, "Make it work. Make it right. Make it fast." Popular practices include immediate fixes and quick prototypes. Programmers work without domain expertise. No thought is given to elegance and efficiency of code.

2003

Eric Evans publishes a book titled *Domain-Driven Design: Tackling Complexity in the Heart of Software*. Evans is credited for coining the term **Domain-Driven Design**. This is also called the "Blue Book".

2009

At QCon, Phil Wills presents a case study about how The Guardian website was almost completely rebuilt following the principles of DDD. He mentions some key points: domain experts got more involved; they kept the model flexible to changes even when deadlines were met; they focused on core subdomains while leveraging on off-the-shelf software for the rest.

May
2011

The term **microservices** gets discussed at a workshop of software architects near Venice, to describe a common architectural style that several of them were exploring at the time. An application is decomposed into loosely coupled services, each being self-contained and managing its own data. From the perspective of DDD, a microservice maps to a subdomain and bounded context.

2013

Vaughn Vernon publishes a book titled *Implementing Domain-Driven Design*. This is also called the "Red Book".

Sep
2017

Eric Evans notes at the *Explore DDD* conference that DDD remains relevant today, fourteen years after he coined the term. Many tools have come to support and adopt DDD. Though DDD is not about technology, it's not indifferent about technology. With technology's support, we can focus on building better models. He gives some examples. NoSQL databases make it easier to implement aggregates. With modern functional languages, it's easier to implement immutable value objects. Microservices have come to represent bounded contexts.

Jan
2019

A report published by InfoQ shows DDD in *Late Majority* stage of technology adoption. This is proof of its effectiveness in software development. This stage implies that more than half the software folks have adopted DDD and the skeptics are starting to adopt them too. Only a year earlier, DDD was in the *Early Majority* stage.

Domain-Driven Design


# DevOps Metrics

## Summary

DevOps encourages incremental changes and faster releases while also improving quality and satisfaction. But how do we know if DevOps is making an impact? How do we decide what needs to change? We need to measure and this is where DevOps metrics come in.

Metrics give insights into what's happening at all stages of the DevOps pipeline, from design to development to deployment. Metrics are objective measures. They strengthen the feedback loops that are essential to DevOps. Collecting metrics and displaying them via dashboards or scorecards should be automated. It's important to map these metrics to business needs.

## Discussion

### What factors make for a good DevOps metric?

A good DevOps metric must ideally be all of these:

    + **Obtainable**: A metric that can't be measured is useless.
    + **Reviewable**: It must be relevant to the business and stand up to scrutiny.
    + **Incorruptible**: It should be free from influence of teams and team members.
    + **Actionable**: It should suggest improvements to workflows, policies, incentives, tools, etc.
    + **Traceable**: It should be possible to trace the metric to root causes.

### What's the process of working with DevOps metrics?

A typical process involved identifying the metrics, putting in place methods to measure them, measuring and displaying them on dashboards, evaluating the metrics in terms status and trends, acting on the metrics to effect change, and continually assessing if the metrics are aligned to business goals.

Since DevOps is cross-functional (process, people, tools) and cross-teams (dev, ops, testing), metrics should not narrowly focus on only some parts of the value chain. Metrics should capture a holistic view of the entire value chain.

### What are some important DevOps metrics?

There are dozens of metrics spread across all phases of a DevOps pipeline. Some have attempted to group them into categories:

    + **Velocity**: lead time, change complexity, deployment frequency, MTTR
    + **Quality**: deployment success rate, application error rate, escaped defects, number of support tickets, automated test pass percentage
    + **Performance**: availability, scalability, latency, resource utilization
    + **Satisfaction**: usability, defect age, subscription renewals, feature usage, business impact, application usage and trafficAnother grouping can be host-based metrics, application metrics, network metrics, server pool metrics and external dependency metrics.

There are also metrics for application build cycles, metrics for application performance, metrics for delivery performance, metrics organized by infrastructure, system and team health, and metrics for building or running apps.

At a minimum, aim for more deployments per week, shorter lead time from code commit to deployment, lower failure rate in production, and shorter time to repair failures. Have metrics to measure these.

Likewise, a study from 2019 identified lead time, deployment frequency, mean time to restore (MTTR) and change fail percentage as key metrics.

### What are the important metrics in the world of microservices and serverless architectures?

For microservices, metrics to include are number of requests per second, number of failed requests per second, and distribution of request service times.

For serverless, the concern shifts from monitoring infrastructure to the application itself. Metrics include performance such as function runtime; scaling such as concurrency limits or memory limits; tracing event-triggered call flows across services or functions; and errors such as code bug, wrong invocation or function timeout.

For both microservices and serverless, it's important to instrument the code. **OpenTracing** provides a vendor-neutral API for distributed tracing. Observability is an important aspect, which means that communication across services and functions needs to be accessible. A single request must be correlated to the sequence of service calls that followed it. **Istio** is a tool that requires strong observability.

### Could you describe some DevOps metrics adopted from traditional engineering practices?

From traditional engineering, DevOps has adopted the following metrics:

    + **Mean Time To Detect (MTTD)**: This is the average time to discover a problem. It's an indication of how effective is your incident management tools and processes.
    + **Mean Time To Failure (MTTF)**: This is an indication of how long on average the system or a component can run before failing. This can suggest preventive maintenance. This metric relates to improving system uptime.
    + **Mean Time Between Failures (MTBF)**: This is the average time between failures. It's a measure of reliability and availability.
    + **Mean Time To Repair (MTTR)**: This is the average time to repair/resolve/recover after failure is detected. This metric relates to reducing system downtime. Code complexity is one aspect that affects MTTR.The goal is to reduce MTTD and MTTR while increasing MTTF and MTBF. DevOps is about incremental changes. If many changes are introduced at once, it will take longer to detect and fix issues.

### Are there DevOps metrics that one should avoid?

Teams transitioning to DevOps might end up adopting the wrong metrics. In fact, **traditional metrics** such as MTBF could be seen as irrelevant for DevOps where some failures are expected due to the speed of delivery. Look beyond such costs. Instead, improve total economic impact. Others to avoid are metrics that focus on business velocity at the expense of quality or culture; metrics that are optimized for one team and causing negative impact on others.

Avoid **conflict metrics** that promote individuals rather than teams or pit one team versus another. These include ranking individuals or teams based on failure metrics (broken builds, etc.), rewarding top performers who don't collaborate or having different standards for different teams.

Avoid **vanity metrics** that promote quantity or speed over quality: number of lines of code, number of deployments per week, number of bugs fixed, number of tests added.

Don't collect a specific metric just because it's easy. Don't use a metric that encourages negative behaviours. It's been said,

> Human beings adjust behavior based on the metrics they’re held against... What you measure is what you’ll get.

### Could you mention some best practices when using DevOps metrics?

For those new to DevOps metrics, start with metrics that are simpler to collect and manage. Get the momentum going. For better focus, don't apply too many metrics. Choose metrics aimed at broader organizational goals or process health issues. Measure fast to enable real-time feedback loops.

Because automated system-based metric collection is hard to do, you may want to start with surveys. In fact, both these are complementary. Surveys are good for metrics on culture or things outside the system.

Use metrics that suit your business model. Adopt **value stream mapping** in which each metric is mapped to business values.  For example, measuring website responsiveness becomes more useful if you can map it to business outcomes such as customer churn or abandoned shopping carts.

Metrics can also be role-based (business vs engineering): give teams the choice to customize their own dashboards. In fact, dashboards are essential for tracking all metrics in one place. Compare trends, not teams. Look for outliers. Measure lead time to production, not just completion.

Evolve your metrics as new technologies and tools enter your DevOps pipeline.

### Are there tools to help teams collect metrics for DevOps?

Many tools are available for various DevOps tasks. Some of these show metrics and even do real-time monitoring. We briefly mention some of them. In any case, there's a need to provide teams a single unified dashboard regardless of the tool that collects the metrics.

Nagios is widely used for IT infrastructure monitoring. Zabbix, Sensu and Prometheus are alternatives. Prometheus is for service monitoring. It's often used with the visualization and analytics of Grafana.

For application performance monitoring, there are New Relic, AppDynamics, Compuware and Boundary. For deeper integration, cross-platform data aggregation and monitoring, there's BigPanda and PagerDuty.

JIRA Software does issue and project tracking. Code Climate automates code review and analysis. OverOps detects bugs proactively. For build automation, there's Apache Ant. Jenkins is useful for continuous integration and delivery. Ansible, Chef and Puppet help with continuous deployment. Ganglia is for cluster and grid monitoring. Snort is for real-time security. For logging, we have Logstash. Monit does system monitoring and recovery.

Cloud providers offer their own monitoring tools: AWS CloudWatch from Amazon or StackDriver from Google.

## Milestones

2009

DevOps has its beginnings at the O'Reilly Velocity conference where John Allspaw and Paul Hammond present a talk titled *10+ Deploys a Day: Dev and Ops Cooperation at Flickr*. Even in these early days, the importance of metrics is realized. Some metrics identified include CPU load, memory usage, network throughput, and aggregated job queue.

2016

There's a growing realization among practitioners that we can end up collecting a lot of wrong DevOps metrics. It's important to relate metrics to business values, needs or outcomes. One such proposal is the value-based approach that measures how value flows through the DevOps pipeline.

Jul
2017

Gartner publishes a report titled *Data-Driven DevOps: Use Metrics to Guide Your Journey*. This report includes a pyramid of metrics for DevOps.

DevOps Metrics


# In-Memory Database

## Summary

An **In-Memory Database (IMDB)**, also called Main Memory Database (MMDB), is a database whose primary data store is the RAM. This is in contrast with traditional databases which keep their data in disk storage and use RAM as a buffer cache for frequently/recently used data.

IMDB has gained traction in recent times because of increasing speeds, reducing costs and growing size of RAM devices, combined with powerful multi-core processors. Since only RAM is accessed and there is no disk operation for an IMDB query, speeds are extremely high.

However, RAM storage is volatile. So there will be data loss when the device loses power. To overcome this, IMDB employ several techniques such as checkpoints, transaction logging, and non-volatile RAM.

Among commercially available IMDB is SAP HANA. Open source options are Redis, VoltDB, Memcached, and an extension of SQLite.

## Discussion

### What's the context for the growing interest in IMDB?

From 1995 to 2015, RAM got 6000 times cheaper. Prior to that, disk drives were the main option to store databases. Disks were great for sequential access but poor for random access since lot of time was spent in rotating the disk and seeking the exact location.

Meanwhile, computing power has increased via faster clocks and multi-core processors. Networking speeds have also gone up. However, disk access speeds have not gone up fast enough. In fact, they've become the bottleneck in computing systems. Worse still, the amount of data being generated has grown exponentially. There's also a need to analyse all this data (via Machine Learning) in almost real time.

This is where in-memory databases become suitable. They're now affordable enough to store large amounts of data, particularly compressed columnar data. They're fast enough for real-time analytics. We can continue to use disks where sequential access is desired, such as for logging. The notable database scientist Jim Gray once supposedly said,

> Memory is the new disk

### What are the key features of IMDB? How do they vary from disk-based RDBMS?

Disk access is sequential. In disk-based DBs, the seek time to locate records on the physical disk is the biggest contributor to query time. In an IMDB, since entire data is in memory, this burden is entirely eliminated.

While RAM is volatile, data persistence is achieved through multiple methods and it’s done very efficiently. These safeguard from data loss due to power failure. Since only a minority of DB operations are data change operations (about 10-15%), disk operations are minimal.

Most IMDB support columnar data storage, where table records are stored as a sequence of columns, in contiguous memory locations. This speeds up analytical querying greatly and minimizes CPU cycles. When data is stored in columnar form, column-wise compression techniques (such as sparse columns) are used to minimise memory footprint. Moreover, there's less dependence on indexing. IMDB delivers performance similar to having an index on every column, but with much less transactional overhead.

### What are the popular applications of IMDB?

IMDB works well for applications that require very fast data access, storage and manipulation. Real-time embedded systems, music and call databases in mobile phones, telecommunication access networks, programming data in set-top boxes, e-commerce applications, social media sites, equity financial services are the most prevalent applications of in-memory databases. IMDBs have also gained a lot of traction in the data analytics space.

### How do the performance characteristics compare between IMDB and traditional DB?

    + **CPU and Memory** - Accessing data in memory is much faster than writing to/reading from file systems. IMDB design is simpler than on-disk databases, so they have significantly lower memory/CPU requirements. Even if a machine hosting an RDBMS has enough memory on board to fit the entire data, IMDB would be faster. That's because it performs fewer copy operations and has more advanced in-memory data structures, optimized for working with memory. However, IMDB scales poorly to multiple CPU cores.
    + **Data Query and Update functions** - Applications requiring random data access under 1ms (such as online/real-time applications) benefit from IMDB. If access time over 100ms is acceptable, traditional RDBMS works fine. For sequential access, the difference is even more pronounced. Persistence operations don’t affect data update times in IMDB since they happen offline.
    + **Size constraints** - Generally, not more than 200-300GB RAM is installed on a machine in order to keep machine start-up time acceptable. So when DB size is in TB range supporting millions of transactions per second, memory sharding is done to partition one logical DB into multiple physical DBs.

### What are the ways in which persistence is supported in IMDB?

There are many methods by which IMDB might persist the data state on disk. End goal is to ensure complete recovery of data but without compromising on query speeds.

    + **Transaction Logs** - Each data update is applied to the IMDB and also on a transaction log on disk. Change entries done at the end of the append-only log file. When the file size rolls over, its contents are archived.
    + **Checkpoint Images** - Checkpoint files contain an image of the database on disk. Some IMDB use dual checkpoint files for additional safety, in case the system fails while a checkpoint operation is in progress. For recovery, the database checkpoint on disk is merged with the latest transaction log entries.
    + **High Availability** - To protect against memory outages in data centers, the data cluster is replicated asynchronously into a second read-only cluster. If outage occurs, hot swap gets triggered to configure the secondary as primary.
    + **Non-volatile RAM** - Using battery powered RAM devices or supercapacitors, all write operations can be persistent even after power loss. These are slightly slower than DRAM, but much faster than RDBMS disk operations.

### What are some commercial IMDB products?

Without being exhaustive, we describe three commercial IMDB products:

    + **SAP HANA** - In-memory, column-based data store from SAP. Available as local appliance or cloud service. The in-memory data is CPU-aligned, no virtual expensive calculation of LRU, logical block addresses, just direct (pointer) addressing of data. Supports server scripts in SQLScript, JSON and R formats. Good support for predictive analytics, spatial data processing, text analytics, text search, streaming analytics, graph data processing, and ETL operations. Guarantees microsecond response and extremely high throughput performance.
    + **Oracle TimesTen** - In-memory OLTP RDBMS acquired by Oracle. Guarantees microsecond response and extremely high throughput performance. Provides application level data cache for improved response time. High availability through replication.
    + **eXtremeDB** - Combines on-disk and in-memory data storage in a single embedded database system. It can be deployed as an embedded database system or elastically scalable client/server distributed database.

### What are some open source IMDB platforms?

Here are some open source IMDB platforms to consider:

    + **Apache Ignite** – Java-based middleware that forms an in-memory layer over any existing DB. Can work in a single or distributed environment. Seamless integration with MapReduce and Hadoop systems.
    + **Altibase** - Hybrid database that combines an in-memory database and an on-disk database into a single product to achieve the speed of memory and the storage capacity of disk.
    + **SQLite** - Instruct an SQLite database to exist purely in memory using the special filename `:memory:` instead of the real disk filename.
    + **Redis** - Key-value store based system with support for key data structures. Schema free. Data durability feature is optional. Good programming language support.
    + **VoltDB** - Traditional RDBMS with schema support. Works using Java Stored Procedures that applications can invoke through JDBC. The company is collaborating with Samsung for Scaling In-Memory Data Processing with Samsung DRAM/SSD devices.

### What are the use cases where an IMDB is not suitable?

 Volatile memory in affordable and servers with support for 24TB of RAM are now available. But IMDB cannot replace the traditional RDBMS in all scenarios. Following are some of the use cases where IMDB is not suitable when:

    + **Persistence is critical** - Applications with confidential/critical data undergoing frequent updates might be at risk in IMDB. Unless persistence features are in the IMDB, there's risk of data loss during power failure.
    + **Very small scale data** - Small and medium enterprises can simply run on low-cost server with acceptable performance.
    + **Memory-intensive applications** - When IMDB is used, the bulk of RAM is going be occupied by the DB itself. So if the application itself requires high memory (such as 3D games, live streaming), then memory costs will rise significantly.
    + **Very large scale data applications** - Memory requirements would be prohibitively expensive, hence not recommended.
    + **Non-mission-critical operations** - Backend operations or applications where data can be batch processed offline don't require millisecond query response times of IMDB.

## Milestones

1970

The term "relational database" is invented by E. F. Codd at IBM in 1970.

1978

IBM's IMS/VS FastPath is one of the earliest in-memory engines.

1980

In telecom and defence domains, some companies start using in-memory databases. However, these are internal to those who used them and generally not available for purchase.

1992

**Oracle 7** version supports data buffers where a snapshot of a data block would be taken from disk and stored in RAM for faster access.

1997

**TimesTen** releases its first version in-memory database. TimesTen is later acquired by Oracle.

2009

**Redis**, an in-memory data structure project, makes its first release with a BSD license. It becomes one of the most popular key-value store databases.

2012

Early versions of **SAP HANA** were present from 2005. In 2012, SAP promotes its commercial version for cloud based applications. HANA now includes a business suite of applications covering ERP and CRM domains under one umbrella.

2014

Oracle releases its **Oracle 12c** cloud RDBMS business suite with in-built support for in-memory DB operations.

In-Memory Database


# API Testing

## Summary

Applications today rely on APIs. Whether it's a web client requesting a service from a web application server, or one microservice requesting data or operation from another microservice, APIs play a key role. Via APIs, developers give others access to their service.

At the same time, organizations are embracing Agile methodology and making frequent product releases. It's therefore important to test these APIs.

API testing is useful to validate a solution and to find errors. API testing complements unit testing and end-to-end testing. It enables more efficient use of test resources. Problems can be caught earlier in the development cycle.

HTTP RESTful API is the most widely used architecture. However, this article describes API testing in general and is therefore relevant to other API types such as SOAP or GWT RPC.

## Discussion

### Do I need API testing for my application?

A web application typically consists of three layers: user interface, business logic and database. End-to-end testing would test all layers of the app but it's also slower. Problems are hard to isolate. Business logic may need many tests, for which we will end up unnecessarily exercising the UI in the same way. Moreover, end-to-end testing can begin only when all layers are available. API testing solves this problem by bypassing the UI. It executes tests at the service or business layer.

While unit tests are typically written by developers and take a white-box testing approach, API tests are usually written by the QA team and view the system under test (SUT) as a black box. However, not everyone agrees on this division of roles. Some feel that developers should do API testing since they created the APIs. Others say that since APIs specify a contract, they need to be validated by testers.

While unit tests exercise the business logic directly, API tests go through the API layer.

### What's the flow for a typical API test?

An API test first calls an API endpoint, which is really a URL. HTTP headers are set as required for the test. The type of HTTP request may be GET, POST, PUT, DELETE, etc. With each of these, the necessary data is sent to the API endpoint.

Once a response is received, the response code and contents are validated. HTTP headers that specify access control, content type, or server might be validated.

In a sequence of API calls, some parts of a response may be used for the next API call. For example, a POST request might return an identifier. A subsequent GET request might verify that the response includes this identifier.

API testing is generally black-box testing. We don't look at what happens behind the API server. We only validate the responses. But sometimes we may want to validate if an API request triggers another API request or updates the database. For example, an API request may trigger a request to Google Maps API. During testing, we could mock Google Maps API and validate the request made to the mocked API.

### What are the possible benefits of API testing?

Since data is exchanged in standard formats (XML, JSON), API testing is language agnostic. Any programming language can be used to create API tests. API responses can be easily validated since most languages have libraries to compare data in these formats.

End-to-end testing can't be done unless all parts of the application are ready. With API testing, the business logic can be tested early on even when the GUI is still under development. This also facilitates easier end-to-end testing at a later stage.

Because APIs are usually well specified, API testing leads to high test coverage. Moreover, UI changes often during development. API tests based on well-defined specifications are easier to maintain.

API testing is faster than UI testing. More tests can be performed in a shorter time. Releases can happen faster. When an API test fails, it's easier to find the source of failure. API testing also enables automation of CI/CD pipelines.

### What types of tests are possible at the API layer?

A wide variety of tests can be done at the API layer, both functional and non-functional:

    + Validation and functional tests ensure that APIs behave as desired and deliver specific functionalities.
    + Security and penetration tests would consider user authentication or authorization, threat detection, and data encryption.
    + Load testing checks app performance at normal and peak loading conditions, or if throttling is correctly applied at theoretical maximum load. For example, we may want to know how many API requests can be served per minute with a specific response time.
    + Tests can be designed to check for runtime errors, resource leaks and general monitoring of the app.
    + Fuzz tests include random data in API requests. App is expected to be robust against such requests.
    + UI events and interactions trigger API calls. Thus, UI testing is also an approach to API testing.
    + Since APIs interface various services, they play a key role in integration testing. However, they're also useful in end-to-end testing to validate dataflows across services.

### What are some best practices for API testing?

Before creating API tests, document and understand the API. This should include API purpose, application workflows, supported integrations, and so on.

Some tools for API testing include ReadyAPI, AcceIQ, Katalon, SoapUI, Postman, Apigee, JMeter, REST-assured, and more. Manual API testing could be a starting point. Tools such as Postman can help create tests manually, save them, and replay them later. For automated API testing, adopt an automation framework such as Robot Framework.

Create client code and components that can be reused across many tests. Write clear tests so that debugging and maintenance is easier. Organize each test into three parts: setup, execution and teardown. It should be possible to configure tests for different environments or customer requirements.

Write tests in a modular fashion. For example, user authentication and password change can be two separate tests, and the latter can be made dependent on the former.

Measure how long each test takes. This can help in scheduling tests. Schedule tests to execute every day. When a test fails, make the failure state explicit in the response or report. Test system should record failures for later analysis.

## Milestones

2000

While APIs existed in earlier decades, the early 2000s mark the birth of modern APIs. During this time companies such as Salesforce, eBay and Amazon popularize the use of APIs. In these APIs, the use of XML data format becomes common.

Dec
2009

Mike Cohn makes the point that test automation must be done at the correct level. He identifies three levels: unit, service and UI. He visualizes these into a **test automation pyramid**. We wish to do lots of unit testing and as little UI testing as possible. API testing happens in between and avoids unnecessary repetitions of the same UI actions. Although he calls the middle layer the service layer, it's not restricted to just service-oriented architecture (SOA).

2018

Since API specifications are formal, and with the recent progress of Natural Language Processing (NLP), some tools such as Functionize explore the possibility of automatically generating API tests from the specifications. This takes test automation to another level so that human testers can focus on exploratory and security tests.

API Testing


# RISC-V Instruction Sets

## Summary

The design of RISC-V instruction sets is modular. Rather than take the approach of a large and complex monolith, a modular design enables flexible implementations that suit specific applications.

RISC-V defines base user-level integer instruction sets. Additional capability to these are specified as optional extensions, thus giving implementations flexibility to pick and choose what they want for their applications. The specifications of the base ISA has been frozen since 2014. Some of the extensions are also frozen while many others are being defined.

## Discussion

### Could you give an overview of RISC-V instruction set?

RISC-V comprises of a base user-level 32-bit integer instruction set. Called **RV32I**, it includes 47 instructions, which can be grouped into six types:

    + R-type: register-register
    + I-type: short immediates and loads
    + S-type: stores
    + B-type: conditional branches, a variation of S-type
    + U-type: long immediates
    + J-type: unconditional jumps, a variation of U-typeRV32I has `x0` register hardwired to constant 0, plus `x1-x31` general purpose registers. All registers are 32 bits wide but in RV64I they become 64 bits wide. RV32I is a **load-store architecture**. This means that only load and store instructions access memory; arithmetic operations use only the registers. User space is 32-bit byte addressable and little endian.

Correspondingly, **RV64I** is for 64-bit address space and **RV128I** is for 128-bit address space. The need for RV128I is debatable and its specification is evolving. We also have **RV32E** for embedded systems. RV32E has only 16 32-bit registers and makes the counters of RV32I optional.

### What RISC-V extensions have been defined?

RISC-V defines a number of extensions, all of which are optional. Some of them are frozen and these are noted below:

    + M: Integer multiplication and division.
    + A: Atomic.
    + F: Single-precision floating point compliant with IEEE 754-2008.
    + D: Double-precision floating point compliant with IEEE 754-2008.
    + Q: Quad-precision floating point compliant with IEEE 754-2008.
    + C: Compressed instructions (16-bit instructions) to yield about 25-30% reduced code size. "RVC" refers to compressed instruction set.Among the evolving or future extensions are L (decimal float), B (bit manipulation), J (dynamically translated languages), T (transactional memory), P (packed SIMD), V (vector operations), N (user-level interrupts), and H (hypervisor support).

When multiple extensions are supported, that ISA variant can be described by concatenating the letters; such as, RV64IMAFD. To represent the standard general purpose ISA, "G" is defined as a short form for "IMAFD".

RV32I uses one-eighth of the encoding space. This means there's plenty of room for custom extensions.

### What are pseudo-instructions?

To ease the job of an assembly language programmer or a compiler writer, some base instructions can be represented by what are called **pseudo-instructions**. For example, a no operation is `addi x0, x0, 0` for which `nop` is the pseudo-instruction. Likewise, branch if zero is `beq rs, x0, offset` for which `beqz rs, offset` is the pseudo-instruction.

### What are privileged instructions?

Application code usually runs in user mode or **U-mode**. RV32I and RV32G are user mode ISAs. Two more modes are available:

    + **Machine mode (M-mode)**: For running trusted code. This is the most privileged mode in RISC-V and has complete access to memory, I/O and anything else to boot and configure the system. It's most important feature is to handle synchronous exceptions and interrupts. The simplest RISC-V microcontrollers need to support only M-mode.
    + **Supervisor mode (S-mode)**: For supporting operating system needs of say Linux, FreeBSD or Windows. This is more privileged than U-mode but less privileged than M-mode. Where OS needs to process exceptions/interrupts, *exception delegation* is used to pass control to S-mode selectively. S-mode also provides a virtual memory system.

### Could you describe some technical considerations in the design of RISC-V instructions?

Design of RISC-V ISA considered cost, simplicity, performance, implementation-independence, room for growth, program size, and ease of use.

RV32I includes generously 32 integer registers, making it easier for compilers to use them more often than memory. By keeping instructions simple, RISC-V instructions typically require only one clock cycle and deliver predictable performance. For dynamic linking, it adopts PC-relative branches.

Instructions offer three register operands, avoiding the extra move required by ISAs with only two register operands. These are also in the same positions so that access can begin before decoding the instruction.

The design was also informed by mistakes of other ISAs. For example, initial Alpha ISA did not have byte or half-word load/store. The shift operation in ARM can be seen as an overdesign. Delayed branches of MIPS and SPARC affected their ISAs. ARM Thumb and MIP16 added 16-bit instructions in hindsight.

### Are RISC-V instructions without precedents?

When we consider the 122 instructions of RV32G, only 6 of them are without precedents. 98 instructions appear in at least three prior ISAs. 18 instructions appear in one or two prior ISAs. This study included 18 prior RISC ISAs, including the CDC 6600 dating back to 1964.

It's been commented that one cannot design a flawless ISA, nor an ISA with flaws doomed to fail.

## Milestones

1981

RISC-I is defined by David A. Patterson at UC Berkeley as an alternative to increasingly complex instructions of the computers of the day. This is followed by RISC-II (1983), SOAR (1985) and SPUR (1986) projects.

2010

Researchers at the UC Berkeley conceive RISC-V as the fifth generation of their RISC design from the 1980s.

May
2011

Version 1.0 of the RISC-V base user-level ISA is published as volume 1. This version is not frozen. Volume 2 is for supervisor-level ISA.

May
2014

Version 2.0 of the user-level ISA is published. This is a frozen version.

May
2017

Version 2.2 of the user-level ISA is published. This does not modify the ISA base plus extensions IMAFDQ version 2.0. Extension C is frozen in this version.

RISC-V Instruction Sets


# Multi-Access Edge Computing

## Summary

In traditional cloud computing, apps and services are hosted in data centres. Devices connect to the data centres via multiple hops traversing the internet. If devices are smartphones, they probably connect via the RAN and the CN of the mobile network operator. Multi-Access Edge Computing (MEC) brings apps and services closer to the network edge. MEC also exposes real-time radio network and context information. The benefit is better user experience, network performance, and resource utilization.

MEC is just one among many edge computing paradigms, others being cloudlets, fog computing and Mobile Cloud Computing (MCC). MEC is perhaps more popular since it's standardized. ETSI is the main organization standardizing MEC.

One possible definition is that,

> MEC offers application developers and content providers cloud-computing capabilities and an IT service environment at the edge of the network.

## Discussion

### What are the benefits of MEC for users, service providers and network operators?

MEC enables many use cases that require high bandwidth, ultra-low latency and high device density. Users get a richer experience. Safety is improved for critical applications such as industrial automation and self-driving vehicles. If 5G promises many novel use cases, it's MEC that delivers them.

MEC helps network operators reduce their CAPEX by purchasing general-purpose equipment rather than specialized telecom equipment. By reducing bandwidth requirements between RAN and CN, OPEX is also reduced. Due to virtualization, reliability and scalability is improved. Heterogenous configurations can be supported. Network entities can be rapidly deployed leading to just-in-time service initiation. Network performance can be optimized by adapting to changing radio conditions. Operators can offer innovative applications and unlock new revenue streams by safely opening up their networks to third-parties.

Traditionally, it made sense only for big service providers such as Netflix to deploy edge servers for their traffic. With MEC, even smaller service providers and independent software vendors can get their applications to the edge on multi-vendor platforms. This is mainly because MEC is standardized with open interfaces and protocols.

### What are some use cases that can benefit from MEC?

MEC brings proximity, ultra-low latency, high bandwidth and virtualization. These characteristics can benefit **many use cases**: data/video analytics, location tracking services, augmented reality, IoT, local hosting of video content, data caching, remote surgery, patient management, radio-aware video optimization, autonomous vehicles, and more.

As more and more applications become **virtualized**, MEC will become increasingly important. Applications can be dynamically deployed, scaled and moved between the cloud and the edge.

On college campuses, business parks, hospitals or factories **private/local networks** can be deployed. For **mission critical communications**, MEC can continue delivering services even when backhaul communications fail.

**IoT applications** require ultra-low latency, mobility management, geo-distribution, location awareness and scalability. MEC is able to provide these for diverse IoT applications such as smart home, smart city, smart agriculture, smart energy, healthcare, wearables and industrial internet.

### Which are the main enablers for MEC?

MEC is made possible by the following technologies:

    + **Network Functions Virtualization (NFV)**: NFV is about deploying network functions in virtual environments rather than with dedicated hardware. MEC reuses the NFV Infrastructure (NFVI) and NFV Management and Orchestration (MANO). In other words, MEC platform and applications appear as VNFs and run on the same infrastructure as RAN or CN VNF components.
    + **Software-Defined Networking (SDN)**: Involves separating the control plane and user plane, logically centralizing the control plane and programming the control plane in a more flexible manner using APIs. An SDN controller can be in the MEC server. SDN brings scalability, availability, resilience, interoperability and extensibility to MEC operation.
    + **Service Function Chaining (SFC)**: Built from NFV, operators can use SFC to interconnect multiple NFs (virtual or physical) in a specific order to achieve end-to-end services and steer traffic flows.
    + **Network Slicing**: A network slice is a logical network set up for specific performance requirements. Using SDN/NFV, slices can be dynamically instantiated, modified or terminated.
    + **Information-Centric Networking (ICN)**: Replaces client-server model of the internet with publish-subscribe model involving caching, replication and optimal content distribution.

### Could you describe ETSI's MEC reference architecture?

ETSI's MEC reference architecture has the following main entities:

    + **MEC Host**: Contains MEC platform. Virtualized infrastructure providing compute, storage, and network resources for running MEC applications. Routes traffic among applications, services and access/local/external networks. Executes traffic rules received by MEC platform.
    + **MEC Application**: Runs on a Virtual Machine (VM) or container on the MEC host. Configured and validated by MEC management.
    + **MEC Platform**: Has essential functionality needed to run MEC applications. Enables applications to discover, advertise, offer or consume MEC services. Platform can also provide services. Receives traffic rules from MEC platform manager. May be interfaced with an API gateway for apps to access MEC service APIs.
    + **MEC Management**: Comprises of system-level and host-level management. The former oversees the complete MEC system and includes Multi-Access Edge Orchestrator (MEO) as a core component. The latter manages a particular host and its applications and includes MEC Platform Manager (MEPM) and Virtualisation Infrastructure Manager (VIM).The reference architecture contains three groups of reference points among entities: **Mp** for platform functionality, **Mm** for management, and **Mx** for external entities.

### What are the different MEC deployment models?

MEC servers can be deployed at base station sites, aggregation points in the radio network, mobile EPC sites or regional data centres. A distributed data centre or a gateway at the edge of the CN could be MEC deployment sites. There's a trade-off: closer to the edge, greater are the benefits but so is the cost of deploying at many locations. It's therefore expected that most operators will initially deploy at a few EPC sites and central offices. As more applications emerge demanding 1ms latency, operators will start deploying closer to the edge. To reduce CAPEX, operators may even share MEC infrastructure.

An MEC server can be indoors at a multi-RAT cell aggregation site serving an enterprise; or it could be outdoors for public coverage scenarios such as stadiums and shopping malls. Ultimately, deployment depends on scalability, physical constraints, performance criteria, cost of deployment, etc. Some MEC services may be unavailable in some deployment scenarios.

Vendors offer equipment optimized for specific deployment locations. For example, GIGABYTE's H242-Z10 is for base station tower whereas G242-Z10 is for aggregated sites.

### Could you mention some resources to learn more about MEC?

MEC specifications from ETSI are available for download via a search feature.

Apart from the specifications, via the DECODE Working Group, ETSI provides API implementations, testing, a proof-of-concept framework and a sandbox environment. This makes it easier for vendors, operators and application developers to implement MEC.

A beginner can start with ETSI's white papers on MEC. There's also an MEC blog for latest updates.

The main MEC wiki page is an entry point for MEC ecosystem, testing, sandbox, proof-of-concept updates, deployment trials and hackathons. The MEC Ecosystem page lists MEC applications and solutions.

Among the many open source projects related to edge computing, two important ones are Akraino and EdgeX Foundry. These come under LF Edge, an umbrella organization under the Linux Foundation. A useful resource is LF Edge's own Wiki page.

## Milestones

Sep
2014

ETSI forms the Mobile Edge Computing Industry Specification Group (ISG).

Dec
2014

The **first meeting** of MEC ISG takes place and is attended by 24 organizations, including network operators, vendors, technology suppliers and Content Delivery Network (CDN) providers.

2015

ETSI's MEC ISG publishes two specification documents: *Proof of Concept Framework* and *Service Scenarios*. The group develops three PoC scenarios. These relate to video optimization and orchestration by adapting to RAN or radio conditions.

Sep
2016

At the MEC World Congress, at the ETSI MEC PoC Zone, six **multi-vendor proofs of concept** are demonstrated based on the MEC PoC framework. It's hoped that such PoCs lead to a diverse and open MEC ecosystem.

2017

ETSI renames MEC from *Mobile Edge Computing* to *Multi-Access Edge Computing*. The new name reflects its relevance to mobile, Wi-Fi, and fixed access networks.

2018

ETSI's MEC group works on **Phase 2 activities** to address charging, regulatory compliance, mobility support, containerization, support of non-3GPP mobile networks, automotive vertical, and more. This year sees approval of 12 MEC PoCs and 2 MEC Deployment Trials (MDTs). A new working group named *DECODE* is also created to focus on deployment and ecosystem development.

Jun
2019

**Akraino** Release 1 is released with ten "ready and proven" blueprints. Blueprints are tested by Akraino community members on real hardware. They serve as an easy starting point for real-world edge implementations and edge use cases. Akraino comes under LF Edge, an umbrella organization founded in January 2019 to bring together multiple edge-specific projects. In Release 3 (Aug 2020), **MicroMEC** is specified. By Release 4 (Feb 2021), Akraino has 27 blueprints.

Multi-Access Edge Computing


# PBX Hacking

## Summary

PBX (Private Branch Exchange) is a private telephone network that handles an organization's internal and external communications. For external connections, the PBX connects to the Public Switched Telephone Network (PSTN) using a Telecommunication Service Provider (TSP) or an Internet Service Provider (ISP).

Hackers target PBX networks in ways that can impact the company. Using the PBX, they might place long-distance calls for free, leaving the company to pay the bills. Hackers might steal data or simply render the network unusable. Such hackers who specialize in hacking phone systems are called *phreakers*.

PBX hacking was traditionally done on analogue PBX systems using various methods. When IP PBX systems were introduced in the 1990s, hacking methods were adapted to these newer networks. There are many best practices that companies can follow to mitigate the dangers of PBX hacking.

## Discussion

### Why is it important to know about PBX Hacking and its role worldwide?

PBX is a system connecting the communication lines such as switches, hubs, telephone adapters and routers. There are different types of PBX Systems from which digital PBX or IP PBX/VoIP PBX is preferred in the business phone systems.

According to the survey done by CFCA(Communications Fraud Control Association), around 2013-2017the cases of PBX and IP-PBX was the top five frauds that you can see in the table given in fig.1. In France, the PBX toll fraud losses in companies estimate $220 million a year. Even before 2013, PBX Hacking cases has been there since the switchboards introduction to the world.

Tom Mulhall, in his paper “Where have all the hacker gone”,1997, states that though it might look like hacking cases has deteriorated while they have migrated from computer hacks to PBX/Voicemail attack.

### How does PBX Hacking affect the company or an organisation?

PBX Hacking/PBX Toll Fraud or PBX Fraud are some terms mentioned concerning phreakers cracking loopholes of the PBX System.

In different types of PBX Systems, some functions are the same like managing calls with also being connected with companies' systems that contain sensitive data(call records of customers). The system hacked through viruses or worms results not only the loss of sensitive data but an extra charge to fix the system. Also, there’s a possibility where the hacker can listen to phone conversations or voicemails or shut down PBX entirely.

After hacking PBX, phreakers usually make free long-distance calls or run call-sell operations from phone booths or private phones which builds a mode to generate funds illegally. By offering the calls for less than the actual cost to dial directly, the result is the companies owning that PBXs end up paying the bill.

It is also difficult to track these call-sellers as they hide their activities from law enforcement officials by PBX-looping(using one PBX to place calls out through another PBX).

### What are the ways the PBX System can be hacked?

DISA and voice mail are classic ways for phreakers to penetrate the organisation PBX System. Also, DoS acts as one of severe attacks towards the PBX System.

    + Brute force attack: It is a trial and error method until the password or log-in credentials or encryption keys.
    + DISA and Voice Mails: DISA(Direct Inward System Access) is a service provided by PBX where the user’s dials into PBX and then give information for authorisation to use PBX service dial local user. The authorisation process comprises the user’s account, user’s password and callerID. Fraudsters with tricks and hacks try to obtain this information and retrieve it, which then helps them to use PBX to generate outbound calls. While in the voicemail, fraudsters plan to reach voicemail boxes, obtain passwords and retrieve them to gain access to the system.
    + Denial of Service Attack(DoS): The denial of service attack makes telephony service unavailable by causing physical damage or software strategies and denying telephones to place or receive calls. It works by flooding networks with fake traffic or server request generated by machines compromised by viruses and malware.

### What are the other famous ways for the PBX System to be hacked?

    + Internal Enemy: One such example to comprehend the internal enemy is when the employee of that organisation forwards their work number to their private number either overseas or in their country, where for per call the organisation foot up the bill. In some instances, revenge (due to a poorly treated employee) becomes the cause for them to turn against the company.
    + Poor Port Management: A poorly protected port linked with PBX can offer hackers a chance to create a “back door” into critical assets such as customer databases and business applications. Also giving them a path to target modem.
    + Social Engineering: It is a technique where hackers just by impersonating or manipulating the person or the entire organisation get the sensitive data. One such example is Frank Abagnale, who is known as the infamous social engineer in history.

### How the ‘SIP’ has an impact on IP-PBX?

SIP(Session Initiation Protocol) is a protocol that initiates a session in an IP network while aiming to provide functions(similar to traditional PSTN) over the internet. It deals with signalling messages, address resolution, user management and packet transfer and services and is as vulnerable as HTTP or any available service which is public on the Internet.

It can be easily hacked through DoS attacks where :

‘SIP Register Flooding’ attack happens to create traffic by sending streams of SIP REGISTER messages and ‘Call Flooding’ attack which is a way of forming traffic by sending streams of the SIP INVITE requests. These attacks make it difficult to get any legitimate calls.

SIP Injection attacks are also a way where phreaker exploit the SIP by injecting malicious code where Buffer Overflow attack, RTP(Real-Time Transport) Injection Attack and SQL Injection Invites are some examples.

Due to SIP not enforcing any SIP source message validation mechanisms, it creates an opportunity for the attackers to have their request processed without authentication, making a path for spoofing or modification of the SIP control messages.

### How to analyse whether we have become the victim of PBX Hacking?

There are some indications to notify the victim whether they have been hacked:

    + Overload occurring in the incoming and outgoing trunks.
    + Sudden change of call patterns mostly the increase in international calls.
    + Any kind of lengthy calls or calls to premium rate services during late-night hours or weekends and holidays.
    + Any strange messages left in the voicemail boxes.
    + Any signs of war dialing like short calls or wrong number calls or any sign of social engineering.
    + Any kind leakage found with respect to the business secrets and sensitive data indicating that the phone conversations are intercepted(hacker listened to the phone conversation).And lastly, webcams and microphones activate automatically in the case of VoIP systems.

### What can we do to prevent them?

    + A regular audit must happen in the organisation to assess its vulnerability to fraudulent exploit.
    + In case of securing DISA, Call Detail Recording help in identifying call activity linked with individual authorisation codes, while also keeping limited print copies of these records.
    + Always change security codes regularly, and give access limit of the administration of authorisation codes to some carefully chosen employees of the company or organisation.
    + With help of control functions within the firewall control the traffic covering the subsystem, which will protect network resources and confidentiality of all traffics, also encrypting signal and media streams will increase the security level.
    + A strong Firewall to evaluate SIP message contents for attacks and block non-SIP traffic.
    + Regular maintenance and analysis of logs.
    + Give PBX users Security advice like never leaving phone-set unlocked, not sharing any type of security codes and regular change of security codes should be done and never having any kind of sensitive data in the phone’s memory.
    + Having strong passwords are always recommended.
    + Channels or services that are not in use should be disabled.
    + Further securing system VPN (in case of VoIP PBX) for remote access and enable endpoint filtering.

### What techniques or tools do PBX vendors use to mitigate hacking?

Some of the PBX vendors have already working on providing any network security in their systems such as Network firewalls, DDoS(Distributed Denial of Service) prevention, network posture assessments and encryption during data transfer, while at the time while purchasing they try to brief all the risk of hacking and the ways you can avoid it.

There are certain analyses to see whether the provider is doing their part such as checking for accreditations(certificates to show whether the industry meets the security standards) like TEC certification, and prevention measures in the system to prevent hacking, also whether the regular updates are available and are they up to date and also look for call encryption.

## Milestones

1878

At a time when telephone calls are routed via **manual switchboards**, two switchboard operators of the Bell Telephone Company intentionally disconnect or misdirect calls. Historically, this is probably the first attempt at hacking telephone networks. Hackers in these early days are simply practical jokers who don't cause serious damage. These operator hacks disappear once switchboards get automated in the 1890s via electro-mechanical switching.

Nov
1963

MIT's student newspaper, *The Tech*, features an article titled *Telephone Hackers Active*. It describes how student hackers are occupying tie-lines between MIT and Harvard, or making long-distance calls for free. A PDP-1 computer searches for outside lines by listening for a dial tone. Some students are even expelled for this. Subsequently, the MIT phone system is updated to prevent calls over tie-lines.

1968

In the U.S., a community of phone hackers begins to take shape during 1968-1969. About this time, the term **phone freak** comes into use. By the end of the decade, whistles and flutes are readily available to aid hackers. For instance, since the mid-1960s, Cap'n Crunch whistle that came with a box of cereals was used by hackers. It emitted 2,600 Hz, a useful tone for hacking. Years later, phone companies install digital filters to block this and other specific tones.

1971

In June, a newsletter of the Youth International Party Line, uses the word **phreek** in its first issue. In October, an Esquire article titled *Secrets of the Little Blue Box* uses the term **phreak** in what may be the first published use of the term. The article describes how toll-free 800 numbers and a tone at 2,600 Hz could be used to occupy tandem lines and place long-distance calls for free.

1972

**Semiconductors** are used in PBX equipment and switching. Through the 1970s, dropping cost of hardware and lower operational expenses make PBX attractive. More and more companies install PBX systems.  This also implies more opportunities for phreakers.

Apr
1992

For greater awareness, the NIST publishes **NISTIR 4816**, *PBX Administrator’s Security Standards*. This discusses the different techniques that PBX hackers use and what PBX administrators can do to mitigate them. These include setting passwords, educating users, protecting voicemails, monitoring PBX options, reviewing billing records and limiting outgoing international calls. A related publication from 2001 is **NIST 800-24**, *PBX Vulnerability Analysis*.

1993

In the U.K., the National Computing Centre includes in its regular survey report a new item name "PBX hacking" that has cost organizations £10,000. Although computer hacking dropped from 29.7% (1987) to 8.8% (1993), it's clear that hackers have simply moved to hacking PBX and voicemail systems. This is the result of traditional PBX systems giving way to **IP PBX systems**.

Nov
2011

Four hackers in the Philippines are arrested for PBX hacking funded by a terrorist organization. The phreakers used PBX systems to call Premium Rate Service (PRS) numbers. The revenue from this was split between the phreakers and the terrorists. One of the phreakers had previously hacked PBX systems by exploiting default passwords and unused extensions. He and his associates then offered long-distance call services to customers at low rates, placing calls worth $55 million during 2005-2008.

Dec
2014

A Pakistani national named Qasmani is arrested for scamming American telecom companies to the loss of $19.6 million during 2008-2012. Qasmani has been an active phreaker since the late 1990s. He and his associates exploited unused extensions of PBX systems and placed calls to pay-per-minute PRS numbers that they had set up. In June 2017, he's sentenced to four years in prison.

2019

A survey of Communications Service Providers (CSPs) by the Communications Fraud Control Association reveals that the estimated global loss in 2019 due to communications fraud is $28.3 billion. Of this, PBX hacking and IP PBX hacking account for $1.82 billion each.

PBX Hacking


# Orthogonal Frequency Division Multiplexing

## Summary

Orthogonal Frequency Division Multiplexing (OFDM) is a key wideband digital communication method used in wireless transmission. Data is split into several streams and transmitted on multiple narrowband channels to reduce interference and crosstalk.

Due to its good spectral efficiency and relatively less complexity, it's one of the popular techniques used in telecommunications. It's used in multiple standards including DAB, HDTV, WLAN (802.11a/g/ac), WiMAX, and LTE. It enables transmission of high data rates (in the order of 1 Gbps) on a wireless channel.

## Discussion

### Could you give an overview of OFDM?

OFDM is a **Multi-Carrier Modulation (MCM)** scheme, which uses closely spaced multiple subcarriers to transmit data. Data to be transmitted is split and transmitted using multiple subcarriers instead of using a single carrier.  The key idea is instead of transmitting at a very high bit rate, the data is transmitted over multiple subchannels each carrying lower bit rates.

Unlike the traditional Frequency Division Multiplexing (FDM), the OFDM does not use guard bands to separate the various subchannels. One of the key features of OFDM is the orthogonality of the subcarriers used to transmit data. The orthogonality of subcarriers results in more subcarriers in a given bandwidth. This improves spectral efficiency. It also eliminates the interference between subcarriers, often called **Inter-Carrier Interference (ICI)**.

### Why is OFDM used in wireless transmission?

One of the key challenges in wireless transmission as compared to wired transmission is the phenomenon of multipath fading and **Inter-Symbol Interference (ISI)**. OFDM helps in mitigating both these effects, making it one of the key technologies to be used in wireless transmission. OFDM uses spectrum in a more efficient way compared to some of the other techniques used to overcome multipath fading and Inter Symbol Interference.

### How does OFDM ensure subcarriers do not interfere with each other?

OFDM splits the available spectrum into multiple subbands and transmits data using multiple subcarriers. The subcarriers are chosen such that they are orthogonal to each other. This ensures that data from one subcarrier does not interfere with the data on the other. To maintain orthogonality between subcarriers, the subcarriers are chosen such that they are all integer multiples of the base frequency. If the total bandwidth of the system is B Hz. Then the base frequency (f0) is given by B/N, where N is the number of subcarriers in the system. The subcarriers used are f0, 2f0, 3f0 ... (N-1)f0.

The spectrum of each transmitted subcarrier in OFDM system is a `sinc` function with side-lobes that produce overlapping spectra between subcarriers. Since the carriers are orthogonal, the peak of each subcarrier coincides with nulls of other subcarriers. Even though there's overlap of spectra between subcarriers, there's no interference between subcarriers.

### What's the role of FFT and IFFT in OFDM implementation?

OFDM system involves mapping of symbols onto a set of orthogonal subcarriers that are multiples of the base frequency. This can be implemented in digital domain using Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT). These transforms are important from OFDM perspective as they can be viewed as mapping digital input data onto orthogonal subcarriers.

The IFFT takes frequency-domain input data and converts it to the time-domain output data (analog OFDM symbol waveform). This waveform is transmitted by the OFDM transmitter. The receiver receives the waveform and uses FFT transform to convert the data back from time-domain into frequency domain to recover the data back.

### What's guard band and cyclic prefix in OFDM?

OFDM systems make use of guard band and cyclic prefix (CP) to overcome the issue of ISI. While guard band is not required to achieve orthogonality of subcarriers, it helps in overcoming ISI in a multipath channel. The duration of the guard band should be more than the channel spread of the wireless medium.

The cyclic prefix is transmitted during the guard band interval. After IFFT, some end bits of the OFDM symbol are copied to the guard band before the symbol to form the CP.

Receivers often have a channel equalizer to combat channel distortion. CP simplifies equalizer implementation. Essentially, CP converts a linear convolution to a circular convolution. Circular convolution in time domain is equivalent to a simple multiplication in the frequency domain. The channel equalizer in the receiver multiplies the received symbol by the inverse of the channel coefficients in frequency domain to recover the original transmitted symbol, assuming that fading is constant over the subband.

### How is a typical OFDM transmitter and receiver implemented?

In an OFDM transmitter, the input bits are first grouped into symbols in frequency domain by using a serial-to parallel-converter. These frequency domain symbols are then taken as input by the IFFT block. The IFFT block converts the input symbol into time domain symbol by doing an IFFT operation on the input. The cyclic prefix is added to the output of IFFT block by the cyclic prefix block. This symbol is then converted back to series of bits by the parallel-to-serial converter and transmitted.

In the OFDM receiver the input signal is passed through the channel equalizer block first, to cancel any impairments introduced by the wireless channel. The output of the equalizer is then input to the prefix extraction block to remove the cyclic prefix. The output of the prefix extraction block is then given to the FFT block. This block converts the input to frequency domain output by doing an FFT operation. Thus the OFDM receiver recovers the original bits back by doing a parallel-to-serial operation.

### What are the advantages of OFDM?

Here are some advantages:

    + **High spectral efficiency**: Compared to other schemes like spread-spectrum, OFDM uses the available spectrum in a more efficient way.
    + **Robust against multipath fading and Inter-Symbol Interference**: Due to low data-rate in each subchannel, OFDM is more resilient to inter-symbol interference caused by multipath propagation.
    + **Simpler channel equalizer in receiver**: In case of OFDM the channel equalization can be done in frequency domain and is a multiplication operation of the received symbol with the channel equalizer.
    + **Efficient implementation**: Implementation can be done using IFFT and FFT, thus eliminating the need for multiple mixers in transmitter and receiver.
    + **Robustness against selective fading**: Since the transmission is done using multiple smaller subbands, frequency selective fading appears as flat fading for each subband.
    + **Resilience to narrow band interference**: Due to narrow band interference, contents in some of the subchannels will be lost. It is possible to recover it by using channel coding and interleaving data before transmission.
    + **Tuned subchannel receivers not required**: Unlike conventional FDM, tuned subchannel receivers are not required, thus simplifying the receiver design

### What are the disadvantages of OFDM?

Here are some disadvantages:

    + **Sensitive to Carrier offset and frequency drift**: In case there is an offset between the transmitter and receiver carrier frequencies, the orthogonal property of the OFDM is lost. Thus OFDM systems are very sensitive to carrier frequency offsets.
    + **High Peak-To-Average power ratio**: Since the output of multiple subbands are combined to get the OFDM signal, the OFDM signal has a very high dynamic range of amplitude. This leads to complex RF design as the amplifiers need to be linear for the entire amplitude range. This also leads to lower efficiency of the RF amplifier.

## Milestones

1870

Alexander Graham Bell is initially funded by his future father-in-law Gardiner Hubbard to work on *harmonic telegraphy*, which is an **FDM** transmission of multiple telegraph channels. With FDM, more than one low rate signal is carried over a relatively wide channel using a separate carrier frequency for each signal.

1957

Collins Radio Company develops the **Kineplex** system to overcome multipath fading. 20 tones are modulated by differential 4-PSK without filtering. Tones can be separable by bank of filters at the receiver.

1961

Franco and Lachs propose a multitone, code-multiplexing scheme using a 9-point QAM constellation for each carrier. Sine and Cosine waves used to generate orthogonal signals.

1966

What could be termed as the birth of modern OFDM, Robert W. Chang publishes *Synthesis of Band-Limited Orthogonal Signals for Multichannel Data Transmission*. He uses Fourier transform to make the subcarriers orthogonal. His system is able to transmit signals in parallel without either ISI or ICI.

1967

B.R. Saltzberg extends Chang's work to complex data, that is, Quadrature Amplitude Modulation (QAM). He shows that I and Q streams should be staggered by T/2, and adjacent channels the other way. Zimmerman and Kirsch publish a paper on the design of an HF (high frequency) radio OFDM transceiver (KATHRYN). This uses 34 subchannels in a 3kHz bandwidth. KATHRYN uses analog hardware to generate orthogonal signals using Discrete Fourier transform (DFT).

1971

Weinstein and Ebert use **Fast Fourier Transform (FFT)** implementation of DFT. This greatly reduces the cost and complexity of OFDM systems. However, Weinstein notes later Bell Labs didn't show much interest in this. The big applications of OFDM (ADSL, wireless communications, digital audio/video broadcasting) came years later. Weinstein and Ebert also introduce the **guard band** for multipath channels.

1980

Although earlier work made the subcarriers orthogonal, in a time dispersive channel the orthogonality was lost resulting in ISI. Peled and Ruiz solve this by introducing **Cyclic Extension (CE)**. Today we use the more familiar term **Cyclic Prefix (CP)**. Effective data rates are reduced but the gain in terms of zero ISI is worth it.

1985

For better channel estimation, L.J. Cimini introduces a pilot-based method to reduce interference from multipath and co-channels. This work becomes important in the context of cellular mobile systems where channels experience fast selective fading.

Jan
1993

Amati's prototype of an ADSL modem wins a competition with Carrierless Amplitude-Phase (CAP) Modulation in a Bellcore-sponsored test. The technique used is **Discrete Multi-Tone (DMT)**, which is essentially OFDM. Soon ADSL becomes the first major consumer-oriented application of OFDM. It uses 256-point DFT with subcarriers separated by 4.3125 kHz and a (block) symbol rate of 4000/s. Early deployments using Amati equipment happen with British Telecom in late 1993 and early 1994, offering 2 Mbps downstream.

1999

**802.11a** WLAN standard is published as an amendment of 802.11 from 1997. It uses OFDM in the physical layer for data transmission. 802.11a's OFDM has 52 subcarriers (4 pilot + 48 data), 64-point FFT, and 312.5 kHz of subcarrier spacing.