# Requirements Development

## Summary

It's hard to build the right product without knowing why it's needed or what features are expected of it. Requirements Development provides answers to these questions. It involves many stakeholders: users, designers, developers, architects, and managers. It identifies user needs, studies technical feasibility, derives exact requirements, and validates those requirements.

Requirements are various: business vs technical, functional vs non-functional, system vs component, hardware vs software, etc. Getting the requirements right mitigates risk, avoids costly rework, and improves customer satisfaction.

Traditionally requirements were identified at the start of the project. In Agile or in any iterative process, requirements are developed continuously as features are added/updated or bugs need to be fixed. Requirements development is so important that Karl Weigers once said,

> If you don't get the requirements right, it doesn't matter how well you do anything else.

## Discussion

### What's the requirements engineering process?

Requirements Engineering (RE) has two parts:

    + **Requirements Development (RD)**: Requirements are developed by understanding customer needs and stakeholder expectations. These are refined and documented as formal specifications. Once specifications are validated, they set the baseline or the starting point for the architecture and the design phases of the project. Getting the requirements right is an iterative process consisting of elicitation, analysis, specification and validation. With each iteration, gaps and defects are addressed.
    + **Requirements Management (RM)**: Requirements are rarely static. Changes to baseline requirements are common. These changes are managed with a clear process. Requirements are tied to project plans and schedules. Requirements are traced to design, code and tests.RD precedes RM when baseline requirements are defined. Thereafter, requirements are continuously refined even in later phases of the software development life cycle.

It's useful to clarify verification versus validation. **Requirements verification** checks that requirements are well formed. **Requirements validation** checks that requirements specify the right system fulfilling the needs of stakeholders.

### What are the different types of requirements?

For architects and developers, the word "requirements" typically means product requirements. For project managers, it may instead mean project requirements. Project requirements will include development/testing infrastructure, tools, licenses, staff training, compliance, and more.

Product requirements can be defined at different levels. Business requirements capture why the product is being built. Managers and marketing folks are involved. Then business analysts come in to define user requirements from the perspective of users. Finally, functional requirements are those that the product should satisfy. Functional requirements are inputs to designers, developers and testers.

Then there are non-functional requirements such as usability, reliability, performance, safety, etc. Data requirements, interface requirements and design constraints are also non-functional requirements.

For example, "customer can pay for gas at the pump" is a business requirement. User requirements include "swipe credit card" and "request receipt". Functional requirements include "prompt user for card swipe" and "parse card's magnetic strip". A non-functional requirement could be "complete the entire payment procedure within 60 seconds".

### What's the expected output of requirements development?

**Requirements specification** is the main output of requirements development. This may be a single document or organized as many documents: Business Requirements Specification, Software Requirements Specification (SRS), Stakeholder Requirements Specification, and System Requirements Specification.

Specifications may be captured as Microsoft Word or PDF documents. Alternatively, they're in spreadsheets, a database or in a custom requirements management tool. In any case, specifications must be well-organized, amenable to tracing and easily accessible by all stakeholders.

During requirements analysis, various models, diagrams or tables might be employed. Use case diagrams, class diagrams, data flow diagrams, and state transition diagrams are examples. Specifications must link to these diagrams.

Finally, let's understand **requirements versus specifications**. Requirements can be treated as *questions* and specifications as *answers*. We start with requirements that're rough statements. Through requirements analysis and validation, we refine these into formal specifications.  In another interpretation, specification is a structured collection of requirements.

### Could you explain requirements elicitation?

Requirements elicitation identifies users, user groups and other stakeholders. It attempts to understand their real needs. This in turn sets the product scope.

It's not as simple as asking users what they want. Henry Ford once said, "If I had asked people what they wanted, they would have said faster horses." Empathy is needed to see things from the user perspective. Build a rough prototype. Observe how users interact with it. This will likely provide useful insights.

Interviews and user focus groups can help. For diverse perspectives from all stakeholders, *Joint Application Design (JAD)* workshops could be organized. User stories, use cases (aka scenarios) and storyboards are some techniques to elicit requirements. These are effective since they dive into the level of actors, needs and interactions.

Sometimes implementations may be wrongly written down as requirements. Asking "why" questions to understand the context will reveal the hidden requirements. In fact, requirements elicitation is really about answering the questions what, why, and who.

Requirements elicitation is in fact nuanced. To call it requirements capture or gathering is perhaps an oversimplification.

### Could you explain requirements analysis?

Requirements analysis refines and adds details to the requirements identified via requirements elicitation. It's done iteratively until requirements can be clearly written down as specifications. High-level requirements are translated into technical product requirements at various levels.

Modelling is done, perhaps using Unified Modelling Language (UML). Visualizations help analyse the system from various perspectives. Some of these are data flow diagrams, entity relationship diagrams, state transition diagrams/tables, class diagrams, sequence diagrams, activity diagrams, process flow diagrams, decision trees, and event/response tables. Through analysis, incomplete, inconsistent or incorrect requirements can be uncovered.

Different aspects or viewpoints can be modelled: organizational, data, behavioural, domain, and non-functional requirements.

### What roles do business rules play in requirements development?

Business requirements frame the problem and the motivation for starting the project. On the other hand, business rules are policies, standards, practices, regulations or guidelines that are defined at the organizational level. Products are expected to adhere to these rules. Business rules affect use cases. System state, input, and user role are some factors that're used to define business rules.

One study identified five types of business rules: facts, constraints, action enablers, inferences and computations. Atomic business rules are better than one complex rule that combines multiple rules.

Consider a firm making a payment gateway. Due to an ongoing litigation, there may be a business rule prohibiting integration with a specific bank. The firm and another bank may be part of the same parent company. For this bank, there may be a rule waiving transaction fees. Due to local regulations, there may be a rule that sets the maximum value of a transaction. Such rules must be translated into functional requirements that developers need to implement.

### Could you give some tips for better requirements development?

Biases can affect requirements development. Some of these are optimism bias, overconfidence bias, strategic misrepresentation, anchoring, and more.

Avoid copying a competitor and jumping straight into implementing features. Likewise, avoid building the opposite of a failed product. Real failure here is skipping requirements engineering and assuming requirements can be transferred from one product to another.

A product packed with features but solicits user feedback very late is risky. Don't be afraid to involve customers to test prototypes and clarify requirements. Early adopters often like to give feedback and ideas.

Avoid "gold plating", that is, implementing features that developers thought would be nice to have. Delivering something that the user doesn't need wastes resources. It could also confuse users.

Document requirements clearly in the specifications. Each requirement must be unambiguous, validated, necessary, complete, consistent, feasible, traceable, verifiable and understandable.

### What resources can help me learn more about requirements development?

*Software Requirements* by Wiegers and Beatty (2013) is a good book to read. Other books from Wiegers include *Software Requirements Essentials* and *More About Software Requirements*. An older book is *The Requirements Engineering Handbook* by Young (2004).

The International Requirements Engineering Board (IREB) certifies engineers in the RE discipline. One study guide is *Requirements Engineering Fundamentals* by Pohl and Rupp (2015).

*ISO/IEC/IEEE 29148:2018* is the main standard to read. Also worth reading are its related standards *ISO/IEC/IEEE 15288:2023* (system life cycle processes) and *ISO/IEC/IEEE 12207:2017* (software life cycle processes).

Penzenstadler's video lectures on Requirements Engineering are available on YouTube. Coursera's Requirements Engineering: Secure Software Specifications Specialization has five courses in RE with security as a focus.

CMU SEI and INCOSE are two useful sources for guides and white papers. Search for the keyword "requirements" at their websites. As an example, we mention INCOSE's Guide to Writing Requirements (2023).

## Milestones

1975

Brooks in his book *Mythical Man-Month* writes about the need to **consult users** when writing system specifications. Some important ideas include user feedback, iterative development, separation of specification and design, and user manual as the external specification of the product. Until then, users weren't considered as important to either identifying requirements or writing specifications.

1977

Ross and Schoman propose the use of the term "requirements definition" over the more widely used but rather ill-defined term "requirements analysis". To them, **Requirements Definition** "encompasses all aspects of system development prior to actual system design." It deals with three things: context analysis (why the system is needed), functional specification (what the system is to be), and design constraints (how the system is to be constructed within defined boundary conditions). Multiple viewpoints (technical, operational, economic) must be considered.

1979

The TRW Defense and Space group proposes the **Software Requirements Engineering Methodology** to address project failures due to poor requirements. The early evolution of requirements engineering up to this point is from systems engineering.

1983

The IEEE Std 729-1983 **defines the word "requirement"**. The definition encompasses user need, contractual necessity, and the starting point for system development. This definition is revised in 1990 to include the aspect of documentation.

1984

IEEE publishes **IEEE Std 830-1984** titled *IEEE Guide to Software Requirements Specifications*. This standard is updated in 1993 and 1998 as *IEEE Recommended Practice for Software Requirements Specifications*. Also in 1984, the European Space Agency describes the Waterfall Model of software development. They separate user requirements from software requirements. They also state that user requirements is an input to defining software requirements.

1993

The first **IEEE Symposium on Requirements Engineering** is held. The following year, the **IEEE International Conference on Requirements Engineering** is held. In 2002, these two are merged into the **IEEE International Requirements Engineering Conference**, which is subsequently organized annually.

1994

Pohl proposes a **three-dimensional framework** for RE. Requirements have to be approached in the dimensions of specification, representation and agreement. They also note five factors that influence the RE process: methods/methodologies, tools, social aspects, cognitive skills, and economical constraints.

1995

Carroll's *Scenario-Based Design* describes how **user scenarios** can be used in requirements analysis. The idea of using scenarios is also seen in the research of other authors about the same time. Other complementary analysis techniques such as state transition diagrams and entity relationship diagrams were invented in the late 1970s.

2010

Jarke et al. point out that RE has changed over the last 30 years. RE is no longer about software alone. It's includes business development, software engineering and industrial design. Challenges include massive reuse, COTS components, system integration, vendor-led requirements, fluid design, short iterations, and distributed requirements. Some of these are reiterated by Jantunen et al. (2019) and noted earlier by Reifer (2000).

2011

**ISO/IEC/IEEE 29148** titled *Systems and software engineering — Life cycle processes — Requirements engineering* is published as an international standard. This replaces earlier standards IEEE 830-1998, IEEE 1233-1998, and IEEE 1362-1998. It's updated in 2018 as **ISO/IEC/IEEE 29148:2018**.

2022

Hehn and Mendez propose an artefact-based model that combines design thinking and RE. Design thinking focuses on the user perspective and uses low-fidelity prototypes to understand the problem better. RE uses methodologies and tools to specify the requirements in greater detail. The two approaches complement each other. While design thinking is diverging and multi-disciplinary, RE is converging and technical.

Requirements Development


# CSS Modules

## Summary

CSS was invented to style web pages. A typical web page contains many elements or components such as menus, buttons, input boxes, and so on. Styles defined in a CSS file are accessible to the entire web page in which that file is included. In other words, all style definitions have global scope. What if we want some styles to be visible only to one component of the page?

CSS was defined to style documents, not UI components. The lack of modularity in CSS language makes it hard to maintain complex or legacy code. Developers are afraid to make code changes since it's easy to break something when CSS definitions are global.

CSS Modules solves these problems by limiting the scope to components. CSS Modules is not an official standard. It's a community-led effort popularized by the ReactJS ecosystem.

## Discussion

### What are the problems that CSS Modules solve?

The problem of **global scope** is solved by CSS Modules since class names are local to the component. The same class name can be used in another component with different styling. Even though CSS as a standard has only global scope for names, CSS Modules leverages on tooling. Tools convert local names to globally unique names.

Deeply nested CSS selectors result in what we call **high specificity**. This impacts performance. With CSS Modules, names are globally unique and locally specific. Hence, flat selectors are more than sufficient. Because CSS styles are now encapsulated within components, code becomes more **maintainable**. Code can be refactored. Dead code can be removed.

### What are the essential features of CSS Modules?

CSS Modules has the following features:

    + **Local Scope**: Class names and animation names are locally scoped by default.
    + **Global Scope**: Using `:global` switch, developer can choose to reuse some styles globally.
    + **Composition**: A selector can extend styles from another, thus promoting reuse of styles within local scope. Composition can't be used for global scope. A selector can be composed from a selector in global scope or selectors in other files.
    + **Naming**: Names should be in camelCase. Though traditional CSS naming convention is to use kebab-case, hyphens can cause problems when accessed within JavaScript. The use of camelCase simplifies the syntax.
    + **Integration**: CSS Modules should have tooling support to compile CSS to a low-level format called *Interoperable CSS (ICSS)*. It should also work nicely with CSS processors (Less, Sass, PostCSS), bundlers (Webpack) and JS frameworks (Angular, React).

### Could you explain how CSS Modules work?

CSS as a language doesn't support local scopes. All names have global scope. To overcome this limitation, developers use tools to automatically transform local names to globally unique names. These names are generated by the CSS Modules compiler. The compiler also generates a JS object to map old names to new names. Therefore, developers can continue to use local names in their component code.

For example, let's take a UI component called "Cat". Its styles are in file "Cat.css" where `.meow` is defined. It's possible that the same class is used in another component, let's say, "WildCat". To avoid this name conflict, CSS Modules compiler makes each name unique.

CSS Modules is flexible. It doesn't impose a specific naming convention. In our example, the new class name is `.cat_meow_j3xk`, where the module name is used as a prefix and a hash value is used as a suffix. The mapping from old to new names goes into a JS object.

### What tools and plugins help with implementing CSS Modules?

Webpack's *css-modules* or PostCSS's *postcss-modules* can be used to implement CSS Modules. PostCSS can be used along with task runners such as Gulp or Grunt. PostCSS is really using JS plugins to transform CSS.

To tell Webpack how to load CSS files, there are *css-loader* and *style-loader*. These may be enough to use CSS Modules with Angular, TypeScript and Bootstrap.

With Browserify, the plugin to use is *css-modulesify*. For use in Rails, there's *cssm-rails*. With JSPM, there's *jspm-loader-css-modules*.

CSS Modules can be used within JS frameworks. For React, there's *react-css-modules* or *babel-plugin-react-css-modules*. CSS preprocessors such as SCSS can be used along with CSS Modules using *sass-loader*. Gatsby is another example where CSS Modules can be used.

To catch problems early, there's eslint-plugin-css-modules.

### Are there alternatives to using CSS Modules?

Historically, CSS started as a single file for an entire app. Thanks to tooling such as Webpack's *css-loader*, it became possible to use one stylesheet file per component. Each component's folder contained its CSS and JS files. However, styles still had global scope.

Global scope was partially solved by OOCSS, SMACSS, BEM, etc. **Block-Element-Modifier (BEM)** brought some modularity to CSS via a naming convention. It solved CSS specificity problem. However, class names were long and had to be named by developers. With CSS Modules, naming is taken care of by the build tools. During coding, developers use simpler and intuitive names. We can customize class names, including the use of BEM-style naming if desired.

**CSS in JS** embeds CSS directly in a JS file. The CSS model is at component level rather than at document level. For example, in *JSS*, styling is defined as JS objects. In another library called *styled-components*, tagged template literals are used. Michele Bertoli has a curated comparison list of many CSS-in-JS techniques.

### Isn't W3C standardizing CSS Modules?

No. CSS Modules is not a standard. It doesn't change the CSS specifications in any way. CSS definitions retain global scope but CSS Modules makes them "behave locally" by renaming them with unique names, which are then referenced by their components in HTML or JS. This is possible because of tooling support.

Beginners may confuse CSS Modules with another concept called *CSS3 modules*. In June 2011, CSS2.1 became a W3C Recommendation. However, this process took 13 long years, mainly because the specification was a monolith. When work started on CSS3, it was therefore decided to split the specifications into separate modules. Each module can take its own path of standardization. These modules are not related to CSS Modules.

A related concept is the use of *namespaces* in CSS. These refer to XML namespaces and how these can be used within CSS selectors. HTML `style` element has the `scoped` attribute for local scoping but this is now deprecated.

### What are some criticisms of CSS Modules?

It's been said that CSS Modules does nothing more than automate class naming so as to avoid name collisions. Developers can still do wrong things without CSS Modules warning them about it. It's not clear if typo errors will be caught at compile time or runtime. You can't share constants across CSS and JS files. **TypeStyle** is one approach that solves some of these issues. It mimics CSS Modules pattern using JS objects.

Back in 2016, one developer commented that CSS Modules is not ready for prime time. He mentions some limitations of using `@value` and `composes`. He states that using pseudo-selectors can be unreliable.

The use of camelCase that's not typical of CSS naming convention is seen as a limitation. While some frameworks may have plugins to solve some of these issues (such as *babel-plugin-react-css-modules*) it may prove difficult to make CSS Modules work with others.

## Milestones

2011

Facebook releases a JavaScript library called **ReactJS**. The design approach is modular, with a web page being built from a hierarchy of UI components. Two years later the project is open sourced.

Apr
2012

Version 0.1.0 of **css-loader** for Webpack is released on NPM repository. This gives control over how CSS is loaded into a project. This becomes useful when CSS is used directly within a UI component implemented in JS.

2014

Christopher Chedeau, a frontend developer at Facebook, lists a number of problems with CSS at scale. He states that while JS recommends avoiding global variables, CSS still uses global names. Facebook initially solves many of the issues by extending CSS and later by using **inline styles**. Styles are specified as JS objects within UI components. Mixing content and styling might seem like a bad approach but every component encapsulates its own styling without relying on global variables. The common name for this approach is **CSS in JS**.

Apr
2015

Webpack's css-loader starts supporting local scope with the use of `:local` switch. A month later some folks implement a module for PostCSS called **postcss-local-scope**, so that CSS styles are local by default even without using the switch. Only globals need to be explicitly specified with the `:global` switch.

May
2015

The PostCSS module postcss-local-scope gets integrated into css-loader of Webpack. Meanwhile, initial commit of **CSS Modules** happens on GitHub.

Jun
2015

The low-level file format that enables CSS Modules is specified separately as **Interoperable CSS (ICSS)**. This becomes the common format for module systems. The specifications is meant for loader implementers, not end users. Meanwhile, ICSS is implemented in loaders including css-loader (webpack), browersify, and jspm.

CSS Modules


# Robot Framework

## Summary

Robot Framework is a framework that automates acceptance testing and acceptance test-driven development. Being generic in nature, the framework can also be used to automate business processes, often called Robotic Process Automation (RPA).

The core of Robot Framework is written in Python but libraries extending it can be in Python or Java. The framework is independent of operating system and application. It's open source and is sponsored by the Robot Framework Foundation.

Test cases in Robot Framework are written using keywords. Keywords themselves are abstracted away from their implementation. This promotes reuse of keywords across tests and easier maintenance of tests, particularly in large projects.

## Discussion

### What are the benefits of using Robot Framework?

Since Robot Framework is keyword-driven, it has the benefit of separating high-level descriptions of tests from the low-level implementation details. Test writers may not be programmers. Since keywords are closer to English than programming, test writers find it easier to write and maintain tests. Keywords can be reused across tests.

Tests are easy to read since they often take a tabular form. In this form, the first column contains keywords. Other columns contain arguments to each keyword. For example, "Create User" is a keyword that takes username and password as arguments. When implemented, this can translate to launching a web browser, clicking a button, entering the data, and confirming the request to create a new user.

Keywords are composable. This means that keywords themselves can depend on other keywords. This allows us to create tests at different levels of abstraction.

Since implementation is kept separate, test cases can be developed agnostic of any programming language. Testing can be planned at an early stage even before any implementation is ready.

### What are the main features of Robot Framework?

While keyword-driven tests are common, Robot Framework can also be used to create data-driven and behaviour-driven tests. It can also be used for automating any business process, and thus not limited to testing.

Test cases can be organized into test suites. Execution can be based on test suites, test cases, or tags, all of which can be specified by name or pattern. Test suites and test cases can have setup and teardown.

The framework produces neat reports and logs. It also allows for debugging.

Developers can extend the framework with their own keywords and libraries.

Test cases can be created in a single text file but they can also be created within the Robot Framework IDE (RIDE). Some popular IDEs can integrate with the framework including PyCharm, RED (Robot Editor) for Eclipse, and Robot Framework Intellisense for VS Code.

### Could you list some built-in keywords used in Robot Framework?

Online documentation gives complete descriptions of all built-in keywords along with accepted arguments. We list a few of them:

    + **Conditionals**: Keyword Should Exist, Length Should Be, Should Be Empty, Should Be Equal, Should Contain, Variable Should Exist
    + **Control Flow**: Continue For Loop, Exit For Loop, Repeat Keyword, Return From Keyword
    + **Conversions**: Convert to Binary, Convert to Boolean, Convert to Bytes, Convert to Integer
    + **Executions**: Call Method, Evaluate, No Operation, Run Keyword
    + **Library**: Get Library Instance, Import Library, Reload Library, Set Library Search Order
    + **Logging**: Log, Log Many, Log To Console, Log Variables, Set Log Level, Set Test Message
    + **String Operations**: Catenate, Should End With, Should Match, Should Match Regexp, Should Not Be Equal As Strings
    + **Timing**: Get Time, Run Keyword If Timeout Occurred, Sleep, Wait Until Keyword Succeeds
    + **Variables**: Get Variable Value, Import Variables, Set Global Variable, Set Suite Variable, Set Test Variable
    + **Verdict**: Fail, Fatal Error, Pass Execution, Pass Execution If

### Where does Robot Framework fit within a test automation architecture?

The job of the framework is to read and process data, execute test cases, and generate reports and logs. The core of the framework doesn't know anything about the system under test (SUT). Actual interaction with SUT is handled by various libraries. Libraries themselves rely on application interfaces or low-level test tools to interact with SUT.

Let's take the example of testing a web application. Suppose Selenium is used for testing the application. Robot Framework doesn't directly control interactions with the web browser or databases. *SeleniumLibrary* and *DatabaseLibrary* are two libraries that manage these interactions. These libraries expose relevant keywords that can be used in test cases without worrying about how they are implemented.

### Could you share some best practices when writing tests in Robot Framework?

Test cases should be easy to understand. Use descriptive names for test suites and test cases. Likewise, keyword names should clear and descriptive. A common convention is to use title case for keywords. For example, use `Input Text` rather than `Input text`. For variables, use lowercase for local scope and uppercase for global scope.

Document each test suite. Describe the purpose of tests in that suite and the execution environment. Include links to external documents. At test case level, documentation may not be required. The use of suitable tags is often more useful.

Tests within a suite should be related. Don't have too many tests in a single file, unless they are data driven. Each test should be independent of others and should rely only on setup and teardown. Each test case should test something specific. A large test can possibly cover an end-to-end scenario.

If there are user-defined keywords, document the arguments and return values. Between keywords, assign return values to variables, and then pass these variables as arguments.

Avoid sleeps. Instead, wait for an event with a timeout.

### What are some useful resources and tools for Robot Framework?

The official Robot Framework website is a good starting point. Those new to writing tests, can study the examples there. The User Guide is another useful resource.

Beginners should get familiar with some of the standard libraries including Builtin, OperatingSystem, String, Process, DateTime, Collections, Screenshot, and more. The framework has been extended by many third-party libraries. Choose what suits your application. There are libraries to interface or work with Android, Eclipse, databases, Selenium, Django, SSH, FTP, HTTP, MQTT, and more.

Among the useful tools are Rebot for generating logs and reports; Tidy for cleaning or formatting test data files; Libdoc and Testdoc for documentation. Pabot is a tool for parallel execution. Robot Corder can record and playback browser test automation. You can choose among various editors and build tools. There's also a Jupyter kernel.

## Milestones

2004

At the Helsinki University of Technology, Paul Laukkanen/Klärck starts looking into large-scale test automation frameworks. As part of his Master's thesis (published in 2006), he creates a **keyword-driven** prototype framework.

2005

Nokia Networks is on the lookout for a generic test automation framework. Robot Framework is born based on Klärck's work.

Jun
2008

Nokia Networks open sources Robot Framework with the release of version 2.0.

Dec
2012

Version 1.0 of **Robot Framework IDE (RIDE)** is released on GitHub. RIDE simplifies the writing and execution of automated tests. Earlier version v0.40, also available on GitHub, can be traced to January 2012. Code before this was hosted at Google Code.

Dec
2015

Robot Framework 3.0 is released. With this release, **Python 3** is supported.

Nov
2016

Standard *ISO/IEC/IEEE 29119-5:2016, Part 5: Keyword-driven testing* is published. This underscores the growing importance of keyword-driven testing.

Jan
2018

Called **RoboCon**, the first annual Robot Framework Conference is organized in Helsinki, Finland.

Dec
2018

Robot Framework 3.1 is released. With this release, **Robotic Process Automation (RPA)** is supported. This means that the framework can be invoked to automate *tasks* and not just *tests*.

Robot Framework


# Chaos Engineering

## Summary

Distributed software systems are an integral part of today's world. Many organizations such as Amazon and Netflix are serving billions of users through such distributed systems. These systems are inherently complex and chaotic. They have many interacting parts whose collective behaviour can be unpredictable. Moreover, frequent deployments add to the uncertainty. The most significant weaknesses should be addressed proactively, before they affect customers in production and lead to loss of revenue.

**Chaos Engineering** addresses these challenges by injecting failures into the system in a controlled manner. We then observe how resilient is the system. Where we find significant flaws, we seek solutions. Chaos Engineering leads to more failure-resistant systems.

It's been said that,

> Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

## Discussion

### What's the context in which Chaos Engineering is relevant?

In a distributed system, even when each component is well-tested, there could be problems when these components come together. Distributed systems are by nature vulnerable to network connectivity issues of low bandwidth, high latency, packet loss or complete link failure. Each component can also fail in unexpected ways. It's hard to test for many real-world failures in a typical software development lifecycle.

It's possible to design a system to be fault tolerant. Even when one component fails, this should affect the system minimally. The system as a whole is designed to be better than the sum of its parts. But design alone isn't enough. We need to be confident that the system will perform despite failures. This is where Chaos Engineering becomes relevant.

Chaos Engineering starts by establishing normal steady state behaviour in terms of observable metrics. Then we inject failures in a controlled manner and record the same metrics. The hypothesis is that such failures don't affect the system. If the hypothesis fails, we know that we need to improve the system to handle those specific failures.

### What are the principles of Chaos Engineering?

The main principles of Chaos Engineering are as follows:

    + **Build a Hypothesis around Steady State Behavior**: Measure system output over a short period of time. This is the steady state. It indicates normal behavior. Hypothesize that the steady state will continue during experiments. Steady state should be based on measurable output, not internal attributes of the system. Throughput, error rates, latency percentiles, etc. could be metrics that define steady state behavior.
    + **Vary Real-world Events**: Inject real-world events such as server crashes, malformed responses, or traffic spikes. Observe what happens. Compare against the steady state. Any deviation disproves the hypothesis.
    + **Run Experiments in Production**: Prefer to experiment directly in production. This ensures authenticity and relevance of the currently deployed system.
    + **Automate Experiments to Run Continuously**: Running experiments manually is unsustainable. Automate the process.
    + **Minimize Blast Radius**: Ideally, experiments shouldn't affect customers. At least, minimize the negative impact on customers.

### Could you share more details into implementing Chaos Engineering?

Amazon uses number of orders as a metric since they found that page load times affect this metric . Netflix uses the number of times a user clicks play, which is affected by system failures. Given that metrics are central to Chaos Engineering, they should be easy to measure, report and visualize.

When metrics show problems, monitor how long it took to notice the problem, notify engineers, and do self-recovery. Do a post-mortem of every problem: what happened, what was the impact, why it occurred, and how to prevent it. Prioritize fixing these problems over developing new features.

Experiments could inject complete failures or affect performance in any component. Terminate the recommendation engine, the caching service or the load balancer. Increase inter-service latency or packet drops. Failures can even be at the UI layer. If a UI widget fails to load, do the other widgets expand to fill up the extra space? Get the whole team involved since each one is likely to propose a different type of experiment. Study the specifications to identify possible experiments.

### What are some myths about Chaos Engineering?

The following will describe and dispel some myths about Chaos Engineering:

    + **Chaos**: It's not about introducing chaos. Experiments are done in a controlled manner. Engineers select which systems should be affected and to what extent. They monitor continuously. If customers get affected, the experiment is stopped.
    + **Reliability**: Apart from adding new features, developers also care about reliability. Developers perform unit testing. Chaos Engineering complements their efforts by finding problems in real-world scenarios. But it's not about testing in production. We expect well-tested systems to come into production.
    + **Tooling**: Chaos Engineering doesn't require any specialized hardware/software tools. In most cases, access to the host OS or containers is adequate.
    + **Observability**: Basic metrics can suffice, often readily available in cloud platforms. We don't need to wait to implement complex metrics collection.
    + **Scale**: Chaos Engineering is not just for large distributed systems. It could be applied to even monoliths that need to be understood better.
    + **ROI**: Teams need to invest time upfront but this is worthwhile. It saves time later by minimizing production outages and troubleshooting. Chaos Engineering gives insights into optimal use of resources, increase productivity, and grow the business.

### What is Chaos Monkey?

Chaos Monkey is a software tool developed at Netflix that randomly simulates failures of production instances. In the world of microservices, it should be possible to lose an instance, and replace that with another instance without loss of application functionality or consistency. Instances are meant to be stateless; that is, they don't store data. Depending on the traffic load, instances are meant to be created and destroyed at will. Chaos Monkey validates this capability.

Imagine a wild weaponized monkey "randomly shooting down instances and chewing through cables." This is where we get the name Chaos Monkey. In the early years of Netflix, it was common practice to run Chaos Monkey during business hours, monitor for changes and identify weaknesses. This led to better auto-recovery mechanisms.

The myth that Chaos Engineering breaks things in production comes from the early use of Chaos Monkey. Today, we have many more tools to exercise Chaos Engineering. The randomness of Chaos Monkey is a useful starting point when the system's a black box. With better knowledge of how the system works, more fine-grained and controlled experiments can be done.

### What is the Simian Army and what are its different members?

The Simian Army is a suite of failure-inducing tools designed to add more functionalities beyond Chaos Monkey. Some tools inject failures while others take proactive actions to avoid future failures. Its members include:

    + **Latency Monkey**: It causes artificial delays or even service shutdowns in RESTful client-server communications.
    + **Conformity Monkey**: It finds out instances that don't adhere to predefined rule sets and shuts them down.
    + **Doctor Monkey**: It performs health checks (CPU load, memory usage, etc.) to detect unhealthy instances and removes them.
    + **Janitor Monkey**: It ensures that the cloud environment is running free of clutter and waste.
    + **Security Monkey**: It finds security violations or vulnerabilities and terminates the violating instances.
    + **10–18 Monkey**: It detects configurations and runtime problems in instances that are accessible across multiple geographic regions, involving multiple languages and character sets.
    + **Chaos Gorilla**: It simulates an outage of an entire AWS availability zone. Services should automatically rebalance to the other active availability zones.
    + **Chaos Kong**: It simulates region outages. An AWS Region has multiple availability zones.

### What are the benefits of Chaos Engineering?

Chaos Engineering has different benefits depending on the perspective:

    + **Customer**: Chaos Engineering predicts or prevents failures before they happen. This results in increased availability and durability of services.
    + **Business**: Chaos Engineering helps in preventing extremely large losses in revenue and maintenance costs, which can occur due to failures in production hours of businesses.
    + **Technical**: The insights from experiments help reduce incidents, increase understanding of system failure modes, improve system design and detect high-severity incidents faster.A 2021 study reported that increased availability, lower mean time to resolution (MTTR), lower mean time to detection (MTTD), fewer bugs in production, and fewer outages are some of the benefits. Those who practise Chaos Engineering can achieve more than 99.9% service availability.

In 2015, Amazon’s DynamoDB service experienced an availability issue in their US-EAST-1 region. This issue caused many other additional AWS services to fail. It resulted in the unavailability of some of Internet's biggest sites and applications. However, Netflix services were affected only in a minor way, thanks to their Chaos Engineering practices.

### Beyond Netflix, how has the industry adopted chaos engineering?

In 2019, a DevOps and Cloud InfoQ Trends report showed that chaos engineering was emerging from the "innovator adoption" stage to the "early adoption" stage. The primary adopters of chaos engineering have been e-commerce and big tech. Downtime directly impacts revenue in e-commerce. Big tech is about better returns from better reliability.

By 2020, many businesses from large financial institutions to health care organizations were adopting Chaos Engineering in their culture. Big organizations such as Uber, JP Morgan Chase and GrubHub have also adopted chaos engineering to maximize their service quality.

Singapore's DBS Bank tried out tools Pumba, Toxiproxy and Vizceral. Their approach was to do Chaos Engineering in non-production environments and collect only metrics in production. They still got many insights and made their applications more fault tolerant. They applied Chaos Engineering across the entire software development lifecycle.

### What are some best practices in Chaos Engineering?

Before starting on Chaos Engineering, design the system to be resilient at many levels: infrastructure, networking, data, application, people and culture. Adopt architectural patterns to achieve this. Three techniques adopted at Netflix are timeouts, retries and fallbacks.

When proposing a hypothesis, start with what you know and what you understand. Then move towards what you don't know and don't understand. A useful question to ask is, "What could go wrong?" Inject only one failure at the time.

A company can have a dedicated Chaos Engineering team of 2-5 engineers. But Chaos Engineering is not the responsibility of this team alone. It's been observed that some teams are quick to embrace it: Traffic Team (e.g. Nginx, Apache, DNS), Streaming Team (e.g. Kafka), Storage Team (e.g. S3), Data Team (e.g. Hadoop/HDFS), and Database Team (e.g. MySQL, Amazon RDS, PostgreSQL).

## Milestones

2000

In the early part of this decade, Amazon creates a program named *GameDay*. Its purpose is to inject failures into critical systems, and then use monitoring tools and alerts to obtain greater insight into how well Amazon's retail website responds to these failures. GameDay brings out architectural flaws and other defects. While initially successful, the program is later halted due to its impact on customer-facing services.

2010

Netflix Engineering Team creates **Chaos Monkey**. It's created in response to Netflix's move from physical infrastructure to AWS cloud infrastructure. Chaos Monkey is meant to test the capability that the loss of an instance doesn't affect the Netflix streaming experience. Chaos Monkey has its limitations. It requires Spinnaker and MySQL. It lacks recovery capabilities and a user inteface.

2011

After the success of Chaos Monkey, Netflix Engineering team develops **Simian Army**. The Simian Army adds additional failure injection modes allowing developers to test the system with different failures. In 2012, Netflix makes Chaos Monkey publicly available by sharing its source code on Github.

2014

Netflix decides to create a new role in organization: **the Chaos Engineer**. Bruce Wong coins the term and Dan Woods shares it with the greater engineering community via Twitter. In October, Netflix shares in blog post that they're using an internal solution called **Failure Injection Testing (FIT)**. FIT allows engineers to break things but control with precision the impact.

2016

Kolton Andrus and Matthew Fornaciari establishes *Gremlin*, the world's first **managed enterprise Chaos Engineering solution**. Gremlin becomes publicly available in late 2017 with multiple failure injection modes.

Oct
2016

At the *IEEE International Symposium on Software Reliability Engineering*, Netflix engineers present their internal platform that automates chaos experiments. They call it **Chaos Automated Platform (ChAP)**. It improves on FIT by routing traffic between control and experimental clusters. These clusters are created with the same configuration as production clusters. Changes in metrics are more easily observed and compared. Complex experiments can be conducted without impacting customers.

2018

Gremlin launches *Chaos Conf*, world's **first large-scale conference** on Chaos Engineering.

2020

Amazon Web Services (AWS) adds Chaos Engineering to the reliability pillar of the AWS Well-Architected Framework (WAF). AWS announces Fault Injection Simulator (FIS), a fully managed service for natively running chaos experiments on AWS services.

2021

Gremlin publishes the first ever *State of Chaos Engineering* report that shows how the practice of Chaos Engineering has grown among organizations, key benefits of Chaos Engineering, how often top performing teams run chaos experiments, and more.

Chaos Engineering


# Git Hooks

## Summary

Hooks in Git are **executable scripts** that are triggered when certain events happen in Git. It's a way to customize Git's internal behaviour, automate tasks, enforce policies, and bring consistency across the team.

For example, hooks can check that passwords or access tokens are not committed, validate that commit messages conform to an agreed format, prevent unauthorized developers from pushing to a branch, update local directories after a checkout, and so on.

Git supports both client-side and server-side hooks. Hooks can be written in any programming language though it's common to use Bash, Perl, Python or Ruby.

## Discussion

### What use cases can benefit from Git Hooks?

The figure illustrates local add/commit followed by a push to the remote repository. Also shown are the relevant hooks. Each hook is given a specific name. Hooks are invoked **by naming convention** before or after specific events.

The `pre-commit` hook is called when the commit procedure begins. This hook can format the code against the project's styling guide, run linting or execute unit tests. By checking for missing semicolons, trailing whitespace, and debug statements code reviews can focus on more important issues.  Hooks `prepare-commit-msg` and `commit-msg` can be used to format/validate the commit message.

After a successful commit, the hook `post-commit` could be used to send notifications. More commonly, when collaboration is via a remote repository, notifications are sent from the `post-receive` hook that's triggered after all references are updated. This hook can also deploy the latest changes to production.  The `update` hook that runs once per branch can be used to enforce access control, that is, only authorized users can push to a branch.

### What are client-side and server-side hooks?

Client-side hooks run on developer machines whereas server-side hooks run on the server that hosts the repository. Committing, merging, rebasing, applying patches, and checkout are common Git operations that developers do on their development machines. Any of these commands can trigger their relevant client-side hooks. Server-side hooks are triggered by the `git-receive-pack` command, which is a result of developers pushing changes to the remote repository on a cloud-hosted Git server.

Developers have full control of client-side hooks. Developers can choose to disable all client-side hooks if they wish. If a project's best practices need to be strictly enforced, server-side hooks are the way to do it.

Some Git repository providers impose restrictions on the use of server-side hooks on their cloud-based servers. For example, GitHub's free tier doesn't allow developers to define server-side hooks though GitHub Enterprise Server supports them. However, there are alternatives to achieve server-side automation such as webhooks, GitHub Apps/Actions, GitLab CI/CD, GitLab push rules, etc.

### What are pre- and post- Git Hooks?

Client-side and server-side hooks can be either `pre-` and `post-` hooks. The `pre-` hooks can be used to validate stuff before allowing the requested operation. For example, `pre-commit` hook executes at the start of a commit workflow. If the commit is deemed invalid for any reason, the hook can return a non-zero value. This terminates the commit operation. By returning zero, the hook states that the commit operation can continue, including the execution of other relevant hooks.

The `post-` hooks execute after the requested operation is done. Such a hook's return value doesn't matter since the operation is already complete.  For example, `post-commit` executes after a successful commit. These hooks are typically used to send emails or other notifications. They may also be used to trigger CI/CD pipelines. For example, `post-receive` could be used to deploy the updated code to production.

### What options help skip the execution of hooks?

It's possible to skip some hooks if their associated commands are called with certain options. Hooks `pre-commit`, `pre-merge-commit`, `prepare-commit-msg`, and `commit-msg` can be skipped if `commit` and `merge` commands are invoked with `--no-verify` option.

The hook `post-checkout` can be skipped if `clone` and `worktree add` commands are called with `--no-checkout` option. However, this hook is always called for `checkout` and `switch` commands.

### What essentials should a developer know to use Git Hooks?

By default, hooks are stored in `.git/hooks` within the project's repository. Git populates this folder with sample scripts that developers can use as a starting point. However, the file suffix `.sample` must be removed and scripts must be made executable. Developers can rewrite the scripts in their preferred language. Accordingly, the shebang (`#!` first line in script) must be updated with the correct path to the language.

If these sample scripts are deleted by mistake, we can retrieve them from Git templates folder. This may be `/usr/local/git-core/templates/hooks` (Linux) or `C:\Program Files\Git\mingw64\share\git-core\templates\hooks` (Windows). In fact, this copying from templates folder to the project's `.git/` folder is what happens when we run `git init`.

While hooks are triggered in response to specific events, we can also manually trigger the execution of a hook. The command for this is `git hook run <hook-name> <hook-args>`. For example, we could trigger one hook from another hook. This command is a low-level internal helper command.

Beginners can start by reading the official documentation on Git Hooks and Chapter 14 of Loeliger's book on Git.

### What does it take to create and maintain reusable Git Hooks?

It's a good practice to commit hooks as part of the repository so that all developers reuse and execute the same hooks. However, it's not permitted to store them within a `.git/hooks` folder within the repository. One approach is store them in another folder, say `hooks`, and then create a soft link from `.git/hooks` to `hooks` folder after cloning the repository. Another approach is to configure the path to the hooks with the command `git config core.hooksPath <path>`.

To reuse across projects, hooks can be in a separate repository and cloned locally. Configure the hooks path globally to that folder: `git config --global core.hooksPath <path>`. This configuration variable has been available since Git version 2.9.0.

There's also a framework named pre-commit to manage pre-commit hooks. This framework includes dozens of tested pre-commit hooks written in Python that developers can reuse in their own projects. These include syntax checking (`check-json`, `check-yaml`), fixing problems (`fix-byte-order-marker`, `trailing-whitespace`), protecting secrets (`detect-private-key`, `detect-aws-credentials`), enforcing policies (`no-commit-to-branch`, `forbid-new-submodules`), and more.

The githooks.com lists many useful hooks and related tools that developers can use.

### How do I configure my project environment to use Git Hooks?

No special configuration is needed other than updating `.git/hooks` folder or configuring `core.hooksPath` variable. However, many developer tools attempt to simplify this further.

In Maven, a plugin can setup the hooks path. In Gradle, we can have a dependent build script to do this setup. In Node.js, Husky project allows us to wire up all the hooks in the `package.json` file. In PHP, hooks can be set up in `composer.json` file via the `post-install-cmd` configuration.

In different languages, frameworks are available to manage hooks: pre-commit (Python), Overcommit (Ruby), Lefthook (Ruby or Node.js).

### What are some criticisms of Git Hooks?

Hooks change Git's default behaviour. If a hook does something unusual, it may confuse developers new to the project. Moreover, hooks can be buggy and make things counterproductive. For this reason, some recommend using Git aliases and invoking scripts directly rather than relying on hooks.

Git doesn't provide any easy mechanism to share hooks among developers or from one repository to another. No doubt this is more secure but at the expense of convenience and reusability. Hooks problem with installation, maintainability, and reusability is to some extent solved by frameworks such as *pre-commit*.

Hooks are powerful. Developers may be tempted to run unit tests before each commit. This can slow things quite a bit and discourage developers from committing often. This can lead to data loss.

Server-side hooks are not universally enabled by cloud-hosted Git repository service providers.

## Milestones

2005

Linus Torvalds creates Git as a replacement to BitKeeper that's a distributed version control system. Even in Git v1.0.0 (December), **hooks** are included as a feature.

Mar
2007

Hook `post-receive` is introduced to replace `post-update`. The latter is expected to be deprecated from v1.6.0. The hook `post-receive` is more useful since it has access to both old and new references; `post-update` knows only the new references.

Sep
2007

Git 1.5.4 is released. In this release, `git merge` command calls the `post-merge` hook following a successful merge.

Feb
2008

Git 1.6.0 is released. This release ships sample hook scripts with the **suffix** `.sample` since on some filesystems it's not sufficient to ship them as non-executables. In Windows, all scripts are executable by default. Developers can rename the scripts if they wish to enable the hooks.

Dec
2008

Git 1.6.1 is released. In this release, `git rebase` command when used without the `--no-verify` option calls the `pre-rebase` hook. This hook can be used to prevent the rebasing of a branch.

May
2009

Git 1.6.3 is released.  In this release, `git clone` command when used without the `--no-checkout` option calls the `post-checkout` hook. This hook is normally called after an update to the worktree, such as with the `git checkout` command.

Dec
2014

A **critical vulnerability** is discovered pertaining to Git hooks. Usually it's not possible to create a top-level folder named `.git` in a repository. However, in case-sensitive filesystems it's possible to create a folder named `.GIT`, `.Git`, etc. In Windows, which is case-insensitive, such a folder overwrites the default `.git` folder. By including malicious hooks, a hacker could execute arbitrary code on the client's machine. This threat is greatest for developers using third party or untrusted repositories. This vulnerability is patched within a week.

Apr
2015

Git 2.4.0 is released. This release adds a new server-side hook named `push-to-checkout`. By default, pushes to checked out branch would fail. With this hook, we can customize the behaviour by merging with or overwriting server-side changes.

Apr
2016

**GitHub Enterprise Server 2.6** is released. This includes the `pre-receive` server-side hook that can be used to enforce push policies and optimize workflows. However, GitHub has included server-side hooks since 2013, it's first hook tasked with warning users when a repository is renamed.

Jun
2016

Git 2.9.0 is released. The hooks directory can now be configured via the `core.hooksPath` configuration variable.

Nov
2019

Git 2.24.0 is released. This release adds a new hook named `pre-merge-commit`. This hook is called after a successful merge but before obtaining the commit message. It can be bypassed with `--no-verify` option to the merge command.

Feb
2022

On GitHub, it's noticed that pushes to repositories can take as long as 880ms on average. The implementation is in Ruby. A lot of time is spent loading Ruby dependencies. By rewriting the hook as a Go service, average processing time is brought down to 10ms. This improvement also becomes part of GitHub Enterprise Server 3.4.

Git Hooks


# Apache Log4j

## Summary

Apache Log4j is a logging framework for Java. A Java application can use Log4j to log important events during the course of its lifetime. The logs are later used for tracing application behaviour or performing audits. Unlike debugging an application using debuggers, log generation requires no user intervention apart from the initial calls to the logger API. Logs generated from production deployments give lot of useful context. Logs are a permanent record of what happened unlike debugging sessions that are transient.

Log4j is an open-source project managed by the Apache Software Foundation. Log4j 1.x is incompatible with Log4j 2. The latter has better design and richer features.

GUI-based tools are available to view and analyze Log4j logs. Log4j has inspired logging solutions in other languages.

## Discussion

### What's the architecture of Log4j?

`Logger` class is the starting point. Applications use the `LogManager` to return a logger object with a specific `LoggingContext`. If such a logger doesn't exist, it's created.

Every logger has a `LoggerConfig`. These config objects are initialized from configuration files. LoggerConfig has a class hierarchy, that is, one config class can inherit from another.

A LoggerConfig object is associated with one or more `Appender` objects that actually deliver log events. For example, a logger can log to both the console and to a file. Among the appender types are ConsoleAppender, FileAppender, RollingFileAppender, AsyncAppender, SocketAppender, SMTPAppender, and many more.  Because config objects have a hierarchy, events will be sent to appenders of parent classes as well.

Each appender can be configured to log messages in a specific format. Such formatting is the job of the `Layout` class.

A LoggerConfig object is associated with a log level. Levels enable automatic filtering of events. In addition, `Filter` objects can be applied to LoggerConfig objects and Appender objects. Filters give more flexibility at runtime. Only those events that pass the filters are send to appenders.

### What are the essential features of Log4j?

Log4j separates its API from implementations. Hence, applications can swap implementations while the code consistently calls Log4j API.

An application typically has many packages, classes, and methods. Log4j allows us to configure logging for each component separately. However, to save work on managing many configurations, configurations can be inherited and hence reused. Thus, `X.Y.Z` inherits and overrides some configuration from `X.Y`.

Log4j has log levels to control the scope of logging. Developers can also define custom levels. Likewise, custom Message types, Layouts, Filters and Lookups can be created. Because of a plugin architecture, Log4j is easily extensible.

Log4j has built-in support for JMX. Many parts of the Log4j system can be locally or remotely monitored and controlled.

For better performance, Log4j supports asynchronous logging. It does garbage-free or low garbage logging so that pressure on garbage collector is relieved. For better concurrency, it locks resources at the lowest possible level. Log4j won't lose events when appenders are reconfigured. Hence, it can be also be used for audit logging.

### What log levels are available in Log4j?

A log level is assigned to a `LoggerConfig` object. If not, the object inherits the level from one of its ancestors along the class hierarchy. Log4j defines the following built-in levels: ALL (Integer.MAX\_VALUE), TRACE (600), DEBUG (500), INFO (400), WARN (300), ERROR (200), FATAL (100), and OFF (0). Applications can also define custom levels. Levels OFF and ALL are not generally used in API calls.

Every event is logged with an indication of its level. For example, if the application sees an unexpected scenario it may log it as WARN; log it as ERROR if the transaction can't be completed; log it as FATAL if application can't proceed further without a reboot or reinitialization.

We can control how much to log without changing the code. For example, with log level ERROR only errors or more series events will be logged. So the call `logger.error()` will be logged but not `logger.info()`. During debugging, log level can be changed to DEBUG to log more messages.

Log level can be dynamically changed at runtime: `Configurator.setLevel("com.example.Foo", Level.DEBUG);`

### How can I format Log4j messages?

Log4j supports a number of logging formats, what are formally called as **Layouts**: CSV, GELF, JSON (pretty or compact), JSON Template, Pattern, RFC5424 (enhanced Syslog), Serialized (deprecated), Syslog (BSD Syslog and used in Log4j 1.2), XML (pretty or compact), and YAML.

If location information is logged (filename, line number) or location-specific patterns are used (`%C` or `%class`, `%F` or `%file`, `%l` or `%location`, `%L` or `%line`, `%M` or `%method`), there's a performance impact, especially for asynchronous loggers. Default configuration excludes location information.

For the pattern layout, message formatting relies on the *conversion pattern*, which is similar to `printf` formatting in C. For example, the conversion pattern `"[%-5p] %d{yyyy/MM/dd HH:mm:ss,SSS} %t %c: %m%n"` logs right padded five-character width logging level (`%-5p`), datetime in a specific format (`%d{pattern}`), thread name (`%t`), logger name (`%c`), log message (`%m`) and newline (`%n`). The "-5" in "%-5p" is a format modifier. Modifier "20.-30" implies left pad to 20 characters or truncate from the end to 30 characters.

### What are Log4j API essentials that a developer should know?

The main package defining the Log4j API is `org.apache.logging.log4j`. Public message types are in package `org.apache.logging.log4j.message`. An implementation is available in package `org.apache.logging.log4j.simple`.

Essentially, if you're developing a library you should use the API, not the concrete implementation. The application that uses your library will determine whether to use `org.apache.logging.log4j.simple` or another implementation of the Log4j API.

Log4j API has four main interfaces:

    + `LogBuilder`: To construct log events before logging them.
    + `Logger`: Main interface of the `log4j` package. Class `org.apache.logging.log4j.simple.SimpleLogger` implements this interface.
    + `Marker`: Adds filterable information to log messages. Markers are hierarchical. For example, "Error" marker can have children "SystemError" and "ApplicationError".
    + `ThreadContext.ContextStack`: ThreadContext Stack interface.Among the classes are `EventLogger`, `Level`, `LogManager`, `MarkerManager`, `ThreadContext`, and more. `LogManager` is the anchor point for the Log4j system. Methods of this class include `getContext()`, `getLogger()`, `getRootLogger()`, and more.

`LoggingException` is thrown when logging error occurs.

For detailed information, consult the complete Log4j API documentation.

### What's SLF4J and is it better than Log4j 2?

SLF4J is a "simple logging facade", which is really a logging interface that applications can call. SLF4J forwards API calls to a concrete implementation used by the application. Such an implementation could be `java.util.logging`, Log4j, Logback, etc. A logging implementation can conform to SLF4J API and thereby enable developers to easily switch implementations. We may say that SLF4J does for logging what ORMs have done for database interfacing.

Both Log4j 2 and SLF4J separate the API from the implementation. This gives developers flexibility. To use Log4j, application code can call SLF4J API that are then routed to Log4j's specific implementation. An alternative is to make Log4j calls but use the *log4j-to-slf4j* adapter to route calls to any SLF4J-compliant library. Hence, calling Log4j API directly in code doesn't tie you down to only Log4j.

In fact, Log4j has some advantages over SLF4J. SLF4J can log only strings but Log4j can also log objects (implementations do the necessary conversions). Log4j supports Java 8 lambda expressions. Log4j has better support for garbage-free logging.

### What are some best practices when using Log4j?

Logging gives us insights into application use and helps in troubleshooting. Too much logging can affect application's performance. Too little logging can be ineffective. Logs should include descriptive messages along with sufficient context to make them useful.

Don't log sensitive information such as passwords, credit card numbers or access tokens.

Use suitable log levels for your messages and use them consistently.

When an exception in logged, include the exception object for more context, such as `try {...} catch (IOException ioe) { LOGGER.error("Error while executing main thread", ioe); }`. This will include the stack trace, which can be very useful.

Logging in JSON can be better for storing logs in a log management system. JSON is also better for multiline messages such as stack traces. An appender can be configured for logging in JSON.

Where performance is critical, consider logging asynchronously. Logging directly over a network may not be ideal since network errors can cause some log entries to be lost. Inside containers, log files are not permanent. Hence, log to the standard output or over a network to a centralized system.

### What other projects are inspired from or associated with Log4j?

Apache Log4j comes under an umbrella project named *The Apache Logging Services Project*. Apart from Log4j, this includes Apache Log4j Kotlin and Apache Log4j Scala. Both these are APIs in their respective languages for Log4j. There's also Apache log4cxx and Apache Log4Net that are ports of Log4j for C++ and .NET respectively. Apache Log4j Audit is meant for audit logging.

Other ports include log4c (C); log4js, JSNLog (JavaScript); log4perl (Perl); Apache log4php (PHP); and log4r (Ruby). Log4j can also be used in languages Clojure and Groovy.

Apache Chainsaw is a GUI-based log viewer. It simplifies the analysis of logs via colour coding, tooltips, filtering, search function, navigation, etc. In fact, there are many other log viewers out there: OtrosLogViewer, Lilith, Eclipse LogViewer Version, Splunk, LogSaw, LogMX, and more.

Complete log management is possible with SolarWinds Log Analyzer, Sematext Logs, Datadog, Fluentd, LogDNA, Splunk, Elastic Stack, and more.

### As a beginner, how do I get started with Log4j?

The official Apache Log4j 2 project site is an ideal place to get started. It has links to the user's guide, articles, tutorials and FAQ. It also hosts the project's Javadoc API documentation.

Installing and configuring Log4j 2 is also described. You need to update project dependencies in file `pom.xml` used by Maven builder. Log4j 2 configuration can be in XML, JSON, YAML or properties formats. This file would include the appenders, their pattern layouts, the loggers, and the mapping between loggers and appenders.

Documentation informs how to configure Log4j for other builders including Maven, Ivy, Gradle, and SBT. Project dependencies are also listed.

Migration guides are available for those who wish to migrate from Log4j 1.x to 2.x.

## Milestones

1991

James Gosling and others start work on a new language for interactive television. They name it *Oak*. In 1995, this is renamed to *Java*. The first public release of the language happens in 1996.

1996

The E.U. SEMPER project creates its own tracing API for Java, which later evolves to *Log4j*. In these early years of Java, there's no universal logging framework or library. Every application includes its own logging or tracing API.

Jan
1999

The Apache Software Foundation adopts Log4j under its *The Apache Logging Services Project*. This project "creates and maintains open-source software related to the logging of application behavior." First versions of Log4j appears in October.

Jan
2001

**Apache Log4j v1.0** is released. Package hierarchy starts at `org.apache.log4j`. In subsequent years, Log4j proves to be the first of its kind to gain traction in the Java world for logging. It's also credited for introducing hierarchical logging, a concept that other loggers go on to adopt.

Feb
2002

Sun Microsystems releases Java 1.4 that includes `java.util.logging` (available since mid-2001 in its beta release). Its API appears to be similar to Log4j. Open source implementations of this API for Java 1.2 and 1.3 become available.

May
2002

**Apache Log4j v1.2** is released. This release is maintained till May 2012 when v1.2.17 is released. In August 2015, Apache announces **end of life for Log4j 1.x**.

Jul
2012

Work starts on **Apache Log4j 2**. It's a complete rewrite of Log4j and it's incompatible with Log4j 1.x. It's inspired by Log4j 1.x and `java.util.logging`. Log4j v2.0 released in July 2014. Package hierarchy changes in Log4j 2 to `org.apache.logging.log4j`.

Sep
2017

Java 9 is released. This **breaks compatibility for Log4j 1.x** due to a feature called Mapped Diagnostic Context (MDC). MDC's equivalent in Log4j 2 is Thread Context Map. Applications using Log4j 1.x but without MDC may still work. But given Log4j 2's many features, developers will find it worthwhile to migrate to it.

Nov
2021

A member of the Alibaba Cloud Security Team finds a vulnerability in Log4j 2. The team informs Apache. Called **Log4Shell**, the vulnerability is made public in December. Given the widespread use of Java and hence Log4j 2, this is considered a series vulnerability affecting cloud servers, enterprise applications, IoT devices, routers, firewalls, printers, etc. Apache fixes the issue in releases 2.15.0, 2.16.0 and 2.17.0 (Dec 2021). Meanwhile, websites and businesses list affected software and mitigation strategies.

Apache Log4j


# Local Area Network

## Summary

Local Area Network (LAN) is a computer network used for connecting computers within a limited geographical area such as offices, schools and buildings. Interconnected LANs dispersed over a wider area is called a Wide Area Network (WAN).

A LAN consists of cables, switches, routers, and bridges that together enable devices to connect and communicate with one another. Devices connected in a LAN have shared access to the interconnecting medium. When the medium is made of physical wires or cables, we called it **Wired LAN**. If the medium is radio waves we called it **Wireless LAN (WLAN)**.

The most common technology used for wired LANs is Ethernet. For wireless LANs, IEEE 802.11 standards are used and their commercialized technology is called Wi-Fi.

## Discussion

### What are the characteristics of a LAN?

LAN is a low-cost and effective network type capable of connecting multiple devices on a single transmission medium so that every device in the network can communicate with one another.

LAN coverage is over a limited extent, from meters to a few kilometres. The quality of data transmission in LAN is comparatively higher than other network types due to the shorter coverage area. Switches or routers are used to construct a LAN network most of the time. This ensures higher data transmission speeds and reliability.

Setting up a LAN network can be done at low costs. If there's a need for expansion, it can be done quickly.

### What are the different LAN types?

Based on the communication mode, LAN can be generally classified into two types:

    + **Client-Server LAN**: In this mode, only a single central server is used to connect with multiple clients through wired or wireless medium. If a client device needs to send a packet to another client, communication happens via the central server.
    + **Peer-to-Peer LAN**: In this mode, there's no need for a central server. Clients can communicate with each other directly. Sending a file is easier and faster, thus enabling higher speeds.

### What's the difference between Wireless LAN and Ethernet LAN?

Wireless LAN as the name suggests uses radio waves to transmit data whereas Ethernet uses wires to transmit data. Data transfer or communication through Ethernet LAN is more stable and faster, though both have their benefits.

The particular Radio Frequency (RF) spectrum used in WLAN determines the coverage. Signal propagation and hence radio coverage are affected by walls, metal objects and even people. Coverage can be increased with wireless repeaters, bridges and access points. Data rates are steadily increasing for WLAN with the recent Wi-Fi 6 release (Feb 2021) that supports a maximum throughput speed of 9.6Gbps across multiple channels with 75% lower latency.

Ethernet LAN primarily uses electrical signals to transmit data through cables. It has less interference than WLAN. Unshielded Twisted Pair (UTP) cables are commonly used to establish Ethernet connections. 10 Gigabit Ethernet (10GbE) is the latest standard that can offer a maximum throughput speed of 10Gbps.

### Could you describe some limitations of LANs?

LAN networks are only suitable for transmission over a limited distance. The cost of installation rises as the coverage area expands.

LAN requires security to avoid threats from malware and viruses. Since computers are interconnected, any security breach can impact the entire network. Apart from securing devices physically, LAN security can be compromised by misconfigurations including weak passwords and improper allocation of devices. Obviously WLANs are easier to attack because physical access to network devices is not required. However, even in wired LANs, physical access to LAN sockets in hallways and reception areas can pose risks. It becomes important to keep the entire network under a policy guideline to avoid possible threats.

### What type of devices are commonly found on a LAN?

A LAN has many types of devices:

    + **Host**: End device (client or server) that can send and receive data.
    + **Repeater**: Operates at the physical layer. Receives data on one port, boosts the signal and sends it out on another port.
    + **Hub**: Essentially a multi-port repeater. All devices connected to the hub belong to the same LAN segment. A packet coming on one port is forwarded to all other ports. More devices connected to a single hub will result in packet collisions and drop in throughput.
    + **Bridge**: Connects two or more LANs in the network. Isolates its ports to accommodate more devices. Bridges forward frames between different LAN segments by looking at MAC addresses. It works at physical and data link layers.
    + **Switch**: Similar to a bridge but more sophisticated. Whereas a bridge typically implements functionality in hardware, a switch does it in software.
    + **Router**: A layer 3 or network layer device. It routes packets by looking at IP addresses. Essential for connecting to the Internet.
    + **Gateway**: Interconnects to another network type, either LAN or WAN, by translating between their protocols.

### How does Virtual LAN differ from a normal LAN?

Virtual LAN (VLAN) increases the capabilities of a normal LAN. It's a logical way of separating traffic between multiple logical or virtual networks that exists physically on the same network. VLANs are mostly used by organizations with a large number of devices.

In a normal LAN, a broadcast message reaches all nodes whether they need it or not. In a VLAN, only a subset of nodes that belong to that VLAN (aka broadcast domain) will receive the broadcasted packet. Thus, VLANs mitigate packet flooding and network congestion.

VLANs minimize security risks by reducing the number of hosts that receive a packet. Moreover, hosts that hold sensitive information can be on a separate VLAN. VLANs enable flexible network configuration, such as, grouping hosts by department rather than physical location. VLANs can be easily reconfigured just by changing port configurations.

Implementation of VLAN will be cost-effective as it's built using switches. Routers are needed only for sending data outside the LAN.

### Could you describe the different types of LAN topologies?

Topology in networking refers to the way of devices are interconnected. Some major LAN topologies are:

    + **Bus Topology**: Also known as the backbone network. Simplest and widely used network design that needs only a single cable for connectivity. A device known as a terminator is placed at both ends to avoid signals reflecting back to each other.
    + **Star Topology**: A central device acts as a hub and connects to all other devices. Data on the network passes through devices in between before reaching the destination.
    + **Ring Topology**: Nodes are arranged in a ring topology. Data passes through each node before reaching the destination. Traffic can be bidirectional and the chances of packet collisions are less. Broken network links affect bi-directional communication.
    + **Mesh Topology**: Nodes are connected to each other in point-to-point configuration. Each node has multiple paths to reach another node. If one path fails, an alternate path is used, making this topology more fault-tolerant.

### What are the different data transmission methods used in LANs?

In communications, data can be transmitted in a few ways:

    + **Unicast**: It's a one-to-one data transmission method that typically uses a connection-oriented protocol such as TCP (Transmission Control Protocol). A web server that transmits or streams data to a single user via a unique connection is an example of unicast.
    + **Multicast**: It's a one-to-many data transmission method that uses a connectionless protocol such as UDP (User Datagram Protocol). A simple example is an email addressed to a specific group of recipients.
    + **Broadcast**: It's a one-to-all data transmission method. Data originates from a single point and is sent to all users simultaneously. Cable TV is an example of a broadcast network.In LAN, unicasting is predominant with many applications using it (FTP, HTTP, Telnet). Multicasting is implicit in the way a hub operates. VLAN is also multicasting in some sense. Explicit multicasting is possible via the IGMP protocol. Broadcasting is usually employed when a node wants to discover something, such as in ARP and DHCP protocols.

### How does a LAN interface with a WAN?

Routers connect LAN devices to WAN and then the wider internet. Most wireless routers come with two types of physical ports labelled as LAN and WAN. A WAN port is mostly connected to a modem that makes the router capable of accessing the internet. LAN port can be used to connect with devices that don't have Wi-Fi support. For secure communications, the LAN and WAN ports are internally separated by a firewall. Devices connected via WAN ports can't access the LAN devices unless port forwarding is configured.

One of the essential operations done by a router is called **Network Address Translation (NAT)**. NAT is designed to conserve IP addresses. Rather than give every device in the world a unique IP address, LAN devices have unregistered IP addresses that are valid only within that LAN. Router acts as an intermediary and presents to the internet a single public IP address. Router translates between internal private addresses and the public address.

### Could you describe some recent advancements in LAN technology?

LAN has seen significant development since its inception in the 1970s and has grown exponentially with changing demands and needs of users. Availability of next-generation WLAN cloud services, 5G, IoT, smart buildings, and cabling trends are transforming the industry.

**Single-Pair Ethernet (SPE)** has become the trend for being faster and cost-effective. It provides transmission speeds up to 1Gbps with a single pair of twisted copper wires rather than the classic four-pair cables or RJ45 connectors.

**All over IP** is an approach to building automation with IP alone. It's an integrated approach that brings together Ethernet/IP cabling, Power over Ethernet (PoE), and WLAN. Since all devices understand IP, there's no need for protocol translation in between. Such standardization improves reliability and availability.

Adoption of Passive Optical LAN (POLAN) cable has changed the fundamental LAN architecture by replacing copper cables with single-mode fiber optic cables, providing higher bandwidth and numerous other advantages.

## Milestones

1972

Dr. Robert M. "Bob" Metcalfe and his colleagues at Xerox PARC develop the first experimental Ethernet network called **Alto Aloha Network** for connecting with Altos, a personal workstation.

May
1973

Metcalfe renames Alto Aloha Network to **Ethernet**. He clarifies that this network can support any device. In a memo titled *Alto Ethernet* he gives a schematic representation in which personal computers can connect to a printer via a coaxial cable.

1974

Cambridge University initiates the **Cambridge RING Project** for communication of devices through the *Laboratory* within the campus. Information flows on twisted pair cables between the printers and computers.

Jul
1976

David Boggs and Bob Metcalfe publish a paper titled *Ethernet: Distributed Packet Switching for Local Computer Networks*. The same year Ethernet becomes an open networking standard funded by three companies Xerox, DEC and Intel.

Dec
1977

The **first commercial LAN** is installed by Datapoint Corp. at Chase Manhattan Bank in New York. It uses the Attached Resource Computer (ARC) network using a token ring scheme for Ethernet.

Dec
1984

The Institute of Electrical and Electronics Engineers (IEEE) publishes the standard **IEEE 802.3** that specifies the Carrier Sense Multiple Access with Collision Detection (CSMA/CD) access method, which is essential for the working of a LAN.

1991

Kalpana Corporation launches a new device called **LAN switch**. Each port can run at full capacity simultaneously, something not possible with hubs. This is later called an Ethernet switch and it provides high-throughput communications.

1995

**Fast Ethernet (FE)** is introduced with transmission speed up to 100Mbps to meet the increasing the needs of users. It's specified by the IEEE 802.3u standard.

1997

First version of **Wireless Local Area Network (WLAN)** standards IEEE 802.11 is published with two raw data rates of 1 or 2Mbps.

Jun
2003

**Power over Ethernet (PoE)** is defined by the IEEE 802.3af and 802.3at standards. It can pass electric power along with data on twisted pair Ethernet cable. PoE can provide 48 volts over a 4-wire or 8-wire twisted pair. It's useful at locations without a power outlet.

Dec
2013

In WLAN, the standard IEEE 802.11ac is defined. It operates at the 5GHz band with support for MU-MIMO (Multi-User MIMO). This allows access points to communicate with multiple devices simultaneously.

Dec
2017

**400GbE Ethernet** interface is approved by IEEE under the IEEE 802.3bs standard. It can transmit data up to 400Gbps, which is four times faster than 100Gbps standard while also consuming less power.

Feb
2021

The IEEE 802.11ax standard or **Wi-Fi 6** is launched. It provides more speed and greater stability on a higher bandwidth channel than its predecessors. MU-MIMO in Wi-Fi 6 can work in both 2.4GHz and 5GHz bands.

Local Area Network


# IoT Operating Systems

## Summary

The use of operating systems for IoT hardware is often categorized into two groups: end devices and gateways. End devices or nodes are often lot smaller in capability as compared to gateways. As more and more processing is pushed to the network edges (to gateways and nodes), traditional devices that used to run without an OS are embracing new OS implementations customized for IoT.

While IoT OS are an evolution of embedded OS, IoT brings its own additional set of constraints that need to be addressed. A mix of open source and closed source IoT OS exist in the market. Since IoT is varied in terms of applications, hardware and connectivity, we expect the market will sustain multiple OS rather than just a couple of them.

## Discussion

### Do IoT end devices require an OS in the first place?

While having an OS is not mandatory, devices are growing in complexity. This complexity is due nodes having more sensors, more data processing and increased connectivity to send that data out. Some devices have rich user interfaces that might include graphical displays, face recognition and voice recognition. End devices that were often based on 8-bit or 16-bit MCUs are moving to 32-bit architectures as costs drop and complexity increases. Addressing these changes without an OS is not only a challenge but also inefficient. The use of OS, particular an RTOS, simplifies the job of application programmers and system integrators because many of the low-level challenges are taken care of by the OS.

As a thumb rule, a system that consumes less than 16KB of RAM and Flash/ROM does not require an OS. Such systems most often run on 8-bit or 16-bit MCUs. With such systems, we can get away with a single event loop that polls and processes events as they occur. But if more complexity is added to the system, the response times will be limited by the worst-case processing time of the entire loop.

### What are the parameters for selecting a suitable IoT OS?

The following parameters may be considered for selecting an IoT OS:

    + Footprint: Since devices are constraint, we expect OS to have low memory, power and processing requirements. The overhead due to the OS should be minimal.
    + Scalability: OS must be scalable for any type of device. This means developers and integrators need to be familiar with only one OS for both nodes and gateways.
    + Portability: OS isolates applications from the specifics of the hardware. Usually, OS is ported to different hardware platforms and interfaces to the board support package (BSP) in a standard way, such as using POSIX calls.
    + Modularity: OS has a kernel core that's mandatory. All other functionality can be included as add-ons if so required by the application.
    + Connectivity: OS supports different connectivity protocols, such as Ethernet, Wi-Fi, BLE, IEEE 802.15.4, and more.
    + Security: OS has add-ons that bring security to the device by way of secure boot, SSL support, components and drivers for encryption.
    + Reliability: This is essential for mission-critical systems. Often devices are at remote locations and have to work for years without failure. Reliability also implies OS should fulfil certifications for certain applications.

### What parameters are important for selecting a suitable IoT OS?

The short answer is that there's no universal subset of important parameters. While there are many parameters for selection, some of them may be more important than others depending on the hardware type and application. For example, a small memory footprint may be important for end devices but not so for gateways. Compliance to standards may be important for an industrial application but not so for a hobby project. Someone looking at only ARM-based hardware might choose ARM mbed at the expensive of platform portability. A device that has access to a power supply might not expect power optimization from its OS; whereas a battery-powered device might expect the OS to do power management so that the battery can last for 10 years.

### What certifications might an IoT OS require?

This is dependent on the vertical. The following is a non-exhaustive list:

    + DO-178B for avionics systems
    + IEC 61508 for industrial control systems
    + ISO 62304 for medical devices
    + SIL3/SIL4 IEC for transportation and nuclear systems

### What are the open source IoT OS?

The following is a non-exhaustive list: TinyOS, RIOT, Contiki, Mantis OS, Nano RK, LiteOS, FreeRTOS, Apache Mynewt, Zephyr OS, Ubuntu Core 16 (Snappy), ARM mbed, Yocto, Raspbian.

Some of these have come from academic institutions. TinyOS and Contiki are among the oldest. RIOT is more recent and has an active community of developers. FreeRTOS is among the popular ones. LiteOS is from Huawei. ARM mbed is single-threaded, event-driven and modular. It's has good connectivity and low footprint. In Zephyr OS the kernel is statically compiled, which makes it safe from compile time attacks. With Ubuntu Core, rather than having a monolithic build, kernel, OS and apps are expected be packaged and delivered as snaps. Android Things, previously named Brillo, is Google's offering. Yocto is not exactly an embedded Linux distribution. It's a platform to create a customized distribution for your particular application.

### What are the closed or commercial IoT OS?

The following is a non-exhaustive list: Android Things, Windows 10 IoT, WindRiver VxWorks, Micrium µC/OS, Micro Digital SMX RTOS, MicroEJ OS, Express Logic ThreadX, TI RTOS, Freescale MQX, Mentor Graphics Nucleus RTOS, Green Hills Integrity, Particle.

Windows 10 IoT comes in three flavours: IoT Enterprise, IoT Mobile Enterprise and Core. TI RTOS and Freescale MQX target the respective chipsets. Where reliability and safety are important, some of these commercial systems are preferred, particularly in aerospace, automotive, healthcare and industrial systems. Windows 10 IoT and Particle are examples that enable easy integration to cloud services.

### What are the popular IoT OS out there?

From the IoT Developer Survey 2018 conducted by Eclipse Foundation, it was found that 71.8% of the respondents like or use Linux-based OS. Within Linux, Raspbian takes the lead. Windows and FreeRTOS follow at 22.9% and 20.4%. However, the sample size in this survey was small.

It's interesting that 19.9% of developers prefer bare-metal programming. **Bare-metal** is a term that's used when no OS is used. Bare-metal is preferred for constrained devices. It's a cheaper option but development and support costs may increase. If the device's processing, memory and power requirements can allow it, Embedded Linux is preferred. It will have a shorter time-to-market, better security, wider support base and well-tested connectivity solutions.

### Do IoT OS need to be real-time OS?

RTOS will be required where data has to be processed within time constraints, often without buffering delays. Where multiple threads are required that have to meet deadlines and share resources effectively, RTOS will be needed.

There will also be a class of devices that may not have strict real-time constraints. For some, a richer user interface may be more important. Some may buffer data and transmit them occasionally to save power. Such devices need not have an RTOS and may adopt a simpler OS. However, a survey from 2015 has shown that many designers who choose an OS mention real time as one of the top reasons.

Designers may choose to use multiple processors where it makes sense. For example, an 8-bit MCU may be used to interface to sensors and actuators; a 32-bit processor will run the RTOS for connectivity and multithreading.

### What are typical memory requirements for IoT OS?

Sensor nodes will have less than 50KB of RAM and less than 250KB of ROM. Contiki requires only 2KB of RAM and 40KB of ROM. Similar numbers are quoted for Mantis and Nano RK. Zephyr requires only 8KB of memory. Apache Mynewt requires 8KB of RAM and 64 KB of ROM. It's kernel takes up only 6KB. Communication protocols typically take up 50-100KB of ROM.

In all cases, the numbers will increase when more components are added as required by the application. For example, one experiment with SMX RTOS showed 40KB/70KB of RAM/ROM for a node; the same OS for a gateway came up to 200KB/300KB.

Devices based on Ubuntu Core, Windows 10 IoT, Android Things and MicroEJ are likely to require memories in the order of gigabytes. Gateways will typically have a footprint in the order of gigabytes. This is because of extra functionality that they are require to do: device management, security, protocol translation, retransmissions, data aggregation, data buffering, and so on.

### What design techniques are used by IoT OS?

TinyOS and ARM mbed use a monolithic architecture while RIOT and FreeRTOS use a microkernel architecture. ARM mbed uses a single thread and adopts an event-driven programming model. Contiki uses protothreads. RIOT, FreeRTOS and µC/OS use a multithreading model. In static systems (TinyOS, Nano RK), all resources are allocated at compile time. Dynamic systems are more flexible but also more complex. File systems may not be required for the simplest sensor nodes but some OS support single level file systems. With respect to power optimization, this is done for the processor as well as its peripherals.

## Milestones

1999

The first implementation of TinyOS happens within UC Berkeley. It is released to public in 2000.

2003

Adam Dunkels releases Contiki. Dunkels goes on to found Thingsquare in 2012.

2013

RIOT is released to the public. The origins of RIOT lie in FeuerWare (2008) and µkleos (2010).

Jun
2016

Irish firm Cesanta releases 1.0-rc1 version of open source OS named **Mongoose**. Version 2.3 is released in June 2018. Mongoose makes it easy to connect your IoT devices to the cloud and do over-the-air (OTA) updates. Microcontrollers supported include STM32F4, STM32F7, ESP32, ESP8266, CC3220, and CC3200.

Oct
2016

Ubuntu Core 16 beta version, based on snaps, is released.

Dec
2016

Google rebrands Brillo as Android Things. Brillo was announced earlier at Google I/O 2015.

Nov
2017

Amazon announces Amazon FreeRTOS. Based on the FreeRTOS kernel, this makes it easier for developers to connect their IoT devices locally or to the cloud, to make their devices more secure and do over-the-air updates.

IoT Operating Systems