# Carrier Aggregation

## Summary

Carrier Aggregation (CA) is a feature that allows a mobile device to send and receive packets from multiple carriers (aka frequencies) at the same time. Carrier aggregation as a feature is available in Wi-Fi, 3G HSPA+, LTE-Advanced and 5G networks. The many carriers are controlled by a single access point (in Wi-Fi) or base station (in cellular systems).

The figure shows an example of CA in LTE. LTE has a maximum carrier bandwidth of 20 MHz. Via CA, five carriers are combined to achieve an aggregated bandwidth of 100 MHz. CA brings higher data rate, lower latency, better user experience, and better use of network resources.

CA is possible in downlink and uplink, symmetric or asymmetric. Carriers can be aggregated across FDD and TDD bands, licensed or unlicensed bands.

## Discussion

### Could you explain intra-band versus inter-band CA?

There are broadly two types of CA:

    + **Intra-Band CA**: The carriers belong to the same band. Carriers can be contiguous (next to each other) or non-contiguous. CA with contiguous carriers is easier to implement. Operators most likely don't have large contiguous spectrum. Hence, non-contiguous CA is a useful feature.
    + **Inter-Band CA**: The carriers belong to different bands. This is more complex to implement than the intra-band contiguous type. Because cellular operators have licenses to different bands, this has led to the standardization of many inter-band CA combinations to cater to industry needs.

### What technologies use carrier aggregation?

In HSPA+, 8 DL and 2 UL carriers can be aggregated. HSDPA Multiflow enables aggregation across cells controlled from the same RNC.

In LTE-Advanced, up to five carriers can be aggregated for a maximum bandwidth of 100 MHz. Each carrier can use a different channel bandwidth. In LTE-Advanced Pro (Rel-13), up to 32 carriers can be aggregated for a maximum bandwidth of 640 MHz. Carriers from unlicensed bands could be aggregated.

In 5G, intra-band and inter-band CA is possible within FR1, within FR2, and across FR1 and FR2. Up to 16 carriers can be aggregated. Maximum aggregated bandwidth is limited to 1 GHz. Each aggregated carrier can be of a different numerology.

In both LTE and 5G, Supplemental Downlink (SDL) carriers provide additional throughput. This is nothing but CA. On the contrary, Supplemental Uplink (SUL) is not CA. In general, a UE can be assigned more downlink than uplink carriers.

Wi-Fi's channel bonding is intra-band and contiguous. OFDM symbols from adjacent channels are combined to yield a single channel. Hence, this isn't CA as used in HSPA+/LTE/5G. However, Wi-Fi 7 introduced multi-link operation in which a device can aggregate channels from 2.4/5/6 GHz bands.

### What are the possible deployment scenarios of CA?

The figure show five possible deployment scenarios with CA using carriers F1 and F2 (assume F2 > F1):

    + **Scenario 1**: F1 and F2 are in the same band and have similar coverage with co-located cells. This scenario provides higher capacity and more bandwidth to the UE.
    + **Scenario 2**: F2 is in a higher band and hence has smaller coverage. F1 provides coverage while F2 improves the throughput.
    + **Scenario 3**: Cells are co-located but F2 is directed to cover the holes in F1. F1 provides coverage and mobility while F2 improves the throughput.
    + **Scenario 4**: F1 provides macro cell coverage while F2 Remote Radio Heads (RRHs) improve throughput at hot spots. F1 and F2 can be non-contiguous carriers in the same band or inter-band carriers.
    + **Scenario 5**: Similar to scenario 2, but frequency-selective repeaters are used to improve F2 coverage.The above scenarios suggest that the benefits of CA are many: higher throughput, better cell coverage, consistent user experience, higher energy efficiency, and higher ROI for operators.

### What's the basic terminology used in LTE and 5G NR carrier aggregation?

Each aggregated carrier is called a Component Carrier (CC). One of them is called Primary Component Carrier (PCC) and the rest are called Secondary Component Carriers (SCCs). The respective cells are called PCell and SCells. Both PCell and SCell are considered as serving cells. There's one HARQ entity per serving cell.

PCC is essential since cell search/selection, system information acquisition and initial random access happen on it. PCC is UE-specific; that is, different UEs can have different carriers as their PCC. In the uplink, only the PCC carries PUCCH (unless two PUCCH groups are configured). Uplink SCCs provide only PUSCH. In the downlink, PCC carries both PDSCH and PDCCH. Downlink SCCs carry PDSCH and optionally PDCCH.

CA can be used together with Dual Connectivity (DC). A UE may be connected to a Master Cell Group (MCG) and a Secondary Cell Group (SCG). CA can happen in each cell group.

### How are CA band combinations named?

Figure shows examples of LTE CA. *CA\_1C* operates in band 1 with 101-200 number of aggregated resource blocks for CA class C. The maximum aggregated channel bandwidth is 40 MHz from two CCs. Moreover, *CA\_1C* has two **Bandwidth Combination Sets (BCSs)**. Each BCS allows for a certain combination of channel bandwidth in MHz per CC: (15 + 15, 20 + 20) for BCS0 and (5/10/15 + 20, 20 + 5/10/15/20) for BCS1.

*CA\_25A-25A* indicates non-contiguous intra-band CA, hence "25A" is repeated. *CA\_1A-5A* indicates inter-band CA involving bands 1 and 5, both applicable to CA class A.

In 5G NR, CA bandwidth classes are defined separately for FR1 and FR2. We note a few CA NR band combinations:

    + Intra-band Contiguous in FR1: *CA\_n78B* (20 + 50)
    + Intra-band Non-Contiguous in FR1: *CA\_n77(3A)* (3 CCs, 2 BCSs, 1 combination per BCS)
    + Inter-band in FR1: *CA\_n1A-n77A*, *CA\_n1A-n77(3A)* (2 non-contiguous CCs in n77)
    + Inter-band in FR1 and FR2: *CA\_n66-n77-n260* (3 bands), *CA\_n1-n3-n8-n77-n257* (5 bands)Band combinations can be browsed online: LTE CA, NR FR1, NR FR2 and NR FR1-FR2

### Which LTE or 5G NR protocol layers are affected by CA?

The main impact of CA is on MAC and PHY. RLC needs to have larger buffers to handle higher data rates but otherwise RLC is unaffected by CA. RRC signalling is modified. For example, once the UE is connected on the PCell, SCells are set up via RRC signalling.

MAC distributes the RLC PDUs among the many CCs. Each CC handles one transport block per TTI. Each CC is associated with it's own HARQ entity, though all CCs are handled by a single MAC entity.

### What are some additional features of LTE and 5G NR CA?

With **cross-carrier scheduling**, the UE monitors a single PDCCH (for example, on the PCC). This will contain the scheduling information for all the CCs.

Feedback signalling such as HARQ acknowledgements of all CCs are sent on the uplink of the PCell. This suits the asymmetric design of CA in downlink and uplink. However, this may overload the single uplink carrier carrying the signalling. It's therefore possible to configure **two PUCCH groups**. The cell carrying the signalling in the second group may be called Primary Second Cell (PSCell).

In Rel-16, **unaligned frame boundary** and **Tx switching** are allowed. This provides the UE more transmission opportunities and utilize effectively both its Tx chains. In Rel-17, the PUCCH carrier can be **dynamically selected** to reduce latency. Moreover, HARQ retransmissions can be on a different carrier.

In LTE Rel-15 and 5G NR Rel-16, **enhanced utilization of Carrier Aggregation (euCA)** was introduced. This allows the UE to perform early measurements so that SCCs can be activated quickly when RRC connection is resumed from an inactive state.

## Milestones

2009

HSPA+ Rel-8 is released. This introduces a feature called **multicarrier** (aka dual-carrier). Two adjacent 5 GHz carriers can be combined to offer a UE higher data rates. Rel-9 (2010) allows non-adjacent carriers to be combined this way. It also brings dual-carrier feature to uplink. This expands to four DL carriers in Rel-10 (2011) and eight DL carriers in Rel-11 (2012). With eight carriers, a UE is able to obtain about 168 Mbps on 40 MHz of aggregated bandwidth, and 672 Mbps when combined with 4x4 MIMO.

2011

Rel-10 of LTE-Advanced comes out. This is the first LTE release to introduce CA (intra-band contiguous and inter-band). This standardizes 5 x 20 MHz CA, though in practice it's limited to only two carriers. Rel-11 (2012) of LTE-Advanced introduces support for intra-band non-contiguous CA. Rel-13 (2016) of LTE-Advanced Pro allows 32 CCs that can accommodate unlicensed bands as well. Overall, CA in LTE along with other enhancements (256QAM and 4x4 MIMO) enables *Gigabit LTE*.

Mar
2015

Rel-12 of LTE-Advanced reaches functional freeze. This release includes the capability to aggregate carriers across FDD (paired spectrum) and TDD (unpaired spectrum) bands. This benefits operators who have licenses to both FDD and TDD bands. In fact, operators demonstrated FDD+TDD CA at the MWC 2014.

2021

3GPP starts to focus on the specifications for 5G Standalone CA in FR1 with more than two CCs. Meanwhile, Ericsson claims to support CA in both 5G SA and NSA, with DL 8CC and UL 2CC in mmWave band, 2CC in mid-band, and 3CC in low-band. Qualcomm supports CA via its Snapdragon X65 5G modem.

Mar
2022

Rel-17 reaches stage 3 functional freeze. In 5G NR, **PUCCH Carrier Switching** is possible whereby the PUCCH carrier can be dynamically selected to reduce latency. HARQ retransmissions can use a different carrier from the original transmission.

May
2022

Nokia and Optus achieve 3 CC CA over a 5G SA network. This uses the bands 2100 MHz (FDD), 2300 MHz (TDD) and 3500 MHz (TDD). They state that their Samsung S22 customers will soon be able to experience this in commercial service. In August, Nokia and BT Networks demonstrate 4 CC CA over a 5G SA network. Four carriers combined are in the bands 2.1 GHz, 2.6 GHz, 3.4 GHz and 3.6 GHz.

Oct
2022

It's seen that most LTE networks use 2-band CA combinations. The two most popular combinations are *CA\_3A-7A* (1800 MHz and 2600 MHz) and *CA\_3A-20A* (1800 MHz and 800 MHz), achieving an aggregated bandwidth of 40 MHz. The most popular 3-band combination is *CA\_3A-7A-20A*, achieving an aggregated bandwidth of 60 MHz.

Carrier Aggregation


# Dependency Manager

## Summary

A good practice in software design is to build software from smaller, single-purpose modules that expose well-defined interfaces. This is also in the spirit of software reuse. With the widespread adoption of open source software, often our own applications or modules depend on those written by others. Thus, to build our software we need to bring in all parts on which it depends, including language libraries and remote third-party modules.

But it's not trivial to ensure that we have all necessary dependencies, particularly when dependencies themselves depend on others. This is why we need a **Dependency Manager**, often invoked during the software build process. A useful definition states that,

> Dependency management is a technique for declaring, resolving and using dependencies required by the project in an automated fashion.

## Discussion

### Isn't dependency manager same as a package manager?

No. A package manager operates at the system level and most often deals with binaries. It's role is to install libraries, tools, and other applications. It affects the entire system and not just one project. Because of this, it requires root access. Examples of package managers are apt-get (Linux), Brew (MacOS), and Chocolatey (Windows).

A dependency manager operates at source code level. It's role is to setup the right dependencies for a particular application. It therefore implies that a dependency manager is something used by developers, not users or system admin.

Thus, a dependency manager is project specific. It helps in easy management of project dependencies. It has to ensure that same source code is used across environments, for example, when moving from Windows developer machine to Linux production system. There should be end-to-end visibility of dependencies. A dependency manager is essential for Continuous Integration. A good dependency manager should also make it easy to publish new packages.

### What are the main components of a dependency management system?

A dependency management system typically has the following components:

    + **Module**: A software artefact, that we wish to use in our project. Depending on the language, modules are also called *packages* or *libraries*. Modules come with *metadata* to indicate dependencies.
    + **Manifest**: Usually a file that specifies immediate/direct dependencies for our project. It specifies intent, such as, saying that we desire "version 2.3 or above of module A".
    + **Lock**: Usually a file that captures a particular realization of the developer's intent. To extend the above example, version 2.3.2 of module A might be used.
    + **Repository**: A place where modules are stored. Often repos are remote and accessed over the internet. Dependency manager usually knows the location of the repo but location can also be specified in the manifest file.
    + **Dependency Constraint**: Usually in the manifest file, this is a constraint about versions of a module that are allowed for this project.
    + **Resolution Rule**: Rules are applied by the dependency manager to arrive at a suitable module version. Selecting the right version is called *Dependency Resolution*.

### What's the usual Project Dependency Management (PDM) pipeline?

Dependency managers bring together all necessary project dependencies before passing them to the compiler or interpreter. In this sense, their work is somewhat a preprocessing phase before the actual build happens.

Dependency managers start by reading the manifest file, in which direct dependencies are noted. They then read the metadata of these dependencies from their repositories to figure out the next level of dependencies. In other cases, they may download the dependencies right away and then process their dependencies. Either way, all dependencies must be downloaded and installed.

Exact installed versions of dependencies are recorded in a lock file. While the manifest file is important for a developer, the lock file has greater importance for operations folks. The lock file makes the project's dependencies reproducible in any environment.

### What is transitive dependency?

Transitivity is a concept in logic and mathematics. In the context of software dependencies, it can be explained as follows. If module A depends on module B, which in turn depends on module C, we can say that module A also depends on module C. Many levels of dependencies produce what's called a **Dependency Graph**.

When a module appears multiple times in the graph, with different version numbers, the dependency manager must take a decision about which version to use. A similar decision must also be made when dynamic version numbers are specified. This selection process is called **Dependency Resolution**.

A good dependency manager will not only resolve direct dependencies as mentioned in your project's top-level manifest file but also resolve all transitive dependencies. It should be able to handle any level of nesting. Typically, cyclic dependencies will cause problems.

### Could you mention some rules used in Dependency Resolution?

We mention a few sample rules below for specific dependency managers. However, similar rules are likely to exist across many dependency managers.

In Gradle, after taking into account all transitive dependencies and constraints, the highest suitable version is selected. If there's a conflict, an error is thrown. You can also use dependency configurations and make manual adjustments. Maven uses the nearest version but if conflicts are at the same depth, the first one wins based on the order of declaration.

NuGet uses four main rules: lowest applicable version, floating versions, nearest-wins, and cousin dependencies. This rule is applied with a warning since the package at a deeper level might get downgraded to an older version.

NPM actually builds a dependency tree, where different versions of dependent packages can be installed in different branches of the tree. However, using what's called "peer dependencies", NPM allows us to enforce same versions where desired.

### What's the purpose of lock files used by dependency managers?

If dependencies are specified in terms of exact versions, anyone else trying to build your project in any other environment will use the same versions.

When dynamic version numbers are specified in manifest files, such as a range of versions, the version used may depend on the environment or the latest version available in the repository at build time. Dynamic version numbers are good because they allow us to bring in latest features/patches of dependent modules into our project. However, this causes uncertainty for continuous integration and production release. To solve this, versions used are recorded in a lock file. This lock file must be committed to version control so that anyone else using the project is guaranteed to use the same versions.

What if you use only exact versions? Is lock file required in such a case? Yes. Even if direct dependencies are exact, transitive dependencies might be specified in a non-exact manner. Therefore, a lock file is essential in all cases.

Lock file is sometimes called "a manifestation of the manifest".

### What should I look for when choosing a dependency manager?

The purpose of a dependency manager is to allow developers create a consistent and reproducible set of dependencies for their projects. It should be easy to use, have a well-defined behaviour and hooks for customizations. Some essential features to compare are the following:

    + **Organization**: For example, Yarn uses flat mode while NPM builds a dependency tree.
    + **Semantic Versioning**: Has provision to specify a range of versions.
    + **Dependency Locking**: Uses lock files to reproduce exact versions.
    + **Dependency Resolution**: Handles transitive dependencies. Resolves conflict with a well-defined set of rules.
    + **Multiple Configurations**: Allows different versions of the same package. This is enabled by defining different configurations or groups, such as build vs testing.
    + **Performance**: Caches downloaded packages for faster updates. Parallel downloads.
    + **Security**: Does integrity checks via package checksums.
    + **Customizations**: Dependencies can be made optional or versions can be manually specified. Can be configured to fetch from any repository. Supports offline installation.
    + **Support**: Has good community support and adoption, with frequent updates.

### What are some best practices that developers can adopt to better manage project dependencies?

Developers need to version control not only the source code but also the manifest file and the lock file. In particular, always version control your lock file if you're building an application. If you're building a library, it's a good idea to do this as well. However, if you wish to build your library with the latest versions of its dependencies, it's recommended to use a second continuous integration build.

If you're making a Java library, let it not depend on frameworks such as Spring. Changes to frameworks can cause "dependency hell" for users of your library. Don't ship fat JAR files, files that include all dependencies. Only depend on what you need.

Finally, it's best to avoid dependencies if you can implement them yourself. The argument is that this will keep your application lean and secure. Check if there's a native library to do the equivalent. Look for those with less transitive dependencies. Check memory usage and load time. Use fewer high-level interfaces. Prefer low-level function calls. Use abstractions. Test regularly on target environments.

### Could you give examples of dependency managers in various languages?

A few examples of dependency managers are listed below in the format *name: (language, default repo, default manifest, default lock)*:

    + **Composer**: (PHP, Packagist, `composer.json`, `composer.lock`)
    + **Gradle**: (Java, Maven Central, `build.gradle`, `gradle/dependency-locks/*.lockfile`)
    + **Node Package Manager (NPM)**: (NodeJS, NPM, `package.json`, `package-lock.json`)
    + **Yarn**: (NodeJS, NPM, `package.json`, `yarn.lock`)
    + **Bundler**: (Ruby, RubyGems, `Gemfile`, `Gemfile.lock`)
    + **dep**: (Go, -, `Gopkg.toml`, `Gopkg.lock`)
    + **Cargo**: (Rust, Crates.io, `Cargo.toml`, `Cargo.lock`)
    + **Pipenv**: (Python, PyPI, `Pipfile`, `Pipfile.lock`)In many languages, newer, better dependency managers are being preferred over older ones: Gradle over Maven and Any-Ivy (Java); Yarn over NPM and Bower (JavaScript); Bundler over RubyGems (Ruby); Composer over PEAR (PHP); dep over godep, glide and govendor (Go). In some cases, such as dep in Go, there's no default repository. The manifest file includes information on package sources.

Gradle did not initially support locking but locking could be implemented using Netflix's Nebula Plugins project. This plugin inspired Gradle to support locking. While `build.gradle` specifies dependencies at the top level, module metadata files (.module, .pom or ivy.xml) must also be read.

### In which languages are dependencies more often used?

One study of 6.9 million versions of open source packages across 14 repository showed that NPM and CPAN have higher dependency counts; all others have 5 or less dependencies, on average. NPM and CPAN also have a larger spread. In general, higher the count, more complex is the dependency graph. While using third-party packages saves development time, there's also risks in terms of security, licensing and performance. Application health depends on all dependencies that we bring in.

Each repository contains packages of a particular language: NPM (JavaScript), CPAN (Perl), Rubygems (Ruby), Maven (Java), Packagist (PHP), Cargo (Rust), Atom (JavaScript), Pub (Dart), CRAN (R), PyPI (Python), NuGet (.NET/C#/F#/Visual Basic), Elm (Elm), Hex (Erlang), Dub (D). Libraries.io tracks open source packages across different repositories. Since Go language doesn't have a default repository, Go Search provides a handy search functionality.

## Milestones

Mar
2002

Based on Project Object Model (POM), **Maven 1.0-beta-2** is released for managing Java projects. Maven 1.x reaches end of life in 2014. As a rewrite of Maven 1.x, with extra functionality, **Maven 2.0** is released in October 2005. Popular Maven plugins are also converted to the new version. Maven 2.0 can manage transitive dependencies.

Jul
2009

For managing dependencies for Java, including Android projects, **Gradle** v0.7 is released. Version v1.0 follows in June 2012. Version v4.8 is released in June 2018. Version v4.8 includes support for dependency locking.

Sep
2010

As a dependency manager for Ruby, **Bundler** V1.0.0 is released. Earliest release can be traced to 2009. Ruby community is credited with introducing the concept of *dependency locking*.

Oct
2010

While being compatible with Maven 2.x projects and plugins, **Maven 3.0** is released. Among other things, it isolates project and plugin dependencies, and does better dependency resolution.

Mar
2011

Version v1.0.0rc9 of **Node Package Manager (NPM)** for NodeJS/JavaScript is released. Release v0.0.1 can be traced to January 2010.

Sep
2011

As a dependency manager for Xcode projects (Objective-C or Swift), **CocoaPods** 0.0.1 is released. Version 1.0.0 is released in May 2016. An alternative to CocoaPods is **Carthage** that trades configuration for convention.

Jul
2013

Version 1.0.0-alpha1 of **Composer** is released as a dependency manager for PHP. Version 1.0.0 is released in April 2016. Composer has been inspired by Node's NPM and Ruby's Bundler. Composer uses **Packagist** as its default repository but for internal/Intranet use, **Satis** is an alternative.

Sep
2014

To overcome the limitations of NuGet, **Paket** is released as a dependency manager for .NET. NuGet can be seen as a package manager. It doesn't handle transitive dependencies and selects the latest version when there are conflicts. Paket solves these problems of NuGet.

Aug
2015

With initial code commits from March 2014, **Cargo** version 0.4.0 is released as a dependency manager for Rust. As of June 2018, version 0.28.0 is the latest available.

Jun
2016

Version 0.2.0 of **Yarn**, is released for dependency management for NodeJS/Javascript. Version 1.0.0 is released in September 2017. For performance, Yarn caches downloaded packages and does downloads concurrently. It uses checksums to verify integrity of downloaded packages before installing them. It's inspired by Bundler, Cargo and NPM.

May
2017

Version 5.0.0 of **NPM** is released for NodeJS/JavaScript. This version introduces dependency locking. In July, version 5.1.0 is released to fix a problem that ignored changes to packages.json when lock file is present.

Jul
2017

For managing dependencies for Go, **dep** v0.2.0 is released.

Dependency Manager


# 3D Modelling

## Summary

A 3D model is a geometric representation of an object or a surface in a simulated 3D space. 3D modelling is about creating such models using computer software. Designers design completely new models or feed scanned images of existing objects into the computer to create 3D models.

3D modelling is an art. It enables one to visualize the final output before building it. It also allows model derivations for detailed viewing.

3D modelling is the bedrock of 3D printing. It evolved with the 3D printing industry that grew big in the 1990s. Almost every industry uses 3D modelling in some form. Video game development, filmmaking, architecture and engineering industries actively employ 3D modelling. It offers vital support for Industry 4.0 technologies to achieve a fully immersive interactive experience in virtual environments. Forecasts suggest the 3D modelling market will touch $7.6 billion by 2025.

## Discussion

### How is 3D modelling put to use?

The crux of 3D modelling is visualization and creativity. Modellers can create 3D models of a person, atmosphere or object of their choice. Using 3D modelling, one can easily understand complex designs. It allows modellers to fix drawing errors quickly. Modellers can design data-based models quite accurately.

Different modellers use 3D modelling for different purposes. Engineers and designers use it to design new products or redesign existing products. Architects build visual demonstrations of buildings, landscapes and interiors using 3D modelling. Animation and game designers use it to create digital characters and assets. Filmmakers use it to produce visual effects.

Medical industries 3D model designs for manufacturing prosthetics, dental implants and medicines.

3D modelling offers cutaway visualization, surface curvature measurement and multispectral analysis. These are techniques to engage with materials in academic fields.

Additive manufacturing technologies use 3D modelling to construct models for 3D printing or manufacturing items in layers.

Augmented Reality (AR) and Virtual Reality (VR) developers apply 3D modelling to add a sense of reality to the non-existent realities they create.

### Can you discuss some important types of 3D modelling?

**Solid Modelling** drafts models with simple shapes such as spheres, cubes and prisms. It suits best for models involving flat surfaces and simple curves of constant radii. It produces mathematically correct final outputs.

**Wireframe Modelling** stitches the vertices of shapes in a web to make models. Each shape will have three or more vertices. So the web is a network of polygons. The triangle is its smallest polygon. The higher the number of triangles, the closer the 3D model will be to realism.

**Surface Modelling** follows guiding lines to define the shape and curvature of a 3D part. Software tools estimate the smooth surfaces and connect these guiding lines. Surface modelling handles complex shapes. It can produce visuals impossible to replicate in the real world.

Some other important methods are *Polygonal Modelling*, *Sculpting*, *Box Modelling*, *Non-Uniform Rational Basis Spline (NURBS)*, and *Non-Uniform Rational Mesh Smooth (NURMS)*.

Modellers may convert wireframe models into 3D views and add surfaces. It takes specific modelling operations. For example, *Boolean Operations* help to combine objects, subtract objects and create intersections. *Parametric Modelling* is the best fit to create design intent between dimensions, parts and assemblies.

### Can you describe the typical workflow of 3D modelling?

3D modelling starts with conception. Designers use reference images, real-world objects or existing concepts to conceive a model output.

Planning for execution is the next step. The main idea is to fix the model's geometry, proportions and shapes before jumping in.

In the execution stage, 3D modellers or artists use software to create primitive shapes that conform to the planned design. These shapes may not serve their best interests yet. So they add details to refine the design. And the model so refined will be of high polygon count.

Since high-poly models are unsuitable for animation, modellers do a retopology on them. *Retopology* means simplifying complex models by reducing their polygon count. Low-poly models will need texturing. *Texturing* includes applying colours, textures, and material attributes to the surface of the 3D model. Put another way, texturing clothes the 3D model with 2D images called *Texture Maps*. Modellers consider how lights hitting on the 3D object's surface reflect its colour and surface texture.

### What are some 3D modelling terms you need to know?

We note a few essential terms:

    + **3D**: The three spatial dimensions are width, height and depth.
    + **Axis of motion**: It's the imaginary line an object follows in a 3D space.
    + **Axis of rotation**: It's the line in a 3D space about which an object rotates.
    + **Beveling**: Modifying the edges of the 3D objects under design is beveling. It adds a hint of realism to the models.
    + **Extrusion**: An extrusion tool creates 3D objects by forcing a third dimension on a two-dimensional model.
    + **Nodes**: Nodes in 3D modelling are software functionalities packaged into atomic units. Nodes organized in a 3D environment are called node graphs. Modellers manipulate these node graphs via programmable APIs or visual interfaces by hovering a mouse. For instance, a *rotate node* produces rotated 3D textures. Similarly, a *pattern node* mixes patterns to form patterned textures.
    + **Sculpting**: Modellers sculpt objects on a 3D display using software tools. They perform operations such as pushing, pulling, pinching, and grabbing to manipulate shapes.

### What are some hardware considerations for 3D modelling?

Due to rendering, not every computer can perform 3D modelling efficiently. Rendering adds specifications and design enhancements to yield presentable finished model. It requires hardware that can handle intense graphics and heavy software applications.

Some general component requirements are:

    + **Central Processing Unit (CPU)**: Select multiple cores if you need rendering. A single core would be sufficient if you are happy with 3D modelling alone.
    + **Graphics Processing Unit (GPU)**: It processes graphics and complex calculations. It gives good quality outputs and consumes less processing time. Opt for a GPU that suits you best. Currently, NVIDIA GTX 1060 or higher graphics in the NVIDIA series works best for modelling.
    + **Random Access Memory (RAM)**: Ensure you have at least 8GB for smooth operations.
    + **Hard Disk Storage**: Storage capacity will depend on the project size. However, adding a Solid State Device (SSD) storage boosts performance speeds.Apart from these, a monitor, a computer case, a power supply, cooling systems and other peripherals are essential. If the minimum requirements are not met, your system might work slowly. It may even freeze or crash. Overall, it will hamper productivity and frustrate you.

### What are the software applications required for 3D Modelling?

3D modelling involves the following software:

    + **Platform**: We need a browser or an operating system (OS) to run 3D software applications. The OS requirements differ for different file formats and use cases. Most software applications run on top of Windows, macOS, iOS, Linux, or Android.
    + **Specialized Software Applications**: For example, Tinkercad, Meshmixer, and SketcUP Free specialize in direct modelling. SculpGL, Leopoly and ZbrushCoreMini are useful for sculpting. Vectary is good for mesh modelling and parametric modelling. Blender specializes in sculpting.
    + **Extension System**: Designers store 3D designs in standard file formats such as STL, IGS and OBJ. Devices compatible with these formats can access the designs.
    + **Component Library**: This feature offers a single source of truth to store and retrieve design components. It lends consistency to designers working on the same project. Design developers who prefer code components to images will find it helpful.

## Milestones

1963

Ivan Sutherland develops and releases the first 3D drawing software called Sketchpad. It runs on a Lincoln TX-2 machine designed in 1956. Sketchpad becomes a pioneering contributor in the Human-Computer Interaction (HCI) field. It's a forerunner of the CAD software that we use today. It later plays an important role towards the development of Computer Graphics (CG). Graphic designers use sophisticated object-oriented programming (OOP) and Sketchpad to create Graphic User Interfaces (GUIs).

1964

IBM and General Motors join hands to develop DAC-1 (Design Augmented by Computer). It speeds up car production and provides quick and good-quality visualization.

1971

Donald P. Greenberg produces an early, sophisticated CG movie at the General Electric Visual Simulation Laboratory. He calls it Cornell in Perspective.
 Elsewhere, Dr Patrick J. Hanratty designs and introduces Automated Drafting And Machining (ADAM). It's a CAD software application. His company, Manufacturing and Consulting Services (MCS), supplies CAD software to major companies of the time, such as McDonnell Douglas.

1974

Donald P Greenberg founds the Program of Computer Graphics (PCG) at Cornell University, New York. Greenberg's later works become foundational for many of the Computer Graphics applications in the market today.

1975

Dr Kenneth Versprille introduces Non-Uniform Rational B-Splines (NURBS) to CAD while working under ComputerVision. NURBS finds wide application in later years.

1980

The U.S. National Bureau of Standards introduces Initial Graphics Exchange Specification (IGES). It becomes the standard file format to move CAD files between different software. STEP replaces it in 1994.

1982

John Walker founds Autodesk inc and introduces AutoCAD for computers. It's a handy tool for Architects and Engineers. AutoCAD would later introduce a 3D design that combines CAD tools and Building Information Modelling (BIM) for simulation and visualization.

1988

Martin Newell designs Cobalt (CAD Program). It's an early parameter-based CAD program and 3D modelling. It runs on both Macintosh and Microsoft Windows Operating Systems.

1992

Juergen Heimbach sets up Cadenas, an engineering firm that provides software solutions to 3D models. Cadenas' digital catalogues permit access to component information from anywhere in the world.

1994

ISO Standard for the Exchange of Product Data (STEP) takes over as the new format for exchanging 3D data. It's a widely accepted international standard. STEP file is also addressed as ISO 10303-11.

1995

MIT Blackjack member, Jon Hirschtick, launches SolidWorks. It's a 3D CAD software that works on Windows platforms. It's easy to use and affordable.

2010

Autodesk releases AutoCAD 360. It's the mobile version of AutoCAD that enables designers to work from anywhere.

2012

Autodesk moves CAD to Cloud. Cloud enables several teams to work on the same design simultaneously.

2015

Mindesk's co-founder, Gabriele Sorento, presents a prototype that facilitates a Virtual Reality (VR) Interface for CAD software.

2019

Autodesk releases CAD to AR inventor app. It permits the view of inventor models in Augmented Reality (AR).

3D Modelling


# Wi-Fi MAC Layer

## Summary

IEEE 802.11 MAC sub-layer is responsible for coordinating access to the shared physical air interface so that the Access Point (AP) and Wi-Fi stations in range can communicate effectively. MAC takes data from a higher sub-layer called LLC, adds header and tail bytes, and sends them to lower physical layer for transmission. The reverse happens when receiving data from the physical layer. If a frame is received in error, MAC can retransmit it.

Multiple access is based on carrier sensing, channel contention and random backoff. Due to contention, a Wi-Fi network with a large number of active stations can suffer from low throughput and high latency. IEEE 802.11e and its subset Wi-Fi Multimedia attempt to alleviate this problem.

## Discussion

### What's the purpose of MAC layer in 802.11?

MAC stands for "Medium Access Control", which implies that it's main function is to control access to a shared medium. The air interface is a shared medium through which all multiple Wi-Fi stations and access point (AP) attempt to transfer data. MAC implements the control mechanisms that allow multiple devices to reliably communicate by sharing the medium as specified in the standard.

Formally, MAC functions include scanning, authentication, association, power saving and fragmentation.

When it comes to the actual handling of data, MAC sits as a sub-layer within Data Link Layer. It takes a packet of data called *MAC Service Data Unit (MSDU)* from Logical Link Control (LLC) sub-layer. MAC adds necessary header and tail bytes to form *MAC Protocol Data Unit (MPDU)*. MPDU is then sent to the physical layer for transmission. The reverse flow happens when MAC receives a packet from physical layer.

### How does a Wi-Fi station discover and associate with an AP?

There are two modes of discovery and the station is at liberty to use either or both:

    + **Passive scanning**: The station looks for beacon frames that are regularly sent by an AP. These frames contain essential information about the network. If there are multiple APs within range, it selects the strongest one. Station then attempts to connect to the AP. Communication happens on the channel in which the AP is operating.
    + **Active scanning**: The station sends a probe request either to a specific AP or to any AP within range. Either way, it expects a probe response from one or more APs, selects the strongest and attempts to connect to that AP. Since no prior beacon is decoded, station will send its probe on all channels.

### What multiple access scheme is used in 802.11?

MAC uses what's called Point Coordination Function (PCF) or Distributed Coordination Function (DCF). The latter is more prevalent in industry and employs *Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA)*.

There are two parts to CSMA/CA as suggested by the name:

    + **Carrier Sensing**: Wi-Fi stations must first sense the air interface. Only if the channel is idle for a specified amount of time (of duration DIFS), they can transmit. DIFS stands for DCF InterFrame Spacing.
    + **Collision Avoidance**: If channel is sensed busy, the station waits till it's free for DIFS duration, then waits for a random interval before transmitting. This randomness reduces the chances that two waiting stations end up transmitting at the same time. "Collisions" can still occur and they are inferred when no acknowledgement is received by the sender. If ACK is not received, station backs off for a random duration and repeats the process again.

### What's the structure of 802.11 MAC frames?

There are three types of MAC frames: data, control and management. Only data frames contain higher layer data. Control frames include ACK, RTS, CTS, Power Save Poll, etc. Management frames help stations discover and connect to AP.

All frames start with a 2-byte *Frame Control* that specifies the frame type and further details. All frames end with a 4-byte *Frame Check Sequence (FCS)* that's used to detect errors. Frame Control is part of what's called the MAC header, which can vary in length. For data frames, MAC header and FCS are added by MAC layer around the MSDU received from LLC. MSDU is often called MAC payload or frame body. Further details on MAC frame are available online.

### Are there techniques in MAC to improve the reliability of communication?

MAC uses many techniques and here we describe the important ones:

    + **Error Detection**: Every MAC frame contains 4-byte FCS that's used to detect errors.
    + **Retransmission**: For a corrupted frame, receiver will not send an acknowledgment frame. Sender will therefore retransmit the frame.
    + **Fragmentation & Reassembly**: When frames are larger, there's a higher chance of collision. MAC can therefore fragment data/management frames and transmit them as individual MAC frames. Any errors, only fragments need to be retransmitted.
    + **RTS/CTS**: *Request-to-Send* and *Clear-to-Send* are used to solve the hidden-node problem. When stations are widely spread around an AP, some may be hidden from others on the wireless channel. RTS/CTS handshaking alleviates this problem and reduces collision. This is an optional feature.

### What are the MAC enhancements due to High Throughput (HT)?

IEEE 802.11n introduced changes to MAC that are part of High Throughput (HT). The idea was to reduce MAC overhead and increase overall throughput. This is done using *Frame Aggregation*. Two types are possible:

    + **Aggregate MAC Service Data Unit (A-MSDU)**: Multiple MSDUs are packed into a single MPDU that in turn maps to a single transmission frame. In this case, all MSDUs share the same MAC header.
    + **Aggregate MAC Protocol Data Unit (A-MPDU)**: Multiple MPDUs are carried within a single transmission frame. Obviously, each MPDU has its own MAC header.These two are not mutually exclusive: a single transmission can have a combination of A-MSDUs and A-MPDUs, sometimes called *two level frame aggregation*. In fact, research has shown that a combination of the two gives best performance on both 802.11n and 802.11ac devices.

MAC frame is extended: new header field called HT Control plus larger payload size. *Block Acknowledgments (BACK)* is introduced.

### What are the MAC enhancements due to Very High Throughput (VHT)?

IEEE 802.11ac introduced changes to MAC that are part of Very High Throughput (VHT). VHT also impacts deployments in 60 GHz band such as for 802.11ad.

Frame aggregation can happen like in 802.11n but in larger sizes: MPDU size of about 8KB (11n) vs 11KB (11ac); PSDU size of about 64 KB (11n) vs 4 MB (11ac). MAC frame is updated: HT Control is defined for 11ac plus larger payload size. All frames are sent as A-MPDUs even if they contain only one MPDU.

### I've heard of the terms "lower MAC" and "upper MAC". What are these?

Part of MAC that interacts with LLC sub-layer is sometimes called *upper MAC* while the part that interacts with PHY layer is called *lower MAC*. Thus, MSDUs and A-MSDUs are processed in upper MAC while MPDUs and A-MPDUs are processed in lower MAC. In general, lower MAC deals with more time-critical operations such as MPDU transmissions and acknowledgements. For example, association of stations with AP is done by upper MAC but actual medium access (time-critical) is handled by lower MAC.

It's common for lower MAC to be implemented in hardware such as in FPGA.

## Milestones

2005

IEEE 802.11e introduces QoS enhancements to MAC. Hybrid Coordination Function (HCF) is introduced. MAC header includes a 2-byte QoS Control field.

Oct
2009

IEEE 802.11n is released and it includes changes to MAC.

Jan
2014

IEEE 802.11ac is released and it includes changes to MAC.

Wi-Fi MAC Layer


# OpenAPI Specification

## Summary

Services and applications expose interfaces that we call APIs. Every service can define its APIs in any way it wants provided those APIs are properly documented. However, there are benefits in following a standard to define or describe an API. This is exactly where the OpenAPI Specification (OAS) becomes relevant.

OAS defines the rules, syntax and semantics for describing a RESTful API. OAS is language-agnostic, that is, it can be used for writing code in any programming language. It's in a format that can be read by both humans and machines. It's self-contained: no additional source code or documentation is required to understand an API specified in OAS.

OAS is vendor-neutral and maintained by the *OpenAPI Initiative (OAI)* under the Linux Foundation. The specification document uses Apache License V2.0.

## Discussion

### What are the benefits of adopting the OpenAPI Specification?

OAS enables higher efficiency, better collaboration, stable implementation and faster time to market. It makes APIs reusable and maintainable. More people can understand an OAS-based API definition since OAS provides a common vocabulary. Multiple teams can work in parallel since OAS enables a shared understanding of the APIs. Downstream service integration, continuous delivery and application security are further areas improved by OAS.

Because OAS is machine-readable, many tools exist to automatically validate, implement, test, mock or document APIs. Errors can be caught early even before any code is written. Moreover, client and server implementations can be generated automatically. This is commonly called the **API-first** approach whereby APIs are defined first. This is contrary to the traditional code-first approach.

JSON Schema can describe only the data model. OAS can describe the entire API while also being compatible with JSON Schema.

OAS is openly developed and released under an open license. It has wide industry support with more than 40 members at the OAI. It's got good documentation and support.

### Could you share some case studies of OpenAPI?

Two places where OAS has become important are eCommerce platforms (eg. eBay) and the media industry (eg. Twitter). This has led to faster innovation. In banking, OAS has improved API security. At eBay, a transformer service enabled their customers to migrate from legacy contracts to OpenAPI contracts.

One eCommerce company migrated from a monolithic to a microservices architecture. OAS enabled them to generate API mocks from the API specifications and conduct many API acceptance tests. As a result, fewer end-to-end tests sufficed. In another company, API mocks served as reference for real implementations.

OAS integrates with and helps in CI/CD pipelines. For example, changes to the API specification can be automatically validated using GitHub Actions. If the validation succeeds, the pull request can be automatically merged.

An API specification in OAS can be defined by someone and implemented by someone else. For example, 3GPP defined the 5G Core Network APIs using OAS. These APIs will then be implemented by various industry vendors. Network elements can come from different vendors and yet interwork more easily. Off-the-shelf tools have been used to test implementations and discover vulnerabilities.

### Could you give a technical overview of the OpenAPI Specification?

An **OpenAPI Document** can be in either JSON or YAML format. If an API is specified in multiple files, it's recommended that the root file is named `openapi.json` or `openapi.yaml`.

The document has a hierarchical structure starting from a root object called **OpenAPI Object**. Fields of this object are `openapi`, `info`, `jsonSchemaDialect`, `servers`, `paths`, `webhooks`, `components`, `security`, `tags` and `externalDocs`. At least one of `paths`, `components` or `webhooks` must be present. All field names (at the root level or otherwise) are case sensitive unless otherwise stated. Fields are either fixed or patterned with regex. At any level, the specification can be extended with a patterned field that starts with `x-`.

**Data types** in OAS are based on JSON Schema Specification. Examples include `integer`, `number` and `string`. OAS extends some of these with additional formats. Where `description` field is used, it can be formatted with CommonMark markdown.

### Could you describe the root-level fields of the OpenAPI Specification?

We describe briefly the following root-level fields:

    + `openapi`: Specifies the OAS version number. This is a string of the form *major.minor.patch*.
    + `info`: Contains metadata on the API. Its fields include `title`, `summary`, `description`, `termsOfService`, `license`, etc.
    + `jsonSchemaDialect`: Sets the default dialect such as JSON Schema 2020-12.
    + `servers`: One or more Server Objects, each containing at least a URL. If omitted, defaults to a Server Object with URL `/`.
    + `paths`: One or more Path Objects. Each Path Object represents an API endpoint along with operations permitted on it.
    + `webhooks`: Webhooks are called by the API provider and may be implement by the consumer.
    + `components`: Contains all reusable objects. Definitions here are generally referenced (using `$ref`) elsewhere in the document.
    + `security`: Specifies security mechanisms across the API but individual operations can override this. Reuses `securitySchemes` from Components Object.
    + `tags`: Tags provide a way to organize operations on endpoints. Operations may be defined without tags or use tags not defined at the root level.
    + `externalDocs`: One or more URLs to external documentation.

### How can API endpoints be specified in OpenAPI?

In OAS, API endpoints are called **Paths**. Each Path Object is an endpoint and must start with `/` character. This is appended to a server URL to obtain a full endpoint URL. RESTful HTTP methods (get, post, delete, etc.) are defined as Operation Objects within a Path Item Object.

Each **Operation Object** has the field `operationId` that's case sensitive and must be unique. The Operation Object also includes `requestBody` that conforms to HTTP 1.1 specification RFC7231. The field `responses` includes a list of possible responses to the operation's request.

A **Response Object** is associated with an HTTP status code (200, 404, etc.) or `default`. A `200` status response is usually accompanied by the `content` field that can define multiple Media Type Objects. For example, the API can define response content for both `application/json` and `application/xml`. The `description` field of a Response Object is mandatory and is meant to give API consumers a useful message to complement the HTTP status code.

### What tools are available within the OpenAPI ecosystem?

The company SmartBear continues to market products under the name of *Swagger*. Its online Swagger Editor is a useful tool to write API specification. It comes with an example called *Petstore*. The editor validates the specification and updates the documentation almost in real time. Developers can make API calls from the browser provided there's a hosted server implementation of the API. The editor's menu also has links to generate client and server code.

OpenAPI.Tools has a curated list of many tools that work with OAS. There are tools that take code and generate OAS; turn OAS into client SDKs, server or mock server implementations; generate documentation; validate data; exercise requests and validate the responses; offer user-friendly GUIs for editing/viewing OAS; and more. If writing in JSON or YAML is not preferred, Domain-Specific Languages (DSLs) are also available. CUE is an example.

The OAI maintains another list of OpenAPI tools.

### What are some best practices when using OpenAPI?

Beyond OpenAPI, many of the best practices we mention here are applicable for general API design. Developers should adopt an **API-first approach** rather than code-first approach. This helps multiple teams to work in parallel and efficiently, particularly when their apps are composed of microservices.

Both producers and consumers must get involved. In fact, APIs should be **consumer-driven** rather than producers defining them from just the data model. This can be extended to consumer-driven contract testing. This can catch breaking changes early.

Adopt a **style guide** for your APIs. This could describe the naming convention such as using lowercase for server URLs or kebab-case for paths. This could prohibit using HTTP method names in paths. In any case, all these rules can be specified formally and used during validation.

Beginners can **study OpenAPI examples** published by well-known organizations. Publicly shared OpenAPI documents from SendGrid and GitHub are worth studying.

### What are some limitations or criticisms of OpenAPI?

In November 2021, four years after OAS 3.0's release, it was observed that many APIs were still using Swagger 2.0. Many tools that supported Swagger 2.0 have not migrated to the newer specification. This has led developers to continue with the older specification. The name change has also caused confusion. Swagger UI still exists and its only the Swagger Specification that has become OAS.

OAS requires some learning curve and adjustments to workflows. For example, end-to-end testing must make way to consumer-driven contract testing. In one case study, many developers in the team didn't understand the new workflows even 12 months after the change.

There's a danger that developers might treat OAS as a magic bullet. OAS is not a shortcut to understanding an API. Techniques such as design thinking and domain-driven design are still relevant in creating great APIs.

## Milestones

Aug
2011

**Swagger 1.0** is released. Tony Tam started working on Swagger back in 2010 while at Wordnik.

Sep
2014

**Swagger 2.0** is released.

Mar
2015

SmartBear Software acquires the Swagger Specification.

Nov
2015

SmartBear and a few other companies announce the formation of the **OpenAPI Initiative (OAI)**. It's formed as an open source project under the Linux Foundation. In December, the Swagger Specification is donated to the OAI. The idea is to standardize and bring formalism to how APIs are described. While Swagger, RAML and API Blueprint are available to describe APIs, OAI aims to bring out a single open specification that covers all use cases.

Jul
2017

**OpenAPI Specification 3.0.0** is released, the first release of the specification from the OAI. Version identifier uses semantic versioning. Multiple servers (aliases) are supported but can be overridden by a Path Item Object. Reusable definitions are organized under the Components Object. Parameters can be complex types. Increased reusability, content negotiation, enhanced security, updated parameters, improved examples, links, and callback descriptions are some features this release brings.

Oct
2019

At Vancouver, Canada, the first **API Specifications Conference (ASC)** is organized by the OpenAPI Initiative. This conference has its origins in *APIStrat*, an API conference organized for many years by 3Scale. In 2017 and 2018, the OpenAPI Initiative hosted APIStrat that now becomes ASC. ASC welcomes participation from other communities (AsyncAPI, gRPC, GraphQL, OData, etc.).

Feb
2021

**OpenAPI Specification 3.1.0** is released. Top-level field `webhooks` is added. An OpenAPI document must have at least one of the top-level fields `paths`, `components` or `webhooks`. The specification is now fully compatible with JSON Schema 2020-12 draft.

Aug
2021

Postman publishes an analysis of nearly 200,000 OpenAPI documents downloaded from various sources. Most documents in this study use the older 2.0 specifications. They find that most of them don't include license information. About 21% fail validation. About 2% include a `/status` endpoint while 4% include a `/ready` or `/healthcheck` endpoint. HTTP methods GET (44%), POST (27%) and PUT (15%) are common.

OpenAPI Specification


# Wi-Fi Performance Testing

## Summary

Often customers complain to their ISPs about low data rates on their broadband or fibre connections, when the actual source of the problem is the variable performance of Wi-Fi. Wi-Fi performance is influenced by many factors including the quality of devices on the network, other networks in the area that are causing interference, or obstructions such as walls.

When testing Wi-Fi for performance, it's important to measure beyond just the throughput involving only a single active client. Testing should mimic real-world scenarios as much as possible. Performance testing for Wi-Fi clients is usually simpler than that of access points.

The purpose of such testing is to validate a deployment or identify performance issues so that corrective actions can be taken.

## Discussion

### What are the main metrics to measure during Wi-Fi performance testing?

A good reference for Wi-Fi performance testing is TR-398, *Indoor Wi-Fi Performance Test Standard*, defined by the Broadband Forum. This looks at six metrics: RF performance, bandwidth, stability, interference, capacity, and coverage. Bandwidth deals with throughput and fairness of resource allocation across stations.

We should check for seamless connectivity as clients move across access points. To identify bottlenecks, we need to look at CPU utilization, memory usage, and buffer status.

Application-level metrics matter as well. Packet loss, latency and jitter are important metrics for audio and video streaming applications. These can be translated into an objective Mean Opinion Score (MOS) to determine how the user perceives quality.

### In a real-world Wi-Fi deployment, what performance tests should I run?

In a typical deployment, engineers check for signal levels, coverage and throughput to ensure basic performance requirements are met. Usually a single client is used but this is far from real-world scenarios. It's therefore important to check performance when the network is loaded with multiple busy clients. Since it's hard to use hundred of real clients, an emulation-based approach can be adopted.

Wi-Fi environment is too dynamic to reproduce performance issues. In fact, performance testing doesn't help with troubleshooting problems. A better approach is to collect data from live networks and analyse this data. With modern Wi-Fi networks being controlled from the cloud, it's become possible to adjust a configuration, collect performance data, send it to the cloud and then evaluate if the adjustment had a positive effect.

Some performance problems could be due to misconfiguration such as access point connected to the wrong switchport, poor quality cables, wrong channel assignments, or bandwidth limits unknowingly imposed.

### What traffic models should I use for Wi-Fi performance testing?

To mimic real-world scenarios, a variety of traffic types coming from multiple devices must be used. This may include a mix of TCP and UDP data, streaming video or audio, voice calls, and a mix of mission-critical and recreational apps.

The idea of mixing different traffic types is that pattern of packet arrivals is different. Voice is periodic while video starts with a surge and then a constant rate. HTTP traffic comes in bursts as pages are loaded. We should thus monitor buffers and CPU utilization for bursty traffic as well.

A classroom scenario could include multicast video streaming, parallel file downloads, and background social media traffic of small packets. A dormitory scenario could include parallel video streaming, two-way video chats, online gaming, web browsing, and social media traffic. A stadium scenario would be heavy on video streaming or audio commentary involving thousands of clients. Performance testing should consider these diverse real-world deployment scenarios.

### Could you share some insights discovered by others during Wi-Fi performance testing?

Before starting any tests, check all cabling and configuration. Devices should be configured to exercise their maximum capability. Turn off other devices that are not part of the tests. Avoid a VPN while testing. Restart devices, stop background tasks or updates, disable antivirus, or even do a clean OS installation before starting the tests. Update devices to latest drivers.

One suggestion is to establish a baseline application throughput. This can be done by copying a large file from a server connected via Ethernet to the wireless router, to a client device connected via Wi-Fi to that router. No other clients should be on the network. Then you can add more clients to the network or try streaming traffic to see how these affect the baseline.

It's been noted that CPU utilization is higher than usual and varies the most when packet sizes are small, such as for VoIP traffic. Another finding is that device OS plays a role as well. For example, Windows clients have a longer inter-packet idle periods, thus resulting in lower throughput compared to Linux clients.

### What tools are available for doing Wi-Fi performance testing?

While there are many online sites to help you measure your Internet speed, this is really an end-to-end performance test. To test the performance of only the Wi-Fi network, you should do an internal test. Likewise, the speed shown in Windows OS is the transfer speed between the PC's adapter and the router, not the throughput seen by applications.

For network deployment, there are tools for site survey and generation of heat maps of signal strengths: Ekahau Site Survey, iBwave, NetScout AirMagnet, Tamograph Site Survey, SolarWinds, VisiWave Site Survey, AirMagnet Survey, Acrylic Wi-Fi, and NetSpot.

For throughput testing, iPerf3 (and its variants), Tamosoft, HE.NET Network Tools, and Ixia's IxChariot are examples. iPerf3 is open source and many tools such as WiFiPerf Professional are built on top of iPerf3.

Spirent offers products that can emulate hundreds of Wi-Fi clients with different traffic types and also emulate movement across access points. Similar emulation is offered by Alethea and LANforge WiFIRE.

## Milestones

2010

ESNet begins work on **iperf3** since iperf2 was not updated for a while. The goal is to keep the tool simple and maintainable. Hence it's designed as a single thread as opposed to the multithreaded nature of iperf2.

2014

Bob McMahon of Broadcom revives **iperf2**, fixing many of its problems and adding new features. Version 2.0.8 is released in 2015. Since iperf2 is multithreaded, it may be a better choice for generating parallel traffic streams.

Feb
2019

Broadband Forum releases TR-398, Indoor Wi-Fi Performance Test Standard. Prior to this standard, it's been generally difficult to compare performance of different devices. TR-398 enables systematic quantitative testing and performance comparisons. In it's first release, TR-398 doesn't cover 802.11ax.

Wi-Fi Performance Testing


# Greedy Algorithms

## Summary

A greedy algorithm is a type of algorithm that follows the problem-solving heuristic of making the locally optimal choice at each stage with the hope of finding a global optimum. While it may not find the global optimum, greedy algorithms are often simpler and faster while being not too far from the global optimum.

Greedy algorithms are being used in many areas of computer science. They're also used in operations research, engineering, finance, machine learning, bioinformatics, game theory, etc. They've become a well-established technique for solving optimization problems.

The key to developing an effective greedy algorithm is to identify the problem's specific structure and design a heuristic that makes the optimal local choice at each step. While they do have some limitations, there's ongoing research to address these limitations.

## Discussion

### Could you explain greedy algorithms with a simple example?

Suppose you are given an array of positive integers and you want to find the subset of those integers whose sum is as large as possible, but does not exceed a given value. For instance, if you were given the array `[3, 4, 5, 6, 8]` and the limit 13, the optimal subset would be `[8, 5]`, with a sum of 13.

A greedy algorithm for this problem would be to sort the array in decreasing order, then start selecting the largest numbers that don't exceed the limit, until the limit is reached. So for the example above, the steps of the algorithm would be:

    + Sort the array in decreasing order: `[8, 6, 5, 4, 3]`
    + Initialize an empty subset and a variable to track the current sum: subset = `[]`, sum = 0
    + Start the iterations: (i) subset = `[8]`, sum = 8; (ii) subset = `[8, 5]`, sum = 13.
    + Return the subset: `[8, 5]`

### Why should I use greedy algorithms when the solution is not guaranteed to be optimal?

While greedy algorithms don't guarantee an optimal solution, they have their benefits:

    + **Simplicity**: Greedy algorithms are often simple to implement and understand, making them a good choice for problems with large inputs or when efficiency is a concern.
    + **Speed**: Greedy algorithms are often very fast, especially when compared to more complex algorithms. This makes them a good choice for problems with large inputs.
    + **Approximation**: Even though the greedy algorithm does not always guarantee the optimal solution, it can often give a very good approximation of the optimal solution. In many cases, the difference between the optimal solution and the solution found by the greedy algorithm is not significant.
    + **Starting Point**: Greedy algorithms can be a good starting point for more complex algorithms. By using a greedy algorithm to quickly find a good solution, you can then refine the solution using other techniques.

### What are some applications of greedy algorithms?

Here are some examples of greedy algorithms:

    + **Minimum Spanning Tree (MST)**: MST is useful in network design and transportation planning. Kruskal's and Prim's algorithms are greedy approaches to this problem.
    + **Huffman Coding**: This is applied in data compression and transmission. It assigns shorter codes to more frequently occurring characters.
    + **Knapsack Problem**: This is considered a classic optimization problem. It asks what's the best way to fill a bag of fixed capacity with items of different sizes and values. A greedy algorithm selects items based on their value-to-size ratio.
    + **Activity Selection**: In scheduling problems, there's a need to select the maximum number of non-overlapping activities. A simple greedy algorithm solves this by selecting activities based on their finish time.
    + **Shortest Path Algorithms**: Dijkstra's algorithm is an example. It selects the shortest path from a given vertex to all other vertices in a graph.
    + **Coin Changing Problem**: This asks what's the minimum number of coins needed to make change for a given amount of money. This is solved by selecting the largest coin possible at each step.

### How are greedy algorithms different from dynamic programming?

Greedy algorithms and dynamic programming are two popular techniques for solving optimization problems, but they differ in several key ways.

Greedy algorithms make the **locally optimal choice** at each step in the hope of finding a global optimal solution. Dynamic programming breaks down a problem into smaller subproblems and then solves them in a bottom-up fashion. It stores the solutions to the subproblems and reuses them to solve the larger problem.

Greedy algorithms may give a **sub-optimal solution**, whereas dynamic programming always finds the optimal solution. Greedy algorithms typically make choices based only on the current state of the problem, while dynamic programming considers all possible subproblems and their solutions.

Greedy algorithms typically require **less memory** because they don't need to store the solutions to all possible subproblems.

Greedy algorithms are typically **faster** due to fewer calculations. However, the time complexity of greedy algorithms can be higher in certain cases.

Greedy algorithms are useful when there is a **unique optimal solution**. Dynamic programming can solve a wider range of problems, including problems with multiple optimal solutions or overlapping subproblems.

### What are some practical limitations of greedy algorithms?

Because greedy algorithms make the locally optimal choice at each step, this may not lead to the global optimal solution. In some cases, making a suboptimal choice at an early stage can lead to a better global solution. The figure shows an example in which the objective is to maximize the sum of the nodes on a top-to-bottom path. Greedy algorithm leads to a sub-optimal solution.

Where the objective function has multiple conflicting objectives, or changes non-monotonically over time, greedy algorithms will not work well. The same can be said of problems with complex constraints or a large search space. For these cases, greedy algorithms would incur a larger time complexity as well.

Many optimization problems are NP-hard, which means that finding the optimal solution is computationally intractable. Greedy algorithms are not suitable for solving them, as they can't guarantee the optimal solution in a reasonable amount of time.

### What are some recent advances with greedy algorithms?

**Adaptive greedy algorithms** adjust their choices dynamically based on the problem's structure and input data. They have outperformed traditional greedy algorithms in many real-world optimization problems.

**Hybrid algorithms** combine greedy techniques with other optimization techniques, such as dynamic programming or local search. These hybrid algorithms can often provide better results than either technique used alone.

There are greedy algorithms capable of **multi-objective optimization**. They can be used to find a Pareto-optimal set of solutions.

In many real-world applications, input data is received incrementally over time. **Online optimization algorithms** must make decisions in real-time, without having access to all the input data in advance. Greedy algorithms have been shown to be effective in this setting because they can make quick decisions based on the available data.

Researchers have also been working on developing new methods for analysing the performance of greedy algorithms. There are new theoretical frameworks for understanding the behaviour of greedy algorithms in different types of optimization problems.

## Milestones

500
AD

Indian mathematician Aryabhata uses a greedy algorithm to solve the problem of finding the largest number that can be written as the sum of two squares.

1930

The modern concept of greedy algorithms was first introduced in the 1930s by mathematicians and computer scientists working on the design of algorithms for optimization problems.

1952

The term **greedy algorithm** is coined by mathematician and computer scientist David A. Huffman in reference to the Huffman coding algorithm. The term "greedy" refers to the algorithm's strategy of always choosing the option that seems best at the time, without considering the possible consequences of that choice on future steps.

1956

Computer scientist Joseph Kruskal develops a greedy algorithm for finding the **minimum spanning tree** of a graph. In 1957, Robert Prim develop's his own greedy algorithm for the solving the same problem. Also in 1956, Edsger Dijkstra develops a greedy algorithm for finding the **shortest path** in a graph. The same year L. R. Ford and D. R. Fulkerson develop a greedy algorithm for solving the **maximum flow problem** in a network.

1980

In the 1980s, researchers develop a variety of **approximation algorithms** based on greedy techniques, which provide near-optimal solutions to NP-hard problems. This research continues into the 1990s.

Greedy Algorithms


# k-Means Clustering

## Summary

The k-means Clustering method is an unsupervised machine learning technique that groups unlabelled dataset into different clusters.

The algorithm starts with a group of randomly selected 'k' centroids as the beginning points for every cluster. Then, iterative calculations optimize the centroid positions.

The clusters in the example are distinguished by color and current cluster seeds (centroids) are marked by a starburst. In the first round, each point is assigned to its closest seed, and a new seed is chosen for each cluster based on the average of all points in that cluster. As a result, the blue cluster seed moves to the right. In the second round, both cluster seeds drift to their correct locations in a proper division. After two rounds, the clusters have reached a steady-state, and wouldn't change further through an infinite number of iterations.

## Discussion

### What similarity measures are applicable for clustering?

Clustering is done on the basis of a similarity measure to group similar data objects together. Similarity is an amount to reflect the strength of relationship between two data items, to represent how similar two data patterns are. A similarity measure is the distance between various data points.

To group objects in clusters, this similarity measure is most commonly based on distance functions such as Euclidean distance, Manhattan distance, Minkowski distance, Cosine similarity, etc. The clusters are formed such that any two data objects in a cluster have a minimum distance value and any two data objects across different clusters have a maximum distance value. Clustering using distance functions, called distance based clustering, is popularly used to cluster objects.

### What are the main objectives of the k-means clustering algorithm?

The main objectives of the k-means algorithm are:

    + Determine the best value of k center points (centroids) iteratively.
    + Assign each data point to its closest k-center. The data points closer to a particular k-center (with minimum Euclidean distance) form a cluster.
    + Minimize the sum of distances between the data points and their corresponding cluster centroid.

### Could you share some use cases of k-means clustering?

This algorithm is used across a variety of applications such as image segmentation, image compression, cyber-profiling criminals, document classification, market/customer segmentation, clustering of IT alerts, delivery store optimization, insurance fraud detection, etc.

For example, in image compression, k-means clustering groups similar colors together into 'k' clusters (say k=64) of different colors. Hence, each cluster centroid represents the three-dimensional color vector in RGB color space of its respective cluster. Then, these 'k' cluster centroids will replace all the color vectors in their respective clusters.

In the original image, each pixel takes 24 (8x3) bits to store its corresponding color vector. After k-means algorithm is applied, each pixel takes only 6 bits to store as it stores only the label of the cluster to which it belongs. K=64 different color vectors can be represented using 6 bits only. The final image will have 64 different colors in its RGB color space.

### Are there any variations of the k-means algorithm?

Yes. The following are some of the variations of the k-means algorithm:

    + k-medians clustering - uses the median in each dimension (instead of the mean) for optimal clustering.
    + k-medoids (Partitioning Around Medoids) - uses medoid instead of mean, and minimizes the sum of distances for arbitrary distance functions.
    + Fuzzy C-Means clustering - A soft version of k-means, where each data point has a fuzzy degree of belonging to each cluster.
    + k-means++ - This is the standard k-means algorithm coupled with a smarter initialization of the centroids.

### What are the steps of the k-means clustering algorithm?

The first step in k-means clustering algorithm is to randomly choose k centroids (k is the number of clusters to be formed). Each data point is assigned to its nearest centroid. The mean of all points in each cluster is computed and a new centroid is selected. Through iterations, the last 2 steps are repeated until the centroid positions do not change.

### What are the major challenges associated with k-means clustering?

    + Initialization sensitivity - Since the initialization process of the standard k-means algorithm is random, it leads to the problem of initialization sensitivity. For larger and more complex datasets, the run-time of the algorithm is longer and it may not reach optimal clustering.
    + Clustering data of varying sizes and densities - k-means has trouble clustering data where the clusters are of varying sizes and densities. Usually, to cluster such data, you need to generalize k-means. While the k-means algorithm clusters the given data sets uniformly when used for uniform data sets, it fails to form clusters as per requirements when used for data sets of varying densities or distances between data sets.
    + Clustering outliers - Centroids can be dragged by outliers, or outliers might get their own cluster instead of being ignored. To overcome this challenge, outliers could be clipped or removed before clustering.
    + Scaling with number of dimensions - With increasing number of dimensions (variables), more data-points become nearest neighbours of a selected centroid, until the choice of nearest neighbour is nearly random. The distances between a data point and its nearest and farthest neighbours could become equidistant with increases dimensions.

### What are the ways to avoid the problem of initialization sensitivity in k-means algorithm?

There are two ways to avoid the problem of initialization sensitivity:

    + Repeat k-means: The algorithm is executed repeatedly. The centroids are initialized and the clusters are formed to result in smallest intra-cluster distance and larger inter-cluster distance.
    + k-means++: This is a smart centroid initialization technique. Only one centroid is initialized randomly, and other centroids are chosen such that they are very far from the initial centroid/s. This results in faster convergence and lower possibility of the centroid being poorly initialized.Between the two techniques that can be implemented to avoid the problem of initialization sensitivity, K means++ provides comparatively better results.

### How to decide the optimal number of k in k-means algorithm?

    + Elbow method - The sum of squared distance (SSE) is used to select an ideal k value. This is based on the distance between the data points and respective clusters. A value of k is chosen such that the SSE begins to flatten out and an inflection point is seen. Visually, this graph looks similar to an elbow, hence the name.
    + Average Silhouette method - This measures the quality of clustering and determines how well each object lies within its cluster. The similarity/dissimilarity score is measured between the assigned cluster and the next best/nearest cluster for each data point. The average silhouette of observations is calculated for different values of k. The optimal k is one that maximizes the average silhouette over a range of possible values for k.
    + Gap statistic method - The total intra-cluster variation is compared for different k values with their expected values under null reference distribution of data (i.e. a distribution with no obvious clustering). The optimal k value is one that maximizes the gap statistic value.

### What are the possible stopping criteria in k-means algorithm?

The following are possible stopping conditions in k-means algorithm:

    + Stability: The centroids of new clusters do not change.
    + Convergence: The algorithm has converged at the minima i.e., the points stay in the same cluster.
    + Maximum number of iterations has been reached: While the number of iterations limits the run-time of the algorithm, the quality of the clustering could be poor due to an insufficient number of iterations.
    + When Residual Sum of Squares(RSS)/ within-cluster sum of squares falls below a threshold: This ensures the desired quality of clustering after termination. The combination of this with a bound on the number of iterations guarantees convergence.

### How to perform k-means on larger datasets to make it faster?

For large datasets, an alternative to k-means is the mini-batch k-means. Unlike k-means, this doesn’t iterate over the entire dataset. It uses small, random, fixed-size batches of data to store in memory. With each iteration, a random sample of data is obtained and used to update the clusters. The means are calculated and the cluster centres are re-calculated using gradient descent. This is repeated until convergence. Mini-batch k-means has been known to provide faster convergence than k-means, with a slightly different cluster output.

## Milestones

1956

The mathematician Hugo Steinhaus publishes an article titled "Sur la Division des Corps Materials en Parties". He presents the problem of partitioning a heterogeneous solid by the adequate selection of partitions.
 The article's conclusion mentions that diverse questions about types in anthropology or industrial object normalization require a solution based on the determination of n fictitious representatives of a numerous population, chosen so as to minimize as much as possible the deviations between population elements and those from the sample. The deviation is measured between every actual element and the closest fictitious element.

1957

The concept of k-means algorithm was first suggested by Stuart Lloyd as a code for pulse-code modulation. In his article, he mentions that this algorithm may result in local optima since it is critical to initial cluster heads so formed arbitrary. The partitioning method comprises of k partitions of data, where each partition denotes a cluster with respective cluster heads such that k ≤ n(data objects). Categorizing the data into k clusters must fulfil the two requirements: i) Every cluster must comprise of one or more data objects. ii) Every object must belong to exactly one cluster.

1958

In the paper 'On Grouping for Maximum Homogeneity', Walter D Fisher discusses the practical procedure for grouping a given set of arbitrary numbers, so that the variance within groups is minimized. For problems up to size of 200 numbers to be placed in 10 groups, he considers two types of problems. First is the unrestricted problem, where no restrictions or side conditions are imposed on the partitions allowed. Second is the restricted problem, where such conditions are imposed a priori on the basis of previous knowledge, theory, or for convenience.

1963

In Principles of Numerical Taxonomy, Sokal and Sneath use Operational Taxonomic Units (OTUs) to measure and count as many characters as possible of the organisms to be compared. Based on a matrix of variation in all features across all OTUs, the OTUs are then compared by overall similarity, affinity, or phonetic distance. The results are displayed by means of a network or OTUs are linked to each other by a clustering algorithm to produce a phenogram. Following this, many different ways have been proposed to measure pair-wise similarity or dissimilarity, and many different clustering methods have also been devised.

1967

James MacQueen, from the Department of Statistics of the University of California, introduces the term k-means in his article titled "Some Methods for Classification and Analysis of Multivariate Observations". He proposes an algorithm to partition an instance into a set of clusters whose variance is small for each cluster. Prior to the term k-means being coined by him, it was known by names like dynamic clustering method, iterative minimum-distance clustering, nearest centroid sorting, and h-means, among others.

1969

Engelman and Hartigan, in their article 'Percentage Points of a Test for Clusters', state that a set of observations may be partitioned into two clusters to maximize the ratio of between sum of squares to within sum of squares. This maximum ratio is the likelihood ratio test statistic for the observations to come from two normal populations, with different means but the same variance. They consider the null hypothesis, that the observations are a sample from a single normal population.

1973

In the book 'Cluster Analysis for Applications', various methods and applications of cluster analysis has been covered, as well as the category solving problems and the need for cluster analysis algorithms. This includes variables and scales to measures of association among variables and among data units. The conceptual problems in cluster analysis is discussed, along with hierarchical and non-hierarchical clustering methods.

k-Means Clustering


# Metaprogramming

## Summary

Programs typically read input data, operate on that data and give some output data. **Metaprograms** read another program, manipulate that program and return a modified program. Sometimes a metaprogram can change its own behaviour by updating itself. The act of writing metaprograms is called metaprogramming. The language used to write metaprograms is called **metalanguage**.

In general, where code itself needs to change with the data, metaprogramming can be an effective approach. Metaprograms treat code as data, and data as code.

While metaprogramming has been around for decades, it only from the mid-1990s that it received increasing attention. More and more languages are supporting metaprogramming. Principles of metaprogramming have led to higher levels of abstractions such as meta-modeling, metadesign, metaengineering and Model-Driven Engineering (MDE).

## Discussion

### Could you explain metaprogramming with an example?

Assume you're asked to write a program to print numbers 1 to 1000 but without using any looping constructs in the code. The obvious way to do this is to explicitly print each number. Writing such a code becomes tedious for the programmer. For example, in a shell script, you would have to type `echo 1`, `echo 2`, and so on till `echo 1000`. However, it's possible to write another shell script with a for loop that outputs the desired program containing these 1000 echo commands. In other words, we've written a program to produce another program.

In the above example, a shell script produces another shell script. This need not be the case. The metaprogram could be in a different language than the target language. For example, a Ruby script could be used to output C source code.

### What are some use cases of metaprogramming?

Metaprogramming has many applications: compiler generation, application generation, code analysis, generic component design, program transformations, software maintenance/evolution/configuration, anticipatory optimization, design patterns, partial evaluation, and more.

Compilers, transpilers, assemblers and interpreters are examples of metaprograms. They take programs in one form and transform them into machine code, bytecode or even source code in another language. Lex and Yacc are metaprograms. Programs that read database metadata and output EJB or XML code are further examples. Dynamic or interpreted languages usually have an `eval()` function that allows execution of code supplied as strings. This is also metaprogramming.

ActiveRecord is a database ORM framework used in Ruby on Rails. When calling a method that's not defined, due to predefined conventions, the method gets created on the fly. This implies that a developer writes less manual code.

Metaprogramming has been applied to numerical algorithms such as for solving Ordinary Differential Equations (ODEs).

Another example is the Hone modular graphing library written in Julia. Data points, axes and legends are composed as strings and executed to create the plot.

### What's the formal definition of metaprogramming?

One definition sees metaprogramming as introducing a higher level of abstraction: "the technique of specifying generic software source templates from which classes of software components, or parts thereof, can be automatically instantiated to produce new software components."

We could define metalanguage thus: "any language or symbolic system used to discuss, describe, or analyze another language or symbolic system is a metalanguage." This leads to the definition of metaprogramming: "creating application programs by writing programs that produce programs."

Another definition that emphasizes program generation: "metaprograms manipulate object-programs" where metaprograms "may construct object-programs, combine object-program fragments into larger object-programs, observe the structure and other properties of object-programs." A simpler definition is, "Metaprogramming is writing programs that themselves write code."

### What are the benefits of metaprogramming?

Metaprogramming has many benefits:

    + **Extensible**: Since code is treated as data, it's easy to extend programs. Code can be added by simply adding metadata.
    + **Performance**: Instead of having variables in memory and passing them around, everything is packed into a concatenated string and executed. Metaprogramming enables abstractions without runtime penalty. It can be useful in pushing computations from runtime to compile time. Metaprograms can help tune programs to specific architectures. Specialized efficient programs are better than generic inefficient ones.
    + **Less Code**: In the long term, metaprogramming automates code writing and reduces manual effort. The use of macros (such as in C or C++) saves development time and promotes reusable code. Recurring code patterns can be abstracted and reused even when functions, generics or classes are unable to this.
    + **Correctness**: Compiling code from a high-level language to machine code or bytecode is not something we wish to do manually. If done manually, it's easy to introduce errors. Metaprograms help in such tedious but necessary tasks.
    + **Reasoning**: Metaprograms such as flow analyzers and type checkers help us discover properties about programs, validate behaviour or improve performance.

### How do we classify the various types of metaprogramming?

In **Homogeneous MP**, the object language and metalanguage are the same. Examples include Racket, Template Haskell, MetaOcaml, and Converge. In **Heterogeneous MP**, these are different. For example, a Java compiler is written in C. In **Generative MP**, the metaprogram generates an output program. In **Intensional MP**, the metaprogram analyzes the input program, by way of reflection. Combining the above two classifications, we note an important class called **Homogeneous Generative Meta-Programming (HGMP)**.

Another classification is based on when metaprograms execute: **Compile-Time MP (CTMP)** or **Run-Time MP (RTMP)**. CTMP examples include the Lisp family, Template Haskell, Converge and C++. RTMP examples include the MetaML family, JavaScript and printf-based MP. Converge and Scala support both. There are also implicit and explicit flavours of compile-time evaluation.

CTMP and RTMP could be called static and runtime respectively. With RTMP, the metaprogram produces new code, which is immediately executed. If the new code is also a metaprogram, we called this **multi-stage programming**.

Metaprogramming requires us to annotate static code fragments (the metaprogram) from dynamic ones generated by the metaprogram. This gives rise to two flavours, **manually annotated MP** versus **automatically annotated MP**.

### In metaprogramming, how are programs represented as data?

Common ways of representing programs include:

    + **Strings**: Simplest and terse. Typically runtime evaluation but some languages offer compile-time support.
    + **Abstract Syntax Trees (ASTs)**: Code fragments represented as a tree. For example, \(2+3\) can be written as \(ast\_{add}(ast\_{int}(2), ast\_{int}(3))\).
    + **Up MetaLevels (UpMLs)**: Represent AST-like structures as quoted chunks of normal program syntax. \(2+3\) can be written as \(\uparrow\{2+3\}\).
    + **Down MetaLevels (DownMLs)**: Used to express holes in UpMLs. Evaluated first to yield an AST. If function \(f\) returns the AST for \(2+3\), then \(\uparrow\{\downarrow\{f()\} \*4\}\) is a valid expression.Adopting a suitable representation depends on syntactic overhead, expressivity, support for the target language and support for generating 'valid' programs. Hygiene is also important, which is about managing free and bound variables. Typically, we want the terseness of strings (ASTs are verbose) and the syntactic correctness of ASTs. UpMLs (aka quasi-quotes, backquotes) and DownMLs (aka splices, inserts) offer the best of both worlds.

Lilis and Savidis (2020) give a more complete discussion of metaprogramming models include macro systems, reflection systems, Meta Object Protocols (MOPs), Aspect-Oriented Programming (AOP), generative programming and multi-stage programming.

### Which languages offer support for metaprogramming?

Many languages support metaprogramming including Python, Ruby, JavaScript, Java, Go, Clojure, and Julia. Groovy, Java, Racket, Common Lisp and Scheme are examples that support both CTMP and RTMP. CPP, M4, Racket, Reflective Java and AspectC++ are examples that support Preprocessing-Time MP (PPTMP). Interested readers can refer to *A Survey of Metaprogramming Languages* by Lilis and Savidis (2020).

Many languages such as Scala, Rust or JavaScript support metaprogramming by design. Others such as C++ evolved to support metaprogramming.

Interpreted languages usually have the `eval()` function. Examples include Lisp, Perl, Ruby, Python, PHP and JavaScript. In such languages, we could generate the code within the program and pass this code into `eval()` for execution. For example, we could do this in Ruby: `x = 3; s = 'x + 1'; puts eval(s)`.

UpMLs are supported in Racket, MetaOCaml and Converge. Popular languages including Python, Ruby, Groovy and Perl support MOP based on metaclasses.

### What are some challenges or shortcomings of metaprogramming?

Metaprogramming, both in theory and practice, has many open problems. C++ template metaprogramming is not pretty. Scala had to rewrite it's implementation for metaprogramming. MetaOcaml has typing problems. There's no consistent terminology. Tooling is not mature. For example, it's hard to debug metaprograms.

Executing strings as code can lead to insecure code, particularly when the strings come from untrusted sources. However, languages may provide some support to make it safer. For instance, Python's decorators are safer than strings for metaprogramming.

Writing code as strings reduces readability and is likely to introduce bugs. Strings might use lots of regular expressions. When strings can't be parsed due to some programming error, we might see unexpected output.

Metaprogramming is hard and harder for beginners. Someone who doesn't know the language constructs (such as macros in Lisp) will find it very hard to understand metaprograms written by others.

## Milestones

1940

Willard Van Orman Quine invents **quasi-quote and splicing**. These concepts would later prove to be useful for metaprogramming. In fact, the term **quine** is coined decades later to refer to a computer program that takes no input and outputs its own source code. Quines can be considered as special metaprograms.

1963

In a short MIT memo, Timothy P. Hart introduces **macros** into Lisp. In the 1970s, Lisp becomes possibly the first language to support HGMP.

1997

Two paradigms of metaprogramming emerge: Aspect-Oriented Programming (AOP) and Multi-Stage Programming. It's also during the years 1995-2000 that many metalanguages and metaprogramming systems emerge. Mainstream languages such as Java and C++ provide better support for metaprogramming. The arrival of MetaML proves that HGMP is possible beyond syntactically simple languages like Lisp.

Metaprogramming