# Deep Learning Frameworks

## Summary

Deep Learning (DL) is a neural network approach to Machine Learning (ML). While it's possible to build DL solutions from scratch, DL frameworks are a convenient way to build them quickly. Such frameworks provide different neural network architectures out of the box in popular languages so that developers can use them across multiple platforms.

Choosing a framework for your problem depends on a number of factors. Therefore, it's not possible to name just one framework that should be preferred over another. Many frameworks are open source. Cloud providers also provide easy ways to deploy and execute a framework on their infrastructure.

Sometimes the term "framework" is used interchangeably with terms "toolkit" or "library".

## Discussion

### Could you mention some popular DL frameworks?

Among the popular open source DL frameworks are TensorFlow, Caffe, Keras, PyTorch, Caffe2, CNTK, MXNet, Deeplearning4j (DL4J), and many more. Many of these frameworks support Python as the programming language of choice. DL4J is for Java programmers but models written in Keras can be imported into DL4J. Among those supporting C++ include Caffe, DL4J, CNTK, MXNet and TensorFlow. Torch was written in Lua and C, and PyTorch extends and improves on it with Python support. Paddle is a framework from Baidu.

These frameworks are not all at the same level of abstraction. For example, Keras provides a simpler API for developers and sits on top of TensorFlow, Theano or CNTK. Likewise, Gluon is an API that can work with MXNet and CNTK. Gluon can be seen as competition to Keras.

Among those not open sourced are Intel Math Kernel Library, MATLAB Neural Network Toolbox, Neural Designer, and Wolfram Mathematica NeuralNetworks.

A curated list of ML frameworks is available online.

### How should I go about selecting a suitable DL framework?

Some obvious factors to consider are licensing, documentation, active community, adoption, programming language, modularity, ease of use, and performance. Keras and PyTorch are said to be easy to use but you can also consider TensorFlow for its popularity.

More specifically, you should check the following:

    + **Style**: Imperative or symbolic.
    + **Core Development Environment**: Programming language, intuitive API, fast compile times, tools, debugger support, abstracting the computational graph, graph visualization (TensorBoard), etc.
    + **Neural Network Architecture**: Support for Deep Autoencoders, Restricted Boltzmann Machines (RBMs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTMs), Generative Adversarial Networks (GANs), etc.
    + **Optimization Algorithms**: Gradient Descent (GD), Momentum-based GD, AdaGrad, RMSProp and Adam, etc.
    + **Targeted Application Areas**: Image recognition, video detection, voice/audio recognition, text analytics, Natural Language Processing (NLP), timeseries forecasting, etc.
    + **Hardware Extensibility**: Support for multiple CPUs, GPUs, GPGPUs or TPUs across multiple machines or clusters.
    + **Optimized for Hardware**: Execute in optimized low-level code by supporting CUDA, BLAS, etc.
    + **Deployment**: Framework should be easy to deploy in production (TensorFlow Serving).

### Which DL frameworks are symbolic and which ones are imperative?

Let's briefly understand the difference between the two. Imperative programs perform computations as they are encountered along the program flow. Symbolic programs define symbols and how they should be combined. They result in what we call a **computational graph**. Symbols themselves might not have initial values. Symbols acquire values after the graph is compiled and invoked with particular values.

Torch, Chainer and Minerva are examples of imperative-style DL frameworks. Symbolic-style DL frameworks include TensorFlow, Theano and CGT. Also in symbolic style are CXXNet and Caffe that define the graph in configuration files.

Imperative frameworks are more flexible since you're closer to the language. In symbolic frameworks, there's less flexibility since you write in a domain-specific language. However, symbolic frameworks tend to be more efficient, both in terms of memory and speed. At times, it might make sense to use a mix of both framework styles. For example, parameter updates are done imperatively and gradient calculations are done symbolically. MXNet allows a mix of both styles. Gluon uses an imperative style for easier model development while also supporting dynamic graphs.

### From a mathematical perspective, what are the main features expected of a DL framework?

All data is represented as **tensors** (multi-dimensional arrays). A DL framework must therefore support tensors and operations on them.

The ability to define **dynamic computational graphs** is desired by developers. To be dynamic means that graph nodes can be added or removed at runtime. With PyTorch and Chainer, graphs can be defined dynamically. With TensorFlow, you have to define the entire computation graph before you can run it. TensorFlow more recently has TensorFlow Fold for dynamic graphs and *eager execution* for immediate execution.

An essential operation in a DL network during learning is function differentiation. **Automatic Differentiation** is a feature that must be supported by a DL framework. This is straightforward when the framework uses a computational graph.

### In terms of hardware support for DL frameworks, what should I look for?

Deep Learning involves training and inference. Training is often done on cloud clusters. Inference can at times happen on constrained IoT devices, embedded systems or smartphones. Thus, trained models should be able to run on ARM-based hardware. For example, Caffe2 is suitable for smartphones while TensorFlow is for research and server-side deployment. However, there's TensorFlow Lite for inference on constrained devices.

When running on NVIDIA GPUs, you should check if there's support for **CUDA** that enables most efficient use of GPUs. An alternative to CUDA is **OpenCL** but most DL frameworks don't support it. **OpenMP** is more widely supported and it enables multiplatform shared memory multiprocessing.

In addition, hardware vendors provide a number of libraries and optimizers so that DL frameworks make the best use of their hardware. Among these are NVIDIA's cuDNN, cuBLAS, cuSPARSE, TensorRT, DeepStream, and many more. Intel offers Math Kernel Library for Deep Neural Networks (MKL-DNN), Data Analytics Acceleration Library (DAAL), OpenVINO, nGraph, and others.

### How do the popular DL frameworks compare in terms of performance?

Since DL frameworks are always improving, what's mentioned here should be only a starting point for your study.

One study from early 2018 compared some DL frameworks on three datasets. Nvidia GPUs were used along with cuDNN, an Nvidia library tuned for common DL computations. Among the faster frameworks are TensorFlow, PyTorch, MXNet, CNTK and Julia-Knet. It was seen that CNTK and TensorFlow are much faster than Keras-CNTK and Keras-TF respectively. While Keras simplifies development, the tradeoff is performance.

In other studies, TensorFlow is said to be slower than CNTK and MXNet. MXNet stands out in terms of scalability and performance.

Ultimately, the framework itself may not matter much when cuDNN is used to provide acceleration on NVIDIA's GPUs. Any difference in performance will come from how frameworks scale to multiple GPUs and machines.

## Milestones

1965

While the theory of neural networks inspired by the human brain was first formulated in 1943, Alexey Ivakhnenko and his team create the first working Deep Learning network in 1965. In 1971, they succeed in building an 8-layer network.

2000

The term *Deep Learning (DL)* is used for the first time to mean a neural network with many layers. By 2017, a DL network is as much as 1000 layers deep.

2007

At the Montreal Institute for Learning Algorithms (MITA), **Theano** is developed in Python for efficient math operations that can run on either CPU or GPU architectures. It enables developers to run rapid experiments in Deep Learning. In later years, Theano goes on to inspire other frameworks. Ten years later (in 2017), it's announced that Theano will no longer be actively maintained.

2013

Invented by Berkeley Vision and Learning Center at UC Berkeley, **Caffe** framework is released. In 2017, Facebook open sources an evolution of Caffe called **Caffe2**.

2015

This year may be considered the turning point for DL frameworks. A number of DL frameworks are released: Chainer, Keras, Apache MXNet and TensorFlow.

Jan
2016

Microsoft releases and open sources **Computational Network Toolkit (CNTK)**, a DL toolkit it's been using internally for speech and image recognition. CNTK supports multiple GPUs across multiple machines.

Apr
2016

Python-based open source library **Keras 1.0** is released. It's a major re-write of Keras while being backward compatible. It's been said that Keras has an "API designed for human beings, not machines".

2017

Among the frameworks to come out in 2017 is **Caffe2** from Facebook. **Deeplearning4j** for the Java and Scala communities becomes part of Eclipse Foundation although it's origins can be traced to 2014. **PyTorch** is open sourced by Facebook and it starts to become popular when DL course *fast.ai* adopts it.

Dec
2017

With so many DL frameworks, the landscape can look very fragmented for developers. **Open Neural Network Exchange (ONNX)** is released as an open format that enables developers to export/import models from/to frameworks. For example, you can build a PyTorch model, export it in ONNX format, and import into MXNet where it can be used for inference.

Feb
2018

An analysis of arXiv 45,000 ML papers show the popularity of TensorFlow, having overtaken Caffe in 2017. Adoption of Keras and PyTorch also appear to be growing.

May
2018

Caffe2 and PyTorch, both out of Facebook, plan to come together into a single platform. The intent is to "combine the flexible user experience of the PyTorch frontend with scaling, deployment and embedding capabilities of the Caffe2 backend".

Oct
2018

Based on PyTorch, v1.0 of **fastai** is released as a free open source library for DL.

Deep Learning Frameworks


# 5G UE RRC States

## Summary

Radio Resource Control (RRC) is a layer within the 5G NR protocol stack. It exists only in the control plane, in the UE and in the gNB. The behaviour and functions of RRC are governed by the current state of RRC. In 5G NR, RRC has three distinct states: RRC\_IDLE, RRC\_CONNECTED and RRC\_INACTIVE.

For each UE RRC state, applicable functions are clearly defined in the standard. Moreover, it's also defined how state transitions will happen, not only within 5G but also to 2G/3G/4G systems via handover or PLMN/cell reselection.

RRC\_INACTIVE is not applicable for Non-Standalone (NSA) mode of operation.

## Discussion

### Why was RRC_INACTIVE state introduced in 5G?

Before 5G, RRC had only two states, idle and connected. In connected state, radio resources are allocated to the UE and typically active communication (user plane or control plane) is taking place between the UE and the network. Otherwise, UE is in idle state.

While releasing an RRC connection is good for capacity utilization and power saving, it's not ideal from a latency perspective. For Machine Type Communications (MTC) and IoT applications, it's typical for devices to send small amounts of data. The overhead in establishing an RRC connection to do this is bad from a power perspective. The extra signalling also introduces delay that's not ideal for URLLC use cases.

Since 5G caters to new cases such as mMTC and URLLC, RRC\_INACTIVE has been introduced. While entering this state, both UE and NG-RAN save radio and security configurations. This saved UE Inactive Access Stratum (AS) context can be quickly restored with minimal signalling when moving to connected state.

Essentially, RRC\_INACTIVE is UE RRC's way of implementing an "always on" radio connection with the network.

### What are the functions in each of the 5G UE RRC states?

In RRC\_INACTIVE and RRC\_CONNECTED, UE and NG-RAN store **AS Inactive context** and **AS context** respectively. In RRC\_IDLE, UE may be registered with the Core Network (CN) but no AS context is stored.

In both RRC\_IDLE and RRC\_INACTIVE, UE does measurements of neighbouring cells and can do **reselection**. In RRC\_CONNECTED, UE mobility is controlled by the CN and **handovers** can be initiated. Data transfer, CA, DC and measurement reporting are supported.

In RRC\_IDLE, **UE paging** is initiated by the CN. In RRC\_INACTIVE, UE paging is initiated by NG-RAN. To page a UE, it's location must be known. In RRC\_IDLE, this is the Tracking Area (RA). In RRC\_INACTIVE, this is the RAN-based Notification Area (RNA) and UE may initiate RNA updates.

In RRC\_IDLE, **Discontinuous Reception (DRX)** is configured by higher layers. In RRC\_INACTIVE, configuration comes from higher layers and RRC. In RRC\_CONNECTED, DRX configuration is for discontinuously monitoring DL PDCCH.

### How does UE RRC transition from one state to another in 5G NR?

RRC\_IDLE to RRC\_CONNECTED happens via the RRC Connection Setup procedure. This consists of three messages: RRCSetupRequest (UE initiated), RRCSetup, and RRCSetupComplete.

RRC\_CONNECTED to RRC\_IDLE is via RRC Connection Release procedure with network-initiated RRCRelease message. Upper layers in the UE may also request a release. RRC connection is also released due to radio link failure, handover failure or cell not meeting cell selection criteria.

RRC\_CONNECTED to RRC\_INACTIVE is network initiated. It's entered via RRCRelease message with `suspendConfig` information element (IE). When UE is using a Dual Active Protocol Stack (DAPS) bearer or is redirected to an inter-RAT carrier frequency, this IE is not configured.

RRC\_INACTIVE to RRC\_CONNECTED is triggered by the network via RAN paging. A paged UE will start with RRC Connection Resume procedure consisting of three messages: RRCResumeRequest, RRCResume (or RRCSetup), RRCResumeComplete (or RRCSetupComplete). This procedure can also be initiated by UE for uplink transfer, including RNA update.

RRC\_INACTIVE to RRC\_IDLE happens when network responds to RRCResumeRequest with RRCRelease. Alternatively, the UE may be asked to remain in RRC\_INACTIVE for some more time.

### How do UE RRC states map to states in other layers of the protocol stack?

When RRC is in either RRC\_INACTIVE or RRC\_CONNECTED states, there's a Non-Access Stratum (NAS) connection between the UE and the CN. Thus, Connection Management (CM) is in the Connected state.

In RRC\_CONNECTED, Mobility Management (MM) may be in either Deregistered or Registered state. If the UE is in the process of registering to the network (attach procedure), then MM is in Deregistered state. Otherwise, it's in Registered state.

When a UE is just powered up, RRC is in RRC\_IDLE and MM is in Deregistered state. After MM registration, RRC can move from RRC\_CONNECTED to RRC\_INACTIVE, and return to RRC\_IDLE only as part of deregistration (detach procedure). It's also possible for a UE to move to RRC\_IDLE and still be available for CN-initiated paging. For simplicity, this case in not shown in the figure.

### Could you describe RRC states and transitions across different cellular generations?

**Reselection** can happen in idle states. In 2G, GSM\_Idle and GPRS Packet\_Idle are the idle states. In 3G, reselection is also possible from connected states Cell\_FACH, Cell\_PCH and URA\_PCH.

Reselection is possible from GPRS Packet Transfer Mode. Cell Change Order (CCO) is possible from 2G idle state, GPRS Packet Transfer Mode and E-UTRA RRC Connected. For reselection from UTRAN to GERAN, CCO can be assisted with Network Assisted Cell Change (NACC).

In connected states, **handovers** are possible. For handovers, relevant states are GSM\_Connected, GPRS Packet Transfer Mode (2G); Cell\_DCH (3G); E-UTRA RRC Connected (4G); and NR RRC\_CONNECTED (5G).

## Milestones

2015

In the 5G-PPP European project *METIS-II*, the main 5G pre-standards project, there's discussion and effort towards supporting **an RRC inactive state**. In June 2016, a draft titled *Draft Asynchronous Control Functions and Overall Control Plane Design* is published with mention of **RRC Connected Inactive** state.

Aug
2016

LTE RRC specification TS 36.331, Release 13, version 13.2.0 is published. In this release, LTE RRC has only two states, RRC\_IDLE and RRC\_CONNECTED. This version includes **suspend/resume of RRC connection**. When suspended, RRC goes to RRC\_IDLE. However, the connection can be resumed with minimal signalling due to stored UE AS context. This feature caters to NB-IoT requirements.

Dec
2017

3GPP publishes Release 15 "early drop". In 5G RRC specification TS 38.331, version 15.0.0, there are **three states**: RRC\_IDLE, RRC\_CONNECTED and RRC\_INACTIVE. RRC\_INACTIVE state applies only in SA mode, that is, it's not applicable in EN-DC (E-UTRA-NR Dual Connectivity).

Mar
2018

3GPP publishes version 1.0.0 of specification TS 38.304 titled *User Equipment (UE) procedures in idle mode and in RRC Inactive state*. In June, this is re-versioned to 15.0.0 for Release 15. Version 16.3.0 comes out in January 2021. It's mentioned that, "The UE initiates RRC Connection Resume procedure upon receiving RAN initiated paging. If the UE receives a CN initiated paging in RRC\_INACTIVE state, the UE moves to RRC\_IDLE and informs NAS."

Oct
2018

In LTE RRC specification TS 36.331, version 15.3.0, **RRC\_INACTIVE state** is introduced. At the same time, 5G RRC specification TS 38.331, version 15.3.0 includes ASN.1 definition for RRCRelease message. This include `suspendConfig` IE that indicates transition to RRC\_INACTIVE. Periodic RNA Update timer can range from 5-720 minutes.

Dec
2018

Hailu et al. publish performance analysis showing improvements due to RRC\_INACTIVE state. They use the term "RRC Connected Inactive" for this and refer to LTE Release 13 suspend/resume procedure as "RRC Suspended". Compared to LTE idle state, RRC\_INACTIVE brings 8x latency improvement, 5x power efficiency and 3.5x less signalling.

Jul
2020

3GPP publishes Release 16 specifications. In 5G RRC specification TS 38.331, version 16.1.0, `preferredRRC-State` IE is introduced as part of UE's **release preference**. UE can indicate preference towards RRC\_IDLE, RRC\_INACTIVE, leave RRC\_CONNECTED without any preference for the next state or revert to an earlier indicated preference.

5G UE RRC States


# Named Entity Recognition

## Summary

**Named Entity Recognition (NER)** is an essential task of the more general discipline of Information Extraction (IE). To obtain structured information from unstructured text we wish to identify named entities. Anything with a proper name is a **named entity**. This would include names of people, places, organizations, vehicles, facilities, and so on.

While temporal and numerical expressions are not about entities per se, they're important in understanding unstructured text. Hence, such expressions are included in NER.

The essence of NER is to **identity** the named entities and also **classify** them. NLP techniques used for POS tagging and syntactic chunking are applicable to NER.

NER models trained on general newswire text are not suitable for specialized domains, such as law or medicine. Domain-specific training is required.

## Discussion

### Which are the common entity types identified by NER?

At its simplest, NER recognizes three entity classes: location, person, and organization.  In practice, at least for newswire text, there's value in extracting geo-political entities, date, time, currency, and percent. For more fine-grained NER, systems may identify ordinal/cardinal numbers, events, works of art, facilities, products, and so on.

Temporal expressions can be absolute or relative. For example, 'summer of 1977' and '10:15 AM' are absolute whereas 'yesterday' and 'last quarter' are relative. Expression 'four hours' is an example of duration.

Specialized domains often require additional or alternative entity types. In biomedical for example, we can broadly define six types: Cell, Chemical, Disease, Gene (DNA or RNA), Protein and Species. For question answering, Sekine defined a hierarchy of more than 200 entities.

### Could you describe some applications of NER?

Given the huge volume of content published online each day, NER helps in categorizing them and thus eases search and content discovery. In fact, for information retrieval, named entities can be used as search indices to speech up search. Likewise, NER helps organize and categorize research publications. For example, from the thousands of papers on machine learning, we might be interested only in face detection that uses CNN.

Content recommendation is possible with NER. Based on the entities in the current article, the reader can be recommended related articles. Another NER application is online customer support. NER can identify product type, model number, store location, and more. Opinion mining uses NER as a pre-processing task. NER can help link related concepts and entities for the Semantic Web.

One developer took 30,000 online recipes but found that the descriptions didn't fit any particular pattern. He therefore applied NER with entity types Name, Quantity, and Unit. He manually labelled 1500 samples of about 10,000 tokens.

Among the NLP tasks that benefit from NER are question generation, relation extraction, and coreference resolution.

### What are the typical challenges with NER?

Ambiguities make NER a challenging task. For example, 'JFK' can refer to former US President John F. Kennedy or his son. These are different entities of the same type. **Coreference resolution** is an NLP task that resolves this ambiguity. More common to NER is when the same name refers to different types. For example, 'JFK' can refer to the airport in New York or Oliver Stone's 1991 movie.

Since entities can span multiple words/tokens, NER needs to identify start and end of multi-token entities. A name within a name (such as *Cecil H. Green Library*) is also a challenge. When a word is a qualifier, it may be wrongly tagged. For example, in *Clinton government*, an NER system may tag Clinton as PER without recognizing the noun phrase.

Same entity can appear in different forms. Differences could be typographical (nucleotide binding vs NUCLEOTIDE-BINDING), morphological (localize vs localization), syntactic (DNA translocation vs translocation of DNA), reduction (secretion process vs secretion/ATP-binding activity), or abbreviated (type IV secretory system vs T4SS).

### What's the typical processing pipeline in NER?

NER is typically a **supervised** task. It needs annotated training data. Features are defined. From the annotated data, the ML model learns how to map these features to the entities. NER can be seen as a sequence labelling problem since it identifies a span of tokens and classifies it as a named entity. Thus, NER can be solved in a manner similar to POS tagging or syntactic phrase chunking. Statistical models such as HMM, MEMM or CRF can be used.

Annotating data manually is slow. A semi-automated approach is to give a file containing a set of patterns or rules. Another approach is to start with an existing model and manually correct mistakes.

A practical approach is for a model to label unambiguous entities in the first pass. Subsequent passes make use of already labelled entities to resolve ambiguities about other entities.

Essentially, iterative approaches that start with patterns, dictionaries or unambiguous entities start with high precision but low recall. Recall is then improved with already tagged entities and feedback. Manually annotated data is typically reserved for model evaluation.

### What sort of features are useful for training an NER model?

The token and its stem are basic features for NER. In addition, POS tag and phrase chunk label give useful information. **Context** from surrounding tokens, and their POS tags and chunk labels, are also useful inputs. For example, 'Rev.', 'MD', or 'Inc.' are good indicators of entities that precede or follow them.

The token's shape or **orthography** is particularly useful. This includes presence of numbers, punctuation, hyphenation, mixed case, all caps, etc. For example, A9, Yahoo!, IRA, and eBay are entities that can be identified by their shape. While this is useful for newswire text, it's less useful for blogs or text transcribed automatically from speech. Also, there's no case information in Chinese.

**Gazetteers** maintain a list of names of places and people. Presence of a token in such a list can be feature. Maintaining such a list is often difficult. Such a list is less useful for identifying persons and organizations. Likewise, domain-specific knowledge bases can be used and string matches with these bases can identify entities. Examples include DBpedia, DBLP and ScienceWise.

### How have neural networks influenced NER?

Linear statistical models used hand-crafted features and gazetteers. These are hard to adapt to new tasks or domains. Non-linear neural network models make use of word embeddings. Early NN models used these embeddings along with hand-crafted features.

A neural network for NER typically has three components: word embedding layer, context encoder layer, and decoder layer. Since NER is basically a sequence labelling task, RNN is suitable. BiLSTM in particular is good at capturing context. Since 2019, transformer architecture has been successfully applied to NER. Multi-task learning (MTL) is an approach in which the model is trained on different tasks and representations are shared across tasks.

For NER, character-level representations obtained via CNN or transformer are useful since they encode morphological features, alleviate out-of-vocabulary issues and overcome data sparsity problem.

Neural networks are used by NeuroNER (LSTM), spaCy (CNN), and AllenNLP (Bidirectional LM).

### How are words or phrases tagged with their identities in NER?

Entities have to be tagged in a manner that's suitable for algorithms to process. There are many tagging schemes available, some of which evolved from those used in chunking:

    + **IO**: The simplest scheme, where *I\_X* identifies the named entity type X and *O* indicates no entity.
    + **IOB**: Where an entity has multiple tokens, this differently tags the begin token (B) from an inner token (I). A variation of this is *IOB2* or *BIO*.
    + **BMEWO**: This differently tags begin token (B), middle token (M), end token (E), and a single-token entity (W). An extension of this with more tags is called *BMEWO+*. Another scheme that's similar is called *BILOU*, where the letters stand for begin, inner, last, out and unit.These encoding schemes have different complexities. For N entities, IO, BIO and BMEWO have complexity of N+1, 2N+1 and 4N+1 tags respectively.

While BIO has been a popular choice, more recent work has shown that BILOU outperforms BIO. spaCy supports IOB and BILUO schemes.

### How do we evaluate algorithms for NER?

Recall and precision are the basic metrics for evaluating NER models. **Recall** gives the ratio of correctly labelled entities to the total that should have been labelled. **Precision** is the ratio of correctly labelled entities to the total labelled. **F-measure** is a single metric that combines both recall and precision.

One caveat in using these metrics is the scope. In a real application, they have to be applied to actual named entities. Typical ML models optimize performance at the tag level, based on a particular encoding scheme. Performance at these two levels can be quite different.

Tagging details also matter. For example, in one study, training data did not include titles (Prime Minister, Pope, etc.) but Amazon Comprehend included these, thus leading to higher recall but lower precision. Different annotated datasets sometimes disagree about entities. For example, in "Baltimore defeated the Yankees", MUC-7 tags Balimore as LOC whereas CoNLL2003 tags it as ORG.

CoNLL2003 and OntoNotes 5.0 are commonly used for training and evaluating models. AllenNLP used One Billion Word Benchmark.

### What resources are available for research into NER?

As part of the GATE framework (University of Sheffield, UK) for text processing, ANNIE is an NER pipeline. Researchers can try out the online demo plus a free API.

displaCy from Explosion is a useful visualization tool. Prodigy is an annotation tool for creating training data. NER is one of the supported tasks. A 2018 survey lists English NER datasets and tools.

Cloud providers also offer APIs for NER. Google's Natural Language API is an example. This API can also figure out the sentiments about the entities. An equivalent offering from Amazon is Amazon Comprehend.

Stanford NER is a Java implementation. It's also called *CRFClassifier* because it's based on linear chain CRF sequence model. Via Stanford CoreNLP, this can be invoked from other languages.

More generally, packages that support HMM, MEMM, or CRF can be used to train an NER model. Such support is available in Mallet, NTLK, and Stanford NER. A Scikit-Learn compatible package that's useful is `sklearn-crfsuite`. Polyglot is capable of doing NER for 40 different languages.

## Milestones

Feb
1991

Lisa Rau implements an algorithm to extract company names from financial news. It's a combination of heuristics, exception lists and extensive corpus analysis. In subsequent retrieval tasks, the algorithm also looks at most likely variations of names. In 1992, she files for a US patent, which is granted in 1994.

1996

The term **Named Entity** is first used at the 6th Message Understanding Conference (MUC). The first MUC was held in 1987 with the aim of automating analysis of text messages in the military. While early MUC events focused on mainly template filling, MUC-6 looks at sub-tasks that would aid information extraction. It's in this context that named entities become relevant. SGML is used to markup entities. During 1996-2008, NER is talked about in MUC, CoNLL and ACE conferences.

2003

Hammerton applies **Long Short-Term Memory (LSTM)** neural network to NER. He finds significantly better performance for German but a disappointing baseline performance for English. Words and their sequences are represented using SARDNET. The algorithm operates in two passes. In the first pass, information is gathered. In the second pass, the algorithm disambiguates and outputs the named entities.

2008

NER is dropped from international evaluation forums. In fact, by 2005, NER was considered a solved problem, with models achieving recall and precision exceeding 90%. However, the best score on ACE 2008 is only about 50%, which is significantly lower that the scores for MUC and CoNLL2003 tasks. This suggests that NER is not a solved problem.

Jun
2009

Ratinov and Roth address some design challenges for NER. They note that NER is knowledge intensive in nature. Therefore, they use 30 **gazetteers**, 16 of which are extracted from Wikipedia. In general, these are high-precision, low-recall lists. They are shown to be effective for webpages, where there's less contextual information. Expressive features and gazetteers enable unsupervised learning. They also note the BILOU encoding significantly outperforms BIO.

Jun
2011

Tweets have the problem of insufficient information. NER models trained on news articles also do poorly on tweets. This can be solved by domain adaptation or semi-supervised learning from lots of unlabelled data. Liu et al. approach this problem with a combination of **K-Nearest Neighbour (KNN) classifier** and **Conditional Random Field (CRF) labeller**. KNN captures global coarse evidence while CRF captures fine-grained information from a single tweet. The classifier is retrained based on recently labelled tweets. Gazetteers are also used.

Aug
2011

To avoid task-specific engineering and hand-crafted features, Collobert et al. propose a unified neural network approach that can perform part-of-speech tagging, chunking, named entity recognition, and semantic role labelling. The model learns representations from lots of unlabelled data. The best results for NER are obtained with **word embeddings** from a trained language model and optimizing for sentence-level log-likelihood. This work motivates other researchers towards neural networks in preference to hand-crafted features.

Jul
2015

Santos and Guimarães extend the work of Collobert et al. by considering **character-level representations** using a convolutional layer. They note that "word-level embeddings capture syntactic and semantic information, character-level embeddings capture morphological and shape information". The use of both gives best results.

Aug
2015

Huang et al. study the use of LSTM and CRF for sequence labelling. They find that **BiLSTM with a CRF layer** gives state-of-the-art results for POS tagging, chunking and NER. With BiLSTM, past (forward states) and future (backward states) features are used. CRF works at the sentence level to predict the current entity label based on past and future labels. This model shows good performance even when Collobert et al.'s word embedding is not used. For faster training, spelling and context features bypass the BiLSTM and go directly to the CRF layer.

2016

Ma and Hovy achieve state-of-the-art F1 score of 91.21 for NER on CoNLL 2003 dataset. Their approach requires no feature engineering or specific data pre-processing. They use CNN to obtain character-level representations to capture morphological features such as prefixes and suffixes. GloVe word embeddings are used. Character embeddings are randomly initialized and then used to obtain character-level representations. Chiu and Nichols present a similar work that also uses word-level features.

Jul
2018

Güngör et al. study NER for morphologically rich languages such as Turkish, Finnish, Czech and Spanish. Word morphology is important for these languages. This is in contrast to English that gets useful information from syntax and word n-grams. They therefore propose a model that uses **morphological embedding**. This is combined with character-based and word embeddings. Both character-based and morphological embeddings are derived using separate BiLSTMs.

Jul
2019

Francis et al. make use of BERT language model for **transfer learning** in NER. For fine tuning, either softmax or CRF layer is used. They find BERT representations perform best in combination with character-level representation and word embeddings. Without BERT, they also show competitive performance when character-level representation and word embeddings are gated via an attention layer.

Dec
2019

Yan et al. note that the traditional transformer architecture is not quite as good for NER as it is for other NLP tasks. They customize transformer architecture and achieve state-of-the-art results, beating prevailing BiLSTM models. They call it **Transformer Encoder for NER (TENER)**. While traditional transformer uses position embedding, directionality is lost. TENER uses relative position encoding to capture distance and direction. Smoothing and scaling of traditional transformer is seen to attend to noisy information. TENER therefore uses sharp unscaled attention.

Named Entity Recognition


# Convention over Configuration

## Summary

It's often necessary to configure a software project correctly before it can be used. For example, the IDE or the build tool should know where to find the source files. It should know where to look for dependencies or save the output of a build. In web applications, it should know how to connect URL paths to application logic, and database tables and columns.

In a **configuration** approach, all this information is supplied via configuration files, often as XML/YAML/JSON files. However, writing and maintaining these files can be tedious. In a **convention** approach, reasonable defaults are used. The conventions are well defined. Once developers learn and use these conventions, they can focus on more important tasks instead of maintaining configuration files.

Many frameworks have adopted Convention over Configuration (CoC) while also giving developers means to override the defaults.

## Discussion

### Could you explain convention over configuration with some examples?

In ASP.NET MVC, the convention is to have the folders `Controllers`, `Models` and `Views`. Accessing the homepage routes to `HomeController` class in file `Controllers/HomeController.cs` where the `Index()` method is called. This method returns a view that's in file `Views/Home/Index.cshtml`. All this "magic" happens due to conventions on folder structure and naming. No configuration files are needed to tell the framework how all these parts are "wired together".

In a Java project built with Gradle, with a single line of code we can apply the Java plugin to this project. Gradle then automatically looks for source code in `/src/main/java`. It creates a `/build` folder where the compiled class and JAR files are saved. This is all by convention. We don't have to tell Gradle these mundane details via configuration files.

In *pytest*, a Python-based test framework, test cases are automatically discovered. The framework looks at the current directory for `test_*.py` or `*_test.py` files. In those files, it picks out functions or methods with prefix `test`. For methods, it picks out classes with prefix `Test`.

### In what aspects of software is convention over configuration applicable?

In Ruby on Rails, CoC can be seen in many areas within its **Model-View-Controller architecture**. Model class names are derived from database table names. Class attributes are based on database column names. View filenames and templates are based on controller action names. A web URL is mapped to controller class by CoC.

Many other **design patterns** benefit from CoC. By implementing an ASP.NET interface according to a defined convention, we can easily create a message handler without any further configuration. To support dependency injection, many frameworks adopt CoC to greatly simplify wiring of dependencies.

**Build systems** such as Maven make use of CoC. For example, paths to source code, resources, tests and many others are specified by defaults. Maven's core plugins also apply defaults for "compiling source code, packaging distributions, generating web sites, and many other processes".

API endpoints designed by different developers can look and behave differently. By adopting CoC, **APIs** become more standardized and simpler to use. JSON-API is an example that specifies shared conventions.

In **test frameworks** such as *pytest*, conventions are used to discover test cases.

### What are the benefits of convention over configuration?

CoC frees developers from making mundane decisions, such as whether to name a database column `id`, `postId`, `posts_id` or `pid`. Developers can focus on application logic or develop deeper abstractions. CoC makes it easier for beginners to get productive without having to read a lot of the existing codebase. It's easier to collaborate since developers follow the same conventions.

When applied to design patterns (such as dependency injection), CoC makes code less verbose, more readable, and more maintainable. Code that does only "housekeeping" can be eliminated. Ultimately, CoC lets developers focus on delivering functionality rather than deal with low-level infrastructure.

Often adding new functionality is easier. There's no need to update configuration files. Conventions either at the language level or framework level take care of the "wiring".

The CoC philosophy can be extended to engineering disciplines. An example is model-driven engineering (MDE). The idea is to abstract away low-level details so that developers can focus on higher-level design and implementation decisions. Code generators can use conventions to generate low-level code.

### Which are some frameworks that adopt convention over configuration?

Among MVC web frameworks that adopt CoC are Ruby of Rails, ASP.NET MVC, CakePHP, Laravel, Symfony, Yii, Grails, Ember.js, Meteor, Nest.js, and Spring Framework.

Among build systems using CoC are Maven and Gradle.

### What are some criticisms of convention over configuration?

Frameworks that adopted CoC are called **opinionated**. One common criticism is that you're forced to follow a convention you may not like. However, frameworks often allow developers to customize.

Too much convention leads to confusion and a steeper learning curve. Too little convention gives more freedom but at the cost of complexity. The key is therefore to achieve the right level of convention.

Unless you know the framework well, reviewing other people's code is harder; and searching through files won't help. For example, finding out how a URL maps to a controller or a view can be challenging. For beginners, the code may not be obvious. As a solution, it's better to limit CoC to conventions that are commonly understood and frequently used. If you override conventions, document them explicitly.

CoC can lead to unexpected behaviour. For example, ASP.NET finds classes inherited from a base Controller class. If the project references an assembly that has controllers, we may unknowingly expose those endpoints. This is one instance why some prefer explicit configuration, such as in Python and Django.

## Milestones

2005

First version of **Ruby on Rails** is released. Convention over configuration is one of the pillars of Ruby on Rails. It brings "default structures for web pages, databases, and web services". Inspired by this, Sensio Framework is launched as a PHP MVC framework that adopts CoC. This is later renamed to **Symfony**.

May
2006

**Enterprise JavaBeans (EJB) 3.0** is released. To overcome the complexity of earlier releases, EJB 3.0 adopts CoC by way of annotations.

2009

Microsoft releases the first version of **ASP.NET MVC**. Although ASP.NET itself started in 2002 and the MVC pattern was common by 2003, ASP.NET MVC formalizes it. ASP.NET MVC adopts CoC, taking inspiration from Ruby on Rails. It's in response to declarative XML configuration that's being overused.

May
2010

Jimmy Bogard releases v1.0 of **AutoMapper** that's described as "a convention-based object-object mapper in .NET".

Convention over Configuration


# Bloom Filter

## Summary

Given a set of elements, suppose we wish to know if a particular element is present in this set. There are many searching algorithms to do this. When faced with a huge set (millions of elements), even with an efficient search algorithm, storage is a problem. There's also latency due to disk access. Bloom filter is one possible solution.

Bloom filter is a data structure that stores the original set in a more compact form with support for **set membership queries**, that is, to query if an element is a member of the set. Bloom filter is a **space-efficient probabilistic data structure**.

With the rise of big data since the mid-2000s, there's been increased interest in Bloom filter. From the original Bloom filter (1970), many variants have been proposed for a diverse range of applications.

## Discussion

### Could you explain the Bloom filter with an example?

A Bloom filter is simply a **bit array** of length \(m\) for storing elements of set \(S=\{x\_1,x\_2,\ldots,x\_n\}\). The filter starts with all zeros, meaning that the set is empty. When an element is added, it is hashed using \(k\) **independent hash functions**. Each function will output a bit position in the filter, which is then set to one. Thus, each element will set \(k\) bits of the filter.

When a query is made, the queried element is put through the \(k\) hash functions. The resulting bit positions are checked in the filter. If at least one of these is a zero, we're certain that the element is not in the set. If all bits are ones, then it *may be* in the set. Thus, the filter is probabilistic. The reason it's probabilistic is that a query may check against bit positions set by more than one stored element.

In the figure, we can say that \(x\_4\) is definitely not in S but \(x\_1\) (*true positive*) and \(x\_5\) (*false positive*) are most probably in S.

### Under what conditions can I use the Bloom filter?

Suppose the records of a large database table are stored on disk and need to be queried. To avoid slow disk access, a Bloom filter representation of the data is kept in fast memory. If the Bloom filter says that a record is absent, we're sure that it's absent and avoid disk access. If the Bloom filter says the record is present, we're not sure. We need to query the database to find out. **False positive** is when the filter says something is present when it's not. False positives should be at a level acceptable to the application. When most queried records are absent in the database, Bloom filter gives performance gains.

Back in 2003, Broder and Mitzenmacher stated the *Bloom filter principle*,

> Wherever a list or set is used, and space is at a premium, consider using a Bloom filter if the effect of false positives can be mitigated.

Multiple elements can set the same bit in the filter. Therefore, elements once added can't be removed from the filter. Neither can we obtain the list of elements added. Standard Bloom filter is not suited for applications that need these features.

### How do I select the parameters of the Bloom filter?

There are essentially three parameters: filter size \(m\), number of hash functions \(k\) and number of elements to be stored \(n\).

For lower false positives, use higher values of \(m\) and \(k\) with a tradeoff. Higher \(m\) means more memory consumption. Higher \(k\) means more computation. Since higher \(k\) sets more bits, there's an optimal value \(k\_{opt}\) beyond which false positives start to increase.

When adding elements beyond the design limit, false positives go up. To achieve a fixed false positive probability, we should linearly scale \(m\) as \(n\) increases. In fact, the bit-to-element ratio is bounded as \(m/n\geqslant1/ln(2)\).

In practice, we estimate \(n\) in advance for our application. We decide on the false positive probability we're willing to tolerate. We select \(m\) and calculate \(k\), or select \(k\) and calculate \(m\). A handy calculator is available online.

If parameters are not chosen correctly, an attacker can corrupt the filter with well-chosen inputs.

### Which hash functions are suitable for use in a Bloom filter?

Hash functions should ideally be independent, meaning that for the same input they set different bits of the filter. Otherwise, hash collisions become more common, leading to higher false positives. Functions should also distribute the input space uniformly.

Hash functions should be fast. The use of cryptographic hash functions is possible but not really required. Among the non-cryptographic hash functions suited for Bloom filter are MurmurHash, Fowler–Noll–Vo (FNV) and Jenkins. One developer showed how replacing MD5 with MurmurHash brought performance improvements.

It's possible to generate \(k\) hash functions from just two hash functions. One researcher pointed out that if the filter size is 32 bits, on a 64-bit machine a single hash function can give two 32-bit hash values, thus simulating two hash functions.

### Which are some real-world applications of the Bloom filter?

For network-related applications, Bloom filter is used to collaborate in peer-to-peer networks, resource routing, packet routing, and measurements. Content Delivery Networks (CDNs) use Bloom filters to avoid caching files seen only once.

For applications that use databases, Bloom filter enables efficient searches, privacy preservation, content synchronization, and duplicate detection.

Medium uses Bloom filter to deduplicate recommendations. In a streaming application processing 17 million events per day per partition, Bloom filter was used to deduplicate events at scale. The filter needed 108Gb across 1024 partitions. Reads and writes were 20x and 3x faster respectively.

Chrome browser uses the filter to represent a set of malicious URLs. When user requests a URL, if the filter indicates a probable match, the request is sent to a server to check if the URL is indeed safe.

In a web application, Bloom filters could be used to keep count of visitors coming from a specific city accessing a webpage.

### How's the performance of the Bloom filter?

A Bloom filter with one hash function is equivalent to ordinary hashing. Bloom filter is a generalization of hashing. It allows for more interesting design tradeoffs.

The complexity to add an element or to query the filter is fixed at \(O(k)\). In other words, it's independent of how many items have been added to the filter.

Bloom filter simplifies some operations. Suppose we have two filters to represents sets \(S\_1\) and \(S\_2\). The union of the these two sets is simply an OR operation of the two filters. Intersection of the two sets can also be approximated with the filters. Another trick is to half the size of the filter with an OR operation of the first and second halves of the filter.

### What are some variants of the Bloom filter?

Bloom filter has inspired dozens of variants. One survey paper from 2019 mentioned more than 60 variants, their capabilities, performance and application areas. A variant might aim to reduce false positives, improve performance or provide additional features.

**Counting Bloom filter** allows us to implement a multiset so that multiple instances of the same element can be stored. Some of these support deletions as well but at the cost of false negatives. Bitwise Bloom filter and spectral Bloom filter improve on the counting Bloom filter.

When elements are added to a filter beyond it's designed capacity, false positives increase. **Scalable Bloom filter** allows us to grow the filter size as more items are added.

When filters have to be exchanged over a network (such as in web caching applications), **Compressed Bloom filter** saves on network bandwidth. By choosing the hash functions and thereby changing the way bits are distributed in the filter, better compression is achieved.

### Could you mention some useful resources on the Bloom filter?

Survey paper by Luo et al. (2019) is worth reading. Jason Davies has published online an interactive visualization of the Bloom filter. JavaScript code for this is also available.

C++ library libbf contains implementations of the Bloom filter and some of its variants.

 A Java implementation of the filter (and other probabilistic data structures) can be found within stream-lib. Guava library in Java also has an implementation.

Packages offering the filter are available in Python as well as in Node.js.

## Milestones

1965

In the construction of a symbol table in compiler design, Batson describes a method that does faster search for symbols. Rather than do a slow linear lookup, an arithmetic transformation is done on the symbol. The result gives the address where the symbol will be stored (or retrieved) in the table. The term **hashing** becomes common by late 1960s for this sort of transformation on the input.

1970

Burton H. Bloom publishes a paper titled *Space/Time Trade-offs in Hash Coding with Allowable Errors*. In conventional hash-coding methods, many lookups are needed when an element doesn't belong to the set. He therefore proposes a novel method to reduce the *reject time*, time taken to declare that an element is a non-member of the set. To allow for smaller hash area, he accepts some fraction of errors. With smaller hash area, hashes can be stored in core memory for faster access.

2000

Fan et al. propose the **Counting Bloom Filter** in the context web proxy caching. Since caches have to be updated, they find it necessary to keep a count of how many times a bit position in the filter has been set. They propose 4 bits per filter bit while keeping false negatives low. When a request comes in, the proxy checks its own cache. If absent, it checks the Bloom filters of participating proxies. Based on the result, it decides whether to query another proxy. Bloom filters are updated among proxies based on defined thresholds.

2004

Shanmugasundaram et al. propose the **Hierarchical Bloom Filter** to address the problem of substring matching. In addition to the substrings, the filter also stores concatenation of substrings, thus creating an hierarchy. It's applied in the context of IP traffic to attribute payloads to source and destination hosts.

2006

Borrowing from standard hashing, Kirsch and Mitzenmacher propose how to simulate multiple hash functions for the Bloom filter from just two independent uniform random hash functions. Their \(k\) hash functions are of the form \(g\_i(x) = h\_1(x) + i \cdot h\_2(x)\,mod\,p\), for \(i\) in range \([0,k-1]\) and where \(p\) is prime, the filter has \(kp\) bits and each function has range \([0,p-1]\).

2007

Almeida et al propose the **Scalable Bloom Filter**. Sometimes it's difficult to estimate upfront the maximum number of elements that need to be stored in a Bloom filter. Designers therefore tend to waste space by overestimating, sometimes by orders of magnitude. By allowing the filter size to scale dynamically, the scalable Bloom filter guarantees that false positives don't exceed a defined design threshold. It's parameterized by initial size, error probability, growth rate and error probability tightening rate.

2008

Bose et al. observe that the original analysis of false positive rate by Bloom and others is not exact. The give an exact formula for the false positive rate plus lower and upper bounds. The rate is slightly larger than originally calculated but the difference is negligible for large \(m\) and small \(k\).

2019

Rae et al. propose the **Neural Bloom Filter** that uses an artificial neural network with one-shot meta-learning. Softmax is computed on content-based attention between the query *q* and a learned address matrix *A*. Thus the network learns where to store elements based on their content. This is more space efficient but at the cost of performance for a single query. It outperforms standard Bloom filter on batch queries parallelized on GPUs.

Bloom Filter


# Algorithmic Complexity

## Summary

Algorithmic complexity is a measure of how long an algorithm would take to complete given an input of size n. If an algorithm has to scale, it should compute the result within a finite and practical time bound even for large values of n. For this reason, complexity is calculated asymptotically as n approaches infinity. While complexity is usually in terms of time, sometimes complexity is also analyzed in terms of space, which translates to the algorithm's memory requirements.

Analysis of an algorithm's complexity is helpful when comparing algorithms or seeking improvements. Algorithmic complexity falls within a branch of theoretical computer science called computational complexity theory. It's important to note that we're concerned about the order of an algorithm's complexity, not the actual execution time in terms of milliseconds.

Algorithmic complexity is also called *complexity* or *running time*.

## Discussion

### Can you give real-world examples of various algorithmic complexities?

Suppose you're looking for a specific item in a long unsorted list, you'll probably compare with each item. Search time is proportional to the list size. Here complexity is said to be *linear*.

On the other hand, if you search for a word in a dictionary, the search will be faster because the words are in sorted order, you know the order and can quickly decide if you need to turn to earlier pages or later pages. This is an example of *logarithmic* complexity.

If you're asked to pick out the first word in a dictionary, this operation is of *constant* time complexity, regardless of number of words in the dictionary. Likewise, joining the end of a queue in a bank is of constant complexity regardless of how long the queue is.

Suppose you are given an unsorted list and asked to find all duplicates, then the complexity becomes *quadratic*. Checking for duplicates for one item is of linear complexity. If we do this for all items, complexity becomes quadratic. Similarly, if all people in a room are asked to shake hands with every other person, the complexity is quadratic.

### What notations are used to represent algorithmic complexity?

*Big-O notation* is the prevalent notation to represent algorithmic complexity. It gives an upper bound on complexity and hence it signifies the worst-case performance of the algorithm. With such a notation, it's easy to compare different algorithms because the notation tells clearly how the algorithm scales when input size increases. This is often called the *order of growth*.

Constant runtime is represented by \(O(1)\); linear growth is \(O(n)\); logarithmic growth is \(O(log\,n)\); log-linear growth is \(O(n\,log\,n)\); quadratic growth is \(O(n^2)\); exponential growth is \(O(2^n)\); factorial growth is \(O(n!)\). Their orders of growth can also be compared from best to worst:

\(O(1) < O(log\,n) < O(\sqrt{n}) < O(n) < O(n\,log\,n) < \\ O(n^2) < O(n^3) < O(2^n) < O(10^n) < O(n!)\)

In complexity analysis, only the dominant term is retained. For example, if an algorithm requires \(2n^3+log\,n+4\) operations, its order is said to be \(O(n^3)\) since \(2n^3\) is the dominant term. Constants and scaling factors are ignored since we are concerned only about asymptotic.

Audrey Nasar gives formal definitions of Big-O. Wikipedia lists orders of common functions.

### What does it mean to state best-case, worst-case and average time complexity of algorithms?

Let's take the example of searching for an item sequentially within a list of unsorted items. If we're lucky, the item may occur at the start of the list. If we're unlucky, it may be the last item in the list. The former is called *best-case complexity* and the latter is called *worst-case complexity*. If the searched item is always the first one, then complexity is \(O(1)\); if it's always the last one, then complexity is \(O(n)\). We can also calculate the *average complexity*, which will turn out to be \(O(n)\). The term "complexity" normally refers to worst-case complexity.

Mathematically, different notations are defined (example is for linear complexity):

    + Worst-case or upper bound: Big-O: \(O(n)\)
    + Best-case or lower bound: Big-Omega: \(\Omega(n)\)
    + Average-case: Big-Theta: \(\Theta(n)\)
    + Non-tight upper bound: \(o(n)\)
    + Non-tight lower bound: \(\omega(n)\)When upper or lower bounds don't coincide with average complexity, we can call them non-tight bounds.

As an example, Quicksort's complexity is \(\Omega(n\,log\,n)\), \(\Theta(n\,log\,n)\) and \(O(n^2)\).

There's also *amortized complexity* in which complexity is calculated by averaging over a sequence of operations.

### Why should we care about an algorithm's performance when processors are getting faster and memories are getting cheaper?

Complexity analysis doesn't concern itself with actual execution time, which depends on processor speed, instruction set, disk speed, compiler, etc. Likewise, the same algorithm written in assembly will run faster than in Python. Programming languages, hardware and memories are external factors. Complexity is about the algorithm itself, the way it processes the data to solve a given problem. It's a software design concern at the "idea level".

It's possible to have an inefficient algorithm that's executed on high-end hardware to give a result quickly. However, with large input datasets, the limitations of the hardware will become apparent. Thus, it's desirable to optimize the algorithm first before thinking about hardware upgrades.

Suppose your computer can process 10,000 operations/sec. An algorithm of order \(O(n^4)\) would take 1 sec to process 10 items but more than 3 years to process 1,000 items. Comparatively, a more efficient algorithm of order \(O(n^2)\) would take only 100 secs for 1,000 items. With even larger inputs, better hardware cannot compensate for algorithmic inefficiency. It's for this reason algorithmic complexity is defined in terms of asymptotic behaviour.

### Are there techniques to figure out the complexity of algorithms?

Instead of looking for exact execution times, we should evaluate the number of high-level instructions in relation to the input size.

A single loop that iterates through the input is linear. If there's a loop within a loop, with each loop iterating through the input, then the algorithm is quadratic. It doesn't matter if the loops process only alternative items or skip a fixed number of items. Let's recall that complexity ignores constants and scaling factors. Likewise, a loop within a loop, followed by another loop, is quadratic, since we need to consider only the dominant term.

A recursive function that calls itself n times is linear, provided other operations within the function don't depend on input size. However, a recursive implementation of Fibonacci series is exponential.

A search algorithm that partitions the input into two parts and discards one of them at each iteration, is logarithmic. An algorithm such as Mergesort that partitions the input into halves at each iteration, plus does a merge operation in linear time at each iteration, has a log-linear complexity.

### If an algorithm is inefficient, does that mean that we can't use it?

Polynomial complexity algorithms of order \(O(n^c)\), for c > 1, may be acceptable. They can be used for inputs up to thousands of items. Anything exponential can probably work for only inputs less than 20.

Algorithms such as Quicksort that have complexity of \(O(n^2)\) rarely experience worst-case inputs and often obey \(\Theta(n\,log\,n)\) in practice. In some case, we can preprocess the input so that worst-case scenarios don't occur. Likewise, we can go with sub-optimal solutions so that complexity is reduced to polynomial time.

In practice, a linear algorithm can perform worse than a quadratic one if large constants are involved and n is comparable to these constants. It's also important to analyze every operation of an algorithm to ensure that non-trivial operations are not hidden or abstracted away within libraries.

### Since algorithmic complexity is about algorithms, is it relevant to talk about data structures?

Data structures only store data but the algorithmic complexity comes into consideration when we operate on them. Operations such as insertion, deletion, searching and indexing need to be analyzed. The intent is to choose the right data structures so that complexity is reduced.

For example, accessing an array by index has constant complexity whereas the same operation with a linked list has linear complexity. Searching by key in a hash table incurs constant average complexity but this operation is linear with stacks, queues, arrays and linked list. A more detailed discussion on what data structures to use is given by Svetlin Nokav and others.

### Can you state the complexity of well-known sorting algorithms?

The simplest sorting algorithm is probably Bubblesort but it's quadratic in the average case and hence not efficient. Better alternatives are those with log-linear complexity: Quicksort, Mergesort, Heapsort, etc. If the list is already sorted, the best-case complexity occurs with Bubblesort, Timsort, Insertionsort and Cubesort, all completing in linear time.

It's been noted that best and worst cases rarely occur. The average case is based on an input distribution model that could be a random sample as well. Analysis of these averages or abstract basic operations can help us pick the best suited algorithm for a specific problem.

### Can you give the complexity of some important algorithms?

Here's a selection:

    + Fast Fourier Transform: \(O(n\,log\,n)\)
    + Multiply two n-digit numbers using Karatsuba algorithm: \(O(n^{1.59})\)
    + Matrix multiplication due to Coppersmith and Winograd: \(O(n^{2.496})\)
    + Prime recognition of an n-digit integer due to Adleman, Pomerance and Rumley: \(n^{O(log\,log\,n)}\)
    + Gaussian elimination: \(O(n^{3})\) but \(O(x^{2^n})\) bit complexity of its operands
    + GCD(a, b) by Euclid's algorithm: \(O(log\,(a+b))\) but \(O({(log\,(a+b))}^2)\) bit complexity when integers a and b are largeBit complexity matters when integers exceed the 64-bit machine capability. In these cases, when arbitrary precision is used, operations such as division and modulo arithmetic are no longer trivial and their complexity must be accounted for. From a theoretical perspective, this doesn't matter to computer scientists but it matters to programmers who have to work within machine limits.

## Milestones

1864

Charles Babbage, while building his Analytical Engine, predicts the importance of studying the performance of algorithms.

1894

Mathematician Paul Bachmann defines the **Big-O notation**. It originally meant "order of". Decades later, Donald Knuth calls it the capital omicron.

1960

The **theory of computational complexity** is born in the 1960s. This is also the decade when problems that can't be solved in polynomial time (NP problems) are recognized.

1975

Tarjan makes the case that when designing data structures, focusing on worst case running time is less important than **amortized running time**, that is, performance averaged over operations on a long sequence of input. His later contributions include the splay tree (1980) and the Fibonacci heap, both achieving good amortized performance.

1976

Donald Knuth introduces new **notations** and clarifies the meaning of \(O\), \(\Omega\) and \(\Theta\).

1998

R. Libeskin-Hadas introduces the term **oblivious algorithm** that's applied to an algorithm whose complexity is independent of the input structure. In other words, the best-case, worst-case and average complexity of the algorithm are all the same.

Algorithmic Complexity


# Programming for IoT

## Summary

Programming for IoT is usually a polyglot (multiple languages) effort since the Internet-of-Things (IoT) is a system of inter-related computing devices that are provided with unique identifiers and the ability to transfer data over a network. The choice of programming-language depends on the capability and purpose of the device. IoT encompasses a variety of devices including edge devices, gateways, and cloud servers.

## Discussion

### What are the popular languages for IOT?

The most popular languages in IoT are **Java, C, C++, Python, Javascript, Node.js, Assembler, PHP, C#, Lua, R, Go, Ruby, Swift and Rust** in descending order of popularity. This is from a 2017 online survey co-sponsored by Eclipse IoT Working Group, IEEE IoT, AGILE IoT and IoT Council in which 713 individuals participated.

Other languages include **Parasail, Microsoft P, Eclipse Mita, Kotlin, Dart, MicroPython, and B#**.

### Which languages are suitable for edge devices, gateways and cloud computing?

    + **Edge Devices** - These are constrained-resource embedded systems. For very small devices (higher constraints), **Assembly and C** are the languages of choice. A better processor and more computing power on the device enables one to use **C, Python, Node.js, Java**. The focus is to minimize instruction count, and maximize execution speed and resource management.
    + **Gateways** - Gateways manage communication and do analysis of data from many devices through several different buses. More languages can be run on these devices because of their increased computing power, including **C, C++, Java, Python, and Node.js**.
    + **Cloud** - With the nearly unlimited computing capability that's available, frameworks like **Apache Hadoop** and **HiveQL** can compute and process large IOT datasets. Statistical computing and visualization can be done using languages like **R** or **Julia.**

### What are the approaches used in IoT application development?

    + **Node-Centric Programming** - Every aspect is programmed by the developer like communication between nodes, collection and analysis of sensor data, issuing commands to actuator nodes. Interaction between devices is explicitly encoded.
    + **Database approach** - Every node is considered part of a database and the developer can issue queries to sensor nodes. Focus is on collection, aggregation and sending of data to the base station.
    + **Macro programming** - Abstractions are provided to specify high-level communication.
    + **Model-driven development** - Application development complexity is reduced by increasing the level of abstraction and by describing the system using different system views. The vertical separation of concerns (SOC) reduces app development complexity by separating platform-independent model (PIM) and platform-specific model (PSM). The horizontal separation of concerns separates different aspects of a system.

### What are the features to consider when evaluating IoT programming frameworks?

The following features are relevant:

    + **Scalability** - Programming frameworks that support diverse programming patterns that are able to perform load-balancing dynamically.
    + **Concurrency** - Real-time communication between millions of devices and applications mean millions of concurrent connections. Thread locking is not efficient in such situations.
    + **Coordination** - Programming language support for explicitly (control driven) or implicitly (data driven) orchestrating the role of computing elements.
    + **Heterogeneity** - Programming framework gives guidance on how computations are mapped to the computing elements.
    + **Fault tolerance** - Applications should be able to gracefully go from online to offline state as networks partition and heal their connections.
    + **Light footprint** - In terms of runtime overhead and in terms of programming effort the framework should be light.
    + **Support for latency and sensitivity** - In geographically distributed applications, pushing all computations to the cloud is not ideal. The programming framework has to handle these requirements dynamically.

### What programming paradigms are suitable for IoT?

A programming paradigm is a way to classify the programming language based on its execution model. It should be kept in mind that a particular language can below to more than one paradigm. The following programming paradigms are suited for IoT:

    + **Functional** - Functional programming helps solve the challenges of scalability and concurrency. Its preference for immutability, function composition, avoiding side-effects, less code, etc. avoid several IoT pitfalls.
    + **Dataflow** - Dataflow programming emphasizes the movement of data. It models programs as a series of connections. Explicitly defined inputs and outputs connect operations. An operation runs as soon as all of its inputs become valid. Dataflow languages are inherently parallel and can work well in large decentralized systems.
    + **Event-Driven** - Event-driven programming (events triggered by sensors, connectivity, or time) is well suited to IoT where devices spend most of their time in power-saving modes, wake up due to some event, process data, send it out, and go back to sleep.

### Are there any specific requirements that the languages should satisfy to be used for IoT?

Developers can consider the following:

    + Languages that support Functional, Dataflow, or Event-driven programming.
    + Interpreted languages (Python) are slower than compiled ones (C/C++/Rust/Go).
    + Type safety, memory management, no null pointer exceptions. Languages such as C and C++ use explicit pointers to reference memory. Incorrect pointer calculations can lead to buffer overruns and memory access violations.
    + Availability of GPIO libraries.
    + Libraries that support IoT protocols like Advanced Message Queuing Protocol (AMQP), Constrained Application Protocol (CoAP), Extensible Messaging and Presence Protocol (XMPP), OASIS MQTT, or Very Simple Control Protocol (VSCP).

### Within languages, are there any specific variants, modules or libraries that enables IOT?

    + In Python, "mraa" and "paho-mqtt" are useful for IoT. . MicroPython can be used on constrained devices.
    + Java first appeared in the 1990s for a resource-tight embedded environment, requiring minimum computing power. Many of the Oracle Java ME Embedded APIs are targeted at the needs of embedded systems.
    + Eclipse Mita is a new programming language designed to feel like modern programming languages in the vein of TypeScript, Kotlin, and Go. It helps scale from prototypes to commercial IoT deployments by transpiling to C code.

### Could you share some best practices for IoT developers?

Choose a programming framework that promotes design reuse, implementation reuse and validation reuse in order to enhance software extensibility, flexibility and portability. To make distributed IoT microservices resilient, developers should be able to specify precise sequence of execution steps to be taken by compiled code, regardless of the order in which asynchronous event messages are processed.

### Could you list IDEs, debuggers and tools for IoT programming?

IDEs and tools for IoT are many and we list some of them below:

    + **Eclipse IOT project (Kura)** - This is a Java-based development framework for IoT applications.
    + **Arduino IDE** - This IDE includes support for the C and C++ programming languages for programmable microcontrollers. It's a complete package with many examples and pre-loaded libraries.
    + **Raspbian** - This IDE comes with many packages and examples created specifically for the Raspberry Pi boards.
    + **OpenSCADA** - This project is a part of Eclipse IOT Industry Working Group along with Eclipse SCADA (Supervisory Control and Data Acquisition). It provides several libraries, interface apps, and configuration tools.
    + **PlatformIO** - This is a cross-platform IDE that supports over 400 embedded boards, and several development platforms and frameworks.
    + **Macchina.io** - This is a toolkit for building embedded applications for IoT using POCO C++ libraries and the V8 JavaScript engine. The core is implemented in C++. JavaScript is used for application development. It enables dynamically extensible modular applications using the plug-in and services model similar to OSGi in Java.

## Milestones

Nov
2005

International Telecommunications Union publishes the 7th in its series of reports on the Internet, titled “The Internet of Things.”

Oct
2013

IBM releases version 0.2.0 of **Node Red**, a visual wiring tool for IoT. In 2016, IBM contributes Node-RED as an open source JS Foundation project.

Nov
2013

Damien George launches **MicroPython**, a software implementation of the Python 3 programming language, written in C, that's optimized to run on a microcontroller.

Oct
2015

Lars Bak and Kasper Lund of Google announce the **Fletch** project for the **Dart** programming language to optimise it for Internet of Things (IoT) devices.

2016

Microsoft open sources **P**, a language designed for asynchronous event-driven programming, which it considers vital for the cloud, artificial intelligence and IoT.

Mar
2017

Team **KinomaJS** starts **Moddable**, a new company building tools for developers to create open IoT products using standard JavaScript on low cost microcontrollers. KinomaJS is a JS application framework optimised for IoT. KinomaJS was previously supported by Marvell Semiconductor.

May
2017

Google makes **Kotlin** (JetBrains), a statically typed programming language for JVM, a first class language for Android Apps and its Android Things IoT Platform.

Mar
2018

Eclipse announces **Mita**, a language that aims to remove the entry barrier to embedded IoT development. Mita is transpiled to C code. It combines imperative programming with model-driven configuration.

Mar
2018

The Linux Foundation announces **ACRN**, an open source reference hypervisor project designed for IoT device development.

Programming for IoT


# Quadcopter

## Summary

**Quadcopter** is an unmanned aerial vehicle (UAV) or drone with four rotors, each with a motor and propeller. A quadcopter can be manually controlled or can be autonomous. It's also called *quadrotor helicopter* or *quadrotor*. It belongs to a more general class of aerial vehicles called multicopter or multirotor.

Small quadcopters are easy to build because of low cost, low inertial force and simple flight control system. Quadcopters provide stable flight performance, making them ideal for surveillance and aerial photography. Other application areas include delivery, land surveys, crop assessment, weather broadcasting, and more. Quadcopters exist in many sizes, from palm-sized ones to those that can carry passengers or heavy cargo.

Civilian use of quadcopters is subject to regulations, which are not mature in many countries. However, quadcopters are an important part of future transportation called Advanced Air Mobility (AAM).

## Discussion

### What's the working principle of a quadcopter?

The main principle behind the flight of a quadcopter is **Newton's Third Law** of motion, which states that for every action there's an equal and opposite reaction. A quadcopter's propellers push air downwards. This causes an opposite reaction called *thrust* that pushes the quadcopter upwards against gravity. Air movement comes from **Bernoulli's Principle**, with larger propeller blades and faster rotation creating more thrust.

When the propellers rotate (for example clockwise), the quadcopter will tend to rotate anti-clockwise. Rotational force is called *torque*. Helicopters solve this by using a tail rotor. Quadcopters solve this by driving two diagonal propellers clockwise and the other two anti-clockwise. Thus, torque from one pair cancels that of the other.

When each diagonal pair of propellers rotate in opposite directions, their thrusts will be in opposite directions. The quadcopter will not be able to lift up or fly. This is solved by having the blades of each diagonal pair of propellers shaped as mirror images of the other pair. Effectively, all propellers will push air downwards regardless of the direction of rotation.

### What are the components of a quadcopter?

A quadcopter has four **motors** that drive four **propellers**. Propeller blades are shaped to create maximum possible thrust. These are attached to the ends of a **frame** made of aluminium, carbon fibre or balsa wood.

Four **Electronic Speed Controllers (ESCs)** drive the motors. ESCs are typically placed along the four arms of the quadcopter. They drive the motors by supplying specific voltage and current. Typically, an ESC must be rated at 1.2-1.5 times the maximum current rating of the motor.

**Flight Controller (FC)** may be considered the "brain" of the quadcopter. It commands the ESCs autonomously or in response to remote commands. It may process inputs from many on-board sensors. For remote control, a quadcopter is equipped with a **wireless transceiver**.

The power to drive the motors and all electronics comes from a **battery** via a **Power Distribution Board (PDB)**. Flight time can be estimated from battery capacity and maximum current drawn by all motors.

Other parts include jumper cables, bullet connectors, LEDs and a buzzer. Quadcopters may have a GPS receiver, a camera and many other sensors as required by specific applications.

### How does a quadcopter achieve different movements?

A quadcopter has four different movements:

    + **Vertical**: Quadcopter can move up against gravity or come down in a controlled manner. These movements are called *throttle*.
    + **Rotational**: Clockwise or anti-clockwise turns about the vertical axis is called *yaw*.
    + **Front-to-back Lateral**: Tilting along the lateral axis, called *pitch*, causes the quadcopter's nose to dip or rise. This in turn enables forward or backward motion.
    + **Left-to-right Lateral**: Tilting along this axis, called *roll*, enables movement towards left or right.The above imply that a quadcopter has **six degrees of freedom**: linear movements in along x, y and z directions; rotational movements called yaw, pitch and roll.

The movements are achieved by controlling the rotational speeds of the four motors. This is shown in the figure. For example, with pitch, when back propellers rotate faster than the front ones, the nose dips down and the quadcopter moves forward. With yaw, when two propellers rotate clockwise faster than the other two, the quadcopter turns anti-clockwise.

### What are the different types of quadcopters?

The different types of quadcopters based on their shape are as follows:

    + **X Quadcopter**: A popular design, it's used for aerial photography, videography, First Person View (FPV) racing and acrobatic stunt flying. Frame variations include true X, square, hybrid X and stretched X.
    + **X-Stretched Quadcopter**: Also called V configuration, it's elongated design provides more stability on the pitch axis.
    + **H Quadcopter**: Has a H-shaped frame.
    + **+ Quadcopter**: This tracks well when going straight. To go straight or sideways, one motor is driven faster than the other on the opposite arm. Propellers are in an aerodynamically efficient position. It's used in acrobatic flying and FPV stunt flying.
    + **Y4 Quadcopter**: Frame has three arms like a tricopter but rear arm has two motors coaxially mounted for better yaw control. While a tricopter use servo motors, Y4 uses BLDC motors. Hence Y4 has more lifting power and is more durable.
    + **V-tail or A-Tail Quadcopter**: Similar to Y4 but with rear motors mounted at an angle in a V or A shape. It has more yaw control because it uses thrust to turn rather than counter torque.

### What are the applications of quadcopters?

We note the following applications of quadcopters:

    + **Aerial Photography**: Used in filming action sequences and sci-fi scenarios, which makes cinematography a lot simpler.
    + **Shipping & Delivery**: Drone delivery is supported by major corporations including Amazon, UPS, and DHL. Used to transport tiny parcels, groceries, medications, and other items across short distances.
    + **Geographic Mapping**: In difficult-to-reach places like coasts, mountaintops, and islands, quadcopters can collect very high-resolution data and download pictures.
    + **Disaster Management**: After a natural or man-made disaster, quadcopters can quickly negotiate debris and gather information in search of injured patients.
    + **Precision Agriculture**: Quadcopters' infrared sensors may be calibrated to assess the health of crops, allowing farmers to react and enhance agricultural conditions locally with fertilizers or pesticides.
    + **Weather Forecasting**: Quadcopters are being developed to track hazardous and unpredictably changing weather. They can be cost-effectively deployed in storms and tornadoes.
    + **Wildlife Monitoring**: Elephants, rhinos, and big cats were photographed using quadcopters. Drones can function at night thanks to their infrared cameras and sensors.
    + **Entertainment**: In fights and cage matches, quadcopters are utilized to collect recordings and images.

### What are the design considerations for a quadcopter?

The essence of quadcopter design is to generate enough thrust to carry the payload. Payload depends on the application. Thrust is provided by good propeller design. Their blades are shaped to minimize drag and maximize thrust. Each rotor produces thrust and torque about its center of rotation. There's gravity. There's also drag in the opposite direction to the flight. These are main forces to consider during design.

Good flight stability and control are essential. Safety and durability are important.

Brushless DC (BLDC) motors are commonly used for quadcopters. These consist of a permanent magnet which rotates around a fixed armature. Compared to brushed DC motors, we get more torque per weight, reduced noise, increased reliability, longer lifetime and increased efficiency.

For remote/radio control (RC), we need to select the correct wireless technology. This determines the range of communication and control.

For power, Lithium Polymer (LiPO) and Lithium Polymer High Voltage (LiHV) batteries are commonly used for quadcopters. They should be shaped for neat mounting within the frame. Battery capacity determines flight time. Higher capacity generally means heavier batteries and hence there's a trade-off.

### How to calculate the thrust of a quadcopter?

**Payload** is weight in addition to the "bare weight" of the quadcopter consisting of its essential components. For a typical drone, the average payload is 0.3-2 kg. A cargo drone can carry 20-200 kg. In either case, thrust should be sufficient to deliver the payload.

To get the quadcopter airborne, the quadcopter needs a certain amount of thrust. Thrust is the force normal to the plane of the propeller. Thrust from a single motor \(r\) for \(r \in [1..4]\) is given by \(T = ρA{V\_r}^2\), where \(ρ\) = real-time air density (kg/m3), \(A\) = cross-sectional area of the propellers, \(V\_r\) = instantaneous peripheral velocity of rotors (m/s).

Total thrust is includes thrust from all four rotors. In the simple case of upwards throttle (vertical take-off), this should exceed gravitational force.

### What are range and endurance of a quadcopter?

**Endurance** is defined as the total time that a quadcopter stays in the air on a full tank of fuel. For a quadcopter, the battery is the fuel. 2 hours, 31 minutes, and 30 seconds is the longest endurance a quadcopter achieved. This record was created by Ferdinand Kickinger of Germany in 2016. It used low discharge rate and high-capacity lithium-ion batteries. He stripped the airframe of non-essential weight to reduce power draw and extend endurance. To calculate the endurance in minutes, take the battery's capacity in amp-hours. Divide it into the average current draw and then multiply it by 60.

**Range** is defined as the distance travelled by an aerial vehicle on a full tank of fuel. A high-end consumer drone can have a drone of about 2.5-4.5 miles (4-8 km). Mid-level consumer drones have a range of about 0.25-1.5 miles (0.4-3 km). **DJI Mavic 2 Pro** is the longest range drone.

### Which type of control system is used in a quadcopter?

Quadcopters require flight control software and hardware elements. These allow it to be controlled remotely. It's either done by a pilot or autonomously by an onboard computer. Flight dynamics are highly variable and non-linear. Maintaining attitude and stability may require continuous computation and readjustment of aircraft's flight systems.

Elements on the ground form a **Ground Control System (GCS)** and use a modem and datalink for communication. Quadcopter collects information from an *Air Data System* (static and dynamic pressure), **GNSS** receiver or Attitude and Heading Reference System/AHRS (roll, pitch, and yaw data). A **Flight Control Computer (FCC)** uses this data to guide the quadcopter to its next waypoint. The FCC also operates payloads and communicates with the GCS.

MATLAB/Simulink are the tools to carry out the simulation, how a quadcopter operates. The three controllers are Feedback Linearization with PD Controller (FBL+PD), Feedback Linearization with LQR (FBL+LQR), and PID Controller are implemented as separate Simulink models and simulated.

## Milestones

1907

The first quadcopter is created by brothers Jacques and Louis Bréguet.

1917

The first pilotless aircraft is developed after the outbreak of World War I. It's named as Ruston Proctor Aerial Target. It's a radio-controlled pilotless airplane, based on RC technology from the inventor Nikola Tesla. The goal of the Aerial Target is to act as a flying bomb, which can pilot into enemies.

1920

Quadcopters are developed by engineers to solve the problems that helicopter pilots had with making vertical flights. The first one is named with the Omnichen 2 by Etienne Omnichen.

1935

The British develop "Queen Bee", a radio-controlled target drone for radio-controlled unmanned aircrafts.

1937

The U.S. Navy starts experimenting with radio-controlled aircaft. This results in the development of Curtiss N2C-2 Drone.

1958

Curtis Wright VZ7 is a VTOL quadrotor helicopter designed by the Curtiss-Wright company for the US Army. It is a "Flying Jeep".

1981

The Israeli IAI Scout drone is operated in combat missions by the South African Defence Force against Angola during Operation Protea.

1999

The Bell Boeing Quad Tiltrotor is a fixed quadcopter combining with the tilt rotor concept for a C-130 size military transport.

2013

Amazon proposes the use of commercial drone technology in delivery system.

2018

Airbus is developing a battery-powered quadcopter to act as an urban air taxi, initially without a pilot, but potentially autonomous in future.

2021

A Cessna 172 of Canadian Flyers International Inc. registered as C-GKWL collides with a drone operated by the York Regional Police while on approach to Buttonville Municipal Airport.

Quadcopter


# Duck Typing

## Summary

Duck Typing is a way of programming in which an object passed into a function or method supports all method signatures and attributes expected of that object at run time. The object's type itself is not important. Rather, the object should support all methods/attributes called on it. For this reason, duck typing is sometimes seen as "a way of thinking rather than a type system".

In duck typing, we don't declare the argument types in function prototypes or methods. This implies that compilers can't do type checking. What really matters is if the object has the particular methods/attributes at run time. Duck typing is therefore often supported by dynamic languages. However, some static languages are beginning to "mimic" it via structural typing.

## Discussion

### What's the origin of the phrase "duck typing"?

The phrase originates from an old saying,

> If it walks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

What does this mean? Even a non-duck entity that behaves like a duck can be considered a duck because emphasis is on behaviour. By analogy, for computing languages, the type of an object is not important so long as it behaves as expected. This behaviour is defined by the object's methods/properties/attributes while expectations are set by those who invoke the methods/properties/attributes.

For example, a `Book` class is expected to have an attribute `numPages` and a method `getPage(number)`, where `number` is an integer. Let's say a function `searchPhrase(book, phrase)` is meant to search for a given phrase in a book. This function calls `book.getPage()` for all `book.numPages` of the book. A `Newspaper` is obviously not a book but with duck typing this doesn't matter. If `Newspaper` has implemented `numPages` and `getPage(number)`, it can be passed into `searchPhrase(book, phrase)` as if it's a book. Though it's not a book, it's sufficient that it behaves like one within the context of `searchPhrase`.

### What's the advantage of having duck typing?

It's been said that dynamic typing in general cuts down development time. There's less boilerplate code. Code is easier to hack. With static typing, compile-time checks slow down the development process. Static typing could be used at a later point to get better performance. Others recommend duck typing only for prototyping and never for production.

While it's true that compile-time checks are useful to catch potential problems early, duck typing enforces following coding conventions, documentation and test-driven methodologies.

Since duck typing isn't exactly a type system, it gives programmers flexibility. In Python, for example, common things are simpler to code. While static typed languages use interfaces, such interfaces may involve excessive refactoring of client code.

### Isn't duck typing more like late binding than a type system?

It can be argued that duck typing is not really a type system in the sense of run-time or compile-time type systems. Type systems typically prescribe rules for defining types, for deducing types from expressions, for determining legality of expressions or storage based on types, and so on. Duck typing doesn't do any of these. Joe Esposito has commented that, "Duck typing is based around conventions, written in the docs instead of code. This allows the language to defer the type system".

Late binding is when an object or method is bound to a name at run time. Duck typing looks similar to late binding except for a subtle difference that it's based on behaviour rather than declared type.

One way to understand this is to recognize that C# (a static typed language) allows for method overloading where methods differ only by their arguments types. This is not possible in Python (dynamic typed language) but duck typing fills in since run-time behaviour is emphasized rather than the type of the object passed to the method. Also, because it's interpreted and does late binding, duck typing is easy to do.

### Can you compare duck typing with structural typing?

Joe Esposito makes a distinction between static duck typing and dynamic duck typing. With the latter, compile-time checks are skipped and this is what we mean when we refer to duck typing. Static duck typing is similar to structural typing that some static typed languages (Scala, F#) support. Structural typing allows for some compile-time checks, not on types but on the supported methods/attributes. Structural typing may be seen as a compiler-enforced subset of duck typing. However, we should also note that structural typing is essentially static typing while duck typing is dynamic.

Just as duck typing is for dynamic typed languages, so is structural typing for static typed languages. Erlang's Behaviours can be seen as static duck typing, that is, with compile-time checking.

Among the languages that support structural typing are Scala, OCaml, Go, Elm and PureScript.

### Can we say that duck typing is another name for implied interfaces?

Static typed languages use explicit interfaces. In C#, we use interfaces: any type that implements an interface can be passed into a method/function that accepts that interface. This was how flexibilty was built into static typed languages but this required upfront interface declaration.

Python doesn't have explicit interfaces: interfaces are implied. For Python, this is not different from duck typing. However, implied interfaces are supported by C++ (for example) in its STL library. Given that C++ does compile-time checks, we can't say in general that implied interfaces is the same as duck typing. Note that when interfaces are implied, there's no formal interface definition, and at best, it may be a shared documentation.

### What is goose typing?

In Python, duck typing is an implied or informal interface. This is sometimes called a **protocol**. For instance, any class or object that implements `__len__` and `__getitem__` is said to conform to the sequence protocol. Therefore, it can be accessed as if it's a sequence. The sequence can be indexed or sliced.

With **goose typing**, the interface or protocol is made explicit. This is done using an Abstract Base Class (ABC). In the call `isinstance(obj, cls)`, the second argument must be an ABC. The ABC class itself can't be instantiated but another class provides an implementation of the expected interfaces. However, this other class is simply registered (via a decorator) as conforming to the ABC's interface; that is, it's not explicitly derived from the ABC. For this reason, we call goose typing as "virtual subclassing of Python ABCs."

Among the languages supporting goose typing are Python, TypeScript and Go.

### What languages support duck typing?

    + Python and Ruby support duck typing, both of which are dynamic typed languages. In general, dynamic typed languages such as JavaScript and TypeScript support duck typing. While C# doesn't, some parts of .NET use duck typing.
    + In Ruby, for example, `StringIO` doesn't have `IO` in its class hierarchy. Yet, it behaves like an `IO` due to duck typing. In fact, we may say that `StringIO` and `IO` share a common *protocol*, which is exactly how the duck typing concept is seen in OOP languages such as Smalltalk and ObjectiveC. This can be said of Python as well. For example, an iterable is one that supports the iterator protocol, which is nothing more than implementing necessary methods for an object.
    + In PHP, PHP's dynamic typing system and object-oriented features enable duck typing to a certain extent.
    + It's interesting to note that duck typing has inspired the creation of a [new language called Duck].(http://ducklang.org/).

### How is duck typing related to LBYL and EAFP?

LBYL expands to *Look Before You Leap* while EAFP expands to *Easy to Ask for Forgiveness than Permission*. Both these concepts are typically discussed in Python (and possibly other languages as well). Python prefers the EAFP approach, which is the opposite of LBYL.

With LBYL, we typically check if an object is of a compatible type before performing operations on it, such as calling its method. Python has a built-in function `isinstance()` to do this. With EAFP, we simply call the object's method, expecting that method to be present. If it's not, an exception will be thrown by the Python interpreter, which the code is expected to handle.

With duck typing, validating an object's type is not Pythonic. Thus, duck typing plays well with Python's preference for EAFP though the use of `hasattr()` can be used instead.

## Milestones

1946

Emil Mazey states at a meeting that "I can’t prove you are a Communist. But when I see a bird that quacks like a duck, walks like a duck, has feathers and webbed feet and associates with ducks—I’m certainly going to assume that he is a duck." The phrase might have originated earlier from James Whitcomb Riley (1849–1916).

1980

Languages of the 1980s such as Smalltalk-80 and Objective-C have the concept of **protocols**, where the interface is more relevant than the class. In 1992, with reference to Smalltalk-80, William R. Cook of Apple Computer states that *interface hierarchy* is more useful that *inheritance hierarchy*.

Jul
2000

What may be the one of the earliest use of the term "duck typing" among programmers, Alex Martelli discusses it on the `comp.lang.python` Google Group as an alternative to method overloading of C++. He states that duck typing is closer in spirit to polymorphism of C++. He clarifies,

> You don't really care for IS-A -- you really only care for BEHAVES-LIKE-A-(in-this-specific-context), so, if you do test, this behaviour is what you should be testing for.

2015

Alex Martelli coins the term **goose typing** in the first edition of his book titled *Fluent Python*.

Mar
2017

PEP 544 proposes to introduce structural subtyping to Python. This is subsequently accepted into Python 3.8 (October 2019).

Duck Typing