# Question Similarity

## Summary

In some applications, there's a need to compare two questions and determine how similar they are. In Information Retrieval (IR), it may be necessary to compare the incoming query against questions stored in the system database. This helps the IR system give a suitable response.

Question similarity involves a few basic aspects: pre-processing to reduce words and phrases to a form suited to the task, representing questions in efficient vector forms, defining features and selecting models to bring out the similarity, and defining a similarity measure to objectively compare the questions.  The task benefits from lexical, syntactic and semantic features.

Question similarity is part of a more general NLP task called **Semantic Textual Similarity (STS)**. STS involves comparing two sentences, two paragraphs or even two documents. Question similarity is also closely related to the task of question answering.

## Discussion

### Could you explain question similarity with some examples?

Consider the pair "How old are you?" and "What is your age?" These are semantically equivalent. Consider the pair "How are you?" and "How old are you?" These are semantically different even when they differ by only one word. Thus, question similarity is usually about the meaning or intent of the questions, not about how many common words they share. A useful definition is,

> Two questions are semantically equivalent if they can be adequately answered by the exact same answer.

Determining semantic equivalent is a challenging problem. Consider the pair "Which Star Wars episode should I watch?" and "I have so much of a life that I'm at home watching Star Wars." Topic "Star Wars" can mislead a model into thinking these two are similar.

Another example is the pair "How to declare an array in Python?" and "How to create a fix size list in python?" In Python terminology, arrays are often referred to as lists. A model that lacks this knowledge, when asked the first question, might instead match "How do I declare and initialize an array in Java?"

### What are some applications of question similarity?

On **knowledge-sharing platforms** such as Quora and StackExchange, people post questions. Question similarity detects similar questions, possibly already answered. Moderators may review this information to mark duplicates. These sites also list related questions as a useful feature for readers.

In **chatbot applications** or with **voice assistants**, a user may ask a question but that exact question may be unavailable in the system. Question similarity identifies questions that're closest in terms of semantics or user intent. The user is then presented these interpretations or if possible directly give an answer.

In general, any **Information Retrieval (IR)** system can benefit from question similarity.  For example, when a user asks Google "how to bake a cake", Google will provide search results but also include a section titled *People also ask*. This section lists a handful of related questions such as "How do you bake a cake in 10 steps?" and "What are the ingredients used in baking cake?"

Where **exam papers** use automatically generated questions or pick questions from a question bank, question similarity ensures that similar questions are not picked for the same paper.

### How should I prepare the data for question similarity?

Like other NLP tasks, question similarity involves data collection, exploratory data analysis, pre-processing, feature extraction, model training and evaluation. The idea of pre-processing data is to make it suitable for a given task and also remove unclean or wrong data. Other steps of the process might also influence the selection of pre-processing steps.

For example, if Jaccard similarity measure is used, we may do case normalization, tokenization, stopwords removal, punctuation removal, and lemmatization. Where there are many stopwords, Soft Cosine similarity will be higher if they're not removed. However, removing stopwords gives poorer results with BM25 (used in IR applications) and Translation-Based Language Model.

On short form Twitter data, Dey et al. suggested topic phrase removal, slang normalization (eg. aaf vs as a friend), named entity boundary correction (eg. Colorado Ravens and Ravens are probably the same concept), named entity tag cleaning, and synonym/hypernym replacement using WordNet (eg. quick vs fast).

### How can I represent questions for the task of question similarity?

Before questions can be compared, we need to represent them in a form that algorithms can efficiently process and compare. This is done after data pre-processing. The simplest techniques are TF-IDF, bag-of-words model, and n-gram model.

Since the mid-2010s, word embeddings are popular. Word embeddings are multi-dimensional vectors containing real values. Words that are similar tend to occur close to one another in the vector space. Such embeddings are learned by training on huge amounts of real-world text. GloVe and word2vec are well-known word embeddings. **Contextualized word embeddings** such as BERT and ELMo capture context as well.

**Sentence embeddings** can be learned from a weighted sum of word embeddings. By using Principal Component Analysis (PCA) and removing the projections of the average vectors on their first principal components, embeddings have shown to perform better on text similarity tasks.

Universal Sentence Encoder is another approach to sentence or question embeddings. Cer et al. have shown that transfer learning via sentence embeddings perform better than via word embeddings. Sentence-BERT used BERT to learn sentence embeddings.

### How should I select features for text similarity?

Features can be lexical, either character-level or word-level n-grams. Syntactic features include words along with their POS tags, named entities and verb similarity. Semantic features include words and phrases that semantically overlap, with help from WordNet.

Some basic features could include number of words or characters, uppercase words, last words, first words, adjectives, nouns, ratio of the common words to words, fuzz ratio (from Python's FuzzyWuzzy package), etc.  Exploratory data analysis may show that only some of these are actually useful. Such features are commonly employed with traditional ML models such as SVM.

With neural networks, extensive feature engineering is not required. For example, when using Sentence-BERT, the sentence embeddings are used as features. In fact, these embeddings and neural networks are meant to capture context and semantics effectively. However, enabling neural networks to use syntactic information is an ongoing research.

Where embeddings such as GloVe are used, correcting spelling errors may give poorer results since the embeddings themselves were trained without correcting for spelling.

### When are two questions said to be semantically equivalent?

Various similarity measures, also called **distance measures**, exist to compare two vectors, each representing one question: cosine distance, Euclidean distance, Word mover's distance, Minkowski distance, Jaccard distance, and many more. We note the calculations for cosine, Euclidean and weighted Manhattan:

$$d\_{cos}(h\_1, h\_2) =\frac{h\_1 . h\_2}{||h\_1|| ||h\_2||}$$

$$d\_{euc}(h\_1, h\_2) = ||h\_1 − h\_2||$$

$$d\_{w−man}(h\_1, h\_2) = w · |h\_1 − h\_2|$$

More generally, in STS, many methodologies are available to calculate semantic similarity:

    + **Knowledge-based**: Such as the use of WordNet and distance measures associated with WordNet graph nodes.
    + **Statistical or Corpus-based**: Techniques include Latent Semantic Analysis (LSA), Pointwise Mutual Information-Information Retrieval (PMI-IR), Normalized Google Distance (NGD).
    + **Character-based**: Longest Common Substring (LCS), Damerau-Levenshtein, Jaro, Needleman-Wunsch, Smith-Waterman, n-gram.
    + **Term-based**: Manhattan, Euclidean, Jaccard, Cosine similarity, Soft Cosine Similarity, Sorensen-Dice index, Simple Matching Coefficient (SMC).While different distance measures are available, it's hard to know which of these is ideal for measuring semantic similarity. Homma et al. addressed this with a 2-layer neural network that learned the correct distance function.

### Which datasets are useful for benchmarking or training Question Similarity models?

We note a few of these:

    + **CQADupStack**: Questions and answers from 12 subforums of StackExchange, each subforum with at least 10,000 threads and 500 duplicate questions. Another StackExchange dataset covering 19 domains is also available. StackExchange data dump is also useful. This dataset is part of SemEval-2017 Task 3.
    + **glue/qqp**: General Language Understanding Evaluation (GLUE) is a benchmark for many NLP tasks. One of them involves Quora Question Pairs2 dataset of about 364K training, 40K validation and 391K testing data samples. GLUE's *glue/stsb* dataset might also be useful. The first Quora dataset is also available.
    + **MQS**: Medical Question Similarity dataset related to COVID-19. Contains 3,048 labelled medical questions pairs.
    + **Microsoft Research Paraphrase Corpus**: 5,800 pairs of sentences from web news sources including annotations to indicate semantic similarity.Since question answering (QA) is a closely related task, QA datasets can also be used. 719 QA pairs from the web apps domain of StackExchange is one such dataset.  Google's Natural Questions has 307,373 training examples, 7,830 development examples, and 7,842 test examples.

Other datasets include SQuAD,  SuperGLUE, , TriviaQA, TREC-QA, WikiQA, YahooCQA, and SemEval CQA (2015, 2016, 2017).

### What are some classical machine learning techniques used in question similarity?

Question similarity can be seen as a classification problem. Many classical ML techniques have been applied to this problem including Support Vector Machine (SVM), Decision Tree, Logistic Regression, Extreme Gradient Boosting, Random Forests, and Adaptive Boosting.  Dey et al. (2016) showed good results using SVMs with hand-picked features and pre-processed data.

Similarity is determined either statistically (TF-IDF) or semantically (meaning and context). Semantic similarity requires knowledge representation for which WordNet can provide shallow lexical semantics. Other NLP tasks such as word sense disambiguation and relation extraction can also be useful.

### What are some neural network approaches used in question similarity?

Till the mid-2010s, neural networks performed worse than SVMs due to limited training on noisy datasets. Subsequently, neutral networks started to perform better. One approach is to use "Siamese" neural network architecture based on CNN, RNN or GRU. Each question is encoded individually using the same network and then compared. For better interaction between the questions, bilateral multi-perspective matching LSTM model is an option.

Where sufficient labelled data is not available, domain adaptation is possible. Shah et al. did this for some StackExchange forums using labelled data from other forums. Joty et al. used Adversarial Neural Networks across languages English and Arabic. Hazem et al. built a multi-domain framework using a StackExchange dataset covering 19 domains.

Transformer models such as BERT have been used for question similarity. In general, model training benefits from not just question-question pairs but also question-answer pairs and other NLP tasks.

Papers With Code curates the latest developments in Question Similarity.

### What tools or libraries are available for question similarity?

In Python, NumPy, Pandas and Scikit-Learn are commonly used libraries for machine learning. NLTK, TextBlob, spaCy, Gensim, Pattern, and Stanford CoreNLP (in Java with Python wrappers) are more specific to NLP. A GitHub repo has a curated list of Python NLP libraries.

For neural network models, PyTorch or TensorFlow libraries are commonly used. Hugging Face is an active community and library that bundles PyTorch, TensorFlow, and models based on them. Embeddings, BERT and its many variants are available within Hugging Face.

For semantic similarity, Java libraries such as Semantic Measures Library, SEMILAR and SEMILAT, and DISCO Builder/API could be useful. There's a Perl module for WordNet-based similarity. UMLS is another Perl module specific to medical domain. RxNLP has an API for text similarity.

## Milestones

1993

To compare the similarity of two signatures, Bromley et al. propose using two identical time delay neural networks that share one output that captures the similarity of the two inputs. They call this the **Siamese Neural Network**. This technique of combining two neural networks is later used specifically for the task of question similarity.

Jul
2015

Bogdanova et al. compare Support Vector Machine (SVM) with their proposed approach of **Convolutional Neural Network (CNN)**. They train their CNN with positive and negative pairs of semantically equivalent questions. The dataset is *Ask Ubuntu* forum on StackExchange. Their definition of a question consists of both title and body. CNN captures local features around each word, which are then combined to produce a fixed-size vector representation for each question. Their results show that CNN outperforms SVM. Moreover, results are better when embeddings are learned from in-domain data.

Sep
2015

Sanborn and Skryzalin try out both **Recurrent Neural Network (RNN)** and **Recursive Neural Network** within a Siamese architecture. They use SemEval-2015 Task 2 as the dataset. For baselines, they use cosine similarity between bag-of-words vectors, cosine similarity between GloVe-based sentence vectors, and Jaccard similarity between sets of words. They find that RNN with 100-dimensional word vectors and 20% dropout gives best performance, although not the state of the art.

2016

Dey et al. apply **SVM** to short-text Twitter data based on a SemEval 2015 dataset. They obtain an F1-score of 0.741 for semantic similarity.

Feb
2017

Bonadiman et al. recognize that question-question similarity relates to question-comment similarity and new question-comment similarity. They therefore train a deep neural network jointly on these three tasks of SemEval 2016 Task 3, a technique that's called **Multi-Task Learning (MTL)**. They use CNN with max pooling as the sentence model. This is followed by a Multi-Layer Perceptron (MLP) with sigmoid output layer. For feature embedding, they use word overlap.

Jul
2017

Conneau et al. show that NLP tasks can benefit from **pre-trained sentence level embeddings** rather than just word embeddings as in GloVe or word2vec. They consider seven different encoder architectures and find best performance with BiLSTM with max pooling with embeddings trained in a supervised manner on Natural Language Inference datasets. They obtain better results than earlier sentence embeddings obtained via unsupervised methods such as SkipThought (2015) and FastSent (2016).

2018

Cer et al. propose the **Universal Sentence Encoder (USE)** via transformer-based encoding with multi-task learning. They also propose a simpler Deep Averaging Network (DAN). The former gives better performance at the cost of higher compute. They also find that transfer learning via sentence embeddings performs better than via word embeddings. For sentence embeddings, USE is publicly available via TensorFlow Hub. It can be fine tuned for specific tasks.

Aug
2019

Reimers and Gurevych propose **Sentence-BERT (SBERT)** as a modification of pretrained BERT to train sentence embeddings. They use a Siamese network architecture and fine tune SBERT using Natural Language Inference (NLI) data. On seven Semantic Textual Similarity (STS) tasks and also SentEval, they achieve better performance than Universal Sentence Encoder.

Aug
2020

In medical domain, specifically COVID-19 FAQs, McCreery et al. show that double fine-tuning during pretraining yields good performance. They try four different pretraining tasks: question-question pairs, question-answer pairs, answer-answer pairs and question-category pairs. They observe best results when the network is pretrained with medical question-answer pairs and then fine tuned on medical question-question pairs. On question similarity task, the model gives 84.5% accuracy. Even with much smaller training set, it achieves 80% accuracy.

Question Similarity


# 5G New Radio

## Summary

5G defines a new radio interface called **5G New Radio (NR)**. Rather than being something new, it should be seen as an evolution of LTE technology. In fact, the term **Next Generation Radio Access Network (NG-RAN)** is commonly used and it covers both 5G NR and LTE/E-UTRA radio access.

5G attempts to address many new use cases not possible in earlier generations. 5G NR plays an important role in fulfilling these new cases. All layers of the 5G NR protocol stack are enhanced but perhaps most changes are within PHY.

While there are many 3GPP specifications that define 5G NR, beginners can start with TS 38.300 and TS 38.401 that give high-level descriptions.

## Discussion

### Which are the 5G NR technical enablers for achieving 5G use cases?

The three broad 5G use cases supported by 5G NR are Enhanced Mobile Broadband (eMBB), Massive Machine-Type Communications (mMTC) and Ultra-Reliable Low-Latency Communications (URLLC).

5G NR spans spectra from sub-GHz to mmWave bands. This enables deployments of macro cells to picocells. Both licensed and unlicensed bands are allowed. Wide bandwidths at mmWave bands enable eMBB. Multiple bands can be combined to offer higher data rates and boost capacity.

For URLLC, reliability is improved via multi-antenna transmission, multiple carriers and packet duplication. Latency is reduced by mini-slot transmission, grant-free uplink access, and eMBB resource pre-emption. Control and reference signals appear at the start of a slot.

To cater to wide range of bands and deployments, OFDM Sub-Carrier Spacing (SCS) is flexible while maintaining time-domain alignment. As SCS goes up, the slot duration shrinks. For example, wider SCS might suit URLLC and a narrower SCS for MBB/mMTC.

There are many ways to aggregate slots with guard periods between uplink and downlink. Thus, we can have short transmissions for URLLC and longer ones for eMBB.

### What are the key design principles behind 5G NR?

We identify the following:

    + **Flexibility**: OFDM sub-carrier spacing and symbol duration are flexible such that services with different requirements of bandwidth and latency can coexist. The design allows for wide range of deployments from outdoor macrocells to indoor picocells. Wide spectrum from sub-GHz to mmWave bands are supported. Symbol allocation within a slot is flexible.
    + **Forward Compatibility**: Design is such that it's easy to introduce new use cases in future. This is enabled by self-contained slots and beams, that is, they can be decoded without dependency on other slots or beams. This also relates to design flexibility: details are configured at runtime rather than fixed in specifications.
    + **Ultra-Lean Design**: LTE transmits regular reference signals, synchronization signals and system broadcasts. 5G is designed to minimize such "always on" transmissions. This improves network energy efficiency and reduces interference in high traffic load conditions. This also assists forward compatibility. Another related approach is device-centric mobility. A mobile sends out periodic reference signals that a base station measures, rather than monitor reference signals from many nearby cells.

### What's the architecture of 5G NR?

**Next Generation Radio Access Network (NG-RAN)** consists of gNB and ng-eNB. *gNB* serves a 5G UE over 5G New Radio (NR), a new air interface developed for 5G. gNB connects to 5G Core, though some can connect to 4G EPC as well. *ng-eNB* connects to 5G Core but serves a 5G UE over E-UTRA radio.

gNB and ng-eNB are interconnected via **Xn interface**. In the user plane, Xn uses GTP-U over UDP/IP. In the control plane, Xn uses XnAP over SCTP. Thus, signalling packets have guaranteed delivery whereas user plane packets don't. Xn-U does data forwarding and flow control. Xn-C facilitates UE mobility management and dual connectivity.

5G NR nodes connect to 5G Core via **NG interface**. Like Xn, NG-U uses GTP-U over UDP/IP, and NG-C uses NG-AP over SCTP. More specifically, NG-U connects to User Plane Function (UPF) and NG-C connects to Access and Mobility Management Function (AMF) in the core. NG-C and NG-U are also called N2 and N3 interfaces respectively.

### What's the 5G NR protocol stack?

The air interface between the UE and gNB is usually described in two planes:

    + **User Plane (UP)**: Carries user data. Consists of PHY, MAC, RLC, PDCP and SDAP. SDAP is new in 5G (doesn't exist in LTE).
    + **Control Plane (CP)**: Carries signalling such as sending system broadcasts, paging UEs, establishing connection, handover, measurement reporting, NAS messaging, etc. Consists of PHY, MAC, RLC, PDCP and RRC. In addition, the UE has Non-Access Stratum (NAS) layer that terminates on the network side in the 5G Core.PHY is commonly called Layer 1. MAC/RLC/PDCP/SDAP are called sub-layers of Layer 2.

When transmitting, SDAP maps QoS flows to data radio bearers. PDCP maps (data or signalling) radio bearers to RLC channels. RLC maps RLC channels to logical channels. MAC does the scheduling and maps logical channels to transport channels. PHY maps transport channels to physical channels. When receiving, the layers map in the opposite direction.

### What are the key principles or features of NG-RAN?

An essential principle of NG-RAN is the logical separation of signalling and data transport networks. NG-RAN and 5GC functions are separated from transport functions, even if they happen to reside in the same equipment. Mobility of an RRC connection is fully controlled by NG-RAN.

NG-RAN nodes can be further disaggregated into Radio Unit (RU), Distributed Unit (DU) and Centralized Unit (CU). By locating these parts at cell site or at cloud edge different deployment options are possible. Operators can decide based on the use case, CAPEX and OPEX. 3GPP has standardized the **F1 interface** between gNB-DU and gNB-CU, the **W1 interface** between ng-eNB-DU and ng-eNB-CU, and the **E1 interface** between gNB-CU-CP and gNB-CU-UP. Other industry bodies might standardize the RU-DU interface.

NG-RAN has many functions: Radio Resource Management (RRM); routing of packets to AMF or UPF; encryption and integrity protection of data; IP and Ethernet header compression; connection setup and release; scheduling and transmission of system broadcasts and paging; QoS flow management; radio access network sharing; dual connectivity; support of network slicing; and more.

### What role does each protocol layer play in NG-RAN?

SDAP maps a QoS flow to a data radio bearer. It's not relevant to the control plane.

PDCP does header compression/decompression using ROHC or EHC protocols. It encrypts/decrypts packets. It adds/checks integrity of packets. It supports both in-order and out-of-order delivery. It discards duplicates.

RLC supports three transmission modes: Transparent Mode (TM), Unacknowledged Mode (UM), and Acknowledged Mode (AM). An RLC entity's functions are based on its mode. In AM, there's error correction through ARQ, duplicate detection and error detection. In AM and UM, SDUs can be segmented and there's sequence numbering; plus, SDUs can be discarded.

MAC maps SDUs from one or more logical channels to transport channels. It does padding. It does error correction through HARQ, one per Component Carrier (CC). It prioritizes transmission towards UEs (downlink only), across logical channels of a UE, and UE resources.

Among the PHY procedures are link adaption, power control, cell search, random access, and HARQ. PHY processing includes code block segmentation and CRC attachment, LDPC coding, rate matching, scrambling, modulation, layer mapping, and mapping to assigned resources and antenna ports.

### How do 5G NR L2 sub-layers differ from LTE's E-UTRA sub-layers?

**SDAP** is new in 5G NR. 5G Core can configure different QoS requirements for different IP flow of a PDU session. SDAP maps IP flows to data radio bearers that can provide the necessary QoS.

**PDCP** does packet duplication when sending, and reordering and duplicate detection when receiving. This is done to satisfy URLLC use case. PDCP also includes integrity protection for user plane data.

Unlike LTE, **RLC** doesn't concatenate RLC SDUs since something similar is done at MAC. Likewise, 5G RLC doesn't do reordering since the reordering done at PDCP is deemed adequate.

**MAC** carries signalling for beam management done at PHY.

Whereas LTE RRC has only two states (*Idle* and *Connected*), 5G **RRC** includes *Inactive* as an additional state. This enables devices to save power and quickly reconnect with minimal signalling. UE can also request specific system information instead of NG-RAN periodically sending the same.

### What are the enhancements to 5G NR in Release 16?

We summarize the main enhancements:

    + **MIMO**: MU-MIMO support of higher rank, multiple transmission and reception points (multi-TRP), better multi-beam management (useful in mmWave band), and extended uplink coverage (full power UL transmission).
    + **URLLC**: Using Coordinated Multi-Point (CoMP) reliability is improved. CoMP uses multi-TRP. Improved HARQ, flexible scheduling, inter-device service multiplexing and intra-device channel prioritization are other changes.
    + **Time-Sensitive Networking (TSN)**: For industrial automation, TSN provides time-deterministic delivery of data packets.
    + **Power Save**: Wakeup Signal (WUS) lets a device skip the next low-power DRX (discontinuous reception) monitoring period.
    + **Integrated Access and Backhaul (IAB)**: IAB allows a base station to provide wireless backhaul to neighbouring base stations.
    + **NR-U**: 5G can operate in unlicensed spectrum, either with a licensed/shared spectrum as anchor or in standalone mode.
    + **Non-Public Network (NPN)**: For industrial IoT, private networks with small cells, dedicated resources and low latency are possible.
    + **Cellular-Vehicle-to-Everything (C-V2X)**: NR-based sidelink is introduced. Multicast groups are defined based on distance and applications.
    + **Positioning**: Meets the accuracy requirements of 3 meters (indoor) and 10 meters (outdoor).

## Milestones

Mar
2017

First drafts of two high-level documents *TS 38.300: NR and NG-RAN Overall description; Stage-2* and *TS 38.401: NR-RAN; Architecture description* are published.

Dec
2017

3GPP approves the first specifications for 5G, called "early drop" of Release 15. Specifically, it ratifies **Non-Standalone (NSA) 5G New Radio (NR)** specification. This enables vendors to start implementing the first 5G products. NSA 5G will allow operators to leverage existing 4G infrastructure. However, it can't support some use cases that require ultra-low latency and higher capacity.

Jun
2018

Also part of Release 15, and called "main drop", 3GPP approves many specifications for 5G including the **Standalone (SA)** option. This allows operators without 4G networks to offer 5G service.

Mar
2019

3GPP approves "late drop" of Release 15. This might aid in migrating from 4G to 5G, or NSA 5G to SA 5G. However, some vendors and operators don't see this as essential since Release 16 or 17 specifications could offer alternatives.

Jul
2020

3GPP finalizes **Release 16** specifications. This adds support for unlicensed spectrum. It improves on latency, power consumption, positioning and cellular-to-vehicle connectivity. Existing features enhanced by Release 16 include MIMO, beamforming, Dynamic Spectrum Sharing (DSS), Dual Connectivity (DC) and Carrier Aggregation (CA).

5G New Radio


# Aphelion Consensus Algorithm

## Summary

A lot of research has been carried out regarding Artificial Intelligence (AI) and its application in the real world. Blockchain also emerged as an important technology in the 2010s. Blockchain's decentralized architecture presents a lot of complex issues, but with AI as the basis of a consensus algorithm, the blockchain system can be redefined to have much higher real-world scalability, decentralization and security. An AI-enabled blockchain can even solve a long-present problem—seamless connection between different blockchains, private and public.

Aphelion is an **AI-governed autonomous blockchain consensus engine**. It's responsible for running many tasks while governing the blockchain, starting from simple ones like extracting features and categorizing nodes that join the network, and ending with detecting security attacks and adjusting network's parameters through time to guarantee the best performance and efficiency at all times.

## Discussion

### What are the key values of decentralized technology?

Three key values of blockchain technology are scalability, decentralization and security. However, blockchain has struggled to strive in all three areas simultaneously. Most of the blockchains right now are trade-off systems.

**Scalability** defines how fast blockchain is in a real-world environment in terms of transactions per second and block finality. Most of the blockchains today are too slow to sustain the use cases and expand.

**Decentralization** is a process of distributing decision-making power from a central authority to the network participants. The main reasons that affect a network's decentralization are governance setup, utility, update model, knowledge and investment requirement, and reward system.

**Security** reflects on 51% attack, double spending, DDOS, poorly written code, off-chain implementations and other issues. Solutions that blockchains propose are very complex. Therefore, many blockchains still face security breaches.

**Interoperability** is another core value. It's the ability of blockchains to communicate with one another. Making transactions from one blockchain to another is a huge milestone, but till now it's not been possible in a decentralized manner, without the use of atomic swap, offline signatures or central authoritative control.

### How is Aphelion solving the blockchain problems?

Aphelion, a concept in blockchain technology that emerged in 2020, is an AI-governed autonomous blockchain consensus engine. With the inclusion of AI in the consensus engine, it enables blockchain to be performant with respect to scalability, decentralization and security.

Aphelion is involved in many tasks, first of which occurs right when nodes join the network. AI evaluates node prior contribution as well as technical characteristics that typically include computing power, disk space, and network speed. This information is combined to understand the nature of the nodes that helps the engine to classify them into different node pools.

Node pools are used to improve the system's efficiency by dividing the workload among them. After each dataset, Aphelion consensus records the performance and learns. Thus in future, it's able to make better predictions on how to split the workload and further increase the efficiency of the system.

### What are other benefits of utilizing Aphelion in blockchain?

Current blockchains lack a fair distribution of nodes in the network. There's always a bottleneck in terms of performance. With multiple pools, Aphelion presents a new mechanism for the blockchain. It accommodates high-performance and low-performance systems, thus not just improving the blockchain's scalability, but also network's decentralization as all nodes on the system, even mobile ones, have a fair chance at contributing and earning rewards.

It's also important to highlight that the major difficulty in utilizing AI in decentralized networks is that artificial agents need to be decentralized as well. Earlier systems lacked this capability and therefore couldn't achieve full decentralization. Aphelion fills that gap successfully with the introduction of an AI-based consensus engine that's completely decentralized.

### What information is gathered from nodes?

To mitigate the issue of taking control of the network and throughput of the blockchain, a feature extraction agent is used to extract system information that can be used to predict not only the performance of the system but also its overall effect on the network in terms of scalability and security. A feature extraction engine extracts node features at the time of their initialization and communicates the extracted data to all the classification agents running on the network.

The four major features that are required for the classification agent includes the node's computing power, bandwidth, contribution and life of a node. For these features, the extraction algorithm, after extracting the data, tags the data with the node's key to remove any kind of duplication and prepares the feature matrix `M = {X1, X2, X3}`:

    + `X1 = {Power Ratio, Life-time, Contribution}`
    + `X2 = {Latency, Connections, Bandwidth}`
    + `X3 = {Assignment probability, Removal probability}`Aphelion uses these features to classify each node to one of the four pools: Power Pool, Exploit Finding Pool, Audit Pool, and Maintenance Pool.

### What else is important for the feature extraction?

**Dataset cleansing**: Once the features are extracted and are received by the classification agent, the dataset for the agent is prepared. The cleansing process includes data detecting and correcting corrupt or inaccurate records, which will be modified according to a specific use case.

**Normalization**: After cleansing the data, it's normalized to avoid any inaccurate learning. Normalization is one of the most important steps before the data is given to any ML or AI algorithm. To normalize the dataset linear scaling formula is used.

**Splitting dataset**: Dataset is split into three portions. 70% of the data will be used for training, 15% of the data will be used for testing and the remaining 15% will be used for validation.

### What is the Power pool?

Power pool is where all the transactions and activities take place. Nodes here are differentiated into three different classes named Lightning nodes, Medium nodes and Lower nodes. With the inclusion of pools and their subclasses Aphelion protocol gives the blockchain an ability to accommodate any kind of node to join the network and participate in validating the transactions.

### What is the Exploit-Finding (EF) pool?

Exploit finding pool serves as the security layer for the blockchain - to save the blockchain from any kind of exploit that could disrupt the whole system. Nodes in the EF pool work collectively to verify and index the blocks included by the power pool.

When any vulnerabilities are exposed, the decision agent is reported and the necessary steps are taken to secure the system beforehand. The dataset is constantly being provided to the decision agent, which predicts the behavior of the nodes in complex scenarios. Exploiting attempts are carried by the algorithm on the isolated copy of the current blockchain, to ensure if there is any kind of vulnerability present and calculate the steps that can be taken by the system for it to behave as it is intended to.

### What is the Audit pool?

Due to inclusion of multi-pool architecture and the decision agent, there is a possibility that invalid data can be provided to the agent. To overcome this issue, an audit pool is included in the system. It serves as the audit layer for the blockchain. The audit pool overlooks the overall working of the pools in the network. To join the audit pool, the node must satisfy the threshold to be regarded as trustworthy, and be a subject to stake in the protocol.

There is also a possibility that a node secretly behaves as a trustful node but after joining the pool it can feed invalid data. To mitigate this issue, the Aphelion protocol includes randomized round trip and multi-layered gossip mechanism. Through this process, nodes are randomly appointed to participate and no node at any instant gets to know whether or not it's the one being appointed.

### How does Audit pool work?

To make a decision, votes are collected by the nodes, but to reduce cost and to be fair, the votes are also cast without the knowledge of the nodes. The multi-layered gossip includes the nested data approach, where the node's data communicated is indexed over the swarm.

By extracting the history of the nodes' index and unmarshalling the data, the protocol calculates the node's acceptance on the data and feeds the calculated value to the decision agent, where the rest of the steps are taken by the decision agent. This ensures privacy, fairness, truthfulness and security. After the round, the extracted data is useless because the algorithm is also programmed with redundancy removal.

After the decision agent takes the necessary steps, audit nodes are programmed to broadcast the rules over the whole network. Firewalls and state machine of each pool are updated with the new rules. This may include an increase in threshold, removal of a node or banning a node from the whole network.

### What is the Maintenance pool?

According to the analysis on computer performance, it's concluded that a system consistently conducting complex operations can reduce its efficiency and life span. The maintenance pool ensures steady performance to save nodes from long and high computation, reduce cost, ensure system's reliability, increase life span and ensure high throughput.

If a node leaves or is removed from the network due to malicious activity, a node from the maintenance pool can be used as a replacement in order to keep the resources intact.

Maintenance pool doesn't refer to a permanent position for nodes. As the Aphelion protocol resolves the issues of fairness, trustfulness and decentralization, each node in the network is given a chance to contribute to the network. The protocol includes the mechanism of randomized round trip and time slots. Whenever a node has processed some amount of transaction and the time slot assigned has been finished, the node is replaced with another node from the maintenance pool. Replacement and movement doesn't make the node to be kept waiting for multiple hours. Instead, nodes are replaced with other nodes in the threshold of 2-3 seconds.

### How does Aphelion ensure fast communication?

For the Aphelion protocol to run as it is intended, it is necessary that the network layer of the protocol supports fast communication and can scale over time.

There is high communication among the nodes and the machine learning models. The whole protocol can be badly affected if uncompressed data is communicated over the network and the same channel is utilized by the nodes. To solve this problem, at the network layer Aphelion utilizes **multi-layered gossip swarming with compression mechanisms**. Nodes interact with one another through the swarm and share the compressed data in the streams of packets.

The gossip communication involves the multi-layered gossip or the nested gossip mechanism, which means that the gossip is layered not only at the swarm level but also at the node level. The protocol is being implemented in Libonomy blockchain, but can allow any blockchain solution to utilize Aphelion network channel.

## Milestones

May
2020

Launch of public Testnet.

Aug
2020

Multi-pool consensus is added to Aphelion algorithm.

Oct
2020

Launch of public Main-net.

Nov
2020

LBY, the main asset of Libonomy blockchain, is released. This uses Aphelion consensus.

Aphelion Consensus Algorithm


# Mobile Robot

## Summary

Mobile robots can move autonomously that is without any assistance from external human operators. A robot is called autonomous when it can determine the actions to be taken to perform a task, using a perception system that helps it. It also needs a control system to coordinate all the subsystems that compromise the robot.

Mobile robots function using a combination of **artificial intelligence (AI)** and physical robotic elements like wheels, tracks and legs. Mobile robots are popular in different sectors. They are used to assist with work processes and even accomplish tasks that are impossible or dangerous for humans. Humanoid robots, entertainment pets, drones, and underwater robots are great examples of mobile robots. They are different from other robots because of their ability to move autonomously. These robots have the intelligence to decide based on algorithms it takes input and gives the expected output.

## Discussion

### What are the features of a mobile robot?

Every mobile robot will incorporate different features that optimize the system to meet a specific goal or to perform a particular task. However, industrial mobile robot systems are mostly used today. They possess several core features that should always be present. These features are:

    + Integrated safety: Safety is the main thing that a mobile robot is
    + Wireless communication
    + Fleet management software
    + FLeet simulation software
    + Integration with supervisory software

### What are the different types of mobile robots?

Mobile robots are classified in two ways based on the environment they work in and on the device they use to work.

    + *Aerial robots*: These are also known as Unmanned Aerial Vehicles (UAVs). They can fly through the air. Eg: drones, helicopters, etc.
    + *Land robots*: They are also known as Unmanned Ground Vehicles (UGVs). These robots can navigate on dry land and in buildings. They are used to supply equipment. Eg: Dirtdog etc.
    + *Underwater Robots*: Autonomous Underwater Vehicles (AUVs) are the other name for underwater robots. They can direct themselves and efficiently travel through water. Eg: Wave gliders.
    + *Wheeled robots*: Wheeled robots are known as Autonomous Intelligent Vehicles (AIVs). These use the wheels for their locomotion. There are different types of wheeled robots like one-wheeled robots, two-wheeled robots, etc.
    + *Humanoid robots*: They have the shape of a human body. They work majorly with the help of sensors. Eg: Androids, etc.
    + *Legged robots*: These robots use articulated limbs like legs and follow a mechanism to provide locomotion. Eg: Haxapad robot etc.

### What are the parts of a mobile robot?

The main components of an autonomous mobile robot are a microcontroller, chassis, motors, and sensors.

    + **Microcontroller / SBC**:A microcontroller is the brain of the robot. Arduino and Raspberry Pi are popular options. Raspberry Pi is a single-board computer (SBC) that can run a full Linux OS. An Arduino is ideal for simple robotics projects. Some microcontrollers are Teensy, BeagleBone, micro: bit, and Raspberry Pi Pico.

    + **Chassis**:The chassis is the body where electronics and motors are mounted. They come in various sizes and materials. Some alternative chassis is made with plastic, metal, wood, or cardboard.

    + **Motors**:Motors are fuel for a robot. Some motors have an inbuilt gearbox to increase torque and drive heavier loads. PWM (pulse-width modulation) controls motor speed.

    + **Power**:The power source is of two types. Use separate power sources for motors and electronics. Another is using a single power source connected to both via a BEC (battery eliminator circuit). USB power banks, battery packs, and LiPo batteries are examples.

    + **Sensors**:Sensors allow the robot to move autonomously. An ultrasonic distance sensor sense an obstruction. An IR sensor detects a dark line.

### What are the types of sensors used in mobile robots?

Sensors are the eyes of a mobile robot. When combined with sensor software algorithms, sensors allow a robot to understand and navigate the environment, detect and avoid collision with objects, and provide location information about the robot. Exteroceptive sensors visualize the world with cameras, lasers, lidar, radar, sonar, infrared, touch sensors such as whiskers or bump sensors, GPS and proximity sensors.

Proprioceptive sensors deal with the robot itself. They include accelerometers, gyroscopes, magnetometers, compass, wheel encoders and temperature sensors.

Some typical sensors used in ground mobile robots and drones include:

Inertial measurement units (IMUs) combine multiple accelerometers and gyroscopes. They also include magnetometers and barometers. GPS provides latitude, longitude and altitude information. Vision sensors like cameras, both 2D and 3D, as well as depth cameras play an important role in mobile robots. Computer vision and deep learning can aid object detection and avoidance, obstacle recognition and obstacle tracking. Visual odometry and visual-SLAM are becoming more relevant for mobile robots operating in both indoor and outdoor environments.

### What is the control system used in a mobile robot?

A robot control system manages commands and directs or regulates the movement and function of various parts of the mobile robot to achieve the desired result. Automatic control is an important requirement for any robotics control installation. The control system of a robot is a feedback control system. It continuously reads from sensors and updates the commands for the actuators to achieve the desired robot behavior. The controller controls and coordinates all aspects of the operation. For high-level performance, the robotics control uses a closed-loop control system like PID controllers, which uses sensor feedback. A feedback control system consists of five basic components like:

    + Input
    + Process being controlled
    + Output
    + Sensing elements
    + Controller and actuating devices

### what does it make a mobile robot autonomous?

Autonomous mobile robots act on their own. It is programmed to respond the outside stimuli. These robots have a bumper sensor to detect obstacles. When a robot hits an obstacle, the impact triggers the bumper sensor. The robot's programming tells it to back up and take another direction. Mobile robots often use infrared or ultrasound sensors to see obstacles. More advanced robots analyze and adapt to unfamiliar environments like rough terrain. These robots may associate certain terrain patterns with certain actions. The best example of an advanced robot is NASA's preservance rover.

### Which areas and sectors can collaboratively mobile robots be applied to?

The basic function of a mobile robot is to move and explore, transport, and complete complex tasks. They are also used in medicine, surgery, personal assistance, and security.

Some uses of mobile robots in many areas and sectors such as:

    + **Health Care**:Robotics has the potential to change many health care practices like surgery, rehabilitation, and therapy. Robotics are used in surgeries like cardiac, gynecologic, thoracic, urologic, etc.

    + **Agriculture**:There is a rise in the experimental use of mobile robots in operations like pruning, thinning, mowing, spraying, and weed removal.

    + **Food and Beverage Industry**:Mobile robots are used on stationary conveyor equipment. These robots do offer a high amount of operational flexibility.

    + **Manufacturing**:Robotics is being used in many aspects of manufacturing to increase productivity and efficiency while lowering production costs. Robotic manufacturing technology became safe to operate.

    + **Military**:Robotic technology is being applied in many areas of the military. An unmanned drone is an example. Military drones flying over areas of war and conflict, in hostage situations, and during natural and manmade disasters can assess danger levels and provide soldiers and first responders with real-time information.

## Milestones

1942

Do you know that robots are used in World War II? Yes, this is the first time that robots are used in real-time war situations. These mobile robots are used as flying bombs to deploy bombs on enemies .

1960

In the present era, we all know that robots can charge themselves when they ran out of power. But this technology was first implemented by Johns Hopkins University. The name of the robot that used this technology is "Beast". One more interesting fact is it uses sound waves to move around .

1976

Till 1976 robots are used in land and water applications. For the first time, NASA used a mobile robot in a space mission to Mars. The name of the program is Viking program .

1989

Mark Tilden used simple analog circuits like comparators instead of microprocessors. He proved that robots can be in the simplest shapes and efficient in doing a task .

1996

After the Viking program, NASA used mobile robots again in its Mars Pathfinder mission. They used the robot in their rover Sojourner to explore the surface of Mars .

1999

Sony designed a robot similar to a dog called Aibo. It is the first robotic dog designed by Sony. It is capable of seeing, walking and interacting with its environment .

2002

Robots are used in wars, on land and water applications, and in space too. Then iRobot organization invented a mobile robot for domestic applications. The name of the robot is Roomba .

2005

Boston Dynamics invented a quadruped robot that can carry heavy loads in terrain areas too .

2016

Mobile robots are used in solving crimes too. US police used a mobile robot named MARCbot. The Multi-Function Agile Remote-Controlled Robot (MARCbot) helped the US police to kill a sniper who killed 5 police officers in Dallas, Texas which raises ethical questions regarding the use of drones and robots by police as instruments of lethal force against a perpetrator.

Mobile Robot


# Race Condition (Software)

## Summary

Race condition in software is an undesirable event that can happen when multiple entities access or modify shared resources in a system. The system behaves correctly when these entities use the shared resources as expected. But sometimes due to uncontrollable delays, the sequence of operations may change due to relative timing of events. When this happens, the system may enter a state not designed for and hence fail. The "race" happens because this type of failure is dependent on which entity gains access to the shared resource first.

One definition of race condition describes it as "anomalous behavior due to unexpected critical dependence on the relative timing of events." This definition is broad in the sense that it can apply to both hardware and software race conditions. In software, race condition happens because of some shared resource whose access is not properly controlled by design.

## Discussion

### Can you give one example of race condition?

Suppose a process calls a function that's supposed to increment a value (v) stored in a database or some shared memory. This operation is not atomic, meaning that it can be broken down into smaller steps. The function will read the current value from database (A), store it in memory as a function variable (B), increment it (C) and finally write the new value to database (D). This function's execution is therefore a sequence of steps A-D.

Race condition will happen if process P1 calls this function and proceeds to step C. Meanwhile, it gets preempted by the operating system and another process P2 gets its chance to run. P2 calls the same function, completes all the steps A-D and returns. When P1 resumes, it continues from step C using an old value (v), not P2's result (v+1).

Race condition would not have happened if P1 had completed all steps without preemption; or P2 had been prevented from executing the function until P1 had completed all steps.

### How is it possible that my one-line code is causing a race condition?

What appears as a single line in a high-level programming language, will actually translate to multiple assembly instructions that involve multiple clock cycles. The executing process or thread can therefore get interrupted anywhere along this sequence of assembly instructions. Thus, race condition is still possible.

Microsoft Support shows on its site how a single line of Visual Basic code such as `Total = Total + val1` translates to six assembly instructions for a x86-based processor.

### What systems are prone to race conditions?

Because race condition is related to timing, it can happen in any system where multiple processes "race" to access a shared resource. Hence, it's not limited to just real-time systems. Any multithreaded or distributed system can have race conditions.

Hardware containing multicore CPUs, common in parallel computing, can suffer from race condition since multiple processes are running at the same time on multiple cores or CPUs. In networked systems, the channel can be considered as a shared resource and if two users attempt to access the shared channel race condition will lead to a packet collision. Network links that have large delays, such as satellite links, can cause race condition. Multithreaded applications are prone to race condition.

Web applications can suffer from race condition since multiple clients are sending requests in parallel to the web server, which may spawn multiple threads or processes to serve the requests. Even for a single user session, race condition is possible. For example, a web chat application may periodically send an AJAX request to check for the latest updates. At the same time, the user may create a new chat message.

### Are there any best practices for avoiding race condition?

The usual solution to avoid race condition is to serialize access to the shared resource. If one process gains access first, the resource is "locked" so that other processes have to wait for the resource to become available. Even if the operating system allows other processes to execute, they will get blocked on the resource. This allows the first process to access and update the resource safely. Once done, it will "release" the resource. One of the processes waiting on the resource will now get its chance to access the resource. Code protected this way using locks is called **critical section**. The idea of granting access to only one process is called **mutual exclusion**.

Having large critical sections will affect performance. Such code may be refactored into smaller critical sections. In real-time systems or kernel code, another solution is to turn off interrupts when entering (and turn on when leaving) a critical section.

### How to detect race condition in a program?

The first symptom is that results are unpredictable. In fact, the random nature of race condition poses a problem for debugging since under debug mode the race condition may not appear. For this reason, code reviews are recommended to catch race condition. This really implies that best way to avoid race condition is by careful design and not by testing and debugging.

There are tools that perform dynamic analysis and flag possible race conditions that may occur. Examples include Coverity's Thread Analyzer for Java and Intel Inspector. Clang Thread Safety Analysis is a static analysis tool to detect race condition. ThreadSanitizer is a data race detector written in C++ used by Clang and Go languages.

### Do real-time operating systems (RTOS) have provisions to mitigate race conditions?

The common mechanism used to allow safe access to shared resource is called **mutex**, which is short for mutual exclusion. Any process entering a critical section will first acquire the mutex. Other processes wanting access to the critical section will get blocked. The owning process will give up the mutex when leaving the critical section.

What separate mutexes from semaphores is the concept of ownership. A mutex can be released only by its owner. This is not so with semaphores that can be used for signalling state change and synchronization from one thread to another. While it's possible to use semaphores to protect critical sections, they come with their own set of problems such as accidental release, recursive deadlock, task-death deadlock, and priority inversion. Mutexes are therefore a better solution.

### Are there different types of race conditions?

Netzer and Miller make this distinction:

    + **General races**: Programs designed to be deterministic behaving in nondeterministic manner.
    + **Data races**: Failure with accessing nonatomic critical sections in nondeterministic programs.They also claim that debugging data races is easier since it's local to the critical section. General races require analysis of entire execution to understand exactly how the expected deterministic behaviour deviated.

A data race need not result in a race condition in the sense that program correctness is not compromised. A race condition that's not a data race implies a general race, for which the accompanying figure is an example.

### Can you give examples of products that failed due to race conditions?

Therac-25 was a software-controlled radiation therapy machine used in the 1980s. Six patients were overdosed and some died. Race condition was one of many reasons for the system's failure. Shared variables were accessed by both data entry subroutine and magnet control subroutine. When a race condition occurred, the magnet control subroutine failed to read new settings by the operator.

In August 2003, Northeastern parts of North America suffered a massive blackout. "There was a couple of processes that were in contention for a common data structure, and through a software coding error in one of the application processes, they were both able to get write access to a data structure at the same time. And that corruption led to the alarm event application getting into an infinite loop and spinning."

NASA's Mars Spirit Rover suffered a race condition whereby an initialization process could not obtain write access to a variable. An exception occurred. In another case, an imaging module was attempting to read from memory while a deactivation process was also triggered. Other problems, possibly involving race condition, have been reported.

### Can race conditions be useful?

Race conditions are something software designers wish to avoid. But some researchers have attempted to show that they can be used to generate random numbers. This is dependent on the operating system's scheduler, which by itself is not random. Randomness is obtained by triggering context switches based on "execution environment, cache misses, the instruction execution pipeline and the imprecision of the hardware clock used to generate timer interrupts."

## Milestones

1954

In an early use of the term "race condition", Huffman notes in his thesis that this can occur in sequential switching circuits. This is a case of race condition in hardware.

1965

Dutch computer scientist Dijkstra introduces the concept of **semaphores** as a tool to prevent race conditions in software. However, semaphores have limitations and don't completely eliminate race conditions.

1968

In describing *THE Multiprogramming System* developed for a Dutch machine called EL X8, Dijkstra explains how semaphores can be used to enable mutual exclusion on critical sections. In an example, he names a semaphore as "mutex". However, this is just a semaphore. It's quite different from the concept mutexes that take shape in the 1980s.

1988

The concept of **mutex** is developed in the late 1980s. The exact history of mutexes is uncertain. They probably came about in the course of specifying the IEEE Std 1003.1, often known as POSIX. This standard is first released in 1988. At this date, it mentions semaphores but not mutexes. Mutexes probably got introduced into the standard in the 1990s.

Race Condition (Software)


# Wrangle (Language)

## Summary

Real-world data is messy. Before we can do any useful analysis on data, we need to clean or format it in a manner that's acceptable to data analysis or visualization tools. This process is often called **data wrangling**. We can do this interactively but it's better to record all actions into a script or write a custom program. This will help us document how the data was wrangled and also repeat the process with new data.

Wrangle is a proprietary language for automating the task of data wrangling. It's owned and managed by Trifacta. Once wrangled, data can be exported in CSV or JSON formats, which are supported by most analysis or visualization tools.

The essence of Wrangle is this:

> Spend less time fighting with your data and more time learning from it.

## Discussion

### In Wrangle, what are transforms, functions and recipes?

Here are the three essentials aspects of Wrangle:

    + **Transform**: This is an action that modifies the dataset in a specific way. Typically, a dataset is in table format with rows and columns. Transforms accept parameters to set the context in terms of rows, columns or conditions. Examples include deduplicate, delete, derive, drop, extract, filter, flatten, nest, pivot, replace, etc.
    + **Function**: Like in any programming language, a Wrangler function is a computation unit. It works on one of more columns of data. Functions can be passed as parameters to transforms. There are lots of functions that are usually organized into function categories such as aggregate (COUNT, MAX), logical (AND, OR), comparison (ISODD, LESSTHAN), math (DIVIDE, SQRT), date (DATEADD, NOW), string (LOWER, FIND), nested (ARRAYCONCAT, LISTSUM), type (IFNULL, ISMISSING), window (FILL, ROLLINGSUM), and others (RAND, RANGE).
    + **Recipe**: This is a sequence of transforms applied to a dataset.It's common to name transforms in lowercase and functions in uppercase. Transforms and functions are the building blocks from which powerful recipes can be built.

### Could you describe the Wrangle syntax with some examples?

Wrangle transforms follow the general syntax of the transform's name followed by optional parameters: `(transform) param1:(expression) param2:(expression)`. Let's look at a few examples to understand this.

To create a new column called "circumference" based on another column of values named "diameter", we can apply the `derive` transform: `derive type:single value: (3.14159 * diameter) as: 'circumference'`

To create a column of Boolean values indicating big orders, we can use the `derive` transform: `derive type:single value:IF(order > 1000000, true, false) as:'bigOrder'`

By passing a regular expression parameter to the `replace` transform, we can delete last two digits from "qty" column: `replace col: qty on: /^\d$|^\d\d$/ with: '' global: true`

Suppose a column named "myCol" is in the format of key-value pairs, such as `{"brand":"Subaru","model":"Impreza","color","green"}`. We can separate this into three columns using the `unnest` transform: `unnest col:myCol keys:'brand','model','color'`

We can use `filter` transform to keep only rows that match a specific condition: `filter row: (row_number >= 25 && firstName == 'Steve') action: Keep`

### Which data types are supported in Wrangle?

Basic data types include `String`, `Integer`, `Decimal`, and `Boolean`. String size is limited by size of a row of data, which is about 1 MB. Integers can be safely used within the range (-2^53 + 1) to (2^53 - 1). Decimals are limited to 15 floating point digits. It's not clear if these are limits of the Wrangle language or the Trifacta Wrangler family of products.

Other data types include Social Security Number, Phone Number, Email Address, Credit Card, Gender Data, Zip Code, State, IP Address, URL, HTTP Code and Datetime. However, some of these won't be useful for non-U.S. data.

`Array` type can be used to group together a sequence of values. Arrays can be *nested*; that is, an array containing another array. Arrays can be *ragged*; that is, the number of items in an array can vary from one data record to the next.

To store key-value pairs, we can use the `Object` type. This can also be nested like in arrays but there's no ordering of items within the type.

### What are some concerns about the Wrangle language?

Wrangle language is not an open standard. It's owned and managed by Trifacta. It's supported in Trifacta's Wrangler family of products. It's also available within partner products such as Google's Cloud Dataprep. While learning the language is useful, it's not essential since these products often generate the language syntax for the user via contextual suggestions and steps.

Because it's not open, there's a vendor lock-in. This means that any data preparation workflows that you build with Wrangle can't be reused when you move to another platform. For example, AWS has its own tool for data preparation called *Glue*. ETL workflows can be programmed in Glue using Python, PySpark extensions and Scala. However, anything written in Wrangle can't be reused in AWS. Only the wrangled data can be exported and imported into other tools or platforms.

## Milestones

2001

Researchers from UC Berkeley publish a paper titled *Potter’s Wheel: An Interactive Data Cleaning System*. They identify a number of transforms that can help with data wrangling.

May
2011

Led by researchers at the Stanford Visualization Group, a paper titled *Wrangler: Interactive Visual Specification of Data Transformation Scripts* is published. While the tool is visual and interactive in nature, the paper also talks about a **declarative transformation language** that has evolved along with the tool. Among the transforms supported are delete, extract, cut, split, lookups, joins, fold, unfold, fill, lag; plus sorting and aggregating functions such as sum, min, max and mean.

Feb
2014

Stanford's Wrangler is commercialized with the release of *Data Transformation Platform*. An alpha version of this was available back in April 2013 and the company behind it, Trifacta, itself was founded in 2012.

Oct
2015

Trifacta releases **Wrangler**, a free desktop application for data wrangling. The has limited features compared to the commercial offering that's renamed to **Trifacta Wrangler Enterprise**.

Mar
2017

Google launches **Cloud Dataprep** for data preparation in the cloud. Under the hood, it makes use of Trifacta Wrangler Enterprise along with the Wrangle language.

Apr
2018

Trifacta Wrangler is released as a cloud product. The desktop version will no longer be updated and cease operation (implying data loss) in August 2019.

Mar
2019

New functions are introduced including ARRAYINDEXOF, ARRAYRIGHTINDEXOF, ARRAYSLICE and ARRAYMERGEELEMENTS.

May
2019

Functions RANK and DENSERANK are introduced.

Wrangle (Language)


# Kerberos

## Summary

Kerberos is a network authentication protocol for client-server applications based on cryptographic keys. It's used in Windows 2000, Windows XP and Windows Server 2003 and later systems. Because it's an open standard, it can also used by non-Windows systems.

Unlike password-based authentication systems, passwords are never sent over the network. Kerberos authenticates by verifying identities of users and servers within a trusted environment. As such, Kerberos is inaccessible for outsiders. Once authenticated, the user or client can access multiple services without needing to authenticate again for each service. Encrypted tickets are used instead of passwords.

Security aspects covered by Kerberos include authentication, access control, and key exchange.

## Discussion

### What are some essential Kerberos terms that a beginner should know?

Here are some essential Kerberos terms:

    + **Principal**: This is a unique identity. Because Kerberos supports mutual authentication, both users and network resources have principals.
    + **Realm**: Called *domain* in Windows, this is a logical grouping of resources and identities.
    + **Ticket**: Ticket is an encrypted block of data that authenticates the user. Because of tickets, we don't need to transmit clear passwords over the Kerberos infrastructure. Tickets come with a configurable time expiry.For example, `jdoe@SALES.WIDGET.COM` is a principal for user `jdoe` within the realm `SALES.WIDGET.COM`. It's quite possible for users to have multiple principals in different realms with different access rights. Another example is users enter the system in one realm and access resources in other realms.

### Could you give an overview of the Kerberos' authentication flow?

Central to Kerberos is the **Key Distribution Center (KDC)** that has a **database** of trusted principals, the **Authentication Service (AS)**, and the **Ticket-Granting Service (TGS)**.

When a user logs into a client machine, she supplies a password. To gain access to network services, Kerberos protocol is invoked to authenticate the user. However, network services don't do the authentication on their own. KDC (and its AS) is the trusted third party that does this. The user's principal, and not the password, is sent to the KDC. KDC checks this against its database, authenticates the user and sends back the Ticket-Granting Ticket (TGT). All this happens transparent to the user when she logs in or invokes the `kinit` command.

When the client wants to access a service, it presents the TGT to the TGS, which generates service session key and a service ticket containing this key. Whenever client wants, it presents the service ticket to the service. The client and the service can now start communicating, with all communication encrypted with service session key.

### What are the different tickets used in Kerberos?

Kerberos has essentially two tickets. One is issued by the AS during authentication. The other is issued by the TGS when a service is requested.

AS issues the **Ticket Granting Ticket (TGT)**. TGT is a ticket to tell the TGS that the client has already been authenticated.

TGS issues the **Service Ticket** after validating the TGT. This enables the client to access a service. Service ticket contains client's IP address, timestamp, and the service session key.

Both tickets are encrypted. Although the client is in possession of these tickets, it can't read them. The TGT is meant for the TGS. The service ticket is meant for the service provider.

### What are the different keys used in Kerberos?

The following keys are used in Kerberos:

    + **Principal Key**: Based on the principal's password, this encrypts the TGT and the session key between client and AS.
    + **Session Key**: This is generated by the KDC during authentication and sent to the client. This is used to encrypt communication between client and TGS. In particular, TGT, service ticket request, service ticket, and service session key are encrypted using this.
    + **Ticket Granting Service (TGS) Key**: This encrypts the TGT. The TGS uses this key to decrypt the TGT. TGS then checks that TGT hasn't expired and generates the service ticket and service session key.
    + **Service Key**: This encrypts the service ticket. It's available within the KDC and also with the service provider.
    + **Service Session Key**: This is used to encrypt all communication between the client and the service. This is generated in the TGS.

### What are the main advantages of using Kerberos?

Here are some advantages of Kerberos:

    + **Mutual Authentication**: Client and server authenticate each other. This protects clients from connecting to a bogus server.
    + **Open Standard**: It's based on RFC 4120. Non-Windows systems that implement this RFC can also use Kerberos.
    + **Delegation**: Also called *authentication forwarding*, a service can access remote resources on behalf of the client using the client's identity.
    + **Smart Card Logon**: Beyond password logon, two-factor authentication using a smart card and PIN is possible.
    + **Passwords & Keys**: User passwords are never sent over communication links. Eavesdroppers can't get hold of passwords. Secret keys are sent only in encrypted form. There's support for both symmetric and asymmetric keys.
    + **Encrypted Sessions**: Service session key shared between the client and a service can be used to encrypt all conversation pertaining to the service.
    + **Integrated Sessions**: Once authenticated, client can access a service without having to re-authenticate, until the ticket expires.
    + **Renewable Sessions**: After the first session, subsequent sessions are setup faster. Server authenticates the client immediately without involving the KDC.

### What's the difference between Kerberos impersonation and delegation?

Clients are authorized to access certain services but the services themselves run in a different security context, such as on a different thread, process or machine. Services acquire client credentials (such as service tickets) and use these to obtain resources on behalf of the clients. In some sense, services therefore impersonate the clients they serve.

There are four impersonation levels: anonymous, identify, impersonation and delegation. If the service is on the same computer as the client process, it can impersonate the client to access network resources. If the service is on a remote computer, it can impersonate the client for accessing resources on the service's computer. With delegation, the service can impersonate the client even when accessing resources on another computer.

Let's assume that a user requests data via a web server but the data resides on a different database server. The web server delegates to the database server to obtain necessary data from the database. Within the security context of the database server, which accesses the database on the same machine, we can say that impersonation happens.

### How is Kerberos related to GSSAPI?

**Generic Security Service API (GSSAPI)** is a standard interface, defined by RFC 2743, that provides a generic authentication and secure messaging interface. Multiple security mechanisms can be plugged in so long as they conform to GSSAPI. IETF has defined two mechanisms: **Kerberos V5** via RFC 1964 and **Simple Public Key Mechanism (SPKM)** via RFC 2025.

GSSAPI is an abstraction that must be accompanied by a security mechanism. The advantage is that applications are not tied to a particular mechanism. Migration is possible, such as, from SPKM to Kerberos V5. In fact, an implementation might support multiple mechanisms and applications can choose the mechanism at runtime. However, both client and server must negotiate to use the same mechanism.

GSSAPI was designed with the following goals:

    + **Mechanism independence**: Applications need not concern with the security mechanism such as the type of cryptographic keys used.
    + **Protocol environment independence**: API is not tied to particular communications protocol.
    + **Protocol association independence**: A single API implementation can be used by different application modules that possibly use different communications protocols.

### What are some limitations of Kerberos?

Kerberos is only as secure as the passwords being used. Weak passwords make the system vulnerable to brute force attacks. During protocol use, encryption keys are stored in memory in unencrypted form. Kerberos can't do anything about compromised user endpoints, authentication servers or KDC. If the authentication server is down, new users can't login. *Pass the Ticket* attack involves getting access to a ticket, moving laterally within the network and gaining access to critical systems.

Kerberos has strict timing requirements. Usually Network Time Protocol (NTP) daemons are used for clock synchronization. NTP therefore becomes a dependency.

If each service requires a different host name, each must use its own keys. This complicates deployment of virtual hosting and clusters. Kerberos works in a trusted environment and therefore hard to use on the public Internet where untrusted or unknown clients may want to connect.

Kerberos has worked well within the enterprise but hasn't been adopted in the cloud. A Kerberos ticket contains an encrypted key that could be hacked. Cloud services prefer to use SAML 2.0, OAuth 2.0 or OpenID Connect. In fact, cloud services have been poor in adopting identity federation standards.

## Milestones

1978

Researchers Roger Needham and Michael Schroeder invent a protocol based on symmetric keys. This protocol aims to securely establish session keys between two parties to protect further communication.
 The **Needham–Schroeder symmetric key protocol** forms the basis for Kerberos.

1983

At MIT, Project Athena is launched with the goal of creating a distributed computing environment for educational purposes. The idea is to have thin clients and let servers do the demanding computations. As part of this project, **Kerberos** is invented for authentication and single sign-on.

1988

At the Usenex conference, the Kerberos v4 protocol is described for the first time. This makes Kerberos one of the oldest authentication protocols since many others come years later: SAML (2002), WS-Federation (2003), OAuth2 (2010) and OpenID Connect (2014). Versions 1-3 of the protocol remained internal to MIT.

Jan
1989

**Kerberos v4** is released. It uses DES for encryption. Due to export restrictions, Kerberos can't be used outside the U.S. An alternative implementation from Sweden called **KTH-KRB** is released years later for non-U.S. markets. However, v4 has limitations: weak DES encryption, misuse of PCBC mode of DES, ticket lifetime can't be longer than about 21 hours, delegation not supported.

1993

**Kerberos v5** is released and standardized by IETF as RFC 1510. It overcomes many limitations of v4. It uses ASN.1 syntax. It adds new modes for DES. AES encryption is supported. Ticket lifetimes are much longer. Delegation is supported. It allows for unkeyed checksums (CRC, MD5, SHA-1) and keyed checksums (HMAC with MD5 or SHA-1). This is also the year when Microsoft decides to adopt Kerberos in its products.

1999

Microsoft releases Windows Server 2000 with Kerberos as the default authentication protocol, thus replacing NTLM. There's no separate KDC database. Instead, information comes from Active Directory (AD). With the subsequent growth of AD and AD-enabled apps, use of Kerberos grows.

2005

**RFC 4120** clarifies many aspects of the v5 protocol and obsoletes the earlier RFC 1510 of 1993. **GSS-API** specification is also released as RFC 4121.

2007

**MIT Kerberos Consortium** is formed for the continued development and promotion of Kerberos in a more open manner rather than being internal to MIT. For some time, this is renamed to MIT Kerberos & Internet Trust Consortium or MIT-KIT, but later this name is retired.

Kerberos


# Hybrid ARQ

## Summary

A typical communication channel has imperfections. Packets get lost or corrupted. When the receiver detects errors, it discards the erroneous packet and requests the sender to retransmit it. This is called **Automatic Repeat Request (ARQ)**. Error detection is usually done using Cyclic Redundancy Check (CRC) bits.

However, if errors are within certain limits, the receiver can not only detect but also correct errors. This is possible if the sender employs Error-Correcting Codes (ECCs). These include parity bits that help in correcting errors in information bits. The technique is called **Forward Error Correction (FEC)**. FEC avoids ARQ retransmissions but parity bits are an overhead.

Hybrid ARQ is a technique the combines both ARQ and FEC. Packet errors are corrected where possible. Where not possible, retransmission is requested. First and subsequent packet transmissions are combined to increase the chance of correct decoding.

## Discussion

### What's the typical HARQ processing pipeline?

The pipeline starts with a packet of information bits coming into the sender's HARQ entity. CRC bits are added to the information bits. Then an FEC encoder uses an ECC and maps its input to a codeword. Multiple such codewords are buffered and interleaved to protect against channel burst errors. Interleaving distributes such errors across codewords so that there's greater chance of recovering all packets. Interleaved bits are finally modulated for transmission on the channel. Modulation type and order are selected according to recent channel conditions.

At the receiver, the demodulation and deinterleaving are followed by HARQ operation. FEC decoder uses either soft or hard decision decoding. It maps its input to closest valid codewords. A CRC check determines if the FEC decoder has got it right. If CRC check fails, a retransmission is requested.

Combiner at the receiver combines all transmissions of the packet so that FEC has a better chance of decoding the packet correctly. Sender and receiver buffers are for retransmission and combining respectively.

### How is Chase Combining different from Incremental Redundancy?

There are two basic ways in which multiple transmissions of a packet can be combined at the receiver. The simpler method is Chase Combining (CC) in which all transmissions contain the same information and parity bits. Each transmission therefore adds to the overall packet energy.

Incremental Redundancy (IR) is a method in which each retransmission pertains to the same information bits but contains a different subset of information and parity bits. Parity bits are also called redundancy bits. IR is therefore about progressively sending more of these bits to help the FEC decoder. Like CC, IR retransmissions add to the overall packet energy. More importantly, IR retransmissions decreases the code rate since each retransmission increases the redundancy. Implementing IR is more complex than implementing CC.

In poor channel conditions (low SNR), CC does well but when the channel is good, parity bits take up bandwidth unnecessarily. IR is a better method since more parity bits are sent only when needed. IR's first transmission can even include only information bits.

### What are the different types of HARQ?

Researchers refer to three types of HARQ:

    + **Type-I**: In the simplest case, errors are corrected if possible. Otherwise, erroneous packets are discarded and retransmission is requested. In an alternative case, all packet transmissions are buffered and combined using Maximal Ratio Combining (MRC). The latter case is basically CC.
    + **Type-II**: This is basically IR in which different redundancy bits are sent with each retransmission. This can be realized via bit puncturing that effectively changes the code rate over all transmissions.
    + **Type-III**: In Type-II, apart from first transmission, more than one buffered version of the packet is required for decoding. In Type-III, each transmission is *self-decodable*. Bits common to transmissions can be chase combined. This is also called *partial IR* since each retransmission includes more redundancy bits.Some use the term Repetition Redundancy (RR) instead of Chase Combining. Chase Combining has also been called Packet Combining while Incremental Redundancy has been called Code Combining in literature.

ARQ protocols called Stop-and-Wait, Go-Back-N and Selective Repeat are applicable to HARQ.

### What's the difference between truncated and persistent HARQ?

In really bad channel conditions, many HARQ retransmissions may be needed. If there's no limit on the number of retransmissions, we call it **persistent HARQ**. Otherwise we call is **truncated HARQ**.  Truncated HARQ is preferred for real-time applications. The maximum can be selected based on throughput efficiency, packet delay, packet drop rate and buffer space requirements.

Some systems may specify a deadline or maximum time limit for the retransmissions. Past this deadline, the packet is dropped and retransmissions are stopped. This is called **time-truncated HARQ**.

5G New Radio doesn't specify a limit on the number of retransmissions.  However, such a limit may be provided by the network equipment vendor and operator configurable. Kawser et al. have proposed truncated HARQ for 4G/LTE for UEs with poor downlink reception.

### What's the difference between synchronous and asynchronous HARQ?

With synchronous HARQ, the timing is regular and known in advance. For example, the time between a packet transmission and its ACK/NACK could be fixed. The time between a NACK and the next retransmission could be fixed.

With asynchronous HARQ, timing is configurable and highly dynamic. Scheduling can be based on current channel conditions and resource requirements. Asynchronous HARQ requires additional signalling compared to synchronous HARQ.

5G NR uses asynchronous HARQ since 5G already has a highly dynamic TDD frame structure. Moreover, 5G can operate in unlicensed spectra where resources can't be assigned in advance to support synchronous HARQ.

### What's the difference between non-adaptive and adaptive HARQ?

With **non-adaptive HARQ**, the modulation and coding remains the same across all transmissions of a packet. With **adaptive HARQ**, these can change. For example, the first transmission might have used 16-QAM but if channel quality is seen to have worsened, subsequent retransmissions could you QPSK with more radio resources allocated to accommodate the packet in the lower modulation.

Adaptive HARQ is more common in wireless systems due to the time-varying nature of wireless channels. Adaptation depends on channel feedback from the receiver. Transmission power, modulation, code rate, bandwidth, packet length, maximum number of retransmissions, and allocated resources are some aspects that can be adapted. There are many techniques in adaptive HARQ but their objectives may differ: throughput, delay, spectral efficiency, power or reliability.

It's been shown that adaptive truncated HARQ is a robust approach for real-time applications.

### What codes are suitable for detecting or correcting errors in HARQ?

Error detection is commonly done using using Cyclic Redundancy Check (CRC) bits. These are computed based on and appended to the information bits. CRC is often implemented in hardware using Linear Feedback Shift Registers (LFSRs). CRC field is typically 8-64 bits long. Since this overhead reduces overall system capacity, CRC-free alternatives to error detection have been researched. However, 4G, 5G and Wi-Fi standards have adopted CRC.

Error-Correcting Codes (ECCs) for HARQ are typically Turbo Codes, Turbo Product Codes (TPCs), Low Density Parity Check (LDPC) Codes, and Polar Codes. At least for HARQ, these have replaced an older generation of ECCs such as convolutional, Bose–Chaudhuri–Hocquenghem (BCH) and Reed-Solomon (RS) codes.

### How have different wireless communication protocol standards adopted HARQ?

HARQ is widely adopted in major wireless systems:

    + **3G/UMTS**: Asynchronous downlink HARQ. Synchronous uplink HARQ. Stop-and-Wait HARQ with up to 8 parallel processes. Supports both CC and IR.
    + **CDMA2000**: Similar to 3G/UMTS but up to 4 processes and maximum of 3 retransmissions.
    + **4G/LTE**: Similar to 3G/UMTS with 8 processes in FDD and 1-15 processed in TDD. Maximum number of retransmissions can be configured. Uplink HARQ is typically non-adaptive but can be made adaptive. TTI bundling of up to 4 TTIs allows UEs to transmit all redundancy versions without waiting for feedback. HARQ operates at MAC and PHY layers.
    + **5G**: Asynchronous and adaptive in both downlink and uplink. Up to 16 parallel processes, each using Stop-and-Wait.
    + **WiMAX**: Supports both CC and IR. Up to 16 parallel processes each in uplink and downlink, each using Stop-and-Wait. Maximum of 4 retransmissions. Asynchronous adaptive downlink HARQ. Synchronous adaptive downlink HARQ but default operation is non-adaptive HARQ.
    + **IEEE 802.15.4**: HARQ Type-I. Maximum of 0-7 retransmissions with 3 as default. Retransmissions also have a timing constraint.
    + **Bluetooth**: Synchronous HARQ Type-I.

### What are some research initiatives on HARQ?

Adaptive Cross-Packet HARQ (XP-HARQ) is a technique that varies the number of bits with each retransmission. A retransmission can contain bits from two packets and joint decoding happens at the receiver. Sender selects codebooks based on current channel conditions.

Cooperative HARQ is where retransmissions can happen from a neighbouring node that managed to receive the first transmission correctly. This gives spatial diversity gain at the intended receiving node. A similar but more sophisticated idea encodes packets meant for different receiving nodes into each retransmission. This is called Network Coded HARQ (NC-HARQ).

Designing HARQ for IoT presents new challenges. IoT devices send short packets and are often battery powered. Some IoT applications require high reliability and low delay. Type-II HARQ outperforms Type-I HARQ though the latter is simpler to implement.

Wi-Fi uses only ARQ though there have been proposals to introduce HARQ. HARQ is being researched in many other areas as well: 5G's URLLC and mMTC use cases, Cloud RAN, NOMA and Cognitive Radio.

## Milestones

1940

Van Dureen introduces the **first electronic retransmission** scheme to telegraphy. His method incorporates error detection via even parity. The term "automatic repeat request" itself appears in 1957. In 1961, Van Dureen publishes one of the earliest performance analysis of ARQ. By early 1960s, the term ARQ is widely accepted.

1960

At the Fourth London Symposium on Information Theory, Wozencraft and Horstein propose HARQ, although they don't use the term "HARQ". In a subsequent MIT Technical Report, they state, "The system is somewhat similar to human communication, in that typical errors are corrected, while grievous ones initiate a request for retransmission."

1974

Mandelbaum proposes **Incremental Redundancy** in which the first transmission contains a punctured codeword. If in error, each subsequent retransmission brings another increment of redundant bits until the receiver can successfully decode the packet.

1977

Sindhu proposes a a modification of ARQ in which erroneous packets are buffered at the receiver and later combined with retransmissions to improve the decoding. He calls this **ARQ-with-Memory (MRQ)** and the decoding procedure as Redundancy Repetition Error Correction (RRC). Essentially, this is packet combining or Type-I HARQ with hard-decision decoding.

1981

Lin and Yu propose the use of two block codes, \(C\_0\) for only error detection and \(C\_1\) for both error correction and detection. In their method, transmissions alternate between message bits and \(C\_0\) parity bits, and \(C\_1\) and \(C\_0\) parity bits. Wang and Lin improve this in 1983.

1984

In a survey paper on ARQ, Lin et al. name the main **types of HARQ**: Type-I and Type-II. This example shows that this sort of naming was current by the mid-1980s if not earlier.

1985

Chase proposes **code combining** in which multiple receptions are weighted and combined to offer better error correcting capability. The method does maximum-likelihood decoding and soft-decision packet decoding. The original packet uses a half-rate convolutional code but with every retransmission, the effective code rate drops.

2003

Cheng et al. propose **adaptive incremental redundancy** in which channel state information can be used to adapt the modulation and coding for each retransmission. They analyze this for a Release 5 3G/UMTS system and find that up to 25% improvement in system throughput can be obtained under certain conditions.

Sep
2005

3GPP publishes Release 6 of 3G/UMTS specification. In poor channels where ACK/NACK feedbacks are not coming through (such as for UEs at the cell edge), a variation called **autonomous or blind HARQ** is employed. The sender transmits multiple versions of the packet without waiting for ACK/NACK.

2015

Kuhn et al. consider many variants of TCP over 4G satellite links in their simulations. They find that in poor channel conditions, HARQ provides up to 22% higher throughput compared to other reliability schemes regardless of the TCP variant used.

2016

Sassioui et al. analyze the interplay between AMC and HARQ. They note that in fast-fading channels HARQ is beneficial only at low SNR and counterproductive at high SNR. In such cases, ARQ at a higher layer is sufficient. In slow-fading channels, throughput gains due to HARQ are moderate particularly when AMC ignores the presence of HARQ.

2021

Nadeem et al. study the effectiveness of HARQ for 5G's URLLC use case. Whereas we have large packets in eMBB use case, HARQ for URLLC has to be analyzed in the Finite Blocklength (FBL) regime. They present results for AWGN and Rayleigh fading channels.

Hybrid ARQ