# Secure Shell

## Summary

In computer networking, there's a need to log into a remote computer. Historically, this was done using *telnet* (since late 1960s) and *rlogin* (since 1980s). However, these early programs were not secure. **Secure Shell (SSH)** is a protocol created to address these deficiencies. It basically establishes a secure connection over an insecure network.

SSH adopts a client-server architecture. Client refers to the machine attempting to connect to a remote server. SSH authenticates both the client and the server. It specifies how client and server can agree on a shared secret for subsequent encryption of packets. Finally, it also provides data integrity protection.

SSH has two versions: SSH-1 and SSH-2. SSH-2 was standardized by the IETF.

SSH has wide adoption. Modern operating systems come with built-in SSH clients.  OpenSSH is a popular open source implementation. Commercial implementations are also available.

## Discussion

### What are the applications of SSH?

A common application of SSH is to **remote login**. In the early days of computer networking, trusted users within a corporate or university network had access to mainframes. For such trusted users, `telnet` was good enough. As networking expanded to WANs and the Internet, there was a need for a more secure protocol. This is what SSH provides. Once remotely logged into a computer, the user can execute programs via a terminal.

**Port forwarding** or **SSH tunnelling** is another useful application of SSH. TCP/IP traffic is encrypted and tunnelled through an SSH connection. This is also useful to bypass firewalls. Telnet, SMTP, NNTP, IMAP and other insecure protocols can be tunnelled through SSH. X forwarding is a special case of port forwarding.

SSH can be used to **transfer or copy files** via SFTP and SCP. SSH can automate server access and therefore enables process automation by cloud-based services. SSH keys can also offer Single Sign-On (SSO) so that users need not repeatedly type passwords.

### What are main features of SSH?

SSH has the following main features or purposes:

    + **Authentication**: Client can authenticate the server and vice versa. Algorithms include RSA and DSA (such as Ed25519 and Ed448).
    + **Key Exchange**: Client and server agree on what keys to use for encryption. Algorithms include Diffie-Hellman and RSA.
    + **Encryption**: SSH can encrypt it's payload, packet length, padding length and padding bytes. Algorithms include chacha-poly1305, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm, aes256-gcm, arcfour128 and arcfour256.
    + **Data Integrity**: By including a Message Authentication Code (MAC) field to each packet, the receiver can check that the packet hasn't been tampered with. MAC is calculated from a sequence number field (not transmitted) and unencrypted bytes. Algorithms include hmac-sha2-256 and hmac-sha2-512.
    + **Compression**: Payload going into an SSH packet can be optionally compressed. Algorithms include zlib.

### How does SSH differ from SSL?

Both SSH and SSL (Secure Sockets Layer) are about secure communications. However they differ in terms the sort of applications they enable. The main use of SSH is remote login and providing secure tunnels for insecure programs. System administrators or tech savvy users use SSH. SSL focuses on securing communications between a web browser and a web server. SSL powers websites and e-commerce.

Like SSH, SSL does authentication, data integrity checks and encryption. While SSH has many ways to authenticate clients, SSL authentication uses certificates. Though SSL offers both server and client authentication, the latter is impractical and rarely used.

For secure file transfer, SSH enables SFTP. SSL provides a similar service via File Transfer Protocol over SSL (FTPS).

### Which are the components of SSH?

SSH-2 includes three distinct protocol components:

    + **Transport Layer Protocol (SSH-TRANS)**: This is a low-level protocol by which client and server negotiate their support for SSH version and algorithms. Key exchange and server authentication are performed.
    + **Authentication Protocol (SSH-AUTH)**: This does client authentication. Client can attempt any authentication method supported by the server. The client may send "none" method to trigger the server to send a list of supported methods.
    + **Connection Protocol (SSH-CONN)**: From a single connection multiple logical channels are setup. For example, one channel might be an interactive user session while another might be an SSH tunnel for an insecure application.

### Could you describe the SSH protocol handshaking?

When a client contacts the server, a TCP connection is established. As the next step, client and server **negotiate** on two things based on their capabilities: (a) the version of SSH to use; (b) the algorithms to use.

The **Key Exchange (kex)** procedure follows. This includes explicit or implicit **server authentication**. This outputs a shared secret K and an exchange hash H. Encryption and authentication keys are derived from these. H from the first key exchange serves as the session identifier. RFC 4253 recommends that keys be re-exchanged every hour or after 1 GB of data transfer.

Next, the client makes a **service request**. The service request can be either SSH-AUTH, SSH-CONN or SSH-AUTH followed with SSH-CONN. With SSH-TRANS, a password could be used for **client authentication**. A more popular method is use pubic-private key pair, called key-based authentication. Another method is host-based authentication where server allows connections from specific hostnames. Once authenticated, client and server can establish a channel and exchange data.

### How is SSH-2 different from SSH-1?

SSH-1 is a monolithic protocol whereas SSH-2 is designed as three separate components: SSH-TRANS, SSH-AUTH and SSH-CONN. With this modular approach, SSH-2 improves on many aspects: algorithm negotiation, multiple methods of key exchange, certificates for public keys, more flexible authentication, stronger integrity checks, and periodic renewal of session key.

For client/server authentication, DSA works with only SSH-2 whereas RSA works with both versions of SSH. Mathematically, DSA is based on the discrete logarithm problem whereas RSA is base on the prime factorization problem. SSH-2 is extensible, that is, it can work with any public-key signature algorithm. For password-based authentication, SSH-2 allows users to change password.

SSH-2 doesn't support DES and IDEA encryption algorithms, either due to their weakness or licensing restrictions. SSH-2 adds SFTP to the SSH suite.

Unlike SSH-1, SSH-2 supports multiple interactive sessions. Many **channels** can be multiplexed over the same SSH connection. Thus, X or port forwarding can be done without opening separate terminal sessions for each.

As per RFC 4253, servers that support both versions are required to identify the `protoversion` as **SSH-1.99**.

### What tools are available for using SSH?

Two common tools that run on top of SSH are **SFTP** for secure file transfer and **SCP** for secure file copy. These are secure counterparts of earlier UNIX/Linux programs FTP and RCP. Popular implementations include OpenSSH, PuTTY and Tectia SSH. Built-in SSH clients are available on Windows, Linux and Mac. SFTP programs include WinSCP (Windows), gFTP (Linux) and Fetch (Mac).

SSH requires key generation and management. The tool `ssh-keygen` generates public-private key pairs. Once generated, the client's public key needs to be copied to the server. This is aided by the `ssh-copy-id` tool. On the server, the clients' public keys are stored at `~/.ssh/authorized_keys`.

When initiating an SSH connection, the user will be prompted for the passphrase. The tool `ssh-agent` stores the client's private key so that the user need not enter the passphrase for subsequent SSH connections. The tool `ssh-add` adds the key to the agent once the correct passphrase is supplied.

### What are some limitations of SSH?

SSH works well in low-latency environments. In high-latency environments such as 4G connections or satellite links, making an SSH connection or relaying user commands can be slow. Mobile Shell (Mosh) has come up as an alternative for these scenarios. It uses SSH for the initial connection but subsequently uses UDP rather than TCP.

SSH keys don't expire. SSH key mismanagement is an issue. Stolen keys, hardcoded root authorized keys, and host key copied from one host to another are all problems. The Sony breach of 2014 was achieved with stolen SSH keys.

Port forwarding occurs at the application layer, not at the network layer. Hence, it's not completely transparent to the application. Some configuration may be needed to get it working. For example, FTP data channels are hard to forward. For complete transparency, alternatives such as IPSec or VPN may be more suitable.

### How can I make SSH connections more secure?

There are a few things system administrators can do on the server side to make SSH more secure. Hackers may try brute-force password attacks. This can be avoided by disabling password-based authentication. Disable remote login for the root user. Be restrictive, that is, allow SSH only for some users. SSH by default uses port 22. Configure the server to use a non-default port.

When creating public-private key pair, use a strong passphrase. Key generation requires randomness. Use a hardware random number generator. Update the seed file after generating the SSH host key.

In large organizations, SSH key management solutions are needed. SSH Communications Security's Universal SSH Key Manager and Venafi's SSH Protect are examples. SSH keys should be at locations accessible by only the root user.

Though SSH sessions are encrypted, traffic analysis can still yield may clues as to what's happening. This could be mitigated by including random amount of padding.

### Which are the RFCs that specify SSH?

The Secure Shell (SECSH) Working Group of the IETF defined a number of documents for SSH. We note the following as essential reading:

    + **RFC 4250**: The Secure Shell (SSH) Protocol Assigned Numbers
    + **RFC 4251**: The Secure Shell (SSH) Protocol Architecture
    + **RFC 4252**: The Secure Shell (SSH) Authentication Protocol
    + **RFC 4253**: The Secure Shell (SSH) Transport Layer Protocol
    + **RFC 4254**: The Secure Shell (SSH) Connection ProtocolThe above documents refer to further RFCs that update them: 6668, 8268, 8308, 8332, 8709, 8758, 9141, 9142.

Two online sources have published useful lists of RFCs pertaining to SSH:  OmniSecu and OpenSSH.

## Milestones

Jul
1995

Ylönen, a researcher at the Helsinki University of Technology, discovers a password-sniffing attack. Plain-text communication protocols are inadequate and underscore the necessity of secure communication. To address these, Ylönen creates **Secure Shell (SSH)** with a focus on security and encryption. Ylönen releases the first version of SSH, named **SSH-1**. Internet Assigned Numbers Authority (IANA) assigns **port number 22** to SSH. By the year's end, about 20,000 users from 50 countries are using SSH. Given this popularity, in December 1995, Ylönen commercializes the technology under the name SSH Communications Security, Ltd.

Nov
1995

Ylonen publishes an IETF Internet-Draft that details SSH. This is the first formal description of the protocol. The draft expires in May 1996.

Sep
1996

SSH Communications Security, Ltd. releases **SSH-2**. SSH-1 had evolved in an ad hoc manner and had limitations and flaws. SSH-2 introduces more secure algorithms. It improves session identifier and key exchange. However, SSH-2 is incompatible with SSH-1. The last free version of SSH-1 is 1.2.12.

Feb
1997

The use of SSH-2 (as released by SSH Communications Security, Ltd.) has licensing restrictions. At the IETF, a new Working Group named Secure Shell, aka **SECSH**, is formed. It's mandate is to standardize SSH that can also be used openly. The first IETF Internet-Draft of SSH-2 comes out in February 1997.

Dec
1999

OpenBSD developers need a secure communication protocol. A few members work to modify and enhance *OSSH*, a fork of SSH created by Swedish programmer Groenvall. OpenBSD releases the first version of **OpenSSH**. In subsequent years, OpenSSH gains wide adoption, particularly in all POSIX-compliant operating systems.

Feb
2002

A vulnerability is discovered in the CBC mode block cipher. One suggested solution is to use more secure cipher modes. Systems could disable weak ciphers (3des-cbc, aes128-cbc, aes256-cbc, des-cbc), weak key exchange algorithms (diffie-hellman-group-exchange-sha1, diffie-hellman-group1-sha1) and weak MAC algorithms (hmac-md5, hmac-md5-96, hmac-sha1-96). For example, HP switches can be configured to block these weak options.

Jan
2006

IETF publishes the main RFCs that standardize SSH-2: 4250, 4251, 4252, 4253 and 4254. In October, the Working Group SECSH is closed.

Mar
2006

IETF publishes RFC 4432 that specifies **RSA Key Exchange**. IETF's SSH-2 uses Diffie-Hellman key exchange, which can take tens of seconds on slow devices. RSA-based key exchange is therefore proposed. The latter was supported in the original SSH-1 protocol.

Jul
2012

IETF publishes RFC 6668 that specifies the use of **SHA-2** for data integrity verification. Specifically, it defines hmac-sha2-256 (recommended) and hmac-sha2-512 (optional). This comes as a response to SHA-1 and MD5 algorithms that are now considered weak. In March 2018, RFC 8332 is published. This adds public key algorithms rsa-sha2-256 (recommended) and rsa-sha2-512 (optional).

Feb
2020

IETF publishes RFC 8709 that formalizes the use of **Ed25519 and Ed448 algorithms**. These are Digital Signature Algorithms (DSAs) used for authenticating clients and servers. In SSH, they're not used for encryption. OpenSSH 6.5 (released in 2014) introduced support for Ed25519. This RFC formalizes this and adds Ed448.

Apr
2020

IETF publishes RFC 8758 that **deprecates RC4**. RC4 encryption with its 128-bit key has known weaknesses. It's replacement are "arcfour128" (128-bit key) and "arcfour256" (256-bit key) that are specified in RFC 4345 (January 2006). In both algorithms, the first 1536 bytes generated by the cipher are discarded.

Jan
2022

IETF publishes **RFC 9142** that recommends which key exchange methods are safe and which ones should be avoided. Specifically, SHA-1 hash should be avoided. It also states, "A key exchange method is considered weak when the security strength is insufficient to match the symmetric cipher or the algorithm has been broken."

Secure Shell


# Natural Language Parsing

## Summary

Natural languages were designed by humans, for humans to communicate. They're not in a form that can be easily processed or understood by computers. Therefore, natural language parsing is really about finding the underlying **structure** given an input of text. In some sense, it's the opposite of templating, where you start with a structure and then fill in the data. With parsing, you figure out the structure from the data.

Natural languages follow certain rules of **grammar**. This helps the parser extract the structure. Formally, we can define parsing as,

> the process of determining whether a string of tokens can be generated by a grammar.

There are a number of parsing algorithms. Statistical methods have been popular since the 1980s. By the late 2010s, neural networks were being increasingly used.

## Discussion

### Could you introduce the basics of natural language parsing?

Text is a sequence of words. This can be simply represented as a "bag of words". This is not very useful. We can obtain a more useful representation by making use of syntactic structure. Such a structure exists because the sentence is expected to follow rules of grammar. By extracting and representing this structure, we transform the original plain input into something more useful for many downstream NLP tasks. Beyond syntax, parsing is also about obtaining the semantic structure.

In a typical flow, input text goes into a lexical analyzer that produces individual tokens. These tokens are the input to a parser, which produces the syntactic structure at the output. When this structure is graphically represented as a tree, it's called a **Parse Tree**. A parse tree can be simplified into an intermediate representation called **Abstract Syntax Tree (AST)**.

Structure can be represented either as a phrase structure tree or in labelled bracketed notation.

### What are the common difficulties in natural language parsing?

Breaking an input into sentences is the first challenge. Input could be formatted as tables or may contain page breaks. While punctuation is useful for this task, punctuation in abbreviations (e.g. or U.S.) can cause problems.

Given an input, a parser should be able to pick out the main phrases. This is not a solved problem. A more difficult problem is to obtain the correct semantic relationships and understand the context of discussion. Word embeddings such as word2vec operate at word level. This work needs to be extended to phrases.

Annotated corpora and neural network models are often about newswire content. Applying them to specific domains such as medicine is problematic.

Unlike parsing computer languages, parsing natural languages is more challenging because there's often ambiguity in human communication. A well-known example is "I shot an elephant in my pajamas." Was I or was the elephant wearing my pajamas? Humans also use sarcasm, colloquial phrases, idioms, and metaphors. They may also communicate with grammatical or spelling errors.

### Could you compare constituency parsing with dependency parsing?

Constituency parsing and dependency parsing are respectively based on Phrase Structure Grammar (PSG) and Dependency Grammar (DG). Dependency parsing in particular is known to be useful in many NLP applications.

PSG breaks a sentence into its constituents or phrases. These phrases are in turn broken into more phrases. Thus, the parse tree is recursive. On the other hand, DG is not recursive, implying that phrasal nodes are not produced. Rather, it identifies a network of relations. Two lexical items are asymmetrically related. One of them is the dependent word, the other is the head or governing word. Relations are labelled.

Consider the sentence, "She bought a car". In constituency parsing, "bought a car" is a verb phrase, which in turn contains noun phrase "a car". Thus, the structure is recursive. With dependency parsing, a directed arc identifies the relation along with a label. Typically, arrows go from head to dependent.

Languages such as Finnish that allow greater freedom of word order will benefit from dependency representation. This doesn't mean that English can't benefit from dependency representation.

### What is shallow parsing and how is it useful?

Constituency parsing is complex. Traditionally, such full parsing was not robust in noisy surroundings (although CoNLL 2007 Shared Task changed that). Some researchers therefore proposed **partial parsing** where completeness and depth of analysis were sacrificed for efficiency and reliability. **Chunking** or **shallow parsing** is a basic task in partial parsing.

Chunking breaks up a sentence into syntactic constituents called *chunks*. Thus, each chunk can be one or more adjacent tokens. Unlike full parsing, chunks are not further analyzed. Chunking is thus non-recursive and fast. Chunks alone can be useful for other NLP tasks such as named entity recognition, text mining or terminology discovery. Chunks can also be a useful input to a dependency parser.

POS tagging tags the words but doesn't bring out the syntactic structure. Chunking can be seen as being somewhere between POS tagging and full parsing. Chunking can include the POS tags. Perceptron, SVM and bidirectional MEMM are some algorithms used for chunking.

### What are the main approaches to text parsing?

Parsing is really a search problem. Search space of possible parse trees is defined by a grammar. An example grammar rule is "VP → VP NP". Broadly, there are two parsing strategies:

    + **Top Down**: Goal-driven. Starts from the root node and expands to the next level of nodes using the grammar. Checks for left-hand side match of grammar rules. Repeat this until we reach the POS tags at the leaves. Trees that don't match the input are removed.
    + **Bottom Up**: Data-driven. Starts from the input sequence of words and their POS tags. Builds the tree upwards using the grammar. Checks for right-hand side match of grammar rules.While bottom-up can waste time searching trees that don't lead to the root sentence, it's grounded on the input and therefore never suggests trees that don't match the input. These pros and cons are reversed with top-down.

Recursive descent parser is top-down. Shift-reduce parser is bottom-up. Recursive descent parser can't handle left-recursive production. Left-corner parser is a hybrid that solves this problem.

Borrowing from dynamic programming, **chart parsing** saves and reuses intermediate results. It also dynamically determines the order in which entries are processed.

### What are the some algorithms for text parsing?

State-of-the-art models in 2019 were based on neural networks. For both constituency parsing and dependency parsing, a combination of self-attention transformer architecture and Head-Driven Phrase Structure Grammar (HPSG) gave the best score. Earlier models used LSTM.

Before the use of neural networks, there were a number of algorithms: Stanford, Charniak, CJ, Bikel, Berkeley (for constituency); and Covington, Nivre, MSTParser, RelEx (for dependency). Most dependency parsers are much faster than constituency parsers. MaltParser implements Nivre and Covington algorithms. MSTParser implements Eisner and Edmonds algorithms.

Algorithms that adopt dynamic programming include Cocke-Kasami-Younger (CKY), Earley, and chart parsing. While CKY is bottom-up, Earley is top-down.

### How do text parsing algorithms disambiguate?

When there's ambiguity, we end up with multiple parse trees. This is where **probabilistic grammar** becomes useful, where rules have associated probabilities. The most probably parse tree is selected. The well-known CKY algorithm becomes probabilistic.

Given a treebank such as the Penn Treebank, it's easy to obtaining these probabilities. If we don't have any such treebank, we can use the Expectation-Maximization (EM) algorithm to iteratively arrive at these probabilities.

Another approach is to let the probabilistic model account for **lexicalized rules**. For example, a rule such as "VP → VBD NP PP" is changed to "VP(dumped,VBD) → VBD(dumped,VBD) NP(sacks,NNS) PP(into,NN)". It's difficult to obtain probabilities when we're so specific but there are techniques to do this. Two lexicalized parsers are Collins parser and Charniak parser.

### Could you mention some text parsers that I can use?

Commonly used Python NLP packages such as NLTK and spaCy have utilities for parsing. FreeLing and Apache OpenNLP support parsing. Berkeley Parser, Stanford Parser and Stanford CoreNLP support parsing. The Stanford Parser includes a shift-reduce constituent parser and a neural network dependency parser.

In R, *udpipe* is the package to use for dependency parsing.

Two open source tools from Google include SLING and SyntaxNet. Both of these are based on neural networks. SLING is trained for semantic frames of interest.

Although written for computer languages, Aho and Ullman's book (1972) is a classic reference that's applicable for natural language parsing as well.

## Milestones

1950

Noam Chomsky introduces **Phrase Structure Grammar (PSG)** in the 1950s. While mainly used for syntax, it has found applications in semantics and phonology.

1959

Historically, **Dependency Grammar (DG)** predates phrase structure grammar by many centuries. However, the modern history of dependency grammar starts with Frenchman Lucien Tesnière whose main work is published posthumously. Subsequently, DG advances through the 1960s.

1967

Patterns in the form of **regular expressions** can be used for parsing. While regular expressions date back to the 1950s, it's in 1967 that Ken Thompson at Bell Labs brings regex from the world of mathematics to computer science for the first time. In the 1970s, regular expressions make their way into some UNIX programs.

1992

**Penn Treebank** is released. It has 4.5 million words that are POS tagged. One-third of these are also annotated with skeletal syntactic bracketing. A treebank is a collection of text released with corresponding parse trees. Treebanks like this one provide annotated datasets for training statistical models.

1994

Pollard and Sag develop a highly lexicalized, constraint-based grammar called **Head-driven Phrase Structure Grammar (HPSG)**. In 2004, Miyao et al. acquire this grammar from the Penn Treebank. In 2019, Zhou and Zhao develop a HPSG parser that gives better predictions on both constituent and dependency tree structures.

1995

Based on Brill's earlier work (1992) on POS tagging using transformation-based learning, Ramshaw and Marcus adapt it to the problem of **chunking** by framing it as a tagging task. Tags are introduced to denote beginning (B), internal (I) and outside part (O) of a chunk. Later this approach is named **IOB tagging**.

2006

Dependency relations are useful in a number of NLP tasks. Existing dependency parsers are not robust or accurate as phrase structure parsers. The Penn Treebank, and other treebanks trained on large corpora, provide only phrase structure parses. As a solution to this problem, Stanford researchers come up with a system that automatically extracts typed dependency parses from phrase structure parses. In time, this becomes popular as the **Stanford Dependencies** and leads to **Universal Dependencies** that can be used for many natural languages.

May
2016

With the release of **SyntaxNet** from Google, what is claimed to be the world's most accurate English parser, **Parsey McParseface**, goes open source. This is built using machine learning. A neural network learns to resolve ambiguities. Multiple possible interpretations of structure are preserved via beam search rather than selecting the first best interpretation. On the Penn Treebank, this model achieves 94% accuracy, which compares well against human accuracy of 96-97%.

Natural Language Parsing


# Magic Methods in Python

## Summary

Magic methods are special methods that are called implicitly by the Python interpreter to execute corresponding operations. They help customize common operators or constructs. They make code more intuitive and Pythonic.

For example, to add two Python objects we might write `a + b`. The actual execution would call the magic method `a.__add__()` implicitly to perform this addition. If this magic method doesn't exist, the Python interpreter would call `b.__radd__()`. Application code where addition is invoked is itself simple and readable.

There are several conceptual groups of magic methods such as object lifecycle, object representation, object comparison, object attributes, class creation, callability, containers, numerics, context management, and more. This article describes many of these and shares how to use them.

## Discussion

### What are the characteristics of magic methods?

Magic methods are simply special methods. They have system-defined names **starting and ending with double underscores**, informally known as *dunder* names. Not all of these are magic methods.

Several built-in classes define various magic methods. Output of `dir()` can be filtered to see only dunder names but not all are magic methods. Further, `callable()` can be used to introspect these names, but not all callables are magic methods. Ultimately, Python documentation clarifies which attributes are magic methods. For example, as illustrated in the figure, `__doc__` isn't one and `__class__` despite being callable isn't one.

The code above shows implicit calls to `__str__()` and `__repr__()` magic methods by `str()` and `repr()` respectively. These return suitable string representations of the `Point2d` instances. Magic methods thus serve these purposes:

    + Enable classes to define their own behaviour corresponding to Python operators or constructs.
    + Enable developers to write Pythonic code. A developer could have written `to_string()` method but magic methods provide a uniform convention. This reduces technical debt.
    + Enable code reuse and design patterns. For example, custom objects can be compared and sorted based on magic methods without extra effort.

### Could you give an overview of the main magic methods?

Python 3.10.1 has dozens of magic methods. We discuss them within a few conceptual groups:

    + **Object Lifecycle**: These relate to creation, initialization and destruction of objects. Eg. `__new__()`, `__init__()`, `__del__()`.
    + **String Representation**: These represent objects as strings. Eg. `__repr__()`, `__str__()`.
    + **Comparisons**: To compare objects via operators `<`, `==`, `>=`, etc. Eg. `__lt__()`, `__eq__()`, `__ge__()`.
    + **Math Operations**: Arithmetic operators `+`, `*`; bit operators `&`, `<<`; and math functions. Eg. `__add__()`, `__rshift__()`, `__floor__()`.
    + **Type Conversions**: Convert from one type to another. Eg. `__bool__()`, `__int__()`.
    + **Type Check**: Check class hierarchy. Eg. `__instancecheck__()`, `__subclasscheck__()`.
    + **Context Manager**: To set up and clean up objects within a context, such as open/close of files. Eg. `__enter__()`, `__exit__()`.
    + **Attribute Access**: Control access or inspect instance attributes. Eg. `__getattr__()` , `__dir__()` , `__set__()` .
    + **Operations on Sequences**: Find length, get/set/delete items or iterate through items of a sequence. Eg. `__len__()`, `__getitem__()`, `__contains__()`.
    + **Asynchronous**: Pertaining to asynchronous operations. Eg. `__await__()`, `__aiter__()`, `__aenter__()`.
    + **Miscellaneous**: Not covered above. Eg. `__call__()`, `__mro_entries__()`.Some Python modules add their own magic methods. Examples include `copy.__deepcopy__()` and `pickle.__reduce__()`.

### Are magic methods Python's approach to operator overloading?

Indeed, magic methods are Python's approach to operator overloading.

However, there are many magic methods that don't correspond to an operator. For example, `__str__()`, `__bool__()` and `__repr__()` are magic methods corresponding to functions rather than operators.

Further, for example, the `object` class implements `__eq__()`, which corresponds to the `==` operator. When we implement this magic method we're technically overriding a base class method in OOP parlance.  However, we're actually overloading the `==` operator.

This is why it's better to think of magic methods as *intuitive interfaces*.

### What magic methods differ between Python 2 and Python 3?

For object comparisons, Python 2's magic method `__cmp__()` is replaced with *rich comparison methods* in Python 3: `__lt__()`, `__le__()`, `__eq__()`, `__ne__()`, `__gt__()`, `__ge__()`. Because it can be extra work to implement all these magic methods for a class, Python 3 provides the decorator `@functools.total_ordering`. We can provide implementations for `__eq__()` and one of the ordering operators. The decorator will supply the rest.

Python 2's magic method `__nonzero__()` is replaced with `__bool__()`. Method `__div__()` is replaced with `__truediv__()`. Method `__unicode__()` is removed since strings in Python 3 are Unicode by default.

Python 2's `__coerce__` is no longer needed since other magic methods do this in Python 3. For example, `2 + 3.5` would do type coercion in Python 2 but in Python 3, `__radd__()` on the float object is called to add the integer.

### Which magic methods are available for object lifecycle constructs?

`__new__()`, `__init__()` and `__del__()` are magic methods that allow customization of object creation, initialization and destruction respectively.

`__new__()` is mainly intended to allow subclasses of immutable types to customize instance creation and to customize class creation in custom metaclasses. `__init__()` is mainly intended for object customization before returning to the caller; that is, `__new__()` deals with creation and is a static class method while `__init__()` deals with initialization and is an instance method. `__del__()` is for runtime clean up just before an instance is destroyed, which only happens if the corresponding reference count becomes zero.

There are magic methods to customize class creation itself. Classes themselves are objects: a class is created from its metaclass. Classes themselves use memory, can be assigned, copied and passed as parameters, and even dynamically created. Magic method `__init_subclass__()` influences the behaviour of future subclasses as this method is called whenever the class is subclassed. Related magic methods include `__set_name__()`, `__prepare__()`, `__instancecheck__()`, `__subclasscheck__()`, and `__class_getitem__()`.

### Which magic methods are available for object representation?

Objects of type `str` are inherently strings but any Python object can be given a string representation. Built-in functions `str()`, `repr()`, `bytes()`, `format()`, and `print()` serve this purpose. However, their default behaviours can be customized by defining suitable magic methods.

Built-in function `repr()` calls magic method `__repr__()` to obtain the *official* string representation. Built-in functions `str()` and `print()` call magic method `__str__()` to obtain an *informal* and printable string. Since `__repr__()` is typically used for debugging, it should return an information-rich and unambiguous string. If possible, it should be a valid Python expression string for data serialization so that the instance can be recreated from the string.

Built-in function `bytes()` calls magic method `__bytes__()` to obtain a byte-string representation, which is primarily useful for storage and network transfers.

Built-in function `format()` and string method `str.format()` call magic method `__format__()` to evaluate formatted string literals, also known as *format-strings*. If this magic method is not defined, `__str__()` is called instead.

### Which magic methods are available for object comparison?

Comparison magic methods allow custom classes to implement specific comparison behaviours. Python defines six object comparison magic methods: `__lt__()`, `__le__()`, `__eq__()`, `__ne__()`, `__gt__()` and `__ge__()`, corresponding to `<`, `<=`, `==`, `!=`, `>`, and `>=` respectively.

The magic methods should return the singleton `NotImplemented` if the comparison is not implemented for the requested operands. They should return a boolean value. Otherwise, Python will call `bool()` on the returned object.

If `__ne__()` is undefined, its default behaviour is to call `__eq__()` and invert the result. No such relation exists among other comparison operators. For example, Python doesn't enforce that `x < y` is same as `y > x`. However, implementations should maintain semantic consistency.

### How can I customize object conversions?

Any Python object can be used in boolean context, such as `if obj: pass`. This operation is trivial for a `bool` type object. For other types, we need a way to evaluate the object in a boolean context. Magic method `__bool__()` does this.

Evaluating an object in a boolean context or directly calling the built-in function `bool()` calls `__bool__()`, which must return either `True` or `False`. Any other return value will cause Python to raise an exception. If `__bool__()` is undefined, `bool()` calls `__len__()`. The result is considered `True` if nonzero and `False` otherwise. If a class doesn't define either of these, all its instances are considered `True`.

Likewise, built-in functions `complex()`, `int()` and `float()` trigger other magic methods. A call of `int()` checks for `__int__()`, `__index__()` and `__trunc__()` in that order; `float()` checks for `__float__()` and `__index__()`; `complex()` checks for `__complex__()`, `__float__()` and `__index__()`.

When present, `__index__()` indicates that the object is of integer type. It must return an integer. Python slicing and functions `operator.index()`, `bin()`, `hex()` and `oct()` rely on this.

### Which magic methods emulate numeric objects?

Numbers can be added, multiplied or bit-shifted. What if we wish to emulate these behaviours on non-numeric objects? For example, we could add two RGB colour objects or left shift beads on an abacus object. Magic methods associated with numeric operators enable this.

There are three categories of magic methods (we describe only the addition operator):

    + **Left Operand**: Given `a + b`, the call `a.__add__(b)` is made.
    + **Right Operand**: Given `a + b`, the call `a.__add__(b)` is attempted first. If this doesn't exist or returns `NotImplemented`, `b.__radd__(a)` is called. For example, `3 + 5.5` will call `3.__add__(5.5)`. Since an integer doesn't know how to add a float, this returns `NotImplemented`. Then `5.5.__radd__(3)` is called and this succeeds with the addition.
    + **In-Place Operation**: Given `a += b`, the call `a.__iadd__(b)` is made. This method should modify the object in-place. If this method doesn't exist, `a.__add__(b)` and `b.__radd__(a)` are tried in that order.Related to these are unary operators `+` and `-` that call `__pos__()` and `__neg__()`; built-in function `abs()` calls `__abs__()`; similar built-in functions (some in `math` module) call `__round__()`, `__trunc__()`, `__floor__()`, and `__ceil__()`.

### Which magic method makes class instances callable?

Given a class `A` and its initializer `__init__(self, i)`, we can instantiate an object by doing `a = A(2)`. Now `a` is an instance. But it can't be called like a method or function. However, if the class `A` includes the magic method `__call__()`, we can write `a()`. Thus, the instance becomes callable (with or without arguments) when the magic method `__call__()` is defined.

Class instantiation itself is just calling the corresponding *class object*. With `__call__()`, we're enabling a similar syntax on instances.

Callable instances have practical use cases. With `__init__()` an object can be initialized in a fundamental way. Suppose it has only one specific behaviour. Hence it defines no methods other than `__call__()`. Different types of instances are then created and called. An example is initializing a `Hasher` object with an algorithm: `md5 = Hasher(hashlib.md5)` and `sha1 = Hasher(hashlib.sha1)`. Actual hashing happens later by calling these instances: `md5('somefile.txt')` or `sha1('somefile.txt')`.

Implementing decorators or presenting a consistent API are other use cases of `__call__()`.

### Which magic methods are available for customization of attribute access?

`__getattr__()`, `__getattribute__()`, `__setattr__()` and `__delattr__()` allow us to customize attribute access for class instances. This includes use of, assignment to or deletion of `a.x`, where `x` is an attribute of instance `a`.

Built-in function `dir()` calls magic method `__dir__()` to customize the list of valid instance attributes. Moreover, `__getattr__()` and `__dir__()` can be used to customize access to module attributes.

A feature of the language called **descriptor** introduces further magic methods: `__get__()`, `__set__()` and `__delete__()`. These are more powerful and present a more modular way to control attribute access.

### How can I make a custom object hashable?

Certain data structures in Python such as sets, frozensets and dictionaries are implemented using hash tables for efficient lookups and insertions. This requires computing a hash value on the object by calling the built-in function `hash()`. Internally, this calls the magic method `__hash__()`.

By defining this magic method on a class, we can compute the hash value for objects of that class. It must return an integer. Objects that compare equal via the method `__eq__()` must have the same hash value. One suggested way to compute a proper hash is to pack comparison-sensitive components of an object into a tuple and hash that tuple. For example, `return hash((self.x, self.y))` as the body of `__hash__()` method on a class `Point2d`. Note that `__hash__()` in general is tightly coupled with `__eq__()` and has atypical behaviour related to `__hash__()` implementations of `str` and `bytes` types.

## Milestones

Sep
2000

Python 1.6 is released. Magic method `__contains__()` is introduced to allow overriding the `in` operator.

Apr
2001

Python 2.1 is released. Magic methods `__lt__()`, `__le__()`, `__eq__()`, `__ne__()`, `__gt__()` and `__ge__()` are introduced to allow overriding the corresponding comparison operators.

Dec
2001

Python 2.2 is released. This introduces **descriptors** into the language via PEP 252. Descriptors themselves have the following attributes: `__name__`, `__doc__`, `__get__()`, `__set__()` and `__delete__()`. Descriptors are now used to support static methods and class methods. New features slots and properties are also new kinds of descriptors.

Sep
2006

Python 2.5 is released. This introduces the `with` statement and its associated magic methods `__enter__()` and `__exit__()`. A common use case of this is to open a file in `__enter__()` when the `with` context is entered and close the file in `__exit__()` as soon as execution comes out of the `with` context. Another new method is `__index__()` that solves a problem faced by NumPy users. It's now possible to use NumPy types as slice indexes.

Oct
2008

Python 2.6 is released. Some features of Python 3.0 are backported to Python 2.6. One of these is the `__complex__()` magic method. Magic method `__dir__()` is introduced. Built-in function `dir()` now looks for it on the object it receives.

Dec
2008

Python 3.0 is released. Magic method `__cmp__()` is removed. Instead use `__lt__()` for sorting, `__eq__()` with `__hash__()` and other rich comparisons. Operator `!=` returns the opposite of `==`. Methods `__getslice__()`, `__setslice__()` and `__delslice__()` are removed. The slice `a[i:j]` is implemented as `a.__getitem__(slice(i, j))`. Methods `__setitem__()` and `__delitem__()` are introduced. Converters `oct()` and `hex()` now use `__index__()`. Method `__nonzero__()` is now `__bool__()`.

Sep
2015

Python 3.5 is released. Magic methods for asynchronous programming are introduced: `__await__()`, `__aiter__()`, `__anext__()`, `__aenter__()`, and `__aexit__()`.

Jun
2018

Python 3.7 is released. Module-level methods introduced are `__getattr__()` and `__dir__()`. These can be used to implement module attribute deprecation and lazy loading. New methods `__class_getitem__()` and `__mro_entries__()` bring 7x improvement to performance in relation to types and the `typing` module. New module `dataclasses` has decorators and functions to automatically add some magic methods to user-defined classes.

Oct
2019

Python 3.8 is released. Constructors of `int`, `float` and `complex` will now use `__index__()` if their corresponding magic methods are not available.

Oct
2021

Python 3.10 is released. If `__ipow__()` returns `NotImplemented`, `__pow__()` and `__rpow__()` are the fallbacks.

Magic Methods in Python


# Java (Language)

## Summary

Java is a high-level programming language, having an object-oriented programming approach. It meets high standards of security. It was developed by James Gosling in 1995 at Sun Microsystems. One of the greatest strengths of Java is portability which allows the programmer to compile the code once and run on any platform.

Java has many real-world applications from Android gaming applications to scientific applications. It's versatility makes it one of the most popular and trusted languages in the world. The syntax of Java is very similar to C++ language. It's designed to let programmers *write once, run anywhere* .

## Discussion

### Why was Java created in the mid-1990s?

Java was developed by James Gosling at Sun Microsystems in 1995 to allow consumer electronic devices to communicate with each other. It is a high-level, portable and object-oriented programming language. It's designed to let programmers *write once, run anywhere* (WORA), meaning that the compiled Java code can run on all platforms that support Java without the need of recompilation. Gosling made the syntax of Java similar to C/C++ language so that the programmers would find it familiar.

### What exactly is a Java Virtual Machine?

The code written in Java language is converted into a specific code called **Java Bytecode** by a Java compiler. Then the bytecode is interpreted by a software called the Java Virtual Machine (JVM), or Java Runtime Environment (JRE).

The JVM acts a virtual computer which translates bytecode into machine instructions for the host computer. This helps in achieving one of the design goals of Java, called **portability**. Portability means that the program written for the Java platform must run similarly on any operating systems regardless of the hardware specifications. Users commonly use a Java Runtime Environment (JRE) installed on their device for running Java applications.

### What are the different editions of Java?

Sun Microsystems defined and supports four editions of Java for different application environments. Java APIs are categorized according to these environments. The different editions are:

    + **JavaFX**: A platform that provides a lightweight user-interface API. JavaFX applications can use hardware-accelerated graphics and media engines for high-performance clients.
    + **Java Micro Edition (Java ME)**: A computing platform for development and deployment of Java code for embedded and mobile devices such as mobile phones, micro-controllers and sensor nodes.
    + **Java Standard Edition (Java SE)**: A computing platform for development and deployment of Java code for desktop and servers.
    + **Java Enterprise Edition (Java EE)**: An extension of Java SE with additional features such as distributed computing and web services for enterprises.

### What sort of applications use Java in the real world?

Java has a very wide and extensive usage in the real world. Java is used for various purposes including Android apps, financial applications such as online trading systems, embedded systems, and many more. Java is used in to develop mobile applications, web applications, scientific applications, web servers, applications servers, server apps in the financial services industry, big data technologies, embedded systems, enterprise applications, trading applications, desktop GUI and business applications.

There are many significant applications like NASA WorldWind, Wikipedia Search, Minecraft, NetBeans, Eclipse IDE and IntelliJ IDEA that are all written in Java.

### What are the main features of the Java language?

The major features of Java are as follows:

    + Simple and Familiar: The syntax of Java is made similar to C/C++ language, eliminating the complexities of C/C++.
    + Compiled and Interpreted: Generally, a language is either compiled or interpreted but Java integrates the concepts of compiled and interpreted language.
    + Portable: Programs written for Java platforms run similarly on all operating systems regardless of system hardware components.
    + Architecturally Neutral: A Java program is independent of platforms or environments. It can run on any other operating system that has a compliant JVM.
    + Object-Oriented: Java strictly follows the concept of Object-Oriented Programming (OOP).
    + Robust: Java can handle runtime errors. It supports automatic garbage collection and exception handling.
    + Secure: Java has access modifiers to provide various levels of access to data.
    + Distributed: A Java program can access any other program running on any other machine.
    + Multi-Threaded: Multi-threading means executing multiple threads (portions) of a program in parallel.
    + High Performance: Java programs are highly efficient.
    + Dynamic and Extensible: Since Java is a object-oriented programming language, it gives freedom to add new classes and methods in our program.

### As a beginner, how can I get started with Java programming?

The Java can be installed on Windows, Linux or Mac. The official Java SE by Oracle can be installed.

Next step is to install an Integrated Development Environment (IDE). IntelliJ, Microsoft Visual Studio and Powerbuilder are a few examples of IDEs. The Java programs can be easily written and run on the IDE. For developing mobile apps using Java, Android Studio is the software to be installed.

Oracle provides beginner to advanced Java courses and certification. It provides many free courses. Oracle Java Training and Certification is a good starting point for more information.

### What are some limitations of using Java?

Although Java is one of the most used programming languages, it has some limitations:

    + Slow Performance: Bytecode interpretation by the JVM makes it slower than native languages such as C/C++.
    + Memory Consumption: Java programs require more memory compared to C/C++ programs.
    + Low-Level Programming: Java doesn't support low-level programming.
    + Unsigned Data types: Unlike C/C++, doesn't have support for unsigned data types. However, Java 8 APIs introduced some support for these.
    + Garbage Collection: Java doesn't give any control over garbage collection.

### What are some alternatives to Java?

Here are some alternatives to Java:

    + Python: Python is a high-level, object-oriented programming language. It's one of the easiest language to learn.
    + C#: C# has been competing with Java over the years. It's also an alternative to Java, having a similar syntax.
    + C++: The developers of Java considered many points of C++ while building Java. C++ is relatively faster than Java and also gives programmers some robust features.
    + Kotlin: Kotlin is the official language for Android development. It's an easy-to-code language compared to Java.
    + Swift: Swift is a faster programming language than Java. It's also a secure language.
    + Go: Go is a Google-supported language made for building fast, reliable, and efficient software. Paypal, Netflix and Microsoft use it.
    + Rust: Rust is a fast and memory efficient language used by many companies including Firefox, Dropbox and Cloudflare.

## Milestones

Jun
1991

James Gosling, Mike Sheridan, and Patrick Naughton start working on the Java language project. The project is named **Oak** after an oak tree standing outside Gosling's office. Later the name is changed to green and finally renamed as **Java** after a type of coffee in Indonesia.

Jan
1996

Sun Microsystems makes **Java 1.0** public, promising write once, run anywhere (WORA) functionality.

May
2007

Sun Microsystems makes all of the JVM's core code available under **free software and open-source** distribution terms.

Jan
2010

Oracle acquires Sun Microsystems. This doesn't change Java's open-source licensing and distribution.

Jul
2011

Java Development Kit (JDK) 7 is released. It contains networking and security enhancements. Support for **non-Java languages** is provided in this version.

Mar
2014

JDK 8 is released. A new language feature called **Lambda Expressions** is released. The new Modena theme is implemented in this release. Now, the *Java* command launches JavaFX applications. The Rhino JavaScript engine is replaced with the Nashorn JavaScript Engine.

Sep
2017

JDK 9 is released. The **module**, a new kind of Java programming component, is introduced. It's a named, self-describing collection of code and data. JDK and JRE runtime images are restructured to accommodate modules, and improve performance, security and maintainability.

Mar
2018

JDK 10 is released. A new method `orElseThrow()` is added to the `Optional` class. This is the preferred alternative of existing `get()` method. Several new APIs are added that facilitate the creation of unmodifiable collections.

Sep
2018

JDK 11 is released. It combines **Unicode 9.0.0 and 10.0.0** versions including 16,018 new characters, 18 new blocks, and 10 new scripts. **Nests** are introduced, an access-control context aligned with the existing notion of nested types in the Java programming language.

Mar
2019

JDK 12 is released. It includes support for **Unicode 11.0.0**. It includes 684 new characters, 11 new blocks, and 7 new scripts.

Sep
2021

JDK 17 is released. **Sealed Classes** are added to the Java Language. Sealed classes and interfaces restrict which other classes or interfaces may extend or implement them. An extension to switch function is implemented. The file system provider implementation on macOS is been updated in this release to support extended attributes. JavaDoc can now generate a page summarizing the recent changes in an API.

Java (Language)


# Border Gateway Protocol

## Summary

Border Gateway Protocol (BGP) is a routing protocol used to transmit routing information so that hosts or computers in one network can communicate with those in other networks anywhere on the internet. BGP is categorized as an external routing protocol since it deals with routing beyond an organization's internal network. BGP is standardized by the IETF.

BGP selects one best path based on a number of rules. BGP also ensures that paths are loop free. By adapting to route failures, BGP ensures network stability. When one path fails, a new path is quickly found. BGP uses TCP on port 179 as its transport protocol.

## Discussion

### Can you explain some commonly used terms in BGP?

A **BGP speaker** is a router that supports BGP. It communicates with other BGP speakers to share routing information. Routers that have established a mutual connection are called **peers**. .

**IP prefix** is a block of IP addresses allocated to an organization. A collection of routers in a network having a group of IP prefixes and sharing a common routing policy is called an **Autonomous System (AS)**. The internet connects many of these systems. The Internet Assigned Numbers Authority (IANA) assigns each AS with a globally unique Autonomous System Number (ASN).

There are broadly two types of routing protocols. An **Interior Gateway Protocol (IGP)** such as RIP, EIGRP, and OSPF does routing within an AS. An **Exterior Gateway Protocol (EGP)** such as BGP does inter-AS routing. However, BGP can also be used for intra-AS routing. For this reason, some use the terms Internal BGP (iBGP) and External BGP (eBGP) to distinguish between intra-AS and inter-AS usage of BGP.

### Why do we need BGP?

BGP is a routing mechanism that connects and binds the entire internet globally. It's in charge of determining the optimum path for a packet from its origin to its destination through the autonomous systems. BGP controls how data packets are distributed between the massive networks that make up the internet, allowing it to function properly.

It's a critical protocol since the internet would not function without it. The whole path to each destination is included in BGP routing information. BGP maintains a database of network reachability information, which it shares with other BGP systems, using the routing information.

### What are the characteristics of BGP?

BGP **employs TCP** as its transport protocol. Therefore BGP needn't implement functions already implemented at TCP, such as explicit update fragmentation, retransmission, acknowledgement, and sequencing. Since BGP doesn't have any built-in security mechanism, TCP Authentication Option (TCP-AO) was introduced. It improves the security and authenticity of TCP segments transmitted during BGP sessions. TCP-AO can handle both IPv4 and IPv6 traffic.

BGP is a **path vector protocol** that relies on dynamically updated path information to reach a destination. Path vector protocol keeps all paths **loop-free**.

BGP has a mandatory attribute `NEXT_HOP` for all the routes in a BGP table. This value is typically the address of the peer in another AS. For intra-AS routing, we can use OSPF; or use iBGP and update the attribute to point to a neighbouring router within the AS. The figure shows an example where packets from CE2 to 1.0.0.0/8 are routed via 200.0.0.11. Hence CE2 has the `NEXT_HOP` attribute set to 200.0.0.11 for destination 1.0.0.0/8.

### What are the message types in BGP?

A BGP message in BGP consists of header and data. All messages have fixed size header. BGP defines five message types:

    + **Open Messages**: After establishing a TCP connection by a three-way handshake, an open message is issued to establish a BGP connection. Subsequently, they can exchange other messages and data traffic.
    + **Update Messages**: By sending update messages, network reachability information such as route announcements and withdrawals can be shared. Thus BGP maintains a graph of connections.
    + **Keepalive Messages**: This is exchanged to keep the connection alive or to test connectivity. It's frequently exchanged to prevent the hold timer from expiring.
    + **Notification Messages**: When a problem with the BGP session is discovered, such as a hold timer expiring, changing neighbour capabilities, or a request for BGP session reset, a notification message is provided. The BGP connection is then terminated.
    + **Route-Refresh Messages**: It's requested by peers dynamically when a route advertiser needs to resend the update messages.

### How does BGP choose the best path?

BGP selects a best path between two peers. Given below is the list of criteria to select best path:

    + **Weight**: Choose the route which has most weight. Greater the weight value, higher the preference for that path.
    + **Local Preference**: If many routes have same weight, choose the one with highest local preference.
    + **Origin**: Prefer the route which was originated by local router. Next hop for a locally originated route is 0.0.0.0.
    + **AS-Path**: This is used in case two similar path is detected with same preference.
    + **Origin Code**: If AS-Path length is same prefer path with lowest MED (Multi-Exit Discriminator) which selects best path when there are multiple connections between autonomous systems(AS).
    + **eBGP over iBGP**: Prefer eBGP. eBGP's administrative distance is 20 whereas iBGP's is 200.
    + **IGP metric**: Lowest IGP for next-hop is preferred.
    + **External Paths**: If both routes are external select first and oldest.
    + **Router ID**: Select the path to router which has lowest router ID.
    + **Cluster list**: If multiple paths have same router ID select minimum length.
    + **Neighbour Address**: Select path from lowest neighbour address which is the IP address used in BGP neighbour configuration.

### What are the different BGP states?

Establishing a BGP session between speakers requiring a reliable transport protocol. This is provided by TCP. Thus a BGP session is established in two phases: TCP connection establishment phase and BGP session establishment phase. BGP maintains a Finite State Machine (FSM) per peer to track the operational status.

Three BGP states relate to establishing the TCP connection:

    + **IDLE**: Initial state of BGP before any connection happens.
    + **Connect**: BGP initiates TCP connection and waits for successful three-way handshake. If successful, it's transferred to open-sent state.
    + **Active**: Failed to establish a connection and a new three-way handshake is initiated.Three BGP states relate to establishing the BGP session:

    + **Open-Sent**: Open message is sent from the originating router to the peer.
    + **OpenConfirm**: Open message is received from the peer. Wait for Keepalive message.
    + **Established**: Connection is success. Peers can now exchange routes by sending Update message.

### What are the limitations of BGP?

The difficulty with BGP is that it doesn't explicitly incorporate built-in security and relies on network operators to effectively safeguard their systems. BGP hijacking and leak incidents have been a persistent source of concern since the early 2000s.

One of the major challenges that BGP faces is the expansion of the *routing table*. This issue arises when the routing table grows to the point where certain older, less capable routers are unable to meet the requirements for routing table maintenance, resulting in issues such as *512k day*.

BGP is vulnerable to *prefix hijacking*, which is the deliberate generation of incorrect routing information. The attacker announces routes to disrupt a service on an IP space or hijack traffic to sniff any confidential information. The reasons for this are numerous and difficult to comprehend. Prefix hijacking will also have the same effects as *route leaks* caused by any router misconfiguration.

## Milestones

Sep
1982

**Gateway-to-Gateway Protocol (GGP)**, a first experimental internet gateway is implemented by Bolt, Beranek and Newman (BBN) for use in the US department of Defense for a project named ARPANET. This is the early days of the internet. . The gateway forwards datagrams between networks because it contains a dynamic routing table with an entry for every network that may be reached. The closest gateway is determined by the "number of hops" necessary. A gateway has zero hops when connected directly to the network. The number of hops increases as the number of gateways increases.

Apr
1984

DARPA Internet is growing continuously requiring more gateways making GGP less feasible as it requires constant updates. As a result, the **Exterior Gateway Protocol (EGP)** is implemented through RFC904. The EGP includes autonomous systems, each with a unique identifier. It can transmit traffic from one autonomous system to another, making internet flat and uniform. . The protocol uses Hello/I-Heard-You (I-H-U) message exchanges to poll the reachability of neighbouring autonomous systems on a regular basis. The EGP was actually discussed conceptually in 1982.

Jun
1989

Kirk Lougheed and Len Bosack of Cisco, and Yakov Rekhter of IBM write a new protocol with experience gained on EGP. EGP has problems with regional networks providing false information. Intra-regional routing is also isolated from the latest NSFNET, the backbone of internet. They write this new protocol on napkins. For this reason, it's sometimes called the *Two-Napkin Protoco*. In **RFC 1105**, these ideas becomes the Border Gateway Protocol.

Jun
1990

Border Gateway Protocol (BGP) gets a major update with the publication of **RFC 1163** and **RFC 1164**. It has resolved several issues that arose with the first version of BGP described in RFC 1105. Several Messages types and their applications are redefined. The concept of path attributes is introduced to communicate information about traffic routes. In addition, directional topology in routers that can be up, down or horizontal is removed and replaced with arbitrary AS topology.

Oct
1991

**RFC 1267: BGP-3** is published with several improvements and corrections. If two BGP speakers attempt to make a TCP connection to each other at the same time, two parallel connections may be formed. This is referred to as a *connection collision*. A new field BGP identifier is introduced now and been added to message type OPEN to detect and recover from a connection collision. Also, information exchange between previously reachable routes is optimized and simplified.

Mar
1995

**RFC 1771: BGP-4** is published as a Draft Standard based on last year's Proposed Standard **RFC 1654: BGP-4**.  Major concept of Classless Inter-domain Routing (CIDR) is introduced with the support of advertising IP Prefix to the reachable destination. This eliminates the need for network classes. For storage and bandwidth efficiency, it also helps to aggregate route update messages received from several different routes to be advertised as single routing table entry. The handling of connection collision becomes more sophisticated in this upgrade.

Feb
1998

With new RFC 2283, multiprotocol extensions with support for IPv6, IPX, and other network layer protocols are added to BGP-4, which previously only supported IPv4. This BGP-4 expansion is also known as **Multiprotocol BGP (MBGP)** or Multicast BGP. To facilitate this and offer backward compatibility, two new attributes are introduced: Multiprotocol Reachable NLRI (MP REACH NLRI) and Multiprotocol Unreachable NLRI (MP UNREACH NLRI).

Jan
2006

**RFC 4271: BGP-4** is published, thus obsoleting RFC 1771 that was published more than a decade earlier.
 Among many technical modifications, this update clarifies the use of the BGP identification in the AGGREGATOR attribute, the various types of NEXT HOPs, and the use of the ATOMIC AGGREGATE attribute.

Feb
2008

To block YouTube access in Pakistan as a result of government order a Pakistan Telecom (AS17557) begins to advertise to its provider PCCW (AS 3491) a small part of 208.65.153.0/24 prefix owned by YouTube (AS36561). Without verifying ownership, PCCW propagates this wrong route. This results in requests to YouTube coming to Pakistan Telecom's network. When YouTube starts announcing the same prefix, BGP selects Pakistan Telecom because it has a shorter path.

Dec
2012

As the two-octet encoding for autonomous system numbers approaches its limit, **RFC 6793** is introduced with several clarifications and editorial changes obsoleting RFC 4893 (May 2007) to support **four-octet ASNs**. In addition, a BGP capability code with two new attributes AS4 PATH and AS4 AGGREGATOR is introduced to support this. These new attributes are introduced to disseminate this four-octet-based information to BGP speakers who don't support the new feature.

Aug
2014

On 12th August, an Internet Service Provider (ISP) called "Verizon" pushes 15,000 new routes into BGP tables. This unexpectedly surpasses the **maximum of 512,000 routes** that BGP routers can actually hold. BGP routing tables are stored in TCAM (Tertiary Content Addressable RAM), which reached its memory limit on this fateful day. A remedy for this anticipated problem was recommended, and notifications were provided in May 2014. However, most ISPs failed to make the necessary modifications.

Nov
2018

A group of criminal hackers known as *3ve* (pronounced "eve") hacks 1 million IP addresses from reputable organizations such as the US Air Force, as well as from residential and business users in North America and Europe. Using botnets they hijack BGP for an endless supply of highly valuable IP addresses. Hackers generate $29 million via ad fraud involving bots placing bids on counterfeited domains.

Oct
2021

Facebook and its affiliated services such as WhatsApp and Instagram become unavailable for nearly six hours. This is due to an incorrect internal configuration. They stop announcing BGP routes to their DNS Prefix around 15:58UTC. BGP UPDATE messages begin flooding in from Facebook, causing all routes to it to be withdrawn. DNS servers go offline, cutting it off from the internet. As a result, all DNS resolvers stopped resolving their domain names by responding SERVFAIL.

Border Gateway Protocol


# N-Gram Model

## Summary

Given a sequence of N-1 words, an N-gram model predicts the most probable word that might follow this sequence. It's a probabilistic model that's trained on a corpus of text. Such a model is useful in many NLP applications including speech recognition, machine translation and predictive text input.

An N-gram model is built by counting how often word sequences occur in corpus text and then estimating the probabilities. Since a simple N-gram model has limitations, improvements are often made via smoothing, interpolation and backoff.

An N-gram model is one type of a **Language Model (LM)**, which is about finding the probability distribution over word sequences.

## Discussion

### Could you explain N-gram models with an example?

Consider two sentences: "There was heavy rain" vs. "There was heavy flood". From experience, we know that the former sentence sounds better. An N-gram model will tell us that "heavy rain" occurs much more often than "heavy flood" in the training corpus. Thus, the first sentence is more probable and will be selected by the model.

A model that simply relies on how often a word occurs without looking at previous words is called **unigram**. If a model considers only the previous word to predict the current word, then it's called **bigram**. If two previous words are considered, then it's a **trigram** model.

An n-gram model for the above example would calculate the following probability:

P('There was heavy rain') = P('There', 'was', 'heavy', 'rain') = P('There')P('was'|'There')P('heavy'|'There was')P('rain'|'There was heavy')

Since it's impractical to calculate these conditional probabilities, using *Markov assumption*, we approximate this to a bigram model:

P('There was heavy rain') ~ P('There')P('was'|'There')P('heavy'|'was')P('rain'|'heavy')

### What are typical applications of N-gram models?

In speech recognition, input may be noisy and this can lead to wrong speech-to-text conversions. N-gram models can correct this based on their knowledge of the probabilities. Likewise, N-gram models are used in machine translation to produce more natural sentences in the target language.

When correcting for spelling errors, sometimes dictionary lookups will not help. For example, in the phrase "in about fifteen mineuts" the word 'minuets' is a valid dictionary word but it's incorrect in this context. N-gram models can correct such errors.

N-gram models are usually at word level. It's also been used at character level to do stemming, that is, separate the root word from the suffix. By looking at N-gram statistics, we could also classify languages or differentiate between US and UK spellings. For example, 'sz' is common in Czech; 'gb' and 'kp' are common in Igbo.

In general, many NLP applications benefit from N-gram models including part-of-speech tagging, natural language generation, word similarity, sentiment extraction and predictive text input.

### How do we evaluate an N-gram model?

The best way to evaluate a model is to check how well it predicts in end-to-end application testing. This approach is called **extrinsic evaluation** but it's time consuming and expensive. An alternative approach is to define a suitable metric and evaluate independent of the application. This is called **intrinsic evaluation**. This doesn't guarantee application performance but it's a quick first step to check algorithmic performance.

A common metric is to use **perplexity**, often written as PP. Given a test set \(W = w\_1 w\_2 \dots w\_n\), \(PP(W) = P(w\_1 w\_2 \dots w\_n)^{-1/N}\). Because of the inverse relationship with probability, minimizing perplexity implies maximizing the test set probability. Perplexity can also be related to the concept of entropy in information theory.

It's important in any N-gram model to include markers at start and end of sentences. This ensures that the total probability of the whole language sums to one. However, all calculations should include the end markers but not the start markers in the count of word tokens.

### What is smoothing in the context of N-gram modelling?

It's quite possible that some word sequences occur in test data that were never seen during training. When this happens, the probability of the sequence equals zero. Evaluation is also difficult since perplexity metric becomes infinite.

The usual way to solve this is to give non-zero counts to N-grams that are seen in testing but not in training. We can't just add 1 to all the zero counts since the overall probability distribution will not be normalized. Instead, we remove some probability mass from non-zero counts (called **discounting**) and add them to the zero counts. The overall process is called **smoothing**.

The simplest technique is **Laplace Smoothing** where we add 1 to all counts including non-zero counts. An alternative is to add k, with k tuned using test data. Other techniques include Good-Turing Discounting, Witten-Bell Discounting, and Kneser-Ney Smoothing. All of these try to estimate the count of things never seen based on count of things seen once. **Kneser-Ney Smoothing** provides a good baseline and it's based on **absolute discounting**.

### How do backoff and interpolation techniques help N-gram models?

Smoothing solves the zero-count problem but there are techniques to help us better estimate the probabilities of unseen n-gram sequences.

Suppose we want to get trigram probability of a certain word sequence that never occurs. We can estimate this using the bigram probability. If the latter is also not possible, we use unigram probability. This technique is called **backoff**. One such technique that's popular is called **Katz Backoff**.

**Interpolation** is another technique in which we can estimate an n-gram probability based on a linear combination of all lower-order probabilities. For instance, a 4-gram probability can be estimated using a combination of trigram, bigram and unigram probabilities. The weights in which these are combined can also be estimated by reserving some part of the corpus for this purpose.

While backoff considers each lower order one at a time, interpolation considers all the lower order probabilities together.

### What are some limitations of N-gram models?

A model trained on the works of Shakespeare will not give good predictions when applied to another genre. We need to therefore ensure that the training corpus looks similar to the test corpus.

There's also the problem of **Out of Vocabulary (OOV)** words. These are words that appear during testing but not in training. One way to solve this is to start with a fixed vocabulary and convert OOV words in training to `UNK` pseudo-word.

In one study, when applied to sentiment analysis, a bigram model outperformed a unigram model but the number of features doubled. Thus, scaling N-gram models to larger datasets or moving to a higher N needs good feature selection techniques.

N-gram models poorly capture longer-distance context. It's been shown that after 6-grams, performance gains are limited. Other language models such cache LM, topic-based LM and latent semantic indexing do better.

### What software tools are available to do N-gram modelling?

R has a few useful packages including *ngram*, *tm*, *tau* and *RWeka*. Package *tidytext* has functions to do N-gram analysis.

In Python, NTLK has the function `nltk.utils.ngrams()`. A more comprehensive package is nltk.lm. Outside NLTK, the *ngram* package can compute n-gram string similarity.

Written in C++ and open sourced, SRILM is a useful toolkit for building language models. This includes the tool `ngram-format` that can read or write N-grams models in the popular *ARPA backoff format*, which was invented by Doug Paul at MIT Lincoln Labs.

A demo of an N-gram predictive model implemented in R Shiny can be tried out online.

Ngram Viewer is a useful research tool by Google. It's based on material collected for Google Books. Given a sequence of words, it shows how the N-gram counts have changed over the years.

## Milestones

Jul
1948

In his paper titled *A Mathematical Theory of Communication*, Claude Shannon describes an example in which the next letter depends on the previous one based on defined probabilities. This is an application of Markov process to natural languages.

1980

Although smoothing techniques can be traced back to Lidstone (1920), or even earlier to Laplace (18th century), an early application of smoothing to n-gram models for NLP is by Jelinek and Mercer (1980). A better smoothing technique is due to Katz (1987). More smoothing techniques are proposed in the 1990s.

1991

Jelinek and team at the IBM Research Division adapt a trigram LM to match the current document better. By caching a recent history of words, they propose **Cache Trigram Language Model (CTLM)**. They show that for long text of 500-800 words, there's a drop in error rate by about 24%.

1993

N-gram models look at the preceding (n-1) words but for larger n, there's a data sparsity problem. Huang et al. propose a **skipping n-gram model** in which some preceding words may be ignored or skipped. For example, in the phrase "Show John a good time", the last word would be predicted based on P(time|Show \_\_ a good) rather than P(time|Show John a good). Many such skipping models are proposed through the 1990s.

1995

Kneser and Ney propose improved backoff and discounting. It's based on the concept of **absolute discounting** in which a small constant is removed from all non-zero counts. **Kneser-Ney Smoothing** improves on absolute discounting by estimating the count of a word in a new context based on the number of different contexts in which the word has already appeared.

1998

Chen and Goodman at Harvard University give an empirical comparison of different smoothing techniques. Other papers from them around the late 1990s become influential in this area of research. Due to their work, **Interpolated Kneser-Ney** has become a popular language model.

Jul
1999

Version 0.99 of the **SRILM toolkit** is open sourced for public use. Earlier versions of SRILM can be traced back to 1995. Version 1.00 is released in June 2000. Version 1.7.3 comes out in September 2019.

2002

Researchers at Carnegie Mellon University apply N-grams to model whole-genome protein sequences. Their research triggers further application of N-gram models to computational biology.

Sep
2006

For the benefit of research in linguistics, Google releases a dataset of counts of about a billion five-word sequences that appear at least 40 times from a text of trillion words. The dataset becomes available via the Linguistic Data Consortium (LDC).

N-Gram Model


# Standard Library

## Summary

Every programming language defines a basic set of components that can be reused by programs. Every implementation of the language should provide these components. This set of reusable components is what we call a **standard library**.

A good standard library is essential to the long-term success of a programming language. However, there's no consensus on what or how much should be included in a language's standard library. Python and Java have large standard libraries while C takes a minimalistic approach.

The definition of a standard library is not limited to languages. It can be extended, for example, to frameworks. Thus, we can say that Django (web framework) and Robot Framework (test automation framework) have their own standard libraries that are installed by default with the framework.

## Discussion

### What should a standard library typically include?

Since the C standard library is minimal, it gives us an idea of what's typically included. These are either macro constants or functions. In an object-oriented language such as C++, classes, class templates and methods are part of the standard library. Others (Java, PHP) may additionally include interfaces.

Operations that are deemed necessary for most programs are included. A standard library will have functions to access the file system and perform input/output operations; functions that deal with the data types of the language; functions that allow memory allocation and manipulation; functions that enable multithreaded programming; functions to handle date and time; common math functions; functions for error handling and assertions.

Modern languages have lot more than what C standard library offers. C++ has classes for containers, algorithms and regular expressions. Python has packages for data persistence, data compression, cryptography, exception handling, networking, CSV/XML parsing, GUI, testing, profiling, and more.

### What are some advantages of using a standard library?

Developers can focus on their application logic rather than low-level details, which are provided by the standard library. Code can be reused across projects. Code is more maintainable since most developers will be familiar with the standard library. There's also a shorter learning curve for new developers joining a project.

Since implementations of the standard library are optimized and well tested, programs that use them are likely to be more reliable. If the language is open source, developers can also study the implementation of the standard library to learn best practices.

Projects that use the standard library might even be smaller since developers are not introducing custom code for some modules while using the standard library in others.

### How are standard libraries delivered or distributed?

A standard library is not simply one physical document but rather a combination of many things:

    + **Standards document**: This is the main document released with every version of the language. This will include core parts of the language plus what's in its standard libraries. For example, C and C++ standards documents are released by ISO and must be purchased. In other languages, these may be more readily available online.
    + **Header files**: This is typical of C and C++. Header files specify what's available in the standard libraries.
    + **Implementations**: When a language compiler or interpreter is installed on a machine, an implementation of the standard library is also installed. In Python, users have a choice of different implementations of the standard library, with CPython being the default.
    + **Delivery**: In C and C++, implementations are object files. In Python, they are packages containing modules. In Java, we have packages containing classes. In client-side JavaScript, implementation is built into the web browser.Documentation and header files may also be termed as the Application Programming Interface (API) of the language. It's also common for vendors to include useful non-standard functions at the expense of portability.

### Who should read standard libraries?

There are a few types of folks who need to read them: compiler writers, those who implement standard libraries, tool vendors, and users of the standard libraries.

Standards documents are usually hard to read because they are formal, legal and highly detailed. They are meant for compiler writers and those who implement standard libraries. Tool vendors may also read them. For everyone else, it's usually sufficient to read the documentation on how to use the standard libraries in their code. Tutorials and example code showing the use of standard libraries are more useful than reading the language standards.

### What are some considerations when defining a standard library?

What a standard library provides should fulfil the needs of programmers. In 2016, when initiating a revision of C standard, convener David Keaton pointed out that security threats on the Internet are growing, Moore's Law is failing and there's a need to exploit concurrency and parallelism in programs. Languages must therefore evolve to address these new concerns.

Standard library must balance stability and evolution. Design should address safety, security, simplicity, interoperability, demanding application domains, and embedded systems. To add a feature, we must consider how it would interact with other features. Updates to the standard library must be backward compatible to earlier versions. While there could be some platform-dependent features, portability should be improved. Sometimes portability may be sacrificed for performance, at least in C.

One point of view is that most standard libraries lack features and they are incompatible across languages. What if you could write a library in one language and automatically translate it into any desired language? **Loyc Multi-Language Standard Library (MLSL)** aims to do this.

### Should a standard library be exhaustive or minimal?

C and C++ are minimal. Java provides many higher level utilities out of the box. While this benefits developers, it also bloats the language. Some APIs become antiquated and less useful, though they would have to be maintained unless deprecated. Python also adopts the "batteries included" approach and provides many sophisticated packages as part of the standard library. While this is useful, it also means that new implementations have more work to do. A small library implies that multiple implementations are possible and all programmers can rely on it.

There's been some debate about moving C++ towards a "batteries included" approach, such as adding a graphics library. Alternatively, C++ can be kept minimal while finding a better way to distribute libraries and dependencies.

The choice that language designers make must be governed by the its main purpose. Should it a general purpose language or satisfy a niche? It's been said,

> No language can be everything for everybody.

### Could you list some examples of standard libraries?

Here's a short and non-exhaustive list of some standard libraries:

    + C: assert, float, math, stdio, stdlib, string, threads
    + C++: list, map, deque, vector, algorithm, iterator, string, regex, fstream, iostream, exception
    + Python: list, dict, tuple, range, open, enum, collections, math, pickle, zlib, threading, queue, email, json
    + Go: tar, gzip, heap, crypto, hash, image, io, mime, net, os, path, strings, time, unicode
    + Java: java.lang, java.io, java.net, java.math, java.text, java.awt.image, java.security, java.beans

### What happens if commonly required functions are not part of a standard library?

There's nothing in C standard library to help a developer send emails from a C program. The developer is forced to build this feature from low-level networking constructs. Moreover, that code can't be reused by others since it's not part of the standard library.

This is where the power of the community becomes important. Useful functions that are not part of the standard library are curated, reviewed and shared via public repositories. In Java, there's Apache Commons. In Python, there's PyPI. For C++, Boost provides peer-review libraries, some of which were later accepted into the standard library. For C, there's POSIX standard library that enables portability across different variants of UNIX/Linux while adding many functions missing in C standard library.

C++ Boost binaries can be downloaded and installed for your platform. Modern languages implement the concept of package/dependency management. Software is gathered into centralized repositories and tools exist to manage packages and version dependencies as necessary for your project. Example repositories include Packagist (PHP), PyPI (Python), CPAN (perl), NPM (Node.js), and Maven (Java).

### Are there well-defined processes to evolve the standard libraries?

Languages that are proprietary (such as MATLAB or Wolfram) will have closed processes. Customers of these products may request features. Otherwise, they have little control on how the language or its standard library evolves. On the other hand, more open languages are standardized by a process that allows greater community participation.

Both C and C++ are standardized by the International Organization for Standardization (ISO). Meetings are organized regularly to discuss changes to the language. Participants are organized into working groups. Proposals are submitted and refined to produce working drafts. Voting by members will promote these drafts to becoming part of the standard library. Those who wish to make proposals, should study the Standard Library Guidelines.

In the case of Python, the Python Software Foundation (PSF) does regular releases. Any modification to the standard library is done via a Python Enhancement Proposal (PEP). Proposals are discussed and documented in draft stages. Once finalized, they're implemented.

Java has the Java Community Process (JCP) and proposals are managed as Java Specification Requests (JSRs). PHP has an RFC process with clear guidelines on how to participate.

## Milestones

1989

The **ANSI C** standard library is standardized to overcome compatibility problems across variations and implementations of the C language. Effort towards this important milestone started back in 1983 when the American National Standards Institute (ANSI) formed a committee. This standard is also called **C89**. Normative Addendum 1 (NA1) is added to it in 1995.

Sep
1999

With `integer` library as its first entry, an experimental release of **Boost** is introduced for C++.

Apr
2012

As ISO/IEC 30170:2012, **Ruby** is published as an ISO standard.

2017

ISO standardizes **C++17** for the C++ language. C++ was first standardized in 1998. The next planned evolution is named **C++20**, to be released in 2020. The C++ Standard Library was previously called Standard Template Library (STL).

Standard Library