# Python for Scientific Computing

## Summary

Fortran has been the language of choice for many decades for scientific computing because of speed. In the 1980s, when a programmer's time was becoming more valuable than compute time, there was a need for languages that were easier to learn and use. For the purpose of research, code-compile-execute workflow gave way to interact-explore-visualize workflow. In this context were born MATLAB, IDL, Mathematica and Maple.

Modern scientific computing is not just about numerical computing. It needs to be versatile: deal with large datasets, offer richer data structures than just numerical arrays, make network calls, interface with databases, interwork with web apps, handle data in various formats, enable team collaboration, enable easy documentation.

Python offers all of the above. It's been said that,

> Python provides a balance of clarity and flexibility without sacrificing performance.

## Discussion

### What makes Python a suitable language for scientific computing?

Python is not just suited for manipulating numbers. It offers a "computational ecosystem" that can fulfil the needs of a modern scientist. Python does well in system integration, in gluing together many different parts contributed by different folks. Python's duck typing is one of the reasons why this is possible. In terms of data types, `memoryview`, `PyCapsule` and NumPy's `array` aid scientific work.

Python is easy to learn and use. It offers a natural syntax. This enables researchers to express and explore their ideas more directly rather than fight with low-level language syntax.

With Python, performance bottlenecks can be optimized at a low-level without sacrificing high-level usability. A researcher needs to explore and visualize ideas in an incremental manner. Python allows for this via IPython/Jupyter notebooks and matplotlib. A variety of Python tools can work together and share data within the same runtime environment without having to exchange data only via the filesystem.

### I'm used to MATLAB. Why should I use Python?

MATLAB is proprietary, expensive and hard to extend. Python is open, community-driven, portable, powerful and extensible. Python is also better with strings, namespaces, classes and GUIs. While MATLAB, along with Simulink, has vast libraries, Python is catching up as many scientific projects are adopting Python. MATLAB is said to be poor at scalability, complex data structures, memory handling, system tasks and database programming.

However, there are some criticisms of Python (December 2013). Syntax is not consistent since different packages are written by different folks with different needs. There are two ordinary differential equation (ODE) solvers in scipy with incompatible syntax. Duplicated functionality across packages may result in confusion. MATLAB does better with data regression, boundary value problems and partial differential equations (PDE).

In 2014, Konrad Hinsen commented that Python may not be suitable for small-scale projects where code is written once and rarely maintained thereafter. This becomes a problem when Python scientific libraries are upgraded by deprecating older classes/functions/methods.

### Should I worry about performance when using Python for scientific research?

The short answer is no. There are many initiatives that aim to make Python faster. PyPy and Pyston do just-in-time (JIT) compilation for better performance. Nuitka aims to replace the Python runtime to automatically transpile code to languages that run fast natively. Numba speeds up math-heavy Python code to native machine instructions with just a few annotations on your Python code.

While pure Python code is definitely slower when compared to Fortran or C, scientific packages in Python often make use of low-level implementations that are themselves written in Fortran, C, etc. For example, NumPy operations often call BLAS or LAPACK functions that are written in Fortran.

f2py is enabling Python to directly use Fortran implementations. SWIG and Cython allow us to make calls to optimized C/C++ implementations from within Python. For example, Cython is being used by scikit-learn. Intel Math Kernel Library (MKL) and PyCUDA are also bringing Python on par with Fortran on specific hardware platforms.

### What are the essential packages for scientific computing in Python?

Here are some packages that could be considered essential:

    + **numpy**: Multi-dimensional arrays and operations on them. Executes faster than Python.
    + **scipy**: Linear algebra, interpolation, integration, FFT...
    + **matplotlib**: Plotting and data visualization with an API similar to MATLAB. Eases transition from MATLAB to Python.
    + **spyder**: An IDE that includes IPython (for interactive computing).
    + **pandas**: Numerical data structures, data manipulation and analysis.
    + **jupyter**: Web-based sharing of code, graphs, annotations and results.Most Python scientific packages are based on numpy and scipy. If visualization is involved, matplotlib may be used. For higher-level data structures, pandas may be used. Spyder, IPython and Jupyter are simply useful tools for the scientist or engineer.

### Could you name some useful scientific projects/packages in Python?

Here are some that can be applied to any domain:

    + **Image processing**: Pillow, OpenCV, scikit-learn , Mahotas
    + **Visualization**: matplotlib, bokeh, plotly, mayavi, seaborn , basemap, NetworkX
    + **Markov chains**: Markov, MarkovNetwork, PyMarkov, pyEMMA, hmmus
    + **Stochastic process**: stochastic, StochPy, sdeint
    + **Solving PDEs**: FEniCS, SfePy
    + **Convex Optimization**: CVXPY
    + **Units and conversions**: quantities, pint
    + **Multi-precision math**: mpmath, GmPy
    + **Spatial analysis**: cartopy, georasters, PySAL, geopy
    + **Data access**: pydap, cubes, Blaze, bottleneck, pytables
    + **Machine learning**: scikit-learn, Mlpy, TensorFlow, Theano, Caffe, Keras
    + **Natural Language Processing**: NLTK, TextBlob, spaCy, gensim, pycorenlp
    + **Statistics**: statistics, statsmodels, patsy
    + **Tomography**: TomoPy
    + **Symbolic computing**: sympy
    + **Simulations**: SimPy, BlockCanvas, railgun

### Could you name some domain-specific scientific projects/packages in Python?

Since there are dozens of packages for all types of scientific work, we can only give a sample:

    + **Solar data analysis**: SunPy
    + **Astronomy**: astropy
    + **Chemistry**: thermo, chemlab
    + **Biology**: biopython
    + **Neurosciences**: PsychoPy, NIPY
    + **Life sciences**: DEAP
    + **Network analysis**: NetworkX
    + **Quantum dynamics**: QuTiP
    + **Protein analysis**: ProDy
    + **Neuron simulations**: NEURON
    + **Seismology**: ObsPy
    + **Phylogenetic computing**: DendroPy
    + **Software defined radio**: GNU Radio

### What's the recommended Python distribution for scientific computing?

Installation of Python for scientific work used to be a pain earlier but with modern distributions, this is no longer an issue.

Rather than install Python's standard distribution and then install scientific packages one by one, the recommended approach is to use an alternative distribution customized for scientific computing: Enthought Canopy, Anaconda, Python(x,y) or WinPython. Enthought Canopy is commercial but the rest are free. Enthought Canopy claims to include 450+ tested scientific and analytic packages.

Anaconda distribution uses *conda* for package management. The use of virtual environments is recommended so that different projects can use their own specific environments. Powered by Anaconda, Intel offers its own distribution that's optimized for performance.

SageMath is another distribution that offers a web-based interface and uses Jupyter notebooks. It aims to be the free open source alternative to Magma, Maple, Mathematica and Matlab.

### As a beginner in scientific Python, what should be my learning path?

From tools and environment perspectives, get familiar with using IPython, Jupyter Notebook and optionally Spyder.

After learning the basics of Python, the next step is to learn numpy since it's the base for many scientific packages. With numpy, you can work with matrices and do vectorized operations without having to write explicit loops. You should learn about operations such as reshaping, transposing, filling, copying, concatenating, flattening, broadcasting, filtering and sorting.

You could then learn scipy to do optimization, linear algebra, integration, and so on. For visualization, matplotlib can be a starting point. For dealing with higher-level data structures and manipulation, learn pandas.

If you wish get into data science, scikit-learn and Theano can be starting points. For statistical modelling, you can learn statsmodels.

### What useful developer resources are available for scientific computing in Python?

One good place to start learning is the SciPy Lecture Notes.

SciPy Conference is an annual event for Python's scientific community. It also happens in Europe as EuroSciPy and in India as SciPy India.

EarthPy is a collection of IPython notebooks for learning how to apply Python to Earth sciences.

The Hacker Within, Software Carpentry and Data Carpentry are some communities that bring together research and scientific folks. Although these are not exclusive to Python, Python programmers will find them useful.

## Milestones

1995

*Numeric* is released to enable numerical computations. This is the ancestor of today's *NumPy*.

2001

Many scientific modules are brought together and released as a single package named *SciPy*. The same year, IPython is born.

2002

"Python for Scientific Computing Workshop" is organized at Caltech. In 2004, this is renamed as *SciPy Conference* and is now an annual event. In 2008, EuroSciPy is held for the first time. In 2009, 1st SciPy India is held.

2005

NumPy is released based on an older library named *Numeric*. It also combines features of another library named *Numarray*. NumPy is initially named *SciPy Core* but renamed to *NumPy* in January 2006.

Jul
2017

*Anaconda Accelerate* is split into *Intel Distribution for Python* and open source Numba's sub-projects pyculib, pyculib\_sorting and data\_profiler.

Python for Scientific Computing


# Systems Programming

## Summary

Systems are built from hardware and software components. Systems programming is about implementing these components, their interfaces and the overall architecture. Individual components perform their prescribed functions and at the same time work together to form a stable and efficient system.

Systems programming is distinct from application programming. System programs provide services to other software. Via abstractions, they expose API to simplify the development of applications. They're often optimized to low-level machine architecture. Unlike application software, most system software are not directly used by end users.

Assembly and C language have been historically used for systems programming.  Go, Rust, Swift and WebAssembly  are newer languages suited for systems programming.

## Discussion

### How is systems programming different from application programming?

Operating systems, device drivers, BIOS and other firmware, compilers, debuggers, web servers, database management systems, communication protocols and networking utilities are examples of system software.   As part of operating systems, memory management, scheduling, event handling and many more essential functions are done by system software. Examples of application software are Microsoft Office, web browsers, games, and graphics software.

Application software generally don't directly access hardware or manage low-level resources. They do so via calls to system software. We may thus view system software as aiding application software with low-level access and management.  Application developers can therefore focus on the business logic of their applications.

While application developers may not build system software, they can become better developers by knowing more about the design and implementation of system software. Specifically, by knowing and using system APIs correctly they can avoid implementing similar functions in their applications.

For example, a C application on UNIX/Linux calls the system API `getpid()`. The OS creates the process, allocates memory and assigns a process identifier.

### What are the main concerns of systems programming?

An operating system such as Linux is a collection of system programs. These deal with files and directories, manage processes for executable programs, enable I/O for those programs and allocate/release memory as needed. The OS manages users, groups and associated permissions. The OS prevents normal user programs from executing privileged operations. The shell is a special system program that allows users to interact with the system. If processes need to communicate or respond to external events, signals (aka software interrupts) facilitate this.

There are systems programs that transform other programs into machine-level instructions for execution: compilers, assemblers, macro processors, loaders and linkers. Their aim is to generate instructions optimized for speed or memory. Use of registers, loops, data structures, and algorithms are considered in these system programs.

Programming languages, editors and debuggers are also system programs. These are tools to write good and reliable system programs. They have to be easy to learn and productive for a developer while also being efficient and safe from a system perspective.

### Is a system programmer same as a system administrator?

A system programmer is one who write system software and this task is called system programming or systems programming. But there's an older definition that's been used in the context of mainframes since the 1960s.

On a mainframe, a system programmer installs, upgrades, configures and maintains the operating system. She does capacity planning and evaluates new products. She's also skilled in optimizing the system for performance, troubleshooting problems and analyzing memory dumps.

On the other hand, a system administrator handles day-to-day operations such as adding/removing users, configuring access, installing software, and monitoring performance. She deals with applications whereas a system programmer is more well-versed with the mainframe hardware.

In small IT organizations, both roles may be performed by a single individual.

### What languages are suited to systems programming?

Wikipedia lists more than a dozen languages: C/C++, Ada, D, Nim, Go, Rust, Swift, and more. Notably, the following popular languages are absent: JavaScript, Java, C#, Python, Visual Basic, PHP, Perl, and Ruby. Implemented in Haskell, COGENT is a functional programming language that's also suited for systems programming.

System programming languages are required to provide direct access to hardware including memory and I/O access. Performance, explicit memory management and fine-grained control at bit level are essential capabilities. Where they also offer high-level programming constructs, system programming languages (C, Rust, Swift, etc.) are also used for application programming.

Since such languages give access to low-level hardware functions, there's a risk of introducing bugs. Rust was created to balance aspects of both safety and control. C sacrifices safety for control while Java does the opposite.

Scripting languages such as Python, JavaScript and Lua are not for systems programming. However, the introduction of static typing (for safety) and Just-in-Time (JIT) compilation (for speed) has seen these languages being used for systems programming.

### What's systems programming in the context of the web?

On the web, applications adopt the client-server architecture. Server-side logic may be implemented as many microservices distributed on a cloud platform. Applications make use of APIs served by different endpoints. In this context, we have systems programs that help create distributed applications for the cloud. System programs must be designed to address network topology changes, security concerns, high latency, and poor network connections.

Rob Pike commented that developers thought that Go was for systems programming. In fact, it was for writing any server-side code. Later he saw anything running on the cloud as systems software. Ousterhout commented in 1998 that on the web Java was being used for systems programming.

Bunardzic commented that "the only way to program a system is to program a network". Systems should be designed for concurrency, fault encapsulation/detection/recovery, upgrade without downtime, observability, and asynchronous communication. Developers who use system services and APIs must be free to choose their own technology stack. One service shouldn't depend on others.

### What are the best practices in systems programming?

A systems programmer needs to know the system APIs well. In addition, she should know how the OS kernel, programs and users interact with one another.

Designing a good system is not a one-time task. The system should be designed for iteration and incremental improvements. System designers must be open to feedback. Historically, many Linux system programmers blamed application developers for program crashes instead of seeing these as opportunities to improve the Linux kernel or system tools.

All assumptions must be made explicit. Systems software should get the abstractions right, minimize if not avoid leaky abstractions. In other words, its users shouldn't need to know implementation details. For instance, ORM frameworks that interface between applications and databases are often leaky due to a conceptual mismatch between objects and relations.

Before implementation, it's beneficial to do system modelling, analysis and simulations. Unified Modelling Language (UML) can help. Small and simple systems are amenable to formal analysis. Anything else, we need to apply statistical analysis. More components there are, more complex becomes the system.

## Milestones

1960

Till early and even mid-1960s, the first concern in designing computer systems is the hardware itself. Programming them becomes a secondary concern. Programming techniques are chaotic. Often they're not as intended by the hardware designers. Systems programming as a discipline is only starting to emerge.

1966

Shaw considers assemblers, interpreters, compilers, and monitors as **translators**; that is, they translate code in one form to another. Referring to the figure, translator T translates A to B. He defines the following,

> Systems Programming is the science of designing and constructing translators.

Oct
1968

At the NATO Conference on Software Engineering, the merits of **high-level languages** are discussed: cost, maintainability, correctness. System programmers however object to high-level languages. They don't like anything to get in between themselves and the machine. Reconciling these two concerns is the main challenge in design a suitable systems programming language.

1969

**Unix** operating system is invented at Bell Laboratories, first in assembly on PDP-7 (1969) and subsequently migrated to C on PDP-11 (1971). A decade later one of its inventors, Dennis Ritchie, comments that only the assembler is still written in assembler. Most other system programs are written in C language.

1971

Researchers at the Carnegie-Mellon University invent a new systems programming language that they name **BLISS**. They describe it as a general purpose high-level language for writing large software systems for specific machines. They enumerate the requirements of any good systems programming language: space/time economy, access to hardware features, data structures, control structures, understandability, debugging, etc. The figure shows an example of how BLISS enables bit-level access. Such low-level access is typical of systems software.

1984

Weicker proposes **Dhrystone as a benchmark** for systems programming. Currently we have the Whetstone benchmark that's tuned to measure floating-point arithmetic. Systems programs often use enumerations, records and pointer data types. Compared to numerical programs, systems programs have fewer loops, simpler compute statements, more conditional branching and more procedure calls. The Dhrystone benchmark accounts for these.

1986

An IBM research report details the challenges and pitfalls in programming for a multi-CPU and multi-threaded environment. They refer to the System/370 architecture. Shared memory and data structures have to be handled carefully. Therefore, **parallelism** is a new concern for systems programming. However, multi-programming was considered in the design of Unix back in the early 1970s.

1990

This decade sees the wide adoption of **scripting languages** such as Perl, Tcl, Ruby, PHP, Python and JavaScript. They're seen as languages that "glue together" various components whereas systems programming languages are used to create those components. In the 2010s, the boundaries blur as some scripting languages are used to build large systems software.

1992

Wolfe proposes changes to how systems programming must be taught to students. The curriculum would include small challenges in various aspects of systems programming and not just focus on assemblers and compilers. Currently, students perceive systems programming as boring, the techniques themselves stale, and the courses disconnected from systems programming jobs out there.

2005

Brewer et al. note that 35 years after the invention of C language, we still don't have a suitable alternative for systems programming. Shapiro voices a similar complaint in 2006. Advances in programming languages have not motivated systems programmers to adopt them.

2009

Mozilla begins sponsoring the development of **Rust**. Rust is announced to the world in 2010. The first stable release of the language happens in May 2015. Today Rust is well-recognized as a systems programming language.

2019

Crichton at Stanford University proposes that students of programming languages must be taught systems programming. Courses must not be about theory and formal methods alone. They should include practical programming while also providing effective mental models of computation and formal reasoning. He includes WebAssembly and Rust in the proposed curriculum.

Systems Programming


# MongoDB Query Language

## Summary

Whereas relational databases are queried using Structured Query Language (SQL), MongoDB can be queried using MongoDB Query Language (MQL). It's the interface by which clients can interact with the MongoDB server. Developers and database administrators can write MQL commands interactively on the MongoDB Shell. For client applications, drivers are available in popular programming languages to execute MQL commands.

MQL supports CRUD operations. Results can be sorted, grouped, filtered, and counted via aggregation pipelines.  Special operations such as text search and geospatial queries are possible. Multi-document transactions are supported.

This article covers MongoDB Shell commands except aggregations. Commands in Node.js are similar. For commands in other languages, refer to the documentation of language-specific drivers. This article doesn't cover administrative commands that relate to users, backups, replica sets, sharded clusters, etc.

## Discussion

### What are the basic commands to get started with MQL?

Start by launching the MongoDB Shell `mongosh`. Type `help()` to see a list of **shell commands**. Here are some useful commands: `show databases`, `show collections` (of current database), `use <db-name>` (to switch to a database), `version()` (for shell version), `quit` and `exit`.

The `Database` class has many methods. View them with the command `db.help()`. Here are some useful methods: `getMongo()` (get current connection), `getCollectionNames()`, `createCollection()`, `dropDatabase()`, and `stats()`.

The `Collection` class has many methods. View them with the command `db.coll.help()`. Here are some useful methods: `insert()`, `find()`, `update()`, `deleteOne()`, `aggregate()`, `count()`, `distinct()`, `remove()`, `createIndex()`, `dropIndex()`, and `stats()`.

The command syntax is same as JavaScript. Commands can be saved into a JavaScript file and executed using the `load()` command.

### Could you introduce the commands for CRUD operations in MongoDB?

Database operations are conveniently referred to as *CRUD (Create, Read, Update, Delete)*. MongoDB supports CRUD with the following methods of `Collection` class:

    + **Create**: Add documents to a collection. Methods include `insertOne()` and `insertMany()`. If the collection doesn't exist, it will be created. Unique `_id` field is automatically added to each document if not specified in the method calls.
    + **Read**: Retrieve documents from a collection. Main method is `find()`, called with query criteria (how to match documents) and projection (what fields to retrieve).
    + **Update**: Modify existing documents of a collection. Methods include `updateOne()`, `updateMany()` and `replaceOne()`. Method calls include query criteria and what to update.
    + **Delete**: Remove documents from a collection. Methods include `deleteOne()` (first matching document is deleted) and `deleteMany()`, called with query criteria.It's also possible to insert new documents with the option `upsert: true`. Some methods that do this are `update()`, `updateOne()`, `updateMany()`, `findAndModify()`, `findOneAndUpdate()`, `findOneAndReplace()`, and `bulkWrite()`.

### How do I find documents in a MongoDB collection?

The main method is `find(query, projection)`. Query parameter specifies the filter. If omitted, all documents returned. Projection parameter specifies what fields to retrieve. If omitted, all fields are retrieved. The method returns a cursor to the documents. Method `findOne(query, projection)` returns the first matching document, not a cursor.

Here are some sample commands on a collection named `people`:

    + `db.people.find()`: Retrieve all fields of all documents.
    + `db.people.find({status: "A", {age: {$gt: 25}}})`: Retrieve all fields of documents of status 'A' and age exceeding 25. Multiple query fields are combined with `$and` operator by default.
    + `db.people.find({}, {user_id: 1, status: 1, _id: 0})`: Retrieve only two fields of all documents.
    + `db.people.find({$or: [{status: "A"}, {age: {$gt: 25, $lte: 50}}] })`: A complex filter specifying status 'A' or age within a range.
    + `db.people.find({status: {$ne: "A"}).sort({age: 1}).skip(10).limit(5)`: Filter, sort, skip and limit the results.Comparison operators include `$eq`, `$gt`, `$gte`, `$in`, `$lt`, `$lt`, `$lte`, `$ne`, and `$nin`. Logical operators include `$and`, `$not`, `$nor` and `$or`.

Since MongoDB is schema-less, documents may include only some fields or fields of differing types. Operators `$exists` and `$type` check if field exists and if the type matches.

### How do I write queries when arrays are involved?

Consider an inventory collection with two array fields `tags` and `dim_cm`. Here are some example queries:

    + `db.inventory.find({ tags: ["red", "blank"] })`: Documents with both 'red' and 'blank' tags. Order of tags matters.
    + `db.inventory.find({ tags: { $all: ["red", "blank"] } })`: As above, but order of tags doesn't matter.
    + `db.inventory.find({ tags: "red" })`: Documents tagged 'red'.
    + `db.inventory.find({ dim_cm: { $gt: 15, $lt: 20 } })`: Documents where both conditions are satisfied but not necessarily by the same array item.
    + `db.inventory.find({ dim_cm: { $elemMatch: { $gt: 15, $lt: 20 } } })`: Documents where both conditions are satisfied by the same array item.
    + `db.inventory.find({ "dim_cm.1": { $gt: 25 } })`: Second item of array satisfies the condition.
    + `db.inventory.find({ tags: { $size: 3 } })`: Documents with exactly three tags.Operators `$in` and `$nin` check presence or absence in an array. For example, `db.inventory.find({dim_cm: {$in: [15, 18, 20]}})` finds documents with specific dimensions.

### How do I find documents based on values in nested documents?

It's possible to find documents based on the content of inner documents, either with a full or a partial match. Assume documents with the field `size` that itself is a document with fields `h`, `w` and `uom`. Here are some examples:

    + `db.inventory.find({ size: { h: 14, w: 21, uom: "cm" } })`: Exact document match. Field order matters.
    + `db.inventory.find({ "size.h": { $lt: 15 } })`: Nested field query.Assume documents with the field `instock` that's an array of documents. Here are some examples:

    + `db.inventory.find({ "instock": { warehouse: "A", qty: 5 } })`: At least one item in array matches the query. Field order matters.
    + `db.inventory.find({ 'instock.qty': { $lte: 20 } })`: Nested field query. At least one item in array matches the query.
    + `db.inventory.find({ 'instock.0.qty': { $lte: 20 } })`: Check the first item in array.
    + `db.inventory.find({ "instock.qty": 5, "instock.warehouse": "A" })`: Two nested fields but they need not match the same item within the array.
    + `db.inventory.find({ "instock": { $elemMatch: { qty: 5, warehouse: "A" } } })`: Two nested fields and both should match at least one item within the array.

### Could you share more details about MongoDB projection?

The projection parameter is used in `find()` and `findOne()` methods. It controls what fields are sent by MongoDB to the application. It contains field-value pairs. Values can be boolean (1 or true, 0 or false), array, meta expression or aggregation expression.

The `_id` field is always included in returned documents unless explicitly suppressed with `_id: 0`.

Here are some examples of projections:

    + `db.inventory.find({status: "A"}, {item: 1, _id: 0})`: Retrieve only `item`. Suppress `_id`.
    + `db.inventory.find({}, {status: 0, instock: 0})`: Retrieve all fields except `status` and `instock`.
    + `db.inventory.find({}, {"size.uom": 1})`: Given `size` is a nested document, retrieve `_id` and nested field `size.uom`. An alternative form is `db.inventory.find({}, {size: {uom: 1}})`.
    + `db.inventory.find({}, {_id: 0, "instock.qty": 1})`: Given `instock` is an array of documents, retrieve only the `instock.qty` field.
    + `db.inventory.find({}, {instock: {$slice: -2}})`: Retrieve only the last two items of `instock` and all other fields of inventory.
    + `db.inventory.find({}, {instock: {$slice: [3, 5]}})`: Retrieve five items after skipping first three items of `instock`. Include all other fields of inventory since no explicit inclusion is specified.
    + `db.inventory.find({}, {total: {$sum: "$instock.qty"}})`: Using aggregation expression in projection, get total stock quantity in each document.

### What are cursor methods in MongoDB?

Cursor methods modify the execution of the underlying query. Since the collection method `find()` returns a cursor, we can list all cursor methods on the shell by calling `db.coll.find().help()`.

Here are a few examples, some showing how to chain method calls:

    + `db.products.find().count()`: Return number of documents rather than the documents themselves.
    + `db.products.find().limit(2)`: Limit results to first two documents.
    + `db.products.find().min({price: 2}).max({price: 5}).hint({price: 1})`: Query only documents within the specified index range. Index `{price: 1}` must exist.
    + `db.products.find().pretty()`: Show results in a user-friendly format.
    + `db.products.find().sort({price: 1, qty: -1})`: Sort by price and then reverse sort by quantity.
    + `db.people.find().skip(3).limit(5)`: Skip first three documents and show only the next five.

### What are the update operators in MongoDB?

There are three groups of update operators:

    + **Field Update**: Operators include `$currentDate` (Date or Timestamp), `$inc` (increment), `$min` (updates only if it's less than current field value), `$max` (updates only if it exceeds current field value), `$mul`, `$rename`, `$set`, `$setOnInsert` (only if update results in an insert), and `$unset`.
    + **Array Update**: Operators include `$`, `$[]`, `$[<identifier>]`, `$addToSet`, `$pop` (-1 to remove first item, 1 to remove last item), `$pull` (removes items based on a query condition), `$push` (append to array), and `$pullAll` (removes items based on a list of values). Operator modifiers include `$each`, `$position`, `$slice` and `$sort`.
    + **Bitwise Update**: The only operator is `$bit`. It supports AND, OR and XOR updates of integers.

### As a SQL developer, how do I get started with MongoDB Query Language?

SQL's `(tables, rows, columns)` map to MongoDB's `(collections, documents, fields)`. SQL's `GROUP BY` aggregations are implemented via aggregation pipelines in MongoDB. Many other features such as primary keys and transactions are equivalent though not the same.

In SQL, we can drop or add columns to a table. Since MongoDB is schema-less, these operations are not important. However, it's possible to add or drop fields using the method `updateMany()`.

SQL developers can refer to these useful guides:

    + SQL to MongoDB Mapping Chart
    + SQL to Aggregation Mapping Chart
    + Logan's Gist on SQL to MongoDB Mapping Chart

## Milestones

Feb
2009

**MongoDB 1.0** is released. By August, this version becomes generally available for production environments.

Mar
2010

MongoDB 1.4 is released. Among the improvements are `$all` with regex, `$not`, `$` operator for updating arrays, `$addToSet`, `$unset`, `$pull` with object matching, and `$set` with array indexes.

Aug
2012

MongoDB 2.2 is released. Projection operator `$elemMatch` is introduced. It returns only the first matching element in an array. For bulk inserts, it's now possible to pass an array of documents to `insert()` in the shell.

Mar
2015

MongoDB 3.0 is released. This includes a new **query introspection system** towards query planning and execution. Command `explain` and methods `cursor.method()` and `db.collection.explain()` are relevant here.

Dec
2015

MongoDB 3.2 is released. Query operators to test bit values are introduced: `$bitsAllSet`, `$bitsAllClear`, `$bitsAnySet`, `$bitsAnyClear`. Many CRUD methods on Collection class are added to correspond to drivers' APIs: `bulkWrite()`, `deleteMany()`, `findOneAndUpdate()`, `insertOne()`, and many more. **Query modifiers** are deprecated in the shell. Instead, **cursor methods** should be used.

Nov
2016

MongoDB 3.4 is released. Type `decimal` for 128-bit decimal is introduced. To support language-specific rules for string comparison, **collation** is introduced.

Nov
2017

MongoDB 3.6 is released. With `arrayFilters` parameter, we can selectively update elements of an array field. Positional operators `$[]` and `$[<identifier>]` allow multi-element array updates. For `$push` operator, `$position` modifier can be negative to indicate position from end of an array. Deprecated `$pushAll` operator is removed. New query operators include `$jsonSchema` and `$expr`. MongoDB shell now supports sessions.

Aug
2019

MongoDB 4.2 is released. We can now use the **aggregation pipeline for updates**. **Wildcard indexes** support queries against fields whose names are unknown or arbitrary. Some commands (`group`, `eval`, `copydb`, etc.) are removed.

Jul
2020

MongoDB 4.4 is released. Projections in `find()` and `findAndModify()` are made consistent with `$project` aggregation stage. Method `sort()` uses the same sorting algorithm as `$sort` aggregation stage. Compound hashed indexes and hidden indexes are introduced.

Jul
2021

MongoDB 5.0 is released. It provides better support for field names that are `$` prefixed or include `.` characters.

MongoDB Query Language


# Sorting Algorithms

## Summary

Sorting is defined as the rearrangement of the given data in a particular order. This order can be related to numerical values i.e. ascending or descending order, alphabetical i.e. case sensitive or insensitive and can be based on length of the string or on different coding styles like ascii or unicode.

Sorting is an extra step that is carried out to increase the efficiency of the operation that is being performed. For example: sorting a data structure like an array in advance may speed up a search operation. Also, it is necessary for some algorithms to function, for example, binary search works only on sorted data.

We have a variety of sorting algorithms ranging from bubble sort, selection sort, merge sort, quick sort, counting sort, radix sort, etc. Each one of them has its own set of weaknesses and strengths, based on which they can be employed by the developers.

## Discussion

### Why do we need sorting?

In school, the children of a class are sorted in the increasing order of height while standing in a queue for the morning assembly. Similarly, the attendance register has the names of children sorted in alphabetical order. The telephone directories also have the contact numbers sorted by owner's name alphabetically. The sorted data is preferred over unsorted data because it is easier to handle and search for.

Small amount of data, a class of 60 in school, can be manually sorted but large data, for example, employee data in big company cannot be sorted manually. So, several sorting algorithms came into existence that could help human sort the data in seconds.

The data can be sorted in various ways like an ascending order, descending order, alphabetical order (in case of names). It can also be sorted using multiple keys for example sorting the employee data first by their respective department and then in an alphabetical order.

Sorting optimizes the efficiency of various algorithms like searching and merging, etc. Some algorithms like Prim's algorithm , Kruskal's algorithm, dijkstra's algorithm require the data to be already sorted before they could be applied.

### What are various properties of a sorting algorithm?

A sorting algorithm has different properties like:

Inplace or not: An in-place sorting algorithm does not use any extra space to arrange the given elements in order i.e. sorts the elements by swapping them. For example: insertion sort, selection sort, etc.

Stability: A stable sorting algorithm keeps the equal elements in same order as in the input. For example: insertion sort, merge sort, bubble sort, etc.

Comparison-based or not: As the word suggests, a comparison-based algorithm compares elements to each other to sort them like bubble sort, insertion sort, selection sort, etc. The non-comparison based sorting algorithms work on assumptions like counting sort, bucket sort and do not sort using comparison operator.

Internal or External: Internal sorting can be carried out while residing in the computer's memory whereas when there are a large number of elements external sorting has to be carried out by storing data in files. Sorting algorithms can be implemented both internally and externally depending on memory requirement.

### Explain some basic sorting algorithms.

The basic sorting algorithms include bubble, insertion and selection sort which make the sorting concept easier to understand. These algorithms are capable of sorting small amount of data and are not efficient with large data.

**Bubble sort** swaps two consecutive out-of-order elements, do this repeatedly until no more swaps occur. It bubbles(moves) up the elements to its proper place that they belong in specified order.

**Insertion sort** is an in-place, stable sorting technique that maintains two sub lists i.e. sorted and unsorted and progresses by comparing each consecutive element and inserting it in the correct order.

**Selection sort** selects the maximum element from the data and keeps replacing it with the elements from the end for maintaining an ascending order.

### What are some efficient sorting algorithms?

Some efficient sorting algorithms include merge sort, quick sort and shell sort, heap sort that work well with large amount of data.

**Merge sort** is based on the divide and conquer technique i.e. it partitions the given data into groups until each group contains single element and then sorts and merges each group iteratively to produce sorted list of data.

**Quick sort** chooses a pivot element to partition the data. Both the sides are sorted by iteratively splitting and sorting and then merged in a way that the elements before the pivot have a value lesser than that of a pivot and vice versa.

**Shell sort** is an improved insertion sort. An insertion sort exchanges consecutive elements but a shell sort can exchange far apart elements with each other which leads to an increase in the efficiency of the algorithm.

### What are some distribution-based sorting algorithms?

Distribution based sorting algorithms are those which divide the data into structures, sort it individually and then combine to get the sorted output. These include counting sort, bucket sort and radix sort.

**Counting sort** is a non-comparison-based algorithm that groups all the elements into bins in the range 0 to k and then counts the number of elements less than or greater than a particular element to place it at the correct position.

**Radix sort** uses any stable sort to sort the input according to the least significant digit followed by the next digit iteratively until we reach the most significant digit. It assumes the input consists of d-digit integers.

**Bucket sort** assumes that the input is uniformly distributed and falls in the range [0, K). It uses a hash function to assign each element to a bucket. Then all the buckets are sorted using any sorting algorithm and then merged to get sorted data.

### What is tree sort and heap sort?

**Tree sort** utilizes the property of binary search trees to sort the elements. A binary search has the property that the value of the left child node is less than the parent node and the value of the right child node is greater than that of the parent node. So, this sorting algorithm first builds a binary search tree with the elements and then ouptuts the in order traversal of the binary search tree which gives the elements in sorted order.

**Heap sort** first inserts the given elements into a heap and then deletes the root element to get the elements in sorted order. Max-heap has the property that the value of each node is greater than equal to its child nodes and vice versa for the min-heap. For example, a min-heap is build using the elements. The root is deleted repeatedly while maintaining heap property to get ascending order as the root is smallest element.

### What is the time and space complexity of different sorting algorithms?

Time complexity is the amount of time required by algorithm to execute as a function of input size. Similarly, space complexity refers to the amount of space required by algorithm to execute as function of input size.

The symbols to denote best-case time complexity is \(Ω(n)\), average-case time complexity is \(Θ(n)\) and worst-case time complexity is \(O(n)\). Time complexity order is same as mathematics 1>logn>n>nlogn>n^2.

Bubble sort and Insertion sort offer \(Ω(n)\) best-case time complexity and \(O(n^2)\) worst-case time complexity. The worst-case space complexity is \(O(1)\) as they are in place sorting.

Selection sort offers \(Ω(n^2)\) best and worst-case time complexity as it is independent of the prior order of elements and \(O(1)\) space complexity.

Quick sort and Merge sort offer \(Ω(n \cdot log(n)\) best-case time complexity whereas worst-case time complexity of merge sort is \(O(n \cdot log(n)\) but \(O(n^2)\) for quick sort because it depends on position of the pivot element. . The worst-case space complexity of merge sort is \(O(n)\) to store an extra array and that of stable quick sort is \(O(log(n)\) because of recursive calls.

### When to use which sorting algorithm?

Choosing a sorting algorithm entirely depends on the problem at hand.

Bubble sort is used for basic understanding of sorting algorithms and can detect already sorted data. . Bubble sort It is used to sort the TV channels according to user's viewing time.

Selection sort is quite easy to implement and offers in-place sorting but does not perform well when the size of data increases.

Insertion sort sorts the data as soon as it receives it and is quite adaptive i.e. works well with already sorted data.

Merge sort is stable algorithm that can be implied when an efficient algorithm is needed. It is used where the data is accessed in sequential manner like tape drives and also in field of e-commerce.

Quick sort works well with arrays and medium sized datasets. It is considered one of the fast sorting algorithms if the pivot element is chosen randomly and is employed by sport channels to sort scores.

Heap sort is used when there is restriction on space complexity as it offers O(1) worst-case space complexity, i.e. utilises no extra space. It is employed in security and embedded system problems and used to sort linked list.

## Milestones

1890

Sorting originated in the late 1800s when Herman Hollerith was tasked with determining the population count, so he invented the sorting machine to sort one column at a time.

1945

John Von Neumann develops merge sort in 1945 and bottom-up merge sort in 1948.

1946

John Mauchly introduces insertion sort.

1954

Radix sort is developed at MIT by Harold H. Seward and is considered one of the first sorting algorithms.

1956

Bubble sort initially called **'Sorting by Exchange'** is developed in 1956. Iverson coins the word bubble sort in 1962.

1960

Tony Hoare invents quick sort in Moscow while working on a machine translation project and wanted to sort Russian sentences.

Sorting Algorithms


# C++ Inheritance

## Summary

Inheritance is one of the most important principles of object-oriented programming. In C++, inheritance takes place between classes wherein one class acquires or inherits properties of another class. The newly defined class is known as **derived class** and the class from which it inherits is called the **base class**. Class inheritance reflects heredity in the natural world, where characteristics are transferred from parents to their children.

In C++, there are many types of inheritance namely, single, multiple, multilevel, hierarchical, and hybrid. C++ also supports different modes of inheritance. These are public, private, and protected.

Inheritance promotes *code reuse*. Reusing code not only makes code easy to understand but also reduces redundancy. Code maintenance is easier. Code reliability is improved.

## Discussion

### Could you explain C++ inheritance with an example?

Consider two different classes, `Dog` and `Cat`. Both classes might have similar attributes such as breed and colour. In C++, such attributes are called **data members**. The classes might also have similar behaviours or actions such as eat, sleep and speak. In C++, such actions are implemented as **member functions**. Instead of writing similar code in two different classes we can create an `Animal` class that brings together common attributes and behaviours. Classes `Dog` and `Cat` can inherit from `Animal` class. They can add their own functionalities.

In this example, `Animal` is the base class, `Dog` and `Cat` are derived classes. We may say that (Animal, Dog) represents a parent-child relationship. Dogs are inheriting characteristics of animals. We may also say that `Animal` is a generalization of `Dog` and `Cat`; and `Dog` and `Cat` are specializations of `Animal`.

Derived classes can specialize by adding extra data members and member functions. They can also **override** the general behaviour defined in the base class.

Inheritance in C++ follows a *bottom up* approach. Derived classes acquire properties from base classes but the reverse is not true.

### What are different types of inheritance in C++?

C++ supports many types of inheritance based on the number of base or derived classes and the class hierarchy:

    + **Single Inheritance**: A derived class inherits from only one base class. Eg. `class B` inherits from `class A`.
    + **Multiple Inheritance**: A derived class inherits from more than one base class. Eg. `class C` inherits from both `class A` and `class B`.
    + **Multilevel Inheritance**: The class hierarchy is deeper that one level. Eg. `class C` inherits from `class B` that itself is inherited from `class A`.
    + **Multipath Inheritance**: A derived class inherits from another derived class and also directly from the base class of the latter. Eg. `class C` inherits from `class B` and `class A` when `class B` itself is inherited from `class A`.While the above types are about a single class, there are others that are about the structure of the class hierarchy. These are more design patterns than types:

    + **Hierarchical Inheritance**: A base class is specialized in many derived classes, which in turn are specialized into more derived classes.
    + **Hybrid Inheritance**: This combines both multiple and hierarchical inheritance.

### How does C++ deal with the diamond problem or ambiguity?

The **diamond ambiguity** can occur with multiple inheritance. It happens when the base classes themselves are derived from another common base class. The final derived class ends up with multiple copies of the distant base class.

Consider D inheriting from classes B1 and B2, B1 inheriting from A, and B2 inheriting from A. Thus, D has multiple copies of A. The problem occurs when a call is made to a member of A from an instance of D. C++ compiler can't determine which copy of A to use. Compilation will fail.

To solve this problem, C++ uses the concept of **virtual inheritance**. When inheriting A, B1 and B2 will specify the `virtual` keyword. The compiler sees this and creates a single instance or copy of A within D. No `virtual` keyword is needed when defining D.

Having multiple copies of a base class is not really an error. In fact, it's possible to define a class that uses both virtual and non-virtual inheritance. Ultimately, the compiler should be able to determine member access *unambiguously*.

### What's the concept of name hiding in C++?

**Name hiding** happens when a derived class has a member with the same name as a member of the base class. The derived class definition hides the base class definition. Thus, `Animal::age` is hidden by `Dog::age`.

The concept is also related to but different from **overriding**. For example, `Dog::speak()` if defined overrides `Animal::speak()`. In addition, `Dog::speak()` can explicitly call the base class implementation with the statement `Animal::speak()`.

C++ also allows functions to be **overloaded**, which is about having multiple member functions of the same name that differ in their parameter types. So, we could have `Animal::speak()` and the `Dog` class that defines only `Dog::speak(const string&)`. Unfortunately, in this case, `Dog` class doesn't have access to `Animal::speak()`. In this case, we have name hiding, not overriding.

It's possible for `Dog` class to have access to `Animal::speak()` by simply including the line `using Animal::speak;` in its class definition. With this `using` directive, all the overloaded `speak()` functions of the base class become accessible in the derived class.

### What is object slicing in C++?

When a derived class object is typecast into one of its base classes, we're doing what's called **object slicing**. Conceptually, it's similar to typecasting a decimal number to an integer type, whereby the number loses it's decimal portion. With object slicing, the object loses members specialized in the derived class and retains only those parts of the base class to which it's been typecast.

Consider a base class A and a derived class B. Consider object b of type B. Object slicing happens with the assignments `A a = b` and `A& a_ref = b`. Assignment operator is not virtual in C++. Hence, in these assignments the assignment operator of A is called and not that of B.

The figure shows another example. Base class defines data member `a` and virtual functions `bar1()` and `bar2()`. Derived class overrides `bar1()`, and adds `bar3()`, `bar4()` and `b`. An A-type slice of B can access `a`, derived class function `bar1()` and base class function `bar2()`.

### What are compile-time and runtime bindings in the context of inheritance?

If the C++ compiler can determine what function to call at compile time, we call it **compile-time binding**, early binding or static dispatch. However, where virtual functions are defined in a base class and overridden by derived classes, the compiler can't know which function to call. This information is available only at runtime, giving rise to the terms **runtime binding**, late binding or dynamic dispatch.

Suppose classes `Square` and `Triangle` are derived from `Shape`. Consider member functions `virtual void Shape::area() {...}`, `void Square::area() {...}` and `void Triangle::area() {...}`. Also defined is the function `paintArea(Shape& shape) { int area = shape.area(); ... }`. Exactly which area function is called inside `paintArea()`? This is known only at runtime.

Under the hood, runtime binding relies on *vpointers* and *vtables*. Every class with virtual functions has a *vtable* that stores pointers to virtual function definitions. When a virtual function is overridden, the pointer would point to the derived class implementation. Every class with a *vtable* also has a *vpointer* that points to the correct *vtable*.

### What are the rules of inheritance involving C++ abstract classes?

An abstract class in C++ has at least one pure virtual function, which essentially defines the interface but doesn't supply an implementation. Abstract classes are meant to be used as base classes for other class definitions. Abstract classes themselves can't be instantiated. Their very purpose is inheritance that conforms to an interface.

An abstract class can't be used as a parameter type, a function return type or the result of an explicit conversion. Pointers and references to an abstract class are allowed.

If a derived class inherits from an abstract class without supplying implementations, the derived class is also an abstract class. Thus, "abstraction" can be inherited. On the other hand, it's possible to inherit from a non-abstract base class, add pure virtual functions or override a non-pure virtual function with a pure virtual function, and thereby define an abstract derived class.

When a pure virtual function is called from a constructor, the behaviour is undefined.

### What are the different visibility modes in inheritance in C++?

C++ supports three access specifiers: public, protected and private. These access specifiers are used on data members and member functions. If not explicitly mentioned, private access is the default.

Likewise, a derived class can use an access specifier on each of its base classes. Available access specifiers are public, protected, and private. Where multiple inheritance is used, each base class can have a different access specifier. If not explicitly mentioned, private inheritance is the default.

Regardless of the access specifier on the base class, base class private members will remain private; that is, derived class can't access them. For public and protected members of base class, we summarize how access specifier on the base class affects access to members of the derived class:

    + **Public**: Public and protected members of base class become public and protected members of derived class respectively.
    + **Protected**: Public and protected members of base class become protected members of derived class.
    + **Private**: Public and protected members of base class become private members of derived class.

### What's the influence of access specifiers on virtual functions?

The figure shows three examples involving the member function `A::name()`. In (a), the function is non-virtual. Hence, when it's called via the base class pointer, `A::name()` is called. This is really compile-time binding.

In (b), the function is made virtual. Now when the same call is made, runtime binding happens and `B::name()` is called. Although `B::name()` is private, it's called via the base class pointer and `A::name()` itself is public.

In (c), we make the inheritance protected. This makes `A::name()` protected in the derived class. This means that it can't be called from outside the class, such as from `main()`. Hence we get a compile-time error just as if `B::name()` had been declared protected or private.

### How is inheritance of struct different from that of class?

Data structure `struct` comes from C programming language and is applicable in C++ as well. C++ extends `struct` so that it can include member functions. Like `class`, a `struct` definition can also be inherited.

The main difference is that when access specifiers aren't specified, `struct` members are public by default whereas `class` members are private by default.  Likewise, when access specifiers are omitted in inheritance, `struct` inheritance is public by default whereas `class` inheritance is private by default.

For completeness, we note that there's also `union` that comes from C. It's members are public by default. Unions can't be used as base classes.

### What are the workings of C++ class constructors and destructors?

Constructors and destructors were traditionally not inherited. Since C++11, constructors could be inherited.

Base class constructors are called before derived class constructors. In the case of multiple inheritance, base classes constructors are called in the depth-first left-to-right order of inheritance in the derived class. Destructors execute in reverse order.

Constructors have be defined as public or protected so that derived class constructors can call base class constructors.

Constructors need not be virtual: constructors are always called by name. Destructors have to be virtual. In other words, it's not sufficient to destroy only the base class object. We need to destroy the original derived class object.

### What are the main criticisms of C++ inheritance?

Inheritance in C++ is complex. Even a single inheritance has six variants: private/protected/public and virtual/non-virtual. With multiple inheritance, this complexity increases. The utility of a private virtual function is rather limited.

The designer of a base class decides if a member function must be declared virtual. Derived class can't control this or prevent calls to base class non-virtual functions. Virtual inheritance is also a problem since the decision is made early. It prevents defining a derived class that wants two copies of the distant base class.

C++ implementation of polymorphism via virtual functions can impact performance. Virtual member functions are not directly called. Instead, they've to be looked up via *vpointer* and *vtable*.

## Milestones

Apr
1979

At Bell Laboratories, inspired by Simula, Bjarne Stroustrup conceives the idea of *C with Classes*. The new language would combine the low-level features of C and high-level code organization of Simula. Even in these early days, the language includes classes, class hierarchies and **simple inheritance**. Also included are **private and public inheritance** of a base class.

1983

To support runtime polymorphism, **virtual functions** are added to the language. Only with the introduction of virtual functions, the language claims to support **object-oriented programming**. This is also when the first implementation becomes available to users. Subsequently, the language is renamed to *C++* (1984) and the first commercial release happens (1985).

Jun
1989

C++ 2.0 is released. This includes support for **multiple inheritance** and **abstract class**. Abstract classes provide a "cleaner separation between a user and an implementor" and reduces compile times.

Sep
2011

ISO publishes the C++11 standard, formally named ISO/IEC 14882:2011. This release introduces identifiers `override` and `final`. These help manage complex class hierarchies. They can be applied on virtual member functions when overridden in a derived class. Identifier `final` can also be applied to a class so that it can't be inherited. **Constructors can now be inherited** with the `using` declaration.  This is useful when a derived object needs to be initialized in the same way as the base object.

Dec
2017

ISO publishes the C++17 standard, formally named ISO/IEC 14882:2017. It's now possible to do **aggregate initialization involving derived and base classes**.

C++ Inheritance


# HTTP/2

## Summary

HTTP/2 is an alternative to HTTP/1.x that has been the workhorse protocol of the web for about 15 years. The older version served well in the days when web pages were simple. In February 2017, an average web page is seen to have more than a hundred assets -- JS files, CSS files, images, font files, iframes embedding other content, and more. This means that the browser has to make multiple requests to the server for a single page. User experience is affected due to long load times. HTTP/2 aims to solve these problems.

In particular, HTTP/2 allows multiple requests in parallel on a single TCP connection. HTTP/1.x allowed only a single request. HTTP/2 is also more bandwidth efficient since it uses binary encoding and compresses headers. It allows for Server Push rather than relying only on client requests to serve content.

## Discussion

### Are there any published results to show that HTTP/2 is better?

As of February 2017, a test comparing plain HTTP/1.1 against encrypted HTTP/2 showed that former is 73% percent slower. The server was in Dallas and the client was in Bangalore for this test. Early testing of HTTP/2 done in January 2015 showed that HTTP/2 uses fewer header bytes due to header compression. Its response messages were also smaller when compared against HTTP/1.1. For page loads, HTTP/2 took 0.772 seconds whereas HTTP/1.1 took 0.988 seconds.

### How is the support for HTTP/2 from browsers and servers?

In 2015, popular browsers started supporting HTTP/2. This includes Chrome, Firefox, Safari, Opera and Edge. Chrome for Android started support in February 2017.

At the server side, W3Techs reported that as of February 2017 12% of the servers support HTTP/2. Of these, 77% were on Nginx and 18% were on LiteSpeed. We see that when platforms that host a number of domains start supporting HTTP/2, there's a spike in HTTP/2 traffic. This can be seen when WordPress, CloudFlare, Blogspot and Wikipedia started support for the protocol. Apache introduced it on an experimental basis in version 2.4.17. KeyCDN reported that as of April 2016 68% of its traffic is HTTP/2.

### Are there any security vulnerabilities with HTTP/2?

In August 2016, Imperva, Inc. reported four security vulnerabilities. These should be seen as implementation flaws rather than flaws in the protocol itself. At least one of these vulnerabilities was found in five different server implementations, which were subsequently fixed.

### Is the use of encryption mandatory for HTTP/2?

No. However, major browsers have said that they will support HTTP/2 only over TLS. The use of HTTP/2 over TLS is referred to as *h2*. The use of HTTP/2 over TCP, implying cleartext payload, is referred to as *h2c*.

### Do I have to change my application code when migrating to HTTP/2?

No. The idea of HTTP/2 was to improve the way packets are "put on the wire." For reasons of interoperability, HTTP Working Group was clear that the semantics of using HTTP should not be changed. This means that headers, methods, status codes and cache directives present in HTTP/1.x will remain the same in HTTP/2 as well. The semantics of HTTP remain unchanged. Hence, application code need not be changed.

Of course, if you're building your own server code or custom client code, you will need to update these to support HTTP/2. The binary framing layer of HTTP/2 headers is not compatible with HTTP/1.x but this affects only HTTP clients and servers. A HTTP/1.1 client cannot talk to a server that supports only HTTP/2.

If your application is using any of the well-known "best practices" for better performance over HTTP/1.1, the recommendation is to remove these best practices. The bigger migration challenge may be to use HTTP/2 over a secure connection if your applications are not currently using TLS.

### Didn't HTTP pipelining already solve this problem in HTTP/1.1?

HTTP/1.0 was really a stop-and-wait protocol. The client could not make another request until response to the pending request was received. HTTP/1.1 attempted to solve this by introducing HTTP Pipelining, which allows a client to send multiple requests to the server in a concurrent manner. Requirement from the server was that responses must be sent out in the same order in which requests were received.

In the real world, performance improvement due to pipelining is not proven. Proxies when implemented wrongly can lead to erroneous behaviours. Other factors including round trip time, packet size and network bandwidth affect performance. There's also the problem of head-of-line blocking whereby packets pending from one request can block other pipelined requests. For these reasons, most modern browsers disable HTTP pipelining by default.

### What sort of hacks did developers use to improve performance before HTTP/2?

Some of the "best practices" to improve performance were really hacks to overcome the limitations of HTTP/1.x. These included domain sharding, inlining, image sprites and concatenation. The idea was to minimize the number of requests. With the coming of HTTP/2, these hacks will no longer be required.

There's one trick that browsers have used to improve performance without using HTTP pipelining. Browsers opened multiple TCP connections to enable concurrent requests. As many as six requests were used per domain. This is the reason why some applications deployed their content across domains so that multiple TCP connections to multiple domains will lead for faster page loads. With HTTP/2, multiple TCP connections will not be required. Multiple HTTP requests can happen on a single TCP connection.

### What's the rationale for using binary encoding?

HTTP/1.x was a textual protocol, which made it easier to inspect and debug the messages. But it was also inefficient to transmit. Implementation was also complex since the presence of newlines and whitespaces resulted in variations of the message, all of which had to be decoded properly by the receiver. Binary formats are more efficient since fewer bytes are needed. Since the structure is fixed, decoding becomes easier. Debugging a binary protocol is more difficult but with the right tool support it's not much of a problem. Wireshark supports decoding and analysis of HTTP/2 over TLS.

### Can you give some details of multiplexing HTTP requests on a single TCP connection?

HTTP/2 defines three things:

    + Frame: Smallest unit of transmission that contains its own header.
    + Message: A sequence of frames that correspond to a HTTP request or response.
    + Stream: A bidirectional flow of messages and identified by a unique identifier.Thus, multiple streams are multiplexed (and interleaved) on to a single TCP connection. Each stream is independent of others, which means that even if one is blocked or delayed, others are not affected. HTTP/2 also allows prioritization of streams, which was not possible with HTTP/1.1.

### How is header compression done in HTTP/2?

Headers in HTTP/1.x can typically 500-800 bytes but can grow to the order of kilobytes since applications use headers to include cookies and referers. It therefore makes sense to compress these headers. HPACK is the name give to HTTP/2 header compression. It was approved by IETF in May 2015 as RFC 7541.

Huffman coding is used to compress each header frame, which always appears at the start of each message within a stream. In addition, many fields of the header will remain the same across messages. HPACK removes this redundancy by replacing the fields with indices that point to tables that map these indices to actual values. There are predetermined static tables defined by HPACK. Dynamic tables allow further compression.

A test on CloudFlare gave 76% compression for ingress headers, which translates to 53% savings on total ingress traffic. The equivalent numbers for egress are 69% and 1.4%. Results also showed that with more traffic, the dynamic table grows, leading to higher compression. KeyCDN reported savings of 30% on average.

### How does Server Push improve performance?

In a typical client-server interaction, the client will request one page, parse it and then figure out other assets that need to be requested from the server. Server Push is a feature that enables the server to push assets without the client asking for it. For example, if a HTML page references images, JS files and CSS files, the server can send these after or before sending the HTML page. Each pushed asset is sent on its own stream and therefore can be prioritized and multiplexed individually.

Since Server Push changes the way clients and servers interact, it's still considered experimental. Wrong configuration could lead to worse performance. For example, the server may push an asset even if the client has cached it from an earlier request. Likewise, servers may push too much data and affect user experience. Proper use of cookies, cache-aware Server Push and correct server configuration can aid in getting Server Push right.

## Milestones

2012

IETF's HTTP Working Group begins looking at Google's **SPDY** protocol as a starting point for defining HTTP/2.

Feb
2015

IESG approves HTTP/2 as a **proposed standard**.

May
2015

HTTP/2 is published as **RFC 7540**.

HTTP/2


# WordNet

## Summary

WordNet is a database of words in the English language. Unlike a dictionary that's organized alphabetically, WordNet is organized by concept and meaning. In fact, traditional dictionaries were created for humans but what's needed is a lexical resource more suited for computers. This is where WordNet becomes useful.

WordNet is a network of words linked by lexical and semantic relations. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms, called **synsets**, each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations. The resulting network of meaningfully related words and concepts can be navigated with the WordNet browser.

WordNet is freely and publicly available for download.

## Discussion

### What's the distinction between WordNet and a thesaurus?

A thesaurus provides similar words (synonyms) and opposites (antonyms). WordNet does much more than this. Via synsets, WordNet brings together specific word senses. As a result, words that are found in close proximity to one another in the network are semantically disambiguated.

A synset is also linked to other synsets by semantic relations. Such relations are missing in a thesaurus. These relations are based on concepts and therefore give us valuable information about words. For example, the verbs (communicate, talk, whisper) are all about talking but the manner goes from general to specific. A similar example with nouns would be (furniture, bed, bunkbed). An example of a part-whole relation is (leg, chair). These sorts of relations are captured in WordNet.

The nodes of WordNet are synsets. Links between two nodes are either **conceptual-semantic** (bird, feather) or **lexical** (feather, feathery). Lexical links subsume conceptual-semantic links.

### Could you explain WordNet's synsets with an example?

Consider the word 'bike'. It has multiple meanings. It could be a motorcycle (noun), a bicycle (noun) or bicycle (verb). WordNet represents these as three synsets with unique names: `motorcycle.n.01`, `bicycle.n.01` and `bicycle.v.01`.

Each synset has an array of lemma names that share the same concept. Thus, `motorcycle.n.01` has the words 'motorcycle' and 'bike'. The synset also has a definition, also called **gloss**. It now becomes clear that the word 'bike' must be present in all the three synsets, each representing a different concept or meaning.

It's also possible to move from one synset to another by certain relations. For example, `motor_vehicle.n.01` is a more general concept of `motorcycle.n.01` whereas `minibike.n.01` and `trail_bike.n.01` are more specific concepts of `motorcycle.n.01`. Thus, synsets are linked by relations making WordNet a network of conceptually related words.

### What is WordNet used for?

WordNet is typically used by linguistics, psychologists and those working in the fields of AI and NLP. Among its many applications are word sense disambiguation, information retrieval, automatic text classification, automatic text summarization, and machine translation.

WordNet can be used as a thesaurus except that words are organized by concept and semantic/lexical relations. In NLP, WordNet has become a useful tool for word sense disambiguation. When a word has multiple senses, WordNet can help in identifying the correct sense. WordNet's symbolic approach complements statistical approaches.

Measuring similarity between words is another application. Different algorithms exist to measure word similarity. Such a similarity measure can be used in spelling checking or question answering. However, WordNet is limited to noun-noun or verb-verb similarity. We can't compare nouns and verbs, or use other parts of speech.

Where neural networks are used for NLP work, word embeddings (low-dimensional vectors) are used. However, word embeddings don't discriminate different senses. WordNet has been applied to create **sense embeddings**.

### What are major lexical relations captured in WordNet?

Major lexical relations include the following:

    + **Synonymy**: Synonyms are words that have similar meanings. Often context determines which synonym is best suited.
    + **Polysemy**: Polysemous words have more than one sense. The word bank can mean river bank, where money is stored, a building, or institution. Polysemy is associated with the terms *homonymy* and *metonymy*.
    + **Hyponymy/Hypernymy**: Is-a relation. Robin is a hyponym of bird since robin is a type of bird. Likewise, bird is a hypernym of robin. Thus, hypernyms are synsets that are more general whereas hyponyms are more specific.
    + **Meronymy/Holonymy**: Part-whole relation. Beak is meronym of bird since beak is part of a bird's anatomy. Likewise, bird is holonym of beak. WordNet identifies three types of relations: components (leg, table), constituents (oxygen, water), and members (parent, family).
    + **Antonymy**: Lexical opposites such as (large, small).
    + **Troponymy**: Applicable for verbs. For example, whisper is a troponym of talk since whisper elaborates on the manner of talking.

### What are some limitations of WordNet?

WordNet doesn't include syntactic information, although later work showed that at least for verbs there's correlation between semantic makeup and syntactic behaviour.

Semantic relations are more suited to concrete concepts, such as tree is a hypernym of conifer. It's less suited to abstract concepts such as fear or happiness where it's hard to identify hyponym/hypernym relations. Some relations may also be language specific and therefore can make different wordnets less interoperable.

WordNet's senses are sometimes too fine-grained for automatic sense disambiguation. One possible solution is to group related senses.

WordNet doesn't include information about the etymology. Thus, word origins and how they've evolved over time are not captured. Offensive words are also included and it's left to applications to decide what's offensive since meanings change over time. Pronunciation is missing. There's limited information about usage. WordNet covers most of everyday English but doesn't include domain-specific terminology.

WordNet was created in the mid-1980s when digital corpora were hard to come by. WordNet was assembled by the intuition of lexicographers rather than by a corpus-induced dictionary.

## Milestones

1928

Murray’s Oxford English Dictionary (OED) is compiled "on historical principles". By focusing on historical evidence, OED, like other standard dictionaries, neglects questions concerning the synchronic organization of lexical knowledge.

1969

Collins and Quillian propose a **hierarchical semantic memory model** for storing information in computer systems. They hypothesize that human memory is in fact organized in this manner. They test their hypothesis by measuring retrieval times. For example, if a person is asked if a canary can fly, the actual retrieval might involve inference from a memory that contains "canary is a bird" and "birds can fly". This important work goes on to influence the creation of WordNet almost two decades later.

1976

Miller and Johnson-Laird propose **psycholexicology**, a study of the lexical component of language, which is about words and vocabulary of a language.

1985

Some psychologists and linguists at Princeton University start developing **a lexical database**. While a dictionary helps us search for words alphabetically, a lexical database allows us to search based on concepts. This marks the beginning of **Princeton WordNet**. We can say that it's a dictionary based on psycholinguistic principles.

1991

**WordNet 1.0** is released.

1996

**EuroWordNet** is started as an EU project covering languages Dutch, Spanish and Italian. It's inspired by and is designed to link to the Princeton WordNet. In 1997, more languages are added: German, French, Czech and Estonian. The project is completed towards the end of 1999. One novel feature is the *Inter-Lingual-Index (ILI)* that defines equivalence relations between synsets in different languages. In later years, this work is extended by other projects: EUROTERM, BALKANET, and MEANING. By 2006, it's noted that databases exist for 35 languages globally.

Mar
2005

**WordNet 2.1** is released. There's support for UNIX-like systems and Windows. WordNet 2.1 contains almost 118,000 synsets, comprising more than 81,000 noun synsets, 13,600 verb synsets, 19,000 adjective synsets, and 3,600 adverb synsets.

Dec
2006

**WordNet 3.0** is released. This release has 117,798 nouns, 11,529 verbs, 22,479 adjectives, and 4,481 adverbs. The average noun has 1.23 senses, and the average verb has 2.16 senses.

Jun
2011

**WordNet 3.1** is released. It's available only online. It's possible to download only the database and use the installation from 3.0. This version contains 155,327 words organized in 175,979 synsets for a total of 207,016 word-sense pairs. It's compressed size is 12MB.

Jun
2018

Under the guidance of Global WordNet Association, the **English WordNet** is created on GitHub as a fork of the Princeton WordNet 3.1. Annual updates of this resource happen in April 2019 and April 2020.

WordNet


# OAuth

## Summary

OAuth is an authorization framework that allows a third-party application to access private resources of a user when those resources are stored as part of another web service. For example, a user has an account with GMail or Facebook. The user visits a gaming platform that requests access to the user's basic profile on GMail or Facebook. OAuth allows the user to authorize this gaming platform to access the profile.

The strength of OAuth lies in the fact that the user's credentials (username, password) are not shared with third-party services. Such credentials are used only between the resource owner (user) and the authorization server. Thus, OAuth is for authorization, not authentication. In addition, the user can at any point revoke this authorization.

## Discussion

### Why was OAuth invented in the first place?

Back in 2006, Blaine Cook was investigating a means to grant third-party applications access to Twitter API via delegated authentication. He considered using OpenID for this purpose but found it unsuitable. The idea was that users should not be required to share their usernames and passwords to third-party applications.

There were at the time other frameworks but they were not open standards: Flickr Auth, Google AuthSub and Yahoo! BBAuth. OAuth was thus invented as an open framework for authorization without even specifying the methods of authentication. In fact, it's been said that OAuth was written by studying and adopting the best practices from these early proprietary methods.

### What are the available OAuth roles?

Four roles are defined in the standard:

    + **Client** is the application requiring access to protected resources.
    + **Resource Owner** is the entity owning the resource. It's usually a person, called end-user. Resource owner gives authorization to client to access protected resources.
    + **Authorization Server** does authentication of the resource owner and handles authorization grants coming from the client. It gives out access tokens.
    + **Resource Server** manages protected resources based on access tokens presented by clients.As an example, a gaming application (client) requires the profile of a gamer (resource owner) from her GMail account. Access to the gamer's GMail profile (protected resource) is authorized by the gamer. The GMail service encapsulates both the authorization and resource servers. Implementations have the choice to combine authorization and resource servers into a single server, but they might just as well be different servers.

### Could you briefly describe the protocol flow?

OAuth specifies different authorization grant types, each of which has its own flow. We can summarize the flow as made up of these steps: obtain authorization from user, exchange this authorization for an access token, use the token to access protected resources.

Delving into the details, OAuth flows can be related to the level of confidentiality involved. If the client is a web application, then it's a confidential client since client ID and secret key are kept on the client server. If the client is a user-agent (web browser) or a native application running on a device with the resource owner, then it's a public client. From this perspective, we can look at flows in this manner:

    + **Confidential**: client credentials flow, password flow, authorization code flow
    + **Public**: implicit flow, password flow, authorization code flowThough it's more common to talk of grant types these days, the terms two-legged flow and three-legged flow are still in use. With the former, the resource owner is not involved, as in client credentials flow. With the latter all three (resource owner, client and authorization server) are involved.

### What essential data are exchanged for authorization?

Every client must have a client ID and a client secret key. Sometimes these are called consumer ID and consumer secret key. The client will also have a redirect URI. This is the endpoint where the authorization code will be received by the client and subsequently exchanged for an access token. This tuple of client ID, secret key and redirect URI is generated or configured in advance on the authorization server, what is called client registration. Authorization will fail if these do not match.

Access token is the one that's presented to the resource server after a successful authorization. Access token simply allows access to protected resources. The resource owner may choose at any point to revoke access at the authorization server. A token can also expire, in which case the client can request a token refresh if refresh is allowed.

In the case of password flow, authorization will be based on the resource owner's username and password. From a security standpoint, this flow makes sense only if the resource owner trusts the client enough to give out her username and password.

### Which are the endpoints defined in OAuth?

An endpoint is nothing more than a URI. The standard defines the following endpoints though extra endpoints could be added:

    + **Authorization endpoint**: Located at the authorization server, this is used by the client to obtain authorization from the resource owner. Resource owner is redirected to authorization server via a user-agent (typically the web browser).
    + **Redirection endpoint**: Located at the client, this is used by the authorization server to return responses containing authorization credentials to the client via the user-agent.
    + **Token endpoint**: Located at the authorization server, this is used by the client to exchange an authorization grant for an access token.The typical call flow, at least for authorization code grant type, invokes the endpoints in the order of Authorization, Redirection and Token. In the case of implicit grant type, the order is Authorization and Redirection endpoints. The Token endpoint is not involved since the Authorization endpoint implicitly returns an access token.

### Are there open source implementations of OAuth?

OAuthLib is an open source implementation in Python. Hydra is an implementation in Go. It covers OAuth and OpenID Connect. In Java, we have Spring Security OAuth and Apache Oltu. The latter includes OAuth, JWT, JWS and OpenID Connect. From Google, there's Google OAuth Client Library for Java. Anvil Connect supports OAuth, JWT and OpenID Connect.

We have keep in mind that these implementations may be specific to client or server or both.

### How would you compare OAuth with OpenID Connect and SAML?

OpenID Connect (OIDC) is an identity and authentication layer that uses OAuth 2.0 as the base layer for authorization. As an ID token it uses a signed JSON Web Token (JWT), also called JSON Web Signature (JWS). It uses REST/JSON message flows. OIDC best suited for single sign-on apps while OAuth is best for API authorization.

In some use cases, OAuth 2.0 may be used by implementors as a means of pseudo-authentication, by assuming the entity owning the access token is also the resource owner. Eran Hammer-Lahav summarized the difference between OpenID and OAuth nicely (authentication vs authorization),

> While OpenID is all about using a single identity to sign into many sites, OAuth is about giving access to your stuff without sharing your identity at all.

Security Assertion Markup Language (SAML) offers both authentication and authorization. What is called a token in OpenID and OAuth terminology, is called an assertion in SAML. Assertions (structured in XML) contain statements that fulfil the purpose of authentication and authorization. SAML may not be suited for mobile apps.

As a simplification, we could see OIDC as a combination of OAuth and SAML.

### What are OAuth extensions?

OAuth allows extensibility in terms of authorization grant types, access token types, and more. This will allow OAuth to interwork with other protocol frameworks.

Website OAuth.net considers three RFCs as part of the OAuth 2.0 Core: 6749, 6750, 6819. It then lists a number of OAuth extensions. Likewise, a search on IETF Datatracker yields all documents that pertain to OAuth.

### How does OAuth 1.x compare against OAuth 2.0?

OAuth 2.0 uses SSL/TLS for security. With OAuth 1.x, each request had to be secured by the OAuth implementation, which was cumbersome. In this sense, OAuth 2 is simpler. OAuth 1.0 suffered from session fixation attacks. Such a flaw does not exist in OAuth 1.0a and above.

OAuth 2.0 uses grant types to define different flows or use cases. OAuth 1.x worked well for server-side applications but not so well for browser web apps or native apps. OAuth 2.0 is not compatible with OAuth 1.0 or 1.0a (sometimes called 1.1). A deeper technical comparison of versions 1.x and 2.0 is available at OAuth.com.

Controversially, it's been mentioned that OAuth 2 is "more complex, less interoperable, less useful, more incomplete, and most importantly, less secure."

### What are the problems with OAuth?

One of OAuth's original creators, Eran Hammer, claimed that OAuth 2 is more complex and less useful than its earlier version. It's been "designed-by-committee" with enterprise focus. He claims OAuth should have been a protocol rather than a framework. The end result is too much flexibility and very few interoperable implementations.

In terms of security risks, one analysis showed how developers can mistakenly use OAuth for the purpose of user authentication. In 2014, the "Covert Redirect" vulnerability was discovered where phishing along with URL redirection can be used to gain access to protected resources. This can be solved by having a whitelist of redirect URLs on the authorization server. In 2016, researchers discovered that man-in-the-middle attacks were possible with OAuth and OpenID Connect. Because OAuth 2.0 uses TLS, it doesn't support signature, encryption, channel binding and client verification.

## Milestones

Nov
2006

Blaine Cook starts looking at OpenID but wants a better delegated authentication method for the Twitter API.

Apr
2007

OpenAuth Google group is started but when AOL releases their own protocol named OpenAuth, this group changes its name to OAuth in May 2007.

Dec
2007

OAuth Core 1.0 final draft is released.

May
2009

OAuth Working Group is created at IETF. One of the creators of OAuth later commented that bringing OAuth into IETF was a mistake.

Jun
2009

OAuth Core 1.0 Revision A is released. It fixes the session fixation attack.

Apr
2010

IETF releases RFC 5849 that replaces OAuth Core 1.0 Revision A.

Oct
2012

OAuth 2.0 is released by IETF as RFC 6749.

OAuth