# Dead Code

## Summary

Dead code is any code that's never executed, or if executed, the execution has no effect on the application's behaviour.

Dead code adds unnecessary complexity to the codebase. It makes the codebase more difficult to understand and maintain in the long run. It impacts performance and security. In some cases, due to poor software engineering practices, dead code can "come back to life" and cause catastrophic failures. It's therefore recommended to remove dead code.

Tools and techniques are available to help developers get rid of dead code. But developers tend to leave them around rather than remove them. It's been said that,

> Deleting dead code is not a technical problem; it is a problem of mindset and culture.

## Discussion

### What's the definition of dead code?

Dead code is any code that has no effect on the application's behaviour. This is a simple and useful definition. Looking into the details, we identify some alternatives to this definition.

Code that is unnecessarily executed is called **redundant code**. This would include variable assignments or function returns that are subsequently never used. A subset of this may be called **partially dead code** that refers to code that's essential only on some program paths.

Code that can't be reached or executed on any program path is called **unreachable code**. This could happen if there's an unconditional function return and there's code that follows the return statement. Code that's commented is another example. A superset of this is **unused code** that could be executed but it's not executed because the user never triggers it.

### What are the different types of dead code?

Dead code can be of the following types:

    + **Dead Types**: Types are defined but not used anywhere. This includes types used by only other types but the latter themselves are not used anywhere. Public types should not be considered dead since they maybe called by client code.
    + **Dead Methods**: Like dead types but applicable for methods.
    + **Dead Fields**: Unused fields or fields that are only initialized but not used anywhere in the code. Dead fields include attributes of classes common in object-oriented programming languages.
    + **Dead Parameters**: Maybe considered as a special case of dead fields. For example, a function accepts four parameters but only three are used by the function.Since public types, methods and fields could be called by client code, we don't consider them as dead during static analysis or compilation. However, a linker can still identify dead code and remove them if they see that client code is not calling them. An example of this is *R8* code shrinker that developers use in Android projects.

### What problems can dead code cause?

Dead code introduces unnecessary **complexity** and confusion. People who maintain code are usually not the same as those who created it. With dead code around, it's hard to know what's safe to delete or update. Developers may even maintain dead code (and their test cases) without realizing that it's a waste of effort. From a design perspective, dead code increases or preserves coupling among classes. This in turn makes code refactoring more difficult.

Dead code takes up extra **space**. Program size is larger and so too its runtime footprint.

Dead code that executes without changing application's functional behaviour but may affect **performance**. CPU cycles are spent in executing redundant code.

Dead code that hasn't been touched for a long time may also contain **security** issues that could be exploited by hackers. It's smarter to delete such code than maintain it.

A **patent** is infringed only if the code that implements the claimed invention actually executes. Therefore, patent holders will have a weaker case if some of their code is dead. This is yet another reason to identify dead code.

### Could you share some real-world instances of dead code?

A high-profile example is that of Knight Capital Group that lost $440 million in a single day on the NYSE due to dead code. This happened on August 1st, 2012 though the dead code had been unused since 2003. The problem was triggered because a dead feature flag was repurposed for some new code. This unintentionally activated the dead code that had been designed for only test environments. The codebase had also been refactored without adequate regression testing. Moreover, deployment was done manually without a proper reviewal process.

In 2014, Apple fixed a security vulnerability that was attributed to a copy-paste edit that lead to dead code. A duplicate line of code allowed an attacker to bypass an authentication check, which became dead code. Static code analysis would have caught this problem. Informal peer review, formal code inspection, test coverage (line and path coverage) and training developers are other ways this could have been prevented or caught earlier.

### Why do we have dead code?

Dead code arises due to **software maintenance** and **software aging**. For example, a new fragment of code might make another code fragment dead. Developers are either unaware of this or think that such code may be needed again someday. When software ages, dead code due to obsolete features is a common reason.

Where a codebase is inherited from another company, we can have **inherited dead code**. When developers recognize the dead code, they might comment it out or add comments to flag it. They may remove it only if it's low risk and low cost.

Sometimes code is **created dead** with the expectation that it could be used in future. In classes, the use of getters and setters is such an example. Unclear or incomplete specifications are further reasons for dead code. Code reuse can cause dead code since not all interfaces or functions may be used.

Sometimes dead code is active during **testing** but not in production. Developers therefore don't consider removing it. On the contrary, lack of automated regression testing makes developers wary of any refactoring.

### What are some tools and techniques to eliminate dead code?

An easy way to find and eliminate dead code is a good IDE. Unused files can also be removed. Unnecessary classes can be inlined. Class hierarchy can be collapsed. Eclipse, Findbugs, PMD, and NDepend are some useful IDEs or tools.

Languages such as JavaScript have good linting tools to catch dead code. For Go, Grind package can be considered. For R, there's `rco`. For Python, there's Vulture for static code analysis. Although Python is a dynamic language, Vulture can improve code quality.

Where developers suspect dead code, they can log messages. More generally, it's a good practice to log API calls. If such messages appear at runtime, we can be assured that the code is not entirely dead. However, absence of these messages in the log doesn't guarantee that the code is dead.

For JavaScript frontend code, minifiers remove dead code for production builds. Likewise, there are options to C++ compilers and linkers that eliminate dead code. These reduce program size and perhaps improve performance. However, they don't solve the problem of dead code at the source, which implies that software maintenance continues to be complex.

## Milestones

1970

In this decade, there's research towards better compilers, program certification and program optimization. For example, Rosen's 1977 paper titled *High-Level Data Flow Analysis* adopts directed graphs as an analysis tool. These research initiatives facilitate dead code elimination.

1994

Based on techniques for partial redundancy elimination, Knoop et al. present an optimal method for **partial dead code elimination**. Their method is optimal in the sense that any remaining partially dead code can't be eliminated without changing the code structure or its semantics.

1998

In a book about software refactoring, Brown et al. take note of "unused code frozen in an ever-changing design." They call this **lava flow**.

1999

Martin Fowler in his book titled *Refactoring: Improving the Design of Existing Code* identifies 22 code smells and suggests ways to remove them. However, he doesn't explicitly mention dead code as a code smell.

Dec
2003

Bolton and Ung file for a patent with the title *Partial dead code elimination optimizations for program code conversion*. They identify partially dead register definitions that can be eliminated towards more optimized intermediate representation of the code.

Nov
2018

As part of Android Studio 3.3 beta, Google releases **R8** for code shrinking. With R8, APK size reduces. R8 gets rid of unused code and resources. It also does code minification to save space. Proguard is another tool that does this but R8 does it faster while achieving similar results.

Dec
2020

In an analysis of code smells, dos Reis et al. include dead code as one of the code smells. From the 83 studies selected for the analysis, they find that 4.8% contain dead code. The most prominent code smell is the God Class, present in 51.8% of the studies.

Dead Code


# Li-Fi

## Summary

Li-Fi, or Light Fidelity, uses light as a medium of data communication. It comes under the umbrella of Optical Wireless Communication (OWC) that includes infrared, visible light and ultraviolet spectrum. Li-Fi uses visible light spectrum in the downlink and infrared spectrum in the uplink.

Li-Fi promises to achieve data rates in excess of 100Gbps. In 2014, 10Gbps was achieved in a lab environment. A year later, Li-Fi achieved 224Gbps. These rates are many times what current Wi-Fi technology can achieve. For this reason, Li-Fi can be categorized as gigabit communication technology.

Li-Fi is expected to complement cellular and Wi-Fi technologies, not replace them. While some proprietary Li-Fi products exist in the market, Li-Fi has still some way to go to become standardized and mainstream.

## Discussion

### Why do we need Li-Fi when plenty of wireless technologies exist today?

Among the widely deployed wireless technologies are cellular, Wi-Fi and Bluetooth. These operate in radio frequencies less than 10GHz. This RF spectrum is getting increasingly crowded, and there's a growing demand for more capacity and higher speeds. The visible light spectrum is an appealing alternative. Visible spectrum's capacity is 10,000 times larger. In fact, it's about 670THz of license-free spectrum.

Cellular can offer higher capacity by making cells smaller but this incurs heavy investment on infrastructure such as base station equipment and transmission towers. LED lighting is becoming common worldwide and this infrastructure can be reused for Li-Fi. This also means that off-the-shelf components such as LEDs can be used to keep costs down. In contrast, millimetre wave technology at 60GHz requires specialized components.

From the perspective of availability, we're often asked to turn off wireless transmitting devices in aircraft, hospitals and other places where EMI can cause problems. Light is everywhere and there's no problem using it in these places. Line-of-sight and short range characteristics of Li-Fi imply better security and less co-channel interference.

### What are some use cases of Li-Fi?

Applications that require hundreds of Mbps throughput or microseconds of latency will benefit from Li-Fi. In dense urban areas, Li-Fi indoor attocells can complement cellular infrastructure. Li-Fi can be used for cellular or attocell backhauls.

Li-Fi can enable Augmented Reality applications. Lighting in a museum can give rich information and interaction around exhibits. At malls, offers can be delivered via Li-Fi. Many household appliances can get connected via Li-Fi, thus enabling the Internet of Things. Li-Fi can also be used for indoor location tracking and navigation. Li-Fi can enable smart wearables.

Because Li-Fi cannot get through walls, it suits applications that require high security. Only those within the beam of the transmitter can receive data. In environments where radio waves are considered harmful, Li-Fi can be a safe alternative. Radio waves are easily attenuated underwater but this problem doesn't exist for Li-Fi. Vehicle lights can be used for vehicle-to-vehicle, vehicle-to-traffic-light communication and thereby enable autonomous driving technology.

### What are some important technical details of Li-Fi?

    + Spectrum: Visible spectrum is approximately in the range 400-700nm or 430-750THz. Spectrum above 3THz is unregulated.
    + Bandwidth: LED BW is limited to 20MHz but system BW of 180MHz with 512 subcarriers of DMT waveform has been possible.
    + Area spectral efficiency: At least 0.41 bits/s/Hz/m2, which is already 1000x of RF systems.
    + Modulation: Simple methods such as OOK or PWM experience ISI at high data rates. Instead, OFDM or DMT can be used to control light intensity rather than just turning LEDs on and off. There's been significant research to make these methods unipolar. DMO-OFDM is an example.
    + Multiple access: OFDMA is a suitable method and it's already used in IEEE 802.11 and LTE standards. An alternative method would be Wavelength Division Multiple Access (WDMA).
    + Duplexing: Wavelength Division Duplexing (WDD) is the proposed method. Visible spectrum is used for the downlink while IR or RF spectrum is used for the uplink.
    + Topology: Peer-to-peer, star or broadcast.

### Is Li-Fi a proprietary technology or is there a standard for it?

In 2011, IEEE Task Group (TG) 7 released the 802.15.7 standard titled "Short-Range Wireless Optical Communication Using Visible Light". In March 2017, IEEE Task Group 13 started looking into making a new standard 802.15.13 titled "Multi-Gigabit/s Optical Wireless Communications".

It's been said that IEEE 802.15.7m is focusing on optical camera communications and low rate photodiode communications while IEEE 802.15.13 is for speciality networks for industrial applications.

The IEEE 802.11 is also involved in standardizing Li-Fi to complement Wi-Fi. This groups suggests that using 802.11 as the base would be a plus for mass deployment and to complement RF spectrum.

Meanwhile, existing Li-Fi products and their deployments must be considered as proprietary. Any attempt to standardize Li-Fi will likely require support from patent owners.

### How does Li-Fi stack up against Wi-Fi?

Wi-Fi is based on RF, which relies on electric fields to carry information. Li-Fi relies on optical intensity. RF is complex-valued and bipolar while Li-Fi is real-values and unipolar (non-negative).

Li-Fi has the advantage over Wi-Fi in terms of capacity: 10,000 times more capacity. Visible light spectrum is being used for lighting but not for communication yet. Hence this spectrum is free of interference. Wi-Fi IEEE 802.11ac has a theoretical maximum throughput of about 7Gbps but practical values are at best 200Mbps.

LED lighting infrastructure is already installed in many places and can be reused for Li-Fi communications. The cost of LEDs is also dropping. Moreover, solar panels can be used as Li-Fi receivers for data sent from laser stations or lamp posts. Hence Li-Fi has a better proposition in terms of cost and efficiency. Li-Fi can also be used in places such as aircraft and hospitals where EMI due to RF is usually a concern. For location-aware services, Li-Fi may complement iBeacon.

Li-Fi can be seen as complementing Wi-Fi. Reasearch is ongoing regarding hybrid Li-Fi/Wi-Fi networks.

### What are the common criticisms of Li-Fi?

Li-Fi systems need to emit and capture light, implying that they can't be used at night. IR spectrum could be used at night but at the expense of throughput. In bright sunlight, Li-Fi systems may not work that well due to light interference. Li-Fi has limited range and requires line-of-sight. However, the use of OFDM along with M-QAM can enable non-LOS communications. Concerns about lower LED energy efficiency when used for Li-Fi have been dispelled.

While Harald Haas has claimed that existing lighting infrastructure can be used, the reality is that LEDs come in many varieties and there will be compatibility issues with Li-Fi adapters. Moreover, electrical wiring will need to be converted to network wiring for the transmission of data. Consumer devices will need to be replaced to handle Li-Fi signals.

For immediate adoption, a temporary issue is that Li-Fi is not a mature technology and the standards are not yet in place. When Li-Fi last mile is achieving hundreds of gigabits per second, backhaul infrastructure also needs to scale equivalently. However, backhauls reaching a few terabits per second have been reported.

### What are essential components of Li-Fi?

Li-Fi uses LEDs as transmitters and photodetectors/photodiodes as receivers. The former can be phosphor-coated LEDs or RGB LEDs. For receivers, photodiodes can be p-intrinsic-n (PIN) photodiodes or avalanche photodiodes (APD). Off-the-shelf components can be used for these but they need to be driven by signal processing firmware to make them communicating devices. Recent advancements include Resonant Cavity LEDs and MicroLEDs that achieve higher data rates. LEDs have a trade-off between output power and optical efficiency. To solve this, Laser Diodes (LD) have been proposed to target more than 100Gbps data rates.

## Milestones

Aug
2003

Japanese researchers report the use of white LEDs to transmit information at 400Mbps while also serving the primary purpose of indoor lighting.

Nov
2003

The Visible Light Communications Consortium (VLCC) is established, comprising of major companies in Japan. The aim is to provide communication capability through LED lightings.

2006

OFDM is first used for VLC since the high crest factor that's usually a problem for radio communications is not an issue for VLC when intensity modulation and direct detection is used.

2009

Scottish Enterprise funds a project named "D-Light". Project work begins in January 2010 at the University of Edinburgh. Project objectives mention the use of off-the-shelf LEDs to achieve 100Mbps under normal lighting, BER of 10E-4, 1-4m range and real-time signal processing. Thus, the project has a strong practical rather than academic focus.

Jul
2011

At TEDGlobal event, Harald Haas coins the term *Li-Fi*, which stands for *Light Fidelity*. He explains four limitations with radio communications: capacity, efficiency, availability, security. He claims that Li-Fi overcomes these limitations. A few months later, D-Light project member Gordon Povey announces that project goals have been met with a measured data rate of 102.5Mbps.

Oct
2011

The Li-Fi Consortium is launched to promote Optical Wireless Communications (OWC). While Visible Light Communications (VLC) considers only the visible spectrum, OWC includes infrared as well.

2012

Italian researchers achieve 1Gbps data rate using Discrete Multitone (DMT) modulation.

Feb
2015

Researchers at the University of Oxford report Li-Fi speed of 224Gbps by aggregating six channels with each channel achieving 37.4Gbps.

Sep
2015

Harald Haas demonstrates the use of solar cells as Li-Fi receivers while at the same time converting light energy to electrical energy.

Mar
2017

Company PureLiFi claims to deliver 45Mbps with its Li-Fi products.

Jul
2017

IEEE 802.11 Light Communication(LC) Topic Interest Group (TIG) publishes a report on OWC. The report points out why 802.11 may be better than alternatives that other IEEE groups have proposed. This is an input to IEEE 802.11 Working Group (WG) for potential standardization of OWC.

Li-Fi


# Optimizing Pandas

## Summary

Performance of Pandas can be improved in terms of memory usage and speed of computation. Optimizations can be done in broadly two ways: (a) learning best practices and calling Pandas APIs the right way; (b) going under the hood and optimizing the core capabilities of Pandas. This article covers both these aspects.

As a general observation, Pandas was designed for data analysis and usability. On the other hand, NumPy was designed for performance. However, there are some techniques to make Pandas more performant without sacrificing its ease of use. There are also third-party libraries that build on or work with Pandas to achieve better performance.

## Discussion

### What's an essential approach towards performant Pandas code?

Perhaps the most important rule is to avoid using loops in Pandas code. Looping over a Series or a DataFrame processes data one item or row/column at a time. Instead, operations should be **vectorized**. This means an operation should be performed on the entire Series or DataFrame row/column. Developers should think of all operations as matrix computations that can be parallelized.

The worst or slowest approach is to use `df.iloc[i][j]` within a `for` loop. A slightly better approach is to use `df.iterrows()` that returns a tuple containing a row index and a Series for the row. Even better is to remove loops completely and use `df.apply()` that takes as first argument a function that's applied to each row or column. This internally uses Cython iterators.

By far, the best approach is to use vectorization. For example, if `addr` is a Series containing addresses, `addr.str.upper().str.replace('.', '')` is applied to all items "at once". Register size and the computer's instruction set determines how many items can be parallelized.

### For data manipulation, what are some techniques for faster code?

Indices are commonly used in Pandas for lookup or merging two datasets. It's faster to **merge based the index**. For example, `listings_.merge(reviews_, left_index=True, right_index=True)` is faster than `listings.merge(reviews, on='listing_id')`, given that `reviews_ = reviews.set_index('listing_id')` and `listings_ = listings.set_index('listing_id')`.

When chaining multiple operations, the order is important. For example, it's faster to **filter first and then merge**. Thus, `listings.merge(reviews[reviews[listing_id].isin(listings[listing_id])], on='listing_id')` is faster than `listings.merge(reviews, on='listing_id')`.

### What are some best practices for handling datetime data type?

In general, adopt vectorization over explicit loops. If this is not possible, prefer `df.apply()`, `df.itertuples()` or `df.iterrows()` in that order.

If datetime values are stored as object data type, use `pd.to_datetime()` method to convert to datetime type. Explicitly passing `format` argument to this method speeds up the conversion.

In electricity billing, suppose different tariffs are applied based on time of the day. We can use `df.isin()` to selectively apply the tariff to data subsets. An even better way is to use `pd.cut()`. We can reduce execution time further by converting data to NumPy arrays. In this example, it's also convenient to use the datetime column as the index.

### What are some techniques to improve Pandas performance?

There are a few known techniques to speed up Pandas:

    + **Cython**: Cython is a superset of Python. It's Python code with additional type information. This is translated into optimized C/C++ code and then compiled as Python extension modules. Passing NumPy ndarray (rather than creating and passing Pandas Series) into Cython functions gives further improvement.
    + **Numba**: This is a just-in-time (JIT) compiler. With a few annotations to Python code (`@numba.jit`, `@numba.vectorize`, etc), runtime performance can come close to C/C++ or Fortran. There's support for both CPU and GPU hardware. However, as of Numba v0.20, optimizations work on NumPy arrays and not Pandas objects.
    + **`eval()` and `query()`**: For datasets larger than 10,000 rows, `pandas.eval()`, `DataFrame.eval()` and `DataFrame.query()` evaluate expressions faster. Only some expressions can be optimized. For example, `is`, `lambda`, `yield`, comprehensions and generator expressions can't be optimized. Package *numexpr* needs to be installed.Pandas User Guide documents all the above techniques with useful code examples.

Particularly for large datasets, Pandas official documentation recommends *numexpr* that uses multiple cores, smart chunking and caching; and *bottleneck* that uses Cython routines to speed up some types of `nan` evaluations.

### How does Pandas compare against NumPy in terms of performance?

NumPy is faster than Pandas because most of the calculations happen using precompiled optimized C code. Although Pandas makes use of NumPy, it does a lot of housekeeping: tracking data types and indices, and checking for errors. This makes Pandas slower than NumPy. Therefore, one way to speed up Pandas code is to convert critical computations into NumPy, for example by calling `to_numpy()` method.

One study on selecting a data subset showed NumPy outperforming Pandas by 10x to 1000x, with the gains diminishing on very large datasets. Regardless of DataFrame size, Pandas paid an initial penalty 1ms. Similar results were seen when taking square root of numbers.

Within NumPy, operations within an array are faster than operations that span multiple arrays. Thus, `a1 + a2` is slower than `np.sum(a_both, axis=1)` if we reorganize the data as `a_both[0, :] = a1` and `a_both[1, :] = a2`.

### How do I deal with large datasets in Pandas?

Pandas does in-memory analytics, which makes it difficult to handle large datasets that don't fit in system memory. One approach is to **load less data**, such as loading only columns needed for analysis. For example, `pandas.read_csv()` has argument `usecols` to do this.

Use **efficient datatypes**. Text data of low cardinality are those with lot of values but only a few unique values. Storing each value as a complete string is memory inefficient. Instead, they should be stored as categorical data. Data with all unique values will not benefit from such a conversion. Numerical values can be downcast to smallest types using `pandas.to_numeric()`.

**Chunking** is a technique to split the dataset and process in chunks. This works best when chunks require little coordination. Some methods such as `pandas.read_csv()` has argument `chunksize` to do chunking. Group-by operations are typically harder to do in chunks.

Where **complex expressions** are involved such as `(df.A < 0.5) & (df.B < 0.5)`), each sub-expression creates temporary memory. This can be avoided with `pd.eval()`, `df.eval()` and `df.query()`. This improves performance only on large datasets.

### How does Pandas store a DataFrame under the hood?

Pandas groups columns of the same type into what is called a **block**. A DataFrame is actually stored as one or more blocks. Using metadata, these blocks are composed into a DataFrame by a **BlockManager**. Thus, only a block is contiguous in memory.

It's possible to inspect the blocks using `df['A'].values.base` for column A. The output will also show values of other columns in that block. To see the internal storage of all blocks, call `df._data`.

Referring to the figure, slicing with `df.loc[:'2015-07-03', ['quantity', 'points']]` doesn't involve data copy since both columns are of the same data type. But any cross-dtype slicing will typically involve copying. Another performance killer is appending a row to a DataFrame. This would result in reallocating memory and copying every block. Thus, it's better to concatenate two big datasets rather than append one to the other row by row. When a new column is added, block copy is deferred until an operation requires it.

### Which third-party libraries help improve Pandas performance?

Indexing involves lots of lookups. **klib** is a C implementation that uses less memory and runs faster than Python's dictionary lookup. Since version 0.16.2, Pandas already uses klib.

To run on multiple cores, use **multiprocessing**, **Modin**, **Ray**, **Swifter**, **Dask** or **Spark**. In one study, Spark did best on reading/writing large datasets and filling missing values. Pandas did best for group-by operation. An advantage of Swifter is that it decides what to use (vectorization, Pandas apply or Dask) depending on the dataset size.

Swifter can also be used along with Modin. In fact, Modin partitions data in an optimal way but can work with other libraries that perform parallel execution.

Dask can use multiple threads or processes on a single machine or a cluster of machines. Combining **Numba with Dask** is even better.

PySpark is suitable for very large datasets but Pandas offers better APIs. A suitable approach is to use **Spark with Apache Arrow**. Arrow is an in-memory columnar format that helps to efficiently transfer data between Spark and Pandas. Based on Arrow, **PyPolars** is a highly performant DataFrame library.

## Milestones

Jul
2011

Wes McKinney, creator of Pandas, writes about **BlockManager** in a blog and explains how it works. It's the internal data structure in Pandas. A **block** is really a "homogeneously-typed N-dimensional NumPy ndarray object". Gluing together columns of the same type into a single block is called **consolidation**. In the early days of Pandas (0.1 and 0.2) a DataFrame was stored using Python `dict` until the BlockManager took over.

Jun
2015

With the release of Pandas 0.16.2, klib package is used to improve performance of indexing. This takes less memory and runs faster than Python's dictionary. This should not be confused with klib (first released in April 2020), a library that simplifies visualization of DataFrame.

Sep
2017

McKinney notes on his blog that Pandas requires typically **5-10 times RAM as the size of the dataset**. This is bad for analytics on large datasets. What's needed is a columnar format optimized for analytics with zero-copy access. This is where **Apache Arrow** fits in. Arrow 0.1.0 was released in October 2016.

Nov
2018

As a library for parallel computing in Python, **Dask 1.0.0** is released. Pandas DataFrame becomes Dask DataFrame whose processing can be scheduled on a multiple threads/processes of a single machine or distributed across machines. Dask takes care of the task scheduling. The release of Dask 0.2.0 can be traced to January 2015.

Jan
2020

**Pandas 1.0.0** is released. This provides a number of performance improvements. We mention a few: DataFrame arithmetic and comparison operations with scalars; indexing with a non-unique IntervalIndex; initializing a DataFrame using a range; `DataFrame.select_dtypes()` uses vectorization instead of a loop; comparing a Categorical with a scalar and the scalar is not found in the categories.

Jul
2020

**Swifter 1.0.0** is released. Swifter "efficiently applies any function to a pandas dataframe or series in the fastest available manner". If the function can be vectorized, it vectorizes it. If not, it selects the best of Pandas, Dask or Swifter apply method. Swifter can be traced to April 2018 when version 0.1 was released.

Sep
2020

Package **PyPolars 0.1.0** is released. It's implemented in Rust and uses Apache Arrow as its memory model. It supports lazy and eager evaluations. On datasets larger than 10,000 rows, it outperforms both pandas and pydatatable (another DataFrame library).

Jan
2021

**Pandas roadmap** as on this date notes a few items related to performance. Apache Arrow may be explored as an in-memory array library. At the least, Pandas would interoperate better with Arrow. BlockManager and 2-D blocks are complex. They could be replaced with a simpler collection of 1-D arrays. BlockManager uses both labels and positions for indexing. In future, it might use only the latter. Where Pandas accepts user-defined functions, developers should be able to provide Numba-jitted functions.

Optimizing Pandas


# Factor Analysis

## Summary

When analysing data containing many measured variables, it may happen that some of the variables are correlated. This could be because they share an underlying influence or **common factor**. It would be useful to understand how these variables are correlated and seek an intuitive explanation about what's common among them. This will also simplify further analysis by reducing the dataset into fewer variables or factors. This is what factor analysis tries to achieve.

A good factor is intuitive, easy to interpret, has a simple structure and lacks complex loadings. Factor analysis is in some sense an art. It's been said,

> Factor analysis is not a purely statistical technique; there is always a certain amount of guesswork in it... Factor analysis is certainly a very treacherous tool in inexperienced hands.

## Discussion

### Could you provide an intuitive explanation of Factor Analysis?

Suppose a village survey is conducted and the questionnaire includes 500 questions. This survey therefore results in a large dataset of 500 variables. However, we may discover that many of the variables are correlated. We can probably put related variables into groups such as income, education, healthcare, cleanliness, etc. These are called **factors**. Now our analysis becomes easier from many variables to fewer factors.

Let's say we measure students' abilities in terms of four variables: vocabulary, grammar, arithmetic, and geometry. We can make a hypothesis that vocabulary and grammar abilities must be correlated. Likewise, arithmetic and geometry abilities must be correlated. We can therefore hypothesize two factors: language ability and math ability. Subsequent analysis on the data can either confirm or reject the presence of these factors and to what extent they relate to the variables. In fact, the factors themselves could be correlated with each other and we might identify a single factor that we can call academic ability.

### What are latent variables and factor loadings in Factor Analysis?

What we call factors are in fact *latent variables*. These are variables that can't be measured but in fact influence variables that are measured.

The coefficients of latent variables are called **factor loadings** with respect to a measured variable. In other words, the extent to which a variable is associated with the factor is quantitatively expressed by its factor loading. It's possible that a measured variable is influenced by more than one factor.

To give an example, when income, education and occupation are correlated, the common factor could be "individual socioeconomic status" (F1). On the other hand, house value, neighbourhood crimes and amenities can point to another factor "neighbourhood socioeconomic status" (F2). Consider a loading of 0.65 between income and F1; and a loading of 0.48 between occupation and F1. This implies that F1 influences income more strongly than occupation.

An absolute value of 0.4 or higher can be considered as a high loading.

### What are the main types of Factor Analysis?

The two main types of FA are:

    + **Exploratory Factor Analysis (EFA)**: This is used when we wish to summarize data efficiently, when we want to know how many factors are present and their associated factor loadings. EFA is about revealing patterns in the relationships among variables.
    + **Confirmatory Factor Analysis (CFA)**: This is used when a researcher starts with one or more hypotheses. Each hypothesis may state the presence of certain factors. Analysis on measured data must prove or disprove each hypothesis. A graphical representation of a hypothesis is called *path diagram*. CFA produces *fit statistics* that are used to confirm if data fits a particular hypothesis.**Structural Equation Modelling (SEM)** is similar to CFA but allows us to test complex hypotheses about the structure of variables. SEM may be seen as a method to do CFA.

### What are the some methods of doing Factor Analysis?

All methods of factor analysis are looking for correlations among variables. FA is usually done in one of these ways: Principal Component Analysis (PCA), Principal Axis Factoring (PAF), Ordinary or Unweighted Least Squares (ULS), Generalized or Weighted Least Squares (WLS), Maximum Likelihood (ML). Other methods include Image Factoring (based on ULS) and Alpha Factoring.

PAF is considered the conventional technique. It uses eigenvalue decomposition of a correlation matrix. ULS is considered one of the better methods. It produces the Minimum Residual (MinRes) solution. One study that compared some of these methods found that ULS gave accurate results.

### What are some assumptions for Factor Analysis?

Variables have to be correlated but there shouldn't be perfect multicollinearity among the variables; that is, one variable cannot be predicated accurately from other variables. Data shouldn't have outliers. We assume interval data.

Non-linearity is not allowed but non-linear variables can be transformed to linear ones before applying factor analysis. In fact, in the discipline of statistics, factor analysis is considered as a part of **Generalized Linear Model (GLM)**.

### Why do we do Factor Rotation?

Sometimes we will find that a variable has high factor loadings due to more than one factor. This makes it difficult to interpret the factors. Since factor models are not unique, factor rotation allows us to find another factor model that can perhaps be interpreted better.

There are two rotation types:

    + **Orthogonal**: This uses the *loading matrix* that represents the correlation between variables and factors. In this type, the rotated factors remain orthogonal to one another.
    + **Oblique**: This uses *factor correlation matrix*, *structure matrix*, *pattern matrix*, and *factor coefficient matrix*. In this type, the rotated factors are allowed to become correlated. Oblique rotation may be considered as a subset of orthogonal rotation. If data clusters are in fact uncorrelated, then an oblique rotation will result in orthogonal factors.There are different methods to perform these rotations. For orthogonal rotation, we have *Quartimax*, *Varimax* and *Equamax*. For oblique rotation, we have *Oblimin* and *Promax*.

### Isn't Factor Analysis the same as Principal Component Analysis?

Both PCA and FA achieve dimensionality reduction while minimizing information loss. Both appear to use similar techniques of extraction, interpretation and rotation to reduce many variables to fewer components or factors. Yet, they are fundamentally different.

PCA extracts maximum variance into the first component, then extracts maximum variance into second component, and so on. Factors in FA have no such order. Factors identify *common* variance among variables. For this reason, FA is also called **Common Factor Analysis (CFA)**. FA doesn't capture error or unique variance whereas PCA considers all the variance.

If variables are uncorrelated, PCA will still find suitable components but EFA will be unable to identify useful factors. It's been noted that as the number of variables increases (at least 40 variables), results from PCA and EFA tend to come closer.

### Considering PCA and FA, how are variables related to components or factors?

Factors cause variables. Components are aggregate of variables.

In PCA, components (C) are a linear combination of variables (Y). FA aims to identify latent variables or factors (F). Latent variables themselves can't be measured directly but are seen to cause or influence the measured variables. The extent of influence (b) is called **factor loading**. While components in PCA explain all of the variance in data, factors may not explain all the variance in a variable, thus resulting in a term that's unique (u) to each measured variable.

Mathematically,

$$FA:\ Y\_1=b\_1F+u\_1;\ Y\_2=b\_2F+u\_2;\ Y\_3=b\_3F+u\_3;\ Y\_4=b\_4F+u\_4\\PCA:\ C=w\_1Y\_1+w\_2Y\_2+w\_3Y\_3+w\_4Y\_4$$

### What tools are available to perform Factor Analysis?

In R language (which is free and open source), factor analysis can be performed easily thanks to the `psych` package. EFA can be performed using your choice of method: MinRes, PAF, ULS, WLS or ML. The `scree` function can help in determining the number of significant factors. An alternative to this is **parallel analysis** that can be done using the `fa.parallel` function.

An example of a commercial product is JMP of SAS. The Multivariate platform of JMP can do both PCA and FA. SPSS is another commercial product that can do both PCA and EFA.

## Milestones

1884

The genesis of factor analysis is in human personality psychology: to identify attributes and then categorize them into a structural model. Francis Galton uses a dictionary to identify terms that describe personality. However, he fails to come up with a model.

1901

Spearman gets interested in the work of Galton. In a paper published in 1904, he uses the terms *factor* and *loadings*, although he doesn't describe the methods he used for factorizing.

1934

L.L. Thurstone, considered the *father of factor analysis*, uses 60 terms across 1300 subjects to arrive at five broad factors. He doesn't pursue his analysis further. R.B. Cattell identifies at least a dozen factors in the 1940s but it's Donald Fiske who shows that they reduce to five factors.

1960

Harry Harman introduces **Minimum Residuals (MinRes)**, an approach to factor analysis via least squares.

1966

To determine how many factors to retain, R.B. Cattell proposes a graphical method called the **Scree Plot**, a plot of eigenvalues vs. factors.

1969

The term **Confirmatory Factor Analysis (CFA)** is introduced. Prior to this, factor analysis was exploratory in nature but the "exploratory" prefix was not used.

1980

Following Fiske and other researchers after him, it's in the 1980s that the **Big-Five** factor structure to describe human personality finally takes shape. This goes to prove that finding the correct number of factors, and identifying those factors, is not a trivial problem.

Factor Analysis


# Functions in C++

## Summary

A function is a block of code that typically performs a specific task. Functions help manage complexity in a large program by breaking it down into smaller manageable parts. Functions make programs easier to read, debug and understand. They introduce modularity into a program.

Within a given program scope, each function has a unique name. A function can be called with arguments. After performing its operations on those arguments, a function can return a value. Arguments and return value are optional.

C++ adopts and extends C functions in terms of syntax and functionality. Since C++ is object-oriented, we also have member functions that are part of classes. C++ standard libraries supply a number of functions that developers can use. In addition, developers can define and call their own custom functions. Every program requires and starts execution from the `main()` function.

## Discussion

### What are the different components of function?

A function consists of three components:

    + **Function Prototype/Declaration**: Declares the function before it's used. The prototype tells the compiler about the function's return type and the arguments passed, which is used to validate function definition and function calls. Arguments need not be named in the prototype. The function declaration itself is not necessary if the function is defined before it is called. Syntax: `return_type function_name (parameter_list);`
    + **Function Definition**: The block of statements enclosed within curly braces. Also called function body. The parameters present in the function definition are called *formal parameters*. Syntax: `return_type function_name (parameter_list) {…}`
    + **Function Call**: Actual call to the function for execution of its body. The parameters present in the function call are called *actual arguments*. Actual arguments replace the formal parameters at the time of the function call. Syntax: `function_name (parameter_list)`The figure shows prototypes of function `rad2deg` and member functions `Sector::Sector` (constructor) and `Sector::degrees`. Code shows the common practice of putting declarations in header files and definitions in CPP source files. When `sector.cpp` is compiled, compiler sees the prototype of `rad2deg` via the inclusion of `utils.h`.

### How is a C++ function different from a C function?

In C, if function `func` is called before it's declared, compiler assumes the implicit declaration `extern int func();`. C++ requires explicit function declarations. In terms of function definitions, C++ extends the C syntax with try block for exception handling. For member functions, C++ adds constructor initializer, and `=default;` and `=delete;` specifiers.

In C++ function definitions, parameters need not be named but C requires that all parameters be named. Unnamed parameters can't be referenced in function body. They're simply placeholders for future code evolution.

In old C, functions could be called by any number or type of arguments, which sometimes resulted in undesired output. For better type checking and safer code, standard C/C++ introduced the prototype. A C++ function prototype with empty argument list implies zero arguments but in C this could mean an indeterminate number of arguments. To denote an empty list consistently in both C and C++, we could use "void" instead, `func_name(void)`.

### Could you describe C++ storage class specifier, function specifier and return type specifier?

Storage class specifiers determine linkage. If `static`, such a function is visible only within the current translation unit, which is typically the current file. Thus, other files can't access this function. In C, `inline` specifier also makes the function static. In C++, use of `static` is deprecated. Instead, an unnamed namespace should be used. To call a function from other translation units, `extern` can be used, which is also the default when no storage class specifier is mentioned.

There are many function specifiers. The `inline` specifier inserts code in-place. Member functions can be defined as `virtual`. `constexpr` enables compile-time optimization. `_cdecl` specifies C linkage conventions. `_Export` makes the function available to other modules. `explicit` disables implicit conversions involving constructors. `_Noreturn` indicates that function doesn't return to the caller.

A function return type can be any value except an array or a function. Pointers to an array or a function may be returned. In C++, a reference can be returned and `void` if nothing is returned. When return type is not specified, by default C functions assume `int` is returned.

### Could you explain use of specifiers virtual and inline on functions?

Specifier `virtual` is applicable to member functions. Virtual functions in C++ help implement *runtime polymorphism*, which is about designing a unified interface but with different runtime behaviours. A virtual function in the base class presents a unified interface for derived classes and their users. A derived class can override the base class virtual function with its own specialized behaviour. The virtual specifier tells the compiler to check the type of object at the runtime and then call the correct derived class function.

A pure virtual function is a virtual function without an implementation. It's assigned to zero, `virtual int func () = 0;`. Such a class, called an **abstract class**, can't be instantiated. It's derived class can be instantiated provided the latter defines an implementation of all pure virtual functions of its base classes.

Specifier `inline` replaces the function call with the function body. This avoids the runtime overhead of a function call at the expense of program size. Member functions defined within class declarations are automatically defined inline. Inline is only a suggestion to the compiler. Compiler is free to ignore it.

### What is a const and constexpr function?

The `const` *keyword* in C++ is used to prevent accidental modification of the object that it is applied to. The `const` keyword always binds to the left. In case, there's nothing on the left, it binds to the right. We show two examples:

    + `const T& func();`: Function returns a constant reference to object of type `T`.
    + `int T::func() const;`: Declares a constant member function of class `T`. This function can't modify data members of the class. Note that `const` keyword comes after the function identifier and parameters. In general, this is called *cv\_qualifier\_seq*, referring to `const` and `volatile` member functions. It's part of the function declarator.The `constexpr` *specifier* specifies that the particular value can be evaluated at compile time. By computing the values at compile time, runtime performance can be improved. However, `constexpr` is only a suggestion to the compiler. The compiler may choose to ignore it, particularly when the function doesn't satisfy conditions to yield a constant expression.

In conclusion, `constexpr` is for optimization whereas `const` prevents modification of class data members. `constexpr` can be used for non-member and member functions but `const` is applicable only for member functions.

### What are the ways to pass arguments to a function?

C++ supports three argument-passing mechanisms:

    + **Call by Value**: Value is copied from the caller and passed into the function. The argument variable at the caller and that within the function occupy different memory locations. The function works on its own copy and any changes are visible only inside the function. The caller's argument variable is not affected.
    + **Call by Reference**: Reference of the caller's argument variable is copied into the function. Due to the reference, the function "refers" to the same object as the caller. If any changes are made, the original value is modified and caller sees those changes.
    + **Call by Pointer**: This is similar to call by reference, except for a legacy syntax coming from C. Pointer contains a low-level memory location of the object. When a pointer is passed, the function "points" to the same object as the caller. If any changes are made, the original value is modified and caller sees those changes.A function can use a mix of the above: some arguments are passed by value, some are by reference and the rest by pointer.

### What's the difference between function overloading and function overriding?

In C++, **function overloading** is a concept in which two or more functions can have the same name but different number or types of parameters. Function overloading is not possible when two C++ functions differ only by the return type. It's achieved at compile time. Function overloading is possible with member functions as well.

In the case of inheritance, when a derived class overwrites a virtual function in the base class, the concept is referred to as **function overriding**. When function calls are made on base class objects, execution should call the correct function of the applicable derived class. This information is either not available or hard to achieve at compile time. Thus, function overriding is achieved at runtime.

Where a derived class overwrites a non-virtual function in the base class, then the concept is called **name hiding**. This is achieved at compile time.

### What's operator overloading in C++?

Examples of operators are `+`, `*`, `>>`, and many more. When such operators are applied to standard types such as integers, strings or floating-point numbers, C++ uses different underlying implementations. Thus, addition of two strings is a different function call compared to addition of two integers. We may say that the operator is overloaded. This is similar to function overloading where the overloaded functions differ by the number or types of arguments. An overloaded operator is called an **operator function**.

Suppose we define a custom class named `Complx` and wish to add two objects `c1` and `c2` of this class. This is where operator overloading helps. We can create member function of prototype `Complx operator+(const Complx&) const;`. This function would do the actual addition. The function is invoked by `c1 + c2`, where `c2` is passed as argument to member function `c1`.

### What's a function template in C++?

Suppose we need a function to add two integers and another function to add two floating-point numbers. We can write two uniquely named functions. Or we can write two functions of same name (function overloading) but with differing argument types. With C++ templates, we can achieve this with a single function template. They also enable type-safe code.

Templates in C++ save developers the trouble of writing duplicated code for different data types. The parameters used in the template definition are of a generic data type and can be replaced by actual ones.

Templates are instantiated at compile time. The source code contains only one function template but the compiled code contains many functions, each specific to its data type. Instantiation can be implicit or explicit. Developers can also specialize the template with alternative implementations for specific combinations of template arguments.

### What's a function pointer and what's its usage?

A **function pointer** is used to store the address of a function. With a function pointer, we can dynamically select at runtime a particular function to execute, pass as arguments to other functions or use as return types. It's prototype in C++ takes the form `type(*ptr)(arg_list);`.

Consider the example `int (*g)(int a);`, where `g` is a pointer to a function that accepts one integer argument and returns an integer. The extra bracket around the pointer name is necessary because in its absence the compiler may interpret it as a function that returns a pointer to an integer.

A function pointer can be a function argument, `void func(void (*f)(int)) { (*f)(); }`, where `f` is the function pointer. Given `void print(int a);`, `func(print)` is a valid function call.

C++ also supports **references to functions**. The above example becomes `void func(void (&f)(int)) { f(); }` and the function call is `func(print)`.

### What are some best practices when using C++ functions?

Following best practices will lead to clean maintainable code:

    + A function should do only one task. A function that does multiple tasks is hard read, edit and unit test.
    + Use inline functions instead of `#define` pre-processor directive. Don't inline large functions.
    + Specify the return type of a function explicitly. Public functions mustn't return a reference or pointer to a local variable.
    + When using function overloading, have semantic consistency. All overloaded functions should do the same task.
    + Use `const` on an argument if function is not supposed to modify it. Rather than call by value, use `const &` to avoid the runtime overhead of calling the copy constructor and destructor.
    + Prefer call by reference since the code is more readable than call by pointer.
    + Don't assume call arguments will be processed in left-to-right order. No ordering is specified in C++ standard.
    + Code should be self-explanatory. It makes the code easy to understand. Function and variable names should be descriptive of what they do. Comments should be given wherever needed.

## Milestones

Apr
1979

At Bell Laboratories, inspired by Simula, Bjarne Stroustrup conceives the idea of *C with Classes*. This is later renamed to C++ (1984).

1980

By now, the language includes base and derived classes with member functions, constructors and destructors, type checking and conversions for function arguments, inline functions, default arguments, and operator overloading for the `=` operator. For type checking, the language introduces `f(void)` as a better syntax to C's relaxed use of `f()`. Functions `call` and `return` are available but these are not accepted into C++ standard in the late 1980s.

1983

**Virtual functions** are introduced into the language. With this feature, its creators start calling it a language for object-oriented programming.

1985

Stroustrup publishes first edition of his book *The C++ Programming Language*. This becomes the reference for C++ 1.0. Features described in this book include function and operator overloading, references, the keyword `const`, and scope resolution operator. References are introduced mainly to support operator overloading.

1989

C++ 2.0 is released. This includes support for abstract classes via **pure virtual functions**. Among other additions are better resolution of overloaded functions, `static` member functions, `const` member functions, and overloading of `->` operator.

1990

Ellis and Stroustrup publish *The Annotated C++ Reference Manual*. This becomes a starting point for C++ standardization. This introduces **templates**, which is added into the C++ standard in September 1991.

Sep
2011

ISO publishes the C++11 standard, formally named ISO/IEC 14882:2011. This release includes `constexpr` functions, unnamed function objects called *lambdas*, variadic templates (arbitrary number of arguments), template aliases, and `noexcept` to ensure that the function doesn't throw an exception.

Dec
2017

ISO publishes the C++17 standard, formally named ISO/IEC 14882:2017. This introduces a **uniform function call syntax**. A function could be called in the object-oriented way `x.f(y)` or the functional way `f(x,y)`.

Functions in C++


# Data Structures

## Summary

In today's data-driven world, we need to learn to manage data and perform operations on them to yield solutions to everyday problems. Data structures provide an efficient way to store and organize the data. They enable traversal, sorting, and searching through the data in the minimum time possible.

Data structures are the basic building blocks of a program. Choosing the right ones is important. Many languages provide built-in data structures that have their own properties. Programmers need to pick them carefully.

Historically, data structures were created when it became difficult to process and search through large amounts of data, and when it was hard to concurrently process data.

## Discussion

### What are the types of data structures?

Data structures can be broadly classified into linear and non-linear.

In **linear data structures** data is accessed in a linear order. Examples include array, linked list, stack and queue. Array is a collection of values that are stored sequentially. Linked list is a collection of data stored in non-contiguous locations connected by pointers. Stack is similar to a stack of plates. Insertion and deletion happen only at one end. Queue is similar to people waiting for their turn in a line. It has different ends for insertion and deletion.

**Non-linear data structures** store data in a non-linear order. Examples include tree and graph. Trees exhibit a hierarchical structure starting from the root node to the leaf node. Graph is a collection of nodes called vertices connected by links called edges.

### What are the essential characteristics of data structures?

We note the following:

    + **Structure**: Linear or non-linear. Arrays and linked lists are linear since every item has a unique predecessor and successor (except first and last items). Trees, graphs and sets are non-linear.
    + **Access**: Direct (aka random) or sequential. With arrays, items can be accessed directly via an index or key. With linked lists or graphs, an item is accessed by traversing preceding items.
    + **Data Type**: Homogeneous or heterogeneous. Arrays often contain items of the same data type. Heterogeneous data structures contain items of different data types/structures. Classes in OOP are heterogeneous.
    + **Abstraction**: Concrete or abstract. Concrete ones expose the implementation. Abstract ones define the interface but hide the implementation.
    + **Uniqueness**: In sets, each item is unique while other data structures usually allow duplicates.
    + **Operations**: Items can typically be added, removed, replaced or sorted in mutable data structures. Some operations may be constrained in some data structures. For instance, in static arrays items are not added or removed. Items are only appended at the end to queues. Sets have no order and hence items can't be sorted.

### What is the time and space complexity of operations on data structures?

Common operations on data structure include:

    + **Access**: Read or write of an item requires access. If access is sequential via other items, we call it traversing the data structure.
    + **Insert**: Add another item to the data structure.
    + **Delete**: Remove a specified item from the data structure.
    + **Search**: Attempt to find a particular item in the data structure. Basic searching algorithms include linear search, binary search, etc.
    + **Sort**: Some data can store items in a particular order. Sorting algorithms include selection sort and insertion sort. Sorting a data structure in advance may speed up searches.The choice of a data structure usually depends on the algorithm and most common operations on that data structure. If fast direct access is needed, arrays are preferred but not linked lists. If search is common, hash tables are preferred. If middle insertions/deletions are common, single/double linked lists are preferred but not arrays.  Stacks and queues have same performance as linked lists since they're often implemented from the latter.

### What are the implementation aspects of data structures?

**Memory allocation** for data structures can be static or dynamic. Static memory allocation is usually done at compile time and stack memory is used. Memory is freed when variable goes out of scope. With dynamic allocation, heap memory is used. Its size can grow or shrink during the lifetime of the program.

With arrays, items are stored **contiguously in memory**. Hence access is direct. Cache locality improves performance. With linked lists, items are not stored contiguously in memory. Access is sequential via pointers that take up extra space. In C++, `std::vector` is stored contiguously but not `std::list`.

**Mutability** is another aspect. A data structure variable that can't be modified after initialization is immutable. Python's `tuple` and C++'s `const struct` are examples but any variable can be used in an immutable context. In functional programming, it's customary to accept an input data structure and return a modified one without changing the input.

### Could you explain arrays and linked lists?

Array is defined as a list of values stored at contiguous locations. Its elements are accessed using index, i.e. position of elements. The base address is the address of the first element. There are two types of arrays, **one-dimensional array** and **multi-dimensional array**. Two-dimensional arrays are similar to matrices in mathematics. Arrays are used to implement stacks, queues, etc.

Linked list is a collection of nodes connected by links. Each node contains data and a pointer to the next node. They occupy non-sequential locations in memory, which is an advantage over arrays that need a chunk of contiguous space. The starting node is mostly called the head node, and the last node points to NULL.

There are many types of linked lists, some of which include,

    + **Single linked list**: Starts from one end and ends at NULL, with next pointer at each node pointing to the next node.
    + **Doubly linked list**: In addition to the next pointer, each node has a previous pointer that points to the previous node.
    + **Circular linked list**: The last node connects to the head node instead of NULL.

### What are the differences between stacks and queues?

Stack works on **Last-In, First-Out (LIFO)** principle where the element last inserted is deleted first. It has only one end called *top* where both insertion and deletion happen. Inserting an element in the stack is termed as a *push* operation where the top value is incremented. Deleting the element is termed as a *pop* operation where the top value is decremented.

Queue works on **First-In, First-Out (FIFO)** principle where the element first inserted is deleted first. It has two ends called *front* where deletion happens and *rear* where deletion is performed. Inserting an element into a queue is termed as an *enqueue* operation, which increments rear value. Deletion of an element is called *dequeue* operation, which increments the front value.

If there's no more space left and an element is inserted into stack or queue, *overflow* occurs. If there's no element in the stack or queue and deletion is performed, *underflow* condition occurs.

Both stacks and queues can be implemented using arrays or linked lists. Stacks find application in recursion problems, whereas queues are used in scheduling algorithms.

### Could you compare tree and graph data structures?

Both trees and graphs contain nodes and edges. Node is an entity that contains data. Nodes are also called vertices in graphs. Edges connect adjacent nodes, called adjacency in graphs.

A tree with \(n\) nodes has \(n-1\) edges. Path is the sequence of nodes and edges from one node to another. Height of the tree is the path length from the root node to the deepest leaf node. In a graph, the degree of a vertex is the number of edges at that vertex.

Tree nodes are ordered but graph nodes are not. Tree has a root node and therefore a hierarchy is implied. Graph has no root node or hierarchy. In a tree, there's only one path between any two nodes and no loops. Graphs can contain loops and any pair of nodes may have multiple paths.

Folders and files in computer file system are tree nodes. Facebook users and tags are graph nodes and their connections to other friends are the edges. Basically, tree is a minimally connected graph to represent a hierarchical model whereas graph represents a network model.

### What are ADT and CDT in the context of data structures?

An **Abstract Data Type (ADT)** defines the data and the operations that can be performed on that data without specifying implementation details. Stacks and queues are examples of ADT. Developers use ADTs without bothering about the internal implementation of the operations being performed with the data. We call `push()` to insert new data and `pop()` to delete data, but don't care how these functions work. The idea of hiding the complexities and exposing only the essentials is the essence of *abstraction*.

A **Concrete Data Type (CDT)** implements a simple way to organize data. The implementation is not hidden. Developers need to be aware of the implementation to operate on the data structure. CDTs include arrays, records, linked lists, trees and graphs. They can be combined or specialized to form other data structures. CDTs can be used to implement ADTs. For example, a linked list or a linear array may be used to implement stacks and queues.

It's possible to abstract CDTs into ADTs. For instance, arrays and records can be defined as ADTs so that implementation details are hidden.

### When to use which data structure?

Among the known data structures, the correct data structure is the one that is efficient and is well suited to the problem. For that, one needs to know both strengths and limitations of a particular data structure.

Array should be used if the amount of data is known beforehand, as it uses less memory and also offers the advantage of random access. It shouldn't be used if insertion and deletion of data is frequently involved. Linked list is used when constant insertion/deletion time is required and random access of elements is not needed.

Stack is used mainly for the recursion and backtracking problems, whereas queue is used when data is to be added and deleted from both ends.

Trees are more memory efficient and can be used when more than one value has to be searched but require already sorted data. Graphs are mainly used in problems that involve finding the shortest paths, solving the maze game, networking, etc.

### What are the applications of data structures?

Two-dimensional arrays (aka matrices) are used in image processing, mobile contact lists and online ticket booking systems.

Linked lists are used for traversing images in image viewers. A user's webpage history is saved using a linked list. Music players uses a circular list to play all selected songs in a loop.

Stacks are used for converting infix expression to postfix expression and vice versa. Undo and Redo functions in any editor uses the stack. While writing a program, parentheses are matched using a stack.

Queues are used by call centers to schedule calls. Requests to shared resources such as printers are managed with queues. CPU interrupts are handled using queues.

Trees are used in database indexing, game development, and many decision-making tasks in machine learning. Pathfinding algorithms also utilize trees.

Graphs are used in most social media platforms where every user is a node, and to find the shortest path between two users. Shopping websites use graphs to represent user preferences.

### What data structures are provided by programming languages out of the box?

The basic concepts of data structures are the same across all languages, though with different implementations.

Array exists in most languages. Some index it starting from zero and some from one. In C++, `std::vector` is the implementation of the dynamic array. Java uses `ArrayList` whereas Python uses `list`.

Linked list is implemented using a `std::list` container in C++. In Java, the `LinkedList` class implements the `List` interface.

Stack is implemented as `Stack` class in Java. C++ has `Stack` in its template library.

Queue exists as a module in Python. `LinkedList` class in Java offers a `Queue` interface. C++ has `std::queue` for queues.

Trees and graphs don't exist explicitly in any language but can be implemented using fundamental types. For example, a tree is a kind of graph and a linked list is also a type of graph.

## Milestones

1945

John von Neumann uses **arrays** when implementing the merge sort algorithm. The first digital computers during this period use arrays for data tables and vector or matrix operations. Subsequently, the early languages of the 1950s and 1960s such as Fortran, Algol, COBOL and BASIC use one-indexed arrays with support for multidimensional arrays. A later version of BASIC in 1980 uses zero-indexed arrays.

1955

Klaus Samelson and Friedrich L. Bauer introduce the **stack** data structure. They subsequently patent it in 1957. This invention comes about while creating a translator for ALGOL programs. A program is essentially a sequence of symbols but processing them is complex due to operator precedence and the use of parentheses. There's a need to evaluate some symbols later even when they appear earlier in the sequence. It's in this context that the stack data structure is invented.

1956

Allen Newell, Herbert A. Simon and J.C Shaw invent and implement **linked lists** and list processing for the development of artificial intelligence programs. **List processing** shows that programs need not have a fixed memory structure, that data structures can be chosen to suit the problem independent of hardware, and that symbol manipulation is the essence of computing. In 1959-60, this inspires McCarthy to create LISP to realize the full power of list processing.  By early 1960s, some languages use linked lists as their primary data structure.

1960

In an attempt to improve time and space performance for processing data on magnetic tapes, Windley proposes the **tree** data structure. He defines tree as "a network of points arranged in levels" with the root point at the bottom and branches at higher levels. He shows that rearranging data as trees rather than by merging subsequently improves performance on data processing tasks. In 1961, Iverson at the IBM Research Center publishes a notation for trees.

1975

Tarjan analyzes the performance of find and union operations on disjoint sets. Find operations are interspersed with union operations. Sets are represented as trees. He shows that when designing data structures, focusing on worst case running time is less important than **amortized running time**, that is, performance averaged over operations on a long sequence of input. He later invents the data structures **splay tree** (1980) and **Fibonacci heap** (1985), both of which achieve good amortized performance.

Data Structures


# Linked List (Data Structure)

## Summary

Linked list is a dynamic data structure whose memory is allocated dyamically. It provides constant time complexity when it comes to insertion or deletion of element at any position.  It is the second most used data structure after arrays.

Linked list is a linear data structure, meaning that one data point follows another. It's a list of values that could be stored at non-contiguous locations in memory, called nodes, connected by links. Each node contains data and a pointer to the next node. Unlike arrays, linked lists don't allow random access. All access is sequential.

Just as a train has coaches connected in a sequence, so are the nodes connected linearly in a linked list.

## Discussion

### What are the terms associated with linked lists?

Some terms related to the linked list are as follows:

    + Node: a record in a linked list that contains a data field and a reference, a self-referential structure.
    + `next` pointer: the field of a node that contains a reference to the next node.
    + `prev` pointer: the field of the node that contains a reference to the previous node.
    + Head Node: the first node of the linked list.
    + Tail Node: the last node of the linked list.

### What are the different types of linked lists?

**Singly-Linked List** consists of nodes, starting from head node to `NULL`, where each node contains a data field and a `next` pointer.

**Doubly-Linked List** consists of nodes, where each node contains a data field, a `next` pointer and a `prev` pointer.

**Circular Linked List** is similar to a singly-linked list except that the last node instead of connecting to `NULL` connects to the first node, creating a ring.

**Circular Doubly-Linked List** is a mixture of the circular and doubly-linked list which has two pointers `prev` and `next`, and the last node connects to the first node.

**Multi-Linked List** consists of nodes, where each node contains a data field and two or more pointers. A doubly-linked list is a special case of the multi-linked list, which has two pointers that are the exact inverse of each other. These lists are typically used to implement different sorting of the nodes. For example, given student names and their marks, one set of pointers will sort by name and the other set by marks.

### What's the time and space complexity of linked list operations?

The basic linked list operations with their associated complexity are as follows:

    + **Access / Search**: Access is via the head node and additionally the tail node in case of doubly-linked list. There's no random access like in arrays. Access in the worst case requires O(n) time.  Presence of the head pointer and the tail pointer makes access of first and last node in O(1) time.
    + **Insert**: First we need to reach the position at which the node has to be inserted is to be performed. The worst-case time complexity of insertion of a node into a linked list is O(1).
    + **Delete**: First we need to reach the node that needs to be deleted. The worst-case time complexity of deletion of a node from a linked list is O(1).

### How do insertions and deletions in linked lists work?

For insertions, we need to create a node, assign its value, insert it at the required position while updating the relevant pointers. For inserting node at the start, we need to link it to the current head node and then update the head pointer to the newly inserted node. For inserting node in the middle, we need to update the next pointer of previous node and next pointer of the inserted node. For inserting node at end, we have to traverse the whole linked list till we find the last node and then link it to newly created node.

For deletions, we first need to know the location of the node. For deleting node from the start, we need to head pointers to the next pointer of the first node. For deleting node in the middle, we need to link its previous node to its next node. For deleting a node at the end, we update the next pointer of the second-last node to `NULL`.

The above procedures have to be enhanced for doubly-linked lists and circular linked lists.

### What are sentinel nodes in linked lists?

Implementations of linked lists need to handle the special case when the list is empty. Head and tail pointers will be null in an empty list. These need to be handled specially in code. This complicates code and raises the possibility of bugs. To solve this problem, sentinel nodes are used.

Sentinel nodes are nodes at the ends of linked list and don't contain data. An empty doubly-linked list will have two sentinel nodes, called **header node** at the start and **trailer node** at the end. Head and tail pointers will point to these sentinel nodes.

In a singly-linked list, only the header node will be present. Sometimes the term **Header Linked List** is used for such a list. The last node's next pointer could be null or point to the header node.

Though sentinel nodes simplify code, they take up extra space. This may be an issue if the application uses many short lists.

### What other data structures can be implemented using linked lists?

Linked list, being a linear data structure, can be used to implement other linear data structures. In particular, linked list is a concrete data type. Stacks and queues, which are abstract data types, can be implemented in linked lists. A singly-linked list suffices for a stack (LIFO operation) whereas a doubly-linked list in needed for a queue (FIFO operation).

Tree and graphs are non-linear data structures. These can be implemented with linked list that need to be customized for non-linear storage. Algorithms that operate on these will also be different. For example, a node in a binary tree will have one parent and at most two children. A doubly-linked list node can be modified for this purpose: instead of `next` and `prev` pointers we have `left` and `right` pointers to link to the child nodes. Note that with this structure, we can get to the children from a parent but not vice versa.

Any arbitrary non-linear storage (n-ary trees or graphs) can be implemented by introducing more pointers into each node of the linked list.

### How are linked lists implemented in various programming languages?

Most of the languages have built-in singly-linked lists, some have ADTs or templates to build linked lists.

C++ has a `std::list` container to implement linked lists. It is a doubly-linked list with a head node.

Java has the `LinkedList` class that implements the `List` interface. It's a doubly-linked list.

Python does not have a built-in linked list. Python's `list` is a dynamic array.

C# also has `LinkedList` as Java which is a doubly-linked list implementation.

JavaScript has a list() in collection.js which uses doubly-linked list.

### What are the different storage mechanisms for linked lists?

There are two ways to store node data: internal and external. Internal storage stores data within the node. External storage stores data elsewhere and only a reference to that data is stored within the node.

Internal storage is more cache-friendly, uses less metadata and simplifies memory management. External storage needs more memory to store the reference to the data. Getting to the data requires dereferencing. Otherwise, external storage is more generic since any type or size of data can be stored. Same data can be referenced from multiple linked lists. Linked lists with external storage can also be more easily adapted to new requirements.

### How do arrays and linked lists compare in terms of performance?

Array is a linear data structure by way of storing values in contiguous locations. Linked list is a linear by way of storing reference along with value. Array has better locality of reference than linked list and its memory access is predictable than that of linked list. Linked list utilises memory efficiently as it occupies non-contiguous locations limited only by system memory whereas dynamic array requires contiguous space and needs to be resized. .

Array offers random access in O(1) time whereas linked list offers sequential access in O(n) time. When it comes to insertion and deletion of data, linked list is quite flexible and does it in constant O(1) time. However, if insertion or deletion require first searching for the element, the operations become O(n). Insertions and deletions in an array require O(n) at worst if elements need to be shifted.

Linked list occupies more memory as compared to array because each node stores references to its next node.

### How do I optimize linked lists for performance?

System performance is often limited by memory access times and not CPU speeds. Performance is generally improved using data caches and pre-fetching data into the cache. Since elements of a linked list are often stored in different memory locations, caching and pre-fetching aren't effective.

However, it's possible to improve performance by implementing custom memory management. For example, C++ STL allows use of a custom allocation with `list` and `forward_list`. Same allocator is used for all lists of a type, which is fine if there are many small lists to manage. Otherwise, we would need to implement custom linked list with allocator for each instance. One approach is to implement linked list using STL's `vector`. This can be improved with a *compact* operation to make the list more cache-friendly.

**Unrolled linked list** is also cache-friendly. Multiple data elements are stored sequentially in a single node.

For faster searches, **skip list** maintains layers of pointers, each layer skipping some elements. Where data filtering is a common operation, an STL `vector` backed by a bitset is one solution.

### What are some real-world application of linked lists?

A singly-linked list is used when a user views all the images from the gallery as a slideshow. Collision techniques use a singly-linked list to store all the data items in chains that map to the same value in a hash table. It can be used to keep a track of the user's webpage history.

A doubly-linked list is used to implement the undo-redo function in any editor. It is used to implement the next and prev buttons while a user is browsing and wants to visit already visited web pages. A music player also uses a doubly-linked list to store the next and previous track and a circular linked list to play all the selected songs in a loop.

A circular linked list is used to store all the processes in execution by an operating system. It loops over the list repetitively until all the processes exit the system.

## Milestones

1956

Allen Newell, Herbert A. Simon and J.C Shaw invent and implement **linked lists** and list processing for the development of artificial intelligence programs. **List processing** shows that programs need not have a fixed memory structure, that data structures can be chosen to suit the problem independent of hardware, and that symbol manipulation is the essence of computing.

1963

Bobrow and Raphael at the RAND Corporation compare four prominent list-processing languages: COMIT, IPL-V, LISP 1.5 and SLIP. They note that these are good for handling complex data structures requiring dynamic and unpredictable memory allocations. They compare data representation, storage allocation, stack implementations and recursive operations. In general, by the early 1960s, some languages use linked lists as their primary data structure.

1994

**C++ Standard Template Library (STL)** is released with support for the `list` sequence container, which implements a doubly-linked list. This container can be used to implement stacks and queues. For example, `queue<list<int> >` is a queue of integers using `list` container. In the ISO C++ 11 standard (Sep 2011), STL includes a singly-linked list named `forward_list`. This is more space-efficient compared to `list`.

1998

J2SE 1.2 (aka internal version 1.2) is released. This release includes the `LinkedList` class (a doubly-linked list) that's an implementation of `List` and `Deque` interfaces.

Linked List (Data Structure)


# Queue (Data Structure)

## Summary

Queue is an Abstract Data Type (ADT) based on the First-In-First-Out principle where an element inserted first is accessed/deleted/processed first. A queue has two ends called front and rear. Insertion of an element happens at the rear end. Deletion of an element happens at the front end.

A queue can be linear or circular, bounded or unbounded. Implementations can be from arrays or linked lists. Deque and priority queue are special queue types not based on the FIFO principle and serve their own purpose.

People standing in a queue (line) to buy a train ticket at a counter is an example of a real-life queue. The first person in the line is served first and exits the queue. A new person who wants to buy a ticket joins the end of the queue.

## Discussion

### What are the common operations on a queue data structure?

Inserting an element to a queue is called **enqueue** operation. Element is added at the queue's rear end. For an unbounded queue, there's no limit to the queue length. Memory is dynamically allocated when required. An **overflow** condition occurs when one attempts to enqueue into a full bounded queue or when system memory is exhausted.

Deleting an element from a queue is called **dequeue** operation. Often implementations will support **peek** operation to inspect the element at the front of the queue. Whereas dequeue removes the element, peek doesn't.  An **underflow** condition occurs when one attempts to dequeue from an empty queue.

Implementations may include interfaces to check if queue is full or empty. Implementations may allow us to inspect queue capacity (maximum limit) and queue size (number of elements currently in the queue).

Queue implementations maintain front and rear variables or pointers that are essential to most operations.

### How can queues be implemented?

Queue is an abstract data type. ADT defines the data and the operations that can be performed on that data while hiding the implementation complexities. So, a queue is implemented using concrete data types such as an array or a linked list.

For an **array** implementation of queues, we maintain two variables front and rear. After successful insertions, front points to the first element of the array and rear points to the last. In the case of static arrays, once rear reaches the maximum size, overflow occurs. We can solve this problem using dynamic arrays that are resized according to requirement.

For an **linked list** implementation of queues, each node of the linked list contains data and a pointer to the next node. Nodes need not be stored sequentially in memory. Initially when the queue is empty, front and rear point to NULL. After successful insertions, front points to the first node and rear points to the last node. There's no overflow unless the system memory gets exhausted.

### Could you explain the circular queue?

Consider a case in which elements 1-8 are enqueued. The front points to 1 and rear points to 8. Let's dequeue 1 and 2. In the case of a linear queue, the first two memory locations become unavailable. One way to solve this is to shift the remaining elements forward so that last two memory locations become available for enqueuing. But shifting elements this way impacts runtime performance. Without shifting, the first two memory locations becomes unavailable. This is impractical when queues are based on static arrays.

A better option is to use a circular queue. In a **circular queue**, the first element location follows the last location in a circular structure. It was introduced to efficiently utilize the space that was wasted by a linear queue (without shifting). It can be implemented using a circular array, singly linked list, and doubly linked list.

Both linear and circular queues support the same operations but implementations differ.

### What's a deque?

With FIFO queues, there's no quick way to dequeue from the back of the queue. Deque is a **double-ended queue** that solves this problem. It doesn't follow the FIFO principle. It's a very flexible data structure that allows enqueue and dequeue operations from both front and rear ends.

Operations supported by deque include front enqueue, front dequeue, rear enqueue, and rear dequeue. However, deques could be restricted to implement other data structures. For instance, if only rear enqueue and front dequeue are allowed, we get a queue. If only front enqueue and front dequeue are allowed, we get a stack. In general, there are two types of deques:

    + Input-restricted deque that allows insertion at one end and deletion from both the ends.
    + Output-restricted deque that allows insertion at both the ends and deletion from one end.A deque is implemented using a circular array or a doubly-linked list. Deque finds application in the work-stealing algorithm for multiprocess scheduling. Each processor has a deque of tasks. Self-tasks are front dequeued but when a processor becomes free it rear dequeues from another processor's deque.

### What's a priority queue?

A normal queue works on the FIFO principle, but there are some cases in which the elements need to be processed based on priority. For example, when multiple processes or programs are ready to execute, the highest priority one will be scheduled first.

A **priority queue** enqueues elements along with an associated priority value. When dequeuing, the element with the highest priority is processed first. In a max-priority queue, largest value implies highest priority. In a min-priority queue, smallest value implies highest priority.

Priority queues can be implemented with an array, linked list or tree but retrieval of the highest priority element is expensive. Instead, the heap data structure is used to implement a priority queue. **Max-heap** is a binary tree with a special property that the value of each node is greater than or equal to its children. This is used to implement the max-priority queue. Likewise, min-heap implements the min-priority queue.

A node is enqueued or dequeued keeping in mind that the heap does not violate any of its property.

### What's the time and space complexity of operations on queues?

Since all elements need to be stored, the worst-case space complexity for a queue is O(n).

Enqueuing has O(1) average time complexity since only an element is inserted and the rear pointer is updated. With dynamic arrays, occasionally the operation will take longer if resizing is required. Peeking has O(1) worst-case time complexity. Dequeuing has O(1) worst-case time complexity. Dequeuing from a static array where elements are shifted to the front has O(n) worst-case time complexity.

With circular queues, enqueuing and dequeuing have worst-case time complexity of O(1). Queue capacity is fixed and no resizing happens.

With deques, enqueuing and dequeuing from either end have worst-case time complexity of O(1). Where supported, enqueuing and dequeuing from the middle is at O(n). Peeking at either end is at O(1).

With priority queues implemented as binary heaps, enqueuing and dequeuing have worst-case time complexity of O(log n) since elements have to be moved around to conform to binary heap properties. Peeking is at O(1).

### What are some real-world applications of queues?

**Linear queues** are used to handle hardware and real-time system interrupts. First-Come-First-Serve and Round Robin process scheduling algorithms use linear queues extensively. The uniprocessor computer uses a linear queue to store the different applications to be processed in a queue.

**Circular queues** are used to loop over the list of processes in execution by an operating system until they exit the system. They are treated as buffers for maintaining a playlist in music players and listening to songs on loop.

In computer networking, routers and switches use **priority queues** to process packets. Operating system use priority queues for disk scheduling, memory management, semaphores, I/O buffers, and file access requests. Requests to shared resources like printers are managed using linear queues and priority queues if priorities are defined. Well-known algorithms such as Prim's, Dijkstra, and heap sort use priority queues.

Messaging systems use queues. Producers write to message queues. One or more consumers read from those queues. Apache Kafka, Amazon Simple Queue Service (SQS), and RabbitMQ are some examples.

### What are some variants of queue?

Some applications don't consider overflow and underflow conditions as errors. Instead, they wish to wait for an available space when enqueuing into a full queue or wait for new data when dequeuing from an empty queue. This is exactly what the **blocking queue** provides. Implementations may support no waits, indefinite waits or waits with timeouts. It's used to synchronize producers and consumers that interact via the queue asynchronously.

A special case of a blocking queue is the **synchronous queue**. Enqueuing and dequeuing are synchronized. An element can be inserted only if a consumer is waiting to dequeue it. Thus, a synchronous queue has no storage capacity.

Another type of blocking queue is the **transfer queue** in which producers may wait for consumers to dequeue an element. This is useful in message passing applications.

Sometimes we don't want to overwhelm consumers with too many messages. **Delay queue** is another blocking queue that delays dequeuing operations. Only after the delay time expires, dequeuing can happen.

## Milestones

1960

FCFS and Round Robin scheduling algorithms use **queues** for process scheduling in operating systems.

1964

Williams introduces binary heap that forms the basis of **priority queues**. It supports insertion of an element and extraction of the element with minimum value.

1978

Vuillemin presents a type of priority queue that he calls **binomial queue**. It supports primitive operations such as union, update and search.

1995

**C++ Standard Template Library (STL)** is released with support for three sequence containers: `vector`, `list` and `deque`. STL's `stack`, `queue` and `priority_queue` provide restricted interfaces to these containers. In particular, `list` and `deque` can be used for queues; `vector` and `deque` for priority queues. For example, `queue<deque<int> >` is a queue of integers using `deque` container. The library provides heap-related functions that are used in the implementation of priority queues.

2004

J2SE 5.0 (aka internal version 1.5) is released. New collection interfaces `Queue` and `BlockingQueue` are introduced. Many concrete queue implementations are included. Priority queues are implemented using heaps. Blocking queues, delay queues and synchronous queues are introduced. . In 2006, J2SE 6.0 adds new collection interfaces `Deque` and `BlockingDeque` along with concrete implementations `ArrayDeque` and `LinkedBlockingDeque`. In 2011, J2SE 7.0 adds the interface `TransferQueue` along with the implementation `LinkedTransferQueue`.

Queue (Data Structure)