# Hoisting

## Summary

In many languages, variables and functions have to be declared upfront before they can be used later in code. Attempting to use them before declaration would result in an error.

In JavaScript, it's possible to use variables and functions before they're explicitly declared later in code. This feature of JavaScript is called **Hoisting**. In a sense, the declarations are "hoisted" to the top of the code.

While hoisting is a useful feature, in the interest of code readability and maintenance, it's perhaps a good practice to put declarations before they're used.

## Discussion

### Could you illustrate hoisting with an example?

In the code `console.log(x); var x = 1;`, the variable `x` is used before it's declared. JavaScript allows this because of hoisting.

Hoisting is also possible for functions. In the code `add(3, 4); function add(a, b) { return a + b; }`, the function `add()` is called before it's declared. Again, this is possible due to hoisting.

Hoisting can occur in any execution context. JavaScript has three types of execution contexts: global, function, and eval.

### Is hoisting moving a declaration to the top of the scope?

One common (and incorrect) explanation of hoisting is that the JavaScript engine moves variable and function declarations to the top of their scope. Variable declarations are initialized with value `undefined`. In reality, the JavaScript engine doesn't move code to the top this way.

The correct explanation lies in understanding **execution context**. This is the environment in which the code runs. Such an environment encapsulates `this`, variables, objects and functions that are accessible. Execution context is created in two phases: creation phase and execution phase.

During the creation phase, memory is allocated for variables and functions in that scope. The entire function body is in memory but it's not executed. Likewise, variables become available in memory but are not initialized. It's only later, during the execution phase, that initializing variables and calling of functions happen.

### What can be hoisted?

Variables and functions are hoisted. Functions can be called earlier in the code even if their declarations happen later. It's interesting to note that if a variable and a function have the same name (a practice that should be discouraged), variable initialization takes precedence over function declaration. If the variable in uninitialized, function declaration takes precedence.

Function expressions are not hoisted. Function expressions take the form `var a = function () {...}`. Variable `a` is hoisted, which means that it's initialized to `undefined`. Thus, we can't make the call `a()` since the variable is not initialized with the function object. Similarly, arrow functions, which are of the form `var a = () => ...`, are not hoisted.

Class declarations are "hoisted" only in the sense that memory is allocated but not initialized with the class object. With class expressions, the variable is hoisted but it's initialized to `undefined`. For example, `var Square = new Polygon(); var Polygon = class { ... }` won't work because `Polygon` is `undefined`. This effectively means that we can't instantiate objects without putting the declaration first.

### Could you share more details on hoisting variables?

JavaScript has this default behaviour that a variable initialized without a declaration is automatically declared with global scope. This happens during the execution phase of execution context, not during the creation phase. Therefore, such variables are not hoisted. Hoisting applies only to explicitly declared variables. For example, hoisting works for `console.log(a); var a = 1;` but not for `console.log(b); b = 1;`. For this reason, it's recommended to explicitly declare variables.

Variables declared with keywords `let` or `const` are "hoisted" in the sense that memory is allocated during the creation phase of execution context. Thus, during execution phase, JavaScript engine won't complain that they're undefined. However, the engine will still throw a reference error when the code attempts to use them before they're properly initialized. Unlike the `var` keyword, `let` and `const` keywords don't initialize variables with `undefined`. The solution is therefore to declare before using such variables. Hoisting is not useful for `let` and `const`.

## Milestones

1995

**JavaScript** is announced as a dynamic scripting language that can run within a web browser. Two years later, a more standardized form of JavaScript, called **ECMAScript** is published by ECMA International.

Oct
2010

A video on YouTube explains JavaScript hoisting with examples. This shows that the term "hoisting" is in use by 2010, if not earlier.

Jun
2015

ECMA International releases *ECMAScript® 2015 Language Specification*. This specification uses the term "hoisting". It's said that prior standards didn't use the term explicitly. Hoisting provides us a metaphor to think about execution contexts.

Hoisting


# 5G NR RLC

## Summary

The **Radio Link Control (RLC)** sublayer of 5G NR protocol stack interfaces to PDCP sublayer from above and MAC sublayer from below. It interfaces to PDCP via RLC channels and to MAC via logical channels. There's a one-to-one mapping: RLC SDUs belonging to an RLC channel are mapped to a single logical channel.

An RLC entity in a 5G UE has its peer RLC entity in the gNB, or other UEs in the case of NR sidelink communication. An RLC entity can be configured in one of three transmission modes: Transparent Mode (TM), Unacknowledged Mode (UM) and Acknowledged Mode (AM). Each mode supports a set of functions. Mode is configured by RRC and depends on the higher layer service requirements.

## Discussion

### What's the architecture of 5G NR RLC sublayer?

The RLC sublayer can have multiple RLC entities. Each entity is of a specific mode. In the transmit direction, an RLC entity receives RLC SDUs from PDCP on an RLC channel and sends out RLC PDUs to MAC on a logical channel. An RLC PDU contains RLC header plus RLC SDU, but header is absent in TM.

TM and UM RLC entities are configured as either transmitting or receiving. AM RLC entity has both transmit and receive functionality. Thus, TM and UM RLC entities are **unidirectional** whereas AM RLC entity is **bidirectional**.

A single RLC entity is mapped to a single PDCP entity. But a single PDCP entity can send its PDUs to a maximum of eight RLC entities. For example, a bidirectional PDCP entity is mapped to two UM RLC entities, one for transmit and one for receive. Dual Connectivity (DC), Dual Active Protocol Stack (DAPS) and PDCP duplication are scenarios where a PDCP entity maps to more than one RLC entity.

For Integrated Access and Backhaul (IAB), RLC's upper layer is Backhaul Adaptation Protocol (BAP) sublayer, not discussed here.

### Which logical channels are carried by each RLC transmission mode?

Logical channels BCCH, SBCCH, PCCH, and DL/UL CCCH are carried in **TM**. These correspond to system broadcast, sidelink broadcast, paging and SRB0 signalling.

In **UM**, applicable logical channels are DL/UL DTCH, SCCH and STCH.

Signalling other than SRB0 goes in **AM**. The corresponding logical channel is DL/UL DCCH. User plane traffic on DL/UL DTCH is also carried in this mode. Sidelink logical channels SCCH and STCH can go in AM.

Thus, Signalling Radio Bearers (SRBs) are mapped to either TM (SRB0) or AM (non-SRB0). Data Radio Bearers (DRBs) are mapped to either UM or AM, which is configured via RRC signalling.

### What are the main functions of 5G NR RLC?

All three modes can **transfer upper layer PDUs**.

TM RLC entity is the simplest. It doesn't add any header or segment RLC SDUs. An RLC PDU is simply an RLC SDU.

UM RLC entity **segments** RLC SDUs and adds a header to each RLC PDU. In the receive direction, the entity **reassembles** the segments and delivers a complete RLC SDU to upper layer. The receiving entity may **discard** RLC segments and SDUs based on sequence number (SN), reassembly window size and reassembly timer.

AM RLC entity has the important feature of **error correction through ARQ**. The receiving entity sends feedback on STATUS PDUs. This tells the transmitting entity to retransmit PDUs not correctly received. Thus, bidirectional transfer capability, acknowledgments and retransmission distinguish AM RLC entity. It can **detect and discard duplicate PDUs**. It supports segmentation, reassembly and discard functionalities, similar to UM RLC entity. In addition, AM RLC entity can **re-segment** previously lost or erroneous segments before retransmitting them.

### What are data PDUs and control PDUs in 5G NR RLC?

**RLC data PDUs** carry upper layer PDUs. These are named TMD PDU, UMD PDU and AMD PDU for the respective TM, UM and AM entities. Apart from actual data traffic (DTCH or STCH), these PDUs may be carrying control information from upper layers. For example, all RRC signalling on SRBs are treated as data at RLC sublayer.

**RLC control PDUs** are control information generated and consumed at RLC. These are applicable only for AM RLC entities. In Release 15 and 16, the only such PDU is the *STATUS PDU*. It's sent by the receiving side of an AM RLC entity to inform the sending side about correctly received or lost RLC PDUs. Via a feature call **polling**, indicated via the RLC header, the transmitting side can request the receiving side to send a STATUS PDU.

Control PDUs are sent on the same logical channels as the data PDUs of that AM RLC entity.

### Could you describe the main procedures in 5G NR RLC?

**RLC entity procedures** include RLC establishment, re-establishment and release. These are applicable to all transmission modes, though the details vary. When a new radio bearer is configured, RLC establishment happens. RLC re-establishment is initiated by upper layers to recover from errors. In this procedure, timers are stopped and reset; state variables are reinitialized; RLC SDUs, segments and PDUs are discarded. RLC release procedure happens when the RLC entity is no longer needed.

**Data transfer procedures** include specific transmit and receive procedures for TM, UM and AM entities. This includes segmentation and reassembly; handling of sequence numbers, window sizes and timers; interfacing to PDCP and MAC; and more.

**SDU discard** procedure allows a higher layer to request UM or AM RLC entities to discard a particular RLC SDU, provided the SDU or its segment has not yet been submitted to MAC.

**Data volume calculation** is done at RLC for MAC to send Buffer Status Reports (BSRs). This calculation includes RLC SDUs and segments not yet part of RLC PDUs, RLC data PDUs (initial or retransmit) and the STATUS PDU.

### How is 5G NR RLC different from LTE RLC?

**Concatenation** is when multiple RLC SDUs can be packed into a single RLC PDU. Segmentation is when an RLC SDU is segmented and sent on multiple RLC PDUs. LTE RLC can do both concatenation and segmentation whereas 5G NR RLC is allowed to do only segmentation.

This design change relates to how 5G NR RLC sends its PDUs to MAC. LTE RLC concatenates SDUs and sends a single PDU to MAC per Transmission Time Interval (TTI). 5G NR RLC can send multiple PDUs to MAC in any given TTI. 5G NR MAC packs them into a single MAC PDU for transmission. This improves latency since RLC can prepare its PDUs in advance even before MAC informs how many bytes can be sent.

The other difference is that LTE RLC does **in-sequence delivery** of upper layer SDUs. This is not required of 5G NR RLC. For improved latency, an RLC SDU can be delivered as soon as it's received. Any ordering that may be required is the role of PDCP.

### What RLC parameters are configured by 5G NR RRC?

RLC configuration is **per logical channel**. It's independent of numerologies or transmission durations. RRC provides the mapping between logical channel identify, radio bearer identity and RLC configuration.

For TM, there's no configuration required from RRC.

Sequence number field in RLC header can be 6 or 12 bits for UM, and 12 or 18 bits for AM. For SRBs other than SRB0, AM uses 12-bit SN. For groupcast and broadcast in sidelink, UM uses 6-bit SN.

Reassembly timer is also configured for UM and AM. RRC signalling between UE and gNB defines this only for DL UM and AM. For UL UM and AM, this timer is known within the gNB.

Additional parameters configured for AM include poll retransmit timer, status prohibit timer, pollPDU, pollByte, and maxRetxThreshold. Polling parameters and maxRetxThreshold are on the transmit side of the AM RLC entity. Status prohibit timer is on the receive side.

## Milestones

Apr
2017

An early draft of 5G NR RLC specification **TS 38.322** is released. This evolves to version 1.0.0 by September.

Dec
2017

3GPP publishes **Release 15** "early drop". RLC specification TS 38.322 is updated to version 15.0.0.

Mar
2020

As part of Release 16, TS 38.322 version 16.0.0, RLC specification is updated for **V2X sidelink communication**.

Jul
2020

3GPP publishes Release 16 specifications. RLC specification is TS 38.322 version 16.1.0. For AM, poll retransmit timer and status prohibit timer are updated to include **low timer values** of 1ms, 2ms, 3ms and 4ms.

5G NR RLC


# Constrained Application Protocol

## Summary

HTTP is a widely used web protocol but it's not suited for the Internet-of-Things (IoT). IoT is composed of constrained nodes running on 8-bit microcontrollers with small amounts of ROM and RAM. IoT networks have low bandwidth and low availability. *IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs)*, used in IoT, often have high packet error rates and a typical throughput of tens of kilobits per second. Given these constraints, IETF's *CoRE (Constrained RESTful Environments)* Working Group defined **Constrained Application Protocol (CoAP)**.

Whereas HTTP packets might take up thousands of bytes, CoAP packets are only tens of bytes. CoAP handshaking is also lighter since it uses UDP instead of TCP. CoAP focuses on being lightweight and cost effective. It supports networks with billions of nodes. CoAP uses the *REST (Representational State Transfer)* architectural style.

## Discussion

### What are the main features of CoAP?

Since IoT uses the internet, CoAP has been designed very much in the spirit of HTTP that powers the internet. Like HTTP, CoAP is a document transfer protocol. A client requests a resource at a URI (Uniform Resource Identifier) and the server responds. Moreover, RESTful principles are followed; verbs GET, POST, PUT, DELETE are used. Protocol is indicated with `coap://`.

Where CoAP differs from HTTP is that UDP is used for transport instead of TCP. UDP handshaking is lighter and easier to implement on microcontrollers. CoAP header is only 4 bytes. CoAP can also use UDP's broadcast and multicast features. CoAP communication is using connectionless datagrams. Because datagrams are used, SMS and other packet-based protocols may be used. CoAP uses a subset of MIME types and HTTP response codes. There's also a built-in discovery mechanism. For security, *Datagram TLS (DTLS)* is used.

Since there's no TCP, CoAP takes care of message acknowledgements, retries with congestion control and duplicate detection. To keep things simple, these features are optional.

### Could you describe the architecture of CoAP?

The CoAP protocol is closely aligned to the traditional web stack based on HTTP. UDP is used instead of TCP at the transport layer. CoAP uses binary encoding unlike the textual encoding of HTTP but otherwise both are based on RESTful APIs and request-response method. This one-to-one mapping between the two stacks makes it possible to interwork the two protocols via a proxy. Within the constrained network, CoAP is used but elsewhere CoAP messages can be translated into HTTP. Such proxies may reside in the edge of a cloud. It's also possible to bypass such a proxy and deploy CoAP end to end if cloud platforms support this.

We can also think of CoAP as being composed of two logical sub-layers: a lower messaging layer to deal with UDP and asynchronous interactions; an upper request/response layer that deals with the semantics using message codes and response codes.

### What's a typical message flow in CoAP?

It may be tempting to think of sensor nodes as CoAP clients but in fact they are typically CoAP servers. But they could also be CoAP clients consuming information available at other devices on the network. In a typical CoAP flow, clients discover resources available at the server by asking for URI path `GET /.well-known/core`. Server will reply with content of type `application/link-format`. Now that the client knows what resources are available, it can request the content of a specific resource (such as temperature) by calling `GET /temperature`. Server will respond with the value of that resource.

CoAP specifies four message types: Confirmable (CON), Non-confirmable (NON), Acknowledgement (ACK), and Reset (RST). Requests and responses are carried as part of these message types. If reliable transmission is required, CON and ACK are used with timeout and exponential back-off. If unreliable transmission in sufficient, then NON is used. When ACK is used, response may be piggybacked with ACK or sent separately in a CON.

### Where should I use CoAP?

If you have an existing system that is web service-based, then adding in CoAP is a good option. It's good in resource-constrained environments: energy-limited devices, low-bandwidth links, and lossy or congested networks. On congested networks, CoAP/UDP would work but MQTT/TCP may not even manage a complete handshake. CoAP can be used where broadcast and multicast is needed.

CoAP is suited for building something where a device is deployed in "report only" mode. Once deployed, device simply reports data back to a server. Information appliance, control equipment and communication equipment in Smart Home networks need to be low cost and lightweight. Thus, CoAP is suitable for home communication networks.

When deploying on third-party networks where we don't have control over firewalls and blocked ports, CoAP may not be suitable. In this case, HTTP/REST is a good option. Since sensor nodes are CoAP servers, there could be issues with Network Address Translation (NAT). LWM2M, which is based on CoAP, overcomes this by requiring clients to send a probe packet first so that router can make the association. Although CoAP does not require IPv6, it's easiest to deploy on IP networks.

### What's the security protocol used for CoAP?

CoAP uses **Datagram Transport Layer Security (DTLS)** for security. The DTLS-secured CoAP protocol is termed **CoAPs**. DTLS is based on PSK, RPK and certificate security. There are three main elements when considering security, namely integrity, authentication and confidentiality. DTLS can achieve all of them. DTLS solves two problems: reordering and packet lost. It adds three implements:

    + Packet retransmission
    + Assigning sequence number within the handshake
    + Replay detectionUnlike network layer security protocols, DTLS in application layer protects end-to-end communication. No end-to-end communication protection will make it easy for attacker to access to all text data that passes through a compromised node. DTLS also avoids cryptographic overhead problems that occur in lower layer security protocols.

It's been shown that DTLS adds 11% overhead compared to communication without any security. Moreover, since DTLS is error tolerant, it's more vulnerable to be hacked compared to using TLS.

### What are the alternatives to CoAP for IoT messaging?

Among the alternatives to CoAP are Data Distribution Service (DDS), Advanced Message Queuing Protocol (AMQP), MQTT, HTTP/REST, and eXtensible Messaging and Presence Protocol (XMPP). DDS is from Object Management Group. AMQP and MQTT are OASIS standards. XMPP is defined by IETF.

DDS offers a rich set of QoS and can prioritize traffic based on content. It can use either TCP or UDP. It's model is also flexible: publish-subscribe (like MQTT) or request-response (like CoAP).

A study comparing these protocols concluded that MQTT and RESTful HTTP are popular, possibly because they are stable and mature. MQTT is suitable for constrained devices and RESTful HTTP can fit into fog and cloud computing systems. CoAP is also gaining maturity.

### In terms of performance, how does CoAP compare against its alternatives?

A survey that studied the performance of different messaging protocols concluded that CoAP and MQTT show best latency performance. CoAP performs better for high packet losses. CoAP messages can be transferred three times faster than HTTP messages. However, when comparing HTTP/2 with CoAP, HTTP/2 performed better in high congestion scenarios.

CoAP required less bandwidth compared to MQTT. In a network under heavy load, CoAP used less bandwidth than MQTT whereas DDS took up twice the bandwidth of MQTT. CoAP also had higher efficiency, calculated as a ratio of useful information bytes to total bytes at application and transport layers.

In terms of energy consumption, both MQTT and CoAP are efficient.

We should note that MQTT is based on TCP. There's also a variant of MQTT called MQTT-SN that runs on UDP. Comparing CoAP against MQTT-SN may be more appropriate. Likewise, CoAP running on top of TCP (RFC 8323) can be compared against MQTT.

### Could you compare some implementations of CoAP?

There are plenty of open source implementations of CoAP. In Java, we have Californium and jCoAP. In C we have, Erbium, libCoAP, and OpenCoAP. Some OS support CoAP: mbed, TinyOS and Contiki. When looking at implementations, we need to consider if it's for client or server; for constrained devices or server side; for browsers; or for smartphones. Carsten Bormann maintains a rich list of CoAP implementations. A similar list is available on Wikipedia.

A comparison of some implementations in C, Java, JavaScript and Python has looked at feature support, targeted platforms and developer usability. It found that CoAPy did not interoperate with any other implementation. On the server side, C implementations (libcoap, smcp and microcoap) are fast. It also compared CPU and memory usage. For example, sharedlib/client/server ROM in bytes are 296K/33K/22K for libcoap and 383K/23K/18K for smcp. Peak RAM usage for C implementations is typically 3.2KB.

### What are CoAP standards that developers must read?

Among the standards relevant to CoAP are the following:

    + RFC 7252: This is main CoAP standard.
    + RFC 7959: CoAP is for transferring small payloads but if firmware upgrades are needed *block-wise transfers* option becomes useful. Information blocks are sent without maintaining any state.
    + RFC 7641: What if we want to get notified when resource state changes on the CoAP server? This will save clients from polling the server on a regular basis. *Observing resources* is an option that makes this possible.
    + RFC 8323: CoAP over UDP supports reliable delivery, simple congestion control and flow control. What if we want reliable transport? This RFC talks about CoAP over TCP, TLS and WebSockets.
    + RFC 6690: Defines a simple format for exposing resources as a basis for a resource directory.
    + RFC 7390: Using the multicast feature of CoAP, this RFC details how to use CoAP for group communication.Latest developments of the CoRE Working Group are on IETF website. CoRE parameters, including CoAP method and response codes, are registered with IANA. IANA has assigned port numbers 5683 and 5684 for CoAP and CoAPs.

### What are some useful resources for CoAP developers?

For interoperability testing, coap.me is useful. Wireshark v1.12 has support to decode CoAP packets. To translate CoAP to HTTP and vice versa, such as done on a proxy, *crosscoap* can be used. There's also a handy CoAP cheatsheet.

## Milestones

Jun
2014

Internet Engineering Task Force (IETF) standardizes the **Constrained Application Protocol (CoAP)** as RFC 7252.

Oct
2014

OASIS approves **MQTT Version 3.1.1** as an OASIS Standard.

Feb
2015

Internet Engineering Steering Group (IESG) approves **HTTP/2** as a proposed standard.

Feb
2017

Using CoAP as the underlying stack, **Lightweight Machine to Machine (LWM2M)** technical standard is approved by Open Mobile Alliance. LWM2M defines interfaces Bootstrap, Client Registration, Device management and service enablement, and Information Reporting. It gives details on resource identifiers, security procedures, and transport layer binding and encodings.

Feb
2018

IETF publishes **RFC 8323** titled "CoAP (Constrained Application Protocol) over TCP, TLS, and WebSockets" as a Proposed Standard.

Apr
2018

Eclipse Foundation publishes results of a survey on IoT developers. For messaging, more than 60% of respondents use MQTT. CoAP is in the fifth place with only 20% of respondents using it.

Constrained Application Protocol


# Big Data

## Summary

Big data is a collection of massive data sets that are generated and collected through various sources. Big data can be present in structured, semi-structured and unstructured form. It helps in identifying trends and past behaviours in any field. It has applications in many fields including banking, education, Marketing and business.

However, there are certain challenges associated with big data such as lack of storage capacity and processing technologies. High computing power is required to process big data and find meaningful insights from it.

## Discussion

### Which software technologies are used to manage big data?

In traditional computing systems, organizations had to buy infrastructure and maintain them to have proper computing system. It was very costly as well as time consuming. Cloud computing solved this problem. It offered customized computing systems at affordable costs.

Big data requires massive on-demand computation power and distributed storage. It is often present in petabytes which is too large for traditional computing systems. Cloud computing provides scalable on-demand integrated computer resources, required storage and computing capacity to analyse big data. Users can use these resources and end the session when done. They will be charged only for the resources used.

Apache Hadoop, an open source distributed processing framework is used to perform the processing of big data. It uses Map/Reduce algorithm to process large volume of data. It works on divide and conquer method in which a problem is broken down into many smaller parts. Gradually other processing tools such as Apache Spark and MongoDB Atlas were introduced after Hadoop. Often an ecosystem of different technologies are used for big data processing.

### What are various definitions of big data?

According to Edd Dumbill on O’Reilly, big data is data that can't be processed through conventional database systems. The data is too big and increases too fast to fit the structures of database architectures. An alternative way should be chosen to process it.

According to Microsoft Enterprise Insight Blog, big data is the process of applying high computing power – the latest in machine learning and artificial intelligence – to massive and highly complex sets of information.

According to Networkworld, any amount of data that’s too big to be handled by one computer is big data.

According to Cory Janssen's post on Techopedia, big data is a process that is used when traditional data mining and handling techniques cannot find insights and meaning from data sets.

### What are the characteristics of big data?

Characteristics of big data can be defined by 42 V's. Some major of them are as following:

    + Volume: Size of big data is often larger than terabytes and petabytes. Every second more data is stored on the internet than the total data stored on the internet just 20 years ago.
    + Variety: It denotes the type and nature of the data. Technologies such as RDBMS were capable to handle structured data efficiently but they weren't sufficient to process and analyze semi-structured and unstructured data.
    + Velocity: The speed at which data is generated and processed for analysis. Two kinds of velocity related to big data are the frequency of generation and the frequency of handling and processing.
    + Veracity: It signifies the degree of accuracy and reliability of data. Often big data is the accumulation of data collected from various sources, hence it is important to make sure the data collected is accurate and trustworthy.
    + Value: The useful information retrieved from big data determines its value. All the datasets collected in big data not necessarily provide useful information.

### What are sources of big data?

Big data is made up of text, image, video and audio files. Big data is generated through many sources. Some major sources of big data include Internet of Things (IoT) devices, self quantified data, multimedia data and social media data.

IoT data is generated by GPS devices, mobile phones, intelligent clothing, alarms, intelligent/smart cars, mobile computing devices, PDAs, window blinds, window sensors.

Self quantifying data is generated by the measuring individuals' behaviour. Data from wristbands used to monitor movements and exercise and sphygmomanometers utilized to measure blood pressure are examples of self-quantification data.

Multimedia data is generated from various sources such as text, images, and audio and video. Each individual connected to the internet generates this data. This type of data grows exponentially each day.

Social media data is generated by platforms such as Facebook, Twitter, LinkedIn, YouTube, Instagram and so on. This type of data grows at the highest speed. Excessive usage of social media leads to a lot of social media data generated each day.

### How is big data stored and processed?

Data lakes are used to store big data. Unlike typical data warehouses that are commonly built on relational databases and contain structured data only, data lakes can support various data types and are based on Hadoop clusters (an open source distributed framework that manages data processing and storage for big data applications in scalable clusters of computer servers), cloud object storage services, NoSQL databases or other big data platforms. Consistency, availability and partition tolerance are some important factors of big data storage systems.

Often big data environments are made up of a combination of multiple systems. For instance, a central data lake might be integrated with other platforms, including relational databases or a data warehouse.

Technologies such as Hadoop and Spark are used to process big data. Heavy computing power is required to process big data which is provided by clustered systems that distribute processing workloads across hundreds or thousands of servers. Hence, Cloud is preferred choice for big data systems.

### What are use cases of big data?

Companies use big data to understand different consumer patterns and improve their operations to provide better customer support. Companies that use big data make better and faster decisions than companies which don't.

Big data has many use cases in various fields such as finance, healthcare, education, media, IOT and specially business.  Below are some applications of big data:

    + Product development: Big companies such as Netflix and Procter & Gamble use big data to determine customer demand and needs. They find out the key attributes of past successful products/services and use this data to build the new products/services.
    + Predictive maintenance: Analysis of unstructured data such as log entries, sensor data, error messages and engine temperature can help in predicting the lifespan of products.
    + Customer experience: Big data allows organizations to collect data from social media, web visits, call logs, and other sources and improve the customer experience.
    + Fraud and compliance: Big data helps in finding leakage in security systems by finding similar patterns in fraud cases.
    + Machine learning: Big data allows us to teach machines how to perform instead of just programming them.

### What are challenges faced with big data?

One of the major challenges with big data is its volume. The size of big data is increasing at a very high speed. According to Oracle, data volumes are doubling in size about every two years. Current data processing algorithms are not capable of retrieving the required information on time in case of big data storage. They are designed for limited amount of data.

Data is only useful if it conveys any meaning or gives any results. Organizing data in such a way that it gives insights takes a lot of work. Data scientists spend 50-80 percent of their time curating and preparing data before it can actually be used.

Apache Hadoop was the only technology used to handle big data. Soon Apache Spark was introduced. The combination of Hadoop and Spark framework is a better approach for big data processing. Staying up-to date with big data technologies is also an overhead.

Often NoSQL databases are used for big data. NoSQL databases have many advantages such as flexibility, open source, cost effective and scalability. They have certain limitations too such as lack of maturity and consistency related to performance.

### What are some good big data practices?

Big data can be an expensive burden on the organization if its employees don't know how to make insightful decisions from it. Hence, organizations deploying big data strategy should consider investing in their employees' skills and training.

With the increase in collection and usage of data, data misuse also started increasing. European Union employed General Data Protection Regulation (GDPR). It limits the types of data organizations can collect and makes opt-in consent from individuals compulsory for collecting personal data. A similar law is applied in California, called California Consumer Privacy Act (CCPA).

Organizations should focus on their needs rather than rapidly evolving big data technology. Businesses should find out the problems/opportunities that big data can solve. After that a collaborative effort between data scientist and business executives can lead to fruitful insights from data sets.

Limited and authorised sources of data should be chosen to avoid complexity and useless loads of data.

Having backup of big data is important as data can be lost or corrupted. Also, it protects data from cyber threats.

### How does big data analytics work?

Below are series of steps involved in deriving big data analytics;

    + Data Collection: Organizations can use numerous ways to collect data from from cloud storage to mobile applications to in-store IoT sensors and beyond. The data is stored in data warehouses and data lakes.
    + Data Processing: After the collection of data, it is organized properly for further processing. Batch processing is a data processing option which looks at large data blocks over time. It is useful when there is a longer delay time between collection and analysis of data. Stream processing is a data processing option which looks at small batches of data at a time. It shortens the delay time between collection and analysis of data enabling quicker decision-making.
    + Data Cleaning: Regardless data is big or small, cleansing of data is mandatory before analysis. Any duplicate or irrelevant data should be removed and data should be formatted correctly.
    + Data Analysis: Now advanced analytics processes are used to turn big data into insights. Some of these big data analysis methods include data mining, predictive analytics and deep learning.

### What are the myths about big data?

With evolution of big data, myths about big data has also evolved. Some of them are as following:

**Big data is costly and only for IT department**

The cloud SaaS platforms such as Amazon Web Services, Microsoft Azure and Google Cloud have made big data systems very affordable. No hardware/software purchase or installation is required. The access of right data can improve the performance of employee/organization regardless of department.

**Big data can predict the future**

Big data only tells the possibility of events based on the past data. It does not precisely predict the future. The predictions are based on past events and can be wrong too.

**Big data is all about size**

Volume is an important characteristic of big data but variety and velocity are equally important feature. Data should not only be large in size but also from credible sources.

**Big data will replace existing data warehouses**

Big data fulfils specific requirements but it is not the solution for every data-related issue. It cannot replace the traditional data warehouses or RDBMS.

**Big data is just hype**

Marketers in many industries are using it to increase revenue and reduce the operational costs.

## Milestones

1990

Peter J. Denning describes the need for computing machines in his publication, **Saving All the Bits** to process massively increasing data and find statistical summary out of it.

2000

The Sloan Digital Sky Survey (SDSS) begins collecting astronomical data, it gathers more data in its first few weeks than all data collected in the history of astronomy previously. It collects data at a rate of about 200 GB per night.

2001

Doug Laney introduces the 3Vs concept of big data. The Vs are volume, variety and velocity.

2005

Computer scientists Doug Cutting and Mike Cafarella create an open source framework called **Apache Hadoop**. It is used to store and process large data sets. **Apache Spark** is an open-source data processing framework introduced in 2009. It can quickly perform processing tasks on very large data sets. It provides the computational speed, scalability and programmability required for Big Data. The primary difference between Spark and Hadoop is that Spark processes and retains data in memory for subsequent steps, whereas Hadoop processes data on disk.

2007

The term **big data** is introduced to the masses in the Wired's article “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete”.

2008

Google processes 20 petabytes of data in a single day.

2012

The White House announces a national "Big Data Initiative" that consists of six federal departments and agencies committing more than $200 million to big data research projects. The U.S. state of Massachusetts announces the Massachusetts Big Data Initiative which provides funding from the state government and private companies to a variety of research institutions.
 Harvard Business Review titles Data Scientist as the “Sexiest Job of 21st Century”.

2014

The British government announces the founding of the Alan Turing Institute to focus on new ways to collect and analyze large data sets.

2016

IBM reports that 2.5 quintillion bytes of data is created every day (that's 2.5 followed by 18 zeroes).

Big Data


# DAPS Handover

## Summary

In mobile cellular systems, a UE in an active call might move from one cell to another. Via a procedure called handover the network attempts to maintain the call. Call is handed over from source cell to target cell. However, for some time period the connection is lost. DAPS Handover is an attempt to eliminate this handover interruption time.

During Dual Active Protocol Stack (DAPS) Handover, the UE maintains two protocol stacks. One stack continues to receive and send data on the source cell while the other handles traffic in the target cell. Within the RAN, packets are forwarded from source to target.

In 4G/LTE, Make-Before-Break (MBB) Handover was introduced in Release 14. In 4G/LTE and 5G, DAPS Handover was introduced in Release 16. MBB Handover can be viewed as a simpler form of DAPS Handover.

## Discussion

### Could you explain DAPS handover?

In 4G/LTE, a basic handover procedure incurs an interruption time of about 30-60ms.  In some cases, interruptions of hundreds of milliseconds have been reported. With the introduction of 5G and its URLLC use case, this is no longer acceptable. URLLC demands a latency of 1ms.

With DAPS, UE uses resources of both source and target cells even after receiving the handover command. UE suspends Signalling Radio Bearers (SRBs) towards the source cell. On those Data Radio Bearers (DRBs) configured for DAPS, the UE continues to send and receive data on the source cell while connecting to the target cell. Within the network, source eNB/gNB forwards uplink/downlink packets to the target eNB/gNB.

Once UE connects to the target cell, the latter commands the UE to release source cell resources. If handover failures, UE continues on the source cell and resumes suspended SRBs.

### What's the difference between MBB handover and DAPS handover?

With DAPS handover, UE makes the connection with target cell before breaking connection with the source cell. Hence, it's common in literature to call this "make-before-break". However, the 4G/LTE specifications uses the term MBB handover in a specific way. Consider the following definitions:

    + **MBB Handover**: Maintaining source eNB connection after reception of RRC message for handover before the initial uplink transmission to the target eNB during handover.
    + **DAPS Handover**: A handover procedure that maintains the source eNB connection after reception of RRC message for handover and until releasing the source cell after successful random access to the target eNB.In MBB, source eNB decides when to stop downlink transmission. Once uplink transmission is achieved on the target, UE can break its connection with the source. In DAPS, UE releases its source connection only after the target explicitly signals the UE to do so.

MBB is applicable for a change of SeNB in Dual Connectivity (DC) scenarios. It can even be used with RACH-less handover. However, DAPS can't be used alongside DC or RACH-less handover. These must be released by the RAN before initiating DAPS.

### How does DAPS impact the PDCP layer?

At the UE, PHY/MAC/RLC layers are duplicated, one for source cell and one for target cell. A single PDCP entity deals with both source and target cells. The figure shows the functional view of a 5G UE PDCP layer. In both transmit and receive entities, PDCP has two sets of security functions/keys and header compression protocols.

When transmitting, PDCP does routing, that is, decides whether packet should be sent on source or target cell. PDCP Data PDUs are sent to the target cell. PDCP Control PDUs are sent to source or target as associated.

When receiving, PDCP detects and discards duplicates since the same SDU may be received on both source and target cells.

In 4G/LTE, the PDCP layer for DAPS is similar except for integrity protection. 4G/LTE PDCP applies integrity only for control plane but DAPS is for the user plane.

### What are the technical details about DAPS?

UE, source/target NG-RAN and source/target AMF should support DAPS. DAPS is selectively applied to DRBs that require it.

Inter-RAT DAPS handover is not supported. DAPS FR2 to FR2 is not supported in Release 17.

For basic handover, UE resets the MAC entity and re-establishes the RLC entity. For DAPS, UE creates a new MAC entity. For each DAPS-enabled DRB, UE creates an RLC entity (with DTCH logical channel) and reconfigures the PDCP entity. If handover fails, UE reports the failure via the source cell without triggering RRC connection re-establishment.

Source gNB continues downlink transmission (and forwarding to target gNB) till it receives *Handover Success* from target gNB. Source gNB then responds with *SN Status Transfer* to target gNB. At this point, source gNB stops delivering uplink QoS flows to UPF. Source gNB no longer allocates PDCP SNs for downlink packets.

Even after switching to target gNB, UE continues to send UL layer 1 CSI feedback, HARQ feedback, layer 2 RLC feedback, ROHC feedback, HARQ data (re-)transmissions, and RLC data (re-)transmissions to the source gNB.

## Milestones

Dec
2015

Samsung files a patent that mentions **make-before-break handover** as a technique to reduce data interruption time. They describe this type of handover as "with continued PDUs delivered from the source eNB until the UE is ready to receive data from the target eNB."

Jun
2017

LTE Release 14 reaches functional freeze. The release introduces support for **Make-Before-Break Handover**.

Jul
2020

3GPP finalizes Release 16 specifications. The release introduces support for **DAPS Handover** in both 4G/LTE and 5G.

Feb
2022

MediaTek and Anritsu demonstrate successful DAPS handover. The test system uses MediaTek M80 5G modem and Anritsu's MT8000A Radio Communication Test Station with Rapid Test Designer application software.

Mar
2022

In the specifications of 5G Self-Organizing Network (SON), Mobility Robustness Optimisation (MRO) is defined for DAPS. Measurements that need to be collected are defined. DAPS use case is also described.

DAPS Handover


# Network Security

## Summary

Computer network security ensures the integrity of data stored on a network and restricts who has access to it. The necessity to offer service to users and the need to regulate access to information are balanced by network security policies. A network has numerous entry points including hardware and software, and the devices that connect to the network, like PCs, cellphones, and tablets. With these points of access, network security demands the deployment of many defense measures.

Another layer of protection is provided by processes for authenticating users using user IDs and passwords. Isolating network data makes it more difficult to obtain private/personal information than less vital data. Other network security methods include assuring frequent hardware and software upgrades and patches, educating network users about their participation in security operations, and remaining vigilant against external risks perpetrated by hackers and other hostile actors.

## Discussion

### Why do we need network security?

With the growth of social networks and e-commerce applications, **data transmission** is an important undertaking. This is because device to device communication requires a network interface, which is insecure due to the several types of tools and software available to destroy the existing network. A **security vulnerability** happens during data transit from one node to another in the sphere of network security. And, this is one of the most important difficulties in this field. This network security is justified by the fact that numerous types of **data attacks** occur on a daily basis. Setting up a new network is straightforward, but protecting the entire network is a major concern.

**Communication** is the most important aspect of network security, followed by **data automation**. The problem of network security is closely linked to the concept of data encryption. Due to the advancement of network security, we have gone back to thumb prints instead of signatures. For example, we use a finger print-based lock system, to keep the data secure. Although this technology helps to avoid physical data theft, logical data theft is still a concern in case of data transfer.

### How does network security work?

When it comes to network security in an organisation, there are numerous layers to consider. Attacks can occur at any layer of the network security layers model, so your network security hardware, software, and rules must be built to cover all of them. Three separate controls are usually used to secure a network:

    + **Physical-Level Network Security**: Unauthorized staff gaining physical access to network components such as routers, cabling cabinets, and so on is prevented by physical security mechanisms. In any organisation, controlled access, such as locks, biometric authentication, and other devices, is critical.
    + **Technical-Level Network Security**: Data that is stored on the network or that is in transit across, into, or out of the network is protected by technical security mechanisms. It is necessary to secure data and systems from unauthorised personnel as well as malicious activity by staff.
    + **Administration-Level Network Security**: Security rules and processes that manage user behaviour are referred to as administrative security controls. This includes how users are verified, their level of access, and how IT staff members apply changes to the infrastructure,

### What can network security provide protection against?

**Virus** is a harmful, downloaded file that can remain inactive and reproduce itself by modifying the code of other computer programmes. Once infected, the files can travel from one machine to another, corrupting/ destroying network data in the process.

**Worms** can slow down computer networks by consuming bandwidth and decreasing the computer's ability to handle data.

**Trojan** is a backdoor programme that allows attackers to gain a computer system's access by imitating an original programme but rapidly revealing itself to be dangerous. It can erase files, trigger additional malware, such as a virus, that is concealed on your computer network, and steal valuable data.

**Spyware** is a computer virus that collects information about a person/ organization without their knowledge and may communicate that information to a third party without their agreement.

**Adware** can reroute your search requests to advertising websites while also collecting marketing data about you. This is to present tailored advertisements based on your search and purchasing history.

**Ransomware** is a type of trojan cyberware that encrypts data and blocks access to the user's system This is to steal money from the person/ organization whose computer it is installed on.

### What are types of network security?

Email gateways are the most common source of security breaches. To prevent the loss of critical data, an **email security** solution stops incoming threats and regulates outbound messages.

Malware infects a network and then remains inactive for days or even weeks. **Antimalware systems** scan for malware upon arrival, and track files later to look for anomalies, delete malware, and repair damage.

Wireless networks are insecure as compared to conventional networks. Hence, devices created for **wireless network security** to prevent an exploit from taking hold are needed.

A **web security solution** will restrict your employees' access to the internet, block web-based dangers, and prevent them from visiting hazardous websites.

A **virtual private network (VPN)** encrypts the connection between an endpoint and a network, usually over the Internet, in order to authenticate communication between the device and the network.

An **intrusion prevention system (IPS)** monitors network traffic in order to detect and prevent assaults.

Organizations must ensure that sensitive data is not sent outside the network by their employees. **Data loss prevention (DLP)** technology can prohibit users from sending, transmitting, or even printing sensitive data in an unsafe way.

### What are the most important network security tools?

A **firewall** creates a barrier between a network's trusted and suspicious parts. Also, it uses IP subnets to execute access control and macro-segmentation. It may also perform micro-segmentation, which is more granular segmentation.

A **load balancer** is a device that distributes traffic based on metrics. It can go beyond standard load balancing to provide the ability to absorb certain attacks, like a volumetric DDoS attack, by incorporating specific mitigation strategies.

The conventional **IDS/IPS** installed behind a firewall performs protocol analysis and signature matching on different portions of a data packet, for protection against well-known attacks like SQL injection. Protocol analysis is a check for conformance to the protocol's publicly published specification.

A **sandbox** is similar to an IDS/IPS, however it does not use signatures. It can simulate an end-system environment and detect whether malware objects are attempting to do port scans, for instance.

**NTA/NDR** examines traffic/ traffic records and evaluates irregularities using machine learning algorithms and statistical methodologies to identify if a danger exists. It attempts to establish a baseline first. It detects irregularities like traffic surges or intermittent communication after establishing a baseline.

### What is network security monitoring and is there any real-time application for it?

The term **Network Security Monitoring (NSM)** refers to the process of detecting security incidents by monitoring network activities. Given the increasing sophistication of cyberwarfare, an NSM system is critical for the security of today's networks. The NSM cycle can be described by four stages: 1) Monitoring, 2) Detection, 3) Forensics/Diagnosis, and 4) Response/Recovery. Its purpose is to monitor the state of a given network to detect abnormal events and, when detected, to manage them in a timely manner.

Poor firewall configuration and monitoring continue to plague networked systems. VisualFirewall aims to help with firewall configuration and network monitoring by displaying four simultaneous views with varied degrees of information and time-scales, as well as appropriately depicting firewall responses to individual packets. Real-time traffic, visual signature, statistics, and IDS alarm are the four implemented views that provide the levels of detail and temporality that system administrators require to adequately monitor their systems in a passive or active way.

### What is the role of IDS in network security?

A critical issue for all of the networks is ensuring security, and several academics have proposed various solutions such as firewalls, cryptography, and restricting network access, to address this issue. They have one major flaw: they are unable to detect intrusions and attacks within the network.

**Intrusion detection systems** can detect both internal and external attacks. They monitor inbound network traffic for any unusual activity/ abuse by users. With early identification of intrusions, IDS plays a vital role in mitigating the harm caused by these intruders to the network structure/ the data shared across this network.

IDS is capable of detecting intrusions in two ways:

(a) Data from the history of operations and transactions carried out by authorised users and hackers in the networks is used to establish patterns for normal and deviant behaviour to distinguish unauthorised users from authorised users by matching these patterns/ signatures with the patterns of activity of existing users.

(b) IDS can identify hackers who use new tactics to breach the system by utilizing machine learning to classify users based on their network behaviour, traffic amount generated and content downloaded.

### What is a WSN and what is its relevance in network security?

A WSN is a massive network of resource-constrained sensor nodes that perform numerous pre-programmed roles, such as sensing and processing, to achieve various application goals. The sensor nodes and base stations are the most important components of a WSN. In fact, they might be thought of as the network's "sensing cells" and "brain," respectively.

Along with the development of numerous security-sensitive applications in various industries utilizing WSNs, wireless sensor networks (WSNs) security is of critical relevance. WSNs have a number of additional vulnerabilities compared to traditional wireless and wired networks, including changeable network topology, the broadcast nature of the medium, resource restricted nodes, massive network scale, and a lack of physical infrastructure. Because of the open communication environment, WSNs are more vulnerable to a variety of attacks than wired communications, including passive eavesdropping operations that result in intercepted transmissions and active jamming attacks that result in transmission disruption. These additional flaws enable the attacker to carry out more serious and complex attacks.

### Describe BYOD Security.

Bring Your Own Device (BYOD) refers to a group of linked technologies, concepts, and business practises where employees use their own mobile devices, such as smartphones, laptop computers, and tablet PCs, to access company IT resources, including databases and apps. With the introduction of a new IT environment, such as BYOD and smart-work, operational convenience has improved, but also adds to security dangers. There are many security solutions, including NAC (Network Access Control) and MDM (Mobile Device Management) tools, to counteract those dangers.

Network Method approach or network access control technology, focuses on managing and controlling access to business networks (NAC). It manages the devices that connect to the corporate network and offers a wide range of devices from various locations secure and restricted network access. Some network BYOD solutions, such as those from Cisco Networks and Meru Networks, provide BYOD management guidance.

There are two aspects to BYOD security:

    + Safeguarding corporate assets (networks, corporate data, applications, etc.) from any cybersecurity threats coming from BYOD devices used by employees.
    + Security measures taken by BYOD endpoints themselves to protect themselves from malware infections, phishing scams, and other security risks that could compromise sensitive data and spread laterally.

### What are cloud vulnerabilities and some countermeasures?

**Vulnerabilities:**

    + Increased resource leverage provides attackers with a single point of attack, which can cause crucial damage.
    + Data breaches are highly likely to be intentional, malicious, or accidental.
    + Losing access to a privileged account could result in service interruptions.
    + Numerous IP flaws, including IP spoofing, ARP spoofing, and DNS poisoning, pose serious risks.
    + Certain cloud computing characteristics, such as the usage of the trial period of use to conduct zombie or DDoS attacks, can be utilised for malicious offensive purposes.
    + A malicious insider at the cloud provider has the potential to harm numerous customers severely.**Countermeasures:**

    + Encrypted data cannot be read by the firewall or IDS, so end-to-end data encryption and screening for malicious activity are essential.
    + Validation of cloud consumers is necessary to stop malicious attacks from taking advantage of key cloud functionalities.
    + APIs provide secure interfaces for management, orchestration, and automation.
    + Any vulnerability must be mitigated, according to the cloud provider.
    + To prevent insider attacks, cloud providers should exercise caution when hiring employees and contractors and strengthen internal security measures.
    + The cloud provider has safe shared resources, including a hypervisor, orchestration software, and monitoring tools, under a shared/multi-tenancy architecture.

## Milestones

1971

The first computer worm is generated, with the words "I am the Creeper: catch me if you can" displayed on the screen.

1972

For the United States Air Force, James P. Anderson writes a Computer Security Technology Planning Study. This is split into two parts. The Anderson Report is the name given to this report.

1973

Multics is a timesharing operating system that began as a research project at MIT in 1965. Researchers at MIT investigate the security concerns of Multics running on a Honeywell 6180 computer system in the summer of 1973. They come up with security design principles that are broad in scope. With thanks to J. Anderson, they divide these into three categories: unauthorised release, unauthorised modification, and unauthorised denial.

1982

The first virus, 'Elk Cloner,' is created by a high school student that infects the Apple II operating system.

1986

The United States passes the first Fraud and Abuse Act, which establishes federal computer offences and punishments.

1988

Robert Morris designs a self-propagating virus, and distributes the denial-of-service (DDoS) assault that causes up to $10 million in damage to the early internet.

1990

The Computer Misuse Act is passed in the United Kingdom, making unauthorised efforts to access computer systems illegal.

1999

The Melissa virus attacks users via Microsoft Outlook in 1999, inflicting an estimated $1.2 billion in losses.

2001

The National Institute of Standards and Technology (NIST) releases Underlying Technical Models for Information Technology Security. Availability, Integrity, Confidentiality, Accountability, and Assurance are the five security objectives identified. It emphasises the interdependence of these elements. If confidentiality is broken (for example, a superuser password), integrity is likely to be compromised as well.

2002

Donn B. Parker adds three new items to the CIA Triad: authenticity, possession or control, and utility. It's also essential to grasp these six principles in pairs, according to Parker: confidentiality and possession, integrity and authenticity, and availability and utility. Parkerian Hexad is the name given to these six principles over time.
 Five DNS root servers were knocked out by a DDoS attack. It was the first time someone tried to shut down the internet.

Network Security


# Pandas Data Structures

## Summary

The two main **data structures** in Pandas are `Series` for 1-D data and `DataFrame` for 2-D data. Data in higher dimensions are supported within DataFrame using a concept called *hierarchical indexing*. For storing axis labels of Series and DataFrame, the data structure used is `Index`. These data structures can be created from Python or NumPy data structures.

Pandas data structures store data using NumPy or Pandas **data types**. Pandas has defined new data types where NumPy data types don't satisfy specific use cases. Pandas data types are also called *Extension types*. They're extended from Pandas `array`. Developers can extend `array` to create custom data types.

Pandas includes methods to convert data structures from Pandas to Python or NumPy. There's also implicit or explicit conversion of data types since a Series object or a DataFrame column can store values of only one data type.

## Discussion

### Could you give example usage of Pandas Series and DataFrame?

Consider the *pnwflights14* dataset from 2014. It captures 162,049 flights departing from two major airports Seattle and Portland. A flight is characterized by date and time of departure, arrival/departure delays, carrier, origin, destination, distance, airtime, etc. In a Pandas DataFrame, each flight becomes a row and each attribute becomes a labelled column.

For analysis, it's possible to extract only one column from the DataFrame. The result is a Series data structure. For example, given DataFrame `df`, we can write `df['air_time']` or `df.air_time`. Thus, DataFrame is for 2-D data and Series is for 1-D data. We can also extract multiple columns (such as `df[['origin','dest']]`) resulting in another DataFrame.

It's possible to extract specific rows for analysis. For example, if we wish to analyze all flights originating from only Seattle, we can write `df[df.air_time=='SEA']`. The result is another DataFrame. If we access a single row (such as `df.iloc[0]` or `df.loc[0]`), we get a Series data structure.

In terms of data types, this dataset has a mix of types: integer, float, datetime, string, etc. However, each column has only one type.

### What's the anatomy of a Pandas DataFrame?

A Pandas DataFrame represents tabular data, that is, data in **rows and columns**. All values in a column must have the same data type. Each row can be thought of as representing an entity or thing, with its attributes represented in the columns.

By default, rows and columns are **indexed** with integers, starting from zero. For convenience, it's more common to give **labels** or descriptive names, particularly to columns. Thus, rather than writing `df[0]` we can write `df['homeTeamName']`. Duplicate labels are allowed but can be prevented with method `set_flags(allows_duplicate_labels=False)`.

We can extract a single column, resulting in a Series. A row can also be extracted into a Series but since columns are likely to have different data types, the type of the resulting Series would be `object`, a generic type.

The direction of a Series is called **axis**. When moving from one row to the next, we're on axis-0. When moving from one column to the next, we're on axis-1.

### What are the different data types supported in Pandas?

Pandas mostly uses NumPy arrays and dtypes. These are float, int, bool, datetime64[ns], and timedelta[ns]. Unlike NumPy, Pandas datetime type is timezone-aware. For numerical values, defaults are `int64` and `float64`.

Data types particular to Pandas are BooleanDtype, CategoricalDtype, DatetimeTZDtype, Int64Dtype (plus other size and signed/unsigned variants), IntervalDtype, PeriodDtype, SparseDtype, StringDtype, Float32Dtype, and Float64Dtype.

When reading a data source, Pandas infers the data type for each column. Sometimes we may want to use a different type. For example, 'Customer ID' may be stored as `float64` but we want `int64`; 'Month' may be `object` but we want `datetime64[ns]`. For type conversions, we can use methods `astype()`, `to_numeric()` or `to_datetime()`. There's also `convert_dtypes()` method to infer the best possible conversion.

### What use is string type when there's already object type?

Consider the following examples:

    + `pd.Series(['red', 'green', 'blue'])`: object data type is used by default
    + `pd.Series(['red', 'green', 'blue'], dtype='string')`: string data type is usedThe object is a more general type. A DataFrame column with different types is collectively given the object type. Instead, string type gives developers clarity on the type of values stored. It also prevents accidentally storing strings and non-strings together. Pandas string type, which is an alias for `StringDtype`, is also more "closely aligned" to Python's own `str` class. The underlying storage is `arrays.StringArray`.

In a DataFrame, columns of string type can be more easily separated from columns of object type. This is possible by calling the method `select_dtypes(include='string')`.

While there's no difference between the two types in terms of performance, it's possible that future versions of Pandas might implement optimizations for string type.

### What are sparse data types and why do they matter?

Both Series and DataFrame can be used to store sparse data. Sparse doesn't necessarily mean "mostly zero" values. Rather, sparse data structures can be seen as a memory optimization technique. Any value can be specified. This value is not stored. However, these data structures behave the same way as their dense counterparts.

The underlying storage for sparse data structures is an ExtensionArray called `arrays.SparseArray`. Fill value, which is not stored in sparse arrays, can be specified using argument `fill_value`. We can inspect proportion of non-sparse values with property `density`. Method `to_dense()` yields the dense array. Methods `from_spmatrix()` and `to_coo()` help in interfacing with SciPy matrices.

As an example, consider the storage size of these two, using `df.memory_usage().sum()`:

    + `df = pd.DataFrame([1, 2, 2, 2])`: 160 bytes
    + `df = pd.DataFrame([1, 2, 2, 2], dtype=pd.SparseDtype("float", 2))`: 140 bytes

### What data types deal with missing values in Pandas?

For speed and convenience, it's useful to detect missing values. Traditionally, Pandas has used NumPy `np.nan` (displayed as `NaN`) to represent missing data. For example, Python `None` values will result in `NaN` in Pandas.

However, `NaN` is not used for all data types. It's used in a float context. For datetime objects, `NaT` is used, which is compatible with `NaN`. Python `None` value when converted in a string context becomes string "None", that is, it's not treated as a missing value. List of integers with some `None` values can't be converted into a Pandas Series.

To provide more uniform handling of missing values, since Pandas 1.0.0, there's `pd.NA` (displayed as `<NA>`) value. This is now used for missing values in `string`, `boolean` and `Int64` (and variants) types.

Inserting missing values into a Pandas data structure will trigger the suitable conversion. Infinite values `-inf` and `inf` can be treated as `pd.NA` by calling `pandas.options.mode.use_inf_as_na = True`.

DataFrame/Series methods `fillna()`, `dropna()`, `isna()`, `notna()` and `interpolate()` are useful.

### How do I convert Pandas data structures to/from NumPy or Python equivalents?

Input to the Series constructor can be a Python list or NumPy array. Thus, we can write `pandas.Series([1,2])` or `pandas.Series(numpy.array([1,2,3]))`. Constructor can also accept a tuple, a dictionary or scalar value. For a dictionary input, the keys become the Series index.

Series methods `tolist()` and `to_numpy()` return Python list and NumPy ndarray respectively. Another way to get a NumPy array is to access property `Series.values` but this is not recommended since for some data types it doesn't exactly return an ndarray.

Another property `Series.array` returns the underlying ExtensionArray. This is Pandas' own array. If the data is of NumPy native type, this is a thin wrapper (no copy) over NumPy ndarray.

Likewise, there are many ways to create a DataFrame: from lists, dictionaries, Series, another DataFrame, 1-D/2-D/structured NumPy arrays, or list of dictionaries. For a dictionary input, the keys become column labels. If keys are tuples, we create a multi-indexed frame. Useful methods for conversion include `to_dict()` and `from_dict()` (Python), and `to_records()` and `from_records()` (NumPy).

## Milestones

May
2017

In Pandas 0.20.0 (joint release with 0.20.1), `pandas.Panel` is deprecated. The recommended way to represent 3-D data is to use `MultiIndex` on a DataFrame. Method `to_frame()` converts from Panel to DataFrame. To convert to `xarray` data structure of package *xarray*, the method `to_xarray()` can be used.

Jan
2020

**Pandas 1.0.0** is released. On an experimental basis, Pandas introduces `pd.NA` to represent missing scalar values. This is used by `Int64Dtype` (and variants), `StringDtype` and `BooleanDtype`.

May
2020

In *Scikit-Learn*, a popular Python library for machine learning, version 0.23.0 is released. Dataset loaders in this release now support loading data as a Pandas DataFrame using argument `as_frame=True`.

Dec
2020

**Pandas 1.2.0** is released. This introduces `Float32Dtype`, `Float64Dtype` and `FloatArray`. While the default float uses `np.nan` for missing values, the new float types use `pd.NA`.

Pandas Data Structures


# Stemming

## Summary

Consider different forms of a word, such as *organize*, *organizes*, and *organizing*. Consider also words that are closely related, such as *democracy*, *democratic*, and *democratization*. In many applications, it may be inefficient to handle all the variations individually. **Stemming** reduces them to a common form. Algorithms that do this are called **stemmers**. The output of a stemmer is called the **stem**, which is the root word.

Stemming may be seen as a crude heuristic process that simply chops off ends of words. Unlike lemmatization, stemming doesn't involve dictionary lookup or morphological analysis. It's not even required that the stem be a valid word or identical to its morphological root. The goal is to reduce related words to the same stem.

## Discussion

### What are some applications of stemming?

Stemming is a pre-processing task that's done before other application-specific tasks are invoked. One application of stemming is to count the use of emotion words and perform basic sentiment analysis. When used with a dictionary or spell checker such as Hunspell, stemmers can be used to suggest corrections when wrong spelling is encountered.

One of the first applications of stemming was in Information Retrieval (IR). Searching with keyword "explosion" would fail to retrieve documents indexed by the word "explosives". Stemming solves this problem since indexing would be done using stem words.

Online search engines such as Google Search use stemming. An analysis from 2009 of Google Search showed that some suffixes such as '-s', '-ed', and '-ing' are considered to be strongly correlated with the stems. Suffixes '-able', '-tive', '-ly', and '-ness' are considered less correlated. For better SEO, forms that are poorly correlated could be added to improve search ranking. However, Google may penalize the page if it appears unnatural.

### What are some essential terms concerning stemming?

Stemming is based on the assumption that words have a structure, based on a root word and modifications of the root. The study of words and their parts is called **morphology**. In IR systems, given a word, stemming is really about finding morphological variants. The term **conflation** indicates the combining of variants to a common stem.

Words may contain prefixes and suffixes, which generally are called **affixes**. Stemming usually concerns itself with suffixes. Suffixes themselves are of two types:

    + **Inflectional**: Form is varied to express some grammatical feature such as singular/plural or present/past/future tense. Inflections don't change part of speech or meaning. For example, 'boy' and 'boys'; 'big', 'bigger' and 'biggest'.
    + **Derivational**: New forms are created from words. New ones have a different part of speech or meaning. For example, 'creation' from 'create'. They can occur between the stem and an inflectional suffix, such as 'governments', where '-ment' precedes '-s'. Another example is 'rationalizations', where '-al', '-iz' and '-ation' are derivational, and '-s' is inflectional.

### What are the typical errors while stemming?

Errors occur because rules fail for some special cases. The worst error is when two different concepts conflate to the same stem. For example, when 'Withings' is stemmed to 'with' on a web portal, wrong items are presented to the user. Porter stemmer stems 'meanness' and 'meaning' to 'mean', though they relate to different concepts. On the contrary, 'goose' and 'geese' are equivalent but they're stemmed as 'goos' and 'gees' respectively.

Typical errors of stemming are the following:

    + **Overstemming**: Happens when too much is removed. For example, 'wander' becomes 'wand'; 'news' becomes 'new'; or 'universal', 'universe', 'universities', and 'university' are all reduced to 'univers'. A better result would be 'univers' for the first two and 'universi' for the last two.
    + **Understemming**: Happens when words are from the same root but are not seen that way. For example, 'knavish' remains as 'knavish'; 'data' and 'datum' become 'dat' and 'datu' although they're from the same root.
    + **Misstemming**: Usually not a problem unless it leads for false conflations. For example, 'relativity' becomes 'relative'.

### Could you give an overview of algorithms for stemming?

Stemming algorithms broadly fall into one of two categories:

    + **Rule-based**: Table lookup is a brute force approach where inflected or derived forms are mapped to stems. For Turkish that has lots of inflected forms, this will lead to large tables. Another approach is to apply a series of rules to strip affixes to get to the stem. Yet another approach is to use word morphology, including part of speech. All these approaches require particular knowledge of the language.
    + **Statistical**: By training a model, suffix-stripping rules are implicitly derived. This is good for languages with complex morphology. No expert knowledge of the language is needed. Models can be trained from a lexicon, a corpus or character-based n-grams of the language's words.Among the rule-based stemmers are Lovins stemmer, Dawson stemmer, Porter stemmer, Paice/Husk stemmer (aka Lancaster stemmer), Krovetz stemmer and Xerox stemmer. Porter stemmer in its Snowball implementation is commonly used. Lancaster stemmer is more aggressive, leading to overstemming.

### Could you explain Porter's algorithm?

Porter stemmer applies a set of rules in five steps involving 51 suffixes and 60 rules.

Each rule is given by the form \((condition) \, S1 \to S2\), where S1 and S2 are suffixes. If the condition is matched or null, S1 is replaced with S2. When multiple rules are applicable in a given sub-step, only the rule with the longest S1 is obeyed. For example, step-1a has four rules with null condition:

$$SSES \to SS\\IES \to I\\SS \to SS\\S \to$$

In this case, 'caresses' becomes 'caress' since the first rule gives the longest match for S1; 'caress' remains caress (third rule); 'cares' becomes 'care' (fourth rule).

Porter also defined the generic structure of a word as \([C] (VC)^{m} [V]\), where [ ] denotes arbitrary presence, C is one or more consonants, V is one or more vowels and m is the word's **measure**. Measure is computed on what precedes the rule's suffix. We could use it to ignore rules when stem is too short. For example, the following rule changes 'agreed' to 'agree' (m=1) but retains 'feed' as 'feed' (m=0): \((m>0) \, EED \to EE\)

### What are the common approaches used by stochastic stemmers?

One approach is to segment words into roots and suffixes and selecting the best possible segmentation. This is based on counts of letter successor varieties. Minimum Description Length (MDL) stemmer is also about finding the optimal split between root and suffix. MDL was found to be computationally intensive and gave 83% accuracy for both English and French.

Another approach is to discover suffixes based on their frequencies. Frequencies are adjusted for partial suffixes so that '-ng' doesn't include '-ing'.

Hidden Markov Model (HMM) has been applied to stemming. Word letters are considered as hidden states and the transition from a root state to a suffix state is taken as the split point.

Yet Another Suffix Stripper (YASS) treats stemming as a word clustering problem. Two words having long common prefix are considered similar. Threshold value must be selected carefully to produce meaningful clusters. YASS was shown to perform similar to rule-based stemmers.

Other approaches make use of graphs with weighted edges, word co-occurrences, distribution similarity, and queries along with context.

### How is the performance of stemming?

Algorithmic stemmers are typically fast. For example, a million words can be stemmed in 6 seconds on a 500 MHz personal computer. It's more efficient not to use a dictionary. Rather than complicate the algorithm, it's simpler to ignore irregular forms.

Lemmatization uses a dictionary such as WordNet to get to the correct base forms. However, lemmatization has its limitations too. For example, 'happiness' and 'happy' are left unchanged while the Porter stemmer stems them correctly to 'happi'.

In IR systems, stemming reduces storage requirements by over 50% since we store stems rather than full terms.

### What are some tools to do stemming?

For a quick demo of stemming, online tools are available:

    + JavaScript-based Porter Stemmer: Highlights removed suffixes.
    + JavaScript-based Snowball Stemmer: Includes open source code and unit tests.
    + NLTK-based Stemmers: Two sources are text-processing.com and TextAnalysisOnline. They support Porter, Lancaster and Snowball stemmers. These tools also include lemmatization.For developers, R has the *corpus* package for stemming. This is based on Snowball. This has support for multiple natural languages. In Python, NLTK and TextBlob are two packages that support stemming.

Martin Porter has shared a list of many language implementations of the Porter stemmer. For test purpose, he also shares a sample vocabulary and the expected output.

## Milestones

Jun
1968

Julie Beth Lovins of MIT publishes details of one of the earliest stemmers in the context of information retrieval. The system has 10,000 documents in the field of materials science and engineering. The algorithm has two phases: (a) remove longest possible ending to obtain the stem; (b) handle spelling exceptions. For example, 'absorption' becomes 'absorpt' and 'absorbing' becomes 'absorb' in phase-1; phase-2 matches 'absorpt' and 'absorb' via recoding.

1980

Martin Porter invents an algorithmic stemmer based on rules for suffix stripping. The algorithm runs in five steps. He finds that in a vocabulary of 10,000 words the stemmer gives a size reduction of 33%, with most reduction coming from step-1 that deals with plurals and past participles. The stemmer is implemented in BCPL. This later becomes known as **Porter stemmer**.

1990

Chris Paice at Lancaster University invents a new stemmer that iteratively removes suffixes based on matching rules. Some rules replaces suffixes so that spelling exceptions don't occur. This stemmer is usually called Paice/Husk stemmer or **Lancaster stemmer**.

1994

Stemmers are often evaluated based on their effectiveness towards information retrieval. This however does not help in improving the stemmer. To solve this, Chris Paice proposes using predefined concept groups with sample words. The method counts overstemming and understemming errors. It then computes the metrics overstemming index (OI), understemming index (UI), stemming weight and error rate relative to truncation (ERRT). These help us compare the performance of different stemmers.

2001

Martin Porter finds that many implementations of Porter stemmer contain errors. He also notes that the Kraaij-Pohlmann stemmer for Dutch is released as open source ANSI C software but lacks a description of the algorithm. To solve these issues, Porter invents **Snowball**, a system in which rules can be expressed in a natural way. Then a compiler transforms the description into C code. The Snowball implementation for English is also called **Porter2 stemmer**. By 2019, Snowball supports more than a dozen natural languages with code generated in C, Java, Python, Go, Rust, C#, and more.

2007

John Fairweather files a patent application that treats stemming as a shortest-path problem. Words are modelled in the form `[Prefix] Root {[Infix] Root} [Suffix]`, where [ ] is zero or more occurrences, { } is one or more occurrences. Cost of each path is computed from word component and number of characters in it. The algorithm is language independent.

2009

Ahmet Uyar of Mersin University, Turkey, investigates stemming mechanisms used by Google. Using 18,000 different words, he finds that Google uses a document-based algorithm. Documents are indexed based on word forms that are semantically strongly correlated. This includes singular and plural forms but rarely suffixes of the type '-able' or '-tively'.

Stemming


# Data Preparation

## Summary

Raw data is usually not suitable for direct analysis. This is because the data might come from different sources in different formats. Moreover, real-world data is not clean. Some data points might be missing. Some others might be out of range. There could be duplicates. Data preparation is therefore an essential task that transforms or prepares data into a form that's suitable for analysis.

Data preparation assumes that data has already been collected. However, others may consider data collection and data ingestion as part of data preparation. Within data preparation, it's common to identify sub-stages that might include data pre-processing, data wrangling, and data transformation.

Useful insights from data via analytics is the final goal in today's data-driven world. However, data preparation is an important task. Poorly prepared data can make analytics more difficult and ineffective.

## Discussion

### What's the typical pipeline for data preparation?

The first step of a data preparation pipeline is to **gather** data from various sources and locations. Before any processing is done, we wish to **discover** what the data is about. At this stage, we understand the data within the context of business goals. Visualization of the data is also helpful here. The next stage is to **cleanse** the data of missing values and invalid values. We also reformat data to standard forms. Next we **transform** the data for a specific outcome or audience. We can **enrich** data by merging different datasets to enable richer insights. Finally, we **store** the data or directly send it out for analytics.

In the context of machine learning, **data pre-processing** converts raw data into clean data. This involves iteratively cleaning, integration, transformation and reduction of data. When an ML model is being prototyped, often data scientists may wish to go back to convert the data into a more useful form. This could be called **data wrangling** or **data munging**. This involves filtering, grouping, sorting, aggregating or decomposing data as required for modelling.

### What are some common tasks involved in data preparation?

Data preparation involves one or more of the following tasks:

    + **Aggregation**: Multiple columns are reduced to fewer columns. Records are summarized.
    + **Anonymization**: Sensitive values are removed for the sake of privacy.
    + **Augmentation**: Expand the dataset size without collecting more data. For example, image data is augmented via cropping or rotating.
    + **Blending**: Combine and link related data from various sources. For example, combine an employee's HR data with payroll data.
    + **Decomposing**: Decompose a data column that has sub-fields. For example, "6 ounces butter" is decomposed into three columns representing value, unit and ingredient.
    + **Deletion**: Duplicates and outliers are removed. Exploratory Data Analysis (EDA) may be used to identify outliers.
    + **Formatting**: Data is modified to a consistent form. For example, 2019-Jul-01, 2019-07-1, and 1/7/19 are changed to a single form, such as 2019-07-01.
    + **Imputation**: Fill missing values using estimates from available data.
    + **Labelling**: Data is labelled for supervised machine learning.
    + **Normalization**: Data is scaled or shifted, perhaps to a range of 0-1.
    + **Sampling**: For a quick analysis, select a small representative sample.

### What attributes are important in the context of data preparation?

It's important to measure the data preparation pipeline and assess how well it delivers data for analytics. To start with, **data quality** is measured by its accuracy, completeness, consistency, timeliness, believability, added value, interpretability, and accessibility. The pipeline must be able to validate data, spot problems and give tools to understand why they're occurring.

With the growing demand for near real-time analytics, we expect frequent **data refreshing**. Ideally, data quality is maintained even when refresh rate is high. In practice, there may be a trade-off.

Data must be **easy to access** even with growing data variety and volume. Data lakes and Hadoop are enablers in this regard.

Data must also **conform** to data models, domain models and database schemas. Data preparation must check for conformance.

Data preparation must ensure **consistency** across various datasets. For example, variations due to spelling or abbreviations must be handled. One sports dataset may use the term 'soccer' while another may use the term 'football'.

Unlike ETL systems, data preparation pipeline must be more **flexible** for ad hoc processing.

### What are some challenges involved in preparing data?

A common challenge faced by businesses is the **diversity** of data sources, siloed or proprietary tools, tedious processes, and regulatory compliance. Incompatible data formats, messy data and unbalanced data are further challenges.

When **manual processes** are used, businesses spend more time preparing data than analyzing it. Therefore, organizations must invest in tools and automation. Indeed, data scientists should get involved with teams tasked with data preparation.

It's been said that "bad data is bad for business". Organizations are unsure of data's **quality** and hence lack confidence in using it for decision making. This can be solved by investing early in data collection and preparation. Profile the data landscape. Improve data quality at source.

Some applications (such as fraud detection or industrial IoT apps) may require data preparation in **real time**. With large data volumes, collecting, preparing and storing data at scale is a challenge. In **production**, the preparation pipeline should be repeatable, handle errors and tuned for performance. It should work for initial data, incremental data and streamed data.

### Could you share some best practices for data preparation?

Check data types and formats. Check if data is accurate. Graph the data to get a sense of the distribution and outliers. If these are not as expected, question your assumptions. Label and annotate the graphs. Backup the data. Document or record the steps so that they can be repeated when new data comes in.

Data professionals shouldn't rely too much on IT departments. Adopt data systems and tools that are user friendly.

Data literacy, which is the ability to read, analyze and argue from data, is an essential skill today. Engineers, data scientists and business users must talk in a common language. Prepare data with a good understanding of the context and the business goals.

Profile the data first. Start with a small sample. Iteratively, try different cleaning strategies and discuss the results with business stakeholders. Keep in mind that data may change in the future. Data preparation is therefore not a do-once-and-forget task.

### Which programming languages are well suited for data preparation tasks?

Two well-known libraries for data preparation are `pandas` (Python) and `dplyr` (R). Apache Spark, more a framework than a language, is also suited for data preparation. Apache Spark enables fast in-memory data processing in a distributed architecture.

The main data structure in `pandas` is the DataFrame. The method `df.describe()` is a quick way to describe the data. A more detailed profile can be obtained using the `pandas-profiling` package. Missing values are represented as `NaN` or "not a number". Methods `df.fillna()` and `df.dropna()` help in dealing with NaN values. A couple of useful tutorials using Pandas are by Jaiswal and by Osei.

In R, the equivalent data structure is `data.frame`. A brief structure of the data can be seen using `str()`. Missing data is marked as `NA`. Packages `dplyr` and `tidyr` can be used for data preparation.

Spark provides APIs in Python, R, Java, Scala and SQL. When Spark is used, it's possible to convert between Spark DataFrames and Pandas DataFrames using Apache Arrow.

### Could you list some tools that aid data preparation with minimal coding?

Data analysis can't happen without data preparation. To enable this with minimal or no coding, it's best to use a **self-service data preparation tool**. This helps data analysts, business owners and data scientists. There are many of these in the market today (2019): Altair Monarch, Alteryx, ClearStory, Datameer, DataWatch, Dialogue, Improvado, LavaStorm, Microstrategy, Oracle, Paxata, Qlik, Quest, SAP, SAS, Tableau Prep, TIBCO, Trifacta, and Zaloni.

Many tools can combine data from different sources. They include visualization and exploratory data analysis. For data preparation, they can clean, augment, transform, or anonymize data. Some self-service tools include analytics and cataloguing. Good tools manage metadata and offer a search feature. They can track data sources and data transformations.

While developing data preparation pipelines, tools should support real-time visualizations. This facilitates quick debugging of the processing logic. However, it's been noted that many tools are unable to handle big data or fast enough on complex queries.

When choosing a suitable tool, some aspects to consider are features, pricing, performance, usability, collaboration, licensing model, vendor viability, customer support, enterprise integration, security and ecosystem.

## Milestones

2000

This decade sees the growing use of data in organizations. However, data is controlled and managed by IT departments. Data scientists work with IT staff. Data preparation involves coding and specialized expertise. Meanwhile, Business Intelligence (BI) tools show the benefits of data visualization and reporting. Users shift from spreadsheets to BI tools. This leads to more and more requests for data access and analysis. IT is soon overwhelmed by the workload.

May
2005

In a report titled *Competing on Analytics*, researchers note that companies are competing based on data and analytics. Statistical analysis and predictive modelling have become important. However, the report makes no mention of data preparation.

2010

To cope with the shift towards data-centric operations, IT open up data to other departments. While this brings speed, data also get siloed and inconsistent across datasets. Spreadsheets are error prone. ETL tools are rigid and therefore not suited for rapid prototyping. In this context, **self-service data preparation tools** emerge. These require almost no coding and enable rapid iterations.

Sep
2018

Google Cloud Platform announces the general availability of **Google Cloud Dataprep**, a cloud offering from Trifacta. This is a self-managed data preparation tool that integrates well with other parts of Google Cloud such as BigQuery and Cloud Dataflow. Prepared data can then be analyzed in Google Data Studio.

2019

A study by the International Data Corporation (IDC) finds that 33% of time is spent on data preparation, 32% on analytics and only 13% on data science. Respondents note that "too much time is spent on data preparation". They also recognize that they need to automate data preparation to acceleration their analytics pipeline.

Data Preparation