# Accelerated Mobile Pages

## Summary

To acquire and retain consumers, publishers have realized that content alone is not enough. To deliver rich user experiences, publishers therefore serve content along with interesting animations and engaging interactions. This might work on desktops but not on mobile devices where processing is limited and network connections are slower. Advertisements used for monetization might also worsen the experience. This is where AMP fits.

Accelerated Mobile Pages (AMP) is a library to create web pages that load almost instantaneously. AMP pages are web pages built using HTML but with some constraints. It's supported by multiple platforms and browsers. Developers can reuse their web skills to create AMP pages.

AMP is an open-source project started by Google. It's in active development and adoption is growing. There's also criticism that Google is trying to control the way the web evolves.

## Discussion

### What's the problem that AMP aims to solve?

In 2015, Facebook launched *Instant Articles*. Apple launched *Apple News*. Google followed by launching *AMP* later that year. These developments pointed to the fact that more and more users are accessing content from mobile devices but their experience is marred by the complexity of web technologies and developers using them inefficiently.

Pages load slowly. Content jumps around the page when big images arrive later, forcing rendering engines to recalculate the positioning of page elements. Annoying ads distract users from the main content. When users leave such pages, publishers miss out on readers and ad revenues.

Later research from 2016 also brought some insights about mobile web. 53% of users will leave a page that takes longer than 3 seconds to load. On a 3G connection, the average load time is 19 seconds. Each page makes on average 214 server requests, half of which are ad related.

While the proprietary approaches of Facebook and Apple use native apps, AMP is a mobile web approach. AMP leverages the maturity of the web and its standards.

### If AMP is for mobile, is it irrelevant for desktop users?

AMP was designed with mobile web in mind but it could just as well be used for desktops. It's good to think of AMP as *mobile first* rather than *mobile only*. This means it's easier to develop responsive web apps across devices using AMP.

In the early days of AMP, it made sense to have two versions of your site: non-AMP for desktops and the AMP for mobiles. While this might have enabled initial faster adoption, this had the drawback of sharing links across AMP and non-AMP pages, or showing a mobile-only AMP page to desktop users. The current suggestion is to have a single canonical AMP site that can be served to any device.

As of October 2018, Google Search does not serve pages from AMP Cache when accessed by desktop users.

### Does adopting AMP improve Google search engine ranking?

Google will not rank a page higher just because it's AMP. At best, it will show a lighting-bolt icon next to the search result to indicate that it's AMP and will be served fast from AMP Cache for mobile web users.

However, Google does rank pages higher up because they load faster on mobiles. This implies that having AMP is a good thing. In fact, WompMobile have noted 12% better ranking.

### Could you point me to some example sites using AMP?

The official AMP site has published some case studies. These show that AMP has been adopted by news publishers, advertisers and e-commerce sites. Some of these include Readwhere CMS, Jagran New Media, CNBC, Wired, Gizmodo, The Washington Post, U.S. Xpress, Teads, Tokopedia, Carved, Wego, and Events Ticket Center. Other noteworthy sites using AMP include Vox and Tencent for news; Myntra and Magebit for e-commerce; Tasty.co for food and recipes; Iene for entertainment.

In February 2018, AliExpress launched AMP-first mobile site. Mynet has built a *Progressive Web App (PWA)* that uses AMP pages. Spiegel Daily and Tasty.co have built their entire site using AMP, rather than have separate pages for mobiles and desktops.

A number of platforms (content, ads, analytics) and vendors support AMP by providing custom AMP components or integrating AMP pages to their platforms. Examples are Facebook Pixel, Parsely, Flipboard, Medium, Twitter, Reddit, Pinterest, Weibo, Drupal, Wordpress, Soundcloud, Vimeo, YouTube, and Dailymotion. The biggest ones include Wordpress, Reddit, Bing, Ebay, Pinterest and Google.

### What are the results after implementing AMP?

Washington Post reported +23% mobile search users who return within 7 days. Gizmodo reported 80% of traffic from AMP pages is new traffic and +50% impressions. Wired reported +25% click through rates from search results. Visitors to The Miami Herald spend 10% more time when coming via AMP pages.

At Events Ticket Center, page load times dropped from 5-6 seconds to less than a second. At Myntra, India's largest online fashion retailer, bounce rate reduced by 40%. At AliExpress, load times reduced by 36%, orders increased by 10.5% and conversion rates by 27%.

On the contrary, a Chartbeat study showed that only a third of publishers see increase in traffic. Rockstar Coders has said that AMP decreased conversions by 70%. It's also been noted that results depend on how well AMP is implemented.

### What are the essential AMP components?

AMP consists of three core components:

    + **AMP HTML**: This is HTML with restrictions. Basic HTML is extended with custom properties and elements. Custom elements are also called *AMP HTML components*. With these, we can implement common patterns easily without losing performance.
    + **AMP JS**: This is the AMP runtime in JavaScript. It renders AMP HTML fast. It optimizes resource loading, including asynchronous loading of external resources. Custom JavaScript is not allowed. It does prerendering for resources above the fold. It precalculates layout even before resources are loaded. It minimized style recalculations. It disables slow CSS selectors. CSS must be inline and limited to 50KB. Only animations capable of GPU acceleration are allowed.
    + **AMP Cache**: To get content to users quickly, they are served using Google AMP Cache, a proxy-based content delivery network (CDN). Cache also does page validation. This cache can be proactively updated via an API if original content changes.These components help AMP achieve its goals in areas of content, distribution and advertising.

### What's the data flow for client-server interactions with AMP pages?

When AMP pages are published, search engines and platforms will crawl and index them as usual. For example, Google will identify these pages and caches them in its CDN. When visitors search on Google, Google will mark these AMP pages on the search results page. Meanwhile, Google will also download content in advance above the fold and prerender the page in anticipation of user click. "Above the fold" means that only content visible on the device's viewport on first load will be downloaded. When user finally clicks on the link in the search results page, the loading is fast.

Subsequent clicks on links shown on the served page will be served from the original domain. They do not go via the AMP Cache, that is, the CDN. However, Google Analytics has mechanisms in place to treat the session as the same one.

### Could you list some AMP HTML components?

AMP HTML components start with the prefix `amp-`. For example, to include an image use `amp-img`. Some of them are built into the base library. Others are extensions and must be explicitly included. There are also experimental ones.

Without being exhaustive, we mention a few components:

    + **Ads & Analytics**: amp-analytics, amp-experiment, amp-pixel
    + **Dynamic Content**: amp-access, amp-bind, amp-form, amp-list
    + **Layout**: amp-accordion, amp-carousel, amp-iframe, amp-layout, amp-sidebar
    + **Media**: amp-audio, amp-img, amp-video, amp-youtube
    + **Presentation**: amp-date-countdown, amp-fit-text, amp-font, amp-story
    + **Social**: amp-facebook, amp-reddit

### What are the common criticisms of AMP?

The main criticism is that Google is doing this to increase its ad revenues and get around ad blockers that 16% of Americans use. Caching and pre-loading are not exclusive to AMP.

It's been said that all AMP pages look the same. Cross-domain analytics is hard across AMP and non-AMP pages. When served from AMP Cache, the URL that visitors see is Google's, not the original domain URL. There's disagreement about the effectiveness of ad monetization via AMP pages. Because ad formats are limited on AMP, monetization has been slow. Despite this, publishers are sticking to AMP because they seem to have little choice: Google seems to rank AMP pages higher.

Some see AMP as further fragmenting the web. Content must be formatted for the regular web, for Facebook Instant Articles, for AMP, and so on. However, there's been a change of mindset in which AMP can be served across all devices.

AMP does not respect the platform on which pages are rendered. For example, scrolling on iOS is unnatural or search on Safari browser doesn't work. Hence, Google is really breaking the open web.

### I already have a full-fledged site. How should I adopt AMP?

Creating an AMP page isn't about removing some things from the original page to make it faster. It's about serving the same content but in a manner that conforms to the rules of AMP. Among the errors that new developers make are using disallowed tags/attributes, and invalid/missing URLs.

In sites such as WordPress, there are plugins to convert existing pages to AMP. By applying suitable customizations to the plugins, this can be an easy route to adopting AMP. Hand-coded AMP pages will give better user engagement. Do this first for important pages that are reached via search engines or other platforms.

When users are navigating within your domain, target pages will not benefit from the AMP Cache. This is where the use of **Service Workers** and **Progressive Web Apps (PWA)** can help. In fact, AMP can work nicely with PWA. Developers can choose from a few options.

### What are some AMP resources for beginners?

The main AMP website is a good starting point for documentation, tutorials, guides and case studies. It also has the AMP Validator. To learn AMP via code samples, AMP By Example is a useful resource. To quickstart your project by reusing AMP templates, AMP Start is the place to find them. AMP Blog is the place to track latest news and updates.

Since the project is open source, AMP code is available on GitHub.

Guides are available to help you integrate your AMP pages across various Google products.

## Milestones

Oct
2015

Google announces **Accelerated Mobile Pages (AMP)** as an open source initiative to load content and ads faster on the mobile web. Google Search has integrated with AMP pages, with Google News to follow soon. About 30 publishers are partnering with Google to integrate AMP HTML pages.

Feb
2016

AMP brings in support for paywalls and subscriptions. Publishers retain control of the sign up and login process. This also the time when AMP pages are highlighted by Google Search as Top Stories carousel.

Sep
2016

Google Search results page now starts to surface AMP pages in standard search results. This means that these results link to validated AMP pages rather than to the original domain.

Mar
2017

Baidu, Sogou and Yahoo Japan launch AMP support. Cloudflare launches an AMP cache.

May
2017

Google Analytics enhances AMP support so that user sessions can be correctly tracked across AMP and non-AMP pages when served from the original domain. Meanwhile, Facebook announces that Facebook Instant Articles can also be published as AMP pages. This is to address publishers' frustration of having to format content for different platforms.

Sep
2017

Google introduces **AMP Client ID API**. Using this, Google can determine if a user arriving at the original domain is coming from an AMP page served by Google from AMP cache. This gives more accurate analytics in terms of unique users, session time, bounced rate, etc. In September 2018, this is further enhanced using **AMP Linker**.

Oct
2017

More than 4 billion pages and 25 million website domains are using AMP. The median time to load an AMP page via Google search is less than a second. A study by Forrester Consulting reports that AMP leads to +10% traffic with 2x time spent on page. For e-commerce sites, there's +20% sales conversions compared to non-AMP pages. On GitHub, the project has more than 10k stars.

Feb
2018

The project announces **AMP Stories** as a format for visual storytelling on the web. It also announces **AMP for Email**. Meanwhile, Google's earlier announcement of "same content" policy comes into effect. This requires that AMP pages have the same content as the original pages (also called canonical pages). This is to discourage publishers showing just a teaser on AMP pages and then asking users to navigate to the canonical pages.

Oct
2018

Bing launches AMP viewer and AMP cache.

Accelerated Mobile Pages


# 3GPP Release

## Summary

3GPP regularly publishes Technical Specifications (TSs) and Technical Reports (TRs) pertaining to mobile cellular systems (2G/3G/4G/5G). These documents go by the generic term **specifications**. New specifications are created and existing specifications are updated as the technologies evolve. Specifications are grouped into what are called **Releases**. Given a particular release, a mobile system can be constructed. By adding new functionality to a release, 3GPP partners and members construct a new release.

Coming up with a new release normally takes about 2.5 years. This excludes the time to conceptualize and finalize what should go into the release.

Companies develop products conforming to a specific release. Release is therefore an important concept for feature support and interoperability. Moreover, each release is designed with backward and forward compatibility in mind.

## Discussion

### How are specifications organized and versioned?

Specifications are organized into **series**, versioned and change-tracked. Documents within a series are closely related technically. For example, 38-series covers 5G, 36-series covers 4G and 25-series covers 3G. Within the 38-series, 38.1xx covers UE and base station requirements, 38.2xx covers PHY, 38.3xx covers radio protocols, etc.

Every specification is versioned with a version number of the form `x.y.z`. Field `x` refers to the release. Field `y` increments every time technical changes are introduced and approved by the TSG. Field `z` increments every time editorial changes are made. Fields `y` and `z` reset to zero if the preceding fields increment.

For example, *TS 38.101-4 V17.6.0* is a TS belonging to 38-series and of Release 17. The document is in its sixth technical version with no editorial changes to that version. The suffix "-4" indicates part 4 (parts and subparts are used only for long specifications published as multiple documents).

Field `x` has a few special values: 0 (draft), 1 (presented to TSG for information), and 2 (presented to TSG for approval). Once approved, `x` takes the value of the release number.

### What's the typical 3GPP release process?

Every release starts with a vision or concept of what is desired from the release. Proposals are made to add new features or services. Work formally begins when some of these proposals are accepted. Features are typically broken down into smaller elements.

Each 3GPP TSG meets quarterly to approve Work Items. Work Items with sufficient complexity may be preceded by Study Items. Study Items result in Technical Reports (TRs) that in turn becomes inputs to writing the Technical Specifications (TSs). Either new TR/TS documents are created or existing ones are updated via Change Requests (CRs). Essentially, specifications are change controlled and can be updated only via CRs.

A new release of a specification is either via CRs, via a request to the TSG, or automatic. If a specification is not updated during a release, it's automatically upgraded from the previous release without any changes.

The decision-making process is non-linear, iterative and via consensus. There's no voting process at meetings. Delegates discuss and negotiate. In most cases, original proposals are refined before accepted.

### What's meant by stage 1, 2 and 3 of a 3GPP release?

A release is developed via a three-stage methodology that comes from ITU‑T Recommendation I.130:

    + **Stage 1**: Overall service description from a user's perspective.
    + **Stage 2**: Overall description of network functions. These map service requirements into network capabilities.
    + **Stage 3**: Definition of switching and signalling capabilities to support services defined in stage 1.A feasibility study is often conducted before defining the specifications. This may be termed *Stage 0*. Likewise, stage 3 may be followed with test specifications. This may be termed *Stage 4*.

The figure shows Release 18 as an example. From package approval (December 2021) to protocol coding freeze (March 2024), the duration is 2 years and 3 months. Conceptualizing new features or enhancements for the release may begin as early as 12 months before the package approval. We also note that Release 18 overlaps with adjacent releases. This overlap enables continuous progress.

Stage 3 freeze is usually 1-2 TSG meetings before protocol freeze (release end date). End date is indicative since release may be refined/corrected later.

### Could you share more details on Change Requests?

CRs add new features, clarify/correct/enhance features of an ongoing release, or correct error in functionally frozen specifications. A CR is on a TR or TS and relates to a Work Item. It should highlight sections of the specification changed and include the changes.

CRs can undergo multiple revisions. Multiple CRs could be merged. CRs are agreed first within WGs before sent to TSG for approval. CRs can be rejected, withdrawn or postponed either at WG or TSG level. The figure shows an example of three CRs on 23.456 (fictional) specification. CRs are numbered 4, 5 and 6. Apart from CR 6, the other two have undergone revisions. When presented to the TSG plenary, all CRs of a given specification are combined into a single document along with a cover sheet.

CRs fall into a few categories: A (change to an earlier release), B (add/delete feature), C (functional change), D (editorial change, no impact on implementation), E (not used), and F (correction). When a release is frozen, category F is permitted but other categories are generally not permitted.

### What online resources are available to 3GPP release contributors?

Every release has a full release description document named *TR 21.9xx*. For example, the document for Release 17 is *TR 21.917*. It can be downloaded from the 21-series page.

The following resources are helpful:

    + 3GPP portal for members
    + 3GPP FAQ page
    + List of all series
    + Current status of specifications by release
    + Current Work Plan listing applicable Work Items
    + Guidelines for writing Work Items
    + Work Items download
    + Instructions to write a CR
    + CR search service by Netovate or on 3GPP portal
    + Database of all CRs submitted to TSG plenary

### Which are the main 3G releases?

The first 3G release was called **Release 99**. It supported both circuit-switched and packet-switched bearers. It met IMT-2000 requirements set by ITU. Subsequent 3G releases were Rel-4 through Rel-7. Rel-8 was the start of 4G/LTE but 3G continued to evolve in this and later releases.

Rel-5 introduced High Speed Downlink Packet Access (HSDPA) that supported shared channel transmission, HARQ and higher-order modulations. HSDPA brought lower latency, higher data rate and higher capacity. Rel-6 introduced Enhanced Uplink, aka High Speed Uplink Packet Access (HSUPA), with benefits and techniques somewhat similar to HSDPA. HSDPA and HSUPA are often together called **HSPA**.

HSPA evolved to Rel-7 with MIMO and higher-older modulations. This was called **HSPA+**. From Rel-8, HSPA+ started supporting Carrier Aggregation (CA). Rel-11 marked the start of **HSPA+ Advanced** with the Multiflow feature. Rel-12 introduced HetNets and Wi-Fi interworking.

The names 3.5G (HSPA), 3.75G (HSPA+) and 3.9G (LTE) are sometimes used, although this usage is not consistent across the industry.

### Which are the main 4G releases?

The first 4G/LTE release was Rel-8. It's main features were OFDMA waveform, FDD/TDD operation and an all-IP core network. Rel-9 added femtocells in the form of Home eNodeB (HeNB), Location Services (LCS), Evolved Multimedia Broadcast/Multicast Service (eMBMS), VoLTE and Circuit Switched FallBack (CSFB).

Rel-10 introduced CA, HetNets and advanced MIMO. It marked the start of **LTE Advanced**. These features were further enhanced in Rel-11 and Rel-12.

Rel-13 marked the start of **LTE Advanced Pro** with enhancements to CA and MIMO. The standard could achieve 1 Gbps in the downlink, giving rise to the term *Gigabit LTE*. It also introduced unlicensed access (LAA) and IoT (eMTC and NB-IoT). Enhancements to the specifications continued in Rel-14 and beyond.

### Which are the main 5G releases?

The first 5G release was Rel-15, launched in three phases: early drop with support for Non-Standalone (NSA) operation, main drop with support for Standalone (SA) operation, and late drop. Among its main features were mmWave band support, scalable OFDMA, Dual Connectivity (DC) with LTE, FDD/TDD support, short TTI, mini-slots, massive MIMO, beamforming, and more. The 5G Core (5GC) adopted a Service-Based Architecture (SBA) with network functions, microservices and APIs.

Rel-16 enhanced MIMO, beamforming, DC/CA, UE power saving, positioning and Dynamic Spectrum Sharing (DSS). It also added new features: Integrated Access and Backhaul (IAB), operation in unlicensed spectrum, time-sensitive networking for industrial IoT, NR sidelink for Vehicle-to-Everything (V2X) communications, and more. Apart from many enhancements, Rel-17 added 60 GHz unlicensed band support, satellite communications and sidelink relaying.

Rel-18 marks the start of **5G Advanced** meant to enhance MIMO, mobility, AI-based decisions, sidelink, positioning, satellite communications, multicast, DSS, multi-SIM, and more.

## Milestones

Dec
1998

The 3GPP Partnership Project Agreement is signed by **five Organizational Partners**: ARIB (Japan), ETSI (Europe), ATIS Committee T1 (USA), TTA (South Korea) and TTC (Japan). Their goal is to produce the 3G specifications. Subsequently, they expand their scope of work to include GSM/GPRS/EDGE specifications (since 2000) and evolution of 3G (since 2007).

2000

Release 99 for **UMTS/3G** is launched. It evolves to Rel-4 (2001).

2002

Rel-5 comes out. It's the first release of **High Speed Packet Access (HSPA)**. Also part of HSPA is Rel-6 (2004). This further evolves to **HSPA+** in Rel-7 (2007), Rel-8 (2008), Rel-9 (2009), and Rel-10 (2011). The next evolution is **HSPA+ Advanced** in Rel-12 (2015) and Rel-13 (2016).  The slightly different dates in the figure reflect dates of commercial deployment.

2008

With Rel-8, the first release of **4G/LTE** is launched. The LTE project work was started in December 2004. Originally named Evolved Universal Terrestrial Radio Access (E-UTRA) for the radio access part, it becomes significantly different from the UMTS system. LTE evolves to **LTE Advanced** in Rel-10 (2011), Rel-11 (2012) and Rel-12 (2015). This evolves to **LTE Advanced Pro** in Rel-13 (2016) and Rel-14 (2017). The slightly different dates in the figure reflect dates of commercial deployment.

2018

Rel-15, the first release of **5G**, comes out in 2018, although an "early drop" occurred in December 2017. This is followed by Rel-16 (2020) and Rel-17 (2022). This evolves to **5G Advanced** in Rel-18, which is expected in 2023.

2027

The first release of **6G**, in Rel-21, is expected in 2027.

3GPP Release


# Topic Modelling

## Summary

The growth of the web since the early 1990s has resulted in an explosion of online data. In an effort to organize all this unstructured data, topic models were invented as a text mining tool. Topic modelling uncovers underlying themes or topics in documents.

Consider a document in which the words 'dog' and 'bone' occur often. We can say that this document belongs to the topic of *Dogs*. Another document with words 'cat' and 'meow' occurring frequently is of topic *Cats*. In another example, based on words in the content, emails can be topically labelled as *Personal*, *Project* or *Financial*. An email can also belong to multiple topics. Topic modelling can be seen as a form of tagging.

Topic modelling is an unsupervised task. LDA and its many variants have been popular. LDA is a probabilistic generative model.

## Discussion

### What are some typical applications of topic modelling?

Early invention and application of topic models was in the field of text mining and information retrieval. Since then, topic modelling has been used in various applications including classification, categorization, summarization, and segmentation of documents. More unique applications include computer vision, population genetics and social networks.

In information retrieval, topic modelling helps in query expansion. It also personalizes search results or makes recommendations by mapping user preferences to topics.

When analyzing scientific literature, it's been noted that topics often correspond with scientific disciplines. We can also track how topics evolve over time. For example, the topic 'string' in Physics (for string theory) would be more common from the 1970s.

In social sciences, topic modelling enables qualitative analysis. Sentiment analysis and social network analysis are two examples.

In software engineering, topic modelling has been used to analyze source code, change logs, bug databases, and execution traces.

In bioinformatics, compared to traditional data reduction techniques, topic modelling is seen to be more promising since it's more easily interpretable.

### How is topic modelling different from text classification or clustering?

**Text classification** is a supervised task that learns a classifier from training data. Topic modelling is an unsupervised task where topics are not learned in advance. Topics are induced from the actual data.

**Text clustering** and topic modelling are similar in the sense that both are unsupervised tasks. Both attempt to organize documents for better information retrieval and browsing. However, there's a difference.

Text clustering looks at the similarity among documents and attempts to form similar clusters of these documents. These similarity measures could be based on TF-IDF weighting. In topic modelling, we don't look at document similarity. Instead, we treat a document as a mixture of topics in which a topic is a probability distribution of words. Soft clustering (where a document can belong to multiple clusters) can be viewed as being similar to topic modelling, though the approaches still differ.

Thus, the clusters from text clustering are not quite the same as topics in topic modelling. They're however seen as complementary. Research from mid-2000s explore combining both techniques into a single model.

### What's the typical pipeline for topic modelling?

Topic models perform a statistical analysis of words present in each document from a collection of documents. The model is expected to output three things: (a) clusters of co-occurring words each of which represents a topic; (b) the distribution of topics for each document; (c) a histogram of words for each topic.

To build a model, we must balance different aspects: fidelity (how well the model reflects the real world), performance, tractability (discrete models are preferred), and interpretability.

In the **bag-of-words model** the ordering of words in each document is ignored. Such a model is simple but it ignores phrase-level co-occurrences. An alternative is the **unigram model** in which words are randomly drawn from a categorical distribution. A mixture of such unigram models is also possible. For example, each topic has a distribution of words. We randomly draw words conditioned on the topic.

Essentially, words are being generated by latent variables of the model. Thus, a topic modelling algorithm such as LDA is a **generative model**.

### Could you explain how documents, words and topics are related?

The basic approach towards topic modelling is to prepare a document-term matrix. For each document, we count how many times a particular term occurs. In practice, not all terms are equally important. For this reason, **TF-IDF weighting** is used instead of raw counts. TF-IDF effectively gives more weight to frequent terms in a document that's rarer in the rest of the corpus.

The next step is to decompose this matrix into document-topic and term-topic matrices. We don't in fact identify the names of these topics. This is something the analyst can do by looking at the main terms of the topic. In the figure, we can see that *T1* is probably about sports because of the terms Lebron, Celtics and sprain.

Since number of topics is far fewer than the vocabulary, we can view topic modelling as a *dimensionality reduction technique*.  To determine exactly how many topics we should look for, the **Kullback-Leibler Divergence score** is a useful measure.

### Could you describe the main algorithms for topic modelling?

We mention three main algorithms:

    + **Latent Semantic Analysis (LSA)**: Also called *LSI*, this algorithm constructs a semantic space in which related words and documents are placed near one another. It uses SVD as the technique.
    + **Probabilistic LSA (pLSA)**: Also called *aspect model*, this is a probabilistic generative model. It doesn't use SVD. It looks at the probability of a topic given a document and the probability of a word given a topic. These are multinomial distributions that can be trained with EM algorithm.
    + **Latent Dirichlet Allocation (LDA)**: This is a Bayesian approach. Document is modelled as a finite mixture of topics. Each topic is modelled as an infinite mixture of topic probabilities. Topic probabilities make up a document's representation. Topic mixture is a Dirichlet distribution.

### What are some common challenges with topic modelling?

Topic modelling doesn't provide a method to select the optimum number of topics. LDA has many free parameters that can cause overfitting. LDA uses Bayesian priors without suitable justification. Statistical properties of the data (such as Zipf's Law) may also differ from the assumptions.

LSA is a linear model. It might not fit non-linearities in the dataset. It assumes Gaussian distribution of terms. SVD is also computationally expensive. LSA uses less efficient representations. Its results are not easily interpretable.

LDA can't directly represent co-occurrence information since words are sampled independently. A model based on Generalized Pólya urns or Bayesian regularization can solve this.

High-frequency non-specific words will result in topics that users may find too general and not useful. For examples, documents on artificial intelligence might have the words 'algorithm', 'model' or 'estimation' occurring frequently. Low-frequency specific words are equally problematic.

We could end up with topics that all look nearly the same. This can be solved by conditioning the prior topic distribution with data.

### What techniques have been used to improve the performance of topic modelling?

When pre-processing text, it's common to do lemmatization, remove punctuation and stop words. In addition, we can remove low frequency terms since they represent weak features. We could also make use of POS tags and remove terms that are not contextually important. For example, all prepositions could be removed.

Another technique is to do batch-wise LDA. Each batch will provide a different set of topics and an intersection of these might give the best topic terms.

Through user interactions, we could add constraints, or merge or separate topics. For example, for two highly correlated words we can add the constraint that they should appear in the same topic.

Some approaches to improve upon LDA include integrating topics with syntax; looking at correlation between topics such as Pachinko Allocation Model (PAM) or Correlated Topic Model (CTM); using metadata such as authors; accounting for burstiness of words (word once used in a document is more likely to appear again). Non-parametric models such Pitman-Yor or negative binomial processes have tried to address Zipf's Law.

### What are some useful resources for research into topic modelling?

Developers can use the *MALLET* Java package for topic modelling. A wrapper for this in R is available via the *mallet* package. Another R package is *topicmodels*. The latter package can do LDA (VEM or Gibbs estimation) or CTM (VEM estimation).

Python developers can use *nltk* for text pre-processing and *gensim* for topic modelling. Package *gensim* has functions to create a bag of words from a document, do TF-IDF weighting and apply LDA.  If the intent is to do LSA, then *sklearn* package has functions for TF-IDF and SVD. MALLET package is also available in Python via *gensim*.

pyLDAvis is a Python library for topic model visualization. An analyst can use this to look at terms of a topic and decide the topic name. For automatic topic labelling, Wikipedia can be a useful data source.

Those who wish to use a cloud service can look at **Amazon Comprehend**. It uses LDA on a collection. It returns terms-topics and documents-topics associations. A job should include at least 1000 documents.

## Milestones

1990

Deerwester et al. apply Singular Value Decomposition (SVD) to the problem of automatic indexing and information retrieval. They note that users want to retrieve documents not by words but rather by concept. By applying SVD on a document-term matrix they bring together terms and documents that are closely related in the "semantic space". Their idea of semantics is nothing more than a topic or concept. They call their method **Latent Semantic Indexing (LSI)** or **Latent Semantic Analysis (LSA)**. The name LSI reflects its origins in information retrieval.

May
1998

Papadimitriou et al. present the first mathematical analysis to rigorously explain why LSI works so well for information retrieval. Since LSI uses statistical properties of the corpus, they start with a probabilistic model of the corpus.

Aug
1999

Hofmann presents a statistical analysis of LSA, perhaps independently of the work of Papadimitriou et al. He coins the term **Probabilistic Latent Semantic Analysis (pLSA)**. It's based on *aspect model*, which is a latent variable model. It associates unobserved class variables (topics) with each observation (words). Unlike LSA, this is a proper generative model.

2003

First presented at NIPS 2001 conference, Blei et al. describe in detail a probabilistic generative model that they name **Latent Dirichlet Allocation (LDA)**. They note that pLSA lacks a probabilistic model at the document level. LDA overcomes this.

Dec
2005

Blei and Lafferty present **Correlated Topic Model (CTM)** to overcome a limitation of LDA. LDA is unable to capture correlations among topics. For example, a document about genetics is more likely to be about disease than x-ray astronomy. CTM captures these correlations via the logistic normal distribution. One of their results shows that CTM can handle as many as 90 topics whereas LDA peaks at only 30 topics.

Oct
2007

Researchers at the University of Massachusetts apply topic modelling to social network analysis. They note that previous analysis looked at only links between network nodes. Their work also looks at topics on those links. They call their model **Author-Recipient-Topic (ART)**. It's based on LDA.

2010

Newman et al. study a number of methods to automatically measure *topic coherence*. High coherence implies better interpretability. As an upper bound for comparison, they use inter-annotator agreement (IIA) as the gold standard. They find that **Pointwise Mutual information (PMI)** performs best and comes close to IIA. PMI is calculated between word pairs in a document on a 10-word sliding window. PMI measures statistical independence of two words occurring in close proximity.

Apr
2012

Arora et al. note the limitations of SVD: only one topic per document or recover only spans of topic vectors rather than the vectors themselves. Instead, they propose the use of **Non-negative Matrix Factorization (NMF)**. They assume that every topic has an *anchor word* that separates it from other topics. This aspect of separability using NMF was studied by the machine learning community at least a decade earlier.

May
2012

Chuang et al. present *Termite*, a tool for analyzing the performance of topic modelling. It's a term vs topic visualization on a grid, with size of circles depicting importance. In a technique called **seriation**, terms are ordered to show how they cluster for a topic. The visualization also helps us see if topics use lots of words or only a handful of them. We can also see what words are shared across topics.

Jun
2012

Chaney and Blei present a method to visualize topic models. Given a topic (such as defined by three most prominent words), their system displays associated words, most relevant documents matching this topic and a list of related topics. This is more useful that showing just a **word cloud**. Word clouds typically show only the topics and they make visual search difficult.

Sep
2013

Xie and Xing propose a model that integrates document clustering and text modelling into a single unified framework. Performing these two tasks separately fails to exploit the correlations between them. They note that a flat set of topics is not useful across domains. For example, topics applicable for computer science will be different from topics for economics. Their model also includes global topics that cut across domains. Their model has N documents, J groups, K group-specific topics per group, and R global topics.

2016

Word2vec came out in 2013. It's a word embedding that's constructed by predicting neighbouring words given a word. LDA on the other hand looks at words at the document level. Moody proposes **lda2vec** as an approach to capture both local and global information. This combines the power of word2vec and the interpretability of LDA. Word vectors are dense but document vectors are sparse.

Jul
2018

Gerlach et al. note that topic modelling and community detection have evolved independently. Because they are conceptually similar, they can be combined into a single unified model. Community detection is about identifying groups of nodes with similar connectivity patterns. When applied to topic modelling, we're inferring topics by way of inferring communities of words and documents. Stochastic Block Model (SBM) is popular for community detection. This is adapted as **hierarchical SBM (hSBM)** for topic modelling.

Topic Modelling


# Data Science

## Summary

Data is no longer scarce. In fact, businesses have an abundance of data and its growing. This has given rise to the term Big Data. Data science enables businesses to discover valuable insights from data and apply that profitably. Data science is therefore complementary to Big Data.

Historically, statisticians had a mathematical focus. They evolved into data analysts who applied their expertise to solving business problems. They did this by visualizing data and searching for patterns. When dealing with vast amounts of data, there was a need to apply Machine Learning algorithms and programming. This is where a data scientist comes in.

A data scientist is really a first-class scientist who's curious, asks questions and makes hypotheses that can be tested with data.

## Discussion

### What exactly is the definition of the term "Data Science"?

One possible definition is that "data science is a multifaceted discipline, which encompasses machine learning and other analytic processes, statistics and related branches of mathematics, increasingly borrows from high performance scientific computing, all in order to ultimately extract insight from data and use this new-found information to tell stories."

At a high level, "data science is the study of the generalizable extraction of knowledge from data." It's a combination of multiple disciplines that have been around for decades.

Data Science Association's Professional Code of Conduct states that a "Data Scientist means a professional who uses scientific methods to liberate and create meaning from raw data".

### What's a typical Data Science process?

In science, one starts with a hypothesis, conducts experiments and makes observations to either prove or disprove the hypothesis. In Data Science, the scientific process is similar except that the use of data and algorithms becomes central to the process.

The process starts with an interesting question, often aligned to business goals. Available data is then cleaned and filtered. This may also involve collecting new data as relevant to the question. Data is analyzed to discover patterns and outliers. A model is built and validated, often using machine learning algorithms. Model is often refined in an iterative manner. The final step is to communicate the results. The results may inspire the data scientist to ask and investigate further questions.

### What's a typical Data Science pipeline?

The data science pipeline may be treated as that part of the data science process that deals specifically with data. It starts with the gathering of raw data, processing it, analyzing it via algorithms and finally visualizing the results. Thus, the pipeline basically transforms data into useful insights.

An important aspect of this pipeline is data engineering. It can be broken down into three steps (not standardized):

    + **Data Wrangling**: Raw data is cast to a form suitable for analysis. This could involve combining multiple datasets, removing inconsistencies, converting datasets to a common format, etc.
    + **Data Cleansing**: Real-world data is messy with missing values, bad delimiters or inconsistent records. Cleansing ensures and even repairs data to syntactic and semantic correctness. Data points could be dropped if they cannot be repaired.
    + **Data Preparation**: This makes the data suitable as input to algorithms. This may involve range normalization, conversion of categorical data to numerical values, etc.When building ML models, the typical approach is to partition available data into training and testing datasets. The former is used for learning and the latter is used for validation.

### Could you give some examples of questions that data science answers?

Here are a few examples: Will this tire fail in the next 1000 miles? Is this bank transaction normal? What will be my sales for the next quarter? What sort of customers are not coming back to my store? Which printer models are failing the same way? As a self-driving car, should I now slow down or brake completely?

In the real world, data science has been successfully used by LinkedIn to increase growth in user connections. Google uses it in a number of products. GE uses it to optimize service contracts. Netflix uses it to improve movie recommendation. Kaplan uses it to uncover effective learning strategies. All these are a result of data scientists asking the right questions.

### How is a data scientist different from a data analyst/architect/engineer?

A data scientist is multidisciplinary in terms of skills and expertise. She may embody some of the other related roles:

    + **Data Analyst**: Collects relevant data, visualizes data with various tools and tries to find patterns and insights. Knows basic statistics. Has business/domain knowledge. Probably doesn't deal with big data.
    + **Data Architect**: Architects a system to manage big data. Often this role is embodied within a data engineer since tools and technologies overlap. This role becomes redundant with MLaaS (Machine Learning as a Service).
    + **Data Engineer**: Develops and manages infrastructure that deals with big data. Well versed with tools such as Hadoop, NoSQL and MapReduce. Sets up data pipelines.Where a data scientist stands out is in her use of ML algorithms, which requires both statistics and computational skills. A data scientist augments these skills with her ability to deal with large datasets and domain knowledge.

### Should a data scientist start with a problem statement or explore available data?

If you're new to data science, without much domain knowledge, defining a problem statement can be difficult. In such a case, you could start with exploratory analysis. This can then guide you towards asking the right questions.

Given enough data, exploration is likely to yield patterns and correlations. These could even occur due to measurement errors or data processing artifacts. But are these findings relevant? Asking the right questions and defining a problem statement will give better focus. Some even claim that lack of a problem definition could lead to disaster because you don't know what you're looking for.

### What skills must a data scientist have?

Data scientists are required to be multidisciplinary with knowledge and expertise in statistics, programming, application domain, analytics and communication. However, unicorns who have all of these are rare.

Often specializing in a couple of areas with exposure to others is desirable. Companies don't rely on a single all-knowing data scientist; they form a data science team. A data science team may include the Chief Data Officer, business analyst, data analyst, data scientist, data architect, data engineer and application engineer. Some of these may be combined in a single person. For example, a single person may fulfil the roles of data architect and data engineer.

Anyone with strong data and computational abilities can do well as a data scientist. An essential skill is to turn unstructured data into a form suitable for analysis. This is not something that a traditional quantitative analyst can do.

Technical skills must be complemented with business acumen, creativity and reasoning. This will help a data scientist ask relevant questions, assess the suitability of available data and present results the right way.

### Should a data scientist learn cloud computing?

Data science workflows typically happen on a local computer. There are however scenarios where cloud computing makes sense. The dataset could be too large for local memory; or local computational capability is insufficient for the problem; or the workflow's output feeds into a larger production environment.

When dealing with large datasets, a data scientist needs to get familiar with cloud technologies, platforms and tools. This may include storage, running database queries and managing Apache Spark clusters.

### As a beginner, what should be my learning path to become a data scientist?

One approach is to be practical and hands-on from the outset. Pick a topic in which you're passionate and curious. Research available datasets. Tweet and discuss so that you get clarity. Start coding. Explore. Analyze. Build data pipelines for large datasets. Communicate your results. Repeat this with other datasets and build a public portfolio. Along the way, pick up all the skills you need.

You may instead prefer a more formal approach. You can learn the basics of languages such as R and Python. Follow this with additional packages/libraries particular to data science: (R) dplyr, ggplot2; (Python) NumPy, Pandas, matplotlib. Get introduced to statistics. From this foundation, start your journey into Machine Learning. To relate these to business goals, some recommend the book Data Science for Business by Provost and Fawcett. But you should put all this knowledge into practice by taking up projects with datasets and problems that interest you.

At the University of Wisconsin, statistics is covered first before programming. To become inter-disciplinary, you may choose to learn aspects of data engineering (data warehousing, Big Data) and ethics.

### Could you give some tips for a budding data scientist?

The following tips might help:

    + Data science is about answering scientific questions with the help of data. Don't focus just on the aspect of handling data, dataset size or the tools.
    + Understand the business, its products, customers and strategies. This will help you ask the right questions. Have constant interaction with business counterparts. Communicate with them in a language they can understand.
    + Consider alternative approaches before selecting one that suits the problem. Likewise, select a suitable metric. Sometimes derived metrics may yield better prediction compared to available metrics.
    + Understand the pros and cons of various ML algorithms before selecting one for your problem.
    + Craft machine learning models from scratch. Don't just rely on premade templates and libraries. Test them to their limits to understand what's going to work, & where.
    + Find a compromise between speed and perfection. On-time delivery should be preferred over extreme accuracy.
    + Useful data is more important than lots of data. Use multiple data sources to better understand data and its discrepancies.
    + Be connected with the data science community, be it via blogs, meetups, conferences or hackathons.
    + Practice with open datasets. Learn from the solutions of others.

## Milestones

1962

John W. Tukey publishes "The Future of Data Analysis". He explains that statistics has mostly been about making inferences but his interest is in *data analysis*, which has more to do with science than mathematics. The availability of computers makes data analysis possible. His influential paper is sometimes today referred to as FoDA.

1974

The term *data science* is used for the first time, by Peter Naur in his "Concise Survey of Computer Methods". He defines it as the "science of dealing with data". His definition does not consider data semantics (domain knowledge). Thus, it's different from modern definition of the term. An alternative term *datalogy* is also used.

1976

John Chambers at Bell Labs create programming language *S*. This lays the basis for statistical computing and quantitative programming environments (QPE) that use scripts and workflows. In the 1990s, *S* inspires the creation of an open source language called *R*, which is today the dominant QPE.

1977

The International Association for Statistical Computing (ISAC) is formed. This underscores the increasing use of computing in statistical work "to convert data into information and knowledge". The same year John Tukey publishes "Exploratory Data Analysis" where he states that we should use data to form hypotheses to test. *Exploratory Data Analysis* and *Confirmatory Data Analysis* should both be used.

1989

The first Knowledge Discovery in Databases (KDD) conference is held. By the mid-1990s, this evolves into ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Researchers clarify that "KDD refers to the overall process of discovering useful knowledge from data, and data mining refers to a particular step in this process. Data mining is the application of specific algorithms for extracting patterns from data".

1997

Professor C. F. Jeff Wu calls for statistics to be renamed data science and statisticians to be renamed data scientists. The same year the journal Data Mining and Knowledge Discovery is launched.

2001

William S. Cleveland publishes "Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics". He proposes the term *Data Science* in its modern sense. To Cleveland, a data analyst is good at programming but has limited knowledge of statistics. A data scientist on the other hand comes from statistics background but has to work more closely with computer specialists.

2001

Leo Breiman publishes "Statistical Modeling: The Two Cultures". He compares *Generative Modeling* with *Predictive Modeling*. The former is dominant among statisticians. Breiman calls them to adopt predictive modeling and algorithms, which have developed in other fields.

2008

*Data Science* as a term creates interest thanks to the work of D. J. Patil (LinkedIn) and Jeff Hammerbacher (Facebook).

2009

Google's chief economist, Hal Varian, states that data is plenty but there's a scarcity of experts who can extract value from it. The sexy job for the next decade will be statisticians. He states his expectation of a data scientist,

> The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades.

2010

By the start of this decade, researchers and writers attempt to explain data science to the public. Data scientist is claimed to be the sexiest job of the 21st century. This decade also sees shifting terminology. *Data Mining* is now referred to as *Machine Learning*. The work of a data analyst is called *Business Intelligence* but when she uses big data it's called *Big Data Analytics*.

Sep
2015

The University of Michigan announces a $100 million Data Science Initiative (DSI) to hire 35 new faculty. It's press release says, "Data science has become a fourth approach to scientific discovery, in addition to experimentation, modeling, and computation."

2017

Monica Rogati puts AI at the top of a pyramid that signifies data science hierarchy of needs. She makes the point that companies can't have a successful AI strategy without basic data literacy, collection and infrastructure. It's also not a good idea to over-engineer the infrastructure. Instead, build an MVP spanning all layers of the pyramid and then scale horizontally.

Data Science


# Regression Modelling

## Summary

Regression is a method to mathematically formulate relationship between variables that in due course can be used to estimate, interpolate and extrapolate. Suppose we want to estimate the weight of individuals, which is influenced by height, diet, workout, etc. Here, *Weight* is the **predicted** variable. *Height*, *Diet*, *Workout* are **predictor** variables.

The predicted variable is a **dependant** variable in the sense that it depends on predictors. Predictors are also called as **independent** variables. Regression reveals to what extent the predicted variable is affected by the predictors. In other words, what amount of variation in predictors will result in variations of the predicted variable. The predicted variable is mathematically represented as \(Y\). The predictor variables are represented as \(X1\), \(X2\), \(X3\), etc. This mathematical relationship is often called the **regression model**.

Regression is a branch of statistics. There are many types of regression. Regression is commonly used for prediction and forecasting.

## Discussion

### What's a typical process for performing regression analysis?

First select a suitable predicted variable with acceptable measurement qualities such as reliability and validity. Likewise, select the predictors. When there's a single predictor, we call it **bivariate** analysis; anything more, we call it **multivariate** analysis.

Collect sufficient number of data points. Use a suitable estimation technique to arrive at the mathematical formula between predicted and predictor variables. No model is perfect. Hence, give error bounds.

Finally, assess the model's stability by applying it to different samples of the same population. When predictor variables are given for a new data point, estimate the predicted variable. If stable, the model's accuracy should not decrease. This process is called **model cross-validation**.

### I've heard of Least Squares. What's this and how is it related to regression?

**Least Squares** is a term that signifies that the square of errors are at a minimum. The error is defined as the difference between observed value and predicted value. The objective of regression estimation is produce least squared errors as a result. When error approaches zero, we term it as *overfitting*.

**Least Squares Method** provides linear equations with unknowns that can be solved for any given data. The unknowns are regression parameters. The linear equations are called as *Normal Equations*. The normal equations are derived using calculus to minimize squared errors.

All other algorithms (*Artificial Neural Network (ANN)*, *K-Nearest Neighbour (KNN)*, etc.) too attempt to minimize squared error unless the objective states otherwise.

### Could you explain the difference between interpolation and extrapolation w.r.t. regression?

Simply put, interpolation is estimation in familiar territory and extrapolation is estimation where not much of data is available due to various reasons—not collected or cannot be collected.

We can interpolate missing data points using regression. For instance, we want to estimate height given weight and data collection process missed out certain weights, we can use regression to interpolate. This missing data can estimated by other means too. The missing data estimation is called *imputation*.

The height and weight data is bound by nature and can be sourced. Say, we want to estimate future weight of an individual given historical weight variations of the individual. This is extrapolation. In regression, we call it *forecasting*. This is solved using a distinct set of techniques called as **Time Series Regression**.

### What is correlation? How is it related to regression?

Correlation helps identify variables that can be applied for regression modelling. Correlations between each predictor and predicted variable are identified to decide on the predictors that need to be included in the model.

Correlation is defining the association between two variables. The effect of \(X\) (or \(X1\), \(X2\), \(X3\)...) on \(Y\) can be thus quantified:

    + **Positive Correlation**: \(Y\) goes up/down as \(X\) goes up/down. Correlation coefficient will be in the range [0,1].
    + **Negative Correlation**: \(Y\) goes up/down as \(X\) goes down/up. Correlation coefficient will be in the range [-1,0].
    + **No Correlation**: \(Y\) doesn't go up/down as \(X\) goes up/down. Correlation coefficient will be close to 0.*Correlation coefficient* \(r\) has the following formula:

$$r=\frac{\sum\_{i=1}^n(x\_i-\bar x)(y\_i-\bar y)}{\sqrt{\sum\_{i=1}^n(x\_i-\bar x)^2 \sum\_{i=1}^n(y\_i-\bar y)^2}}$$

An equivalent formula that substitutes the mean values \(\bar x\) and \(\bar y\) with their individual sample points \(x\_i\) and \(y\_i\) is published in Wikipedia. More formally, \(r\) is called **Pearson Product Moment Correlation (PPMC)**.

### What's the right interpretation of correlation coefficient?

Correlation coefficient \(r\) is measure of *linear* association strength. It doesn't quantify non-linearity. A correlation coefficient of 80% (0.8) means that 80% of variation in one variable is explained by variation in the other variable. Example, 80% of variation in rainfall is explained by the number of trees; 20% is due to factors other than the number of trees.

It will be apparent from the formula that \(r\) factors in the sample variance. On a X-Y scatterplot, the regression line may have different slopes due to different sample variance even when all of them share the same correlation coefficient. In other words, \(r\) is not simply the slope of the regression line.

### Could you give examples of non-linear correlation?

A non-linear correlation is where the relationship between the variables cannot be expressed by a straight line. We call this relationship *curvilinear*.

Non-linear relationship can exhibit monotonous positive, monotonous negative, or both patterns together.

### How can we do data analysis when relationships are non-linear?

The correlation coefficient formula applies for only linear relationships. One common approach for non-linear correlations is to transform them into linear forms. If the relationship is curvilinear, we can apply transformations directly. Common transformations include logarithmic or inverse transformations.

If the relationship is non-linear but not curvilinear, we can split the data into distinct segments. Data within some segments may be linear. In other segments, if it's curvilinear, transformations can be applied to make them linear. Analysis is thus segment-wise, sometimes called **segmented regression**. As an example, yield of mustard is not affected by soil salinity for low values. For salinity above a threshold, there's a negative linear relation. This dataset can be segmented at the threshold.

### What is causal relationship in regression?

One study about college education, showed positive correlation between SAT scores of incoming students and their earnings when they graduate. Moreover, we can state that graduating from elite colleges (high SAT scores) had a role in higher salaries.

Causality or causation refers to the idea that variation in a predictor \(X\) *causes* variation in the predicted variable \(Y\). This is distinct from regression, which is more about predicting \(Y\) based on its correlation with \(X\). Regression does not claim that \(Y\) is caused by \(X\).

Here are some possible examples of causality. High scores lead to higher earnings. Regular exercise results in better health. Current season influences power consumption. All pairs of variables that have causal relationship will exhibit significant correlation.

### Does strong correlation always imply causal relationship?

No. Sometimes correlations are purely coincidental. For example, non-commercial space launches and sociology doctorates awarded are completely unrelated but the image shows them to be strongly correlated. This is called a **Spurious Correlation**. This is a clear case where correlation does not imply causation.

Another example is when ice cream sales are positively correlated with violent crime. However, violent crime is not caused by ice cream sales. It so happens that there's a **confounding variable**, which in this case is weather. Hot weather influences both ice cream sales and violent crimes. It's therefore obvious that,

> correlation does not always imply causation

Correlation shouldn't be mistaken for causation. Look at the physical mechanism causing such a relationship. For example, is rain driving the sale of your product? Data may show a correlation. It need not be causal unless your product is an umbrella. However, proving causality is hard. At best, we can do randomized trials to establish causality.

Regression is a useful tool in either predictive or causal analysis. With the growth of Big Data, it's being used more often for predictive analysis.

### Could you explain the regression model?

We call it a *model* when the relationship between variables is in a well-defined mathematical form: \(Y\)=\(f(X)\).

For instance, a linear relationship can be written as \(f(X)=a+b\_1X\_1+b\_2X\_2+b\_3X\_3\), where \(a\) is a constant and \(b\_1,\,b\_2,\,b\_3\) are regression coefficients. \(a\) is constant effect while a unit change in \(X\_1\), will result in \(b\_1\) unit change in \(Y\).

It's important to note that linearity is in terms of the coefficients, not in terms of predictor variables. For example, this model is still linear though it's quadratic in terms of \(X\_1\): \(f(X)=a+b\_1X\_1+b\_2X\_1^2\).

### How do we measure the accuracy of a regression model?

The accuracy of regression model is relative to base model. Called **R-Squared**, this measure is squared deviation from the expected value, which is mathematically defined below:

    + For base model, the sum of squared deviation of actual value \(Y\) from mean value \(E(Y)\) is referred to as *Total Variance* or *SST (Total Sum of Squares)*. $$SST=\sum\_{i=1}^n(y\_i-\bar y)^2$$
    + For regression model, the sum of squared deviation of estimated value \(\widehat Y \) from mean value \(E(Y)\) is referred to as *Explained Variance* or *SSR (Regression Sum of Squares)*. $$SSR=\sum\_{i=1}^n(\widehat y\_i-\bar y)^2$$
    + The accuracy of the model is called *R-Squared*. $$R^2=\frac{\text{Explained Variance}}{\text{Total Variance}} = \frac{SSR}{SST}$$Higher the \(R^2\), larger the explained variance and lower the unexplained. Hence, higher \(R^2\) value is desired. For example, if \(R^2=0.8\), 80% of variation in data is explained by model.

### What are some challenges with regression and how to overcome them?

**High multicollinearity** is a challenge. It basically means one or more independent variables are highly linearly dependent on another independent variable. This makes it difficult to estimate the coefficients. One possible solution is to increase the sample size.

Another challenge is non-constant error variance, also called **heteroscedasticity**. An example of this when the observations "funnel out" as we move along the regression line. One solution is to use a Weighted Least Squares (WLS).

Regression assumes that errors from one observation are not related to other observations. This is often not true with time series data. **Autocorrelated errors** are therefore a challenge. One approach is to estimate the pattern in the errors and refine the regression model.

Another problem is **overfitting** that occurs when the model is "too well-trained". Such a model will not fit any other data. *Regularization* is the technique used to avoid overfitting. For parametric models, there are regression routines that address overfitting concerns. Lasso regression and ridge regression are a couple of such routines.

### Could you share some tips for beginners getting into regression modelling?

Here are a few useful tips:

    + It's known that when not enough data is collected, \(R^2\) is overestimated. Collect sufficient data.
    + Use *partial F-test* to identify predictors that can explain most of the variance in the predicted variable. Try to select as few predictors as possible to simplify analysis.
    + Try different techniques for cross-validation, such as independent samples or split samples.
    + Starting with lots of predictors might result in bad analysis. Start with a narrower focus.
    + Analysis is sensitive to bad data. Be careful about how data is collected.
    + Let decision makers be aware of the error term. See if the predictions make sense. Don't blindly believe in data: combine it with intuition.

## Milestones

1795

Carl Friedrich Gauss invents the **method of least squares**. He doesn't publish the method until much later in 1809. He uses it to predict the position of the celestial body named Ceres. Squared error is easy to compute and the error from this method is also normally distributed.

1805

Adrien-Marie Legendre publishes his invention of the method of least squares independently of Gauss. He uses it for the determination of orbits of comets.

1875

Francis Galton analyzes the sizes of mother and daughter sweet-pea seeds. He also makes a 2D-plot comparing the two, thereby obtaining the first insights into regression. He presents his first **regression line** in 1877. He notices that extreme values are "dampened" in the next generation whose values are closer to the mean. The idea of **regression to the mean** starts with Galton. Galton initially uses the term *reversion* rather than *regression*.

1896

Karl Pearson gives a mathematical treatment of correlation and regression using **product-moment method**.

1898

Francis Galton considers the role of previous generations of ancestors on one individual, thus recognizing that multiple variables can affect the predicted variable. The idea of **multivariate regression** starts here but developed only later by Karl Pearson.

1915

R. A. Fisher gives the exact sampling distribution of the correlation coefficient. Rigorous mathematical treatment of multivariate analysis also starts with Fisher through his **z-transformation** and **F distribution**.

1962

G.E.P. Box and P.W. Tidwell investigate transformations on predictor variables. Such transformations become useful to maintain the assumptions of independence, normality and variance homogeneity.

1970

A.E. Hoerl and R.W. Kennard look into the problem of near linear dependencies in the predictors. They propose **ridge regression** as a solution that uses suitable *biasing parameters*.

Regression Modelling


# Dual Connectivity

## Summary

Dual Connectivity (DC) is a feature that was first introduced in LTE (Release 12) to allow a UE to be connected to two eNBs and send/receive packets via both eNBs. The eNBs are connected via a non-ideal backhaul.

In 5G (Release 15), this concept was expanded so that a UE can be connected to both LTE E-UTRA and 5G NR nodes. The Core Network (CN) is either LTE EPC or 5G Core. This was later expanded so that both cells can belong to 5G NR, in which case the CN is exclusively 5G Core. These various options come under the general term **Multi-Radio Dual Connectivity (MR-DC)**. MR-DC is a generalization of Intra-E-UTRA Dual Connectivity.

MR-DC can offer a UE more resources for higher throughput. More commonly, it helps operators improve mobility robustness and handovers in macro/micro-cell deployments. It aids in migrating their networks from 4G to 5G.

## Discussion

### Why do we need MR-DC?

UL/DL separation, user/control plane separation, and user plane traffic aggregation via multiple nodes are some benefits of DC. DC can help operators integrate WLAN with their cellular networks. DC can also be used for redundancy for URLLC services with traffic duplication. Duplicated traffic may go via the same node or a second node.

An E-UTRA node can provide mobility and initial access to the UE while an NR node joins in later to offer higher throughput. Essentially, E-UTRA master node handles the control plane and both E-UTRA and NR nodes handle the user plane. Thus a more stable LTE macro layer is overlaid on top of a high-throughput NR small cell layer. We could say that LTE provides the anchor while 5G provides more bandwidth.

### How can MR-DC help operators in their 4G-to-5G migration?

With **Non-Standalone (NSA)**, operators can extend the life of their 4G equipment while deploying 5G. Regardless of the type of Core Network, NSA allows both 4G and 5G nodes to interwork. These options are named Option 3 (and variants 3a and 3x), Option 4 and Option 7.

The path of least investment is probably Option 3. Operators deploy only 5G gNB nodes to roll out 5G NR. These connect to EPC core via existing 4G eNB nodes. Thus, neither 5G core nor new backhauls need to be installed. A gNB connects to an eNB via a cheaper non-ideal backhaul, called the X2 interface.

From Option 3, an operator could migrate the core from EPC to 5G Core (Options 4 or 7). Eventually, all eNBs can be replaced with gNBs, leading to **Standalone (SA)** deployment Option 2. In any case, eNBs and gNBs could coexist for a long time since Dynamic Spectrum Sharing (DSS) allows spectrum to be used flexibly between 4G and 5G.

### What's the terminology used in MR-DC?

In MR-DC, a UE is connected to two nodes. One is called the **Master Node (MN)** and the other is called the **Secondary Node (SN)**. MN is the node to which the UE first connects. Subsequently, using RRC signalling messages via the MN, UE connects to the SN.

Historically, only MN provided the control plane connection between the UE and the Core Network (CN). SN provided only additional resources to carry user plane traffic. However, with the later introduction of split SRB and SRB3, SN can also carry signalling messages.

A MN can be an eNB (in EN-DC), a ng-eNB (in NGEN-DC) or a gNB (in NR-DC and NE-DC). Similarly, a SN can be en-gNB (in EN-DC), a ng-eNB (in NE-DC) or a gNB (in NR-DC and NGEN-DC).

MR-DC can be used alongside Carrier Aggregation (CA). Thus, both MN and SN may be associated with multiple cells or carriers. These aggregated carriers are collectively called **Master Cell Group (MCG)** and **Secondary Cell Group (SCG)**. Each group has one primary cell (SpCell) and potentially many secondary cells. The primary cell of MCG is called PCell and of SCG is called PSCell. It carries the PUCCH.

### What are the different configuration options in MR-DC?

MR-DC has four configurations:

    + **EN-DC (E-UTRA-NR Dual Connectivity)**: UE connects to a LTE eNB as MN and a 5G en-gNB as SN. CN is LTE EPC. en-gNB may connect to EPC via S1-U interface.
    + **NGEN-DC (NG-RAN E-UTRA-NR Dual Connectivity)**: UE connects to ng-eNB as MN and gNB as SN. CN is 5GC.
    + **NE-DC (NR-E-UTRA Dual Connectivity)**: UE connects to gNB as MN and ng-eNB as SN. CN is 5GC.
    + **NR-DC (NR-NR Dual Connectivity)**: UE connects to one gNB as MN and another gNB as SN. Alternatively, a single gNB can function as both MN and SN. CN is 5GC.EN-DC corresponds to Option 3, NGEN-DC to Option 4, NE-DC to Option 7, and NR-DC to Option 2.

We note that EN-DC uses LTE EPC whereas other MR-DC configurations use 5G Core. MN and SN are connected via the X2 interface (in EN-DC) and Xn interface (NGEN-DC, NE-DC, NR-DC). MN must have a connection to the CN. MN and SN must be connected even if it's a non-ideal backhaul.

### What's the difference between synchronous and asynchronous DC?

LTE specifies both synchronous and asynchronous DC. Synchronous DC implies that the two eNBs involved in DC need to be time synchronized. UE can cope with 33μs of maximum reception time difference and 35.21μs maximum transmission time difference. With asynchronous DC, no such time synchronization is required.

Likewise, synchronous and asynchronous DC are applicable for MR-DC configurations. Asynchronous NR-DC was specified in Release 16.

### What bearers are used in MR-DC?

An MCG bearer is a radio bearer with an RLC bearer only in MCG. Likewise, a SCG bearer has an RLC bearer only in SCG. A Split Bearer is a radio bearer with RLC bearers in both MCG and SCG. Where the bearer is used for signalling, we use the terms MCG SRB, SCG SRB and Split SRB.

In the control plane, RRC PDUs from SN go via MN unless SRB3 is configured. SRB3 is possible only when SN is a gNB or en-gNB. With SRB3, RRC PDUs can be sent directly from SN to UE. With Split SRB, RRC PDUs from MN can be duplicated and sent via both MCG and SCG. Split SRB is possible for all MR-DC configurations. RRC PDUs from SN can't be duplicated this way.

In the user plane, MCG/SCG/Split bearer may be terminated either in MN or SN. For example, an SCG bearer can be terminated in MN by passing PDCP PDUs between SN RLC and MN PDCP over the X2 or Xn interface. With split bearers, PDCP does routing and duplication.

### What's the higher layer signalling for MR-DC?

The UE has a single RRC state based on the MN RRC state. UE connects to the CN via single control plane connection. Initial access (SRB0) and RRC configuration (SRB1) are via the MN but later reconfigurations can be from either MN or SN. EN-DC starts with E-UTRA PDCP but this can be reconfigured to use NR PDCP.

If there's a link failure in SCG but MCG link is fine, no re-establishment procedure is triggered. Release 16, extended this concept to include fast MCG link recovery; that is, even when MCG link fails but SCG link is fine, no re-establishment procedure will be triggered. Instead, SCG link is used to recover MCG link.

Full details of signalling for Intra-E-UTRA DC and EN-DC are specified in TS 36.300. RRC signalling for other MR-DC configurations are specified in TS 38.331.

### How is uplink power control achieved in MR-DC?

Uplink power control procedures defined for E-UTRA and NR continue to be applicable with some additional constraints. The UE is required to conform to the configured maximum transmission power in each cell group. UE is also configured with total maximum transmission power for the applicable MR-DC configuration. Power sharing can be dynamic or semi-static.

In Intra-E-UTRA DC, power control modes 1 and 2 are available. UE will reduce its power in SCG if subframes in MCG and SCG overlap and total power exceeds a threshold.

In EN-DC and NE-DC, if the UE supports dynamic power sharing, and \(P\_{MCG}(i\_1) + P\_{SCG}(i\_2) > P\_{Total}\) for slots \(i\_1\) and \(i\_2\), then UE reduces in the power in the NR node (MCG or SCG) so that total transmit power doesn't exceed the total configured value.

In NR-DC, power control is done independently for MCG and SCG when they use different ranges (FR1 and FR2). Where inter-CG power sharing is applicable, transmission powers for MCG and SCG are determined per frequency range.

### How is DC different from CA?

DC and CA are not mutually exclusive and can be combined. In both DC and CA, a UE is connected to more than one cell. With CA, all cells belong to the same node (eNB or gNB). Scheduling decisions are made jointly for all cells. With DC, the co-ordination between MCG and SCG is a lot looser and scheduling decisions are made in different nodes.

Cells from different radio access technologies (RATs) can be used for DC but not for CA. DC is essential for 5G Non-Standalone operation but CA is not.

A UE has a single MAC entity for CA. For DC, a UE has two MAC entities. A DC UE also requires 2 TX. In CA, MAC splits the traffic among Component Carriers (CCs). In DC, traffic is split at PDCP.

All traffic due to CA are transported via ideal backhauls. This is not the case with DC. Non-ideal backhauls (with higher latency) may be used for the X2 or Xn interface. In fact, it's been said that,

> Dual connectivity can be seen as carrier aggregation extended to the case of non-ideal backhaul.

### What are the engineering challenges with implementing MR-DC?

Self-interference is one of the challenges. In DC, a UE will be transmitting on multiple carriers. Their intermodulation products may fall within the receiver band. To avoid this interference, RF circuitry will need to be designed with less non-linearity, which may increase cost and power consumption. One way to avoid this problem is to constrain the UE to **single-TX DC** for some band combinations. UE is not allowed to do simultaneous transmissions towards eNB and gNB. Schedulers in these nodes coordinate to satisfy this constraint.

The figure shows an example of B41/n41 EN-DC spectral mask measured at one of the two antennas. We can see that the transmission fails spectral mask requirements due to low frequency intermodulation product.

### Which are the main specifications relevant to Dual Connectivity?

Overall descriptions of Dual Connectivity can be found in **TS 36.300** (Intra-E-UTRA DC) and **TS 37.340** (MR-DC involving E-UTRA and NR, or NR only).

For Intra-E-UTRA-DC, DC valid band combinations are specified in **TS 36.101**, section 5.5C. For example, *DC\_1-20* combines DL bands 2110-2170 MHz with 791-821 MHz (and their paired UL FDD bands).

Valid band combinations for EN-DC, NGEN-DC and NE-DC are specified in a series of documents: **TS 37.716**, **TS 37.717** (Release 17), and **TS 37.718** (Release 18). Likewise, for NR-DC we have **TS 38.716**, **TS 38.717** and **TS 38.718**. These documents also include bands for inter-band Carrier Aggregation (CA). Some band combinations may result in self-interference and these are noted in these documents.

## Milestones

Dec
2013

As part of Release 12 work, 3GPP publishes the report **TR 36.842** that describes small cell enhancements. Dual connectivity is introduced in this report as one of the possible solutions.

Mar
2015

Release 12 of LTE-Advanced is largely completed and functionally frozen. This release introduces **Intra-E-UTRA Dual Connectivity**.

Aug
2015

Ericsson in partnership with KT demonstrates Dual Connectivity. The demo uses two LTE cells, a macro cell and a small cell.

Mar
2016

Release 13 3GPP specifications is published. This introduces **LTE-WLAN Aggregation (LWA)**. Although called aggregation, it's more like DC than CA. This is the first time that a UE can connect simultaneously to two different RATs.

Dec
2017

3GPP publishes Release 15 "early drop". **Dual connectivity between E-UTRA and NR** is supported in this release. This enables DC in Non-Standalone deployment scenarios but not in Standalone scenarios.

Jun
2018

3GPP publishes Release 15 "main drop". **NR-DC** is supported in this release, thus enabling DC even in 5G Standalone deployment scenarios.

May
2020

Nokia claims to achieve a world record speed of 4.7 Gbps on a commercially deployed network in the U.S. This is achieved using EN-DC with 40 MHz of LTE spectrum and 8 x 100 MHz of millimeter wave spectrum in 28 GHz and 39 GHz bands. A 5G cloud RAN is used for this test.

Jun
2020

Release 16 is completed and functionally frozen. This includes many enhancements or additions to DC including idle/inactive mode measurements, SCG/SCell configuration during transition from idle/inactive state, dormant SCell in NR, fast recovery from MCG link failure, and support for asynchronous NR-DC operation. UL Tx Switching (EN-DC) is another new feature in this release.

Apr
2021

Ericsson and MediaTek demonstrate NR-DC and achieve 5.1 Gbps using 8 x 100 MHz (@ 28 GHz, n261) and 60 MHz (@ 3.7 GHz, n77). In December 2021, Ericsson and Singtel achieve 5.4 Gbps using NR-DC at 3.5 GHz (mid-band) and 28 GHz (mmWave). This sort of speed enables high-performance low-latency applications including immersive gaming, AR/VR, autonomous vehicles and robotic control.

Dual Connectivity


# Linux Signals

## Summary

A Linux computer system has many processes in different states. These processes belong to either user applications or the operating system. We need a mechanism for the kernel and these processes to coordinate their activities. One way to do this is for a process to inform others when something important happens. This is why we have **signals**.

A signal is basically a one-way notification. A signal can be sent by the kernel to a process, by a process to another process, or a process to itself.

Linux signals trace their origins to Unix signals. In later Linux versions, real-time signals were added. Signals are a simple and lightweight form of interprocess communication, and therefore suited for embedded systems.

## Discussion

### What are the essential facts about Linux signals?

There are 31 standard signals, numbered 1-31. Each signal is named as "SIG" followed by a suffix. Starting from version 2.2, the Linux kernel supports 33 different real-time signals. These have numbers 32-64 but programmers should instead use `SIGRTMIN+n` notation. Standard signals have specific purposes but the use of SIGUSR1 and SIGUSR2 can be defined by applications. Real-time signals are also defined by applications.

Signal number 0, which POSIX.1 calls *null signal*, is generally not used but `kill` function uses this as a special case. No signal is sent but it can be used (rather unreliably) to check if the process still exists.

Linux implementation of signals is fully POSIX compliant. Newer implementation should prefer to use `sigaction` rather than the traditional `signal` interface.

Just as hardware subsystems can interrupt the processor, signals interrupt process execution. They are therefore seen as *software interrupts*. While interrupt handlers process hardware interrupts, signal handlers process signals.

Some signals are mapped to specific key inputs: SIGINT for `ctrl+c`, SIGSTOP for `ctrl+z`, SIGQUIT for `ctrl+\`.

### How does a signal affect the state of a Linux process?

Some signals terminate the receiving process: SIGHUP, SIGINT, SIGTERM, SIGKILL. There are signals that terminate the process along with a core dump to help programmers debug what went wrong: SIGABRT (abort signal), SIGBUS (bus error), SIGILL (illegal instruction), SIGSEGV (invalid memory reference), SIGSYS (bad system call). Other signals stop the process: SIGSTOP, SIGTSTP. SIGCONT is a signal that resumes a stopped process.

A program could override default behaviour. For example, an interactive program could be written to ignore SIGINT (generated by `ctrl+c` input). Two notable exceptions are SIGKILL and SIGSTOP signals, which can't be ignored, blocked or overridden this way.

Let's consider the example of a parent process and its child process. Suppose the child sends SIGSTOP to itself, child process will be stopped. This in turns triggers SIGCHLD to the parent. Parent can then signal the child to continue using SIGCONT. When child comes out of stopped state, another SIGCHLD is sent to parent. If later, the child process exits, the final SIGCHLD is sent to parent.

### Aren't signals similar to exceptions?

Some programming languages are capable of exceptions using constructs such as `try-throw-catch`. Signals are not similar to exceptions. Instead, failed system or library calls return non-zero exit codes. When a process is terminated, it's exit code will be 128 plus signal number. For example, a process killed by SIGKILL will return 137 (128+9).

### Are Linux signals synchronous or asynchronous?

When signals are generated, they can be considered as synchronous or asynchronous.

**Synchronous** signals occur as a result of instructions that have led to an unrecoverable error such as an illegal address access. These signals are sent to the thread that caused it. These are also called *traps*, since they also cause a trap into the kernel trap handler.

**Asynchronous** signals are external to the current execution context. Sending SIGKILL from an another process is an example of this. These are also called *software interrupts*.

### What's the typical lifecycle of a signal?

A signal goes through three stages:

    + **Generation**: A signal can be generated by the kernel or any of the processes. Whoever generates the signal, addresses it to a specific process. A signal is represented by its number and has no extra data or arguments. Thus, signals are lightweight. However, extra data can be passed along for POSIX real-time signals. System calls and functions that can generate signals include `raise`, `kill`, `killpg`, `pthread_kill`, `tgkill`, and `sigqueue`.
    + **Delivery**: A signal is said to be *pending* until it's delivered. Normally a signal is delivered to a process as soon as possible by the kernel. However, if the process has blocked the signal, it will remain pending until unblocked.
    + **Processing**: Once a signal is delivered, it is processed in one of many ways. Every signal has an associated default action: ignore the signal; or terminate the process, sometimes with a core dump; or stop/continue the process. For non-default behaviour, *handler function* can be called. Exactly which of these happens is specified via `sigaction` function.

### Could you explain the concept of blocking and unblocking signals?

Signals interrupt the normal flow of program execution. This is undesirable when the process is executing some critical code or updating data that's shared with signal handlers. Blocking solves this problem. The tradeoff is that signal handling is delayed.

Every process can specify if it wants to block a specific signal. If blocked and the signal does occur, the operating system will hold the signal as pending. The signal will be delivered once the process unblocks it. The set of currently blocked signals is called **signal mask**.

There's no point blocking a signal indefinitely. For this purpose, the process can instead ignore the signal once it's delivered.

A signal blocked by one processes doesn't affect other processes, who can receive the signal normally.

Signal mask can be set using `sigprocmask` (single-threaded) or `pthread_sigmask` (multi-threaded).  When a process has multiple threads, a signal can be blocked on a per thread basis. Signal will be delivered to any one thread that hasn't blocked it. Essentially,

> Signal handlers are per process, signal masks are per thread.

### Can we have more than one signal pending for a process?

Yes, many standard signals can be pending for a process. However, only one instance of a given signal type can be pending. This is because pending and blocking of signals are implemented as bitmasks, with one bit per signal type. For example, we can have SIGALRM and SIGTERM pending at the same time but we can't have two SIGALRM signals pending. The process will receive only one SIGALRM signal even if raised multiple times.

With real-time signals, signals can be queued along with data,  so that each instance of the signal can individually delivered and handled.

POSIX doesn't specify the order of delivery for standard signals, or what happens if both standard and real-time signals are pending. Linux gives priority to standard signals. For real-time signals, lower numbered signals are delivered first, and if many are queued for a signal type, earliest one is delivered first.

## Milestones

1990

Signals are described in the POSIX.1-1990 standard. These can be traced to the IEEE Std 1003.1-1988.

1993

Real-time extensions are released as POSIX.1b. This includes real-time signals.

1999

Linux starts supporting real-time signals with the release of kernel version 2.2.

2001

More signals are added in the POSIX.1-2001 standard: SIGBUS, SIGPOLL, SIGPROF, SIGSYS, SIGTRAP, SIGURG, SIGVTALRM, SIGXCPU, SIGXFSZ.

Linux Signals


# BEM Methodology

## Summary

When updating web/mobile user interfaces, frontend developers often worry about breaking existing code or introducing inconsistencies. Selecting elements on a UI is often problematic because of long cascading rules, nested styles, rules that are unnecessarily high in specificity, and naming everything in the global namespace. BEM provides a modular approach to solve these problems.

BEM is a methodology, an approach and a naming convention to write better CSS styling and achieve consistent JS behaviour. **BEM** stands for *Block*, *Element* and *Modifier*. These three are called **BEM Entities**.

BEM is not a W3C standard but it's recommended by those who have adopted it and reaped its benefits. Among its adopters are Yandex, Google, BBC, Alfa Bank, and BuzzFeed. BEM also works nicely with CSS languages (Sass, etc.), and frontend frameworks (React, etc.).

## Discussion

### Could you explain BEM with an example?

BEM is a component-based approach to design a user interface for the web/mobile. For example, we can identify components such as header, menu, logo, search and login on a typical web page. In BEM, we call these components **blocks**, each having a well-defined purpose or semantics. It's also apparent that a block can contain other blocks within it. However, each block is also independent of other blocks and can be reused elsewhere. For example, it's not necessary that the login block must be part of the header block. It could be moved to a sidebar in a redesigned page layout.

Blocks can also contain **elements** that cannot exist outside their parent blocks. For example, a menu block is composed of menu items, each of which is an element in BEM terminology.

Finally, an element within a block can be selectively modified depending on the context. For example, in a menu block, the active menu item could be styled differently. This modification is done using **modifiers**. Modifiers are optional.

### What are the advantages of adopting the BEM methodology?

BEM enables flexible maintainable code. Components can be reused. Code becomes more consistent since the rules are clear. It's easier to scale to large projects. Teamwork is easier because everyone is familiar with BEM conventions. Since the approach is based on independent blocks, work can be clearly partitioned, with each team member working on a different block.

More specifically, every BEM CSS class is unique and self-sufficient. BEM doesn't need nesting of styles. We don't need long and inefficient selectors to match specific elements. BEM avoids name collisions due to its consistent naming convention. The naming convention also makes the CSS self-documenting: the relationships among DOM nodes is easier to see. DOM nodes are selected more efficiently with managed specificity.

### What's the default naming conventions followed in BEM?

Let's consider the following: `block-name__elem-name_mod-name_mod-val`. We note that names are all in lowercase, underscores are used as separators between BEM entities, and hyphens are used within names for better readability. We use double underscore to separate block from element; single underscore to separate block/element name from its modifier; single underscore to separate modifier name and modifier value.

Consider these examples:

    + `main-menu_hidden`: main-menu is the block, hidden is the modifier.
    + `menu_theme_islands`: menu is the block, theme is the modifier, islands is the theme being specified (modifier value).
    + `menu__item_type_radio`: menu is the block, item is its element, type is item's modifier, radio is the modifier's value.
    + `main-menu__item_black-listed`: main-menu is the block, item is its element, black-listed is item's modifier.

### What other naming conventions are common in the BEM community?

While there's a default naming convention in BEM, you can create your own convention. Most importantly, the convention should clearly separate blocks, elements and modifiers. Other than the default one, the following conventions are used in the BEM community:

    + **Two Dashes**: `block-name__elem-name--mod-name--mod-val`: double hyphens are used between elements, modifiers and values.
    + **CamelCase**: `blockName-elemName_modName_modVal`
    + **React**: `BlockName-ElemName_modName_modVal`
    + **No Namespace**: `_modname`: no reference to block or element and hence of limited usefulness.

### What technologies assist in implementing BEM?

A block is not just about CSS. It encapsulates behaviour (JavaScript), templates, styles (CSS), documentation, and other implementation technologies. Developers can adopt any implementation technology of their choice. As a sample, we note the following:

    + **Styling**: CSS, Stylus, Sass
    + **Behaviour**: JavaScript, CoffeeScript
    + **Templates**: BEMHTML, BH, Pug, Handlebars, XSL
    + **Documentation**: Markdown, Wiki, XML

### Could you share some BEM best practices?

An important principle of BEM is to avoid selectors based on ID or tag name. Only class names are used, and these can be reused easily regardless of the tag. It's also allowed to use multiple BEM entities on the same DOM node.

We can use nesting in BEM but it's not recommended. Nesting increasing code coupling and inhibits reuse. Likewise, combined selectors (eg. `.btn.btn_error`) are not recommended.

Use only a single double underscore in a name: `c-card__img` rather than `c-card__body__img`. Use namespaces to improve readability: `c-` for component, `l-` for layout, `h-` for helpers, `is-` or `has-` for states , `js-` for JavaScript hooks.

External geometry and positioning (such as padding) should be done on parent node so that the current block can be reused elsewhere. Use **mixes** to apply the same style to multiple DOM nodes. This approach is better than using group selectors.

Take an object-oriented approach to design. Design in terms of reusable blocks. Use principles such as single responsibility principle and open/closed principle. Don't repeat yourself (DRY). Prefer composition over inheritance. Organize your files into a clear folder structure.

### What are some criticisms of BEM?

Since BEM avoids the use of IDs or tag names, class names have to be unique. Giving these names can be a difficult task. Long class names bloat markup. To reuse UI components, we need to explicitly extend them.

As an example, a modifier will need its own class name but giving names for each DOM node that needs to be styled with this modifier can be cumbersome. An alternative is to use utility classes, named with `u-` prefix. The trade-off is that this will result in nested CSS. Thus, a DOM node is selected as `menu__item u-active` rather than `menu__item-active`.

Since DOM nodes are marked with BEM entities, BEM creates a semantic overlay on the DOM. This is called **BEM Tree**. While some regard this as a useful thing, others claim that BEM unnecessarily makes the markup semantic.

## Milestones

Dec
1996

The first version of **Cascading Style Sheets (CSS)** comes out as a W3C Recommendation. The idea of having style sheets can be traced to Tim Berners-Lee, who used it in his NeXT browser (1990), and Pei Wei's Viola browser (1992). Håkon Wium Lie released an initial draft in October 1994.

2006

At Yandex, developers find that for large projects, layouts are becoming difficult to maintain. Long cascading rules and dependencies cause changes in one web page to affect many others. To solve this, they introduce the concept of **blocks**. A block is part of a page or layout defined semantically or visually. A block is given a unique CSS class name. Nodes within blocks are called **elements**. For example, logo is an element with a header block.

2007

Block names use prefixes to emulate namespaces. Each prefix has its own semantics: `b-` for blocks, `h-` for block wrappers, `l-` for layouts, `g-` for global styles. This is also when **modifiers** are introduced to indicate block state or property. Modification can be context-dependent such as location; or context-independent.

Mar
2008

With the release of Lego 2.0, the unified styling framework at Yandex, **BEM** is formally born. Blocks are primary units. A block can be used anywhere on the page. Each block is stored in a separate folder with its own documentation. Blocks used are noted in XML files and used to generate CSS files.

2009

Folks at Yandex invent **Absolutely Independent Blocks (AIB)** to solve the problem of slow DOM updates. Each DOM node has its own class. Tag rules are avoided in CSS.

2010

BEM is open sourced along with some essential BEM tools.

2015

**CSS Modules** is born with the initial commit happening in May 2015. BEM requires developers to choose unique class names and does not provide encapsulation. CSS Modules solves this by dynamically creating class names and localising styles. Thus, tooling is just as important as conventions. CSS Modules doesn't replace BEM. It makes it easier to implement BEM.

BEM Methodology