# Search Engine Optimization

## Summary

Users online use search engines to find information or resources. If you're the owner of a website, you always want to bring more visitors to your site. If search engines give better visibility to your site in response to relevant search queries, this would translate to more visitors. But how do you tell search engines that your site is better than others? This is where **Search Engine Optimization (SEO)** plays an important part.

SEO is the process of applying a set of techniques so that search engines rank your site or page higher, without you paying them to do so. We could optimize on content, organization, design, performance, etc. To know what to optimize is not obvious and requires a good understanding of the many factors that search engines use to rank web pages.

## Discussion

### What are some important factors that search engines use for ranking web pages?

As of December 2018, it's recognized that Google uses more than 200 ranking factors. These can be categorized into domain related, site level or page level factors. For example, a keyword appearing in domain name can boost ranking. At site level, content uniqueness, content freshness, site trust rank, site architecture, use of HTTPS, and usability are some factors. At page level, use of keywords in title tag, content length, topic coverage, and loading speed are some factors.

Search results that get more clicks may be ranked better in future. Bounce rate may be used to adjust ranking. Repeat visits to a page and higher dwell time (time spent on a page) may boost ranking. Likewise, pages with lots of comments may get a boost.

Search engines may also apply algorithmic rules. For example, priority may be given to recently published pages due to freshness. For diversity, results may include different interpretations of the keyword. User's search history is used to give contextual results. Geo-targeted matches may be given priority for some queries.

### If there's only one important SEO technique, what would it be?

In the words of Google SEO Starter Guide 2017,

> Creating compelling and useful content will likely influence your website more than any of the other factors discussed here.

Content comes first. It should be unique and of high quality. It should add value to those who visit your site. When it's backed by great design, we can achieve great user experience across devices. This means that aiming for higher search ranking is not the primary goal. The focus should be on content and user experience. Every other SEO technique should not stray from this focus.

Apart from this, SEO is about keywords, links, relevance, reputation and trust.

### What's the meaning of link building for SEO?

Link building is the process of getting more links to your own webpages. Incoming links, also called **backlinks**, is an important SEO signal that leads to better ranking. Total number of links, number of linking root domains, number of links from separate C-class IPs are all important. Age and authority of pages or domains linking to your site are also important. The anchoring text used for a backlink is also relevant.

The recommended approach is to earn backlinks rather than buy them. Earning backlinks is a slow process but this builds your site's reputation in a natural way for better long-term results. To earn backlinks, create unique high-quality content, promote your content, get positive reviews from influencers, and partner with relevant sites without resorting to link scheming.

**Internal link building** is something on which you have greater control. Build internal links based on keyword research and information architecture of your site. Map each page to keywords and user intent. Use this to create an SEO wireframe to help you build internal links. Use navigation bars, menus and breadcrumbs to link to key pages.

### How can I make use of social signals for better SEO?

Social signals are basically user engagement on social media platforms. These include likes, shares, comments, and so on.

Social signals are important for ranking in Bing search engine. Google Search has said that social signals don't directly influence ranking. However, research has shown good correlation between ranking and social signals. This is because social signals amplify other SEO factors that in turn affect ranking.

It therefore makes sense to optimize your social presence to indirectly affect ranking. Your profile and branding should be consistent across platforms. Link your profiles to your site. Post regularly across platforms. Post per day about 15 tweets, 1 LinkedIn post, 1 Facebook post, and 2 Instagram posts. Use viral headlines and images in posts. Use hashtags, which are essentially keywords. Explicitly ask for shares. On your web pages, use social media plugins to make sharing easier. For best possible engagement, share only your best posts.

### What should web designers keep in mind for better SEO?

Designers should **design for SEO** rather than just user experience (UX). One approach is to make good use of interactions (such as mouseover or mouse click) and expandable elements. For example, page can load with a minimalistic view but via interactions give more information that's also crawlable.

Image-based banners and call-to-action elements are not crawlable. Image's `alt` attribute is plain text and of limited length. Instead, go for crawlable **text-based design** using webfonts, HTML and CSS, along with schema markup.

Design should be responsive. More than just fitting to different screen sizes, aim for consistent UX across devices. For example, have a fluid design that maintains proportions. Google's Mobile-Friendly Test can inform if your site's well designed for mobiles.

Use **header tags** to organize content both for visitors and search engines. Using header tags in sidebars, header and footer is not a good SEO practice. Pop-ups and banners are penalized in mobile SEO. Due to voice interfaces, prioritize your design to be heard rather than seen.

### What is meant by black hat SEO versus white hat SEO?

**White Hat SEO** is when an SEO professional doesn't deviate from the rules defined by search engines. Getting results via white hat SEO takes time but they also last longer. Sites using only white hat SEO will not be penalized by search engines since they play by the rules.

**Black Hat SEO** is when an SEO professional try to game the system to get better rankings. Search engine guidelines and rules are flouted. Black hat techniques may give quick results but there's a risk of getting penalized. However, we should point out that breaking search engine rules is not illegal.

Buying links is a black hat technique. Automatically following someone who follows you on social media is another one. Cloaking is a black hat practice of showing one version of a web page to search crawlers and another version to normal visitors. Another one is keyword stuffing where irrelevant keywords are forced into a page.

In practice, no site is 100% white hat. When white hat SEO professionals try to acquire links, they're crossing into "gray hat" territory.

### Are there SEO techniques that I should avoid?

Google has shared some techniques to avoid: automatically generated content, scraped content, pages with little original content, cloaking, sneaky redirects, hidden texts or links, doorway pages, using irrelevant keywords, abusing rich snippets markup, sending automated queries to Google, etc. Too many ads on a page lowers the ranking.

Bing has shared a similar list to avoid: link buying or spamming, social media schemes, meta refresh redirects, duplicated content, keyword stuffing and misleading markup.

Don't optimize on a single keyword since search engines focus on search intent. Don't design for just desktops. Make your design mobile friendly, fast, and responsive. Don't focus on quantity by duplicating or copying content; quality of your content is more important. Don't put important content in formats that most likely won't be crawled, such as, PDF files or images. Worse still, using `noindex` in meta tags or robots.txt file wrongly can tell crawlers to ignore your content.

Finally, don't think that SEO is a one-time task. Keep adding new content to your site. If you change your site's structure, properly redirect old URLs. Stay updated on new SEO trends.

### What are some common myths about SEO?

Here are some SEO myths and clarifications of the same:

    + Boost your rankings overnight by hiring an SEO agency: improving rankings takes time.
    + Guest blogging is obsolete: avoid spamming blogs and publish quality content.
    + SEO is a one-time effort: it's a continuous process.
    + Link building is dangerous: grow links naturally without being manipulative.
    + CTR is no longer relevant: they still influence ranking but clicks from bots will be penalized.
    + Keywords and keyword research are dead: they're important but think in terms of context and search intent; keyword ratio is useless.
    + Keyword-optimized anchor text is bad: it's bad if it's overoptimized; hence aim for diversity (natural, brand, URL, generic).
    + Paid rankings will improve organic rankings: they're in fact treated separately.
    + XML sitemap improves rankings: it doesn't but search engines will index the site faster.
    + Meta tags are irrelevant: they don't affect rankings but they help search engines understand your site better.
    + H1 tags are important for rankings: not really but use them to organize your content and help users navigate more easily.

### What are some tools that help with SEO?

Google Search Console is an essential tool. Formerly called Google Webmaster Tools, it provides rankings and traffic reports for keywords and pages.

For backlink analysis, AHREFs and Majestic are useful. With these, we get to know who's linking to our site or to sites of our competitors. With this information, we can plan for link building. Similar tools include Buzzsumo, FollowerWonk, and Little Bird.

For keyword research, use Google's Keyword Planner, although this is useful only for paid search. For organic search, consider using Moz Keyword Explorer tool and SEMRush’s Keyword Magic Tool. Google Trends is also useful for competitive analysis on keywords.

Since performance is an SEO factor, use Google PageSpeed Insights, Pingdom, or WebPageTest to know areas where performance can be improved. For local SEO, use Moz Local or Whitespark.

SEO platforms bring together many tools for analyzing and optimizing our site. Moz, BrightEdge, Searchmetrics, Linkdex, and SEO PowerSuite are some examples.

To know if SEO is giving better results, use Google Analytics.

## Milestones

1990

The history of SEO is naturally tied to the history of **search engines**. What's probably the world's first search engine, Archie is released in 1990. The World Wide Web Wanderer, later called Wandex, is released in 1993 as one of the first web crawlers. By 1994, many search engines are operational including Alta Vista, Infoseek, Lycos, and Yahoo. Google itself is founded in 1998 but it started in 1996 with the name BackRub.

1997

Danny Sullivan launches *Search Engine Watch*, a site for news on search industry and tips for better ranking.

1998

For sponsored links and paid search, Goto.com is launched. Advertisers bid on the site to rank higher than organic results. In fact, organic results were bad since search engines didn't succeed against black hat practices. The alternative was paid search or getting listed on popular directories such as Yahoo and DMOZ.

1999

The first conference focused on search marketing, called *Search Engine Strategies*, takes place.

2000

**Google Toolbar** becomes available within Internet Explorer browser. This allows SEO practitioners to see their PageRank score. Meanwhile, some folks start sharing SEO related information at a London pub. This later evolves into *Pubcon*, a regular search conference.

Nov
2003

With Google leading the search market, it releases the Florida update. Sites that were earlier ranked higher due to **keyword stuffing** (a black hat practice), lose their ranking. The Florida update gives better ranking for quality content and authentic backlinks. This partly solves the problem created earlier by Blogger.com and Google's AdSense that content creators gamed to make a quick buck.

2004

Google starts looking at local search intent and thus is born **local SEO**. This year is also when Google personalizes results based on search history and interests.

Jan
2005

Google, Yahoo and MSN jointly create the `nofollow` attribute to combat spammy links and comments.

2011

With Google's Panda update (delivered over many months), the search engine penalizes **content farms** created solely for driving search engine results. This affected sites having scraped and unoriginal content. Panda also penalized pages with high ad-to-content ratios.

2012

With the Penguin update of Google Search, sites with overoptimized anchor text, or spammy hyperlinks are penalized.

2013

With the Hummingbird update of Google Search, search is based on keywords but results are more about **search intent**. This update affects 90% of searches worldwide. Design for user intent among information (to know), location (to go), action (to do), or shopping (to buy). To help search engines figure out intent, let each page have a singular purpose.

2015

In April, Google Search gets a **mobile-friendly** update named Mobilegeddon. In October, Google releases an AI-powered search named *RankBrain*. It's initially used to interpret 15% of searches that Google has never seen before. In later years, RankBrain plays a more central role in all of Google searches.

Search Engine Optimization


# Word Embedding

## Summary

Word embedding is simply a vector representation of a word, with the vector containing real numbers. Since languages typically contain at least tens of thousands of words, simple binary word vectors can become impractical due to high number of dimensions. Word embeddings solve this problem by providing dense representations of words in a low-dimensional vector space.

Since mid-2010s, word embeddings have being applied to neural network-based NLP tasks. Among the well-known embeddings are word2vec (Google), GloVe (Stanford) and FastText (Facebook).

## Discussion

### Why do we need word embeddings?

Consider an example where we have to encode "the dog DET", which is about 'dog', its previous word 'the' and whose part of speech is determiner (DET). If we represent every word and every part of speech in its own dimension, we would require a high-dimensional vector since our vocabulary will have lots of words. The vector will mostly be zeros except in three places that represent 'dog', 'the' and 'DET'. Called *One-Hot Encoding*, this a **sparse** representation.

Instead, word embeddings give a **dense** representation in a lower-dimensional space. Each entity gets a unique representation in this vector space. As shown in the figure, both words have six dimensions each and the part of speech has four dimensions. The entire vector representation is now only 16 dimensions. This makes it practical for further processing.

More importantly, word embeddings capture similarities. For example, even if the word 'cat' is not seen during training, it's embedding would be similar to that of 'dog'. Likewise, different tenses of the same verb are correlated.

### Is word embedding related to distributional semantics?

Yes. The term "word embedding" has been popularized by the deep learning community. In computational linguistics, the more preferred term is **Distributional Semantic Model (DSM)**, which comes from the theory of distributional semantics. Other equivalent terms include distributed representation, semantic vector space or word space.

Essentially, words are not represented as a single number or symbol. Rather, the representation is distributed in a vector space of many dimensions. The notion of semantics emerges because two words that are close to each other in the vector space are somehow semantically related. Similar words form clusters in the vector space.

### How can we extract semantic relationships captured within word embeddings?

Word embeddings are produced in an unsupervised manner. We don't inform the model anything about syntactic or semantic relationships among words. Yet, word embeddings seem to capture these relationships. For example, country names and their capital cities form a relationship. Relations due to gender or verb tense of words are other examples.

To see this in practice, consider the following vector equations:

$$king\,–\,man\,+\,woman = queen\\Paris\,–\,France\,+\,Germany = Berlin$$

The relationship between 'king' and 'man' is same as that between 'queen' and 'woman'. This is captured in the vector space. This means that given the word vectors for Paris, France and Germany, we can find the capital of Germany. The term **word analogy** is often used to refer to this phenomenon.

### What are some applications of word embeddings?

Word embeddings have become useful in many downstream NLP tasks. Word embeddings along with neural networks have been applied successfully for text classification, thereby improving customer service, spam detection, and document classification. Machine translations have improved. Analyzing survey responses or verbatim comments from customers are specific examples.

Word embeddings help in adapting a model from one domain to another, such as from legal documents to news articles. In general, this is called **domain adaptation** that's useful for machine translation and transfer learning. In addition, pretrained word vectors can be adapted to domains where large training datasets are not available.

In recommendation systems, such as suggesting a playlist of songs, word embeddings can figure out what songs go well together in a particular context.

In search and information retrieval applications, word embeddings have been shown to be insensitive to spelling errors and exact keyword matches are not required. Even for words not seen during training, machine learning models work well provided such words are in the vector space. Thus, word embeddings are being preferred over older approaches such as TF-IDF or bag-of-words.

### What traits do we expect from a good word embedding?

Different models give different word embeddings. A good representation should aim for the following:

    + **Non-conflation**: A word can occur in different contexts giving rise to variants (tense, plural, etc.). Embedding should represent these differences and not conflate them.
    + **Unambiguous**: All meanings of the word should be represented. For example, for the word 'bow', the difference between "the bow of a ship" and "bow and arrows" should be represented.
    + **Multifaced**: Words have multiple facets: phonetic, morphological, syntactic, etc. Representation should change when tense changes or a prefix is added.
    + **Reliable**: During training, word vectors are randomly initialized. This will lead to different word embeddings from the same dataset. In any case, the final output should be reliable and show consistent performance.
    + **Good Geometry**: There should be a good spread of words in the vector space. In general, rare words should cluster around frequent words. Frequent unrelated words should be spread out.

### Which are the well-known word embeddings?

One of the first word embeddings is the **Neural Network Language Model (NNLM)** in which word embeddings are learnt jointly with the language model. Embeddings can also be learnt using Latent Semantic Analysis (LSA) or Latent Dirichlet Allocation (LDA).

NNLM has high complexity due to non-linear hidden layers. A tradeoff is to first learn the word vectors using a neural network with a single hidden layer, which is then used to train the NNLM. Other log-linear models are **Continuous Bag-of-Words (CBOW)** and **Continuous Skip-gram**. An improved version of the latter is Skip-gram with Negative Sampling (SGNS). These are part of the *word2vec* implementation.

CBOW and Skip-gram models use only local information. **Global Vectors (GloVe)** is an approach that considers global statistical information as well. Word-to-word co-occurrence counts are used. GloVe combines LSA and word2vec.

Rare words can be poorly estimated. **FastText** overcomes this by using subword information.

Other models include ngram2vec and dict2vec. **Embeddings from Language Models (ELMo)** is a representation that captures sentence level information. Based on ELMo, BERT and OpenAI GPT are two pretrained models for other NLP tasks that have been proven effective.

### What's the role of the embedding layer and the softmax layer?

The general architecture of word embeddings using neural networks involves the following:

    + **Embedding Layer**: Generates word embeddings from an index vector and a word embedding matrix.
    + **Hidden Layers**: These produce intermediate representations of the input. LSTMs could be used here. In word2vec, there are no hidden layers.
    + **Softmax Layer**: The final layer that gives the distribution over all words in the vocabulary. This is the most computationally expensive layer and much work has gone into simplifying this. Two broad categories are softmax-based approaches and sampling-based approaches.

### What's the process for generating word embeddings?

We can use a neural network on a supervised task to learn word embeddings. The embeddings are weights that are tuned to minimize the loss on the task. For example, given 50K words from a collection of movie reviews, we might obtain a 100-dimensional embedding to predict sentiment. Words signifying positive sentiment will be closer in the vector space. Since embeddings are tuned for a task, selecting the right task is important.

Word embeddings can be learnt from a standalone model and then applied to different tasks. Or it could be learnt jointly with a task-specific model. For good embeddings, we would need to train on millions or even billions of words. An easier approach is to use pretrained word embeddings (word2vec or GloVe). They can be used "as is" if they suit the task at hand. If not, they can be updated while training your own model.

In biomedical NLP, it was noted that bigger corpora don't necessarily result in better embeddings. Sometimes intrinsic and extrinsic evaluation methods don't agree well. Hyperparameters that we can tune include negative sampling size, context window size, and vector dimension. Gains plateau at about 200 dimensions.

### Could you share some practical tips for applying word embeddings?

Predictive neural network models (word2vec) and count-based distributional semantic models (GloVe) are different means to achieve the same goal. There's no qualitative difference. Word2vec has proven to be robust across a range of semantic tasks.

For syntactic tasks such as named entity recognition or POS tagging, a small dimensionality is adequate. For semantic tasks, higher dimensionality may prove more effective. It's also been noted that pretrained embeddings give better results. It's been commented that 8 dimensions might suffice for small datasets and as many as 1024 dimensions for large datasets.

In 2018, selecting the optimal dimensionality was still considered an open problem. Too few dimensions, embeddings are not expressive. Too many dimensions, embeddings are overfitted and model becomes complex. Commonly, 300 dimensions are used.

### What are some challenges with word embeddings?

Models such as ELMo and BERT, capture surrounding context within word embeddings. However, word embeddings don't capture "context of situation" the way linguist J.R. Firth defined it in the 1960s. To achieve true NLU, we would have to combine the statistical approach of word embeddings along with the older linguistic approach.

More generally, it's been said the deep learning isn't sample efficient. Perhaps we need something better than deep learning to tackle language with compositional properties.

Word embeddings don't capture phrases and sentences. For example, it would be misleading to combine word vectors to represent "Boston Globe". Embeddings for "good" and "bad" might be similar, causing problems for sentiment analysis.

Word embeddings don't capture some linguistic traits. For example, vectors for 'house' and 'home' may be similar but vectors of 'like' and 'love' are not. In general, when a word has multiple meanings, called homographs or polysemy, its vector is an average value. One solution is to consider both the word and its part of speech. Inflections also cause problem. For example, 'find' and 'locate' are close to each other but not 'found' and 'located'. Lemmatization can help before training the word vectors.

### What software tools are available for word embeddings?

Both word2vec and GloVe implementations are available online. In some frameworks such Spark MLlib or DL4J, word2vec is readily available.

Some frameworks that support word embeddings are S-Space and SemanticVectors (Java); Gensim, PyDSM and DISSECT (Python).

Deeplearning4j provides the `SequenceVectors` class, an abstraction above word vectors. This allows us to extract features from any data that can be described as a sequence, be it transactions, proteins or social media profiles.

A tutorial explaining word embeddings in TensorFlow is available.

You can also download pretrained word embeddings. Note that many use lemmatization while learning word embeddings.

## Milestones

1950

In contrast to the formal linguistics of Noam Chomsky, researchers in the 1950s explore the idea that context can be useful for linguistic representation. This is based on structuralist linguistics. **Distributional Hypothesis** by Zellig Harris states that word meanings are associated with context. Another linguist John Firth states,

> You shall know a word by the company it keeps!

1960

Early attempts are made in the 1960s to construct features to represent semantic similarities. Hand-crafted features are used. Charles Osgood's semantic differentials is an example.

1990

Deerwester et al. note that words and documents in which they occur have a semantic structure. They exploit this structure for information retrieval to match documents based on concepts rather than keywords. They map words to documents as a matrix with word counts, giving a sparse representation. They attempt at the most 100 dimensions and employ the technique of Singular Value Decomposition (SVD). They coin the term **Latent Semantic Indexing (LSI)**.

2003

Bengio et al. propose a language model based on neural networks, though they're not the first ones to do so. They use a feed-forward NN with one hidden layer. Words are represented as feature vectors. Model learns vectors and joint probability function of word sequences. However, they don't use the term "word embeddings". Instead, they use the term **distributed representation** of words. Note that here we're interested in similar words whereas LSI is about similar documents due to its application to information retrieval.

2008

Collobert and Weston show the usefulness of pretrained word embeddings. Using such word embeddings, they show that a number of downstream NLP tasks can be learned by a neural network. They consider both syntactic tasks (POS tagging, chunking, parsing) and semantic tasks (named entity recognition, semantic role labelling, word sense disambiguation). In their approach, a word is decomposed into features and then converted to vectors using lookup tables.

2013

At Google, Mikolov et al. develop **word2vec** that helps in learning standalone word embeddings from a text corpus. Efficiency comes from removing the hidden layer and approximating the objective. Word2vec enabled large-scale training. Embeddings from the skip-gram model is shown to give state-of-the-art results for sentence completion, analogy and sentiment analysis.

2014

Stanford researchers release **GloVe** word embedding. This has vectors of 25-300 dimensions learned from up to 840 billion tokens.

2018

Conneau et al. apply word embeddings to language translation by aligning monolingual word embedding spaces in an unsupervised way. They don't require parallel corpora or character-level information. This can therefore benefit low-resource languages. They achieve better results as compared to supervised methods. Earlier in 2016, Sebastian Ruder published a useful survey of many cross-lingual word embeddings.

Word Embedding


# Open Source Hardware

## Summary

Many developments have contributed to the interest and growth of Open Source Hardware (OSHW): free and open source software (FOSS), 3D printing, crowdfunding, the maker movement, and Moore's Law reaching its limits.

While OSHW is commonly thought to include electronics and mechanical designs, OSHW today has a much broader reach including fashion, furniture, musical instruments, farm machinery, bio-engineering, and more.

OSHW is not a standard, nor is there a single organization tasked with leading the OSHW movement. However, the Open Source Hardware Association (OSHWA) hopes to become the hub of this movement. There's also the Open Source Hardware Definition, which forms the basis for defining licenses for OSHW.

## Discussion

### What's the historical context for open source hardware?

In the ham radio community sharing knowledge was a common practice. Then in the 1970s, computers were shipped as kits with schematics included. This resulted in a hacking culture among computer enthusiasts. This culture was about tinkering, experimenting, sharing, and collaborating.

FOSS that started in the 1980s contributed to OSHW, although open hardware need not be about either electronics or software. The web of the 1990s made it easier to share designs and best practices. By early 2010s, OSHW became more widely known due to the following reasons:

    + **3D Printing**: This brought down prototyping and production costs. It made design iterations easier and faster.
    + **Maker Movement**: Started in the mid-2000s, this established magazines, platforms, and fairs/exhibitions for people to come together, collaborate and co-create. Maker labs brought essential tools under roof, gave people affordable access to these tools, and thereby democratized production.
    + **Crowdfunding**: Those with good ideas could get upfront funding from potential users of the product without depending on traditional investment routes or financial institutions.
    + **Moore's Law**: With the Law reaching its limits, there's a need to create application-specific silicon and open designs can keep costs down.

### What aspects of hardware can be open sourced?

Open source hardware is about sharing the design files of hardware. This may include architecture/design drawings, schematics, PCB layout, bill of materials, HDL code, production/assembly instructions, and anything else that can enable others to replicate the hardware.

There's no use claiming open design if the file format is proprietary and can be opened only with closed tools. Thus, the definition of open source can be expanded to include design file formats and/or access to tools or software to manipulate the files. Along with original proprietary files, intermediate files in open formats should be made open. Examples of file formats include PDFs of circuit schematics, Gerbers for circuit board layouts, and IGES or STL files for mechanical objects. OpenCAD, KiCAD and Eagle are examples of open tools.

In the spirit of open source, users should be able to study the design, modify it for their specific needs, or distribute it. They also have the freedom to make and sell hardware based on the design.

### Won't people misuse my designs and get rich at my expense?

Maybe but not necessarily. Firstly, just because you adopted OSHW licensing, doesn't mean you cannot also sell your products commercially and be successful in the process. Even if others have access to your design, there's cost and effort involved in sourcing, manufacturing, assembling, testing, distributing and providing technical support. If you establish your own brand value, provide a quality product at a good price, it's difficult for others to compete.

The advantage of being open is that you benefit from community support and contributions. You establish an ecosystem around your product. While commercial vendors offer free reference designs, OSHW is a community effort. An open source design is likely to be more robust. It enables faster prototyping and it's continuously improved via multiple contributors.

In conclusion, the case for keeping your hardware design proprietary is weaker than for software. Investment towards manufacturing and distribution is still a barrier to entry. Hardware can also be differentiated based on the firmware that need not necessarily be open.

### What business models are possible with OSHW?

Your design may be open and freely shared but you may charge for products made from open designs. Technical support, maintenance and upgrades can be part of your services model and customers will be willing to pay for them.

You could create innovative services around open designs. You could then set up an online marketplace, aggregate and analyze data, etc. Openness becomes an incentive for customers because they have the freedom to tinker and customize. You can provide them training, resources, and additional services.

Your product and its open design should be a market enabler. You can build an entire platform based on your open design. You can partner with others for peripherals that may or may not be open. In any case, a healthy ecosystem based on your open design will likely lead to better adoption and sales of your core products. This has been the case with Arduino and add-on boards called *shields*; and Raspberry Pi and add-on boards called *hats*.

### What's the recommended licensing for OSHW?

In the days before any OSHW licenses were defined, people simply used FOSS licenses for CAD drawings or firmware. To call something OSHW, it should be completely open without restrictions. Any license that prevents commercial use is not compatible with OSHW.

"Creative" works are protected by copyright and "useful" or functional works are protected by patents. Thus, copyrights don't apply to hardware. If hardware is not patented, anyone can copy, modify or build upon the hardware. But copying or modifying hardware is lot easier if the design files are available. Thus, when we talk about open licensing or copyleft for hardware, we are referring to the design files and related documentation.

In the world of software, there are plenty of licenses with different degrees of openness. Among the copyleft (share-alike or viral) licenses are GPL, CC BY-SA, CERN Open Hardware License (OHL) and TAPR Open Hardware License (OHL). Permissive licenses that allow for closed derivatives include FreeBSD, MIT, CC Attribution, and Solderpad Hardware License.

### What are some examples of OSHW projects?

Well-known examples that use CC BY-SA include Arduino, mBed HDK, BeagleBoard, Particule (formerly Spark), and Tessel. mangOH is an example that uses Creative Commons Attribution license.

Back in 2013, some successful OSHW projects included Arduino, Raspberry Pi, OpenROV (remote-operated underwater robot), DIY Drones, LittleBits, and Makerbot Replicator 2, Lasersaur, Robo3D, and Console II.

Noteworthy projects of 2016 included the Global Village Construction Set (fabricate industrial machines), Open Source Beehives (bee home and sensor kits for tracking), AKER garden kits, WikiHouse (building system), FarmBot (CNC farming machine), OpenDesk (make furniture), OSVehicle, RepRap (3D printer), OpenKnit (digital knitting), Defense Distributed (3D firearms), APM:Copter, and Open Hand Project (robotic prosthetic hands).

Some OSHW boards include Arduino Due, Freescale Freedom, Microchip ChipKIT Uno32, and Beaglebone Black. Mouser website also lists dozens of other boards. Olimex offers OSHW boards including Linux-based OLinuXino boards.

At chip level, RISC-V offers an open architecture from which customized SoCs can be designed. Other include lm32, mor1kx, and blocks from the OpenCores project. There's talk of even building an open source supercomputer.

### What are some online resources for OSHW?

Design files, particularly for 3D printing, can be downloaded from Thingiverse, MyMiniFactory, Pinshape, and Cults. Thingiverse was launched in 2008 and it encourages folks to modify and re-upload designs to the site. Other sources are Hackster.io, Hackaday, Open Electronics and the Open Circuits Institute.

For crowdfunding, try CrowdSupply, Kickstarter, Goteo and Tindie. Adafruit and SparkFun, while selling proprietary products, also promote OSHW. Olimex, Pandaboard.org, and SolderCore are suppliers of OSHW that are also available from Mouser.

If you're open sourcing your own product, you may release the design files on your own website, on Thingiverse or similar sites. If you wish to make it open during the design and development stages, GitHub or its alternatives can be a place to share. Prusa Mendel and Mendel90 are just two examples of projects that have received lots of community contributions, what in the tech speak are called "pull requests".

Element14 has an online community forum for discussions on OSHW.

## Milestones

Mar
1975

At a garage in Menlo Park, California, some computer hobbyists have the first meeting of their newly formed **Homebrew Computer Club**. At such a meeting, Steve Wozniak gets inspired to build his own computer and share the blueprints with others. This leads to Apple-1. However, Steve Jobs convinces Wozniak to sell Apple-1 rather than share them freely, thus marking the birth of Apple in 1977.

1997

Bruce Peters launches the **Open Hardware Certification Program** to allow vendors to self-certify their products as open. This implies availability of documentation so that anyone could write their own device drivers.

1998

Mozilla releases source code of Netscape browser suite. Not wanting to call it free software (in the spirit of the Richard Stallman's Free Software Foundation), the term **Open Source** is coined. The **Open Source Initiative (OSI)** is also formed by Eric Raymond and Bruce Peters. At this point, it's all about software, not hardware.

2003

Hernando Barragán creates **Wiring** so that designers and artists can approach electronics and programming more easily. This work leads to the creation of **Arduino** in 2005.

2005

The birth of the modern maker movement starts with the launch of **Make: magazine**. In August, **Instructables** launches as an online platform to share step-by-step instructions to make something.

2006

Dale Dougherty organizes the first **Maker Faire** for makers to showcase their creations. In October, as an open access DIY workshop, **TechShop** opens in Menlo Park, California.

2010

At a open hardware workshop in March, some folks define the Open Source Hardware Definition 0.1. In July, v0.3 is made public. These are based on the definition of open source (from 1998). As on June 2019, Open Source Hardware (OSHW) Definition 1.0 is current. Open-Source Hardware Definition is not itself a license but OSHW licenses are written so as to be compatible with the definition. In September, the first **Open Hardware Summit** is organized, in New York City. Since then it has become an annual event.

2011

The original gear logo of OSHW is selected via a community contest. A modified version of the winning logo is announced at the Open Hardware Summit. In July, the **CERN Open Hardware License (OHL)** is announced. To facilitate collaboration and sharing, CERN had already set up the Open Hardware Repository in January 2009.

Feb
2012

The first **Raspberry Pi Model B** is released as a credit-card-sized single-board computer (SBC) retailing at only $35. The idea is to make computers affordable, accessible and fun to a new generation of programmers.  Within two years, 2.5 million units are sold. By 2018, 22 million are sold worldwide. Also in 2012, the **Open Source Hardware Association (OSHWA)** is formed. Certification of OSHW is also done by the Association.

Jul
2018

The U.S. DARPA announces funding of $1.5 billion over five years for what it calls the Electronic Resurgence Initiative (ERI). This includes the **POSH (Posh Open Source Hardware)** project meant to create a Linux-based platform for the design and verification of open source hardware IP blocks for SoCs.

Mar
2019

Esperanto, Google, SiFive, and Western Digital get together to form the **CHIPS Alliance**. The purpose is to foster open-source chip designs. The alliance is committed to RISC-V architecture but wishes to encourage more such open designs.

Open Source Hardware


# Slim Framework

## Summary

Slim is a minimalistic PHP framework for implementing APIs. At its core, we can see it simply as a "routing library". It receives HTTP requests, routes them to the correct callback functions, and forwards the HTTP responses. By being minimal, Slim is fast and efficient. Unlike fullstack frameworks, developers don't end up installing lots of features that they may never use.

Slim uses the concept of **middleware**. Each middleware can implement something specific. By chaining together many such middleware, complex web applications can be built. Many third-party middleware are available that implement authentication, security, view templating for generating HTML responses, caching, and more.

Slim is open source and maintained by a community of developers. It can be used on its own (server-side rendering) or used alongside frontend frameworks such as React or Vue (client-side rendering).

## Discussion

### Why should I prefer Slim over so many other PHP frameworks?

Slim is a good choice if you're looking for "performance, efficiency, full control and flexibility". Slim supports standard interfaces such as PSR-7 and PSR-15. This prevents vendor lock-in. Slim framework plays well with good software practices including SOLID principles, design patterns, security principles and dependency injection.

Slim's easier to learn than fullstack frameworks such Laravel or Symfony. Fullstack frameworks provide many things out of the box: authentication, security, database integration, template engines, session management, and more. With Slim, these are outside the core framework. They're available as middleware from third-party vendors. Developers can install them via Composer only if and when their app needs them.

Slim is suited for building modern RESTful APIs and services where data is exchanged in JSON or XML formats.   Complex apps can also be built since the core framework can be extended with middleware. Complete web pages can be rendered on the server side with templating engines such as Twig-View or PHP-View. This brings Slim into the domain of MVC frameworks.  Developers have used Slim backend with Next.js frontend. SPAs with Slim and Vue.js are possible.

### How's the performance of Slim?

In a performance study from 2016, Slim v2.6 ranked fifth in terms of throughput (requests per second), peak memory usage and execution time. Phalcon and Ice were the best. In another study from 2017, Slim 3.0 ranked third behind Phalcon and CodeIgniter. It performed better than microframeworks Silex and Lumen.

Slim's routing is based on FastRoute, which optimizes the parsing of regular expressions via chunking.

### What are the key features of Slim?

The key features of Slim include these:

    + **Routing**: HTTP requests are efficiently routed to mapped callbacks. Routes are matched using regular expressions that can also capture and pass parameters into the callbacks. A callback is usually a function or a method of a class.
    + **Middleware**: Built-in or third-party middleware allow specific handling of requests and responses. Developers can also write custom middleware. The middleware approach is modular and flexible: middleware can be reordered or disabled if needed.
    + **PSR-7 Support**: HTTP messages interfaces conform to PSR-7 specifications. Slim provides an implementation of this interface but developers are free to adopt any alternative implementation.
    + **Dependency Injection**: This makes the app more testable or configurable. The interface is PSR-11 compliant.

### Could you share details of how Slim middleware work?

In Slim, middleware can be visualized as concentric circles. Middleware are called from the outermost to the innermost circles. At the center, the application executes and routing takes place. Subsequently, middleware are again processed but in innermost-to-outermost circles.  Another useful visualization is the LIFO stack.

Suppose we have two middleware: authentication and caching. Authentication middleware is executed first. If authentication fails, app skips further processing. If authentication succeeds, then caching middleware is executed. If response exists in the cache, app uses the cache and skips the main callback processing. If cache doesn't exist, the main callback is executed that generates the response. Then caching middleware is called to store the response in the cache if necessary. Finally authentication middleware is called to complete or modify the response if needed.

Slim middleware deal with PSR-7 request/response objects and PSR-15 request handler object. Middleware can be added to the app object and thus potentially called with every request. Or it can be added to only specific routes, in which case such middleware are called after routing is done. In both cases, outermost middleware is added last.

### Which are the packaged middleware in Slim?

Default Slim installation comes packaged with a few middleware:

    + **Routing**: Implements routing based on *FastRoute* but developers can choose to use a different implementation.
    + **Error Handling**: Useful during development but also in production where errors can be logged for later analysis.
    + **Method Overriding**: Process `X-Http-Method-Override` HTTP header. For example, a `POST` request with `X-HTTP-Method-Override:PUT` header can be treated as a `PUT` request.
    + **Output Buffering**: Append to the current response body by default but this can be changed to "prepend" mode.
    + **Body Parsing**: When implementing APIs, response is often in JSON or XML. This middleware handles these formats.
    + **Content Length**: Appends `Content-Length` header to the response.

### What's the application lifecycle in Slim?

A Slim app is first instantiated with the call to `Slim\App()`. It's common to do this via the factory method `AppFactory::create()`. For each of the HTTP methods (such as GET, POST, DELETE, etc.) routes can be defined. Then middleware are added. Finally, the application is executed by calling the app object's `run()` method.

App middleware execute in outermost-to-innermost order. When it reaches the innermost layer, application despatches the HTTP request to its suitable route object. If this route has its own middleware, they're executed in the same outermost-to-innermost order. Then control passes from the innermost to outermost middleware. Finally, application serializes and returns the HTTP response.

### How can I create routes in Slim?

Slim framework allows us to define routes for various HTTP methods: GET, POST, PUT, DELETE, OPTIONS, PATCH. It's possible to define a route for all methods using `$app->any()` method or for multiple methods such as in `$app->map(['GET', 'POST'])`.

Each route method accepts a **pattern** that's used to match against the URI of the HTTP request. **Placeholders** within the pattern are delimited using `{}`. Moreover, placeholders can be regular expressions. Optional parts of the pattern are delimited using `[]`. For instance, `$app->get('/users[/{id:[0-9]+})` wold match "/users" (since `id` part is optional) and "/users/23" (since `id` is a valid positive integer).

Each route method also accepts a **callback**, which is a PHP callable with three arguments: Request, Response and Arguments. The values captured by placeholders are passed as arguments. In the above example, `id` is accessed as `$args['id']`. Callback can be specified as an anonymous function, a method defined in a class, a class that implements the `__invoke()` method, etc.

Routes can be grouped to make code more maintainable. Thus, URIs `/users/{id:[0-9]+}` and `/users/{id:[0-9]+}/reset-password` can be grouped into `/users/{id:[0-9]+}`.

### What are some online resources to get started with Slim?

Beginners can start with the official Slim documentation. This introduces Slim with a simple example and then goes into the details. For latest updates, visit the Slim blog. To get help from the community, head to the discussion forum.

For step-by-step guides to developing APIs or apps with Slim, see the Slim Walkthrough and Rob Allen's Building Modern APIs with Slim Framework.

For ideas on how to organize a complex Slim-based app, see an example filesystem layout (slide 18) . Read about architecting a layered structure. A Slim v4 starter project is also available.

Curated list of PSR-15 middleware are available at psr15-middlewares and awesome-psr15-middlewares.

Those who wish to contribute the framework itself, can find Slim's codebase on GitHub.

## Milestones

Sep
2010

Josh Lockhart invents the Slim framework with a focus on simplicity and rapid application development. He feels that alternatives such as Symfony, Cake and CodeIgniter have steeper learning curves and too big for his needs.

Sep
2011

The Slim blog illustrates a simple "Hello World" example with the Slim framework. App is initialized with the call `$app = new Slim()`. Routes are configured using `$app->get()`, `$app->post()`, and so on. Application execution is trigger with the call `$app->run()`. All of this can be placed in the file *index.php*. Via Apache's *.htaccess* file, requests can be directed to *index.php*.

Apr
2012

**Slim v1.6.0** is released. Framework architecture is based on the Rack Protocol. Application-wide **middleware** is introduced. Request/response interfaces, session handling, cookie handling and logging are improved. It's also announced that Slim can now be installed via Composer.

Sep
2012

**Slim v2** is released. It requires at least PHP 5.3. It's PSR-2 compliant and uses **namespaces**.

Jul
2013

**Slim v2.3.0** is released in a backward compatible manner. Environment, request, response, view and log objects can be accessed as public properties of the application instance. Route groups are added. Headers and cookies can be easily accessed in Request objects and easily set in Response objects.

Dec
2015

**Slim v3.0.0** is released. New features include support for dependency injection and PSR-7. Methods expect interfaces rather than concrete implementations. By November 2019, this evolves to Slim v3.12.3.

Aug
2019

**Slim v4.0.0** is released. This release requires PHP 7.1 or newer. It supports PSR-15 compliant middleware. Framework is decoupled from built-in PSR-7 implementation. Thus developers can use alternatives such as Zend, Nyholm and Guzzle. Likewise, routing, error handling and response emitting implementations are decoupled from the framework core. Routing is via `Slim\Middleware\RoutingMiddleware`. App factory is introduced. By March 2022, this evolves to Slim v4.10.0.

Slim Framework


# Information Security Principles

## Summary

Information can be private or public, personal or generic, valuable or commonplace, online or offline. Like any other asset, it has to be protected. This is more important online where hackers can steal or misuse information remotely even without any physical access to where that information resides.

In line with evolving technology, data security practices have evolved from high-level principles into more detailed set of practices and checklists.  In practice, there's no single list of principles that everyone agrees on. Many lists exist, each one customized for its context.

## Discussion

### Which are the three main information security principles?

The three main security principles include:

    + **Confidentiality**: Protect against unauthorized access to information.
    + **Integrity**: Protect against unauthorized modification of information. Even if an adversary can't read your data, they can either corrupt it or selectively modify it to cause further damage later on.
    + **Availability**: Protect against denial of access to information. Even if an adversary can't access or modify your data, they can prevent you from accessing it or using it. For example, they can destroy or congest communication lines, or bring down the data server.These principles have also been called security goals, objectives, properties or pillars. More commonly, they are known as the **CIA Triad**.

Security practitioners consider these principles important but vague. This is because they're about the "what" but not the "how". They have to be translated into clear practices based on context. They have been applied to IT infrastructure, cloud systems, IoT systems, web/mobile apps, databases, and so on. Actual practices may differ but can be related to the CIA triad.

### What are some variations of CIA?

It's been said that the CIA Triad is focused on technology and ignores the human element. The **Parkerian Hexad** therefore addresses the human element with three more principles:

    + **Possession/Control**: It's possible to possess or control information without breaching confidentiality.
    + **Authenticity**: This is about proof of identity. We should have an assurance that the information is from a trusted source.
    + **Utility**: Information may be available but is it in a usable state or form?Another variation is the **McCumber Cube**. It includes the CIA Triad but also adds three states of information (transmission, storage, processing) and three security measures (training, policy, technology).

Other published security principles have come from OECD, NIST, ISO, COBIT, Mozilla, and OWASP.

### What are some means of achieving the CIA security goals?

Authorization, authentication and the use of cryptography are some techniques to achieve the CIA security goals. These have been sometimes called **Security Mechanisms**. These mechanisms are designed to protect assets and mitigate risks. However, they may have vulnerabilities that threats will attempt to exploit.

Confidentiality is often achieved via encryption. Hackers in possession of encrypted data can't read it without the requisite decryption keys. File permissions and access control lists also ensure confidentiality. For integrity, a hash of the original data can be used but this hash must itself be provided securely. Alternatively, digital certificates that use public-key cryptography can be used. For availability, there should be redundancy built into the system. Backups should be in place to restore services quickly. Systems should have recent security updates. Provide sufficient bandwidth to avoid bottlenecks.

People must be trained to use strong passwords, recognize possible threats and get familiar with social engineering methods.

### What are some common approaches to enhancing information security?

Complex systems are hard to secure. Keep the design simple. This also **minimizes the attack surface**. For example, a search box is vulnerable to SQL injections but a better search UI will remove this risk. Use **secure defaults** such as preventing trivial passwords. Give users or programs the **least privilege** to perform their function. When failures occur, ensure they're handled with correct privileges.

There's better **defence in depth**. This means that multiple levels of control are better than a single one. Security at application layer alone is not enough. Secure server access, network communications, wireless access, user interface, and so on. Don't trust third-party services. Have a clear **separation of duties** to prevent fraud. For example, admin users shouldn't be allowed to login to the frontend with same privileges and make purchases on behalf of others.

**Avoid security by obscurity**. This means that we shouldn't rely on hidden secrets. For example, even if source code is leaked or encryption algorithms are known, the system should remain secure.

Prefer **decentralized systems** with replication to centralized ones.

### Could you mention some threats or attacks by which hackers can compromise the security principles?

Sniffing data communications, particularly when it's not encrypted, is an example of breach of confidentiality. ARP spoofing is an example of sending false ARP messages so that traffic is directed to the wrong computer. Phishing is a breach of integrity since the hacker's website tricks a visitor into thinking it's the genuine website.

Repeatedly sending a request to a service will overload the server. Server will become progressively slower to response to requests and even crash. This Denial-of-Service (DoS) attack make the service unavailable.

For databases, SQL injection is a big threat allowing hackers access to sensitive data or extra privileges. Buffer overflow vulnerabilities can be exploited to modify data. DoS attacks are possible with databases and their servers.

In any case, record all transactions and events. This leads to better detection of intrusions and future preventions. Have a good recovery plan. Perform frequent security tests to discover vulnerabilities.

## Milestones

1950

Information Security or InfoSec doesn't exist in the 1950s or even in the 1960s. Security is all about physically securing access to expensive machines. Reliability of computers is the main concern. As hardware and software becomes standardized and cheaper, it's only in the 1970s that there's a shift from computer security towards information security.

1970

In the early years of the ARPANET, the US Department of Defense commissions a study that's published by the Rand Corporation as *Security Controls for Computer Systems*. It identifies many potential threats and possible security measures. The task force was chaired by Willis H. Ware. In time, this report becomes influential and is known as the Ware Report.

1972

James P. Anderson authors *Computer Security Technology Planning Study* for the USAF. This is published in two volumes. In time, this comes to be called the Anderson Report.

1973

Multics was a timesharing operating system that started in 1965 as a MIT research project. In the summer of 1973, researchers at MIT look at the security aspects of Multics running on a Honeywell 6180 computer system. They come up with broad security design principles. They categorize these into three categories with due credit to J. Anderson: unauthorized release, unauthorized modification, unauthorized denial.

1980

Prior to the 1980s, security was influenced by the defence sector. In the 1980s focus shifts from **Confidentiality** to commercial concerns such as costs and business risks. Among these is the idea of **Integrity** since it's important for banks and businesses that data is not modified by unauthorized entities.

1988

**Morris Worm** becomes the first DoS attack on the Internet. Thus, **Availability** is recognized as an essential aspect of information security.

1989

In the *JSC-NASA Information Security Plan* document we find the use of the term **CIA Triad**. However, the term could have been coined as early as 1986.

1998

To complement InfoSec, **Information Assurance (IA)** emerges as a discipline. This is more about securing information systems rather than information alone. With the growth of networks and Internet, **Non-Repudiation** and **Authentication** become important concerns. Non-repudiation means that parties can't deny having sent or received a piece of information.

2001

NIST publishes Underlying Technical Models for Information Technology Security. It identifies five security objectives: Availability, Integrity, Confidentiality, Accountability and Assurance. It points out that these are interdependent. For example, if confidentiality is compromised (eg. superuser password), then integrity is likely to be lost as well.

2002

Donn B. Parker expands on the CIA Triad by adding three more items: authenticity, possession or control, and utility. Parker also states that it's best to understand these six principles in pairs: confidentiality and possession, integrity and authenticity, and availability and utility. In time, these six principles have come to be called **Parkerian Hexad**.

Information Security Principles


# Apache Spark

## Summary

When processing large amounts of data, it's common to distribute and parallelize the workload across a cluster of machines. Apache Spark is a framework that sits between the applications above and the cluster of resources below. Spark doesn't manage the low-level storage and compute resources directly. Instead, it makes use of other frameworks such as Mesos or Hadoop. In fact, Apache Spark is described as "a unified analytics engine for large-scale data processing".

Applications written in many popular languages can make use of Spark. Meanwhile, support for more languages is coming. Since Spark comes with many useful libraries, different types of processing are possible in a single application. Spark is being popularly used for Machine Learning workloads.

Spark is open source and is managed by Apache Software Foundation.

## Discussion

### In which application scenarios is Spark useful?

Hadoop MapReduce is great for batch processing where typically data is read from disk, processed and written back to disk. But MapReduce is inefficient for multi-pass applications that read more than once. Performance drops due to data duplication, serialization and disk I/O. Apache Spark was created to solve this and is useful for the following:

    + **Iterative Algorithms**: This includes machine learning algorithms that by nature process data through many iterations.
    + **Interactive Data Mining**: Data is loaded into RAM once and then repeatedly queried. Interactive or ad-hoc analysis often includes visualizations. The language for querying the data must also be expressive.
    + **Streaming Applications**: For real-time analysis, data must be analyzed as they come into the system. There's a need to maintain aggregate state over time.Spark can be used in gaming, e-commerce, finance, advertising, and more. Many of these involve real-time analysis and unstructured data sources.  Uber used Spark for feature aggregation in its ML data pipeline. Spark can help in scaling data pipelines for genomics. One researcher analyzed NASA server logs (~300MB) using database-like queries and regular expressions via Spark.

### What makes Spark better than Hadoop MapReduce?

Like MapReduce, Spark is scalable, distributed and fault tolerant, but it's also more efficient and easy to use. While MapReduce reads and writes to disk between tasks, Spark does in-memory caching and thereby improves performance.

Spark does this by providing a data abstraction called **Resilient Distributed Dataset (RDD)**. Interfacing to RDD is enhanced with *DataFrame* and *Dataset* APIs. This is just one example of Spark's better usability via rich APIs and a functional programming model.

MapReduce jobs are executed as JVM processes but Spark is able to execute multiple tasks inside a JVM process. Another advantage of Spark is that each JVM process lives for entire duration of the application, unlike in MapReduce where the process exits once execution completes. This means that new tasks submitted to a Spark cluster can start much faster. There's better CPU utilization. The tradeoff is that resource management is coarse grained but cluster managers can overcome this (such as YARN's container resizing).

However, MapReduce may still be relevant for linear processing of huge datasets that can't fit into memory, particularly for join operations.

### What's the architecture of Apache Spark?

A Spark application runs in a distributed manner on a cluster of nodes. It consists of two parts: the main program called **driver program**, and **executors** or processes on worker nodes where actual execution happen. Driver program contains the *SparkContext* object for coordinating the application.

A **worker node** is any node that runs application code. Such code could be JAR or Python files, for example. Application code available in SparkContext is passed to executors. SparkContext also schedules and sends tasks to executors. Each application get its own executors. This cleanly isolates one application from another. Multiple tasks can run within an executor due to multi-threading.

Since the driver program schedules tasks, it should be close to the worker nodes, preferably on the same local area network. It should also be network addressable from worker nodes and listen for incoming connections.

Driver program connects to a **cluster manager** that's responsible for managing resource allocation. The cluster manager could be anything: Spark's default manager, Apache Mesos, Hadoop YARN, Kubernetes, etc. The manager needs to only acquire executor processes and these communicate with one another.

### What are some essential terms to know in Apache Spark?

Here are some essential terms:

    + **Task**: A unit of work sent to one executor.
    + **Job**: Parallel computation involving multiple spawned tasks for some actions such as `save` or `collect`.
    + **Stage**: A smaller set of tasks of a job, useful when one stage depends on another.
    + **RDD**: A fault-tolerant collection of elements that can be processed in parallel.
    + **Partition**: A smaller chunk into which an RDD is divided. A task is launched per partition. Thus, more partitions imply greater parallelism. Having 2-4 partitions per CPU is typical.
    + **Transformation**: An operation performed on an RDD. Since RDDs are immutable, transformations on an RDD result in a new RDD.
    + **Action**: An operation on an RDD that returns a result to the driver program.

### What's the Spark software stack?

The Spark Core is the main programming abstraction. It gives APIs (in Java, Scala, Python, SQL, and R) to access and manipulate RDDs. To ease development, Spark comes with component libraries including Spark SQL/DataFrame/Dataset, Spark Streaming, MLlib and GraphX. Each of these serves specific application requirements but they all rely on Spark Core's unified API. This approach of a modular core plus useful components makes Spark attractive to developers.

User applications sit on top of these components. The Spark Shell is an example app that facilitates interactive analysis.

Spark Core itself doesn't manage cluster resources. This is done by a cluster manager. Spark comes with a standalone cluster manager but we are free to choose alternatives such as Hadoop YARN, Mesos, Kubernetes, etc. Spark Core also doesn't deal with disk storage for which we can use Hadoop HDFS, Cassandra, HBase, S3, etc. For example, in one deployment we could choose to use the YARN along with HDFS while the computing is managed via Spark Core.

## Milestones

2009

Researchers at UC Berkeley's AMPLabs create a cluster management framework called **Mesos**. To showcase how easy it is build something on top of Mesos, they look at the limitations of MapReduce and Hadoop. These are good at batch processing but not at iterative or interactive processing, such as machine learning or interactive querying. To overcome these limitations, they build **Spark**. Spark is written in Scala and exposes a functional programming interface.

2010

Spark is open sourced. It's creators publish the first paper on Spark, titled Spark: Cluster Computing with Working Sets.

2013

Spark is incubated under the **Apache Software Foundation**. In February 2014, it becomes a top-level Apache project.

2015

With Spark 1.4 release, there's support for both Python 2 and 3. However, it's announced later to deprecate Python 2 support in the next major release of 2019.

Jul
2016

Spark 2.0.0 is released. To enable optimization, DataFrame API was introduced in v1.3. Dataset API introduced in v1.6 enabled compile-time checks. From v2.0, **Dataset** presents a single abstraction although language bindings that don't have type checks (Python, R) will internally use DataFrame, which is an alias for Dataset[Row].

2018

In February, Spark 2.3.0 is released. With this release, native Spark jobs can be managed by Kubernetes. Data source API V2 improves over V1. Meanwhile, a ranking of distributed computing packages for data science shows Apache Spark at the top, followed by Apache Hadoop. In the enterprise, **Apache Spark MLib** is most adopted for ML and Big Data analytics. This is followed by TensorFlow.

Apr
2019

For using Spark from C# and F#, **.NET for Apache Spark** is launched as a preview project. Direction will come from .NET Foundation. This effort replaces and deprecates Mobius.

Apache Spark


# Profiling Python Code

## Summary

Python documentation defines a *profile* as "a set of statistics that describes how often and for how long various parts of the program executed." In addition to measuring time, profiling can also tell us about memory usage.

The idea of profiling code is to identify bottlenecks in performance. It may be tempting to guess where the bottlenecks could be but profiling is more objective and quantitative. Profiling is a necessary step before attempting to optimize any program. Profiling can lead to restructuring code, implementing better algorithms, releasing unused memory, caching computational results, improving data handling, etc.

## Discussion

### What are the Python packages or modules that help with profiling?

Python's standard distribution includes profiling modules by default: `cProfile` and `profile`. `profile` is a pure Python module that adds significant overhead. Hence `cProfile` is preferred, which is implemented in C. Results that come out from this module can be formatted into reports using the `pstats` module.

`cProfile` adds a reasonable overhead. Nonetheless, it must be used only for profiling and not for benchmarking. **Benchmarking** compares performance between different implementations on possibly different platforms. Because of the overhead introduced by `cProfile`, to benchmark C code and Python code, use `%time` magic command in IPython instead. Alternatively, we can use the `timeit` module.

To trace memory leaks, `tracemalloc` can be used.

Other useful modules to install include `line_profiler`, `memory_profiler` and `psutil`. The module `pprofile` takes longer to profile but gives more detailed information than `line_profiler`. For memory profiling, `heapy` and `meliea` may also help. `PyCounters` can be useful in production.

### Are there visualization tools to simplify the analysis of profiler output?

`vprof` is a profiler that displays the output as a webpage. It launches a Node.js server, implying that Node.js must be installed. On the webpage, you can hover over elements to get more information interactively.

The output of cProfile can be saved to a file and then converted using `pyprof2calltree`. This converted file can then be opened with KCachegrind (for Linux) and QCachegrind (for MAC and Windows). Likewise, gprof2dot and RunSnakeRun are alternative graphical viewers.

*PyCallGraph* profiles and outputs the statistics in a format that can be opened by Graphviz, a graph visualization software.

*nylas-perftools* adds instrumentation around code, profile it and export the results in JSON format. This can be imported into Chrome Developer Tools to visualize the timeline of execution. This can also be used in production since the app stack is only sampled periodically.

### What are the types of profilers?

Three types of profilers exist:

    + **Event-based**: These collect data when an event occurs. Such events may be entry/exit of functions, load/unload of classes, allocation/release of memory, exceptions, etc. `cProfile` is an example.
    + **Instrumented**: The application profiles itself by modifying the code or with built-in support from compiler. Using decorators to profile code is an example of this. However, instrumented code is not required in Python because the interpreter provides hooks to do profiling.
    + **Statistical**: Data is collected periodically and therefore what the profiler sees is a sample. While less accurate, this has low overhead. Examples of this include Intel® VTune™ Amplifier, Nylas Perftools and Pyflame. On the contrary, we can also say that `cProfile`, `line_profiler` and `memory_profiler` are *Deterministic Profilers*.In general, deterministic profiling adds significant overhead whereas statistical profiling has low overhead since profiling is done by sampling. For this reason, the latter can also be used in production. We should note that the profiling overhead in Python is mostly due to the interpreter: overhead due to deterministic profiling is not that expensive.

We can also classify profilers as **function profilers**, **line profilers** or **memory profilers**.

### What are the IPython magic commands that help with profiling?

IPython has the following magic commands for profiling:

    + `%time`: Shows how long a one or more lines of code take to run.
    + `%timeit`: Like `%time` but gives an average from multiple runs. Option `-n` can be used to specify the number of runs. Depending on how long the program takes, the number of runs is limited automatically. This is unlike the `timeit` module.
    + `%prun`: Shows time taken by each function.
    + `%lprun`: Shows time taken line by line. Functions to profile can be specified with `-f` option.
    + `%mprun`: Shows how much memory is used.
    + `%memit`: Like `%mprun` but gives an average from multiple runs, which can be specified with `-r` option.Commands `%time` and `%timeit` are available by default. Commands `%lprun`, `%mprun` and `%memit` are available via modules `line-profiler`, `memory-profiler` and `psutil`. But to use them within IPython as magic commands, mapping must be done via IPython extension files. Or use the `%load_ext` magic command.

When timing multiple lines of code, use `%%timeit` instead of `%timeit`. This is available for `%prun` as well.

### How to interpret the output of cProfile?

The output of cProfile can be ordered using the `-s` option. In the accompanying image, we see that the results are ordered by cumulative time in descending order. Everything starts at the module level from where `main()` function is called. We can also see that most of this is from `api.py:request()`, `sessions.py:request()` and `sessions.py:send()`. However, we are unable to tell if these are part of the same call flow or parallel flows. A graphical viewer will be more useful.

The column "tottime" is the time spent within a function without considering time spent in functions called within. We also see that `readline()` function from `socket.py` is called many times. It takes 1.571 seconds of 4.481 seconds of cumulative time. However, the total time within this function is 0. This means that we need to optimize functions that are called by `readline()`. We suspect it might be the read operation of SSL Socket. The entry "4/3" indicates that `sessions.py:send()` is called 3 times plus 1 recursive call.

### What should I look for when analyzing any profiling output?

You should identify which function (or lines of code) is taking most of the execution time. You should identify which function is getting called most often. Look for call stacks that you didn't expect. Know the difference between *total time* spent in function and *cumulative time*. The latter enables direct comparison of recursive implementations against iterative ones.

With respect to memory usage, see if there's a memory leak.

### Could you share some tips for optimizing profiled code?

Keep in mind that speeding up a function 100 times is irrelevant if that function takes up only a few percent of the program's total execution. This is the essence of profiling. Optimize only what matters.

If profiling shows that I/O is a bottleneck, threading can help. If your code uses a lot of regex, try to replace these with string equivalents, which will run faster. To avoid repeat computations, earlier results can be cached. This is called *memoization*. Where applicable, this can help in improving performance. If there are no obvious places to optimize the code, you can also consider an alternative runtime such as PyPy or moving critical parts of the code into Cython or C and calling the same from Python. Likewise, vectorize some operations using NumPy. Consider moving stuff from inner loops. Remove logging or make them conditional. If a particular function is called many times, code could be restructured.

Whether during profiling or after optimizing your code, don't ignore your unit tests. They must continue to pass. Also, keep in mind that optimizing code can compromise on readability and maintainability.

### Are there Python IDEs that come with profilers?

With IPython, Jupyter and Anaconda's Spyder, `%time` and `%timeit` can be used by default. For line profiling or memory profiling, the necessary modules need to be installed before invoking them.

PyCharm Community Edition does not support profiling but the professional version supports it. In fact, cProfile, yappi and vmprof are three profilers that are supported. vmprof is a statistical profiler.

Visual Studio comes with a built-in profiler. However, this works only for CPython and not for IronPython for which .NET profiler should be used.

### What's the decorator approach to profiling Python code?

It's possible to decorate a function for profiling purpose. We can timestamp the entries and exits and thereby calculate time spent within the function. The advantage of this approach is that there's no dependence on any profiling tool. Overhead can be kept low. We can choose to profile only specific parts of our program and ignore core modules and third-party modules. We'll also have better control of how results will be exported.

### What's Pyflame and why may I want to use it?

Pyflame is a profiler for Python that takes snapshots of the Python call stack. From these, it generates flame graphs. Pyflame is based on Linux's `ptrace`. Hence it can't be used on Windows systems. Pyflame overcomes the limitations of cProfile, which adds overhead and doesn't give a full stack trace. Pyflame also works with code not instrumented for profiling.

Because Pyflame is statistical in nature, meaning that it doesn't profile every since function call, it can also be used in production.

## Milestones

Apr
1992

In CPython implementation, first code commit is made for `profile` module.

Jun
1994

In CPython implementation, first code commit is made for `pstats` module.

Feb
2006

In CPython implementation, first code commit is made for `cProfile` module.

Sep
2008

Initial code commit for `line_profiler` is done by Robert Kern. Version 2.1 of this module is released in December 2017.

Sep
2016

First version of Pyflame is released by the engineering team at Uber.

Profiling Python Code


# Transformer Neural Network Architecture

## Summary

Given a word sequence, we recognize that some words within it are more closely related with one another than others. This gives rise to the concept of **self-attention** in which a given word "attends to" other words in the sequence. Essentially, attention is about representing context by giving weights to word relations.

Transformer is a neural network architecture that makes use of self-attention. It replaces earlier approaches of LSTMs or CNNs that used attention between encoder and decoder. Transformer showed that a feed-forward network used with self-attention is sufficient.

Influential language models such BERT and GPT-2 are based on the transformer architecture. By 2019, transformer architecture became an active area of research and application. While initially created for NLP, it's being used in other domains where problems can be cast as sequence modelling.

## Discussion

### How is the transformer network better than CNNs, RNNs or LSTMs?

Words in a sentence come one after another. The context of the current word is established by the words surrounding it. RNNs are suited to model such a time-sequential structure. But an RNN has trouble remembering long sequences. LSTM is an RNN variant that does better in this regard. CNN architectures WaveNet, ByteNet and ConvS2S have also been used for sequence-to-sequence learning.

Moreover, RNNs and LSTMs consider only words that have gone before (although there's bidirectional LSTMs). Self-attention models the context by looking at words before and after the current word. For instance, the word "bank" in sentence "I arrived at the bank after crossing the river" doesn't refer to a financial institution. Transformer can figure out this meaning because it looks at subsequent words as well.

The sequential nature of RNNs implies that tasks can't be parallelized on GPUs and TPUs. Transformer's encoder self-attention can be parallelized. While CNNs are less sequential, complexity still grows logarithmically. It's worse for RNNs where complexity grows linearly. With transformers, the number of sequential operations is constant.

### What's the architecture of the transformer?

The transformer of Vaswani et al. basically follows the encoder-decoder model with attention passed from encoder to decoder. Both encoder and decoder stack multiple identical layers. Each encoder layer uses self-attention to represent context. Each decoder layer also uses self-attention in two sub-layers. While the encoder's self-attention uses both left and right context, the lower sub-layer of decoder masks out the future positions while predicting the current position.

In each layer we find some common elements. Residual connections are made. These are added and normalized with connections flowing via the self-attention sub-layers. There are no recurrent networks, only a fully connected feed-forward network.

At the input, source and target sequences are represented as embeddings. These are enhanced with positional encodings. At the output, a linear layer is followed with softmax.

The transformer's encoder can work on the input sequence in parallel but the decoder is auto-regressive. Each output is influenced by previous output symbols. Output symbols are generated one at a time.

### How is self-attention computed in a transformer network?

Every word is projected on to three vectors: query, key and value. Respective weight matrices \(W\) to do this projection are learned during training. Suppose we're calculating the attention on a particular word. A dot-product operation of its query vector with the key vector of each word is calculated. Dot-product attention is scaled with \(1/\sqrt d\_k\) to compensate large dot-product values. The value vectors are weighted with weights from the dot product and then summed.

For better results, **multi-head attention** is used. Each head learns a different attention distribution, similar to having multiple filters in CNN. For example, if the model dimension is 512, instead of a large single attention layer, we use 8 parallel attention layers, each operating in 64 dimensions. Output from the layers are concatenated to derive the final attention. Mathematically, we have the following:

$$MultiHead(Q,K,V) = Concat(head\_1,...,head\_h)W^O\\head\_i = Attention(QW^{Q}\_i, KW^{K}\_i, VW^{V}\_i)\\Attention(Q,K,V) = softmax(\frac{QK^T}{\sqrt d\_k})V$$

The original transformer of Vaswani et al. uses self-attention within encoder and decoder, but also transfers attention from encoder to decoder as is common in traditional sequence-to-sequence models.

### How does the transformer network capture the position of words?

In RNNs, the sequential structure accounts for position. In CNNs, positions are considered within the kernel size. In transformers, self-attention ignores the position of tokens within the sequence. To overcome this limitation, transformers explicitly add **positional encodings**. These are added to the input or output embeddings before the sum goes into the first attention layer.

Positional encodings can either be learned or fixed. In the latter case, Vaswani et al. used sine and cosine functions for even and odd positions respectively. They also used different frequencies for different positions to make it easier for the model to learn the positions:

$$PE\_{(pos,2i)}=sin(pos/10000^{2i/d\_{model}})\\PE\_{(pos,2i+1)}=cos(pos/10000^{2i/d\_{model}})$$

While Vaswani et al. (2017) considered absolute positions, Shaw et al. (2018) looked at the distance between tokens in a sequence, that is, relative positioning. They showed that this leads to better results for machine translation with the trade-off of 7% decrease in steps per second.

### Could you share some applications of the transformer network?

In October 2019, Google announced the use of BERT for 10% of its English language search. Search will attempt to understand queries the way users tend to ask them in a natural way. This is opposed to parsing the query as a bunch of keywords. Thus, phrases such as "to" or "for someone" are important for meaning and BERT picks up these.

We can use transformers to generate synthetic text. Starting from a small prompt, GPT-2 model is able to generate long sequences and paragraphs of text that are realistic and coherent. This text also adapts to the style of the input.

For correcting grammar, transformers provide competitive baseline performance. For sequence generation, Insertion Transformer and Levenshtein Transformer have been proposed.

Transformers have been used beyond NLP, such as for image generation where self-attention is restricted to local neighbourhoods. Music Transformer applied self-attention to generate long pieces of music. While the original transformer used absolute positions, the music transformer used relative attention, allowing the model to create music in a consistent style.

### Which are the well-known transformer networks?

**BERT** is an encoder-only transformer. It's the first deeply bidirectional model, meaning that it uses both left and right contexts in all layers. BERT showed that as a pretrained language model it can be fine-tuned easily to obtain state-of-the-art models for many specific tasks. BERT has inspired many variants: RoBERTa, XLNet, MT-DNN, SpanBERT, VisualBERT, K-BERT, HUBERT, and more. Some variants attempt to compress the model: TinyBERT, ALERT, DistilBERT, and more.

The other competitive model is **GPT-2**. Unlike BERT, GPT-2 is not bidirectional and is a decoder-only transformer. However, the training includes both unsupervised pretraining and supervised fine-tuning. The training objective combines both of these to improve generalization and convergence. This approach of training on specific tasks is also seen in **MT-DNN**.

GPT-2 is auto-regressive. Each output token is generated one by one. Once a token is generated, it's added to the input sequence. BERT is not auto-regressive but instead uses context from both sides. XLNet is auto-regressive while also using context from both sides.

### What are some variations of the transformer network?

Compared to the original transformer of Vaswani et al., we note the following variations:

    + **Transformer-XL**: Overcomes the limitation of fixed-length context. It makes use of segment-level recurrence and relative positional encoding.
    + **DS-Init & MAtt**: Stacking many layers is problematic due to vanishing gradients. Therefore, depth-scaled initialization and merged attention sublayer are proposed.
    + **Average Attention Network (AAN)**: With the original transformer, decoder's self-attention is slow due to its auto-regressive nature. Speed is improved by replacing self-attention with an averaging layer followed by a gating layer.
    + **Dialogue Transformer**: Conversation that has multiple overlapping topics can be picked out. Self-attention is over the dialogue sequence turns.
    + **Tensor-Product Transformer**: Uses novel TP-Attention to explicitly encode relations and applies it to math problem solving.
    + **Tree Transformer**: Puts a constraint on the encoder to follow tree structures that are more intuitive to humans. This also helps us learn grammatical structures from unlabelled data.
    + **Tensorized Transformer**: Multi-head attention is difficult to deploy in a resource-limited setting. Hence, multi-linear attention with Block-Term Tensor Decomposition (BTD) is proposed.

### For a developer, what resources are out there to learn transformer networks?

To get a feel of transformers in action, you can try out Talk to Transformer, which is based on the full-sized GPT-2.

HuggingFace provides implementation of many transformer architectures in both TensorFlow and PyTorch. You can also convert them to CoreML models for iOS devices. Package spaCy also interfaces to HuggingFace.

TensorFlow code and pretrained models for BERT are available. There's also code for Transformer-XL, MT-DNN and GPT-2.

TensorFlow has provided an implementation for machine translation. Lilian Weng's implementation of the transformer is worth studying. Samuel Lynn-Evans has shared his implementation with explanations. The Annotated Transformer is another useful resource to learn the concepts along with the code.

## Milestones

2014

Sutskever et al. at Google apply **sequence-to-sequence** model to the task of machine translation, that is, a sequence of words in source language is translated to a sequence of words in target language. They use an **encoder-decoder** architecture that has separate 4-layered LSTMs for encoder and decoder. The encoder produces a fixed-length context vector, which is used to initialize the decoder. The main limitation is that the context vector is unable to adequately represent long sentences.

2015

Bahdanau et al. apply the concept of **attention** to the seq2seq model used in machine translation. This helps the decoder to "pay attention" to important parts of the source sentence. Encoder is a bidirectional RNN. Unlike the seq2seq model of Sutskever et al., which uses only the encoder's last hidden state, attention mechanism uses all hidden states of encoder and decoder to generate the context vector. It also aligns the input and output sequences, with alignment score parameterized by a feed-forward network.

Jun
2017

Vaswani et al. propose the **transformer** model in which they use a seq2seq model without RNN. The transformer model relies only on **self-attention**, although they're not the first to use self-attention. Self-attention is about attending to different tokens of the sequence.

Jan
2018

For multi-document summarization, Liu et al. propose a **decoder-only transformer** architecture that can attend to sequences longer than what encoder-decoder architecture is capable of. Input and output sequences are combined into single sequence and used to train the decoder. During inference, output is generated auto-regressively. They also propose variations of attention to handle longer sequences.

Jun
2018

OpenAI publishes **Generative Pre-trained Transformer (GPT)**. It's inspired by unsupervised pre-training and transformer architecture. The transformer is trained on large amount of data without supervision. It's then fine-tuned on smaller task-specific datasets with supervision. Pre-training involves a standard language modelling and uses Liu et al.'s decoder-only transformer. In February 2019, OpenAI announces an improved model named **GPT-2**. Compared to GPT, GPT-2 is trained on 10x the data and has 10x parameters.

Oct
2018

Google open sources **Bidirectional Encoder Representations from Transformers (BERT)**, which is a pre-trained language model. It's deeply bidirectional and unsupervised. It improves state-of-the-art in many NLP tasks. It's trained on two tasks: (a) Masked Language Model (MLM), predicting some words that are masked in a sequence; (b) Next Sentence Prediction (NSP), binary classification that predicts if the next sentence follows the current sentence.

Jan
2019

Combining Multi-Task Learning (MTL) and pretrained language model, Liu et al. propose **Multi-Task Deep Neural Network (MT-DNN)**. Lower layers of the architecture are shared across tasks and use BERT. Higher layers do task-specific training. They show that this approach outperforms BERT in many tasks even without fine-tuning.

Jan
2019

Just as AutoML has been used in computer vision, Google researchers use an evolution-based neural architecture search (NAS) to discover what they call **Evolved Transformer (ET)**. It performs better than the original transformer of Vaswani et al. It's seen that ET is a hybrid, combining the best of self-attention and wide convolution.

Transformer Neural Network Architecture