# Hypothesis Testing and Types of Errors

## Summary

Suppose we want to study income of a population. We study a sample from the population and draw conclusions. The sample should represent the population for our study to be a reliable one.

**Null hypothesis** \((H\_0)\) is that sample represents population. Hypothesis testing provides us with framework to conclude if we have sufficient evidence to either accept or reject null hypothesis.

Population characteristics are either assumed or drawn from third-party sources or judgements by subject matter experts. Population data and sample data are characterised by moments of its distribution (mean, variance, skewness and kurtosis). We test null hypothesis for equality of moments where population characteristic is available and conclude if sample represents population.

For example, given only mean income of population, we validate if mean income of sample is close to population mean to conclude if sample represents the population.

## Discussion

### What are the math representations of population and sample parameters?

Population mean and population variance are denoted in Greek alphabets \(\mu\) and \(\sigma^2\) respectively, while sample mean and sample variance are denoted in English alphabets \(\bar x\) and \(s^2\) respectively.

### What's the relevance of sampling error to hypothesis testing?

Suppose we obtain a sample mean of \(\bar x\) from a population of mean \(\mu\). The two are defined by the relationship |\(\bar x\) - \(\mu\)|>=0:

    + If the difference is not significant, we conclude the difference is due to sampling. This is called **sampling error** and this happens due to chance.
    + If the difference is significant, we conclude the sample does not represent the population. The reason has to be more than chance for difference to be explained.Hypothesis testing helps us to conclude if the difference is due to sampling error or due to reasons beyond sampling error.

### What are some assumptions behind hypothesis testing?

A common assumption is that the observations are independent and come from a random sample. The population distribution must be Normal or the sample size is large enough. If the sample size is large enough, we can invoke the *Central Limit Theorem (CLT)* regardless of the underlying population distribution. Due to CLT, sampling distribution of the sample statistic (such as sample mean) will be approximately a Normal distribution.

A rule of thumb is 30 observations but in some cases even 10 observations may be sufficient to invoke the CLT. Others require at least 50 observations.

### What are one-tailed and two-tailed tests?

When acceptance of \(H\_0\) involves boundaries on both sides, we invoke the **two-tailed test**. For example, if we define \(H\_0\) as sample drawn from population with age limits in the range of 25 to 35, then testing of \(H\_0\) involves limits on both sides.

Suppose we define the population as greater than age 50, we are interested in rejecting a sample if the age is less than or equal to 50; we are not concerned about any upper limit. Here we invoke the **one-tailed test**. A one-tailed test could be left-tailed or right-tailed.

Consider average gas price in California compared to the national average of $2.62. If we believe that the price is higher in California, we consider right-tailed test. If we believe that California price is different from national average but we don't know if it's higher or lower, we consider two-tailed test. Symbolically, given the **alternative or research hypothesis** \(H\_1\), we state,

    + \(H\_0\): \(\mu = \$ 2.62\)
    + \(H\_1\) right-tailed: \(\mu > \$ 2.62\)
    + \(H\_1\) two-tailed: \(\mu \neq \$ 2.62\)

### What are the types of errors in hypothesis testing?

In concluding whether sample represents population, there is scope for committing errors on following counts:

    + Not accepting that sample represents population when in reality it does. This is called **type-I** or **\(\alpha\) error**.
    + Accepting that sample represents population when in reality it does not. This is called **type-II** or **\(\beta\) error**.For instance, granting loan to an applicant with low credit score is \(\alpha\) error. Not granting loan to an applicant with high credit score is (\(\beta\)) error.

The symbols \(\alpha\) and \(\beta\) are used to represent the probability of type-I and type-II errors respectively.

### How do we measure type-I or \(\alpha\) error?

The p-value can be interpreted as the probability of getting a result that's same or more extreme when the null hypothesis is true.

The observed sample mean \(\bar x\) is overlaid on population distribution of values with mean \(\mu\) and variance \(\sigma^2\). The proportion of values beyond \(\bar x\) and away from \(\mu\) (either in left tail or in right tail or in both tails) is **p-value**. If p-value <= \(\alpha\) we reject null hypothesis. The results are said to be **statistically significant** and not due to chance.

Assuming \(\alpha\)=0.05, p-value > 5%, we conclude the sample is highly likely to be drawn from population with mean \(\mu\) and variance \(\sigma^2\). We accept \((H\_0)\). Otherwise, there's insufficient evidence to be part of population and we reject \(H\_0\).

We preselect \(\alpha\) based on how much type-I error we're willing to tolerate. \(\alpha\) is called **level of significance**. The standard for level of significance is 0.05 but in some studies it may be 0.01 or 0.1. In the case of two-tailed tests, it's \(\alpha/2\) on either side.

### How do we determine sample size and confidence interval for sample estimate?

**Law of Large Numbers** suggests larger the sample size, the more accurate the estimate. Accuracy means the variance of estimate will tend towards zero as sample size increases. Sample Size can be determined to suit accepted level of tolerance for deviation.

Confidence interval of sample mean is determined from sample mean offset by variance on either side of the sample mean. If the population variance is known, then we conduct z-test based on Normal distribution. Otherwise, variance has to be estimated and we use t-test based on t-distribution.

The formulae for determining sample size and confidence interval depends on what we to estimate (mean/variance/others), sampling distribution of estimate and standard deviation of estimate's sampling distribution.

### How do we measure type-II or \(\beta\) error?

We overlay sample mean's distribution on population distribution, the proportion of overlap of sampling estimate's distribution on population distribution is **\(\beta\) error**.

Larger the overlap, larger the chance the sample does belong to population with mean \(\mu\) and variance \(\sigma^2\). Incidentally, despite the overlap, p-value may be less than 5%. This happens when sample mean is way off population mean, but the variance of sample mean is such that the overlap is significant.

### How do we control \(\alpha\) and \(\beta\) errors?

Errors \(\alpha\) and \(\beta\) are dependent on each other. Increasing one decreases the other. Choosing suitable values for these depends on the cost of making these errors. Perhaps it's worse to convict an innocent person (type-I error) than to acquit a guilty person (type-II error), in which case we choose a lower \(\alpha\). But it's possible to decrease both errors but collecting more data.

Just as p-value manifests \(\alpha\), **Power of Test** manifests \(\beta\). Power of test is \(1-\beta\). Among the various ways to interpret power are:

    + Probability of rejecting the null hypothesis when, in fact, it is false.
    + Probability that a test of significance will pick up on an effect that is present.
    + Probability of avoiding a Type II error.Low p-value and high power help us decisively conclude sample doesn't belong to population. When we cannot conclude decisively, it's advisable to go for larger samples and multiple samples.

In fact, power is increased by increasing sample size, effect sizes and significance levels. Variance also affects power.

### What are some misconceptions in hypothesis testing?

A common misconception is to consider "p value as the probability that the null hypothesis is true". In fact, p-value is computed under the assumption that the null hypothesis is true. P-value is the probability of observing the values, or more extremes values, if the null hypothesis is true.

Another misconception, sometimes called **base rate fallacy**, is that under controlled \(\alpha\) and adequate power, statistically significant results correspond to true differences. This is not the case, as shown in the figure. Even with \(\alpha\)=5% and power=80%, 36% of statistically significant p-values will not report the true difference. This is because only 10% of the null hypotheses are false (base rate) and 80% power on these gives only 80 true positives.

P-value doesn't measure the size of the effect, for which **confidence interval** is a better approach. A drug that gives 25% improvement may not mean much if symptoms are innocuous compared to another drug that gives small improvement from a disease that leads to certain death. Context is therefore important.

## Milestones

1710

The field of **statistical testing** probably starts with John Arbuthnot who applies it to test sex ratios at birth. Subsequently, others in the 18th and 19th centuries use it in other fields. However, modern terminology (null hypothesis, p-value, type-I or type-II errors) is formed only in the 20th century.

1900

Pearson introduces the concept of **p-value** with the chi-squared test. He gives equations for calculating P and states that it's "the measure of the probability of a complex system of n errors occurring with a frequency as great or greater than that of the observed system."

1925

Ronald A. Fisher develops the concept of p-value and shows how to calculate it in a wide variety of situations. He also notes that a value of 0.05 may be considered as conventional cut-off.

1933

Neyman and Pearson publish *On the problem of the most efficient tests of statistical hypotheses*. They introduce the notion of **alternative hypotheses**. They also describe both **type-I and type-II errors** (although they don't use these terms). They state, "Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong."

1949

Johnson's textbook titled *Statistical methods in research* is perhaps the first to introduce to students the Neyman-Pearson hypothesis testing at a time when most textbooks follow Fisher's significance testing. Johnson uses the terms "error of the first kind" and "error of the second kind". In time, Fisher's approach is called **P-value approach** and the Neyman-Pearson approach is called **fixed-α approach**.

1993

Carver makes the following suggestions: use of the term "statistically significant"; interpret results with respect to the data first and statistical significance second; and pay attention to the size of the effect.

Hypothesis Testing and Types of Errors


# Polygonal Modelling

## Summary

Polygonal modelling is a 3D modelling approach that utilizes edges, vertices and faces to form models. Modellers start with simple shapes and add details to build on them. They alter the shapes by adjusting the coordinates of one or more vertices. A polygonal model is called faceted as polygonal faces determine its shape.

Polygonal or polyhedral modelling fits best where visualization matters more than precision. It's extensively used by video game designers and animation studios. Assets in video games form whole worlds for gamers. Features of these assets are built using polygonal modelling.

Computers take less time to render polygonal models. So, polygonal modelling software run well on browsers. For higher precision, advanced 3D models such as NURBS are suitable. However, NURBs can't be 3D printed unless they are converted to polygons. Many industrial applications easily handle polygonal model representations.

## Discussion

### Can you describe the basic elements of polygonal modelling?

A **vertex** is the smallest component of a 3D model. Two or more edges of a polygon meet at a vertex.

**Edges** define the shape of the polygons and the 3D model. They are straight lines connecting the vertices.

Triangles and quadrilaterals are the polygons generally used. Some applications offer the use of polygons with any number of edges (N-gons) to work with.

Faces of polygons combine to form polygonal **meshes**. One can **deform** meshes. That is, one may move, twist or turn meshes to create 3D objects using deformation tools in the software. The number of polygons in a mesh makes its **polycount**.

**UV coordinates** are the horizontal (U) and vertical (V) axes of the 2D space. 3D meshes are converted into 2D information to wrap textures around them.

Polygon density in the meshes is its **resolution**. Higher resolution indicates better detailing. Good 3D models contain high-resolution meshes where fine-detailing matters and low-resolution meshes where detailing isn't important.

### How are polygonal meshes generated?

Polygonal meshes are generated by converting a set of spatial points into vertices, faces and edges. These components meet at shared boundaries to form physical models.

Polygonal mesh generation (aka meshing) is of two types: **Manual** and **Automatic**. In manual meshing, the positions of vertices are edited one by one. In automatic meshing, values are fed into the software. The software automatically constructs meshes based on the specified values. The automatic method enables the rapid creation of 3D objects in games, movies and VR.

Meshing is performed at two levels. At the model's surface level, it's called **Surface meshing**. Surface meshes won't have free edges or a common edge shared by more than two polygons.

Meshing in its volume dimension is called **Solid meshing**. The solid surfaces in solid meshing are either polyhedral or trimmed.

There are many ways to produce polygonal meshes. Forming primitives from standard shapes is one way. Meshes can also be drawn by interpolating edges or points of other objects. Converting existing solid models and stretching custom-made meshes into fresh meshes are two other options.

### What are free edges, manifold edges and non-manifold edges?

A **free edge** in a mesh is an edge that doesn't fully merge with the edge of its neighbouring element. The nodes of meshes with free edges won't be accurately connected. Such edges within the geometry will affect the overall output. Therefore, unwanted free meshes should be removed.

A **manifold edge** is an edge shared utmost by two faces. It means, when there is a third face sharing the edge, it becomes a **non-manifold** edge.

A non-manifold edge cannot be replicated in the real world. Hence it should be removed while modelling. In the event of 3D printing, non-manifold edges will produce failed models.

### How would you classify the polygonal meshing process based on grid structure?

A grid structure works on the principle of Finite Element Analysis (FEA). An FEA node can be thought of as the vertex of a polygon in polygonal modelling. An FEA element shall represent an edge, a shape and a solid in three different dimensions.

Dividing the expanse of a polygonal model into small elements before computing forms a grid. Grid structure-wise, meshing is of two types:

    + **Structured meshing** displays a definite pattern in the arrangement of nodes or elements. The size of each element in it is nearly the same. It enables easy access to the coordinates of these elements. It's applicable to uniform grids made of rectangles, ellipses and spheres that make regular grids.
    + **Unstructured meshing** is arbitrary and forms irregular geometric shapes. The connectivity between elements is not uniform. So, unstructured meshes do not follow a definite pattern. It requires that the connectivity between elements is well-defined and properly stored. The axes of these elements are unaligned (non-orthogonal).

### How are mesh generation algorithms written for polygonal modelling?

Mesh generation algorithms are written according to the principles of the chosen mesh generation method. There are many methods to generating meshes. It depends on the mesh type.

A mesh generation method serves the purposes of generating nodes (geometry) and connecting nodes (topology).

Let's take the Delaunay triangulation method for instance. According to it, the surface domain elements are discretized into non-overlapping triangles. The nodes are so created that the angles between them when triangulated are the least. The circumcircle drawn about each triangle cannot accommodate an additional triangle within it.

Delaunay triangulation is applied through several algorithms. Boyer-Watson algorithm is one of them. It's an incremental algorithm that adds one node at a time in a given triangulation. If the new point falls within the circumcircle of a triangle, the triangle is removed. Using the new point a fresh triangle is formed.

### How does one fix the polygon count for models?

Polygon count or polycount gives a measure of visual quality. Detailing needs a high number of polygons. It gives a photorealistic effect.  But high polycount impacts efficiency. It may take more time to load and render. When a model takes more time to download, we may run out of patience. Real-time rendering delays cause a video or animation to stop and start. So, a good polygonal model is a combination of high visual quality and low polycount.

The threshold number to call a polygon count high is subjective. For mobile devices, anywhere between 300 to 1500 polygons is good. Desktops can comfortably accommodate 1500 to 4000 polygons without affecting performance.

These polycount numbers vary depending on the CPU configuration and other hardware capabilities. Advanced rendering capabilities smoothly handle anywhere between 10k to 40k polygons. Global mobile markets are vying to produce CPUs that can render 100k to 1 million polygons for an immersive 3D experience.

Higher polycount increases the file sizes of 3D assets. Websites will have upload limits. So it's also important to keep file sizes in mind while fixing the polygon count.

### What are some beginner pitfalls to polygonal modelling?

**Irregular meshes**: As beginners, we may miss triangles and create self-intersecting surfaces. Or we may leave holes on mesh surfaces or fill in with backward triangles. Irregular meshes will affect the model's overall appearance. Eyeball checks and use of mesh generation software will help us avoid mesh-related errors.

**Incorrect measurements**: It may distort the model's proportionality and ruin the output. It's best to train our eyes to compare images and estimate the difference in depths. Comparing our model with the reference piece on the image viewer tool will tell us the difference.

**Too many subdivisions early in the modelling**: It will disable us from making changes without tampering with the measurements. So, we may end up creating uneven surfaces. Instead, it's better to start with fewer polygons and add to them as we build the model.

**Topology error**: We may get the edge structure and mesh distributions wrong. We need to equip ourselves by learning how to use mesh tools. It's important to learn where to use triangles, quads and higher polygons. Duplicates are to be watched out for. Understanding the flow of edges is vital.

## Milestones

1952

Geoffrey Colin Shepherd furthers Thomas Bradwardine's 14th-century work on non-convex polygons. He extends polygon formation to the imaginary plane. It paves the way for the construction of complex polygons. In polygonal modelling, complex polygons have circuitous boundaries. A polygon with a hole inside is one example.

1972

Bruce G Baumgart introduces a paper on **winged edge data structure** at Stanford University. Winged data structure is a way of representing polyhedrons on a computer. The paper states its exclusive use in AI for computer graphics and world modelling.

1972

Newell introduces the **painter's algorithm**. It's a painting algorithm that paints a polygon. It considers the distance of the plane from the viewer while painting. The algorithm paints the farthest polygon from the viewer first and proceeds to the nearest.

1972

Edwin Catmull and Fredrick Parke create the **world's first 3D rendered movie**. In the movie, the animation of Edwin's left hand has precisely drawn and measured polygons.

1992

Fowlery et al. present *Modelling Seashells* at ACM SIGGRAPH, Chicago. They use polygonal meshes among others to create comprehensive computer imagery of seashells.

1998

Andreas Raab suggests the **classification of edges** of a polygonal mesh. They shall be grouped as sharp, smooth, contour and triangulation edges. It solves the problem of choosing the right lines to draw.

1999

Deussen et al. successfully apply Adreas Raab's algorithm that constructs a skeleton from a 3D polygonal model. They use it in connection with the intersecting planes.

Polygonal Modelling


# Relation Extraction

## Summary

Consider the phrase "President Clinton was in Washington today". This describes a *Located* relation between Clinton and Washington. Another example is "Steve Balmer, CEO of Microsoft, said…", which describes a *Role* relation of Steve Balmer within Microsoft.

The task of extracting semantic relations between entities in text is called **Relation Extraction (RE)**. While Named Entity Recognition (NER) is about identifying entities in text, RE is about finding the relations among the entities. Given unstructured text, NER and RE helps us obtain useful structured representations. Both tasks are part of the discipline of Information Extraction (IE).

Supervised, semi-supervised, and unsupervised approaches exist to do RE. In the 2010s, neural network architectures were applied to RE. Sometimes the term **Relation Classification** is used, particularly in approaches that treat it as a classification problem.

## Discussion

### What sort of relations are captured in relation extraction?

Here are some relations with examples:

    + *located-in*: CMU is in Pittsburgh
    + *father-of*: Manuel Blum is the father of Avrim Blum
    + *person-affiliation*: Bill Gates works at Microsoft Inc.
    + *capital-of*: Beijing is the capital of China
    + *part-of*: American Airlines, a unit of AMR Corp., immediately matched the moveIn general, affiliations involve persons, organizations or artifacts. Geospatial relations involve locations. Part-of relations involve organizations or geo-political entities.

**Entity tuple** is the common way to represent entities bound in a relation. Given n entities in a relation r, the notation is \(r(e\_{1},e\_{2},...,e\_{n})\). An example use of this notation is *Located-In(CMU, Pittsburgh)*.

RE mostly deals with binary relations where n=2. For n>2, the term used is **higher-order relations**. An example of 4-ary biomedical relation is *point\_mutation(codon, 12, G, T)*, in the sentence "At codons 12, the occurrence of point mutations from G to T were observed".

### What are some common applications of relation extraction?

Since structured information is easier to use than unstructured text, relation extraction is useful in many NLP applications. RE enriches existing information. Once relations are obtained, they can be stored in databases for future queries. They can be visualized and correlated with other information in the system.

In question answering, one might ask "When was Gandhi born?" Such a factoid question can be answered if our relation database has stored the relation *Born-In(Gandhi, 1869)*.

In biomedical domain, protein binding relations can lead to drug discovery. When relations are extracted from a sentence such as "Gene X with mutation Y leads to malignancy Z", these relations can help us detect cancerous genes. Another example is to know the location of a protein in an organism. This ternary relation is split into two binary relations (Protein-Organism and Protein-Location). Once these are classified, the results are merged into a ternary relation.

### Which are the main techniques for doing relation extraction?

With **supervised learning**, the model is trained on annotated text. Entities and their relations are annotated. Training involves a binary classifier that detects the presence of a relation, and a classifier to label the relation. For labelling, we could use SVMs, decision trees, Naive Bayes or MaxEnt. Two types of supervision are feature-based or kernel-based.

Since finding large annotated datasets is difficult, a **semi-supervised** approach is more practical. One approach is to do a phrasal search with wildcards. For example, `[ORG] has a hub at [LOC]` would return organizations and their hub locations. If we relax the pattern, we'll get more matches but also false positives.

An alternative is to use a set of specific patterns, induced from an initial set of seed patterns and seed tuples. This approach is called **bootstrapping**. For example, given the seed tuple *hub(Ryanair, Charleroi)* we can discover many phrasal patterns in unlabelled text. Using these patterns, we can discover more patterns and tuples. However, we have to be careful of **semantic drift**, in which one wrong tuple/pattern can lead to further errors.

### What sort of features are useful for relation extraction?

Supervised learning uses features. The named entities themselves are useful features. This includes an entity's bag of words, head words and its entity type. It's also useful to look at words surrounding the entities, including words that are in between the two entities. Stems of these words can also be included. The distance between the entities could be useful.

The **syntactic structure** of the sentence can signal the relations. A syntax tree could be obtained via base-phrase chunking, dependency parsing or full constituent parsing. The paths in these trees can be used to train binary classifiers to detect specific syntactic constructions. The accompanying figure shows possible features in the sentence "[ORG American Airlines], a unit of AMR Corp., immediately matched the move, spokesman [PERS Tim Wagner] said."

When using syntax, expert knowledge of linguistics is needed to know which syntactic constructions correspond to which relations. However, this can be automated via machine learning.

### Could you explain kernel-based methods for supervised relation classification?

Unlike feature-based methods, kernel-based methods don't require explicit feature engineering. They can explore a large feature space in polynomial computation time.

The essence of a kernel is to compute the **similarity** between two sequences. A kernel could be designed to measure structural similarity of character sequences, word sequences, or parse trees involving the entities. In practice, a kernel is used as a similarity function in classifiers such as SVM or Voted Perceptron.

We note a few kernel designs:

    + **Subsequence**: Uses a sequence of words made of the entities and their surrounding words. Word representation includes POS tag and entity type.
    + **Syntactic Tree**: A constituent parse tree is used. Convolution Parse Tree Kernel is one way to compare similarity of two syntactic trees.
    + **Dependency Tree**: Similarity is computed between two dependency parse trees. This could be enhanced with shallow semantic parsers. A variation is to use dependency graph paths in which the shortest path between entities represents a relation.
    + **Composite**: Combines the above approaches. Subsequence kernels capture lexical information whereas tree kernels capture syntactic information.

### Could you explain distant supervised approach to relation extraction?

Due to extensive work done for Semantic Web, we already have many knowledge bases that contain `entity-relation-entity` triplets. Examples include DBpedia (3K relations), Freebase (38K relations), YAGO, and Google Knowledge Graph (35K relations). These can be used for relation extraction without requiring annotated text.

Distant supervision is a combination of unsupervised and supervised approaches. It extracts relations without supervision. It also induces thousands of features using a probabilistic classifier.

The process starts by linking named entities to those in the knowledge bases. Using relations in the knowledge base, the patterns are picked up in the text. Patterns are applied to find more relations. Early work used DBpedia and Freebase, and Wikipedia as the text corpus. Later work utilized semi-structured data (HTML tables, Wikipedia list pages, etc.) or even a web search to fill gaps in knowledge graphs.

### Could you compare some semi-supervised or unsupervised approaches of some relation extraction tools?

DIPRE's algorithm (1998) starts with seed relations, applies them to text, induces patterns, and applies the patterns to obtain more tuples. These steps are iterated. When applied to *(author, book)* relation, patterns take the form `(longest-common-suffix of prefix strings, author, middle, book, longest-common-prefix of suffix strings)`. DIPRE is an application of Yarowsky algorithm (1995) invented for WSD.

Like DIPRE, Snowball (2000) uses seed relations but doesn't look for exact pattern matches. Tuples are represented as vectors, grouped using similarity functions. Each term is also weighted. Weights are adjusted with each iteration. Snowball can handle variations in tokens or punctuation.

KnowItAll (2005) starts with domain-independent extraction patterns. Relation-specific and domain-specific rules are derived from the generic patterns. The rules are applied on a large scale on online text. It uses pointwise mutual information (PMI) measure to retain the most likely patterns and relations.

Unlike earlier algorithms, TextRunner (2007) doesn't require a pre-defined set of rules. It learns relations, classes and entities on its own from a large corpus.

### How are neural networks being used to do relation extraction?

Neural networks were increasingly applied to relation extraction from the early 2010s. Early approaches used **Recursive Neural Networks** that were applied to syntactic parse trees. The use of **Convolutional Neural Networks (CNNs)** came next, to extract sentence-level features and the context surrounding words. A combination of these two networks has also been used.

Since CNNs failed to learn long-distance dependencies, **Recurrent Neural Networks (RNNs)** were found to be more effective in this regard. By 2017, basic RNNs gave way to gated variants called GRU and LSTM. A comparative study showed that CNNs are good at capturing local and position-invariant features whereas RNNs are better at capturing order information long-range context dependency.

The next evolution was towards **attention mechanism** and **pre-trained language models** such as BERT. For example, attention mechanism can pick out most relevant words and use CNNs or LSTMs to learn relations. Thus, we don't need explicit dependency trees.  In January 2020, it was seen that BERT-based models represent the current state-of-the-art with an F1 score close to 90.

### How do we evaluate algorithms for relation extraction?

Recall, precision and F-measures are typically used to evaluate on a gold-standard of human annotated relations. These are typically used for supervised methods.

For unsupervised methods, it may be sufficient to check if a relation has been captured correctly. There's no need to check if every mention of the relation has been detected. Precision here is simply the correct relations against all relations as judged by human experts. Recall is more difficult to compute. Gazetteers and web resources may be used for this purpose.

### Could you mention some resources for working with relation extraction?

Papers With Code has useful links to recent publications on relation classification. GitHub has a topic page on relation classification. Another useful resource is a curated list of papers, tutorials and datasets.

The current state-of-the-art is captured on the NLP-progress page of relation extraction.

Among the useful datasets for training or evaluation are ACE-2005 (7 major relation types) and SemEval-2010 Task 8 (19 relation types). For distant supervision, Riedel or NYT dataset was formed by aligning Freebase relations with New York Times corpus. There's also Google Distant Supervision (GIDS) dataset and FewRel. TACRED is a large dataset containing 41 relation types from newswire and web text.

## Milestones

1998

At the 7th Message Understanding Conference (MUC), the task of extracting relations between entities is considered. Since this is considered as part of template filling, they call it **template relations**. Relations are limited to organizations: employee\_of, product\_of, and location\_of.

Jun
2000

Agichtein and Gravano propose *Snowball*, a semi-supervised approach to generating patterns and extracting relations from a small set of seed relations. At each iteration, it evaluates for quality and keeps only the most reliable patterns and relations.

Feb
2003

Zelenko et al. obtain **shallow parse trees** from text for use in binary relation classification. They use contiguous and sparse subtree kernels to assess similarity of two parse trees. Subsequently, this **kernel-based** approach is followed by other researchers: kernels on dependency parse trees of Culotta and Sorensen (2004); subsequence and shortest dependency path kernels of Bunescu and Mooney (2005); convolutional parse kernels of Zhang et al. (2006); and composite kernels of Choi et al. (2009).

2004

Kambhatla takes a **feature-based** supervised classifier approach to relation extraction. A MaxEnt model is used along with lexical, syntactic and semantic features. Since kernel methods are a generalization of feature-based algorithms, Zhao and Grishman (2005) extend Kambhatla's work by including more syntactic features using kernels, then use SVM to pick out the most suitable features.

Jun
2005

Since binary classifiers have been well studied, McDonald et al. cast the problem of extracting **higher-order relations** into many binary relations. This also makes the data less sparse and eases computation. Binary relations are represented as a graph, from which cliques are extracted. They find that probabilistic cliques perform better than maximal cliques. The figure corresponds to some binary relations extracted for the sentence "John and Jane are CEOs at Inc. Corp. and Biz. Corp. respectively."

Jan
2007

Banko et al. propose **Open Information Extraction** along with an implementation that they call *TextRunner*. In an unsupervised manner, the system is able to extract relations without any human input. Each tuple is assigned a probability and indexed for efficient information retrieval. TextRunner has three components: self-supervised learner, single-pass extractor, and redundancy-based assessor.

Aug
2009

Mintz et al. propose **distant supervision** to avoid the cost of producing hand-annotated corpus. Using entity pairs that appear in Freebase, they find all sentences in which each pair occurs in unlabelled text, extract textual features and train a relation classifier. The include both lexical and syntactic features. They note that syntactic features are useful when patterns are nearby in the dependency tree but distant in terms of words. In the early 2010s, distant supervision becomes an active area of research.

Aug
2014

Neural networks and word embeddings were first explored by Collobert et al. (2011) for a number of NLP tasks. Zeng et al. apply **word embeddings** and **Convolutional Neural Network (CNN)** to relation classification. They treat relation classification as a multi-class classification problem. Lexical features include the entities, their surrounding tokens, and WordNet hypernyms. CNN is used to extract sentence level features, for which each token is represented as *word features (WF)* and *position features (PF)*.

Jul
2015

Dependency shortest path and subtrees have been shown to be effective for relation classification. Liu et al. propose a recursive neural network to model the dependency subtrees, and a convolutional neural network to capture the most important features on the shortest path.

Oct
2015

Song et al. present *PKDE4J*, a framework for dictionary-based entity extraction and rule-based relation extraction. Primarily meant for biomedical field, they report F-measures of 85% for entity extraction and 81% for relation extraction. The RE algorithm uses dependency parse trees, which are analyzed to extract heuristic rules. They come up with 17 rules that can be applied to discern relations. Examples of rules include verb in dependency path, nominalization, negation, active/passive voice, entity order, etc.

Aug
2016

Miwa and Bansal propose to **jointly model the tasks of NER and RE**. A BiLSTM is used on word sequences to obtain the named entities. Another BiLSTM is used on dependency tree structures to obtain the relations. They also find that shortest path dependency tree performs better than subtrees of full trees.

May
2019

Wu and He apply **BERT pre-trained language model** to relation extraction. They call their model *R-BERT*. Named entities are identified beforehand and are delimited with special tokens. Since an entity can span multiple tokens, their start/end hidden token representations are averaged. The output is a softmax layer with cross-entropy as the loss function. On SemEval-2010 Task 8, R-BERT achieves state-of-the-art Macro-F1 score of 89.25. Other BERT-based models learn NER and RE jointly, or rely on topological features of an entity pair graph.

Relation Extraction


# React Native

## Summary

Traditionally, *native mobile apps* have been developed in specific languages that call platform-specific APIs. For example, Objective-C and Swift for iOS app development; Java and Kotlin for Android app development. This means that developers who wish to release their app on multiple platforms will have to implement it in different languages.

To avoid this duplication, *hybrid apps* came along. The app was implemented using web technologies but instead of running it inside a web browser, it was wrapped and distributed as an app. But it had performance limitations.

React Native enables web developers write code once, deploy on any mobile platform and also use the platform's native API. **React Native** is a platform to build native mobile apps using JavaScript and React.

## Discussion

### As a developer, why should I adopt React Native?

Since React Native allows developers maintain a single codebase even when targeting multiple mobile platforms, development work is considerably reduced. Code can be reused across platforms. If you're a web developer new to mobile app development, there's no need to learn a new language. You can reuse your current web programming skills and apply them to the mobile app world. Your knowledge of HTML, CSS and JS will be useful, although you'll be applying them in a different form in React Native.

React Native uses ReactJS, which is a JS library invented and later open sourced by Facebook. ReactJS itself has been gaining adoption because it's easy to learn for a JS programmer. It's performant due to the use of *virtual DOM*. The recommended syntax is ES6 and JSX. ES6 brings simplicity and readability to JS code. JSX is a combination of XML and JS to build reusable component-based UI.

### How is React Native different from ReactJS?

React Native is a framework whereas ReactJS is a library. In ReactJS projects, we typically use a bundler such as *Webpack* to bundle necessary JS files for use in a browser. In React Native, we need only a single command to start a new project. All basic modules required for the project will be installed. We also need to install Android Studio for Android development and Xcode for iOS development.

In ReactJS, we are allowed to use HTML tags. In React Native, we create UI components using React Native components that are specified using JSX syntax. These components are mapped to native UI components. Thus, we can't reuse any ReactJS libraries that render HTML, SVG or Canvas.

In ReactJS, styling is done using CSS, like in any web app. In React Native, styling is done using JS objects. For component layout, React Native's *Flexbox* can be used. CSS animations are also replaced with the *Animated* API.

### How does React Native work under the hood?

Between native and JavaScript worlds is a bridge (implemented in C++) through which data flows. Native code can call JS code and vice versa. To pass data between the two, data is serialized.

For example, a UI event is captured as a native event but the processing for this is done in JavaScript. The result is serialized and sent over the bridge to the native world. The native world deserializes the response, does any necessary processing and updates the UI.

### What are some useful developer features of React Native?

React Native offers the following:

    + **Hot Reloading**: Small changes to your app will be immediately visible during development. If business logic is changed, Live Reload can be used instead.
    + **Debugging**: Chrome Dev Tools can be used for debugging your app. In fact, your debugging skills from the web world can be applied here.
    + **Publishing**: Publishing your app is easy using CodePush, now part of Visual Studio App Center.
    + **Device Access**: React Native gets access to camera, sensors, contacts, geolocation, etc.
    + **Declarative**: UI components are written in a declarative manner. Component-based architecture also means that one developer need not worry about breaking another's work.
    + **Animations**: For performance, these are serialized and sent to the native driver. They run independent of the JS event loop.
    + **Native Code**: Native code and React Native code can coexist. This is important because React Native APIs may not support all native functionality.

### How does React Native compare against platforms in terms of performance?

Since React Native is regularly being improved with each release, we can except better performance than what we state below.

A comparison of React Native against iOS native programming using Swift showed comparable performance of CPU usage for list views. When resizing maps, Swift was better by 10% but React Native uses far less memory here. For GPU usage, Swift outperforms marginally except for list views.

React Native apps can leak memory. Therefore, `FlatList`, `SectionList`, or `VirtualizedList` could be used rather than `ListView`. The communication between native and JS runtimes over the bridge is via message queues. This is also a performance bottleneck. For better performance, ReactNavigation is recommended over Navigator component.

When comparing against Ionic platform, React Native outperforms Ionic across metrics such as CPU usage, memory usage, power consumption and list scrolling.

### Are there real-world examples of who's using React Native?

Facebook and Instagram use React Native. Other companies or products using it include Bloomberg, Pinterest, Skype, Tesla, Uber, Walmart, Wix, Discord, Gyroscope, SoundCloud Pulse, Tencent QQ, Vogue, and many more.

Walmart moved to React Native because it was hard to find skilled developers for native development. They used an incremental approach by migrating parts of their code to React Native. They were able to reuse 95% of their code between iOS and Android. They could reuse business logic with their web apps as well. They could deliver quick updates from their server rather than an app store.

Bloomberg developed their app in half the time using React Native. They were also able to push updates, do A/B testing and iterate quickly.

Airbnb engineers write code for the web, iOS and Android. With React Native, they stated,

> It's now feasible for us to have the same engineer skilled in JavaScript and React write the feature for all three platforms.

However, in June 2018, Airbnb decided to move away from React Native and back to native development due to technical and organizational challenges.

### What backend should I use for my React Native app?

React Native provides UI components. However, the React Native ecosystem is vast. There are frameworks/libraries for AR/VR, various editors and IDEs that support React Native, local databases (client-side storage), performance monitoring tools, CI/CD tools, authentication libraries, deep linking libraries, UI frameworks, and more.

Specifically for backends, **Mobile Backend as a Service (MBaaS)** is now available. Some options include RN Firebase, Baqend, RN Back, Feather and Graph Cool. These services make it easy for developers to build their React Native apps.

The more traditional approach is to build and manage your own backend. Some developers choose Node.js or Express.js because these are based on JavaScript that they're already using to build React Native UI. This can be paired with a database such as Firebase, MySQL, or MongoDB. Another option is to use Django with GraphQL. Even WordPress can be used, especially if the app is content driven. These are merely some examples. Developers can use any backend that suits their expertise and app requirements.

### Could you point me to some useful React Native developer resources?

Here are some useful resources:

    + Expo is a free and open source toolchain for your React Native projects. Expo also has a collection of apps developed and shared by others. The easiest way to create a new app is to use the create-react-native-app codebase.
    + If you wish learn by studying app code written by others, React Active News maintains a curated list of open source React Native apps.
    + React.parts is a place to find reusable components for React Native.
    + Visual Studio App Center is a useful tool to build and release your app.
    + Use React Navigation for routing and navigation in React Native apps.
    + React Native provides only the UI but here's a great selection of tools to complement React Native.

## Milestones

2011

At Facebook, Jordan Walke and his team release ReactJS, a JavaScript library that brings a new way of rendering pages with more responsive user interactions. A web page can be built from a hierarchy of UI components.

2013

React Native starts as an internal hackathon project within Facebook. Meanwhile, ReactJS is open sourced.

Mar
2015

Facebook open sources React Native for iOS on GitHub. The release for Android comes in September.

2016

Microsoft and Samsung commit to adopt React Native for Windows and Tizen.

2017

React Native sees a number of improvements over the year: better navigation, smoother list rendering, more performant animations, and more.

React Native


# Web of Things

## Summary

Web of Things (WoT) is a set of building blocks that seeks to make the Internet of Things (IoT) more interoperable and usable. It simplifies application development (including cross-domain applications) by adopting the web paradigm. Web developers will have a low barrier to entry when programming for the IoT.

The key concepts of WoT include Thing Description, Thing Model, Interaction Model, Hypermedia Controls, Protocol Bindings, Profiles, Discovery and Binding Templates. IoT devices (aka Things) are treated as web resources, which makes WoT a Resource-Oriented Architecture (ROA).

WoT is standardized by the W3C. There are developer tools and implementations. As of December 2023, widespread industry adoption of WoT is yet to happen. Highly resource-constrained devices that can't run a web stack will not be able to adopt WoT.

## Discussion

### Why do we need the Web of Things (WoT)?

The IoT ecosystem is fragmented. Applications or devices from different vendors don't talk to one another due to differing data models. Consumers need to use multiple mobile apps to interact with their IoT devices. While IoT has managed to network different devices via various connectivity protocols (Zigbee, IEEE 802.15.4, NB-IoT, Thread, etc.), there's a disconnect at the application layer.

For developers, this disconnect translates to more effort integrating new devices and services. Each application exposes its own APIs. This results in tight coupling between clients and service providers. It's more effort maintaining and evolving these services.

WoT brings interoperability at the application layer with a unifying data model. It reuses the web paradigm. IoT devices can be treated as web resources. Just as documents on the web are interlinked and easily navigated, Things can be linked, discovered, queried and acted upon. Mature web standards such as REST, HTTP, JSON, AJAX and URI can be used to achieve this.   This means that web developers can become IoT developers. They can create reusable IoT building blocks rather than custom proprietary implementations that work for limited use cases.

### What integration patterns does WoT cover?

An IoT device can directly expose a WoT API. This is the simplest integration pattern. It's also challenging from a security perspective or if the device is behind a firewall. For more resource-constrained devices running LPWAN protocols, direct access is difficult. They would connect to the cloud via a gateway, which exposes the WoT API. When devices spread over a large area need to cooperate, they would connect to the cloud in different ways and the cloud exposes the WoT API.

Let's consider specific use cases. A remote controller connects directly to an electrical appliance in a trusted environment. Similarly, a sensor acting as a control agent connects to an electrical appliance. A remote control outside a trusted environment connects to a gateway or a edge device which then connects to an electrical appliance. Connected devices are mapped to digital twins that can be accessed via a client device. A device can be controlled via its digital twin in the cloud. These various integration patterns can be combined through system integration.

### What's the architecture of WoT?

WoT standardizes a layered architecture of four layers (lower to higher): Access, Find, Share and Compose. The protocols or techniques used at each of these layers are already widely used on the web. These four layers can't be mapped to the OSI model, nor are they strictly defined at the interfaces. They're really a collection of services to ease the development of IoT solutions.

At the access layer, solution architects have to think about resource, representation and interface designs. They should also define how resources are interlinked. At the find layer, web clients can discover root URLs, the syntax and semantics of interacting with Things. At the compose layer, tools such as Node-RED and IFTTT can help create mashups.

### What are Thing Description (TD) and Thing Model (TM) in WoT?

TD is something like the business card of the Thing. It reveals everything about the Thing. It informs the protocol, data encoding, data structure, and security mechanism used by the Thing. TD itself is in JSON-LD format and is exposed by the Thing or can be discovered by consumers from a Thing Description Directory (TDD).

In object-oriented programming, objects are instantiated from classes. Likewise, a TD can be seen as an instantiation of a TM. A TM is a logical description of a Thing's interface and interactions. However, it doesn't contain instance-specific information such as an IP address, serial number of GPS location. A TM can include security details if those are applicable for all instances of that TM.

Both TD and TM are represented and serialized in JSON-LD format. Whereas a TD can be validated against its TM, a TM can't be validated.

### What's the WoT interaction model?

Apart from links, a Thing may expose three types of interaction affordances:

    + **Properties**: Property is a state of the Thing. State may be read-only or read-write. Properties can be made observable. Sensor values, stateful actuators, configuration, status and computation results are examples.
    + **Actions**: Action invokes a function of the Thing. Action can be used to update one or more properties including read-only ones.
    + **Events**: Event is used to asynchronously send data from the Thing to a consumer. Focus is on state transitions rather than the state itself. Examples include alarms or samples of a time series.Like documents on the web, WoT also links and forms. These are called **hypermedia controls**. Links are used to discover and interlink Things. Forms enable more complex operations than what's possible by simply dereferencing a URI.

### What are protocol bindings in WoT?

WoT's abstractions make it protocol agnostic. It doesn't matter if a Thing uses MQTT, CoAP, Modbus or any other connectivity protocol. WoT's interaction model unifies all these so that applications talk in terms of properties, actions and events. But abstractions have to be translated into protocol actions. This is provided by **protocol bindings**. For a door handle for example, protocol binding tells how to open/close the door at the level of knob or lever.

W3C has published a non-normative document called **WoT Binding Templates**. This gives blueprints on how to write TDs for different IoT platforms or standards. This includes protocol-specific metadata, payload formats, and usage in specific IoT platforms. The consumer of a TD would implement the template, that is, the protocol stack, media type encoder/decoder and platform stack.

### Who has implemented WoT?

W3C maintains a list of developer resources. This include tools, implementations, TD directories and WoT middleware. For example, Eclipse Thingweb is a Node.js implementation to expose and consume TD. From other sources, there are implementations in Python, Java, Rust and Dart. Among the TD directories are TinyIoT Thing Directory and WoTHive.  Major WoT deployments during 2012-2021 have been documented.

Krellian Ltd. offers WebThings Gateway and WebThings Framework. WebThings was initially developed at Mozilla. However, its API differs from W3C specifications in many ways.

The sayWoT! platform from evosoft (a Siemens subsidiary) gives web and cloud developers an easy way to develop IoT solutions. One study compared many WoT platforms including WoT-SDN, HomeWeb, KNX-WoT, EXIP, WTIF, SOCRADES, WoTKit, µWoTO, and more.

WoT is being leveraged to create digital twins. WoTwins and Eclipse Ditto with WoT integration are examples of this. Ortiz et al. used WoT TD effectively in real-time IoT data processing in smart ports use case. WoTemu is an emulation framework for WoT edge architecture.

### What standards cover WoT?

The W3C is standardizing WoT. The following are the main normative specifications:

    + WoT Architecture 1.1 (Recommendation)
    + WoT Thing Description 1.1 (Recommendation)
    + WoT Discovery (Recommendation)
    + WoT Profile (Working Draft)Informative specifications include WoT Scripting API, WoT Binding Templates, WoT Security and Privacy Guidelines, and WoT Use Cases and Requirements.

Beginners can start at the W3C WoT webpage for latest updates, community groups, documentation and tooling.

At the IETF, there's a draft titled *Guidance on RESTful Design for Internet of Things Systems*. This is relevant to WoT.

### What are some limitations of WoT?

WoT depends on the web stack. Hence, it's not suited for very low-power devices or mesh deployments.

**Matter** protocol, known earlier as Project CHIP, is an alternative to WoT. This is promoted by the Connectivity Standards Alliance (CSA), formerly called Zigbee Alliance. Matter is based on Thread, IPv6 and Dotdot. While Matter is not web friendly like WoT, it appears to have better industry traction.  However, Matter devices that expose WoT TDs can talk to WoT devices.

There's a claim that WoT hasn't adequately addressed security, privacy and data sharing issues. This is especially important when IoT devices are directly exposed to the web. Devices are energy inefficient since they're always on. They're vulnerable to DoS attacks.

WoT alone can't solve complex problems such as optimize workflows across many IoT devices or applications. Hypermedea and EnvGuard are two approaches to solve this. Larian et al. compared many WoT platforms. They noted that current IoT middleware and WoT resource discovery need to be improved. Legacy systems would require custom code to interface to the WoT architecture.

## Milestones

Nov
2007

Wilde uses the term "Web of Things" in a paper titled *Putting Things to REST*. He makes the case for treating a Thing (such as a sensor) as a web resource. It could then be accessed via RESTful calls rather than the more restrictive SOAP/WSDL API calls. Web concepts of URI, HTTP, HTML, XML and loosely coupling can be applied effectively towards Things.

2011

Guinard publishes his Doctor of Science dissertation in the field of Web of Things. In 2016, he co-authors (with Trifa) a book titled *Building the Web of Things*. Guinard sees WoT as

> A refinement of the Internet of Things (IoT) by integrating smart things not only into the Internet (the network), but into the Web (the application layer).

Jul
2013

**Web of Things Community Group** is created. Subsequently in 2014, a workshop is held (June) and an Interest Group is formed (November).

Dec
2016

Following the first in-person meeting and a WoT Plugfest in 2015, the **W3C WoT Working Group** is formed. It's aim is to produce two normative specifications (Architecture, Thing Description) and two informative specifications (Scripting API, Binding Templates).

Jun
2018

From the Eclipse Foundation, the first commit on GitHub is made for the **Eclipse Thingweb** project. The project aims to provide Node.js components and tools for developers to build IoT systems that conform to W3C WoT standards. The project releases v0.5.0 in October.

Apr
2020

W3C publishes WoT Architecture and WoT Thing Description as separate **W3C Recommendation** documents.

Jul
2022

Tzavaras et al. propose using **OpenAPI** descriptions and ontologies to bring Things closer to the world of Semantic Web. Thing Descriptions can be created in OpenAPI while also conforming to W3C WoT architecture. They argue that OpenAPI is already a mature standard. It provides a uniform way to interact with web services and Things.

Nov
2022

Markus Reigl at Siemens comments that WoT will do for IoT what HTML did for the WWW in the 1990s. TD is not a mere concept. It leads to executable software code. He predicts IoT standardization will gain momentum.

Dec
2023

W3C publishes WoT Architecture 1.1 and WoT Thing Description 1.1 as W3C Recommendation documents. In addition, WoT Discover is also published as a W3C Recommendation.

Web of Things


# TensorFlow

## Summary

TensorFlow is an open source software library for numerical computation using **data flow graphs**. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) that flow between them. This dataflow paradigm enables parallelism, distributed execution, optimal compilation and portability.

The typical use of TensorFlow is for Machine Learning (ML), particularly Deep Learning (DL) that uses large scale multi-layered neural networks. More specifically, it's best for classification, perception, understanding, discovery, prediction and creation.

TensorFlow was originally developed by researchers and engineers working on the Google Brain team within Google's Machine Intelligence Research organization for ML/DL research. The system is general enough to be applicable in a wide variety of other domains as well.

## Discussion

### For which use cases is TensorFlow best suited?

TensorFlow can be used in any domain where ML/DL can be employed. It can also be used in other forms of AI, including reinforcement learning and logistic regression. On mobile devices, applications include speech recognition, image recognition, object localization, gesture recognition, optical character recognition, translation, text classification, voice synthesis, and more.

Some of the areas are:

    + **Voice/Speech Recognition**: For voice-based interfaces as popularized by Apple Siri, Amazon Alexa or Microsoft Cortana. For sentiment analysis in CRM. For flaw detection (noise analysis) in industrial systems.
    + **Text-Based Applications**: For sentimental analysis (CRM, Social Media), threat detection (Social Media, Government) and fraud detection (Insurance, Finance). For machine translation such as with Google Translate. For text summarization using sequence-to-sequence learning. For language detection. For automated email replies such as with Google SmartReply.
    + **Image Recognition**: For face recognition, image search, machine vision and photo clustering. For object classification and identification within larger images. For cancer detection in medical applications.
    + **Time-Series Analysis**: For forecasting. For customer recommendations. For risk detection, predictive analytics and resource planning.
    + **Video Detection**: For motion detection in gaming and security systems. For large-scale video understanding.

### Could you name some applications where TensorFlow is being used?

TensorFlow is being used by Google in following areas:

    + RankBrain: Google search engine.
    + SmartReply: Deep LSTM model to automatically generate email responses.
    + Massively Multitask Networks for Drug Discovery: A deep neural network model for identifying promising drug candidates.
    + On-Device Computer Vision for OCR - On-device computer vision model to do optical character recognition to enable real-time translation.
    + Retinal imaging - Early detection of diabetic retinopathy using deep neural network of 26 layers.
    + SyntaxNet - Built for Natural Language Understanding (NLU), this is based on TensorFlow and open sourced by Google in 2016.Outside Google, we mention some known real-world examples. Mozilla uses TensorFlow for speech recognition. UK supermarket Ocado uses it for route planning for its robots, demand forecasting, and product recommendations. A Japanese farmer has used it to classify cucumbers based on shape, length and level of distortion. As an experiment, Intel used TensorFlow on traffic videos for pedestrian detection.

Further examples were noted at the TensorFlow Developer Summit, 2018.

### Which platforms and languages support TensorFlow?

TensorFlow is available on 64-bit Linux, macOS, Windows and also on the mobile computing platforms like Android and iOS. Google has announced a software stack specifically for Android development called TensorFlow Lite.

TensorFlow has official APIs available in the following languages: Python, JavaScript, C++, Java, Go, Swift. Python API is recommended. Bindings in other languages are available from community: C#, Haskell, Julia, Ruby, Rust, Scala. There's also C++ API reference for TensorFlow Serving. R's `tensorflow` package provides access to the complete TensorFlow API from within R.

Nvidia's **TensorRT**, a Programmable Inference Accelerator, allows you to optimize your models for inference by lowering precision and thereby reducing latency.

### How is TensorFlow different from other ML/DL platforms?

TensorFlow is relatively painless to setup. With its growing community adoption, it offers a healthy ecosystem of updates, tutorials and example code. It can run on a variety of hardware. It's cross platform. It has APIs or bindings in many popular programming languages. It supports GPU acceleration. Through TensorBoard, you get an intuitive view of your computation pipeline. Keras, a DL library, can run on TensorFlow. However, it's been criticized for being more complex and slower than alternative frameworks.

Created in 2007, **Theano** is one of the first DL frameworks but it's been perceived as too low-level. Support for Theano is also ending. Written in Lua, **Torch** is meant for GPUs. It's Python port released by Facebook, called **PyTorch**, is popular for analyzing unstructured data. It's developer friendly and memory efficient. **Caffe2** does well for modeling convolutional neural networks. **Apache MXNet**, along with its simplified DL interface called **Gluon**, is supported by Amazon and Microsoft. Microsoft also has **Microsoft Cognitive Toolkit (CNTK)** that can handle large datasets. For Java and Scala programmers, there's **Deeplearning4j**.

### Which are the tools closely related to TensorFlow?

The following are closely associated with or variants of TensorFlow:

    + **TensorFlow Lite**: Enables low-latency inferences on mobile and embedded devices.
    + **TensorFlow Mobile**: To use TensorFlow from within iOS or Android mobile apps, where TensorFlow Lite cannot be used.
    + **TensorFlow Serving**: A high performance, open source serving system for machine learning models, designed for production environments and optimized for TensorFlow.
    + **TensorLayer**: Provides popular DL and RL modules that can be easily customized and assembled for tackling real-world machine learning problems.
    + **TensorFlow Hub**: A library for the publication, discovery, and consumption of reusable parts of machine learning models.
    + **TensorFlow Model Analysis**: A library for evaluating TensorFlow models.
    + **TensorFlow Debugger**: Allows us to view the internal structure and states of running TensorFlow graphs during training and inference.
    + **TensorFlow Playground**: A browser-based interface for beginners to tinker with neural networks. Written in TypeScript and D3.js. Doesn't actually use TensorFlow.
    + **TensorFlow.js**: Build and train models entirely in the browser or Node.js runtime.
    + **TensorBoard**: A suite of visualization tools that helps to understand, debug, and optimize TensorFlow programs.
    + **TensorFlow Transform**: A library for preprocessing data with TensorFlow.

### What's the architecture of TensorFlow?

TensorFlow can be deployed across platforms, details of which are abstracted away from higher layers. The core itself is implemented in C++ and exposes its features via APIs in many languages, with Python being the most recommended.

Above these language APIs, is the **Layers** API that offers commonly used layers in deep learning models. To read data, **Datasets** API is the recommended way and it creates input pipelines. With **Estimators**, we can create custom models or bring in models pre-made for common ML tasks.

**XLA (Accelerated Linear Algebra)** is a domain-specific compiler for linear algebra that optimizes TensorFlow computations. If offers improvements in speed, memory usage, and portability on server and mobile platforms.

### Could you explain how TensorFlow's data graph works?

TensorFlow uses a **dataflow graph**, which is a common programming model for parallel computing. Graph nodes represent **operations** and edges represent data consumed or produced by the nodes. Edges are called **tensors** that carry data. In the example figure, we show five graph nodes: `a` and `b` are placeholders to accept inputs; `c`, `d` and `e` are simple arithmetic operations.

In TensorFlow 1.x, when a graph is created, tensors don't contain the results of operations. The graph is evaluated through **sessions**, which encapsulate the TensorFlow runtime. However, with **eager execution**, operations are evaluated immediately instead of building a graph for later execution. This is useful for debugging and iterating quickly on small models or data.

For ingesting data into the graph, **placeholders** can be used for the simplest cases but otherwise, **datasets** should be preferred. To train models, **layers** are used to modify values in the graph.

To simplify usage, high-level API called **estimators** should be used. They encapsulate training, evaluation, prediction and export for serving. Estimators themselves are built on layers and build the graph for you.

### How is TensorFlow 2.0 different from TensorFlow 1.x?

It makes sense to write any new code in TensorFlow 2.0. Existing 1.x code can be migrated to 2.0. The recommended path is to move to TensorFlow 1.14 and then to 2.0. Compatibility module `tf.compat` should help.

Here are the key changes in TensorFlow 2.0:

    + **API Cleanup**: Many APIs are removed or moved. For example, `absl-py` package replaces `tf.app`, `tf.flags`, and `tf.logging`. Main namespace `tf.*` is cleaned up by moving some items into subpackages such as `tf.math`. Examples of new modules are `tf.summary`, `tf.keras.metrics`, and `tf.keras.optimizers`.
    + **Eager Execution**: Like Python, eager execution is the default behaviour. Code execute in order, making `tf.control_dependencies()` redundant.
    + **No More Globals**: We need to keep track of variables. Untracked `tf.Variable` will get garbage collected.
    + **Functions, Not Sessions**: Functions are more familiar to developers. Although `session.run()` is gone, for efficiency and JIT compilation, `tf.function()` decorator can be used. This automatically invokes *AutoGraph* to convert Python constructs into TensorFlow graph equivalents. Functions can be shared and reused.

## Milestones

2011

Google Brain invents **DistBelief**, a framework to train large models for machine learning. DistBelief can make use of computing clusters of thousands of machines for accelerated training. The framework manages details of parallelism (multithreading, message passing), synchronization and communication. Compared to MapReduce, DistBelief is better at deep network training. Compared to GraphLab, DistBelief is better at structured graphs.

Nov
2015

Under Apache 2.0 licensing, Google open sources TensorFlow, which is Google Brain's second-generation machine learning system. While other open source ML frameworks exist (Caffe, Theano, Torch), Google's competence in ML is supposedly 5-7 years ahead of the rest. However, Google doesn't open source algorithms that run on TensorFlow, not its advanced hardware infrastructure.

Apr
2016

Version 0.8 of TensorFlow is released. It comes with distributed training support. Powered by gRPC, models can be trained on hundreds of machines in parallel. For example, Inception image classification network was trained using 100 GPUs with an overall speedup of 56x compared to a single GPU. More generally, the system can map the dataflow graph onto heterogeneous devices (multi-core CPUs, general-purpose GPUs, mobile processors) in the available processes.

May
2016

Google announced that it's been using **Tensor Processing Unit (TPU)**, a custom ASIC built specifically for machine learning and tailored for TensorFlow.

Jun
2016

TensorFlow v0.9 is released with support for iOS and Raspberry Pi. Android support has been around from the beginning.

Feb
2017

Version 1.0 of TensorFlow is released. The API is in Python but there's also experimental APIs in Java and Go.

Nov
2017

Google releases a preview of **TensorFlow Lite** for mobile and embedded devices. This enables low-latency inferences for on-device ML models. In future, this should be preferred over **TensorFlow Mobile**. With TensorFlow 1.4, we can build models using high-level **Keras** API. Keras, which was previously in `tf.contrib.keras`, is now the core package `tf.keras`.

Sep
2019

TensorFlow 2.0 is released following an alpha release in June. It improves workflows for both production and experimentation. It promises better performance on GPU acceleration.

TensorFlow


# Wi-Fi Calling

## Summary

Wi-Fi Calling is a technology that allows users to make or receive voice calls via a local Wi-Fi hotspot rather than via their mobile network operator's cellular radio connection. Voice calls are thus carried over the Internet, implying that Wi-Fi Calling relies on VoIP. However, unlike other VoIP services such as Skype or Viber, Wi-Fi Calling gives operators more control.

Wi-Fi Calling is possible only if the operator supports it, user's phone has the feature and user has enabled it. Once enabled, whether a voice call uses the cellular radio link or Wi-Fi link is almost transparent to the user. With cellular networks going all IP and offering VoLTE, Wi-Fi Calling has become practical and necessary in a competitive market.

Wi-Fi Calling is also called *Voice over Wi-Fi (VoWi-Fi)*.

## Discussion

### In what scenarios can Wi-Fi Calling be useful to have?

In places where cellular coverage is poor, such as in rural residences, concrete indoors, basements, or underground train stations, users will not be able to make or receive voice calls. In these scenarios, the presence of a local Wi-Fi network can serve as the "last-mile" connectivity to the user. Wi-Fi can therefore complement the cellular network in places where the latter's coverage is poor.

For example, a user could be having an active voice call via the cellular network and suddenly enters a building with poor coverage. Without Wi-Fi Calling, the call might get dropped. With Wi-Fi Calling, the call can be seamlessly handed over to the Wi-Fi network without even the user noticing it. Astute users may notice that their call is on Wi-Fi since smartphones may indicate this via an icon. More importantly, user intervention is not required to switch between cellular and Wi-Fi. Such seamless handover has become possible because cellular network's IP and packet switching: VoWi-Fi can be handed off to VoLTE, and vice versa.

### Isn't Wi-Fi Calling the same as Skype, Viber or WhatsApp voice calls?

Many smartphone apps allow voice (and even video) calls over the Internet. They are based on VoIP technology. We normally call them over-the-top (OTT) services since they merely use the phone's data connection and operators bill for data usage and not for the service itself. However, many of these systems require both parties to have the same app installed. Even when this constraint is removed, the service is controlled by the app provider.

Wi-Fi Calling gives cellular operators greater control. Driven by competition from OTT services, Wi-Fi Calling gives operators an opportunity to regain market share for voice calls. Voice packets are carried securely over IP to the operator's core network, thus allowing the operator to reuse many resources and procedures already in place for VoIP calls. Likewise, messages and video–*Video over LTE (ViLTE)*–can also be carried over Wi-Fi.

From an architectural perspective, Wi-Fi Calling is served by operator's IP Multimedia Subsystem (IMS), whereas Skype calls are routed out of the operator's network into the Internet.

### Isn't Wi-Fi Calling the same as Wi-Fi Offload?

Not exactly. Wi-Fi Calling can be seen as a form of offload but they have different motivations. Wi-Fi Offload came about to ease network congestion and improve QoS for users in high-density areas. The offload is transparent for users whose devices are authenticated via EAP-SIM/AKA.

Wi-Fi Calling is in response to OTT services stealing revenue from mobile operators. Even VoLTE was deployed by operators, voice calls couldn't be made over Wi-Fi and OTT services was what users used when they had access to Wi-Fi. Wi-Fi Calling aims to overcome this problem.

### What are the possible benefits of Wi-Fi Calling?

For subscribers, benefits include seamless connectivity and mobility between cellular and Wi-Fi. The selection is automatic and transparent to users. Data is protected using IPSec from mobile to core network, along with traditional SIM-based authentication. Users can potentially lower their monthly bills through service bundles and reduced roaming charges. Sometimes calling home from another country could be free depending on the subscribed plan and operator.

Moreover, the user's phone will have a single call log (likewise, for message log). The default dialler can be used along with all saved contacts. Those receiving the call will see the caller's usual phone number. These are not possible with a third-party installed app.

For operators, Wi-Fi complements cellular coverage and capacity. T-Mobile was one of the early adopters because it had poor indoor coverage. Wi-Fi Network performance is optimized by allowing bandwidth-intensive traffic to be offloaded to Wi-Fi when so required. All their IMS-based services can now be extended to Wi-Fi access rather than losing out to OTT app/service providers.

### How does the network architecture change for Wi-Fi Calling?

Two network functions are involved:

    + **Evolved Packet Data Gateway (ePDG)**: Serves an untrusted Wi-Fi network. An IPSec tunnel protects data between mobile and ePDG, from where it goes to Packet Gateway (PGW). The mobile needs an update with an IPsec client. No changes are needed for the access point.
    + **Trusted Wireless Access Gateway (TWAG)**: Serves a trusted Wi-Fi network, which is typically under operator's control. In this case, data between mobile and TWAG is encrypted at radio access and IPSec is not used. From TWAG, data goes to PGW. No changes are needed for the mobile but Wi-Fi access point needs to be updated.If the network is not an Evolved Packet Core (EPC), then Tunnel Termination Gateway (TTG) is used instead of ePDG; Wireless Access Gateway (WAG) is used instead of TWAG; GGSN is used instead of PGW.

The untrusted mode is often used for Wi-Fi Calling, since public hotspots can be used without updating the access point. It's the operator who decides if a non-3GPP access can be considered trusted.

### How is an end-user device authenticated for Voice over Wi-Fi service?

Within the network, *3GPP AAA Server* is used to authenticate end devices. Authentication is based on SIM and the usual network functions located in the Home Subscriber Server (HSS). 3GPP AAA Server does not maintain a separate database and relies on the HSS.

Vendors who sell AAA servers usually give the ability to do authentication of devices that don't have SIM. For legacy networks, they can interface with HLR rather than HSS. They support AAA protocols such as RADIUS and Diameter. They support various EAP methods including TLS, PEAP and CHAP.

### What are the 3GPP standards covering Wi-Fi Calling?

Documents that specify "non-3GPP access" are applicable to Wi-Fi Calling. The following are some relevant documents (non-exhaustive list):

    + TR 22.814: Location services
    + TR 22.912: Study into network selection requirements
    + TS 23.402: Architectural enhancements
    + TS 24.234: 3GPP-WLAN interworking: WLAN UE to network protocols, Stage 3
    + TS 24.302: Access to EPC, Stage 3
    + TS 29.273: 3GPP EPS AAA interfaces
    + TS 33.402: System Architecture Evolution: security aspects
    + TR 33.822: Security aspects for inter-access mobilityIn addition, GSMA has released a list of Permanent Reference Documents on VoWi-Fi.

Wi-Fi Calling is a technology that comes from the cellular world. From Wi-Fi perspective, there's no special IEEE standard that talks about Wi-Fi Calling.

### Are there commercial services offering Wi-Fi Calling?

In June 2016, it was reported that all four major operators in the US support Wi-Fi Calling, with T-Mobile supporting as many as 38 different handsets. In November 2016, there were 40+ operators offering Wi-Fi Calling in 25+ countries. Moreover, even affordable phones or devices without SIMs are supporting Wi-Fi Calling. An operator will normally publish a list of handsets that are supported, which usually includes both Android and iPhone models. In September 2017, it was reported that AT&T has 23 phones and Verizon has 17 phones that support Wi-Fi Calling.

Wi-Fi Calling may involve regulatory approval based on the country's licensing framework. For example, India's TRAI commented in October 2017 that Wi-Fi Calling can be introduced since licensing allows telephony service to be provided independent of the radio access.

### Within enterprises, how can IT teams plan for Wi-Fi Calling?

Some access points have the ability to prioritize voice traffic and this can be repurposed for Wi-Fi Calling. Examples include Aerohive, Aruba, Cisco Aironet and Ruckus. Enterprises can also work with operators to deploy femto/pico cells or distributed antenna systems.

A minimum of 1 Mbps may be needed to support Wi-Fi Calling although Republic Wireless in the US claims 80 kbps is enough to hold a call, although voice quality may suffer. In reality, voice needs just 12 kbps but can scale down to 4.75 kbps.

### How will users be billed for Wi-Fi Calling?

This is completely operator dependent and based on subscriber's current plan. For example, Canada's Rogers says that calls and messages are deducted from airtime and messaging limits. Roaming charges may apply only for international roaming. Verizon Wireless states that a voice call will use about 1 MB/minute of data; a video call will use 6-8 MB/minute. Billing is linked to user's current plan.

### What are some practical issues with Wi-Fi Calling?

Back in 2014, T-Mobile had handoff problems but it was improved later. The service was also not offered by other operators and not supported by most handsets. Even when a handset supports it, operators may not offer the service if the handset has not been purchased from the operator.

Since any Wi-Fi hotspot can be used, including public ones, security is a concern. For this reason, all data over Wi-Fi must be protected and subscriber must be authenticated by the cellular operator. Seamless call continuity across cellular and Wi-Fi could be a problem, particularly when firewalls and VPNs are involved. Some users have reported problems when using Wi-Fi behind corporate firewalls. Likewise, IT teams in enterprises may have the additional task of ensuring Wi-Fi coverage and managing traffic.

Since Wi-Fi Calling often uses public hotspots, there's no QoS control. However, it's argued that in places where cellular has poor coverage, QoS cannot be guaranteed anyway. In addition, QoS on Wi-Fi can often be achieved implicitly because of excess capacity. With the coming of 802.11ac and the ability to prioritize traffic via Wi-Fi Multimedia (WMM), QoS is unlikely to be a problem.

## Milestones

2007

T-Mobile in the US launches something called "HotSpot @ Home". This is based on a technology named *Unlicensed Mobile Access*, which is a commercial name of a 3GPP feature named *Generic Access Network*. GAN operates in the IP layer, which means that access can be via any protocol, not just Wi-Fi. UMA does not take off because of lack of handsets that support it. It also had other operational issues related to interference, handover and configuration setup.

Nov
2011

Republic Wireless, a mobile virtual network operator (MVNO) in the US, rolls out "Hybrid Calling". Calls are primarily on Wi-Fi and cellular will be used as a fallback option. Their General Manager, Brian Dally, states,

> Every other mobile carrier talks about offloading to Wi-Fi, we talk about failing over to cellular.

Sep
2014

T-Mobile introduces Wi-Fi Calling in the US. This comes at the heels of the operator's rollout of VoLTE. Meanwhile, Apple iPhone starts supporting Wi-Fi Calling.

Apr
2015

Sprint introduces Wi-Fi Calling in the US. EE does the same in the UK. Meanwhile, Google gets into telecom by launching *Project Fi*, which allows seamless switching between Wi-Fi and cellular. Google doesn't have its own cellular network but uses those of Sprint, T-Mobile, and US Cellular.

Oct
2015

In the US, AT&T obtains regulatory approval to launch Wi-Fi Calling. By 2016, all four major US operators rollout Wi-Fi Calling nationwide.

Jun
2017

UMA, which may be called first generation Wi-Fi Calling, is decommissioned by T-Mobile in the US.

Nov
2018

Researchers discover security vulnerabilities with Wi-Fi Calling due to various reasons. They propose possible solutions to overcome these.

Wi-Fi Calling


# Design Thinking

## Summary

Design thinking is a problem-solving method used to create practical and creative solutions while addressing the needs of users. The process is extremely user centric as it focuses on understanding the needs of users and ensuring that the solutions created solve users' needs.

It's an iterative process that favours ongoing experimentation until the right solution is found.

## Discussion

### Why is the design thinking process important?

Design thinking helps us to innovate, focus on the user, and ultimately design products that solve real user problems.

The design thinking process can be used in companies to reduce the time it takes to bring a product to the market. Design thinking can significantly reduce the amount of time spent on design and development.

The design thinking process increases return of investment as the products are user-centric, which helps increase user engagement and user retention. It's been seen that a more efficient workflow due to design thinking gave 75% savings in design and development time, 50% reduction in defect rate, and a calculated ROI of more than 300%.

### When and where should the design thinking process be used?

The design thinking process should especially be used when dealing with **human-centric challenges** and **complex challenges**. The design thinking process helps break down complex problems and experiment with multiple solutions. Design thinking can be applied in these contexts: human-centred innovation, problems affecting diverse groups, involving multiple systems, shifting markets and behaviours, complex societal challenges, problems that data can't solve, and more.

A class of problems called **wicked problems** is where design thinking can help. Wicked problems are not easy to define and information about them is confusing. They have many stakeholders and complex interdependencies.

On the contrary, design thinking is perhaps an overkill for obvious problems, especially if they're not human centred. In such cases, traditional problem-solving methods may suffice.

### What are the principles of the design thinking process?

There are some basic principles that guide us in applying design thinking:

    + **The human rule**: All design activity is social because all social innovation will bring us back to the "human-centric point of view".
    + **The ambiguity rule**: Ambiguity is inevitable, and it can't be removed or oversimplified. Experimenting at the limits of your knowledge and ability is crucial in being able to see things differently.
    + **The redesign rule**: While technology and social circumstances may change, basic human needs remain unchanged. So, every solution is essentially a redesign.
    + **The tangibility rule**: Making ideas tangible by creating prototypes allows designers to communicate them effectively.

### What are the typical steps of a design thinking process?

The process involves five steps:

    + **Empathy**: Put yourself in the shoes of the user and look at the challenge from the point of view of the user. Refrain from making assumptions or suggesting answers. Suspend judgements throughout the process.
    + **Define**: Create a challenge statement based on the notes and thoughts you have gained from the empathizing step. Go back to the users and modify the challenge statement based on their inputs. Refer to the challenge statement multiple times throughout the design thinking process.
    + **Ideate**: Come up with ideas to solve the proposed challenge. Put down even the craziest ideas.
    + **Prototype**: Make physical representations of your ideas and solutions. Get an understanding of what the final product may look like, identify design flaws or constraints. Take feedback from users. Improve the prototype through iterations.
    + **Test**: Evaluate the prototype on well-defined criteria.Note that empathy and ideate are divergent steps whereas others are convergent. Divergent means expanding information with alternatives and solutions. Convergent is reducing information or filtering to a suitable solution.

### What are the specific tools to practice design thinking?

Design thinking offers tools for each step of its five-step process. These are summarized in the above figure. These tools offer individuals and teams something concrete to effectively practice design thinking.

New Metrics has enumerated 14 different tools: immersion, visualization, brainstorming, empathy mapping, journey mapping, affinity mapping, rapid iteration, assumption testing, prototyping, design sprints, design criteria, finding the value proposition, and learning launch. They describe each tool briefly and note the benefits. More tools include focus groups, shadowing, concept maps, personas, positioning matrix, minimum viable product, volume model, wireframing, and storyboards.

For specific software tools, we note the following:

    + **Empathize**: Typeform, Zoom, Creatlr
    + **Define**: Smaply, Userforge, MakeMyPersona
    + **Ideate**: SessionLab, Stormboard, IdeaFlip
    + **Prototype**: Boords, Mockingbird, POP
    + **Test**: UserTesting, HotJar, PingPong
    + **Complete Process**: Sprintbase, InVision, Mural, Miro

### What should I keep in mind when applying the design thinking process?

Every designer can use a variation of the design thinking process that suits them and customize it for each challenge. Although distinct steps are defined, design thinking is not a linear process. Rather, it's very much **iterative**. For example, during prototyping we may go back to redefine the problem statement or look for alternative ideas. Every step gives us new information that might help us improve on previous steps.

Adopt Agile methodology. Design thinking is strong on ideation while Scrum is strong on implementation. Combine the two to make a powerful hybrid Agile approach.

While the steps are clear, applying them correctly is not easy. To identify what annoys your clients, ask questions. Empathy means that you should relate to their problems. Open-ended questions will stimulate answers and help identify the problems correctly.

At the end of the process, as a designer, reflect on the way you've gone through the process. Identify areas of improvement or how you could have done things differently. Gather insights on the way you went through the design thinking process.

### What do I do once the prototype is proven to work?

The prototype itself can be said to "work" only after we have submitted it to the clients for feedback. Use this feedback to improve the prototype. Make the actual product after incorporating all the feedback from the prototype.

Gathering feedback itself is an important activity. Present your solution to the client by describing the thought process by which the challenge was solved. Take notes from users and ensure that they are satisfied with the final product. It's important not to defend your product. It's more important to listen to what users have to say and make changes to improve the solution.

Present several versions of the prototype so that users can compare and express what they like and dislike. Consider using *I Like, I Wish, What If* method for gathering feedback. Get feedback from regular users as well as extreme users with highly opinionated views. Be flexible and improvise during testing sessions. Allow users to contribute ideas.

Recognize that prototyping and testing is an iterative process. Be prepared to do this a few times.

### How is design thinking different from user-centred design?

On the surface, both design thinking and user-centred design (UCD) are focused on the needs of users. They have similar processes and methods. They aim for creative or innovative solutions. To elicit greater empathy among designers, UCD has been more recently called human-centred design (HCD).

However, design thinking goes beyond usability. It considers technical feasibility, economic viability, desirability, etc. without losing focus on user needs. While UCD is dominated by usability engineers and focuses on user interfaces, design thinking has a larger scope. Design thinking brings more multi-disciplinary perspectives that can suggest innovative solutions to complex problems. While it borrows from UCD methods, it goes beyond the design discipline.

Some see UCD as a framework and design thinking as a methodology that can be applied within that framework. Others see these as complementary: a team can start with design thinking for initial exploration and later shift to UCD for prototyping and implementation.

### What are some ways to get more ideas?

Design thinking is not about applying standard off-the-shelf solutions. It's about solving difficult problems that typically require creative approaches and innovation. More ideas, the better. Use different techniques such as brainstorming, mind mapping, role plays, storyboarding, etc.

Innovation is not automatic and needs to be fostered. We should create the right mindsets, an open and explorative culture. Designers should combine both logic and imagination. Teams should be cross-disciplinary and collaborative. Work environments must be conductive to innovation.

When framing the problem, think about how the challenge can be solved in a certain place or scenario. For example, think about how one of your ideas would function differently in a setting such as a kitchen.

Write down even ideas that may not work. Further research and prototyping might help refine it. Moreover, during the prototyping and testing steps, current ideas can spark new ideas.

## Milestones

Sep
1962

*The Conference on Systematic and Intuitive Methods in Engineering, Industrial Design, Architecture and Communications* is held in London. It explores design processes and new design methods. Although the birth of design methodology can be traced to Zwicky's *Morphological Method* (1948), it's this conference that recognizes design methodology as a field of academic study.

1966

The term **Design Science** is introduced. This shows that the predominant approach is to find "a single rationalised method, based on formal languages and theories".

1969

Herbert A. Simon, a Nobel Prize laureate and cognitive scientist, mentions the design thinking process in his book *The Sciences of the Artificial* and further contributes ideas that are now known as the principles of design thinking.

1970

This decade sees some resistance to the adoption of design methodology. Even early pioneers begin to dislike "the continual attempt to fix the whole of life into a logical framework".

1973

Rittel publishes *The State of the Art in Design Methods*. He argues that the early approaches of the 1960s were simplistic, and a new generation of methodologies are beginning to emerge in the 1970s. Rather than optimize through systematic methods, the **second generation** is about finding a satisfactory solution in which designers partner with clients, customers and users. This approach is probably more relevant to architecture and planning than engineering and industrial design.

1980

This decade sees the development of **engineering design methodology**. An example is the series of *International Conferences on Engineering Design*. The American Society of Mechanical Engineers also launches a series of conferences on Design Theory and Methodology.

Oct
1982

Nigel Cross discusses the problem-solving nature of designers in his seminal paper *Designerly Ways of Knowing*.

1987

Peter Rowe, Director of Urban Design Programs at Harvard, publishes his book *Design Thinking*. This explores the underlying structure and focus of inquiry in design thinking.

1991

IDEO, an international design and consulting firm, brings design thinking to the mainstream by developing their own customer-friendly technology.

Design Thinking