# TensorFlow

## Summary


TensorFlow is an open source software library for numerical computation using **data flow graphs**. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) that flow between them. This dataflow paradigm enables parallelism, distributed execution, optimal compilation and portability. 

The typical use of TensorFlow is for Machine Learning (ML), particularly Deep Learning (DL) that uses large scale multi-layered neural networks. More specifically, it's best for classification, perception, understanding, discovery, prediction and creation. 

TensorFlow was originally developed by researchers and engineers working on the Google Brain team within Google's Machine Intelligence Research organization for ML/DL research. The system is general enough to be applicable in a wide variety of other domains as well.

## Discussion

### For which use cases is TensorFlow best suited?

TensorFlow can be used in any domain where ML/DL can be employed. It can also be used in other forms of AI, including reinforcement learning and logistic regression. On mobile devices, applications include speech recognition, image recognition, object localization, gesture recognition, optical character recognition, translation, text classification, voice synthesis, and more. 

Some of the areas are: 

    + **Voice/Speech Recognition**: For voice-based interfaces as popularized by Apple Siri, Amazon Alexa or Microsoft Cortana. For sentiment analysis in CRM. For flaw detection (noise analysis) in industrial systems.
    + **Text-Based Applications**: For sentimental analysis (CRM, Social Media), threat detection (Social Media, Government) and fraud detection (Insurance, Finance). For machine translation such as with Google Translate. For text summarization using sequence-to-sequence learning. For language detection. For automated email replies such as with Google SmartReply.
    + **Image Recognition**: For face recognition, image search, machine vision and photo clustering. For object classification and identification within larger images. For cancer detection in medical applications.
    + **Time-Series Analysis**: For forecasting. For customer recommendations. For risk detection, predictive analytics and resource planning.
    + **Video Detection**: For motion detection in gaming and security systems. For large-scale video understanding.

### Could you name some applications where TensorFlow is being used?

TensorFlow is being used by Google in following areas: 

    + RankBrain: Google search engine.
    + SmartReply: Deep LSTM model to automatically generate email responses.
    + Massively Multitask Networks for Drug Discovery: A deep neural network model for identifying promising drug candidates.
    + On-Device Computer Vision for OCR - On-device computer vision model to do optical character recognition to enable real-time translation.
    + Retinal imaging - Early detection of diabetic retinopathy using deep neural network of 26 layers.
    + SyntaxNet - Built for Natural Language Understanding (NLU), this is based on TensorFlow and open sourced by Google in 2016.Outside Google, we mention some known real-world examples. Mozilla uses TensorFlow for speech recognition. UK supermarket Ocado uses it for route planning for its robots, demand forecasting, and product recommendations. A Japanese farmer has used it to classify cucumbers based on shape, length and level of distortion. As an experiment, Intel used TensorFlow on traffic videos for pedestrian detection. 

Further examples were noted at the TensorFlow Developer Summit, 2018. 


### Which platforms and languages support TensorFlow?

TensorFlow is available on 64-bit Linux, macOS, Windows and also on the mobile computing platforms like Android and iOS. Google has announced a software stack specifically for Android development called TensorFlow Lite. 

TensorFlow has official APIs available in the following languages: Python, JavaScript, C++, Java, Go, Swift. Python API is recommended. Bindings in other languages are available from community: C#, Haskell, Julia, Ruby, Rust, Scala. There's also C++ API reference for TensorFlow Serving. R's `tensorflow` package provides access to the complete TensorFlow API from within R. 

Nvidia's **TensorRT**, a Programmable Inference Accelerator, allows you to optimize your models for inference by lowering precision and thereby reducing latency. 


### How is TensorFlow different from other ML/DL platforms?

TensorFlow is relatively painless to setup. With its growing community adoption, it offers a healthy ecosystem of updates, tutorials and example code. It can run on a variety of hardware. It's cross platform. It has APIs or bindings in many popular programming languages. It supports GPU acceleration. Through TensorBoard, you get an intuitive view of your computation pipeline. Keras, a DL library, can run on TensorFlow. However, it's been criticized for being more complex and slower than alternative frameworks. 

Created in 2007, **Theano** is one of the first DL frameworks but it's been perceived as too low-level. Support for Theano is also ending. Written in Lua, **Torch** is meant for GPUs. It's Python port released by Facebook, called **PyTorch**, is popular for analyzing unstructured data. It's developer friendly and memory efficient. **Caffe2** does well for modeling convolutional neural networks. **Apache MXNet**, along with its simplified DL interface called **Gluon**, is supported by Amazon and Microsoft. Microsoft also has **Microsoft Cognitive Toolkit (CNTK)** that can handle large datasets. For Java and Scala programmers, there's **Deeplearning4j**. 


### Which are the tools closely related to TensorFlow?

The following are closely associated with or variants of TensorFlow:

    + **TensorFlow Lite**: Enables low-latency inferences on mobile and embedded devices.
    + **TensorFlow Mobile**: To use TensorFlow from within iOS or Android mobile apps, where TensorFlow Lite cannot be used.
    + **TensorFlow Serving**: A high performance, open source serving system for machine learning models, designed for production environments and optimized for TensorFlow.
    + **TensorLayer**: Provides popular DL and RL modules that can be easily customized and assembled for tackling real-world machine learning problems.
    + **TensorFlow Hub**: A library for the publication, discovery, and consumption of reusable parts of machine learning models.
    + **TensorFlow Model Analysis**: A library for evaluating TensorFlow models.
    + **TensorFlow Debugger**: Allows us to view the internal structure and states of running TensorFlow graphs during training and inference.
    + **TensorFlow Playground**: A browser-based interface for beginners to tinker with neural networks. Written in TypeScript and D3.js. Doesn't actually use TensorFlow.
    + **TensorFlow.js**: Build and train models entirely in the browser or Node.js runtime.
    + **TensorBoard**: A suite of visualization tools that helps to understand, debug, and optimize TensorFlow programs.
    + **TensorFlow Transform**: A library for preprocessing data with TensorFlow.

### What's the architecture of TensorFlow?

TensorFlow can be deployed across platforms, details of which are abstracted away from higher layers. The core itself is implemented in C++ and exposes its features via APIs in many languages, with Python being the most recommended. 

Above these language APIs, is the **Layers** API that offers commonly used layers in deep learning models. To read data, **Datasets** API is the recommended way and it creates input pipelines. With **Estimators**, we can create custom models or bring in models pre-made for common ML tasks. 

**XLA (Accelerated Linear Algebra)** is a domain-specific compiler for linear algebra that optimizes TensorFlow computations. If offers improvements in speed, memory usage, and portability on server and mobile platforms. 


### Could you explain how TensorFlow's data graph works?

TensorFlow uses a **dataflow graph**, which is a common programming model for parallel computing. Graph nodes represent **operations** and edges represent data consumed or produced by the nodes. Edges are called **tensors** that carry data. In the example figure, we show five graph nodes: `a` and `b` are placeholders to accept inputs; `c`, `d` and `e` are simple arithmetic operations.

In TensorFlow 1.x, when a graph is created, tensors don't contain the results of operations. The graph is evaluated through **sessions**, which encapsulate the TensorFlow runtime. However, with **eager execution**, operations are evaluated immediately instead of building a graph for later execution. This is useful for debugging and iterating quickly on small models or data. 

For ingesting data into the graph, **placeholders** can be used for the simplest cases but otherwise, **datasets** should be preferred. To train models, **layers** are used to modify values in the graph. 

To simplify usage, high-level API called **estimators** should be used. They encapsulate training, evaluation, prediction and export for serving. Estimators themselves are built on layers and build the graph for you. 


### How is TensorFlow 2.0 different from TensorFlow 1.x?

It makes sense to write any new code in TensorFlow 2.0. Existing 1.x code can be migrated to 2.0. The recommended path is to move to TensorFlow 1.14 and then to 2.0. Compatibility module `tf.compat` should help. 

Here are the key changes in TensorFlow 2.0: 

    + **API Cleanup**: Many APIs are removed or moved. For example, `absl-py` package replaces `tf.app`, `tf.flags`, and `tf.logging`. Main namespace `tf.*` is cleaned up by moving some items into subpackages such as `tf.math`. Examples of new modules are `tf.summary`, `tf.keras.metrics`, and `tf.keras.optimizers`.
    + **Eager Execution**: Like Python, eager execution is the default behaviour. Code execute in order, making `tf.control_dependencies()` redundant.
    + **No More Globals**: We need to keep track of variables. Untracked `tf.Variable` will get garbage collected.
    + **Functions, Not Sessions**: Functions are more familiar to developers. Although `session.run()` is gone, for efficiency and JIT compilation, `tf.function()` decorator can be used. This automatically invokes *AutoGraph* to convert Python constructs into TensorFlow graph equivalents. Functions can be shared and reused.


## Milestones

2011

Google Brain invents **DistBelief**, a framework to train large models for machine learning. DistBelief can make use of computing clusters of thousands of machines for accelerated training. The framework manages details of parallelism (multithreading, message passing), synchronization and communication. Compared to MapReduce, DistBelief is better at deep network training. Compared to GraphLab, DistBelief is better at structured graphs. 

Nov  
2015

Under Apache 2.0 licensing, Google open sources TensorFlow, which is Google Brain's second-generation machine learning system. While other open source ML frameworks exist (Caffe, Theano, Torch), Google's competence in ML is supposedly 5-7 years ahead of the rest. However, Google doesn't open source algorithms that run on TensorFlow, not its advanced hardware infrastructure. 

Apr  
2016

Version 0.8 of TensorFlow is released. It comes with distributed training support. Powered by gRPC, models can be trained on hundreds of machines in parallel. For example, Inception image classification network was trained using 100 GPUs with an overall speedup of 56x compared to a single GPU. More generally, the system can map the dataflow graph onto heterogeneous devices (multi-core CPUs, general-purpose GPUs, mobile processors) in the available processes. 

May  
2016

Google announced that it's been using **Tensor Processing Unit (TPU)**, a custom ASIC built specifically for machine learning and tailored for TensorFlow. 

Jun  
2016

TensorFlow v0.9 is released with support for iOS and Raspberry Pi. Android support has been around from the beginning. 

Feb  
2017

Version 1.0 of TensorFlow is released. The API is in Python but there's also experimental APIs in Java and Go. 

Nov  
2017

Google releases a preview of **TensorFlow Lite** for mobile and embedded devices. This enables low-latency inferences for on-device ML models. In future, this should be preferred over **TensorFlow Mobile**. With TensorFlow 1.4, we can build models using high-level **Keras** API. Keras, which was previously in `tf.contrib.keras`, is now the core package `tf.keras`. 

Sep  
2019

TensorFlow 2.0 is released following an alpha release in June. It improves workflows for both production and experimentation. It promises better performance on GPU acceleration.