# Computer Vision

## Summary

Computer Vision is about enabling computers to see, perceive and understand the world around them. This is achieved through a combination of hardware and software. Computers are trained using lots of images/videos and algorithms/models are built. An understanding of human vision also informs the design of these algorithms/models. In fact, computer vision is a complex interdisciplinary field at the intersection of engineering, computer science, mathematics, biology, psychology, physics, and more.

Since the early 2010s, neural network approaches have greatly advanced computer vision. But given the sophistication of human vision, much more needs to done.

Computer vision is an important part of Artificial Intelligence. This is because we "see" less with our eyes and more with our mind. It's also because it spans multiple disciplines.

## Discussion

### How does Computer Vision compare with human vision?

Computers first see every image as a matrix of numbers. It's the job of algorithms to transform these low-level numbers into lines, shapes and objects. This isn't so different from human vision where the retina triggers signals that are then processed by the visual cortex in the brain, leading to perception.

CV uses Convolutional Neural Network (CNN). This is a model inspired by how the human visual cortex works, processing visual sensory inputs via a hierarchy of layers of neurons. While more work is needed to achieve the accuracy of human vision, CNNs have brought us the best results so far. CNNs lead to Deep CNNs where the idea is to match 2D templates rather than construct 3D models. This again is inspired by our own vision system.

Among the obvious differences are that CV can see 360 degrees; that CV is not limited to just visible light; that CV is not affected by fatigue or physiology; CV sees uniformly across the field of view but our peripheral vision is better in low-light conditions; CV has its own biases but they're free from biases and optical illusions that affect humans.

### Isn't Computer Vision similar to image processing?

Image processing comes from the disciplines of Electrical Engineering and Signal Processing whereas computer vision is from Computer Science and Artificial Intelligence. Image processing takes in an image, enhances the image in some way, and outputs an image. Computer vision is more about image analysis with the goal of extracting features, segments and objects from the image.

Adjusting the contrast in an image or sharpening the edges via a digital filter are image processing tasks. Adding colour to a monochrome image, detecting faces or describing the image are computer vision tasks. It's common to combine the two. For example, an image is first enhanced and then given to computer vision. Computer vision can detect faces or eyes, then image processing improves facial skin tone or removes red eye.

Let's note that Machine Learning can be used for both CV and image processing, although it's more commonly used for CV.

### How is Computer Vision (CV) related to Machine Vision (MV)?

Machine vision is more an engineering approach to enable machines to see. It's about image sensors (cameras), image acquisition, and image processing. For example, it's used on production lines to detect manufacturing defects or ensure that products are labelled correctly. Machine vision is commonly used in controlled settings, has strong assumptions (colour, shape, lighting, orientation, etc.) and therefore works reliably.

Computer vision incorporates everything that machine vision does but adds value by way of image analysis. Thus, machine vision can be seen as a subset of computer vision. CV makes greater use of automation and algorithms, including machine learning but the line between CV and MV is blurry. Typically, vision systems in industrial settings can be considered as MV.

### What are some applications of Computer Vision?

CV has far-reaching applications. Wikipedia's category on this topic has many sub-categories and dozens of pages:

    + **Recognition Tasks**: Recognition of different entities including face, iris, gesture, handwriting, optical character, number plate, and traffic sign.
    + **Image Tasks**: Automation of image search, synthesis, annotation, inspection, and retrieval.
    + **Applications**: Enabling entire applications such as augmented reality, sign language translation, automated lip reading, remote sensing, mobile mapping, traffic enforcement camera, red light camera, pedestrian detection and video content analysis.Facebook uses CV for detecting faces and tagging images automatically. Google's able to give relevant results for an image search because it analyzes image content. Microsoft Kinect uses stereo vision. Iris or face recognition are being used for surveillance or for biometric identification. Self-driving cars employ a variety of visual processing tasks to drive safely.

Gauss Surgical uses real-time ML-based image analysis to determine blood loss in patients. Amazon Go uses CV for tracking shoppers in the store and enabling automated checkout. CV has been used to study society, demographics, predict income, crime rates, and more.

### Could you describe some common tasks in Computer Vision?

**Image Segmentation** groups pixels that have similar attributes such as colour, intensity or texture. It's a better representation of the image to simplify further processing. This can be subdivided into semantic or instance segmentation. For instance, the former means that persons and cats are segmented; the latter means that each person and each cat is segmented.

**Image Classification** is about giving labels to an image based on its content. Thus, the image of a cat would be labelled as "cat" with high probability.

**Object Detection** is about detecting objects and placing bounding boxes. Objects are also categorized and labelled. In a two-stage detector, boxing and classification are done separately. A one-stage detector will combine the two. Object detection leads to **Object Tracking** in video applications.

**Image Restoration** attempts to enhance the image. **Image Reconstruction** is about filling in missing parts of the image. With **Image Colourization**, we add colour to a monochrome image. With **Style Transfer** we transform an image based on the style (colour, texture) of another image.

### What's the typical data pipeline in Computer Vision?

A typical CV pipeline includes **image acquisition** using image sensors; **pre-processing** to enhance the image such as reducing noise; **feature extraction** that would reveal lines, edges, shapes, textures or motion; **image segmentation** to identify areas or objects of interest; **high-level processing** (also called post-processing) as relevant to the application; and finally, **decision making** such as classifying a medical scan as true or false for tumour.

### Could you mention some algorithms that power Computer Vision?

Here's a small and incomplete selection of algorithms. For pre-processing, thresholding is a simple and effective method: conventional, Otsu global optimal, adaptive local. Filters are commonly used: median filter, top-hat filter, low-pass filter; plus filters for edge detection: Roberts, Laplacian, Prewitt, Sobel, and more.

For feature-point extraction, we can use HOG, SIFT and SURF. Hough Transform is another feature extraction technique. Viola-Jones algorithm is for object or face detection in real time. There's also the PCA approach called eigenfaces for face recognition.

Lucas-Kanade algorithm and Horn-Schunk algorithm are useful for optical flow calculation. Mean-shift algorithm and Kalman filter are for object tracking. Graph Cuts are useful for image segmentation. For 3D work, NDT, ICP, CPD, SGM, and SGBM algorithms are useful.

Bresenham's line algorithm is for drawing lines in raster graphics. To relate corresponding points in stereo images, use Fundamental Matrix.

From the world of machine learning algorithms we have CNNs and Deep CNNs. We also have SVM, KNN, and more.

### What are the current challenges in Computer Vision?

It's been shown that "adversarial" images in which pixels are selectively changed can trick image classification systems. For example, Google Cloud Vision API thinks it's looking at a dog when really the scene has skiers.

Algorithms are capable of deductive reasoning but are poor with understanding context, analogies and inductive reasoning. For example, CV can recognize a book but the same book when used as a doorstop will be seen only as a book. In other words, CV is incapable of understanding a scene.

While CV has progressed with object recognition, accuracy can suffer if the background is cluttered with details or the object is shown under different lighting in a different angle. In other words, *invariant* object recognition is still a challenge.

There are also challenges in creating low-powered CV solutions that can be used in smartphones and drones. Embedded vision is becoming mainstream in automotive, wearables, gaming, surveillance, and augmented reality with a focus towards object detection, gesture recognition, and mapping functions.

### What software tools would you recommend for doing Computer Vision?

The easy approach to use CV into your applications is to invoke CV APIs: Microsoft Azure Computer Vision, AWS Rekognition, Google Cloud Vision, IBM Watson Visual Recognition, Cloud Sight, Clarifai, and more. These cover image classification, face detection/recognition, emotion detection, optical character recognition (OCR), text detection, landmark detection, content moderation, and more.

**OpenCV** is a popular multiplatform tool with C/C++, Java and Python bindings; but it doesn't have native GPU support. For coding directly in Python, there's also NumPy, SciPy and scikit-image. **SimpleCV** is great for prototyping before you adopt OpenCV for more serious work.

Computer Vision Toolbox from MathWorks is a paid tool but it can simplify the design and testing of CV algorithms including 3D vision and video processing. C# and .NET developers can use AForge.NET/Accord.NET for image processing. For using CNNs, **TensorFlow** is a popular tool. **CUDA Toolkit** can help you get the best performance out of GPUs. Try **Tesseract** for OCR.

For more tools, see eSpace on Medium, ResearchGate and Computer Vision Online.

## Milestones

1957

Russell Kirsch at the National Bureau of Standards (now called NIST) asks, "What would happen if computers could look at pictures?" Kirsch and his colleagues develop equipment to scan a photograph and represent it in the world of computers. They scan a 5cm x 5cm photograph of Kirsch's infant son into an array of 176 x 176 pixels.

1959

Neurophysiologists David Hubel and Torsten Wiesel discover that a cat's visual cortex is activated not by objects but by simple structures such as oriented edges. It's only decades later that we use this in the design of CNNs.

1963

Larry Roberts publishes his PhD thesis at MIT. His idea is to create a 3D representation based on perspectives contained in 2D pictures. This is done by transforming images into line drawings. Soon after this, Roberts joins DARPA and becomes one of the founders of the ARPANET that eventually evolves into the Internet. Roberts is considered at the father of computer vision.

1966

The summer of 1966 is considered the official birth of computer vision. Seymour Papert of MIT's AI Lab defines the "Summer Vision Project". The idea is to do segmentation and pattern recognition on real-world images. The project proves too challenging for its time and not much is achieved.

1971

Research in computer vision continues in the direction suggested by Roberts. David Huffman and Max Clowes independently publish **line labelling algorithms**. Lines are labelled (convex, concave, occluded) and then used to discern the shape of objects.

1974

To overcome blindness, Kurzweil Computer Products comes up with a program to do OCR. This comes at a time when funding and confidence in AI was at its lowest point, now called as the AI Winter of the 1970s.

1979

David Marr suggests a **bottom-up approach** to computer vision (his book is published posthumously in 1982). He states that vision is hierarchical. It doesn't start with high-level objects. Rather, it starts with low-level features (edges, curves, corners) from which higher level details are built up. In the 1980s, this leads to greater focus on low-level processing and goes on to influence deep learning systems. Marr's work is now considered a breakthrough in computer vision.

1980

In 1980, Kunihiko Fukushima designs a neural network called *Neocognitron* for pattern recognition. It's inspired by the earlier work of Hubel and Wiesel. It includes many convolutional layers and may be called the first deep neural network. Later this decade, **math and stats** begin to play a more significant role in computer vision. Some examples of math-inspired contributions include Lucas-Kanade algorithm (1981) for flow calculation, Canny edge detector (1986), and eigenface for facial recognition (1991).

1990

In the early 1990s, in criticism of Marr's approach, **goal-oriented** computer vision emerges. The idea is that we often don't need 3D models of the world. For example, a self-driving car only needs to know if the object is moving away or towards the vehicle. One of the proponents of this approach is Yiannis Aloimonos.

2012

**AlexNet** wins the annual ImageNet object classification competition with the idea that depth of a neural network is important for accurate results. AlexNet uses five convolutional layers followed by three fully connected layers. It's considered a breakthrough in computer vision, inspiring further research in its footsteps. In 2015, Microsoft's ResNet of 152 layers obtains better accuracy.

Computer Vision


# Log Aggregation

## Summary

Log aggregation is the process of collecting, organizing, and analyzing log data from various sources across a system or network. Logs are records of events or messages generated by software, devices, or systems. Such logs are used to diagnose problems, monitor system performance, and identify security issues.

Log aggregation is especially important in complex systems and distributed environments, where logs are generated by a large number of components and can be difficult to access and analyze. By aggregating logs from different sources into a single centralized logging system, administrators can get a holistic view of system health and detect issues that may not be apparent when looking at individual logs.

Logs come in different formats, types and sources. Log aggregation tools are available to handle all aspects of log management including dashboards, alerts, and reports.

## Discussion

### Why do we need log aggregation?

In a monolithic architecture, all the components of the application are running on a single host. The logs are relatively easier to access and analyze. However, as the application grows, the logs can become unwieldy and difficult to manage. So log aggregation is still useful to centralize logs and make it easier to identify and resolve issues.

Log aggregation is even more critical in a microservices architecture because the services are distributed. Services generate logs on different hosts. This makes it harder to troubleshoot issues when they arise. Log aggregation leads to faster and more effective incident response.

### What's a typical log aggregation architecture for microservices?

In a typical architecture, each microservice stores log entries into a file or sends them to a log stream. A **log stream** can be thought of as a real-time, unstructured data feed of the activities and events occurring within a specific component or application.

On each host is an agent software called the **log collector**. It forwards the log to a **central log repository**. The centralized repository may be a database, a file system, or a cloud-based storage service. The log collector itself may be a separate process running on the host. Where the deployment is in a Kubernetes cluster, the log collector may be a separate container (aka sidecar container) running in the same pod as the main container.

Once logs are in the repository, there are **tools** to search, visualize and analyze them. ELK stack is one such tool. Logs can also be analyzed as they're being moved into the repository, what's called streaming analytics. This makes sense for critical applications that require real-time monitoring/metrics/alerts and immediate corrective action.

### What types of logs are involved in log aggregation?

Since it's hard to predict what sort of problems may crop up and where, a well-designed application must collect many types of logs. **Application logs** provide information about the behaviour of the application. **Infrastructure logs** provide information about the underlying system and its components. Both are important for troubleshooting issues. A good practice is to tag and structure the logs so that they can be easily searched and analyzed.

Application logs can be further sub-divided into security logs, audit logs, database logs, application server logs, middleware logs, and so on. Likewise, infrastructure logs can be further sub-divided into security logs, audit logs, operating system logs, network logs, and so on. Each log type must contain sufficient details relevant to its context. For example, security logs would include unauthorized access attempts. Database logs would include transactions. Middleware logs would contain information about message queues.

Metrics complement logs towards better monitoring and observability. **Metrics** are structured data points (CPU usage, memory usage, network traffic, response times, etc) collected at regular intervals. While logs provide a detailed record of individual events, metrics provide a high-level view of overall system performance.

### How are logs from different sources interleaved?

Logs can be interleaved at different stages of the log aggregation process, such as when they are collected by log collectors, stored in a database or file system, or analyzed by log search and analysis tools. The specific interleaving method used depends on the requirements of the system and the use case.

The most method is **time-based interleaving** that orders logs based on the time they were generated. This is useful for correlating events across different services or devices.

**Source-based interleaving** is more suited for analyzing logs from a specific component or service.

**Event-based interleaving** uses events or messages to organize logs. This is useful for identifying patterns and trends in the log data. A related method is **transaction-based interleaving**. This is useful for tracking the progress of a specific transaction or operation across different services.

### What log format should I adopt for log aggregation?

When choosing a log format for log aggregation, there are several factors to consider: level of detail, analysis tools, logging libraries in the selected programming language, etc. Some frameworks and libraries support many different formats. Examples include Log4j (Java logging library) and Fluentd (open-source data collector).

A popular format is **JSON (JavaScript Object Notation)** because of its flexibility and support in many programming languages. It's lightweight and text-based. It's easy to read and parse.

**Syslog** is a standard for logging messages and events. It's supported by many operating systems and network devices. It's commonly used in enterprise environments. Syslog messages are sent over UDP or TCP and can be easily collected by a Syslog server.

Apache's **Common Log Format (CLF)** is used for logging HTTP server requests. It's widely supported by web servers and web application frameworks. Like JSON, it's text-based and easy to parse and analyze.

### What are the best practices for log aggregation?

Define a clear logging strategy. Set out the goal of log analysis: identify bottlenecks, optimize performance, regulatory compliance, etc. Determine what log data you need to collect, how long you need to retain it, and what tools you will use for analysis and visualization. Regularly review the strategy since log aggregation is an ongoing process that requires regular maintenance and updates.

Collect logs in real time, rather than relying on batch processing or manual collection. If real-time analysis is not possible, at least regularly analyze logs. Use a standardized logging format, such as JSON or Syslog, to make it easier to parse and analyze logs.

Storing all logs in a central location is a pre-requisite. Implement security controls to protect log data, such as encrypting logs in transit and at rest, and restricting access to logs based on user roles.

Rotate log files regularly to prevent them from becoming too large. Monitor the system and set up alerts to warn of lost logs, bottlenecks or unusual activity.

Many of these best practices are also part of the CNCF guidelines for logging in cloud-native environments.

### What should developers and log analysts know about log aggregation?

Logs provide a wealth of information about application behaviour, including errors, warnings, performance metrics, trends and patterns. Developers should strive to make logs as useful as possible given this context. Too much logging can impact performance, especially if logging is synchronous. Developers must balance the need for information with the impact on performance.

Log data should be designed for machine consumption. This means logs should be standardized so that tools can process them effectively.

Logs are also data. Like in any data analytics workflow, log analytics has similar challenges: data volume, variety, quality, security, and accessibility. Authentication credentials and personally identifiable information (PII) should not be in the logs, or if present, should be perhaps encrypted. Logs should be clean of errors, duplicates, or other inconsistencies.

When troubleshooting problems with logging, check that the log source is actually generating and sending the logs. Check the log configuration. If logs are not reaching the central repository, check the log collector, network connections or firewall settings. Another technique is to increase the logging level to view detailed logs. To deal will log loss, implementing redundancy (multiple logging systems) may be necessary.

### What are some case studies of log aggregation?

Airbnb uses a combination of open-source tools, including Logstash, Kibana, and Elasticsearch, for log aggregation. The company collects log data from over 50,000 servers and analyzes over 200 GB of log data per day. Airbnb uses log aggregation to monitor application performance, detect security incidents, and troubleshoot issues.

Uber uses a custom-built log aggregation system called Marmaray to collect and analyze log data from its microservices architecture. Marmaray provides real-time log analysis and allows Uber to identify and troubleshoot issues quickly. Uber also uses Apache Kafka to collect log data from its applications and store it in a centralized location.

Netflix uses a combination of open-source tools, including Apache Kafka, Apache Cassandra, and Elasticsearch, for log aggregation. The company collects over 1 trillion events per day and also does real-time log analysis. In-house tools for log aggregation include the Spectator library for collecting application metrics, the Atlas service for storing and querying metric data, and the Mantis service for real-time stream processing.

## Milestones

1984

The **syslog** protocol is developed for UNIX systems. Syslog is a standard for logging messages, and it allows system administrators to collect and store logs from multiple sources. In 2000, a more advanced implementation of syslog called **rsyslog** is launch. It supports features such as filtering, log rotation, and remote logging.

1999

The concept of log aggregation gains popularity with the advent of **distributed systems and service-oriented architectures (SOA)** in the late 1990s. As the number of systems and services in these architectures increase, the need for a centralized approach to log management becomes more critical.

2000

In the early 2000s, the rise of **cloud computing** and the popularity of **containerization** leads to a new wave of log aggregation solutions. These solutions were designed to collect logs from virtual machines and containerized environments and store them in cloud-based log management platforms.

2006

The Apache Hadoop project is launched. Hadoop is a distributed computing platform that includes the Hadoop Distributed File System (HDFS) for storing and processing large datasets. Hadoop includes the **Hadoop Distributed Logging Service (HDFS)** for aggregating and analyzing log data.

2008

**Graylog** is launched. It's an open-source log management platform that includes features such as log aggregation, search, and visualization. Other tools follow: Logstash (2009), Elasticsearch (2010), Fluentd (2012) and Prometheus (2016).

2019

The Cloud Native Computing Foundation (CNCF) releases a set of **guidelines for logging in cloud-native environments**. The guidelines emphasize the importance of structured logging and recommend the use of tools such as Fluentd and Elasticsearch for log aggregation.

Log Aggregation


# Express.js

## Summary

Express is a minimalistic web framework based on Node.js. It's essentially a series of *middleware function calls*, each of which does something specific. Express in not opinionated, which is also why you're free to use it in different ways. For example, it doesn't enforce a particular design pattern or a folder structure.

Express is an open source project that's been managed by Node.js Foundation since 2016. Express has good adoption. It's part of the MEAN stack. It's also being used by many other web frameworks such as Kraken, LoopBack, Keystone and Sails. However, it's been said that Express is not suited for large projects/teams.

## Discussion

### Why should I use Express.js when there's already Node.js?

While it's possible to build a web app or an API service with only Node.js, Express.js simplifies the development process. Express itself is based on Node. It's been said that Express...

> provides a thin layer of fundamental Web application features, without obscuring Node.js features that you know and love.

For example, sending an image is complex in Node but easily done in Express. In Node, the route handler is a large monolith but Express enables more modular design and maintainable code.

Node is a JavaScript runtime for server-side execution. Thus, Node can be used as the app server for your web application. Express is seen as a lightweight web framework. Express comes with a number of "middleware" software that implement out-of-the-box solutions for typical web app requirements. HTTP requests/responses are relayed by Node to Express, whose middleware then do the processing.

### Could you explain the concept of middleware in Express.js?

When client requests are received, the server doesn't handle all of them alike. For example, submitting a form is handled differently from clicking a like button. Thus, each request has a well-defined handler, to which it must be properly routed.

Middleware sits in the routing layer. Express is basically a routing layer composed of many modular processing units called middleware. Requests are processed by middleware functions before sent to the handlers.

A request-response cycle can invoke a series of middleware functions. A middleware can access and modify the request and response objects. Once a middleware function has completed, it can either pass control to the next middleware function (by calling `next()`) or end the cycle by sending a response to the client. Since middleware can send responses, even the request handlers can be treated or implemented as middleware.

Middleware can also be called after the request is processed and response is sent to the client. But in this case, middleware can't modify the request and response objects.

### What are the different types of Express.js middleware?

There are a few types of Express middleware:

    + **Application-level**: These are bound to an instance of the application object `express()`. They are called for every app request. The middleware signature is `function (req, res, next)`.
    + **Router-level**: These are bound to an instance of the router `express.Router()`. Otherwise, they work similar to app-level middleware.
    + **Error-handling**: Any middleware can throw an error. These can be caught and handled by error-handling middleware. These middleware have an extra argument in their signature: `function (err, req, res, next)`.
    + **Built-in**: These are built into default Express installation. These include `express.static`, `express.json` and `express.urlencoded`.
    + **Third-party**: Express has a rich ecosystem of third-party developers contributing useful middleware. Some of these are maintained by the Express team, while others come from the community. For an example list of some Express middleware, visit Express Resources Middleware page.

### What are some useful or recommended middleware for Express.js?

Without being exhaustive here are some useful middleware:

    + *body-parser*: To parse HTTP request body.
    + *compression*: Compresses HTTP responses.
    + *cookie-parser*: Parse cookie header.
    + *cookie-session*: Establish cookie-based sessions.
    + *cors*: Enable Cross-Origin Resource Sharing (CORS).
    + *csrf*: Protect from Cross-Site Request Forgery (CSRF) exploits.
    + *errorhandler*: Development error-handling/debugging.
    + *morgan*: HTTP request logger. Alternatives are *winston* and *bunyan*.
    + *timeout*: Set a timeout period for HTTP request processing.
    + *helmet*: Helps secure your apps by setting various HTTP headers.
    + *passport*: Authentication using OAuth, OpenID and many others.
    + *express-async-handler*: For Async/await support. An alternative is *@awaitjs/express*.
    + *express-cluster*: Run express on multiple processes.

### As a beginner, how can I get started with Express.js?

A beginner should first learn the basics of JavaScript, Node.js and Express.js, in that order. Express website has useful tutorials. The routing guide is a good place to start. For in-depth understanding, or as a handy reference, use the API Reference.

Install Node.js first. Then install Express.js: `npm install express --save`

Since Express is unopinionated, your folder structure can be anything that suits your project. However, *express-generator* is a package that gives beginners a basic structure to start with. This installs the *express* command-line tool, which can be used to initialize your app. You get to choose your preferred templating engine (pug, ejs, handlebars) and styling support (less, stylus, compass, sass).

### Where is Express.js not a suitable framework?

Express is not recommended for large projects.

Although Express is lightweight, for performance-critical apps such as highly scalable REST API services, prefer to use specialized Node frameworks such as fastify, restana, restify, koa, or polka.

### What are some best practices when using Express.js?

Adopt a modular structure to make your code more manageable.

In production, log requests and API calls but minimize debug logs. A package such as debug might help in configuring the logging contexts. Pipe logs from `console.log()` or `console.error()` to another program since saving them to file or printing to console make the calls synchronous.

In general, avoid synchronous calls. Use Node's command-line flag `--trace-sync-io` (during development only) to be warned when synchronous calls happen.

For performance, use compression middleware. For security, use helmet. To run multiple instances on multicore systems, launch your app on a **cluster** of processes. The cluster can be frontended with a **load balancer**, which can be a reverse proxy such as Nginx or HAProxy.

Handle errors using try-catch or promises. Don't listen for `uncaughtException` event to handle errors. This might be a way to prevent your app from crashing but a better way is to use a **process manager** (StrongLoop Process Manager, PM2, Forever) to restart the app automatically.

## Milestones

2009

Ryan Dahl and others at Joyent develop Node.js with initial support for only Linux. The name *Node* is coined in March. Being open source, version 0.1.0 is released on GitHub in May. In November, Ryan Dahl presents Node.js at JSConf in Berlin.

Nov
2010

As a web development framework based on Node.js, v1.0.0 of **Express.js** is released. The development of Express can be traced to January 2010 when v0.1.0 was committed on GitHub.

2011

To manage packages for Node.js, **Node Package Manager (NPM)** is released. An early preview of NPM can be traced to 2009, by Isaac Z. Schlueter.

Apr
2013

Valeri Karpov at MongoDB proposes a full-stack solution that he terms the **MEAN stack**: MongoDB, Express, Angular and Node. Given that three of these are based on JavaScript, this promotes the demand for full-stack developers who can handle both frontend and backend code.

Jul
2014

StrongLoop acquires commercial rights to Express. StrongLoop itself offers an open source API framework called **LoopBack** on top of Node and Express. Some community folks criticize this "sponsorship" or "sale" of what was an open source project.

Feb
2015

Joyent, IBM, Microsoft, PayPal, Fidelity, SAP and The Linux Foundation come together to establish **Node.js Foundation**.

Sep
2015

IBM acquires StrongLoop, a company that specializes in Node.js, Express.js and API management. StrongLoop is said to offer "robust implementation, making Node.js enterprise-grade". This acquisition could mean better adoption of Node and Express in the corporate world.

Feb
2016

Doug Wilson, one of the main contributors to Express, decides to stop working on Express. This is mainly because what was once an open source project is now controlled by IBM/StrongLoop. Express community also state their unhappiness with the way the framework is managed. Meanwhile, in a move towards open governance, Express is handed over to Node.js Foundation as an incubation project.

Oct
2018

Version 4.16.4 of Express is released. The same month version 5.0.0-alpha.7 is released. Version 5.0.0-alpha.1 can be traced back to November 2014.

Express.js


# Git

## Summary

Git is a popular free and open source distributed version control system for managing small to large-scale projects. It keeps track of the changes made to files in a project, making it easy to roll back when necessary. It is compatible with a wide range of operating systems and integrated development environments making it accessible to a large range of developers.

Git is a software that tracks Files and their history. Online hosting providers of git such as Github helps developers to remotely collaborate and combine each other's work.

## Discussion

### Why do developers need to learn Git?

Git is useful for everyone from web developers to app developers who develop code or make changes to their files , if a set of people are working on one project, there is a risk of overwriting or conflicting with each others work to avoid these problems we can use Git which help teams to collaborate by implementing version control system(VCS). The most significant advantage is they can track and compare the modifications made to the files and revert back to their earlier versions if necessary.

Version Control System (VCS) are specialized software which keeps the code in a central location and allows a team of people for sharing, collaborating and to track and make changes to their work which makes it easier to develop software in a continuous and straightforward manner.

### What is a Git Repository?

A Repository in Git stores all the project related files as well as the revised history of all the files. It is virtual storage for your projects as it creates a sub-directory named .git (Usually hidden) inside your project directory to store these files. This directory is the heart of the repository where it also tracks the changes done to the files. So that deleting the.git/ subdirectory deletes the history of your project.

There are different ways to create a repository git init command is used to create a new repository locally and the git clone creates a local copy of a full copy of an existing repository stored on a remote server .

### What is the difference between Bare and Non-Bare git repository?

A bare repository is a git repository that is just used for collaboration among multiple individuals for push and pull, with no active development or code written directly. It can serve as a remote repository for team members to share code. Bare Repository can be created using `git init --bare` command and if you list the contents inside the directory you can see the contents of what is normally found inside the .git directory. Individual users can't make changes or create new versions. .

In a non-bare repository, files under that reside under .git directory retain the snapshot of tracked files and allow you to edit the files under the working directory, whereas in a bare repository, you will not see the .git directory and instead all of the files that normally reside under .git will be present right over in the root directory

`git init` command, by default creates a non-bare repository. \* Except for a few variations,both bare and non-bare repositories are almost identical  .

### What is the Difference between Git and SVN?

Git is a distributed version control system (DVCS) that saves a local copy of the repository on each member's machine. Any modification made to these files, referred to as a commit, is compared with the previous version and updates happen only to the alterations made in the files. Each user has their own repository as well as a working copy. Other members won't be able to see your modifications until you push changes to the central repository.

SVN (Subversion) is a centralised version control system (CVCS) that requires individuals working on a repository to download the most recent version from a central location. Limitations of CVCS is that it need to be connected to the central repo to get the latest version of repository and also it can't perform other repo operations locally like DVCS. Despite the fact that each user has their own working copy, there is only one central repository where other members can see and update your changes as soon as you push them.

### Tell me some interesting features of Git

    + **Troubleshooting**: git blame command shows line-by-line revision history done by an author. git bisect command is also used to assess good or bad commits by going throughout the project history. When we have a little idea of how a certain bug was introduced git bisect can be useful whereas git blame can be used when we couldn't find the exact cause for the problem.
    + **Branching and Merging**:Branches create separate copies of files in a repository for users and changes made to these files by users is not reflected on the main code and is independent of each other. This can help users to test and develop new features and commit to the main code only if required. A Sequence of commits from branches are called merging.
    + **Rebasing**: Changes made to the files can be replayed in the same sequence. Commits can be made on the main branch on top of other different commits made recently by another user. Each commit is made one by one so that if a conflict happens the rebasing is paused until it's fixed.

### What is a Git workflow?

It is a method of using Git in a productive and feasible manner. There are no specific rules these are some common practices and can be considered as guidelines for the development. Some commonly used workflows are described below :

    + **Centralized Workflow**: Similar to SVN a single collaboration model is conceptually centralized though it is decentralized technically. Though the project is contributed by multiple users there is only a single-point-of-entry for the changes made. The work done by each developer is merged before the changes are pushed no overwriting is allowed here.
    + **Trunk-based development**: Development happens on a single branch called Trunk the users divide work into small pieces and merged it into the trunk frequently within few hours. This also decreases the chances of merge conflicts.
    + **Git Flow Workflow**: There are many branches split for different purposes. Developments happen in the Develop branch with feature branches split for new features combining back to develop after getting ready. Release branch contains the code after test and is ready to be merged into the main branch. Final code ready for the release is only available in the main branch.

### Can you describe Git add and commit?

The Git add command includes the new or modified files in our working directory to the staging area.Files in the staging area are the ones ready to be added in the next commit. It asks Git to include these files for the upcoming commit. You can add directories, any files, even a specific part of a file. If a deleted file is added using this command then the deletion will be staged for commit.

The term Committing is similar to saving files. Git commit would never happen without using git add for adding files to the staging area. Commits are like a recent snapshot of the repository and can show the history over a period of time. It also includes metadata like author name, timestamp etc. git revert command can be used to undo or redo a specific commit . git reset removes a specific commit and rollsback to the required commit by user . If you use git reset accidentaly and lose commits, you can use git reflog to get them back .

### Mention the three states of Git?

Git considers each file in a directory initially to be in one of two states: tracked or untracked. Files that have already been captured by git are known as tracked files. Everything else is untracked files, which are any files in your working directory not yet captured or not part of git's version control.

These three file states are possible after the files are tracked by Git .

    + **Unmodified/Modified**: State of the file changes to modified whenever Files in a Repository is being edited until it is committed and stored to the local git database.
    + **Staged/Staging area**: When we've completed all the modifications to the files it will be moved to the staged state and is ready to be committed.
    + **Committed**: All modifications are saved in the local database, and the updated files are committed.

### Can you suggest some best practices in using Git?

Software development teams can adapt their own best practices according to their needs so there are no definite answers. Listed below are some recommended practices followed in the industry.

    + **Keep the changes minimum**: Use the least amount of code possible to solve an issue so it is easy to revert back if that doesn't work as expected. Git is known to have atomic behaviour means if there is an issue git will commit files partially and avoid the rest. So only one fix or one upgrade at a time is recommended
    + **Writing Effective commit messages**: Describe Commit messages in a meaningful way by using a verb in the present tense. Clear messages benefit both teams and self when we look back later.
    + **Doing Code Reviews**: Code shall be evaluated by one or a team of developers to avoid mistakes, potential bugs.
    + **Make use of Branches**: Branches can be used to separate work among developers for efficient workflow. Code is verified and tested before merging into the main or master branch.

### Which should I choose Mono-Repo vs Multi-Repo?

The term "Mono-Repository" refers to the storage of all codebase related to a project such as its libraries, dependencies, or services, in a single repository. This approach helps new contributors to make their initial setup easy since everything related to the project is stored in a single location. Deploying a component independently becomes difficult since all the files are tightly coupled with each other also the size of the repository would be huge in large projects.

In Multi-Repository all the project related services are stored in a separate repository. Libraries, dependencies can run altogether from an isolated environment. Teams can work autonomously with different responsibilities but need to better communicate and sync to avoid breaking the code while all the work happens independently.

There is a Multi-Mono approach known as submodules that keeps one git repository as a subdirectory of another repository.

Though these approaches have the same goal to manage the codebase with their own benefits and drawbacks. Some challenges are managing the final release, collaborating and communication among the team. Finally, it is up to the team to decide which would suit their workflow.

## Milestones

Apr
2005

Linus Torvalds designed the main principles of Git in a week of time to maintain the large Linux Kernel Project. Previously, a free DVCS named Bitkeeper was used until they revoked the free-of cost usage . The Word Git is a random three letter word not used by any UNIX command it doesn't have any meaning officially.

Jul
2005

Linus Torvalds established Git as an open source for contributions, and Junio C Hamano is designated as Git's maintainer.

Dec
2005

Git 1.0 version was launched

Jan
2006

The .dircache directory is renamed to .git which stores all the information about the repository. . A command init-db to initalize repository is changed to git-init-db.

Oct
2007

Preston-Werner and Wanstrath wish to develop a platform where where programmers could host and collaborate in Git and started working on their idea of building Github.

May
2012

Git logo was updated

Nov
2018

git range-diff command is introduced in version Version 2.19 it can be used to compare two sequences of commits and order of the changes made.

Mar
2021

git maintenance is introduced to optimize the repository data by speeding up commands and reducing the storage requirements.

Git