# Binary Search

## Summary

Binary search is an algorithm for efficiently searching for an item within a sorted list of items. The search is performed in steps, with each step reducing the search space by half (with linear or sequential search the search space reduces by only one item). At each step, the midpoint is selected and compared against the search value. If the midpoint is lesser than the search value, only the upper half is retained for the next step.

When formulated in a more abstract way, binary search can be used to solve optimization problems with the use of binary predicates. Whenever the predicate is evaluated, the algorithm will retain only half the search space where the predicate is true. In this case, binary search can be seen as "searching" for an optimum solution rather than searching for a specific item.

## Discussion

### How can we derive the complexity of binary search?

Given a list of \(n\) items, linear search is of complexity \(O(n)\). Binary search has a complexity of only \(O(log\_{2}n)\). For comparison, linear search on a list of 1000 items may take on average 500 steps and 1000 steps in the worst case. The same with binary search will only take about \(log\_{2}1000\approx10\) steps.

A video tutorial by GeeksQuiz explains how to derive this complexity. Briefly, since each step reduces the search space by half, and given that \(k\) steps are required, this implies \(n=2^k\), which translates to \(k=log\_{2}n\). Binary search is said to take the divide-and-conquer approach.

Computational complexity described above is in terms of number of steps to complete the search. An optimized implementation would use only one comparison per step. Algorithms also take up memory to manage operations. The space complexity of binary search is \(O(1)\), which means that it is constant and independent of \(n\).

### What are the prerequisites for applying binary search?

Binary search can be applied only under the following conditions:

    + List is sorted: Since half the search space is discarded based on a midpoint value comparison, binary search works only on a sorted list.
    + Random access is possible: Binary search maintains the boundaries of the search space in terms of left and right indices. Midpoint is calculated based on these indices. To access the value of the current midpoint item, the item must be accessible by its index. This works nicely with data structures that support indexing such as arrays (C, Java) and lists (Python). Using binary search on a data structures such as a linked list is wasteful since we need to traverse the list item by item to get to the midpoint.
    + Size: Size of the search space must be known.

### If my array is unsorted, what search should I use?

There are two approaches: sort the array first and then apply binary search; or simply apply linear search. The question is which of these two is faster. Let's use a fast sorting algorithm such as mergesort. This has a complexity of \(O(n\,log\_{2}n)\). Linear search has a complexity of \(O(n)\). Since \(O(n\,log\_{2}n) > O(n)\), to use linear search is better.

This also implies that if your application does frequent updates to the list, avoid binary search since keeping the list in sorted order can reduce performance.

### Can we do a 3-way or even n-ary search for faster results?

Every comparison is between two alternatives, hence binary in nature. To decide among three alternatives we would need two comparisons anyway. A binary search is therefore superior since with two comparisons it can handle \(2^{2}=4\) alternatives. The same argument works for n-ary searches. An alternative explanation is available from Geeks For Geeks.

When the distribution of data is known and the lists are large, then interpolation search can do better than binary search. Hash tables are faster because they maintain a mapping of key-to-value. However, binary search is more versatile since they can be used to solve optimization problems as well.

### Can you give an example of binary search for solving optimization problems?

TopCoder gives an example called FairWorkload. There are \(n\) filing cabinets each with a variable number of folders and \(k\) available workers to process the folders. Each worker must be assigned one or more contiguous filing cabinets to process. Under the constraint of worker availability, how many workers do we need so that workload is most evenly distributed while minimizing the number of folders each worker processes? This can be solved using binary search. The search space is the number of folders per worker.

The predicate can be stated as, "Can the workload be spread so that each worker has to examine no more than x folders, with the limited number of workers available?" With each step, we can calculate the number of workers required for a certain maximum number of files per worker. If this number is less than or equal to \(k\), it implies that we can move the upper limit to the current midpoint.

Another example is to find the minimum of a mathematical function. Another example is to find the square root of a real number.

### How is binary search different from binary search tree?

Binary search is an algorithm. Binary search tree is a data structure. Other than this fundamental difference, both share the same principle in terms of searching for an item. Search complexity is logarithmic in time. With binary search tree, each node in the tree has a left and a right branch for items of lesser and greater value respectively. This means that when the tree is searched, half the search space is discarded either to the left or right. However, a tree that's not balanced can result in worse performance. Even a balanced binary search tree is not necessarily optimal for searching compared to binary search.

Unlike binary search, binary search trees can't be used to solve optimization problems. Binary search trees also take up more space compared to sorted arrays. Binary search trees have the advantage that insertion and deletion of items are also logarithmic in time, unlike sorted arrays where such operations are of complexity \(O(n)\).

### Should we implement binary search iteratively or recursively?

For algorithms in general, there's no general rule that one is better than the other. Sandeep Gupta shows an example where recursive implementation is easier to read.

With respect to binary search, for large arrays, recursive code can lead to stack overflows. Likewise, recursive code can be slower due to the extra function calls. However, if the code can be refactored to tail recursive, speed and stack problems can be overcome with tail-call optimization.

Which one to adopt might ultimately be a matter of personal choice for developers. Adopt one that's easier for you to understand. There's nothing worse that a recursive code that doesn't have a proper terminating condition and hence goes into infinite recursion.

### What are the variations of binary search?

Variations include the following:

    + Uniform binary search
    + Boundary search
    + Fibonacci search
    + Exponential search
    + Interpolation search
    + Fractional cascading

### What are the usual challenges for implementing binary search?

Typical binary search maintains the left and right indices of the search space and calculates the midpoint. For the next step, these indices and midpoints are adjusted. If one is not careful, the search item may be on the boundary and may get missed out due to "off-by-one" error. Don't overconstrain the bounds. Relax them while adhering to the predicate. Also, when calculating the midpoint, take care that overflow and truncation errors are avoided or handled correctly.

Always test for corner cases. Will the code work for odd and even number of list items? Will it work for 1-item or 2-item lists? How will it behave if there are duplicate entries of the search term? Will it exit cleanly if the search term is not found? Does it work correctly when searching for real numbers? With real numbers, match of the search term should be based on either the number of steps or the relative difference between steps. Avoid using absolute difference since precision is limited when numbers are large.

If a recursive implementation is used, be aware how many recursive calls will be possible due to stack space limits.

### Can you give examples of overflow and truncation errors?

When calculating the midpoint on a large array, indices may be large values. Thus, `mid = (lo + hi)/2` may result in an overflow. Change this to `mid = lo + (hi-lo)/2`. On a 2-element array, with lo=0 and hi=1, midpoint will remain at 0 due to truncation. If the match is at index 1, this will result in an infinite loop. The correct code in such a case would be `mid = lo + (hi-lo+1)/2`.

### What support do we have from languages to use or implement binary search?

While it's possible to do one's own implementation, many languages provide libraries that simplify implementation:

    + C++ Standard Template Library: lower\_bound, upper\_bound, binary\_search, equal\_range
    + Java: Arrays.binary\_search
    + .NET Framework: Array.BinarySearch
    + Python: bisect

## Milestones

1946

John Mauchly mentions binary search in what is probably the first published reference.

1960

D.H.Lehmer publishes a method that can handle any \(N\), not just \(N=2^{n}-1\).

1962

H.Bottenbruch changes the algorithm so that comparisons are reduced at the expense of an extra iteration.

1971

A.K.Chandra of Stanford University suggests the uniform binary search.

Binary Search


# WebRTC

## Summary

WebRTC is a technology that brings real-time communications (RTC) capabilities to the web by natively making these part of a web browser. It also adopts open patent-free components to make this technology available to everyone. For both developers and users, WebRTC lowers the barrier to entry to develop and experience RTC in apps. WebRTC is expected to enable innovative RTC apps for the web.

WebRTC signalling requires a server to set up the connection. Once set up, media and other data can be exchanged directly between peers. Only in some cases when a direct peer-to-peer connection is not possible, another server is needed to relay the media.

## Discussion

### Why was WebRTC invented?

Voice and video calling is something that only telecom companies provided in the past. Later, with VoIP, these calls could be delivered over the internet by even non-telecom companies. Skype, Viber, WhatsApp and WeChat are some examples of voice or video calling services on the internet. But the problem was that to use them users needed to download apps or install plug-ins in their web browsers. For developers, it was difficult to add real-time capabilities to their products and maintain them for different platforms, devices and browsers. Licensing fees meant that only large corporations could afford them.

WebRTC solves this by embedding real-time communication capabilities into web browsers. Since web users already have browsers, they need not install anything extra. Since WebRTC is part of the browser, developers need not build RTC capabilities from scratch. Developers can focus on building innovative applications powered by WebRTC technology. More importantly, building apps with RTC capabilities is no longer limited to VoIP experts. All web developers can get into it.

WebRTC is open and uses no proprietary or patented components. Thus, even small firms or individuals can get into building WebRTC-based solutions.

### How has been the adoption of WebRTC?

As of January 2018, WebRTC is supported by many major browsers: Chrome, Firefox, Edge, Safari, iOS Safari, Samsung Internet. Object RTC API is supported only by Edge. Opera supports WebRTC. On others such as Internet Explorer, or on older versions of browsers, plugins such as OpenTok can be used.

It's been said that WebRTC is already in the background and vendors are focusing on delivering effective use cases. This suggests that WebRTC as a technology is stable and mature. Among real-world solutions, RBS is using it to give personalized customer service. Fluke is using it for their remote teams and service staff to collaborate. Esurance uses it to expedite claims processing. At scale, Google Meet and Hangouts is serving 4.7 million minutes of audio/video per day. On Facebook Messenger, 400 million uses voice/video chat a month. Discord is serving 9.5 billion messages a month. Amazon Chime for videoconferencing serves 24.8 million minutes a month. Peer5 is making 1 billion WebRTC connections a day.

### What are some use cases of WebRTC?

Anything that can benefit from media-rich content, can leverage WebRTC: telehealth, remote classroom, online tutoring, remote interviews, online customer service, online collaboration, cloud-based video conferencing, webinars with audience interaction, adding voice/video capability to traditional social/chat apps, multiplayer gaming, and more.

The use of the low-latency data channel leads to more use cases. By allowing camera access, you can upload your profile picture to a social website. You could sign into a service biometrically. You could record an interview and send the video to a recruitment company. Two devices can get connected directly without needing to trust a server somewhere. Low-latency broadcasting, peer-assisted data delivery, a web-based torrent client and a streaming media server are more use cases with the data channel.

WebRTC can be part of embedded devices such as set-top boxes, smart TVs, IP cameras, media-streaming dongles, etc. In other words, WebRTC has a place in the Internet of Things. Data channel can be used for alerts, collect sensor readings or control devices.

RFC 7478 identifies typical WebRTC use cases and translates them into requirements.

### What are the some benefits of using WebRTC?

WebRTC is an open standardized browser-based technology free of patents. Developers can easily enhance their traditional web apps with RTC capabilities. Browsers that support WebRTC will include APIs to manage peer-to-peer connections, transport layer, media engines and codecs, thus removing most of the complexities for web developers. It's also easy to interwork with widely deployed SIP/H.323 systems by adding a gateway or supporting WebRTC.

WebRTC video quality is better than Flash, besides the obvious cost savings in terms of licensing fees. Latency is reduced. Connection times are faster when JS WebSockets are used. Since media is served via browser, HTML5 can be used for customized UI designs.

Encryption is mandated for all communications including signalling. TLS, DTLS and SRTP are used to secure communications. Since it's part of a browser, the installation process is secure. Access to camera or microphone requires explicit user permission.

### What's the protocol stack of WebRTC?

WebRTC uses UDP at the transport layer due to its real-time requirements. SRTP and SCTP are used at a higher layer to multiplex streams. For security, DTLS is used. Because peers can sit behind firewalls and NATs, ICE/STUN/TURN are needed.

For signalling, TLS and TCP are used. To exchange session parameters, SDP is used.

### What are the standards that define WebRTC?

WebRTC is standardized by two bodies:

    + **IETF**: IETF specifies formats, protocols and security aspects that enable peer-to-peer WebRTC communications. This is the transport part, what goes "on the wire." Visit WebRTC status page for current status.
    + **W3C**: W3C specifies browser APIs that developers can use from their apps. This is the access and usage part. Visit WebRTC status page for current status.As of January 2018, WebRTC 1.0 standard isn't out yet. Browsers currently implement the proposed APIs with special prefixes. Meanwhile, Object RTC, which is being referred to as WebRTC 1.1, uses JS objects to provide low-level control of media parameters. While such control is possible with SDP, it would require browser changes.

WebRTC does not standardize the signalling. Vendors are free to choose any signalling and there are plenty of options: Socket.io, WebSocket, XHR/XMLHttpRequest , SIP, XMPP or even custom signalling. Thus, WebRTC does not replace SIP, which is a common myth. In fact, RFC 7118 talks about SIP over WebSocket, which could be used by WebRTC.

While signalling is not standardized, there's an IETF draft titled JavaScript Session Establishment Protocol that provides APIs to control the signalling.

### Could you describe a typical call flow in WebRTC?

Participating peers exchange signalling data via a server to setup a direct peer-to-peer connection. Once such a connection is setup, real-time media traffic is exchanged directly without involving the server. Since peers often sit behind a firewall or a NAT device, they need to know the public IP address and port of the other peer for direct media exchange. This is done using the *ICE framework*.

ICE uses a *STUN* server to figure out the public address and port of peers. ICE first tries for UDP. If that fails, it attempts for HTTP over TCP. If that fails, it attempts HTTPS over TCP. If STUN succeeds, the media route is direct peer-to-peer. If direct peer-to-peer is not possible, ICE involves a *TURN* server. In this case, media is sent to the TURN server, which simply relays it to the other peer. STUN servers are typically used for asymmetric NAT and TURN servers for symmetric NAT.

### As a developer, what do I need to build a WebRTC-enabled app?

Your app should call essential WebRTC APIs:

    + `MediaStream`: Access content streams from camera and microphone. A `MediaStream` object can contain one or more `MediaStreamTrack` objects. For example, audio is a track and video is another track synchronized to the audio. API `getUserMedia()` informs what the app requires, obtains user permissions and then accesses media streams.
    + `RTCPeerConnection`: Gather and exchange local media capabilities and resolve addresses before sending and receiving audio/video content.
    + `RTCDataChannel`: Send and receive non-media content, if your app deals with arbitrary data.You should decide on a method of signalling suited for your app. You will need a server to handle this signalling to set up peer-to-peer connections. You will also need to implement the ICE framework along with STUN and TURN servers. In the case of multiparty calls, you will also need SFUs and MCUs. If this sounds too complex, you should prefer a platform provider to take care of most server-side stuff: Tokbox, Twilio, Agora.io, Xirsys, Temasys, Kandy, etc.

### What metrics should I look at to assess performance of a WebRTC application?

The following are some common metrics:

    + **Bandwidth**: End-to-end bandwidth matters. Symptoms of low bandwidth include video freezing, frame-rate drops, choppy audio and call drops.
    + **Latency**: A latency that's below 100 ms is usually not noticeable. High latency affects video more severely than audio. Symptoms of a high latency connection include lag in playback, video freezing and frame-rate drops.
    + **Loss**: Packets are sometimes dropped or lost, particularly on wireless links. Symptoms include video freezing, frame-rate drops and choppy audio. In RTC, loss is often preferred over playback lag due to NACKs and retransmissions. With video, NACKs may be used and when those fail a full frame is requested, which incurs a high bandwidth cost. On a congested network, lowering the bitrate is a suitable approach.
    + **Jitter**: This is common in high-latency networks or when packets traverse a number of hops. Symptoms are similar to packet loss but these symptoms are alleviated when the jitter buffer adapts to network conditions.

### Are there any problems or challenges with WebRTC?

There have been some challenges with WebRTC. Many enterprises are still using legacy browsers including Internet Explorer, which don't support WebRTC. Multiparty video is a drain on battery and taxes processing power available on laptops or mobiles. Enterprise deployments are complex and often involve multiple servers that scale very differently.

From a standardization perspective, developers are not happy that Object RTC is being proposed while WebRTC 1.0 hasn't been finalized yet. However, ORTC is an evolution of WebRTC 1.0 and parts of ORTC are already into WebRTC 1.0.

Another challenge may be that different browsers support different video codecs, leading to interoperability problems.

## Milestones

2009

The idea for WebRTC is born in Google to bring real-time capabilities to the web with built-in support from browsers. Alternatives at the time were Flash, which was low quality and needed server-side license; or plug-ins that needed to be developed for various browsers/platforms. Moreover, users needed to download and install plugins or custom apps.

May
2010

Real-time communications need audio/video codecs. In the spirit of openness on the web, Google releases WebM as a patent-free alternative to H.26x codecs. WebM can be traced back to the work of On2 Technologies that Google acquired in 2009. WebRTC uses codecs and media frameworks that are free of patents and royalties.

Oct
2010

A one-day workshop is organized to kickoff the standardization process. Standardization involves IETF to define the formats and protocols, and W3C to define APIs that web apps can use on the browser side.

May
2011

Google open sources WebRTC under a permissive BSD license. Thus, the term "WebRTC" often refer to the standard and also the Google's open source project hosted at webrtc.org.

Nov
2011

Chrome browser adds WebRTC support.

2013

Firefox starts supporting WebRTC. Chrome and Firefox on Android also start supporting WebRTC.

Nov
2017

W3C releases WebRTC 1.0: Real-time Communication Between Browsers as a Candidate Recommendation.

2020

In the year of the COVID-19 pandemic, the use of WebRTC from within Chrome browser sees 100X increase. During this year, Chrome also becomes 30% more battery friendly for video calling.

Jan
2021

W3C publishes WebRTC 1.0: Real-time Communication Between Browsers as a W3C Recommendation. Beyond what's been standardized, improvements continue. Video codec **AV1** that saves up to 50% bandwidth is becoming available in WebRTC implementations. **WebRTC NV** is looking into supplementary APIs to enable new use cases. These are expected to complement 5G and WebAssembly technologies.

WebRTC


# IEEE 802.11ax

## Summary

IEEE 802.11ax standard is an evolution of 802.11ac. Unlike previous standards that focused mainly on increasing raw data rates, 802.11ax focuses on better efficiency, capacity and performance. This translates to a 4x improvement in average throughput per user and better user experience. This is true even for dense indoor/outdoor deployments. Changes that enable these include higher modulation, more OFDM sub-carriers, and longer OFDM symbol; multiplexing users via MU-MIMO, beamforming and OFDMA; scheduling uplink instead of contention; and mitigating co-channel interference via BSS colour codes.

IEEE 802.11ax is also called *Wi-Fi 6* or *High Efficiency WLAN (HEW)*. While 802.11ac used only the 5 GHz band, 802.11ax addresses both 2.4 and 5 GHz bands, thus staying backward compatible and becoming the migration path for 802.11n and 802.11ac devices. With the extension *Wi-Fi 6E*, the standard supports 6 GHz bands.

## Discussion

### Why do we need 802.11ax?

IEEE 802.11ac increased raw data rates but left some problems unsolved. Uplink access is mainly based on contention. When there are many devices in a dense network, or multiple access points closely deployed, there can be collisions, backoffs and therefore reduced effective throughput. User experience is affected for all devices.

The common use cases where this happens is at crowded public hotspots (airports) or event venues (football stadiums). It could also happen in apartment complexes, schools and educational campuses. Also, it's expected that by 2022 number as many as 50 Wi-Fi devices may be present in a smart home. This growth is mainly due to appliances and gadgets becoming IoT-enabled. IEEE 802.11ax aims to solves these problems from the perspective of overall network capacity utilization, efficiency, performance, user experience and reduced latency.

### What are the main PHY changes 802.11ax brings compared to earlier 802.11 amendments?

802.11ax supports both 2.4 GHz and 5GHz bands. Therefore it's backward compatible with both 802.11n and 802.11ac, meaning that legacy clients can also connect to an 802.11ax AP, and vice versa. 802.11ax uses 4x larger FFT by increasing the number of subcarriers but also narrowing the subcarrier spacing, thus preserving channel bandwidth. OFDM symbol duration and cyclic prefix are increased for better performance in outdoor environments. For higher data rate for indoor environments, 1024-QAM and shorter cyclic prefix are introduced.

While in 802.11ac Wave 2 only 4 simultaneous MU-MIMO streams were possible, this is increased to 8. MU-MIMO in the uplink is introduced while in 802.11ac Wave 2 this was possible only in the downlink. AP will send trigger frames to coordinate uplink MU-MIMO.

OFDMA is introduced for the first time in Wi-Fi, similar to how it's done in 4G cellular. With OFDMA, multiple users can be transmitting at the same time, each using its allocated set of OFDM subcarriers. Subcarriers are grouped in *Resource Units (RU)*, which are allocated.

### What are the main MAC changes 802.11ax brings compared to earlier 802.11 amendments?

Since 802.11ax is meant to solve the use case of high density networks, uplink access is scheduled and not based on contention. A new feature named *Target Wake Time (TWT)* is used to let stations sleep, save power and wake up at scheduled times. AP can thus do scheduling a way that minimizes contention among stations. This can also be seen as a load balancing technique to ease congestion.

In dense networks, a neighbouring AP can cause cochannel interference. Stations on the overlapping areas will backoff excessively. This is alleviated in 802.11ax by a feature called *BSS Color*. This helps stations to identify if transmission is from another network and thereby take the right action.

### For multiplexing users on the uplink path, we have both MU-MIMO and OFDMA. How are they different?

MU-MIMO increases overall capacity since multiple streams are transmitted at the same time. This is ideal for bandwidth-demanding applications. Transmission to each user is targeted via beamforming. OFDMA does not increase capacity but uses it more efficiently by allocating subcarriers to users based on their needs. With OFDM, a user would occupy all subcarriers for a given time even if that user doesn't have much to send. With OFDMA, multiple users can be multiplexed at the same time, each using different sets of subcarriers. This implies that OFDMA is suited for low-bandwidth applications. Users will also experience less latency with OFDMA. With OFDMA, multiple users with varying bandwidth needs can be scheduled at the same time.

Thus, MU-MIMO and OFDMA complement each other. In typical deployments, performance of MU-MIMO in 802.11ac Wave2 was found to depend on distance between AP and clients, channel selection, antenna performance, presence and capability of other clients, etc. Some have noted that MU-MIMO may even result in lower throughput.

### What is Wi-Fi 6E?

Wi-Fi 6 is the commercial term for IEEE 802.11ax in 2.4 and 5 GHz spectra. Operation in 6 GHz, first announced in 2020, is known as Wi-Fi 6E.

Wi-Fi 6E brings the main features of Wi-Fi 6 to 6 GHz including OFDMA, Target Wake Time and WPA3. While Wi-Fi 6 can deliver a maximum data rate of 9.6 Gbps (1.5 Gbps/device), Wi-Fi 6E can deliver 9.6 Gbps (2.3 Gbps/device). Latency with Wi-Fi 6E can be below 1 ms.

Unlike at 2.4/5 GHz spectra, there are less interfering devices at 6 GHz. The new spectrum brings greater capacity. It opens up wide contiguous spectrum to enable high-bandwidth ultra-low-latency applications. These include online gaming, virtual reality, high-definition video calls, IoT device management, gigabit connectivity at event venues, and more.  Wi-Fi 6E is expected to power 5G services. Indoors, where GPS is not accurate, Wi-Fi 6E can enable more precise location tracking for applications such as asset tracking, emergency services, and keyless entry.

### What bands are supported by Wi-Fi 6 and Wi-Fi 6E?

IEEE 802.11ax operates in bands 2.4/5/6 GHz. At 2.4/5 GHz bands, the channels are the same as in IEEE 802.11n/ac. However, 802.11ax uses the spectrum more efficiently due to OFDMA.

At 6 GHz spectrum (5925-7125 MHz), 1.2 GHz of contiguous unlicensed spectrum is available. Wi-Fi 6E adds capacity with fourteen 80 MHz channels or seven 160 MHz channels. The spectrum is composed of four channel allocations in the US, termed as U-NNI-5 to U-NNI-8, given Unlicensed National Information Infrastructure (U-NII). U-NNI-5 (5925-6425 GHz) is also allocated in the EU.

Deployments at 6 GHz can be at very low power, low power or standard power with maximum EIRP of 14, 30 and 36 dBm respectively. Low power operation is only for indoor coverage. Standard power operation is at U-NNI-5 and U-NNI-7 and must be controlled by an Automated Frequency Coordination (AFC) system. This is to mitigate interference to and from incumbent devices in the vicinity. AFC is not required for very low power and low power operations.

### What are possible implementation challenges for 802.11ax?

OFDM subcarrier spacing narrower at 78.125 kHz, which implies that oscillators must have better phase noise performance and RF frontends must have better linearity. Since 1024-QAM is possible, EVM requirements are tighter. Good performance requires tight frequency synchronization and clock offset correction. Stations must also maintain frame timing based on their clocks since their transmissions must be in coordinated precisely as noted in trigger frames.

## Milestones

May
2013

Within the IEEE, High Efficiency WLAN (HEW) Study Group is formed. A year later, HEW Task Group is formed to start the development of 802.11ax standard.

Oct
2016

Quantenna announces world's first 802.11ax chipset for APs. Named QSR10G-AX, it supports 8 streams at 5 GHz and 4 streams at 2.4 GHz.

Feb
2017

Qualcomm announces chips IPQ8074 (for routers/APs) and QCA6290 (for clients). It's expected that router devices will come out first (end 2017) followed by client devices (2018).

Aug
2017

Broadcom announces three chips as part of its 802.11ax family without support for MU-MIMO in the uplink.

Sep
2017

**Draft 2.0** of 802.11ax is released. This may be incompatible with Draft 1.0 released in 2016 and mixing these two implementations within a Wi-Fi network might result in sub-optimal performance. By now, chipsets are already available from Broadcom, Qualcomm, and Quantenna. Meanwhile, Asus shows off an 802.11ax router that can deliver 4.8 Gbps but this is not yet in commercial availability.

Jan
2018

Intel announces that it will release 802.11ax chipsets some time this year. The chipsets will be for 2x2 and 4x4 home routers and gateways, supporting as many as 256 devices sharing bandwidth simultaneously.

Jun
2018

Asus launches two 802.11ax routers, one of them capable of aggregate peak throughput of 11 Gbps to cater to high-bandwidth multiplayer gaming.

Feb
2019

Samsung Galaxy S10 becomes the first smartphone to support Wi-Fi 6.

Sep
2019

To certify devices for Wi-Fi 6, Wi-Fi Alliances launches **Wi-Fi Certified 6** program. Subsquently, Samsung Galaxy S10 becomes the first smartphone to be certified under this program. Also in September, Apple iPhone 11 starts supporting Wi-Fi 6.

Nov
2019

**Draft 6.0** is released.

Dec
2019

IEEE announces that 802.11ax meets requirements for 5G Indoor Hotspot and Dense Urban test environments of the enhanced Mobile Broadband (eMBB) usage scenario. This includes minimum acceptable peak performance, average and cell-edge user experience, mobility performance and latency performance.

2020

This year sees the first developments of **Wi-Fi 6E**. Wi-Fi Alliance announces the branding early in the year. Spectrum at 6 GHz is opened up in America, Europe and Asia. First chips are demonstrated. Actual products appear on the market in 2021, with Wi-Fi Alliance launching a certification program in January 2021. In 2022, we expect 350 million Wi-Fi 6E devices to enter the market and 15% of all 802.11ax shipments will be Wi-Fi 6E.

May
2021

IEEE publishes the standard **IEEE 802.11ax-2021** as Amendment 1 of the IEEE 802.11-2020 standard. The amendment was formally approved towards the end of 2020 and early 2021.

IEEE 802.11ax


# Blockchain Consensus

## Summary

Blockchain is a distributed peer-to-peer technology. All nodes in the network have to agree on the state of chain and what are its valid blocks. Since there's no centralized control, and nodes cannot be trusted, reaching this agreement is not trivial. Every blockchain implementation must therefore define what's called a *consensus algorithm* to arrive at an agreement. This is also called *consensus protocol*.

Consensus is not exclusive to blockchain. It's a classical problem in distributed computer systems. In fact,

> Any algorithm that relies on multiple processes maintaining common state relies on solving the consensus problem.

## Discussion

### What's the purpose of a blockchain consensus algorithm?

A permissionless blockchain is open. Anyone can be a node and remain anonymous. It's therefore possible for a node to tamper transactions and include them in a new block. Thus, the blockchain can end up with what we call a *fork*. For example, one fork in the chain contains the tampered transaction while the other contains only valid transactions. The purpose of a consensus algorithm is to avoid such forks so that everyone agrees to a single version of truth.

With a permissioned blockchain, although participating nodes are known and chosen, consensus is still required because we can't assume that the nodes are trustworthy.

From the general viewpoint of distributed systems, consensus is a challenge when nodes are either faulty (gone rogue) or unable to communicate reliably. The former is called *Byzantine Generals Problem* and the latter is called *Two Army Problem*. A consensus algorithm must therefore be fault tolerant.

### How does Bitcoin achieve consensus?

Bitcoin uses *Proof of Work (PoW)* for consensus. Before a new block can be added to the chain, a node must perform an expensive computation to arrive at a valid hash value for the block. All other nodes receiving this block can then quickly verify that this block is indeed a valid one. A valid block is therefore proof that the node submitting it has done the necessary work.

In fact, all nodes do this work in parallel but whoever is fastest (has most compute power) will win the race. The winning node is rewarded with *bitcoins*. For this reason, the work done by nodes is called *mining* and the nodes themselves are called *miners*. We can therefore say that consensus in Bitcoin network is achieved via mining.

Consensus is achieved by a simple rule that only the longest fork will survive. In other words, the fork on which most compute power has been expended (PoW) will survive. If two blocks are mined at the same time, there will be a fork. PoW therefore intentionally slows the mining process so that forks don't happen faster than they are discarded by the network.

### What are some possible attacks on blockchain?

The idea of any attack is to prevent nodes from reaching consensus or mislead them to a wrong consensus. Here are a few common attacks:

    + Denial of Service (DoS): Sending nodes lots of transactions will prevent them from working on legitimate ones. Distributed DoS is a variant of this.
    + 51% Attack: If the attacker controls more than 50% of the nodes, then he can influence the consensus process. He can include modified transactions, cause a fork and make that fork the longest. Even with less than 50% controlling power, some attacks can happen with 50% success rate.
    + Double-Spend: Applicable to cryptocurrencies, this is a case when the same coin is used for multiple transactions.
    + Sybil Attack: This happens when a node assumes multiple identities and tries to pass itself off as multiple nodes in the network.
    + Cryptographic Attack: Quantum computing will bring computing power 100 million times that of conventional computers. This shifts the balance in favour of nodes with such power.
    + Byzantine Attack: A single or few nodes prevent consensus.Bitcoin-specific attacks are noted in a 2017 survey paper.

### What are the different types of blockchain consensus algorithms out there?

There are plenty of them and the following are some well-known ones:

    + **Proof of Work (PoW)**: An expensive computation is required and this can be verified by other nodes. Nodes can remain anonymous and anyone can join. PoW is synonymous with mining. Systems that don't use PoW can be said to be doing *virtual mining*.
    + **Proof of Stake (PoS)**: Stakeholders are those having coins or smart contracts on the blockchain. Only they can participate. Those with high stakes are chosen to validate new blocks. They are rewarded with coins. While coins are "mined" in PoW, they are "minted" in PoS. Blocks may still need to be signed off by other nodes before added to the chain.
    + **Delegated Proof of Stake (DPoS)**: In PoS, those with large stakes can take control. In DPoS, delegated nodes represent the interests of smaller nodes.Further examples of consensus algorithms are proof-of-activity, proof-of-burn, proof-of-capacity, Transaction As Proof Of Stake (TAPOS), delegated Byzantine Fault Tolerance (dBFT), Practical Byzantine Fault Tolerance (PBFT), Federated Byzantine Agreement (FBA), and proof-of-weight.

### Could you explain Intel's proposed Proof of Elapsed Time (PoET)?

Proof-of-work wastes electricity to do mining. Essentially, the idea is to delay the creation of new blocks so that the network has enough time to avoid forking. Intel's idea is to mimic PoW by guaranteeing that each node has waited long enough before creating a new block, as if they were "doing work". They achieve this by building in their hardware chips a *trusted execution environment (TEE)* that adds this delay. It's claimed that proof-of-elapsed-time can scale to thousands of nodes.

One criticism of this consensus is that nodes have to put their trust in Intel, who is supplying the chips. This is against the ethos of blockchains that aims to avoid centralization in an trustless environment.

### Could you name the consensus algorithms used by some well-known blockchains?

Bitcoin uses PoW and so does Ethereum, Litecoin and Dogecoin. Since PoW is computationally expensive, Ethereum plans to move to PoS sometime in 2018. Tendermint, BlackCoin and NXT use PoS. Peercoin and Decred use PoS while Bitshares, Steemit and EOS use DPoS.

Hyperledger uses PBFT while Stellar and Ripple use FBA. MultiChain uses a variant of PBFT where there's only one validator per block and multiple validators work in round-robin fashion. Chinese platform NEO uses Delegated BFT.

Proof-of-authority is used by POA.Network and Ethereum Kovan testnet. Proof-of-weight is used by Algorand, Filecoin and Chia.

### What are some essential criteria of a blockchain consensus algorithm?

The most important one is perhaps decentralization. In this aspect, Bitcoin's PoW does better than other consensus protocols. Consensus algorithms have to consider scalability, privacy, security and fault tolerance. Security implies resistance to attacks. Node identity management, consensus finality, correctness proofs and assumptions about network synchrony are other parameters to consider.

The algorithm must scale with number of nodes/clients. Bitcoin's PoW can support thousands of nodes/clients but suffers in terms of performance. BFT can support thousands of clients at better performance but is limited to less than 20 nodes. Performance relates to throughput (transactions per second or tps) and latency. It must be efficient in terms of power consumption, or in general, consumption of any resource.

Current protocols often involve tradeoffs. PoS is more energy efficient and achieves higher throughput than PoW but is more centralized and vulnerable to Byzantine faults. Prometheus and Matrix are newer algorithms that aim to combine the best of both PoW and PoS.

### I've heard of "blockchain killers". What are these?

**PARSEC** is a blockchain-less open source, fully asynchronous, leaderless algorithm for reaching consensus in the presence of Byzantine faults in an asynchronous network. Algorithm correctness has been proved provided less than a third of the participating nodes are faulty. This model matches an agreement with probability of one.

Directed Acyclic Graph (DAG) is a graph of nodes with topological ordering and no loops. DAGs are inspiring the next generation of "blockchains". Iota, Hashgraph and Nano are examples of DAG.

**Tangle** is the DAG consensus algorithm used by Iota. To generate an Iota transaction, a node has to validate two previous transactions it's received.

**Nano** uses block-lattice as its structure. Every transaction is made of send block on the sender's chain and receive block on the receiver's chain.

**Hashgraph** builds a directed graph of events rather the traditional chain of blocks. Consensus is via gossip: nodes tell neighbours about events, which are ordered by timestamps. Hashgraph is claimed to be fair, faster, provable and resistant to Byzantine attacks. It's also called *Swirlds hashgraph consensus algorithm*. It may not suit open public networks.

## Milestones

1982

Researchers at the SRI International in California publish a paper titled "The Byzantine Generals Problem". The papers starts by stating that "reliable computer systems must handle malfunctioning components that give conflicting information to different parts of the system". The paper then abstracts this scenario for distributed computer systems.

1993

Researchers at the IBM Almaden Research Center propose the use of *pricing functions* and *hash functions* to tackle the problem of spam emails. The idea is that an email sender must be required to perform a moderately expensive computation and the recipient can verify easily that such a computation has been performed before accepting the email. This will deter spammers. The idea of modern proof-of-work can be traced back to this research.

1997

Adam Black invents Hashcash, a PoW system to tackle email spam and denial-of-service attacks. Hashcash can be credited with making PoW popular.

Feb
1999

At the 3rd Symposium on Operating Systems Design and Implementation, researchers from the MIT Laboratory for Computer Science present "Practical Byzantine Fault Tolerance" as a solution to the Byzantine Generals Problem. Unlike previous solutions, this works in asynchronous systems and improves response times. This is further detailed in an ACM paper published in 2002.

2009

Bitcoin is introduced as a peer-to-peer cryptocurrency. It uses *Proof of Work (PoW)* for consensus.

2012

Since PoW used by Bitcoin is wasteful, *Proof of Stake* is proposed as a more energy efficient alternative. This later gives birth to the Peercoin network. PoS however suffers from the "Nothing-at-stake" problem.

Dec
2013

Daniel Larimer, founder of BitShares, proposes a new type of PoS in which stakeholders elect nodes who can sign the blocks. This is the genesis of *Delegated Proof of Stake* consensus. Elected nodes are called *witnesses* and they are trusted to behave correctly. The voting process is decentralized and fair. DPoS scales well because nodes need not wait for a minimum number of untrusted nodes to verify a block, which needs to be only signed by one or more trusted nodes.

Aug
2015

Vlad Zamfir of Ethereum announces that they have been working on a new consensus called *Casper*. It's a PoS type of consensus. Casper has mechanism in place to punish nodes that try to do exploit the "Nothing-at-stake" situation. Ethereum is expected to switch from PoW to PoS in 2018.

Apr
2016

Intel announces details of Sawtooth Lake project, a platform for running distributed ledgers. This talks about two consensus protocols: (a) *Proof of Elapsed Time (PoET)* that avoids the energy inefficiency of PoW. (b) *Quorum Voting* that adapts PBFT protocol and uses it for apps that require immediate transaction finality. In January 2018, Hyperledger Sawtooth 1.0 is released and PoET is part of it.

May
2018

MaidSafe announces a new consensus protocol called *PARSEC (Protocol for Asynchronous, Reliable, Secure and Efficient Consensus)* to power its SAFE Network. PARSEC is a "completely decentralised, open source, highly asynchronous, Byzantine Fault Tolerant consensus mechanism".

Blockchain Consensus


# Bad Requirements

## Summary

It's customary to develop product requirements before they're designed and implemented. A good requirement is clear, correct, complete, concise, well-structured, feasible, traceable, and verifiable. A bad requirement is just the opposite.  Bad requirements (aka poor requirements) can arise when requirements development is not done properly.

Bad requirements are costly, particularly when they're discovered late in the development lifecycle. Many software defects can be attributed to bad requirements rather than implementation bugs.

This article shows examples of bad requirements and how they can be improved. It shares some words and phrases to avoid when describing requirements. By following some best practices when developing and writing requirements, bad requirements can be avoided.

## Discussion

### What's a bad requirement?

Requirements are bad if they're ambiguous, incomplete, inconsistent, incorrect, outdated or technically infeasible. Sometimes they're written in highly technical jargon that only a few understand. Some bad requirements describe system internals without relating them to visible interfaces or scenarios.

Badly structured requirements are hard to trace. Multiple requirements written as a single descriptive sentence is also problematic. Unstructured requirements are hard to verify or validate. Requirements not essential or relevant to the system are equally bad. This practice is sometimes called "gold plating".

Some bad requirements describe how to build the system, thus imposing inappropriate constraints. Others describe system operations. Instead, they should set the expectation from subsequent phases (design, coding, integration, testing).

Requirements without suitable metadata is also a problem. Priority, status, estimated cost/schedule, and unique identification number are examples of metadata.

### What's the cost of bad requirements?

Bad requirements lead to more work, rework, and lower morale. This in turn causes cost overruns, delayed deliveries, missing features, and poor quality products. Requirements that impose constraints or describe a solution prevent innovative thinking. When requirements are not traced, impact analysis becomes difficult. Lack of requirements tracing is a technical debt. It becomes difficult to evolve the system. Highly volatile requirements incur expensive rework.

One study found that requirements and design phases contributed 64% of defect costs. Engineers spent 19 hours per day addressing requirements-related defects. Test engineers spent a lot of time clarifying requirements. As much of 10% of testing budget could have been saved with proper requirements engineering.

It's been found that fixing a requirements defect during integration and testing is as high as 78 times more expensive. Other studies showed that defects discovered in the field cost 50-200 times more. Rework is 50% of total development cost. 50% of defects and 80% of rework effort are due to poor requirements. Poor requirements have caused 40% of accidents in safety-critical systems.

### Could you share examples of bad requirements?

Here are some illustrative examples:

    + *The system must have good usability*: It's vague. It's unclear how to measure usability. Each user group or role interacts with the product differently.
    + *Response time should be less than 5 seconds*: Five seconds is an objective measure but there's no context. Product feature, scenario, system state or configuration are examples of context.
    + *The system has to be bug-free*: No system can guarantee this. Instead there should be well-defined processes for testing and change management.
    + *Round-the-clock availability*: Unrealistic and could be too costly in practice. Requirements should balance business, user, technical, and economic perspectives.
    + *The system shall work just like the previous one, but on a new platform*: Requirements can't easily be transferred across platforms or technologies. Newer technologies often offer better ways to implement features.
    + *Reporting*: Too brief. It's unknown what reports should include, the level of detail, who'll read them and towards what objectives.
    + *Authenticate with user IDs and passwords*: Imposes a particular solution rather than specifying a requirement.

### What words characterize bad requirements?

Even when requirements are correctly identified, they may not be written down correctly. Engineers are not necessarily good at writing. The choice of words matters. Words that're not precise enough should be avoided. One study grouped these and called them **requirements smells**. Here are some examples:

    + **Unquantified**: all, always, every, never.
    + **Ambiguous**: most, many, several, some, sometimes, usually, normally, ideally, optionally.
    + **Vague Adjectives**: easy, appropriate, user-friendly, fast, graceful, flexible, versatile, efficient, robust, adequate.
    + **Vague Verbs**: handle, improve, provide, reject, support, maximize, optimize.
    + **Implied Multiple/Missing Requirements**: and, another, etc., other, including, not limited to, such as, including, for example.
    + **Pronouns**: it, this.
    + **Implied Uncertainty**: should, can, could, if possible, may.Context is important. Consider the use of "and" in "DADS shall display the number of requests pending and requests processed." There are two requirements here. They must be separated. However, the following is a single requirement even when "and" is used: "DADS shall display the combined number of requests pending and requests processed."

### How can I avoid bad requirements?

Requirements development should involve all stakeholders, including testers and operational experts. Discussions should promote diverse perspectives. Involve all user groups. The idea is to fix problems early on. Formal inspections and informal reviews should complement each other.

Techniques for functional requirements may not suit non-functional requirements. Use different techniques to elicit and develop requirements: storyboards, interviews, prototypes, use case diagrams, class diagrams, data flow diagrams, state transition diagrams, etc. Consider both normal and abnormal scenarios. Use these techniques to identify and analyze requirements.

Use checklists and templates. Use tools to identify ambiguous words and phrases. Use tools to structure, track and trace requirements. Store requirements in a single repository. Link models and diagrams to requirements. Store metadata. Tools with these capabilities include Borland CaliberRM, IBM/Rational's RequisitePro, and Telelogic's DOORS.

Requirements and even project scope can change, particularly with long-running projects. Requirements engineering and project schedules must embrace and manage this change. Adopt iterative and incremental development methodologies.

## Milestones

1990

The U.S. Government Accountability Office recognizes that poor requirements are causing projects to fail. Through the 1990s, studies identify poor requirements as the main cause of missed deadlines and cost overruns.

1993

Hooks proposes the following uses for words: "shall" for requirements; "will" for statements of fact; "should" for goals. In 1995, the MIL-STD 961D standard reserves the word "shall" to identify requirements. Words "will", "should" and "must" can be used in descriptions but not in requirements.

1996

Alvarez et al. apply syntactic, semantic and domain analysis to specifications. While they develop some tools to do this, the focus of their research is to develop **metrics** to identify poor requirements. In the late 1990s, a similar research at NASA results in the development of the *Automated Requirements Measurement (ARM)* tool. This tool analyses text structure, specification depth and readability.

2005

Arnold and Martin give the example "The system shall transmit data at least 100 kilobits per second." If the system delivers 99.8 kbps, it clearly fails the requirement. This binary approach to defining requirements unnecessarily constraints the solution space and forces designers to make the wrong trade-offs. Hooks observed this in 1993 and recommended the use of allowable tolerances.

2006

Kasser et al. propose a way to define **metrics** for requirements. They adapt metrics used to assess code quality and apply them to requirements. They also consider metrics or indices commonly used to assess readability of English text. They use the presence of poor words to compute what they call the *Figure of Merit (FOM)*. These ideas are incorporated into tools named FRED, TIGER and TIGER Pro.

Sep
2009

The Requirement Analysis and Modeling Process (RAMP) project is started as a collaboration between academia and industry. It's objectives are to improve the writing of requirements in natural language, requirements analysis and context analysis. This research finds a gap between theory and practice. There are problems in formalization and consistency. Many requirements actually describe solutions.

2010

Hooks observes that people are still writing bad requirements despite the availability of many tools and techniques. Twenty years earlier he wrote an article titled *Why Johnny Can't Write Requirements*. Over these years he has learned that many things have to come together to produce good requirements: tools, techniques, training, good metrics, continuous validation, inspections, ask-not-assume approach.

2017

Femmer et al. coin the **Requirements Smell**. They identify nine smells and relate them to ISO 29148. They note that many automated tools currently available are inadequate in catching these smells. They develop a tool called *Smella*. Among it's techniques are POS tagging, morphological analysis, lemmatization, and dictionary search.

2022

A Deloitte white paper talks about the role of a **Requirements Engineer**. This role overlaps partially with the role of a software architect. The requirements engineer must be from a testing background. He/she must look at user scenarios and testability. He/she must address any gaps between developers and testers. In one project, Deloitte reduces requirements-related defects from 23% to 4%. This includes improvements to requirements, design, test data and test scripts.

Bad Requirements


# Session Hijacking

## Summary

A **session** is an interactive information exchange between two or more communicating devices, or between a computer and a user, in computer science and networking in particular.

A session is started at one point in time and eventually 'torn down' - that is, brought to an end - at a later moment. In a well-established communication session, more than one message may be sent in either direction.

A connection-oriented communication requires the establishment of a session. In connectionless communication modes, a session is also the first stage in transmitting data.

**Session hijacking**, often called **cookie hijacking**, is the act of taking control of an already established trustworthy session in order to steal or compromise the data of the user. In a session hijacking attack, the attacker takes control of the user's valid session, and sends packets to the server while impersonating the real user.

## Discussion

### What's the role of cookies in session hijacking?

**Cookie hijacking**, is the exploitation of a valid computer session in computer science. It is specifically applied to the theft of a magic cookie that is used to authenticate a user to a distant service. It's crucial for web developers because the HTTP cookies used to keep a session on many websites can be stolen by an attacker utilising an intermediary computer or accessing the victim's saved cookies.

For example: Justin receives an email advertising a sale at his favourite online retailer, which he clicks on and logs in to begin shopping. After which, a session is created on the server. This maintains the state and is referenced during his any future requests. The session ID generated is stored in cookies, URLs, and hidden fields of websites, based on predictable parameters like the current time or the user's IP address, making them easy to determine for an attacker. The attacker sends an email with his own session ID in the link to Justin. As Justin has saved credit card details for easy use in the future, the attacker after stealing the session uses the card details to go on a shopping spree.

### What are the different types of session hijacking?

An **active session hijacking** occurs when an attacker takes control of the victim's active session and begins to communicate with the server as a legitimate user. A common way to break a user's connection to the server is to flood the target system with a large amount of traffic. The attacker gets complete control over the session after putting the victim into offline mode. Using packet sniffing tools the attacker remains in stealth mode during this process, listening and monitoring the packets transiting the network.

In a **passive session hijacking attack**, the attacker listens to all of the data and records it for future attacks. In most circumstances, the attacker must start with passive mode to launch any type of hijacking attack. The attacker may not succeed in impersonating a user to the server unless the user session is still active. This is a typical man-in-the-middle attack because the attacker is in-fact between the user and the server modifying and sending the packet.

In **hybrid session hijacking**, the attacker uses both passive and active attack techniques to successfully execute the attacks.

### What are the levels of session hijacking?

Hackers use **Network level** session hijacking to personalise their assault and breach the protocol's data flow shared by all online applications. **TCP (Transmission Control Protocol)** allows dependable end-to-end communication over an unreliable network. The attacker prevents the client and the server from exchanging data by forging acceptable packets for both ends that appear like actual packets, taking command of the session. **UDP (User Datagram Protocol)** is weaker than TCP as it does not employ sequence/numbers. The hijacker fakes a server reply to a client UDP request before the server can react.

The attacker aims to hijack current sessions and establish new sessions with stolen data at the **Application level**. **HTTP (Hyper Text Transfer Protocol)** used by the World Wide Web specifies how messages are formatted and delivered between clients and servers, and what actions Web servers and browsers should take response to various instructions. The first step in hijacking HTTP sessions is obtaining Session IDs. The areas to get a session ID are: in the HTTP GET request URL obtained by the browser, by using cookies saved on the client's computer and within the form fields.

### What are the methods of session hijacking?

**Brute force** – The attacker guesses the session ID and uses it to hijack the session, usually when a website's security is low and the session IDs are short and easy to guess.

**Cross-site scripting** – An attacker finds security flaws in a web server and injects scripts into it, which gives the session ID to the attacker.

**Malware** – To hijack a session, cybercriminals can deceive one into opening a link that installs malware (malicious software) on the device. It finds a session, steals the session cookie, and delivers it to the attacker.

**Session Sidejacking** – An attacker obtains access to a user's network traffic, usually when the user connects to an insecure Wi-Fi network, and further monitors the internet user's network traffic using "packet sniffing".

**Session fixation** - An attacker creates a session ID and deceives the user into using it to establish a session, by sending an email to the user with a link to a login form for a website.

In the **man-in-the-middle (MITM) attack**, the attacker is placed between the victim and another system and acts as an unknown proxy for one/both of the two communicating destinations.

### Which are some session hijacking tools and approaches used by attackers?

**CookieCadger** is a free open-source tool for detecting "information leakage" in online applications. It can look for unencrypted data, including session cookies, on both wired and insecure Wi-Fi networks. CookieCadger is a graphical Java software that automates sidejacking and replay of HTTP requests in order to detect data leakage from applications that use unencrypted GET requests.

**DroidSheep** is an open-source Android app that uses "packet sniffing" to extract session cookies and other exposed data from unsecured Wi-Fi web browsing sessions. DroidSheep is a basic Android tool for stealing web sessions (sidejacking).

In May 2012, Google Play released an app called **WhatsApp Sniffer**. It may show messages from other WhatsApp users who were on the same network as the app user. WhatsApp used an XMPP(Extensible Messaging and Presence Protocol) infrastructure with encryption at the time, rather than plain-text communication.

**FireSheep** is a Firefox browser add-on. The FireSheep addon allowed attackers to identify and copy unencrypted session cookies that might be exploited in session hijacking attacks via "packet sniffing." FireSheep took advantage of security flaws and is no longer compatible with the FireFox browser.

### How can session hijacking attacks be prevented?

The session ID, a piece of data used in network communications to identify a session, is recommended. This lowers the chances of an attacker guessing a valid session ID via trial and error or brute force attacks.

When you use public Wi-Fi, also use a **VPN (virtual private network)** to assist stay safe and prevent session hijackers out of your sessions is one of the best practices on the client-side. The data sent and received using a VPN is encrypted.

Prevent clicking on any unconfirmed link in an email unless it comes from a legitimate sender to avoid falling victim to a scam, on the client-side.

Install reliable **security software** on devices to identify viruses and guard against malware, on the client-side including the malware used by attackers to hijack sessions.

Keep an eye on the site's security. Session hijacking is prevented by measures in place at reputable banks, email providers, online retailers, and social media sites. Smart website owners will enable HTTPS (Hypertext Transfer Protocol Secure) across the board, not only on the homepage. They'll also quickly identify and close security flaws. This a server-side prevention technique.

### How does an attacker identify vulnerabilities?

These vulnerabilities (weaknesses/defects that can be exploited by an attacker) make session hijacking possible:

**Session IDs created using Fragile Algorithm:**

A session ID is a data piece used in network communications to identify a session. Websites generally create them using linear algorithms with easily calculable variables like time or IP address. By submitting a large number of requests, an attacker can observe a chronological pattern and generate a legitimate session ID.

**Undefined Session-Termination Time:**

An attacker has plenty of time to guess a legitimate session ID if the session expiration time is unknown. On obtaining a user's cookie file, the attacker can exploit passive session IDs to get access to the user's web account.

**Plain-Text Transmission:**

The unencrypted data delivered over the network can readily be sniffed for Session ID. SSL is used to encrypt data during transmission and prevent it from being sniffed.

**Small Session IDs:**

They can be easily determined by using a strong cryptographic technique. The attacker can even figure out the session ID pattern.

**Indefinite attempts:**

For websites without an account lockout, the attacker can make multiple attempts with different session IDs until the original session ID is discovered.

## Milestones

2007

Warning of webmail wi-fi hijack. Using public Wi-Fi hotspots has become a lot riskier, as security experts have revealed techniques that may capture login data over the air.

2010

Firesheep, a Mozilla Firefox extension, is launched, providing a simple way for session hijackers to attack users of unencrypted public Wi-Fi. Websites such as Facebook, Twitter, and any others that the user adds to their preferences allow the Firesheep user to quickly access private information via cookies, endangering the personal property of public Wi-Fi users.

2011

Facebook is now SSL-encrypted throughout. Not only during log-in, but also on all other pages of the Facebook social networking site, secure data transmissions via SSL are now available. As a result, even cookies are now transferred in encrypted form, and attackers employing tools like Firesheep can no longer read and exploit them for fraudulent purposes.

2011

Twitter adds "Always use HTTPS" option.Twitter implements a new feature that allows users to always use HTTPS when accessing Twitter.com, delivering secure data transmissions via SSL not only during log-in, but also for other pages. This means that even cookies are now transferred in encrypted form, and attackers employing tools like the Firesheep plugin for Firefox can no longer read and exploit them for fraudulent purposes.

2012

A Google Play app called WhatsApp Sniffer is released. It may show messages from other WhatsApp users who are on the same network as the app user.
 WhatsApp connects using an encrypted XMPP infrastructure rather than plain text.

Session Hijacking


# Developing AR Apps

## Summary

Augmented Reality (AR) started with simple apps that offered either location-based information or visualization based on the scanning of 2D markers. As AR moves into the domain of sophisticated image processing, 3D rendering and interactions, there's a need for better AR authoring tools.

In general, 3D tools are complex and difficult to master. Creating high quality 3D models also require particular expertise. If AR has to take off, authoring tools must offer simple and intuitive workflows. This is already happening. The idea is to enable domain experts, even if they lack 3D modelling expertise, to easily create AR experiences. It's expected that the AR ecosystem will eventually offer tools and SDKs for developers at all levels.

## Discussion

### What sort of data can I use for AR content?

AR content can include a variety of data input sources: CAD files and wireframes, 3D video, holographs, 360-degree images, 3D representations of real environments, and 3D sound. Increasingly, authoring tools will provide ways to integrate 3D design with AR. For example, 3D gaming engine Unity can import ARToolkit for tracking purpose. In 2015, CAD vendor PTC acquired AR authoring system Vuforia.

Authoring systems are also capable of exporting AR content targeting a variety of devices: smartglasses, smartphones, tablets, head-mounted displays, etc.

### What's a typical AR app development workflow?

For any project, it's customary to identify the problem, propose a solution or approach, list the requirements, study technical feasibility, select suitable tools and technologies, etc. Specifically for AR app development, there are some unique things to consider:

    + **Data acquisition**: In medical field, for example, data can come from CT or MRI scans. In retail, data may come from printed catalogues. In engineering, data may come from sensors. To build accurate models, data may need to be cleaned and preprocessed.
    + **Modelling**: 3D models are generated based on the data. If models are already available, then they may be simply imported into AR authoring environments.
    + **Interface development**: Developer defines the scenes and their transitions. Models are imported into scenes. Within scenes, user interactions are enabled, such as clicking, zooming or rotating 3D models. The AR experience is then tested within the development environment and later on target devices.

### As an AR app developer, what tools should I adopt?

Because 3D content is an important aspect of AR, and many game engines already work with 3D content, game engines are commonly adopted by AR developers for authoring AR experiences. Top game engines include Unity and Unreal Engine. The development process can be simplified by using an AR SDK. Popular AR SDKs can be imported into game engines. For example, Vuforia can be used from with Unity.

When selecting an SDK look for these capabilities: licensing, supported platforms (iOS, Android, Universal Windows Platform), target devices (phones, tablet, smartglasses, HoloLens, etc.), supported game engines, recognition (cloud or on-device), 3D tracking, geolocation, Simultaneous Localization and Mapping (SLAM)...

AR SDKs are many: PTC Vuforia, Apple ARKit, Google ARCore, Wikitude, Kudan, MaxSt, EasyAR, and more. Some SDKs may be better at marker-based AR while others may be better at SLAM. Some SDKs such as Wikitude and Vuforia allow the use of other SDKs such as ARKit.

### Do I need specific math skills to be an AR developer?

When creating basic AR experiences, the use of AR authoring tools is more important. Developers can get by with basic math skills. For example, in Unity, you can scale or rotate 3D models by simply changing some settings. With Google's Sceneform SDK and its runtime API, developers can avoid learning the whole 3D development stack. With Blippar SDK, object recognition is made easy by calling computer vision APIs.

For more complex AR experiences, knowledge of linear algebra, vectors and matrices will help. To build or manipulate 3D models, 3D math, 3D design, rendering and UX design become important. When handling sensor data, you need to know about transforms and filters. These are the same skills that game developers possess. Thus, game developers could move into AR development roles.

Increasingly, Machine Learning is being used to process sensor data including live video feeds. This would mean that knowledge of statistics and ML models is relevant.

### Could you share some tips for creating compelling AR apps?

John Fan, CEO and Founder of Kopin, gives us five rules for building great AR:

    + Humans first: User comfort must come before technology. To wear something, users must be convinced of its real value.
    + Physical world first: Too much virtual content can overwhelm users. Use overlays only to complement real-world experiences.
    + Maintain situational awareness: All senses must continue to be in touch with the real world. Physical reality cannot be replaced.
    + Voice is the new touch: Keyboards and touch screens are compromises. Voice should be used effectively for command and control.
    + Balance design with benefits: Don't overdesign. Remove unnecessary features. Design for specific AR benefits.

## Milestones

2014

Google announces **Project Tango**. It enables AR by its ability to map its surroundings using sensors, motion tracking camera, 3D depth sensing, depth perception and area learning. It's available in limited devices that meet its hardware requirements. With the coming of ARCore in 2017, it's discontinued in 2018.

Jun
2017

Apple announces its AR platform named **ARKit**. This uses *Visual Inertial Odometry (VIO)*. Using camera and motion sensors, it marks the environment with a bunch of points and tracks them as the user moves around.

Aug
2017

Google releases **ARCore**, an AR SDK for the Android platform. Unlike Google's Tango, ARCore can work on many Android phones without any specialized hardware. ARCore works with Java/OpenGL, Unity and Unreal. It's capable of motion tracking, environmental understanding and light estimation. While not as advanced as Tango, ARCore has potential to reach mass market.

Oct
2017

Unity 2017.2 is released with support for Vuforia, ARCore, ARKit and Windows Mixed Reality immersive headsets.

May
2018

ARCore 1.2 enables multiplayer AR across both Android and iOS. It's capable of wall detection. *Cloud Anchors* allows virtual objects to be placed in 3D space and synchronized to other devices via the cloud. **Sceneform SDK** is also released.

Developing AR Apps


# Regular Expression

## Summary

Regular expressions are essentially search patterns defined by a sequence of characters. Let's say, we wish to search for the substring 'grey' in a text document. This is a simple string search. What if we wish to search for both 'grey' and 'gray'? With simple string searches, we would need to do two separate searches and collate the results. With regular expressions, we can define a single search pattern that will give us matches for either of the two substrings.

In the above example, any of these patterns will work: `gr[ae]y`, `gr(a|e)y`, `(gray|grey)`, `gray|grey`

Regular expression is commonly known as **regex**. Although regex has a history going back to the 1950s,  it was popularized in computer science in the 1990s by the Perl programming language.  Apart from Perl's regex, many other variants exist.

## Discussion

### Could you give examples where regular expressions are used?

Regex is useful typically when the format or syntax of data is known. It's to this expected syntax that a regex is written. For example, email addresses are of the form `username@domainname.tld`. We can make a regex to extract all emails coming from a specific domain.

A common use of regex is for search-and-replace. For example, developers can use regex to quickly change the order of arguments across hundreds of source code files.

Regex can be used to process log files and filter log messages that match a particular signature. It can be used to filter reports on Google Analytics Dashboard.

Regex can also be used within database commands, say, to obtain usernames that contain non-alphanumeric characters in them.

Suppose we've reorganized the file structure in a web application. In web servers, regex can be used to match requested URL patterns and redirect them to new locations. Regex is also useful in web scraping tasks.

### What are the basic building blocks of regular expressions?

A regex has these basic building blocks:

    + **Anchor**: A regex is processed by a regex engine. Anchors make assertions about the current position of the engine. Common anchors are `^` (beginning of line) and `$` (end of line). For a multiline input, we can use `\A` and `\z` to match beginning and end of string respectively.
    + **Character**: `.` matches any character; `\s` matches any whitespace; `[12a-c]` matches the set of characters '1 2 a b c'; `(a|e)s` matches 'a' or 'e' followed by 's'.
    + **Group**: A sequence of characters grouped within `()`.
    + **Quantifier**: Specify how many matches of the character or group are allowed. For zero or more matches, use `*`; `+` for one or more matches; `?` for zero or one match; `{m,n}` for m to n matches.
    + **Modifier/Flag**: Modify the regex in specific ways. Use `i` for case insensitive match; `m` is for multiline match; `s` for single line match so that `.` matches newlines as well.For example, `/^model{1,2}(ing)? /i` will match any line starting with 'model ', 'modell ', 'modeling ', 'modelling ', and their lowercase versions.

### Which are the special characters in regex?

Regex special characters are often called **metacharacters**. The common ones include `\ ^ $ . * + ? [ { ( ) |` characters.

Note that characters `} ]` have special meaning too when used with their counterparts `{ [`; but on their own, they are treated literally. For example, `a(b{2,})?c` will match 'ac', 'abbc', 'abbbbc' and more; `ab}?c` will match exactly either 'abc' or 'ab}c', and nothing else.

Since metacharacters have special meaning, to use them literally, we would have to escape them with the backslash `\` character. For example, to match the dollar value in string 'This costs $22.50 after discount', the regex would be `\$(\d+(?:\.\d+)?)`, where the dollar symbol is escaped. It's treated literally and not as end of the string.

Metacharacter pair `[]` defines a character class. When range is specified, such as `[0-6]`, hyphen character is a special character. However, if hyphen occurs without either start or end, such as `[-6]`, it's interpreted literally. Likewise, other metacharacters are interpreted literally within a character class. This is therefore an alternative to escaping with backslash.

### What are regex groups and how are they useful?

Parts of a regex can be grouped within parentheses, `()`. A quantifier can be applied to an entire group. Thus, `ab+` will match 'abb' but `(ab)+` will match 'ab' and 'abab' but not 'abb'. Groups are also useful to restrict alternation. Thus, `cat|dogs` will match either 'cat' or 'dogs' but `(cat|dog)s` will match either 'cats' or 'dogs'.

Groups are generally of three types:

    + **Numbered Capturing Group**: Each group within a regex is given a number starting from 1. The number 0 is reserved for the entire regex. This number can later be used within the regex for subsequent matches or in a replacement string. These are called *backreferences*.
    + **Named Capturing Group**: First introduced in Python, naming the groups make it easier to backreference. For example, we match one or more digits into a named group called 'score' using `(?P<score>\d+)` and backreference it as `(?P=score)`.
    + **Non-Capturing Group**: Sometimes we wish to match a group but we're not going to backreference it later. Therefore, there's no need to capture the group. For example, here we match one or more word characters without capture, `(?:\w+)`.

### Could you explain the concepts of lookahead and lookbehind in regex?

Sometimes we wish to match a string but only if it occurs in a certain context. For example, we wish to match all dark versions of colours that start with 'g': darkgreen, darkgrey, darkgray, etc. While `dark(g[a-z]+)` will do the job, an alternative is to match the base colour and look behind to see if it's dark: `(?<=dark)g[a-z+]`. The full match will return only the colour without the 'dark' prefix.

Looking before or after a particular match is called *lookaround assertion*. It's called an assertion because it doesn't consume characters from the input. They only declare if there's a match or not. We illustrate examples of the four lookaround assertions:

    + **Positive lookahead**: Match 'q' that's followed by 'u', `q(?=u)`.
    + **Negative lookahead**: Match 'q' that's not followed by 'u', `q(?!u)`.
    + **Positive lookbehind**: Match 'b' that follows 'a', `(?<=a)b`.
    + **Negative lookbehind**: Match 'b' that doesn't follow 'a', `(?<!a)b`.

### What are some other advanced regex features?

Some advanced regex features include the following:

    + **Laziness**: A regex such as `.*,` is greedy. It will match as many characters as possible preceding a comma. A lazy or non-greedy match is achieved using '?', such as `.*?,`. This will match only till the first comma.
    + **Possessive Quantifier**: Using '+', such as `.*+`, we can tell the regex engine to avoid unnecessary backtracking. This can be seen as a notational convenience for atomic grouping.
    + **Atomic Grouping**: These are special non-capturing groups to prevent unnecessary backtracking. Use of atomic grouping improves performance. An example of this is `(?>his|this)`. If 'his' is not present, obviously we don't need to backtrack and look for 'this'.
    + **Conditionals**: We can use lookaround assertions for specifying conditions. Depending on condition's result (true or false), other parts of the regex can be processed.
    + **Recursion**: Use `(?R)` to match nested constructs.

### Do regular expressions differ across programming languages?

Regular expressions are processed by regex engines, of which there are many flavours. A programming language typically adopts one of these engines. Differences across these flavours are not easy to remember. Programmers should be aware that migrating regex from one engine to another must be accompanied with proper testing.

Some of these flavours include JGsoft, .NET, Perl, PHP, R, JavaScript, Python, Tcl ARE, POSIX BRE, POSIX ERE, GNU BRE and GNU ERE. The oldest of these is the **Basic Regular Expression (BRE)** that has limited power and expressiveness. BRE was later extended to the **Extended Regular Expression (ERE)**. To study how these various engines differ, take a look at Roger Qiu's comparison chart or Wikipedia's entry.

As an example, *sed* tool uses BRE by default, where `+` matches literal plus sign. But with option -E it uses ERE, where `\+` must be used for literal match.

Since Perl popularized regex, one of the popular engines that came about is called **Perl Compatible Regular Expression (PCRE)**. PCRE was later updated to PCRE2. PCRE and its variants have been adopted by many programming languages.

### What are some tips or best practices for using regex?

For readable and maintainable regex, use the flag `x` to ignore whitespace.  Thus, a complex regex can be expanded with useful comments.

To find match in the right place, use anchors. The use of `.*` can make the engine backtrack often. Construct a more specific regex. Likewise, the use of lazy quantifier `{.*?}` can be inefficient. Instead, say what you don't want to match, `{[^}]*}`. Also, atomic groups can save on backtracking. However, lazy quantifiers can be better in some simple scenarios: `*?'s` is faster than `*'s`.

Regex must be designed to fail fast. For example, `(?=.*fleas).*` does a lot of backtracking on lines that don't contain 'fleas'. On the other hand, `^(?=.*fleas).*` has a lookahead anchored at the start of the string and will fail faster.

Use contrast, that is, what characters to match and what not to match. For example, to match 'ABC123' the regex `^.+\d{3}$` won't work since `.` and `\d` are not mutually exclusive. Instead, use `^\D+\d{3}$`.

When using alternations (use of `|`), put the more common patterns earlier.

### Where should I not use regular expressions?

While regex is useful, it can be overused and abused. An extreme example is a 6343-character long regex to match an email address. Here's a quote from 1997, attributed to Jamie Zawinski,

> Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

While URL paths and email addresses can be parsed using regex, there are dedicated and mature libraries to do these. You should prefer to use them instead. Regex is also not the best choice for parsing HTML or code since there are better tools to generate tokenized outputs.

Humans write in a number of different ways. A regex will not adequately capture all the different variations. In general, when code is read and maintained by many developers, complex regex will be problematic. Regex is generally not descriptive enough to match balanced parenthesis, such as, `(aa (bbb) (bbb) aa)`.

### Could you point me to useful resources for working with regex?

Among the places to learn regex are RegexOne, RexEgg, Regex Crossword and Jan Goyvaerts' Regular-Expressions.info. Beginners might like Net Ninja's series of sixteen videos on regex.

For absolute basics, consider reading Mike Malone's blog post from 2007.

A classic book on regex is *Mastering Regular Expressions* by Jeffrey Friedl. Another good book is *Regular Expressions Cookbook* by Jan Goyvaerts and Steven Levithan. Dave Child has published a handy regex cheatsheet.

There are many websites that help with debugging and visualizing regex patterns and their matches. A few recommended ones include Regex101, Debuggex and RegExr. These typically support multiple regex flavours. Regex Storm is particular to .NET regex flavour; Rubular is for Ruby regex. You can also read a comparison of some online regex testers.

## Milestones

1956

Mathematician Stephen Cole Kleene coins the term **regular expressions** as a notation for expressing the algebra of regular sets. The regex metacharacter `*` is named *Kleene star* in his honour.

1967

Ken Thompson at Bell Labs writes a new version of **QED** text editor for the MIT CTSS system. He introduces regular expressions to QED, thus bringing regex from the world of mathematics to computer science for the first time. Regex in QED is also compiled on the fly, for which Thompson receives a US patent.

1970

In this decade, regex makes its way into some Unix programs and utilities such as sed, awk and grep. It's been said that awk is the first language to make regex a first class programming construct. In 1975, Al Aho creates egrep command with a much more expressive syntax that the basic one supported by grep.

1986

Henry Spencer expands the regex syntax and provides an engine for the same. His regex library could be freely included in other programs. He later goes on to create an even better regex engine for Tcl.

1992

IEEE defines **POSIX BRE** and **POSIX ERE** as part the standard IEEE Std 1003.1-1992.

1997

Philip Hazel releases the **PCRE** regex library. This is later adopted by PHP, Apache and many others. This follows the syntax and semantics of Perl5. In 2015, PCRE2 is released.

1998

In the 1990s, Larry Wall, the creator of Perl, adopts and expands on Spencer's library. Perl 5.005 is released in 1998 and it includes enhancements to the regex engine. Perl's innovation on regex include lazy quantifiers, non-capturing parentheses, inline mode modifiers, lookahead, and a readability mode. ColdFusion, Java, JavaScript, the .NET Framework, PHP, Python, and Ruby are some of the languages that have since adopted Perl's regex syntax and features.

Regular Expression