# Siri

## Summary


Siri is a voice-controlled virtual assistant for Apple devices. It can be used for multiple purposes such as getting weather reports, setting an alarm, sending a message to someone, scheduling a meeting, or locking a car. At times, Siri can also respond with wit and sarcasm. Siri Shortcuts and Siri Suggestions are additional features that enhance how users can interact with Siri. 

Siri is available in many countries and in different languages. Initially, it was available only on the iPhone and later expanded to include iPod touch, iPad, Mac, AirPods, Apple Watch, Apple TV, and HomePod. 

Despite being the first mover in the industry, Siri has fallen behind the competition. In fact, users may prefer to use Amazon Alexa or Google Assistant on Apple devices.

## Discussion

### What are the capabilities of Siri?

Siri is an artificial intelligence designed to help Apple users in various tasks. It can **perform actions** such as reading your last email, texting your friend, booking a hotel, or calling your parents instantly. It can tell good hotels nearby, set an alarm, give directions to reach a certain place, or describe tomorrow's weather. 

By asking Siri "What can you do?", Siri will respond with a list of all things it can do. There are already apps to do these things but Siri brings a voice interface to these apps and therefore a better user experience. 

It can also be trained to **give answers** to some specific questions. Siri can be taught to pronounce your name or which of your contacts are your family members. 

Siri can also be funny and sarcastic sometimes. For instance, if you ask it "What's your favourite animal?", it will answer "Software doesn't usually get to choose one, but I'll say birds. What's yours?" To the question "What is the meaning of life?", it will answer "I can't answer that. Ha ha!" 


### How can users launch Siri?

Each Apple device has its unique way to invoke Siri. Typically, a device button (physical or virtual) is pressed and the request is made. Users need to press and hold for longer requests. **"Hey Siri"** is a hands-free option to invoke Siri. It's available in latest versions of most devices. 

On iPhone X or later, press the Side button. In other models, press Home or Top button. With AirPods Pro and AirPods (3rd generation), set the force sensor on either left or right AirPod. On AirPods (1st generation) double-tap the outside of either AirPod and wait for a chime. On recent Apples Watch models, "Hey Siri" prompt is unnecessary. Another method is to press the Digital Crown for a few seconds. 

On Macs, press the Siri button on Touch Bar, menu bar or Dock (macOS Sierra or later). On vehicles that support CarPlay or Siri Eyes Free, hold down the voice-command button on the steering wheel while making a request. On HomePod, press the top of the device. For Apple TV, use Siri button on the Siri Remote. 


### How can users automate tasks using Siri?

A user may want to automate a sequence of frequent tasks rather than perform them manually. For example, for the way home from work, the user may want directions, send ETA to a family member and start listening to music. Another example is to download all images on a webpage, reduce them in size and upload them to Twitter. These tasks can be automated with **Siri Shortcuts**.  

*Shortcuts* is a separate app that's installed by default since iOS 13. It comes with 300+ built-in actions. Users can create or edit shortcuts. Users can browse a gallery of shortcuts and launch any shortcut. Specific shortcuts can also be launched from icons or widgets on the home screen. Or invoke and tell Siri the name of the shortcut to execute.  

For third-party apps that support this feature, users can add shortcuts to automate tasks concerning those apps. 

Since iOS 13, shortcuts can also be triggered automatically. For example, a shortcut can be configured to run at 11am daily. A practical example is to automatically enable Do Not Disturb when the user starts watching Netflix. 


### What are Siri Suggestions?

We may view Siri as just a voice interface to apps on Apple devices but Siri is more than this. Using its AI capabilities, Siri learns how you use your devices and apps. Via a feature called **Siri Suggestions**, it gives personalized suggestions. For this purpose, Siri looks at your browsing history, emails, messages, images, notifications, contacts and information shared by third-party apps on your devices. For privacy, synchronization across devices is done using end-to-end encryption. On-device processing is used. Any information sent to Apple is anonymized. 

Here are some things Siri Suggestions can do: suggests people to include in emails and calendar events based on previous emails or events; based on numbers shared in emails, guesses who may be calling even if the number is not in contacts; gives search suggestions in Safari browser; recommends news stories based on your past reading history; notifies when to leave for an appointment based on current traffic conditions. 

Siri Suggestions also makes use of Shortcuts. In fact, suggestions can be seen as informing users what shortcuts to run. 


### How does Siri work under the hood?

The fundamental technologies used in Siri are Automatic Speech Recognition (ASR) that converts audio waveforms to text, Natural Language Understanding (NLU) that determines user's intent, and Text-to-Speech (TTS) that enables Siri to speak out a response. 

Since iOS 15, on some devices, some of these steps can be done offline without connecting to a server. However, the true utility of Siri comes from connecting to a server. Server-side logic might do a better job with ASR, taking into account regional accent, ambient noise, and linguistic information such as syntax and context. For example, the NLU engine must differentiate between "byte" and "bite" based on the context. The server may query databases to find answers. If an answer can't be found, Siri may prompt the user if it should search the web. 

The "Hey Siri" hands-free prompt is based on an acoustic model implemented as a Deep Neural Network (DNN). In fact, Siri does two-pass detection: a small DNN that's always on, a larger DNN that's triggered for more accurate processing. 


### What patents cover the technologies used in Siri?

Without being exhaustive, we note the following patents along with filing and publication dates:

    + US20120016678A1, Jan 2011 / Jan 2012: Intelligent automated assistant: a conversational agent that uses natural language dialog and external services to perform actions, retrieve information and solve problems rather than simply return search results.
    + US20170358301A1, Sep 2016 / Dec 2017: Digital assistant providing whispered speech: determines that user is whispering and modulates the response to whispers. A patent published April 2021, US20210097980A1, expands this idea to devices that are aware of their environments.
    + US20180329957A1, Aug 2017 / Nov 2018: Feedback analysis of a digital assistant: client device receives user inputs, processes these inputs based on instructions received from a server and sends the result to server.
    + US20180330731A1, Sep 2017 / Nov 2018: Offline personal assistant: an assistant that can receive multiple inputs, rank and score them, and decide which one to act upon. The patent focuses on on-device processing without relying on server backend.
    + US20180329677A1, Mar 2018 / Nov 2018: Multi-modal interfaces: visual interface is used to augment the voice interface.

### What developer tools are available to integrate Siri into apps?

Apple provides developers an API called **SiriKit**. Interactions are enabled using Intents and IntentsUI frameworks that are part of SiriKit. An app can have custom vocabulary and sample phrases so that Siri can interact better with the app. Via an app-specific extension, Siri can be used even when the actual app isn't running. Siri handles user interactions while the extension provides necessary information. 

SiriKit is available in Swift and Objective-C programming languages. It's available for iOS 10.0+, iPadOS 10.0+, macOS12.0+, Mac Catalyst 13.0, tvOS 14.0+ and watchOS 3.2+ platforms. 

Developers can read SiriKit documentation for more information.


### What are the shortcomings of Siri?

Ten years after its launch in 2011, Siri was considered inferior to its competitors Google Assistant and Amazon Alexa. This is despite Siri being the first mover in the industry. New releases came with only insignificant updates. Siri mishears even simple commands. Users become frustrated by its failures to complete basic tasks. Some of these can be blamed on poor management and lack of product focus. 

Apple being a closed ecosystem, Siri doesn't integrate well with third-party apps and services, which is where Alexa trumps. Different versions of Siri on different Apple devices means that users don't get a consistent experience. 

In 2019, it was revealed that recordings of real conversations were given to external analysts to improve Siri's algorithms. However, these recordings contained sensitive information. 

Without an active internet connection, Siri's capabilities are limited. Back in 2019 it was reported that user interaction with maps was limited to English. Background noise, low-quality audio, spelling variations, strong accent and fast speech all caused problems for Siri. Even if the iPhone is password protected, Siri could be invoked via the Home button without unlocking the phone. 

## Milestones

2010

A Norwegian company named Siri, Inc. launches an iOS app based on speech recognition technology. Siri is a virtual assistant. It enables users to interact with their iPhones via a voice interface. In April, **Apple acquires Siri**. 

2011

Apple makes Siri an integral feature of **Apple iPhone 4S**. Despite Apple's best efforts to find a better name, they stick to "Siri". Later, people suggest that Siri stands for "Speech Interpretation and Recognition Interface". Competition soon follows: Samsung's S Voice (2012), Google Now (2012), Microsoft Cortana (2014), Amazon Alexa (2014), and Google Assistant (2016). 

2012

Apple announces many improvements to Siri in iOS 6, mainly based on access to new sources of data such as information about sports, restaurants and movies. Siri is now capable of posting to social media, reading incoming messages and notifications, and opening apps. It's now available in 15 languages. Apple expands support of Siri to iPad and iPod touch. This encourages Apple users to upgrade their devices. 

2013

Siri acquires new capabilities including switching on/off Bluetooth and Wi-Fi, using databases to answer search queries, and using more natural sounding voices. Apple introduces "iOS in the Car", an extension of Siri Hands-free designed to work with in-car systems. 

2014

Hands-free interaction with Siri was first introduced in 2012. This is improved with the **"Hey Siri"** feature. Users can use Siri without needing to press buttons or touchscreens. In July, Siri gets a major upgrade with adoption of ML techniques including Deep Neural Networks (DNNs). Error rates drop by at least a factor of two. 

2015

Siri is added to **Apple TV**. Standalone Siri remote for Apple TV is also released. It gives suggestions based on previous user behaviour such as recommending news stories, music, and apps. It's now 40% faster and 40% more accurate than before. 

2016

Apple releases the **SiriKit API** for developers. This allows developers to integrate Siri with their apps. However, this integration is limited to some apps such as messaging, phone calls, photo search, ride booking, personal payments, workouts in apps and some in-car apps. Siri can do intelligent scheduling, integrate with the QuickType keyboard, react to text conversations and make useful suggestions based on user behaviour. 

2017

Developers get support for deploying more Siri features such as to-do lists, notes and payments in SiriKit. Machine Learning becomes deeply embedded into Apple products, including Siri. Apple reveals that Siri is used by over 375 million devices each month in different countries and languages. Siri can also translate from English to Chinese, French, German, Italian, and Spanish. 

2018

Introduced at WWDC 2017, Apple's intelligent speaker system called **HomePod** becomes available along with iOS 11.2.5. Users can interact with HomePod via Siri. 

2019

**Siri Shortcuts**, though available earlier as a downloadable app, is now installed by default on iOS 13 and iPadOS devices. Shortcuts can also do things automatically without users invoking it explicitly. 

2021

With the release of iOS 15, Siri can now be used **offline** though with limited capabilities. This is available on recent iPhones with A12 Bionic processor and iPads. On-device speech recognition is now possible but questions such as "What is the capital of Switzerland?" or "Will it rain later today?" can't be answered without connecting to the server. While offline, Siri can open apps, create timers, make calls, and adjust volume. But it can't add new calendar entries or select downloaded music albums.