Background:

Voice recognition has existed in somewhat passable form since the late 1990s. But recently it has become more prominent with Apple Siri, the “OK Google” voice assistant, and Amazon Alexa.

The issue:

However, these products are all incredibly limited, and can perform almost no actions beyond a few hard-coded ones that were added at release.

The last 5 years of “voice assistant” development have mainly been focused on extremely specific items of potential commercial interest (e.g. “now you can ask about baseball scores or movie showtimes”), rather than adding core functionality.

If you want to do something besides play music, interact with your calendar, or set a timer, then using a voice assistant is an exercise in frustration.

Proposal:

At every company with a voice assistant division, all senior engineers should have a recurring Monday task where they have to:

• Find the #1 query that users ask for that is both 1) reasonable and 2) totally un-answerable by the system.
• Add a response to that query the ever-growing list of hard-coded phrases that the system recognizes.

To assist, I have run through a list of plausible and reasonable queries to test Siri on iOS 11 and “OK Google” on Android 8, and have provided screenshots of the equivalent queries and disastrous results below. They are divided by category.

CALCULATOR: BASIC

• “What’s 2 + 2”
  • Google: SUCCESS
  • iOS: SUCCESS
• “What’s 10 factorial?”
  • Google: SUCCESS (“10 factorial = 3 628 800”)
  • iOS: SUCCESS (“The answer is 3628800.”)

• “Open the calculator”:
  • Google: Failure (Redirects to a “install this third-party calculator app!” link, despite the default Android Calculator app already being present.)
  • iOS: Failure on iPad (but, similarly provides an app-store link where a third-party calculator could be downloaded.)

BASIC APP INTERACTION:

• “Redo my last question.”
  • Google: Failure (Gives the results of an irrelevant web search).
  • iOS: Failure (“Sorry, [your name], we don’t seem to be navigating anywhere”)

• “Clear my browsing history for the last one hour.”

  • Google: Partial success (It says “No problem!” but then directs you to a link with instructions to do it yourself.)

  • iOS: Failure (“I didn’t find any appointments about ‘Browser history.’ “
  • Apparently, Siri interpreted this as a request to clear a calendar event.)

• “How many new mail messages did I get today?”
  • Google: Failure (Shows a list of recent emails, but does not count them.)
  • iOS: SUCCESS (“Seven new emails were received today.”)

• “Are there any new podcasts?”
  • Google: Failure (Provides irrelevant web search results.)
• iOS: Failure (“Playing podcasts, starting with the newest episode…” and shows an “Open Podcasts” button.)

SYSTEM:

• Version of the OS:
  • Google: “What version of Android am I running” or “What is my Android version” or “What is the system version” –> Failure (gives the results of an irrelevant web search).
  • iOS: “What version of iOS am I running” –> Failure (Refusal: “Sorry, I can’t do that”).

• Although this fails on iOS,  the question below succeeds:
  • iOS, attempt #2: “What version of iOS is running” –> SUCCESS (“You’re running iOS 11.0.3.” Note that it says you are running this version, but will not respond to a question in that format.)

• “Change the system language to Portuguese”
  • Google: Failure (Gives an irrelevant web search result about changing the interface of a Nintendo DS portable game system).
  • iOS: Failure (Refusal: “Sorry, but I’m not able to change that setting” with no additional information).

• “Change the OK Google Voice to UK English” or “Change the Siri Voice to UK English”
  • Google: Failure (Gives a relevant web search result for a change, at least.)
  • iOS: Partial success (“I can’t change my voice, but you can do it yourself in Settings” (Button to “Siri Voice Settings” appears on screen.)

• “How many gigabytes of RAM does this phone have” (Android) or “How much RAM is on this iPad” (iOS) or “how much RAM is on this device” (both)
  • Google: Failure (Provides irrelevant web search results.)
  • iOS: Failure (Refusal: “I’m sorry, I can’t do that here.”)

SCREEN & CAMERA RESOLUTION:

• “What’s the screen resolution?” / “What’s the screen resolution of this phone?”/ “What’s the screen resolution of this iPad?”
  • Google: Failure (Provides relevant-ish web search results for an assortment of different phones.)
  • iOS: Failure (Provides generic Wikipedia link to the definition of “Display resolution.”)

• “What is the resolution of the phone camera?” / “What is the resolution of the iPad camera?” / “What is the camera resolution of the front facing camera?”
  • Google: Failure (Provides irrelevant web search results.)
  • iOS: Failure (Provides irrelevant web search results.)

SLIGHTLY MORE DIFFICULT MATH:

• “What’s the sine of 30 degrees?”
• • Google: SUCCESS (“sine(30 degrees) = 0.5”).
  • iOS: SUCCESS (Even though it says, along with the correct answer, “The answer is approximately 0.5.”)

• “What’s the sine of pi?”
  • Google: SUCCESS (“sine(Pi radians) = 0.” Note that “radians” is (correctly) appended here by Google.)
  • iOS: SUCCESS (“The answer is 0.”)
• “What’s the sine of pi radians?”
  • Google: SUCCESS (“sine(Pi radians) = 0.”)
  • iOS: Failure (Fails where “sine of pi” succeeds. “OK, I found this on the web for ‘What is the sine of pi radians’.”)

• • “What is the sine of pi radians?” fails on iOS, but “What is the sine of pi?” succeeds.
• “What’s the sine of 2 radians?”
  • Google: SUCCESS (“sine(2 radians) = 0.909297427.”)
  • iOS: Failure (“OK, I found this on the web for ‘What is the sine of 2 rad.’.”)
• “What’s ‘sine 2 radians’?”
  • Google: Failure (Provides a web search. Interpreted as “what’s sign to radians.”)
  • iOS: Failure (Provides a web search. Interpreted as “what’s sign to radians.”)

NETWORK:

• “What WiFi networks are available”
  • Google: Failure (Provides irrelevant web search results.)
  • iOS: Failure (Provides irrelevant web search results.)
• “What’s the strength of the current WiFi network?”
  • Google: Failure (Provides somewhat relevant installation link to a program called WiFi analyzer that could probably answer this question.)
  • iOS: Failure (Provides vaguely relevant web search results.)

• “Is my Wi-Fi encrypted?” or “Is my Wi-Fi network encrypted?”
  • Google: Failure (Informs me that Messages are encrypted, which is a different question entirely.)
  • iOS: Failure (“Wi-Fi is on.”)
• “Does this device support 80211-ac?” (A WiFi network standard.)
  • Google: Failure (Informs me that Messages are encrypted, which is a different question entirely.)
  • iOS: Failure (“Wi-Fi is on.”)

BACKUPS AND STORAGE:

• “When was this device last backed up”  or “When was the last backup” (identical results)
  • Google: Failure (gives the results of an irrelevant web search).
  • iOS: Failure (Shows a random calendar event from years ago that happened to have the word “backup” in it).

• “How much space is left on this device?”
  • Google: Failure (gives the results of an irrelevant web search).
  • iOS: Failure (Refusal: “Sorry, I can’t help you with that here).
• “How much disk space is free?” or “How much free space is available” (identical results)
  • Google: Failure (gives the results of an irrelevant web search).
  • iOS: Failure (Refusal: “I’m sorry, I can’t do that here”).

2018 Conclusion / Final Ratings:

• Siri: D-minus
• “OK Google”: D-minus
• Alexa: not tested

If every Apple, Google, and Amazon programmer just spent one entire work day contributing a single answer to the repertoire of easy-but-unanswerable questions, perhaps voice assistants would be more reliable.

PROS: Makes voice assistants more reliable.

CONS: Brings the day even closer when humans are replaced by metal skeleton robots.