It’s no real surprise that ComplaintsHQ has gone all in on IoT. This is really the closest thing to a true “next wave” consumer tech has had since the shift to mobile became a tidal wave back in the early 2000s. And similar to those early days of mobile, there are without a doubt rough edges, but the potential and promise is undeniable. Unfortunately another pattern that is actively repeating is division into warring walled ecosystems. Now it’s probably worth noting before we dive a bit deeper that any of these platforms require accepting a corporate controlled “listening device” into your home. If that throws you into a rage and sends you reaching for copies of 1984 then it’s best to stop reading. This article isn’t going to debate the philosophical questions of the entire concept. That’s what The Verge is for. Ok, back to regularly scheduled programming!

There is a rule of three here that tends to happen and IoT is no exception. Sure Microsoft still flirts with Cortana within its ecosystem, but this is clearly a three horse race between Amazon, Google and Apple. Let’s take a deeper look at each one.

The leaderboard…

• Amazon is the 800lb gorilla here. They are the masters of platform building, have incredible insight into retail trends, and flexed that muscle near perfectly in IoT. Alexa, their assistant brand, may not be backed by the absolute best AI, and they may not have a home grown OS and productivity platform to feed it, but they were near flawless in the execution in two key areas: building a partner ecosystem and providing developer tools. Both of these initiatives have translated to a massive lead specifically in the home automation space, with a huge list of appliance partners, and in extensibility, with a vast catalogue of third party skills (effectively the IoT equivalent of mobile “App Store” wars). With the most units sold by far at something approaching 100M across a nice selection of form factors ranging from full screen models (Echo Show gen 2 pictures below) down to the tiny Spot, it’s clear that their approach is working. Even Apple couldn’t ignore Amazons massive critical mass, recently acknowledging it by bringing Apple Music integration to Alexa.
• Apple is at nearly the opposite end of the spectrum from Amazon. Siri, their assistant brand, lives in every iOS and Mac OS device, a huge advantage, but Apple faces too self inflicted challenges which have stifled growth and maturity for Siri: closed by design ecosystem, and privacy as a bug-turned-feature. The latter may be controversial, but the reality is that Apple did try early on to build their own competitive ad platform around iOS, but failed to compete effectively enough against the juggernauts of Google, Facebook and Amazon in this space (see “closed ecosystem”). As privacy has started to emerge as a potential battle ground (or at least differentiator), Apple has doubled down on the message of “never touching user data). Unfortunately, rapid evolution of AI is heavily dependent on huge sample sets of data. Apples privacy first approach shows in Siri’s relative lack of insight in comparison to her peers. Compounding this is Apples closed ecosystem reflex. Apple services rarely leave Apple devices (Apple Music being one notable exception) and so the reach of Siri is limited to iOS. Large and lucrative to be sure, but smaller than Android on mobile and Windows on traditional form factors. All of that said, Apple was definitely early on home automation with Home Kit and combined with their lucrative and locked-in installed base, this has lead to some interesting partnerships (some even exclusive).
• Google, the leader in AI (quite possibly globally, although the large Chinese firms may be ahead), splits the difference between its two competitors. Having a massive end user computing base in Android and Chrome OS, augmented by productivity tools in both business and consumer like GSuite, Google Maps, et al, and amplified by social platforms like YouTube, GMail and Photos, Google sits on a literal mountain of captive data. In AI data is king and it shows. Google Assistant stands out as noticeably more capable and effective than Siri or Alexa (despite not having a name), and because of Googles broad ecosystem and willingness to collect more data, its advanced capabilities are applied to a wider range of real world user stories. So in other words because it knows all of your habits, and you likely use a ton of services it can look into, it gets much closer to being able to do what you would expect it to do. On the IoT front, Google is nowhere near as good as Amazon at building a partner ecosystem, or creating an easy developer story for that matter, and so lag in terms of absolute coverage; that said, the major players are all there. Amazon’s skills approach remains unique though, so grass roots extensibility (again, the “IoT App Store” approach) has no real analogue in the Googleverse.

The also rans…

• Microsoft and Cortana do deserve a mention here. Microsoft may have thrown in the towel on mobile (for now), but they are killing it in Cloud, Windows remains healthy (albeit small compared to iOS or Android) and XBox holds a tiny (again relatively), but passionate community of enthusiasts. Microsoft also has a stack of services of its own, with Office 365 giving Google a run for its money, which they can tap into. So within the Microsoft ecosystem Cortana remains potentially relevant. She did make it out into the world a bit thanks to Microsoft’s own excellence and experience building partner ecosystems, but lately it seems Microsoft is more interested in pursuing partnerships with either Amazon or Google (Apple remains walled off) than with promoting Cortana as a general assistant. And to be fair, this is possibly Apples view as well, that Siri has no intention of being generally applicable, but since Apple is so closed, it pretty much requires she be compared with Alexa and Google Assistant rather than a platform play like Cortana (or…. Bixby. Anyone? No? Ok then)
• Bixby is worth noting as well. Sure Samsung’s entry in this space makes Cortana feel a little better about herself, but Samsung is a massive player in IoT via the Things platform as well as their own appliances, and they are the single largest Android manufacturer. In Korea, it seems inevitable that Bixby will be a major player. Globally though, while still in play, prospects are more grim.

So how do these platforms stack up in practice and what is it like living with them individually, as well as collectively? Well at ComplaintsHQ we fight the hard battles so you don’t have to and have deployed all of these technology helpers (“also rans” included) into production. With roughly a year of experience under the belt we’ve compiled a set of summary notes in four categories: general AI performance, platform integration and overall usefulness, home automation control and exclusives (partners, features, et al). We’ll also elaborate what we’ve actually deployed and what we’ve integrated with each platform and, just for fun, how each did on our patented “how many lights are there?” question (spoiler alert, there. are. 4. lights!)

Network Environment

Linksys Velop AC2200 (WHW03v1) 4 nodes deployed, Linksys LRT224 firewall, ARIS Surfboard DOCSIS 3.1 cable modem (Xfinity 400Mbps/10Mbs)

Amazon (Alexa)

• Hubs: Echo Show 2
• Clients: None
• Integrated: IoT:
  • • Ring Doorbell 2
    • Phillips Hue (19 lights)
    • TP-Link Kasa Outlet (x2)
    • Insignia Connect Smart Plug w/Metering
    • Nest Thermostat
    • BMW Connected Drive

• • Services: Apple Music
  • • Pandora
  • • Prime Video
• General AI – B+: The story here is an odd sort of mixed bag.  On the one hand, Alexa often comes up short when it comes to general information because it is a bit inconsistent in its effectiveness crawling into the web (it also obfuscates source).  On the other hand, the team does an impressive job of manually injecting the “right answer” to popular queries with our “4 lights” challenge being a noteworthy example.  Understanding is excellent, voice quality is very good (if a bit robotic) and personalization is available.  The skills system is both a blessing and a curse in that it opens the possibility of Alexa reaching into places it isn’t natively welcome (like GMail), and provides endless extensibility via the community, but it also brings a very mixed bag in terms of user experience (which extends to IoT; more on that later).  So while Alexa can be made smarter without effort from Amazon, and Amazon itself is doing a decent job of evolving her, the experience remains a bit uneven.  On the test query “What was Jack Nicholson’s first film” Alexa answered directly “Cry Baby Killer”
• Platform – B+: the Platform question is interesting because Amazons natural platform is shopping, Video and Music.  These services are all present and well integrated.  In addition, Amazon’s pure market weight is building some amazing partner bridges including the quite surprising additional of Apple Music.  With the caveats noted above in General AI, really anything becomes possible via Skills as long as there is an API.  Once again, there may be some rough edges, but the platform is extremely comprehensive.
• Home Automation – A: On the Home Automation front we again find quite a double edged sword.  On the one hand no platform even comes close to matching the connected device support of Alexa, but on the other hand the skills system is a bit context breaking.  It never feels truly seamless unless it is a deeply native integration (like Ring).  The difference between “Alexa, what is the temperature in the Family Room” and “Alexa, ask Nest what the temperature in the Family Room is” may seem minor, but the reality day to day is that it definitely hurts UX.  That said, if you want to be able to randomly pick any wacky connected device and know it will integrate, no one even comes close to Amazon.  Amazon now owns Ring, so integration here can be expected to get stronger.
• Exclusives – A: Prime Video (Prime Music, but at ComplaintsHQ we are not subscribers), Prime itself (shopping), all of the Alexa Skills (this is huge), shipping video from Ring to a display enabled hub (yes, this only works on Echo Show), a long tail of connected devices (although none in use at HQ ironically)

Apple (Siri)

• Hubs: HomePod, AppleTV 4, AppleTV 4K
• Clients: Mac Mini i3 (2018), iPad Pro 11″, iPad Pro 10.5″, iPhone XS Max, iPhone X, iPhone 7 Plus, Apple Watch Series 4, Apple Watch Series 3
• Integrated: IoT:
  • • Ring Doorbell 2
  • • Phillips Hue (19 lights)
  • • Eve Room (x4)
  • Services: Apple Music
  • • Pandora
  • • Calendar
  • • iCloud Drive
  • • iPhoto
• General AI – B: Siri is not great here.  Understanding and voice quality are both excellent, but personalization is not available and Siri sometimes stumbles over simple questions. It also doesn’t do much of a job at all of crawling into the web (which probably feeds into the first problem).   On the test query “What was Jack Nicholson’s first film” Siri could not answer and indicated “this can’t be answered on HomePod”
• Platform – A: this is a really tough call as no one even comes close to the level of seamless platform integration, and ease of use, that Apple brings to their ecosystem, but this is also a closed ecosystem that they generally don’t extend anywhere else, and it isn’t a fully complete ecosystem necessarily.  Overall though, if you are deeply invested in Apple, the point is generally moot and life will certainly be easier in terms of “it just working” (which is, after all, Apple’s main goal)
• Home Automation – B+: this one is also a tough call.  The early lead Apple had in HomeKit shows in terms of overall polish, with the experience being truly fantastic when it works both in how effortless it is to setup, how great the software is, and how easily it integrates into HomePod.  On the other hand, the device ecosystem is small compared to the others (although worth noting is that most categories are represented). HomeKit also doesn’t play very well with the open IoT larger world (tools like IFTTT, more on this later), which isn’t a surprise given Apple’s mission.  Of course HomeKit compatible device manufacturers could still choose to open themselves to extensibility via this route (although notably Eve does not)
• Exclusives – A: The big one here of course is iOS and Mac OS integration, but there are also Home Kit exclusive partners with Eve being a particularly good standout.  The Eve Room is an excellent device for environmental monitoring (including VOC levels) and it only integrates with Home Kit.  In addition, the Apple services stack tends to be Apple OS only, although Apple Music has recently broken the mold there.

Google (Assistant)

• Hubs: Google Home, Google Home Mini (x3)
• Clients: Samsung Galaxy S9+ (Android Pie), Nvidia Shield TV Pro, Sony XBR-55X930D Android TV
• Integrated:IoT:
  • • Ring Doorbell 2
  • • Phillips Hue (19 lights)
  • • TP-Link Kasa Outlet (x2)
  • • Insignia Connect Smart Plug w/Metering
  • • Nest Thermostat
  • • Nest Protect (x4)
  • • Nest Camera
  • • Harmony Hub
  • • August Smart Lock Pro
  • • BMW Connected Drive
  • Services: Play Music
  • • Play Video
  • • YouTube Music
  • • YouTube Red
  • • GMail/Calendar
  • • Google Maps
  • • Drive
  • • Google Photos
• General AI – A: Google is the gold standard here.  Assistant is fantastic and general inquiries, crawls deeply into the web (citing source) and seamlessly leverages its visibility into the pile of other data you’re probably giving it.  Understanding is excellent, voice quality is very good, like Alexa a bit robotic, but probably less so. Voice is configurable, which shockingly accurate natural voice updates coming soon, and deep personalization is available.  On the test query “What was Jack Nicholson’s first film” Google Assistant answered referencing the web and citing source “According to Wikipedia, blah blah blah Cry Baby Killer”.  It also asked if we’d like additional information, which was cool.
• Platform – A:  Google has done a brilliant job of spreading the reach of Assistant across Android, Chrome and all of their services.  They leverage the data a user generates universally to build a strong composite, and then tap into this contextually.  There are still weird rough edges, sometimes surprising considering but no other platform has quite the combination of both assets and integration maturity.  Apple is arguably better at the latter, but lacks the assets, Amazon comes up short in both, but uses the excellent Skills model as a hedge.  As a result, Google clearly differentiates here.
• Home Automation – A: what Google lacks here in terms of absolute reach they make up for in seamless integration.  The connected device catalogue is getting better, well ahead of HomeKit but falling well short of Alexa, and nearly ever category is represented with at least one option.  Because the Google approach to integration is less of a plugin model (likely part of what has slowed down their partner play), once a device is supported it very much feels native from a UX perspective.  Worth noting is that this isn’t necessarily mandatory, as some device experiences will be similar to the Alexa model of “ask [thing] to do [action]”.  Google owns Nest, so integration here is strong.
• Exclusives – B+: Integration with the full Google services stack, including Play content and YouTube, is the big one here.  Worth noting is that the Alexa skills ecosystem is a wildcard which means the sky is pretty much the limit given Googles open approach to providing API access to their services (vs Apple).  That said, native integration tends to be, if nothing else, more dependable/trusted.  In addition, Google Assistant is on every Android service (something Google has started to love pointing out), although how useful it is on, for example, TVs at this time is debatable.  In our testing it didn’t bring much value given that devices like the Shield or an Android TV tend to be set-top-boxes where Netflix, Hulu or Prime Video are going to be primary apps (Google doesn’t crawl into these walls), or YouTube, but YouTube is already maintaining awareness of your interests via your login independently.

Microsoft (Cortana)

• Hubs: None
• Clients: Windows 10 Pro (gaming PC) Windows 10 Enterprise (jump server, lab client), Windows 2016 DC (lab server), XBox One X
• Integrated:IoT:
  • • None
  • Services: Office 365
  • • One Drive
  • • XBox Live
• General AI:  In a nutshell, Cortana is a bit like Siri, but with a clear path to the web.  Unfortunately it’s not a recursive path to the web like Amazon or Google, where it goes and gets you the answer, but rather an iterative one offering to send you there.    On the test query “What was Jack Nicholson’s first film” Cortana asked if we’d like to go check the web.
• Platform:
• Home Automation:
• Exclusives: The big thing here of course is Windows and XBox.

Samsung (Bixby)

• Hubs: None
• Clients: Samsung Galaxy S9+ (Android Pie), Galaxy Watch S3 Frontier
• Integrated:IoT:
  • • None
  • Services: Samsung Cloud
• General AI:
• Platform:
• Home Automation:
• Exclusives: At ComplaintsHQ the lone entry here is the Galaxy Watch S3 Frontier, which is Samsung Tizen powered and not covered by any other assistant, but it is worth noting that with a full Samsung appliance household, this list could potentially be longer (although most of Samsung Things should integrate with Alexa, that doesn’t necessarily mean full integration in every case)

An overall winner is tough to pick here.  If you’re an Apple user, you really can’t avoid Siri and the closer you are to “all in” (a near inevitability based on Apple’s strategy), the more likely you are to get the most value out of their platform.  That said, the final report card is:

Main Contenders:

• Amazon Alexa:  B+
• Apple Siri: A- (with the caveat that this is only if all in on the Apple ecosystem, although with Siri’s substandard AI holding it back from an A even here.  For non-Apple users, the point is really moot as Apple isn’t attempting to court them.)
• Google Assistant: A

Overall winner?  Well… It really comes down to priorities.  If you’re an Apple user, just give up, buy a HomePod, and hope Siri gets better.  If not, it depends what you value most.  If it is automating the heck out of a home (especially a larger one), Alexa is it.  If it’s having a special friend you can talk to all day, or you live in the Google ecosystem and want something approaching a true virtual assistant, Google clearly wins.  Given overall polish, and superiority in AI, given these are supposed to be “smart assistants”, the trophy here is going to Google Assistant.

“Also Rans”:

It didn’t seem fair to give the “also rans” a grade given the limited deployment at ComplaintsHQ as well as their limited scope overall compared to the big 3, but a few final notes are warranted.

• Microsoft Cortana: if you’re an XBox or Windows user, Cortana is basically just there.  You can think of her as the replacement for content search.  And, hey, if she adds extra value, why not? Longer term, Microsoft is making moves, opening its platform, and building partnerships and this is the main thing.  They have extremely valuable assets, they continue to hold a loyal user base and he bleeding has stopped, and they are well positioned for growth moving forward. In addition, they’ve got lots of consumer experience.  TL;DR Cortana may be a bit of a punchline, but Microsoft shouldn’t be counted out as a player here and extending either Amazon or Google would be a powerful move for them.
• Samsung Bixby: like Cortana, Bixby is just there.  The more “all in” you are on Samsung the more it looks (roughly) like Siri, just not as good.  That said, as with Microsoft, Samsung isn’t going anywhere.  They also have a wildcard in their huge presence in IoT in general and their deep appliance catalogue.  In the West Bixby is unlikely to move the needle much (hell, the big “feature” on the S10 is being able to remap the Bixby button!), but in Asia we wouldn’t be surprised to see Bixby evolve and extend in interesting ways.

Oh, and on a final note, in a stunning upset, the winner of the “4 lights challenge” was…

Crossing the Aisle

So that’s each platform in insolation, but what about together? Well the bad news here is each major vendor isn’t really rushing to build lots of bridges, but the good news is native extensibility meeting community creativity has definitely enabled some very interesting possibilities.

The biggest thing to be aware of is the emergence of service bus backends for IoT that allow users to stitch services together via applets.  The gold standard here is IFTTT. When you run out of options natively, services like IFTTT can empower you to create some crazy solutions.  The concept is basically that any service with REST endpoints, or some other accessible end-point model, can have a connection module built for the service bus allowing it to be connected.  Once connected, it can be orchestrated.  For the enterprise folks out there, think almost something like Mulesoft, but for IoT.

One example of what can be accomplished is the case of the (weirdly broken) workflow of the Ring doorbell.  Alexa can display Ring doorbell video, but only when asked.  For some reason it cannot be trigged from the doorbell event.  You also cannot create a custom action that asks Alexa do just do something, replicating a spoken command (like “Alexa, show me the front door”).  Google, on the other hand, can react to door bell alerts, but cannot show Ring video even on a screen enabled Assistant.  So that’s the end of that then until someone some day updates something?  Not so fast! Enter IFTTT!