Google Pixel 3 and Slate, announced in Oct 2018 Source: Google
With the newest announcements from Google and Magic Leap over the past weeks, it’s clear that the keyboard, the trusted interface between human and technology, is nearly gone. User Experience and Interface designers, please take note, your studies should take in gestures, facial recognition, voice recognition (not just “who” is speaking, but meaning in inflection and tone)… in fact, I would recommend a course in AI just to learn how to access processing the buckets of information that will be required to interact fully with the new devices coming, and not just from these two tech giants.
Google’s massive investments in Machine Language, AI, and the constant stream of data in all forms to feed the learnings these systems need is being channeled into devices that can work effectively with little need for keyboard entry. In fact, the new Pixel 3 has a personal assistant feature that will screen calls, converting the opening conversation into text for reading, allowing you to pick up and join the conversation or not engage at all.
Magic Leap adds the visual component to that in their Mixed Reality, the Magicverse (more about that in a moment), with the ability to have not just one’s own avatar, but a fully interactive virtual AI-built personal assistant. At Magic Leap’s first conference yesterday (#LeapCON), the use of AI was described as having the personal assistant serve as the interface, though I wasn’t clear how much of the interaction was driven by voice/vocal inflection/tone with or without facial microexpressions and other body language indicators added to guide the AI assistant. Perhaps I was projecting next steps in their development process.
Returning to Google’s announcements, their other product announcements made more inroads in their engagement with users in the home, again with giving the Pixel3 the ability to work as a Google Home device, while adding more product to the dedicated Home hub line. Amazon may have migrated Alexa into their TV products, but Google took advantage of a handheld device that already had the right size and consistency of use to push this interface across an appropriate platform. Then they took another step into killing the keyboard: visual recognition that identifies objects as retail products, and turns taking pictures into making a sale.
Snapchat added a similar feature about a month ago, Instagram has been driving marketing through images with a click for a while, but Google has made visual recognition of objects native to the interface on the Pixel3 (yet not available throughout all Android devices).
Google’s not given up on the keyboard completely, with a new tablet device (the Slate) that has an extremely easy to use tablet… yet can work off of a stylus as well. If you have experience using a tablet device with a stylus, you will find it becomes annoying to shift from stylus to keyboard and back, and more often than not the keyboard becomes the less-used option, not only for navigation but, with highly accurate voice to text transcription, data input. Dictation was the favored method of executives and professionals for most communication for much of the lifetime of the typewriter in professional use, and do not be surprised for the return of the phrase, “take a letter”.
Observing Magic Leap’s desire to bring equality and creativity everywhere, they are adding 6DoF (Degrees of Freedom) to it’s Magic Leap One controllers, with no keyboard accessory in sight. This isn’t different from all of the other VR and AR devices out there, which have concentrated on the simple handheld controller that is to be gripped, and secondary marketers have tried to come up with controllers that attach to fingers for virtual keyboards, or just putting a keyboard on the virtual screen that you virtually “hunt and peck” with the trigger finger.
Let’s step further into the Magicverse, as it’s called by CEO Rony Abovitz: it’s a place where:
“we want our platform to be a sanctuary for the unique”.. “this is our chance to break free of the past”
and identified “Four North Stars“:
Sensory Field Computing: Right now, the company is developing devices that touch the senses — sight and sound are the primary, but perhaps they will find a way to add touch, smell, and taste. Haptics (touch) alone would put them ahead of the pack.
Lifestream: Sensors trained on the user as one moves inside augmented spaces.
Human-centered AI: Using the computer and the livestream for input, it expects to train AIs on you personally in a very deep and individual way, rather than relying on an AI trained on general content (“cat videos,” Abrovitz mentioned as an example. See the image below of the connection between user/AYA (personal AI-created assistant) and the Environment.
Framework for AI-based personal virtual assistance in the Magic Leap Source: Magic Leap
Layers: Also known as “Verses,” these are digital overlays on the real world, such as “entertainment” within a city-scale space. The overall collection of layers will constitute the Magicverse. While not mentioned specifically, this could become thick enough to actually turn into a complete VR environment, with enough of the real world blanked out of view.
This is a very clear departure from the past of dependence on the user typing directly into the device. It’s a wonderful environment that reduces the learning curve to operate and enhabit these space. However, it might not work for every application, particularly for those accustomed to writing or editing materials that are text-heavy, such as the scripts that will create the stories of these worlds, or the email and social media texts that drag us through using a QWERTY keyboard still over 100 years after its invention. And let’s not forget the coders, programmers, User experience designers, User Interface designers, Marketers, Data Analysts, and others who will have to use keyboards for quite a long time doing the work to get both Google’s Home and the Megaverse. The crowd at the Magic Leap even cheered when they heard there was a bridge program to be able to use Javascript in creating apps and content to run in the Magic Leap OS, called Lumin OS. Yes, cheers for Javascript, almost as big as the cheers for the amazing games and activities demoed for that crowd. Additionally, Unity and Unreal Engine took the stage to confirm their development platforms ability to be used to create for the Lumin OS Magicverse, which can speed up the process of either converting VR and AR experiences/environments already made for other platforms, or creating completely new ones without making the uphill climb of a completely new development environment. However, it still will involve the keyboard.
Magic Leap software roadmap Overview Source: Magic Leap
This post isn’t meant to say the demise of the keyboard is complete or imminent over the next few years. Maybe not even 10 years ahead or 15. Telepathic devices may jump us past keyboard, voice and facial recognition altogether, but not all that quickly.
The average consumer will be the first to benefit, rather than the developers and designers, who might, in other circumstances, be the early adopters. However, User Experience designers have been given the challenge, and both of these announcements are beating on the doors of digital design studios louder than ever.
A final note: Samsung may be trying to do damage control by announcing their Galaxy A9 with four cameras in the back, for a variety of picture options. This could be a stopgap measure to keep anticipation for the Galaxy S10, with many major enhancements announced, nearly all centered on the visual processing capabilities, and nothing really of note regarding Bixby, their answer to Siri, Alexa, Magic Leap’s Aya, and Google’s non-anthropomorphized Duplex.
