It seems all the effort is going into developing headsets for Mixed Reality (XR), and very little is going into figuring out how to interact with these systems. It’s particularly shocking when you see people talking about using XR headsets as a laptop replacement, or a phone replacement. Is anyone going to do serious work on any of the devices available today (without using any external devices)? The activity that is done most on computers, for decades, is typing. But it seems the creators of current devices expect people to passively absorb what others create.
At the moment there are three main interaction methods: controllers, midair gestures, and gaze. Controllers are great for games, but they suck for text entry. If you doubt me try writing a novel with an controller, or just the wifi password. They’re also another thing to carry around and lose. Midair gestures seems futuristic, but they get old quickly. Your arms get tired if you hold them up all the time. It’s difficult to be fast and precise when you’re relying on your large upper arm muscles and moving your whole arm. It’s difficult to use large gestures in confined space like an airplane seat, and you look dumb. If you care privacy you now have to obfuscate your motions to avoid someone reading your password from across the room. Gaze detection is perhaps the most convenient, as long you’re passively consuming. For anything else it seems irritating. It’s slow, nausea-inducing, and you can’t look at anything else while you type which makes it useless in lots of situations. This is speculation, but I bet a study will come out saying something along the lines of "People find it weird to use their eyes to press buttons", we’re used to observing with our eyes and interacting with our hands. Of course people with disabilities can manage, but the fact these systems haven’t taken off shows they’re not worth the hassle.
Seeing this failure, there are two alternatives. Some people have taken to carrying around a bluetooth keyboard to type. This is what failure looks like. Keyboards have all the issues of controllers, and they are even worse for 3d interaction. Alternatively, there are products like the Tap Strap, but they’re only good for typing, and the average person is unwilling to learn a whole new system to type. I ordered one, mainly out of curiosity and they’re a fun toy, but they’re bulky, uncomfortable, and the gestures made my hands start to hurt after 45 minutes. However, the learning process was thrilling. I was locked in and I could feel my entire brain working on it. It was like the feeling of a perfect workout, runners high from my brain. After my first two sessions, each about 45 minutes long, I was so exhausted I had to have a nap.
If AR is the next big thing, if it’s going to replace all our devices, we need better options. We need to be able to type. Of course people will make do if they have to, people have written novels on mobile phones. No one buys a phone to write but when they have one they’ll make do. XR is different, unless someone finds a killer app better than desktop replacement XR needs a good typing method. Perhaps we could look to the past? Except they suck too, for different reasons. There’s a long history of people trying to fix typing, from DVORAK, to chording keyboards, to KALQ. Yet QWERTY reigns supreme. Some of those options actually faster and more comfortable, so why QWERTY?
The average person does not care enough about typing to learn a new typing method, they have better things to do. If a system seems weird or complicated they won’t try it. If a system seems easy they’ll put up with limitations. Stenotypes, as use by court typists are an example of the former. They allow the operator to type faster and more accurately than a traditional keyboard with reduced muscle strain. Unfortunately they require specialised hardware and hundreds of hours of practice to get any good. The hardware issue can be solved with macro layers for traditional keyboards, but it’s hard to get started when it seems intimidating. Traditional keyboards are easy to get started with, press the letter and it gets displayed, over time typists get faster, and if they ever forget where a key is they can just look down. Chording keyboards look confusing and require practice which makes them nonstarters for most. Mobile phone keyboards are an example of the latter, they don’t look too bad and being able to text and browse from anywhere is enough of a selling point for people to learn. There has been a huge amount of work making mobile keyboards better, with tools like autocorrect and predictive text, despite this typing on a mobile with autocorrect is still slower, and predictive text only works for predictable text (of which there is more than enough already in the world).
If we broaden our horizons, the real problem we’re trying to solve isn’t typing, it’s text entry. Input Method Editors (IME) are mainly used for typing Chinese, Japanese, and Korean scripts. The operator types a few letters which are used to filter down a list of all possible characters to a handful the operator can select between. Existing IMEs use letters from a keyboard (mobile or traditional) to filter down the list, but it’s conceivable there could be some alternative system. However the fact that people don’t use IMEs for English suggests they’re not worth the hassle. People would need to learn a new system, which seems unlikely. And then there would need to be some system for typing the words that aren’t in the dictionary.
So that’s the state of things. Approximately everything sucks, and we’re all doomed. Part of me feels conspiratorial about all this. All of the existing ways to interact with XR are fine if you just need to swipe and press buttons. They’re fine if you’re using someone else’s tool to do a limited number of actions. The only time they become a problem is when you try to create something. When you need access to more tools than tapping and swiping. Programming on a phone is painful experience, XR looks to be going down the same path. At the moment plenty of people, or whole families, own a single mobile phone. These people are shut out of creation. They don’t have startups in Silicon Valley competing to solve their problems, and because of inadequate tools they can’t make their own. How can you liberate yourself when the only tools you have are the tools someone else wants you to have? It is fundamentally undemocratic. It’s a world of consumers, not of creators.