New Realities: VR, AR, MR and the Future of Design
Achievements in artificial intelligence, augmented reality, and virtual reality mean new design opportunities for designers.
The digital and technological landscape is constantly changing and accelerating. Designers, tasked with coming up with innovative ideas, must track trends and creative possibilities. Recognizing the changes happening in the industry will help designers stay relevant and make more informed creative decisions.
Technology Adoption Lifecycle
In the technology adoption lifecycle, "Innovators" represent the smallest but most successful group. Following them are the "Early Adopters," then the "Early Majority," and finally the "Late Majority" or everyone else. Great designers often fall into the "Early Adopters" category, either due to circumstances or their own choices. An excellent designer is also a visionary, capable of strategic thinking and trend analysis to develop better products or services.
The New Technological Landscape: AI, Augmented Reality, Virtual Reality, and Mixed Reality
What is Artificial Intelligence?
Artificial Intelligence (AI) gives computers or devices the ability to perform intellectual tasks such as reasoning or learning from experience. Cognitive computing was the term used when this started a decade ago—the idea that, faced with cognitive overload, we would need help in decision-making. Referring to the evident advantages of current AI, we already rely on digital AI assistants like Siri and Alexa to help us make simple decisions.
Product design will have a greater volume of information to inform its design decisions. AI specialists use AI technology to create dynamic user interfaces, personalizing each user experience by analyzing every interaction within an application or website. In the future, computers will become even better at making data-driven decisions to enhance automated tasks and human decision-making processes.
With IBM Watson leading in companies like General Motors, Condé Nast, GSK, Citigroup, and Under Armour, the future of AI will focus on effectively making complex and important decisions.
As creative professionals, we can and should leverage these ideas across various industries. For instance, The North Face utilized Watson to help create an intuitive e-commerce interface that assists customers in making optimal purchasing decisions. To sum up, we are on the brink of a time when UX will be entirely driven by artificial intelligence, and even now, it is delivering significant results.
Virtual Assistants—AI Technology in Action
Virtual assistants such as Siri, Alexa, Cortana, Watson, and others rely on AI to help with everything from finding a restaurant to sending an email or scheduling a meeting. Many of them are voice-controlled, providing a more natural interface for users compared to traditional graphical interfaces with windows and buttons. These technologies are likely to change the current design of search systems.
Voice-Controlled Virtual Assistants - Image Credit: Amazon Echo, Google Home, Apple HomePod
Speaking is a natural and simple way for people to interact. Therefore, it's no surprise that voice-controlled virtual assistants have gained favorable adoption rates in our homes. Users can simply talk to the system instead of "managing" it with their hands or gestures. For example, with an iPhone, using just one quick command, Siri allows us to transfer money through PayPal to someone else.
As AI is integrated into applications, this will help intelligent virtual assistants (IVAs) become smarter and smarter. However, regardless of how intelligent IVAs become, users with poor vision or other physical disabilities, who rely on screen readers and other assistive technologies, may still face significant challenges. It will be up to UX designers working alongside user experience researchers to bridge such important accessibility gaps.
How Computer Intelligence Utilizes a Driverless World
The transit systems of the future will smoothly and efficiently manage the flow of traffic on our streets. Design innovations such as modular, detachable buses, flying taxis, and networks of suspended magnetic pods will transform the dream of a dynamic, driverless world into reality.
What was once the stuff of science fiction news a few years ago is now becoming a reality. Whey’s Moby-Mart is a self-driving store with a holographic shopping assistant that is likely to change the world of eCommerce design.
The next generation of eCommerce designers will require multi-specialization in both physical and digital spaces. Future designers will need to create experiences where many familiar human elements are removed or replaced.
Amazon Go, a cashier-less store, is an excellent example of what the future looks like. They have designed shopping where anyone can walk into the store, browse, pick up their items, and leave without any friction—no lines at the checkout.
The intelligence built into every autonomous vehicle allows it to make the right decisions for us—everything from optimal routing to human safety considerations. Creating such complex design systems sets a template for products that will initiate a new wave of designs developed beyond the transportation vertical.
For designers, automotive interiors will need to create layouts that make the interior space comfortable and adaptive. Interfaces should be intuitive and reassuring to foster passenger trust, with particular attention to the interactions between people traveling together.
Augmented Reality
Augmented reality (AR) acts as a virtual layer over the world in front of you and is entering a new age of discovery and innovation. While early AR applications relied on smartphones or tablets, they will soon expand to wearable devices like smart glasses, requiring different types of UX and design that think beyond smaller screens.
Demo Presentation of Apple Animoji at WWDC17
As the discussion around AR versus VR versus MR continues, augmented reality presents a unique opportunity for designers to create applications using image, object, and color recognition. Just as AI leverages data for decision-making, AR utilizes the recognition of key "anchor" objects and points within a designated space.
A prime example of the ideal combination of AI and AR is Apple Animoji, animated emojis for the iPhone X. According to Apple, Animoji are "personalized animated messages that use your voice and reflect your facial expressions." The real-time rendering and animation work is made possible through the scanning capabilities of the iPhone's TrueDepth camera.
Meta-Building Immersive AR Experiences
This integration of AI and AR not only enhances user engagement but also opens up new avenues for creative expression, allowing users to interact with technology in more personalized and immersive ways.
Meta-Glass AR Demo - www.metavision.com
In the next five years, AR will become more integrated into our daily lives, merging our physical and digital worlds with new wearable products. A product like Meta Glasses, which overlays augmented reality onto the user's reality, is likely to gain popularity as a tool for game developers.
Based on holographic technology, Meta Glasses can recognize user gestures, allowing them to manipulate 3D projections of objects. With so many gesture-based devices emerging, standardization and cultural gaps may become significant challenges for designers working in this field.
Meta AR Demo
In the future, wearable devices will enable employees to work in hands-free mode. Devices of the future may operate using voice commands or gestures. The growing success of AR experiences could predict the end of physical controllers, such as a mouse cursor or keyboard.
In augmented reality, natural gestures can be key factors in the experience. For example, a finger snap could trigger interaction with a virtual link or button. Google has launched the Soli project, which goes even further by creating touch technology focused solely on contactless interactions. To remain user-friendly, such experiences require minimal interfaces with virtually no visible overload.
Notable Resources for AR Development:
- Facebook Frame and AR Studio
- Apple ARKit
AR-Enhanced Living
As part of its conceptual kitchen, IKEA is developing a smart table that suggests recipes based on the ingredients placed on it. This is an excellent example of AR technology working in the real world. The interactive table features a projector with a camera that displays recipes on its surface and recognizes ingredients, giving you insights on what to do with what you have.
IKEA AR Concept Kitchen - Image Credit: RumbleRum
Such immersive experiences are in direct competition with mobile applications, which will soon become outdated due to their more limited capabilities. Mobile app developers will soon create augmented applications that integrate with physical spaces, providing a new and exciting way to interact with our physical environment. Ultimately, every business, restaurant, and store will have its own mobile application, just as every company currently has a website. This will become the new standard in the market, leading to inevitable integration with our physical world.
The Future of Working with AR
Augmented reality can also be used for knowledge transfer, such as training in industrial environments. Instead of merely explaining, imagine showing a worker how to perform a specific task through contextual video. This opens doors for designers in the e-learning sector to create next-generation AR interfaces.
Boeing is utilizing AR glasses powered by Skylight to guide technicians as they assemble hundreds of airplanes each year. The wearable display helps technicians identify and accurately connect hundreds of wires, using only their sight and voice to control the application. As a result, Boeing has reduced production time by 25% and decreased error rates to nearly zero.
AR Glasses by Skylight
A simple voice control system reduces interaction costs to complete complex tasks when working with AR apps. Minimizing interface elements reduces the user’s cognitive load, minimizes interruptions, and reduces distractions.
Augmented reality technology can change the way users and workers learn and familiarize themselves with new information. Imagine training modules like how to change the oil in a car or other “How to” training videos using AR to provide a layer on top of the real-world situation.
From a usability perspective, designers must consider not only what the system presents to the user visually, but also how the brain interprets complex information that overlaps real reality.
Virtual Reality vs. Mixed Reality:
Virtual reality (VR) completely immerses you in a virtual world, while mixed reality (MR), sometimes called hybrid reality, is the fusion of real and virtual worlds. VR can be used for virtual offices, products, events like the 2017 Teen Choice Awards, and more. With VR, designers and their clients will be able to preview designs scaled to real-life environments and depth before they enter production.
In early 2017, Forbes noted that “the hype around virtual reality is fading.” Notable limitations of Face VR include the high price of headsets, discomfort in widespread use, poor image quality, and more. As a result, the mass market adoption of VR technology is slow compared to AR, which is “likely to improve in the short term.”
Virtual Reality Applications
The novelty of virtual reality content offers a unique opportunity to reach a wide audience, as evidenced by National Geographic’s VR documentary The Protectors, Walk in the Ranger’s Shoes . It has received more attention compared to other VR-themed docs.
In addition to transporting viewers to new places, the next opportunity for designers is to help them experience life as another person. By offering a first-person perspective, you can create unique and effective campaigns like “A Walk Through Dementia” that focus on empathy.
A new set of design patterns will eventually emerge for VR designers, but a flat approach to design is unlikely to succeed. In VR, our brains use spatio-temporal orientation and problem-solving capabilities; so it’s important to use textures, lighting, and finer details for a more believable and immersive experience.
VR gives users the ability to quickly and personally get up close and personal with virtual objects, as well as move away from them. Since the eye is very good at picking up depth information, designers’ rendering of objects must be realistic and detailed. Sound and music are other important considerations for VR designers.
MR - Blending Real and Virtual Realities
Microsoft HoloLens blends a realistic mixture of 3D holographic content with the physical world, giving holograms real-world context and scale. MR technology mixes and blends a person's physical environment with objects from the physical and digital worlds together. Using this MR technology, we can interact with both digital content and the surrounding world, as well as interact with 3D holographic projections.
Interactive Holographic Projection Microsoft HoloLens
MR and VR applications offer new creative possibilities for designers. We can create experiences where users can interact with products virtually and visit remote environments, historical events, concerts and more. Within the virtual or mixed realities we create, we can bend the laws of physics with endless possibilities and experience the world like never before.
From a design and usability perspective, a key principle to keep in mind is to reassure users and avoid uncomfortable feelings by keeping them grounded in the real world. To this end, we can use sound, movement and a good sense of scale - the virtual environments and interactions we create should feel natural and ergonomic, not necessarily realistic.
When designing a visual interface that uses a headset and requires close inspection of an object such as a VR organ, we must remember that people are much more susceptible to illness when moving around in a VR headset than when looking at a computer screen.
Brand Experiences in Virtual Reality
VR creates an opportunity for companies to create a more immersive experience of their brand for their customers. This is a new frontier for branded content and storytelling. Audi uses VR so that they can experience their new car model before even physically touching it. By capturing the entire interior of the car with a special 3D camera, they allow customers to see every detail before it goes to market.
Using virtual reality marketing effectively allows brands to leverage the following design components:
A well-thought-out creative strategy
Applying best usability practices
Strong narrative and storytelling
Presentation of ergonomic information
Compulsive visual design and modeling
The definition of usability will be greatly expanded for designers to cover all aspects, including the comfort level of the environment. Designers will need to develop an understanding of the level of potential discomfort in certain situations, such as heights, small spaces (claustrophobia), large spaces (agoraphobia), scale, speed, colliding objects, user fatigue, ergonomics, and more.
To communicate the reliability and safety of its vehicles in challenging situations, Toyota immerses customers in a VR story. They can experience the brand as well as unique, customized content in different environments. This opens the door for companies to show customers a side of the brand they haven’t seen before.
Because designers are deeply involved in creating these experiences, we must question the utility of our creative ideas and brand solutions and ensure we are truly solving specific business problems. We must ask ourselves if the VR experience delivers on the company’s core value proposition.
VR Applications in Unfamiliar Environments
Some nonprofits are using VR to transport visitors to life in refugee camps. This creates greater engagement and a greater willingness to support their initiatives. An idea like this could create a shift in VR project work from big brands into the non-commercial space.
VR Refugee Camp 3D Tour - International Society for Presence Research
As another example, Mark Zuckerberg showed the world a VR tour of Hurricane Puerto Rico to raise awareness of the situation. The ideas and use cases are “virtually endless.”
Using a VR headset and a remote controller, users can explore the environment by moving their heads and interacting with objects using their hands.
High-end VR systems like the HTC Vive or Oculus Rift use sensors to locate users in physical space, and so designers can use the user’s entire body to create more advanced sensory experiences in the real world.
The Future Is a Balanced Mix of AI, AR, VR, and MR
Recent developments in AI, especially deep learning, which facilitates real-time image and speech recognition, have helped bring VR and AI together for new applications. The future of product design will need to consider a combination of all three of these technologies. Notable examples include Eolian apps, which use AI technologies to reduce human error through AR and VR simulations of dangerous tasks. Virtualitics provides data visualization in VR and AR environments through machine learning and AI. These technologies are tools that help us tell stories and share information like never before. They can be used as incredibly powerful tools for emotional engagement, allowing us to create empathy and connect with audiences - beyond what television, computers, or the internet have ever offered.
Virtualization Analytics Data Dashboard in VR/AR - virtualytics.com
It’s an exciting time to be working with these technologies; most of the problems are new, and old recommendations often don’t apply. Institutes like MIT are experimenting radically with new ideas. For example, the MIT Game Lab is creating Play Labs to accelerate launches in AR, VR, and AI. This will undoubtedly inspire many innovators.
The future could be called hyper-reality – a combination of AI, AR, VR, and MR. One interesting innovation, EscapeLabs, uses augmented reality and mixed reality (AR/MR) to create holographic experiences for escape, team building exercises, and room-scale puzzle games. Of course, the need for clunky headsets and the requirement to be connected to a static computer will no longer be necessary. The transition between the virtual and real world will become faster and more fluid.
Setting Up the Fifth Industrial Revolution - Source: CBInsights
The internet was around for years before we discovered that social sharing was the key to driving. Likewise, the combination of these technologies is expected to see similar growth and the opportunities for designers will continue to proliferate in the next few years.
New technologies come with many unknowns, creating a need for developing best practices, standards, and design patterns. This is why UX/UI designers are expected to have long-term job security in the coming years. There will always be a need for designers who use user-centered design methods to add warmth to products. The best part about these technologies is that we are still in the early stages and there is a lot to be figured out by our creative community. Exciting times ahead!
