As evidenced by massive investments from the industry’s largest companies, many believe that spatial computing, augmented reality, and virtual reality (SC/AR/VR) wearables represent the next evolution in personal computing. However, achieving truly immersive, affordable, and performant SC/AR/VR experiences poses some significant challenges to hardware engineering. This case study explores the evolution of SC/AR/VR systems, the rise of on-device processing for these applications, and the growing challenges facing the underlying hardware, including balancing the tradeoff between power efficiency, performance, size, and cost. It also explores how the Alif Ensemble® family of microcontrollers (MCU) and fusion processors can help designers balance these tradeoffs to create the optimal computing platform for modern SC/AR/VR wearables.
The Evolution of SC/AR/VR: From Computers to On-Device Processing
Before augmented reality entered the mainstream, virtual reality was the focus of many. These systems were often characterized by their reliance on external computing resources, where the wearable headset would be tethered to a PC for image rendering and graphics processing.
Now, as AR continues to take the spotlight, the focus has shifted significantly from off-device processing to on-device processing. AR glasses are meant to be portable devices for use in daily life, necessitating an untethering from off-device processing and an embrace of edge computing paradigms. Adding momentum to this shift is the merging concept of ambient computing, which seeks to integrate a variety of peripheral devices together to form a cohesive AR wearable system. From a compute perspective, this shift is marked by a transition from relying on powerful computing resources like GPUs and CPUs to more resource-constrained microcontroller units (MCUs) with dedicated AI accelerators.
Challenges Facing SC/AR/VR Designers
Naturally, the shift from off-device to on-device processing comes with a myriad of design challenges that must be addressed. Below is a look at some of the major design challenges in SC/AR/VR applications:
Real-Time Processing
Spatial Computing and Augmented Reality systems are designed to interact with the real world in real time, meaning that they require real-time processing and decision-making.
From a user-experience perspective, real-time processing is a necessity to maintain an intuitive and comfortable experience. Even minor delays can disrupt the user experience, breaking the illusion of integration between virtual and real elements. Furthermore, AR devices are increasingly reliant on sophisticated machine learning algorithms to make decisions and enhance user interactions, such as recognizing objects in the user’s environment or understanding gestures. Executing these compute-intensive workloads with minimal latency poses a significant challenge to the underlying hardware.
This challenge is further exacerbated when viewed in the context of on-device processing, as the systems rely on comparatively resource-constrained MCUs.
Image Rendering
Image rendering in SC/AR/VR wearables is a significant challenge due to the need for real-time generation of complex visuals that are both high-quality and responsive to user interactions.
This process demands substantial computational resources, often pushing the limits of what compact wearable devices can handle without compromising their form factor and battery life.
Rendering must be efficient enough to maintain a seamless experience, as any latency can disrupt the immersive illusion, causing discomfort or disorientation for the user. Moreover, SC/AR/VR systems must accurately overlay digital content onto the real world in AR or construct entirely virtual environments in VR, requiring sophisticated algorithms to manage depth perception, lighting, and textures in a way that aligns with the user’s natural vision.
Balancing these demands with the constraints of wearable technology requires the optimization of processing capabilities, software, and display technologies to advance the SC/AR/VR experience.
Sensor Integration
SC/AR/VR systems rely on an array of sensors to help understand and interpret the user’s environment. These include IMUs for orientation and positioning, high-resolution cameras for spatial awareness, and microphones for audio inputs from the user.
Integrating these sensors presents a multifaceted challenge due to the need for precision, responsiveness, and power efficiency. Designers must be able to ensure accuracy, minimize latency, and manage the generated data in a way that doesn’t overwhelm the device’s processing capabilities. Additionally, maintaining the compactness and wearability of SC/AR/VR devices while accommodating multiple sensors adds to the design complexity. These elements must be seamlessly integrated without compromising device size or battery life, making it a complex task for designers to maintain the immersive experience SC/AR/VR aims to provide.
Power Efficiency
Achieving power efficiency in SC/AR/VR wearables is a complex task, marred by conflicting needs.
On the one hand, there’s demand for high-performance processing to deliver real-time, immersive experiences. This level of computing requires substantial power, which in turn drains battery life more quickly. However, to maintain user comfort, these wearables need to be lightweight, which limits the size and capacity of the batteries they can house.
Furthermore, the trade-off between battery life and performance is a constant balancing act. Longer battery life requires either larger batteries, which add weight and bulk, or reduced power consumption, which can compromise the quality of the SC/AR/VR experience. Designers are thus faced with the challenge of finding innovative ways to enhance power efficiency without sacrificing the core functionalities that define immersive SC/AR/VR experiences.
Wireless Connectivity
Incorporating low power wireless communications like Bluetooth Low Energy (BLE) into SC/AR/VR wearables introduces complexities, particularly in maintaining robust connections with peripheral devices such as controllers, sensors, and smartphones. BLE, designed for efficient power consumption, is pivotal for continuous device communication without significantly draining battery life. However, ensuring seamless, high-speed data exchange with minimal latency through BLE is challenging, as SC/AR/VR applications demand quick, reliable interactions for an immersive experience. Balancing the energy-efficient nature of BLE with the high bandwidth and rapid response required by SC/AR/VR systems is a critical task for hardware designers.
The Role of Alif Ensemble® in SC/AR/VR
Alif Semiconductor has designed their Ensemble® family of MCUs and fusion processors to help bridge the gap between expensive and power-hungry CPU/GPU processors and low-power MCUs for SC/AR/VR applications. Their Ensemble® family of 32-bit MCUs and fusion processors outperform typical 32-bit MCUs used for TinyML without sacrificing power efficiency, size, or cost. Here are some ways that Alif Ensemble® supports the needs of SC/AR/VR wearables:
Enhancing Real-time Processing for SC/AR/VR
At the heart of the Alif Ensemble® family lies a multi-core architecture, combining Cortex-A32 microprocessors and Cortex-M55 microcontrollers. The Cortex-A32 cores at up to 800 MHz are dedicated to high-level operating systems and graphics rendering, while the Cortex-M55 cores at up to 400 MHz with Helium SIMD vector extensions handle real-time control and responsiveness. The result is a harmonious blend of raw processing power and real-time execution capabilities, ensuring that even the most intricate 3D environments are rendered swiftly and responsively.
Within this, the architecture places a strong emphasis on minimizing latency, a critical factor in SC/AR/VR applications. With real-time cores specifically optimized for low-latency execution, data processing is lightning-fast. This translates to an SC/AR/VR experience that mirrors real-world interactions with an imperceptible delay, effectively mitigating motion sickness and heightening the sense of immersion.
The Alif Ensemble® family is also optimized for executing A/ML for tasks such as gesture recognition, object detection, and predictive rendering, and incorporates Ethos-U55 microNPUs, which deliver remarkable AI/ML acceleration. As a result, complex AI algorithms execute at speeds significantly faster than previous-generation processors, with up to 800x acceleration for image classification compared to traditional Cortex-M cores. The Alif Ensemble® family of Arm-based 32-bit microcontrollers scales up to 3670 DMIPS combined compute performance and over 250 GOPS AI performance.
Graphics and Image Rendering
At the core of Alif Ensemble® lies a dedicated graphics and imaging subsystem. This subsystem is purpose-built to handle the intricate tasks of camera input processing and display output generation all within a compact footprint.
It also features specialized hardware components and interfaces designed to efficiently capture and process visual data from cameras. This ensures that real-world surroundings are accurately integrated into the virtual environment, enhancing the overall realism of the AR/VR experience.
They also feature a dedicated subsystem for generating high-quality display output. Whether it’s rendering lifelike 3D objects or displaying immersive landscapes, the MCU’s graphics capabilities ensure that the visual output is smooth, vibrant, and true to the intended design.
Sensor Integration and Data Processing
Alif Ensemble® is also designed to seamlessly integrate a variety of sensors that are necessary for tracking user movements, capturing environmental data, and enhancing the overall realism of AR/VR simulations.
To this end, a notable architectural feature is their integration of a wide range of communication interfaces, to ensure interoperability with virtually any sensor. Some of these interfaces include I2C, SPI, UART, CAN, PWM, SDIO, ADC, MIPI, 10/100 Ethernet, and high-speed USB. This design choice streamlines the data flow between the MCU and sensors, minimizing latency and simplifying the data acquisition process.
Additionally, by eliminating external components from the Bill of Materials (BOM), Alif Ensemble® significantly reduces the overall footprint, power consumption, and cost of SC/AR/VR devices.
Power Efficiency
From a power-efficiency perspective, Alif Semiconductor’s microcontrollers and fusion processors encompass a wide spectrum of devices, ranging from single-core to quad-core configurations. This variability allows developers to select the optimal level of processing power for their SC/AR/VR applications, ensuring that power consumption is finely tailored to the task at hand.
Within the Alif Ensemble® family, low-power cores are strategically integrated to handle background tasks and environmental interactions. These cores are designed for efficiency, operating at lower clock speeds and power levels. Alif Semiconductor’s exclusive aiPM™ technology dynamically shifts between low-power and high-power cores based on real-time processing demands. When background tasks or sensor interactions are underway, low-power cores take the reins, conserving energy. When the SC/AR/VR application demands intense processing, high-power cores awaken, delivering the necessary performance while still managing power effectively.
By optimizing power usage and intelligently distributing processing tasks, users can enjoy longer usage periods without worrying about frequent recharging, enhancing the overall convenience and portability of such devices.
Conclusion
SC/AR/VR is widely considered the next evolution in personal computing, but before it can reach its full promise, there are many design challenges to be solved. The Alif Ensemble® family revolutionizes SC/AR/VR by enabling on-device processing, bridging market gaps, and redefining device criteria. Additionally, they empower real-time processing, graphics rendering, and sensor integration, enhancing SC/AR/VR experiences. As the industry shifts towards untethered SC/AR/VR, Alif Ensemble® will play a pivotal role in delivering lightweight, powerful, and immersive wearables.