Faster Inferencing with Lower Power Consumption

Introduction

Design engineers who work with microcontrollers tend to find that they quickly run up against the barriers of performance and power when they try to add AI functionality to a typical embedded control system.
The levers to pull when facing performance constraints in a conventional control application are normally CPU frequency and memory: increasing the CPU core’s speed and addressing more memory should result in a predictable uplift in throughput, and reduction in latency.

Ethos-U55 in Ensemble AI MCU Reduces Power Consumption

For common AI applications such as speech recognition or object detection, however, these same levers do not seem to work. Faced with long wait times for a conventional 32-bit MCU to perform inferencing at the edge, design engineers have tried upgrading their hardware – for instance, by migrating from an Arm® Cortex®-M0+ core to a Cortex-M4. In AI applications, however, the effect tends to be negligible.

There is a technical argument to be made for this: it boils down to the parallel nature of AI inferencing operations – a CPU excels in executing sequential instructions. So running a machine learning application at higher speed on a CPU is like spinning the wheels of a car: it uses more energy, but forward progress is slow.

This argument underlies the hybrid architecture of the Alif Semiconductor Ensemble™ family: like a conventional MCU, the Ensemble AI MCU includes a CPU – the Cortex-M55 for the control part of the application. But the Ensemble AI MCUs also have a dedicated hardware accelerator for AI functions, the Arm Ethos™-U55 micro neural processing unit (microNPU). The theory is that an architecture combining a dedicated AI accelerator with a CPU will perform better at machine learning tasks than an architecture based on a CPU alone.

But is the theory borne out in practice?

The answer is in the numbers. In a demonstration at Arm DevSummit 2021, an Ensemble E3 AI MCU (with single Cortex-M55 and Ethos-U55 cores) performed single inference image classification using the MobileNet V2 1.0 model 78 times faster with the microNPU enabled than when the same MCU only used its Cortex-M55 MCU core. Execution time for the Ensemble E3 when the microNPU is enabled was 8ms compared with 624ms for execution on the Cortex-M55 core alone, and please keep in mind that the Cortex-M55 is several times faster than earlier generation Cortex-M class cores for these types of workloads, meaning that an order core, such as the Cortex-M4 or Cortex-M33 would take even longer. The faster execution speed also has a major impact on the energy consumed for each inference. When accelerated by the microNPU, the energy consumed for the inference operation was only 3mJ, 76 times less than the 228mJ used when the inference operation was executed on the MCU core.

In fact, across a broad range of common machine learning functions, the addition of the Ethos-U55 core increases machine learning performance and reduces energy consumption by huge multiples, as the table shows. And the benchmarking provides a comparison to the Cortex-M55 core, which is the CPU that Arm has most modified for compatibility with machine learning operations: the contrast of benchmarking data would be even more stark against other Cortex-M cores.

The extra performance makes it possible to perform applications far more effectively than on a CPU-only MCU: for instance, Alif Semiconductor has demonstrated a facial recognition and tracking application, hosted on the Ensemble E3, running at 148 inferences per second, substantially faster than is possible using conventional MCUs.

High Efficiency (HE) System: Cortex-M55 and Ethos-U55 128MAC at 160MHz
Model	Accelerated inferencing				CPU-bound inferencing (on Cortex-M55)				Improvement w. Acceleration
	Time ms	Power mW (∆)	Current mA (∆)	Energy mJ (∆)	Time ms	Power mW (∆)	Current mA (∆)	Energy mJ (∆)	Time	Energy efficiency
KWS1: MicroNet Medium (ARM)	15.9	8.8	2.6	0.14	326	3.4	1.0	1.27	21x	19x
Object Detection2: YOLO-Fastest (face trained)	18.6	14.2	4.2	0.27	1373	5.4	1.6	8.3	74x	67x
Auto Speech Recognition4: Tiny ASR (Wav2letter)	78.6	10.0	3.0	0.69	8562	7.4	2.2	62.5	109x	104x[TS1]

High Performance (HP) System: Cortex-M55 and Ethos-U55 256MAC at 400MHz
Model	Accelerated inferencing				CPU-bound inferencing (on Cortex-M55)				Improvement w. Acceleration
	Time ms	Power mW (∆)	Current mA (∆)	Energy mJ (∆)	Time ms	Power mW (∆)	Current mA (∆)	Energy mJ (∆)	Time Improve	Energy Improve
KWS1: MicroNet Medium (ARM)	6.8	25.7	7.6	0.17	137	19.3	5.7	2.62	20x	18x
Object Detection2: YOLO-Fastest (face trained)	7.3	33.8	10.0	0.25	657	21.3	6.3	13.7	90x	76x
Image Classification3: MobileNet v2	20.1	43.3	12.8	0.86	2707	24.0	7.1	62.4	135x	108x
Auto Speech Recognition4: Tiny ASR (Wav2letter)	27.9	29.6	8.9	0.89	4534	17.3	5.2	77.8	163x	138x

---

Conclusion

The evidence of the numbers is clear: an architecture optimized for machine learning operations, such as the Ensemble AI MCU, produces markedly superior machine learning performance. So microcontroller users now have a better lever to pull to solve problems with slow or unreliable AI performance at the edge.