In the tech industry, you can scarcely have a conversation without someone mentioning inference, artificial intelligence (AI), and machine learning (ML). However, it's important to note that while all these terms are interconnected, they also differ significantly.
In this article, we'll explain the fundamental differences and highlight the importance of using tensor processing-based edge AI technology, particularly in edge and embedded systems. Compared to solutions based on graphics processing units (GPUs), tensor processing units (TPUs) offer more efficient and cost-effective performance. We'll also provide some example use cases illustrating where you might encounter edge AI solutions in the future.
Fundamentals of ML and Inference
ML refers to the methodology of training models using representative data to enable machines to learn how to perform tasks. This process can be highly computationally intensive, generating trillions of operations per new training data point. The iterative nature of the training process, combined with the enormous training datasets required to achieve high accuracy, drives the demand for extremely high-performance floating-point processing. ML training is best implemented as data center infrastructure, where high capital and operational costs can be justified by amortizing them across numerous customers.
Inference involves using trained models to generate potential matches for new data relevant to the representative data upon which the model was trained. Inference aims to deliver rapid answers within milliseconds. Examples of inference include speech recognition, real-time language translation, machine vision, and advertising insertion optimization decisions. While inference requires only a fraction of the processing power needed for training, it still far exceeds what traditional central processing unit (CPU)-based systems can deliver, particularly for computer vision applications. This is why so many companies are turning to tensor-based acceleration solutions-whether as IP on SoCs or as in-system accelerators-to achieve the sub-second response times required at the edge. The reality is that spending even a minute or a few seconds processing images in a vision system is not very useful. Industrial vision systems are seeking millisecond-level processing speeds.
Separating Training and Inference
Deploying the same hardware used for training to handle inference workloads may result in over-provisioning inference machines with accelerators and CPU hardware. GPU solutions developed for ML over the past decade are not necessarily the optimal choice for large-scale deployment of ML inference technologies. The diagram below perfectly illustrates the comparison between TPU accelerators and GPU accelerators. It clearly shows that TPU accelerators deliver lower power consumption, reduced costs, and higher efficiency compared to GPU-based AGX solutions, while still providing compelling performance levels for inference applications.
Another critical consideration when approaching ML training and inference solutions is the software environment. Today, numerous popular libraries are in use, such as CUDA for NVIDIA GPUs, ML frameworks like TensorFlow and PyTorch, optimized cross-platform model libraries like Keras, and more. These toolkits are essential for developing and training ML models, but inference applications require a different, smaller set of software tools.
Inference toolkits focus on running models on target platforms. They support porting trained models to platforms, which may involve some operator transformations, quantization, and host integration services. However, this represents a relatively straightforward set of functionalities compared to those required for model development and training.
Inference tools benefit from starting with a standardized representation of the model. The Open Neural Network Exchange (ONNX) is the standard format for representing ML models. As the name implies, it is an open standard managed as a Linux Foundation project. Technologies like ONNX enable the decoupling of training and inference systems, granting developers the freedom to choose different optimized platforms for each.
Example Visual Applications
As ML and inference processor technologies continue to advance and evolve, applications are proliferating. Below are just a few places you might encounter this technology in the future.
Edge servers in enterprises such as factories, hospitals, retail stores, and financial institutions. For instance, in industrial settings, AI can assist with inventory management, defect detection, and even predictive maintenance before issues arise. In retail, it enables features like pose estimation, using computer vision to detect and analyze human posture. Data from this analysis helps brick-and-mortar retailers better understand human behavior and foot traffic within their stores, allowing them to optimize store layouts for maximum sales and customer satisfaction.
High-precision/high-quality imaging for applications including robotics, industrial automation/inspection, medical imaging, scientific imaging, surveillance and object recognition cameras, and photonics. For instance, machine learning methods have demonstrated the ability to detect cancer by processing digital X-rays. This process involves developing an ML model designed to process X-ray images, typically using trained semantic segmentation algorithms to identify cancerous lesions. During training, cancer images identified by radiologists are used to teach the network what is not cancer, what is cancer, and how different types of cancer appear. The more an ML model is trained, the better it becomes at maximizing correct diagnoses and minimizing misdiagnoses. This means machine learning relies not only on intelligent model design but equally on vast amounts (tens of thousands to millions) of carefully curated data examples where cancer has been expertly identified.
Smart Shopping Carts-Several companies are developing and deploying intelligent shopping systems that recognize products not by their UPC barcodes, but by the visual appearance of the packaging itself. This feature allows shoppers to simply place items into the cart or onto the checkout system without needing to locate the UPC code and scan it with a UPC laser scanner. This technology makes the shopping process more accurate, faster, and more convenient.
Making the Right Decision
Companies must evaluate all available solutions today and select the optimal one based on their specific use case. They also cannot simply assume all AI solutions are best implemented on GPU devices, as TPU-based solutions offer higher processing efficiency and lower silicon utilization, thereby reducing power consumption and costs.