An Introduction to Embedded Machine Learning
According to Wikipedia, machine learning is a subfield of artificial intelligence concerned with developing and studying statistical algorithms that can learn from data and generalize to unseen information, thus performing tasks without explicit instructions.
These days, it’s often synonymous with cloud computing or powerful servers processing tons of data to perform tasks such as spam email filtering, movie recommendations, and predicting traffic patterns during a commute; however, this approach can be quite limiting.
Since internet connectivity is a prerequisite for cloud applications, not all machine learning can be done in the cloud. Just imagine you’re in a self-driving car on the highway, and it loses connection to the internet. How scary would that be? Sometimes, machine learning applications must perform processing locally to get the job done. This means putting plenty of computing power into small devices.
In recent years, researchers have made significant progress in running machine learning algorithms on tiny microcontrollers. This has led us into the era of embedded machine learning, or TinyML, as it’s often referred to. While performing machine learning tasks on smaller board computers is not new, being able to run them on microcontrollers, which are less powerful and complex than small board computers and contain less advanced processors, opens up a world of opportunities that is likely to foster a new generation of AI-powered electronics.
What is Machine Learning?
One approach to writing a computer program is machine learning (ML). It’s a technique used to build programs by giving computers the ability to learn without requiring explicit programming. It works by processing raw data and transforming it into useful information at the application level.
For example, a program may be designed to determine when an industrial machine is likely to break down or when it has broken down based on historical data collected from various sensors. Another machine learning program may involve collecting raw audio data in anticipation of a specific wake word or phrase, like “Hey, Alexa.” Upon hearing the trigger phrase, a smart home device will be activated – this type of program is known as automatic speech recognition (ASR).
In contrast to traditional computer programs, the developer does not prescribe the exact logic that must be followed in machine learning applications. Instead, machine learning programs use specialized algorithms to extract rules from data throughout the training process.
Traditional software consists of an engineer who explicitly codes an algorithm that takes input, applies the same logic every time, and returns an output. The same output will always come from an identical input. If traditional software were to be used to predict when an industrial machine would break down, an engineer would need to know which metrics in the data indicate a problem and then write code that specifically looks for them to forecast breakdowns. This method works well for several problems.
For instance, developing a program to determine whether water is boiling based on its present temperature and altitude is easy because we know water boils at 100°C at sea level.
However, determining the precise combination of variables that foretells a particular state can be extremely challenging in many cases. Few requirements are as simple as the boiling water example above. In contrast, in our industrial machine example, several combinations of temperature, vibration level, and production rate may point to an issue but are not immediately apparent from the data. Getting the machine to figure out what those combinations look like can save years of guesswork and produce insights leading to better products and superior business economics.
But it’s not as simple as writing a machine learning model. First, you need to train it. Before creating a machine learning program, a sizable amount of data must be collected. This data is then fed to a machine learning algorithm, which learns patterns and invents, creates, and deduces its own specific set of rules about the data to make generalizations on new data.
In other words, machine learning practitioners are not required to understand which metrics in the data to pay attention to. The machine learning algorithm builds a model of the system based on the data it’s supplied and then uses this model to make predictions in a process called inference.
Machine learning serves tasks involving pattern recognition well, especially when dealing with complicated patterns that could be challenging for a human observer to recognize. Still, like with any tool, it’s not without its problems.
Machine Learning on Embedded Devices
Plenty of research has been conducted in the past few years on improving the efficiency of machine learning algorithms. Breakthroughs in this field of research have enabled developers to run complex neural networks and other machine learning algorithms on smaller, more power-efficient devices instead of massive servers in the cloud – this is where embedded systems come into the picture. But before we get there, let’s look at what machine learning models can run on…
Working in the Cloud
Machine learning is a resource and power-intensive task, especially when working with large datasets. For instance, let’s say a machine learning model is being developed to predict fraud…
Millions of rows and dozens of columns of financial data may be used, and the model may need to process hundreds or even thousands of transactions per second at peak activity. This high volume and speed will require large, reliable GPU clusters that can be scaled up or down to meet demand. Machine learning is mostly conducted on these distant computing servers, often called “the cloud.”
Also, training and implementing large machine learning models locally is computationally expensive, especially for the current state-of-the-art (SOTA) models. High-end GPU units are required to scale models to meet large-scale needs, which implies that during periods of low utilization, they will be mainly idle. Consequently, costly servers gather dust, despite requiring extensive maintenance. Utilizing machine learning in the cloud means users only pay for what they need, enabling them to preserve resources.
However, it also means some security is being forfeited. Natural disasters and hacker attacks can always cause issues for one’s data center, including data loss, data leaks, and downtime. Machine learning models can also be compromised. For example, if an attacker obtains access to an organization’s AWS account credentials, they can use them to alter the model’s predictions. This type of attack can often go undetected by administrators and customers.
In addition, the cloud service provider is responsible for data security. Though it’s possible to take legal action against cloud service providers if data is lost, stolen, or compromised, there’s no assurance the data can be retrieved once this occurs – this is because cloud providers may encrypt or erase records to comply with legal obligations. There’s also the matter of latency. Since large amounts of data are being transferred to the cloud, it’s possible for there to be considerable network latency.
Shrinking the Models
You may have read about different LLMs and noticed people referring to the number of nodes or connections or something… as a way of indicating how big they are. Big models need big iron to run, so they are restricted to the cloud and supercomputers.
But what’s happening now is people taking a subset of those models and shrinking them down to management size for tiny devices. For example, you might carve out computer vision and optimize it to run on one of the modern AI chips that is becoming available. Then you would compile your model to run specifically and only on that chip.
In 2017, Google introduced a new version of TensorFlow called TensorFlow Lite. TensorFlow Lite is optimized for use on embedded and mobile devices. It’s an open-source, product-ready, cross-platform deep learning framework that converts a pre-trained TensorFlow model to a special format optimized for speed or storage.
TensorFlow Lite was a new version released by Google in 2017. It has been designed with embedded and mobile devices in mind. This is a cross-platform, open-source, product-ready deep learning framework that transforms a pre-trained model into a unique format that may be tailored for storage capacity or speed.
It eliminates the need to program the matrix multiplications manually and enables high-level neural network operation on microcontrollers. In addition, numerous code optimizations and microcontroller manufacturer efforts have resulted in substantially quicker operation of these frameworks and other ML algorithms on these small devices.
Using a microcontroller to perform matrix operations is not new. The magic in the embedded world starts when we run complex machine learning algorithms on these microcontrollers, which has been made possible in recent years due to the combination of hardware and software optimization and ML frameworks. This creates a whole new set of opportunities; for example, we can build voice-activated systems embedded into our homes or cameras that can identify specific objects or people.
The Benefits of Embedded Machine Learning
Using machine learning on embedded devices has several significant benefits. Jeff Bier’s acronym, BLERP, succinctly captures the main benefits. They are as follows:
- Bandwidth – Little to no internet connectivity is required for inference. This means ML algorithms on edge devices can extract valuable information from data that would otherwise be inaccessible because of bandwidth limitations.
- Latency – On-device machine learning results in reduced latency. This is because the data does not need to be transferred to a server for inference since the model operates on the edge device. In other words, models can respond to inputs in real-time.
- Economics – Since they require so little power, microcontrollers can run continuously for extended periods of time before needing to be recharged. Moreover, as no information transfer occurs, a large cloud server infrastructure is unnecessary, conserving money, energy, and resources.
- Reliability – Embedded machine learning is more reliable overall. At any time, cloud connections can unexpectedly cut out, which reduces their intrinsic reliability.
- Privacy – User privacy is preserved, and misuse is less likely when data is processed on an embedded system rather than in the cloud. Due to the edge computing nature of the architecture, your data is not stored on servers. Since no information is transferred to servers, data privacy is more likely to be upheld.
Conclusion
In recent years, embedded machine learning has gained popularity across various industries thanks to the emergence of ecosystems (e.g., novel advancements in computer architecture and the breakthroughs in machine learning) supporting it with hardware and software. This has facilitated the integration of machine learning models into low-power systems, such as microcontrollers, hence creating a multitude of novel prospects.
For developers looking to create apps for Internet of Things (IoT) devices, embedded machine learning offers several advantages, including reliability, low latency, energy savings, data privacy, and no network dependencies.
Adopting these cutting-edge trends and technologies is necessary for those who want to remain competitive. Having a reliable technology partner who can provide information, guidance, and cooperation throughout the development process is essential to ensuring it’s implemented correctly. By adding experience to the design and implementation process, businesses can ensure they are developing cutting-edge, safe, and embedded machine learning systems that operate to the highest standards.
Give us a quick shout out if you are in need of a reliable development partner for embedded machine learning.
