MLTK Overview

Dataset Acquisition

• The first and most important step of model development is acquiring a representative dataset

• To create a robust model, the data used to train the model must be similar to the data the embedded device will see in the field

• The dataset is typically application-specific

• The MLTK comes with reference datasets

Common Dataset Issues

For many machine learning applications, acquiring a representative dataset may be challenging.

Many times the dataset will suffer from one or more of the following:

• The dataset does not exist - Need to manually collect samples
• The raw samples exist but are not “labeled” - Need to manually group the samples
• The dataset is “dirty” - Bad/corrupt samples, mislabeled samples
• The dataset is not representative - Duplicate/similar samples, not diverse enough to cover the possible range seen in the real-world

 

A clean, representative dataset is one of the best ways to train a robust model. It is highly recommended to invest the time/energy to create a good dataset!

Dataset Analysis and Preprocessing

• Once a dataset is acquired, it typically needs to be analyzed and preprocessed
• Preprocessing can be thought of as a way of amplifying the signals in the dataset so that the machine learning model can more easily learn its patterns and make accurate predictions
• Whatever preprocessing is done to the dataset during model training must also be done on the embedded device at run-time
• The MLTK features:
  • Embedded model parameters so the preprocessing settings used during training may also be used on the embedded device at runtime   
  • Support for custom C++ Python wrappers - these allow for sharing code between Python during model training and the embedded device at runtime                               
  • Audio Utilities to preprocess and visualize audio data

Audio Utilities

• The MLTK features several tools to aid the development of audio-based ML models (e.g. Keyword Spotting)

• AudioFeatureGenerator - a library to convert streaming audio into spectrograms for classification by a Convolutional Neural Network (CNN)

• view_audio - command to visualize real-time audio samples as a spectrogram

• classify_audio - command to classify real-time microphone audio

Audio Feature Generator

• A common audio preprocessing technique is converting audio signals to spectrograms
  • A spectrogram is a 2D, greyscale image of an audio sample
• The MLTK features the AudioFeatureGeneator Python component which converts a raw audio sample into a spectrogram. It features:
  • A C++ Python wrapper that enables the model training scripts to use the exact same source code as the embedded device at runtime
  • A Gecko SDK component so the firmware application can easily convert the microphone audio into spectrograms
  • Embedded model parameters so the settings used to generate the spectrograms during training are also used by the embedded device

View Audio Command

• The MLTK features a utility to view the spectrograms generated by AudioFeatureGenerator in real-time
• This allows for adjusting the AudioFeatureGenerator settings and immediately seeing how the spectrogram changes

> mltk view_audio

Classify Audio Command

• The MLTK features a utility to use a trained keyword spotting model to classify real-time microphone audio
• The audio_classifier application can run on Windows/Linux or on an embedded target

> mltk classify_audio

Model Design

• The standard Tensorflow/Keras API is used to design the machine learning model
• The model can be designed from scratch or existing model architectures may be used
• The MTLK comes with reference models
• The entire model and all configuration parameters are defined in a single Python script called the Model Specification
• With the model specification the:
  • profile command may be used to profile the model to determine latency, energy, and CPU usage
  • summarize command may be used to generate a text summary of the model
  • view command may be used to view the model in an interactive diagram
  • train command may be used to train the model

Create

Model Specification

Profile Model

Train Model

Evaluate Model

Copy .tflite model file

into Gecko SDK Project

Profile the model before training to ensure it fits within the hardware constraints

Evaluate trained model to see how well it can make predictions on unseen data samples

The output of model training is an archive file containing the trained model files

The model specification is a single Python script containing everything needed to design, train and evaluate the model

The .tflite model file is loaded into the Tensorflow-Lite Micro Interpreter which executes on the embedded device

Model Specification

.py

Keras Model File

.h5

TF-Lite Model File

.tflite

Model Archive

.mltk.zip

Train Model

Quantize Model

The configuration defined in the model specification file is given to Tensorflow to train the model

The output of Tensorflow, the trained model, is a Keras .h5 model file

The Keras .h5 file is given to the Tensorflow-Lite Converter to generate a quantized, .tflite model file

The quantized, .tflite model file is programmed into the embedded device

The output of MLTK train command also generates a model archive file which contains all of the model files and training logs

TF-Lite Model File

.tflite

Add parameters

The MLTK adds additional parameters to the .tflite model file which the Gecko SDK parses during project generation