From Lab to Device: Deploying Neural Networks on Embedded Systems

Neural networks achieve astonishing results - mostly powered by cloud infrastructure. But for autonomous or safety-critical systems, a stable connection cannot be assumed. These applications must run reliably on embedded hardware with limited compute, memory, and real-time constraints. This talk walks through the full development pipeline, with a focus on optimizing and deploying neural networks for production use on resource-constrained devices.

Target Audience: Developers, ML Engineers, Architects
Prerequisites:Basic understanding of neural network and training pipelines
Level: Introductory

Extended Abstract:
Outline

1) Intro & Pipeline Overview (5 min)

Outline of the end-to-end ML pipeline: data → training → validation → optimization → deployment → monitoring.

2) Optimization for Low-Resource Targets (15 min)

• Pruning strategies to reduce model size and latency
• Knowledge distillation and transfer learning for lightweight performance
• Quantization: from post-training to quantization-aware training (QAT)

3) Deployment Stack (10 min)

• ONNX as a portable model format
• TensorRT for high-performance inference on NVIDIA edge platforms
• C++ integration considerations

4) Retraining & Fine-Tuning (5 min)

• Handling data drift and domain shifts
• Designing retraining pipelines
• Challenges of finetuning quantized models

5) Q&A (10 min)

Attendees will leave with a solid understanding of the practical steps and trade-offs involved in deploying neural networks on embedded systems, with actionable insights for ML model deployment on edge devices.