Embedded AI · SCUT

Embedded Smart Home Terminal

Local speech recognition, connected device control, and a voice-dialogue interface.

Project period: Sep 2024 – Jan 2025
Team project · Supervised by Prof. Zhicong Huang

35 s · Two excerpts at normal speed. Muted preview; original Mandarin audio available in the controls. Open clip · Full recording

Working prototype

From a spoken command
to an embedded response.

I worked across speech-data collection, MFCC preprocessing, neural-network training, ESP32 firmware, and system integration.

Local command classification, OneNET control, and iFLYTEK Spark dialogue are separate subsystems.

My Contribution

01

Speech data & model

Collected and processed speech data, implemented MFCC features, and trained and evaluated the command-recognition neural network.

02

Embedded integration

Contributed to ESP32 firmware, connected the hardware and software subsystems, and tested the prototype.

03

Team delivery

Led project planning, task coordination, and team-wide integration.

Three connected subsystems

Offline speech training and local inference with separate OneNET device control and Spark dialogue paths.
Offline model training, local classification, device control and the separate dialogue interface. Click to enlarge.

Local recognition

STM32F407 classifies seven Chinese command classes from microphone audio.

OneNET Device Control

ESP32-C3 connects curtain, air-conditioner, and LED commands through MQTT.

iFLYTEK Spark Voice Dialogue

ESP32-Wroom supports conversational responses, including weather and time queries.

The architecture deliberately separated latency-sensitive local commands from cloud-connected functions. Device commands were classified on the embedded platform before being routed to the OneNET control path, while open-ended speech requests entered a separate speech-to-text and language-model pipeline.

System architecture separating local command recognition, OneNET device control, and iFLYTEK Spark voice dialogue
The three-part architecture: local command recognition on STM32F407, OneNET device control through ESP32-C3, and iFLYTEK Spark dialogue through ESP32-Wroom.

Training & validation

89.1%89.1% validation accuracy on the seven-class command task.
7 classesDevice commands and entry into chat mode.
0.41Recorded validation loss.

Microphone audio was captured as PCM data and transferred to the STM32F407 for preprocessing, MFCC feature extraction, and 2D neural-network classification. The model distinguished seven Chinese command classes: turning the light, air conditioner, and curtain on or off, plus entering the chat mode.

The recorded validation result was 89.1% accuracy with a loss of 0.41. The confusion matrix documents class-specific performance and the remaining error modes, keeping the reported result scoped to the validation set.

Audio pipeline from PCM capture and MFCC processing to neural-network command classification Validation dashboard showing 89.1 percent accuracy, 0.41 loss, and a seven-class confusion matrix
Local recognition pipeline and the final seven-class validation result.

What the prototype establishes

The final demonstration connected the complete interaction chain: local recognition of seven device and chat commands, OneNET-based control of demonstrated smart-home devices, and iFLYTEK Spark responses displayed on the TFT or played through the speaker. The result should be understood as a functional team prototype. The project did not include validated measurements of power consumption or long-duration network reliability, so these remain future evaluation goals.

Cloud-Connected Functions

OneNET Device Control

The ESP32-C3 used Wi-Fi AT firmware to connect the embedded controller to OneNET over MQTT. A demonstration interface built with HBuilderX displayed device state and issued control commands for the curtain, air conditioner, and LED.

iFLYTEK Spark Voice Dialogue

The language-model subsystem handled conversational interaction and information requests such as weather and time queries. Speech was transcribed through an STT service, sent to iFLYTEK Spark, and returned as text on the TFT display or audio through the speaker. This subsystem did not generate device-control commands.

OneNET-connected smart-home interface for curtain, air-conditioner, and LED control iFLYTEK Spark interaction flow from speech-to-text through language-model response and text or audio output
The OneNET control interface and the separate iFLYTEK Spark dialogue pipeline.

Hardware Prototype

The prototype combined an STM32F4-Discovery board, ESP32-Wroom and ESP32-C3 modules, a 1.8-inch ST7735S TFT display, an INMP441 digital microphone, a MAX98357 I2S audio amplifier, and a speaker. These development boards and peripheral modules supported local recognition, cloud connectivity, and text and audio feedback within one integrated demonstration.

Prototype hardware including STM32F4-Discovery, ESP32-Wroom, ESP32-C3, TFT display, microphone, amplifier, and speaker ESP32-based audio-interface prototype with TFT display, microphone, amplifier, and speaker TFT display presenting a Chinese-language response from the voice assistant
Development hardware, the ESP32-based audio interface, and TFT response output used in the working prototype.
Future Directions
  • Improve command-recognition accuracy across speakers and acoustic environments.
  • Measure and optimize system power consumption.
  • Expand compatibility with a wider range of smart-home devices.
  • Strengthen data-security and privacy protection across cloud-connected functions.