Opening Remarks

The excitement surrounding on‑device generative AI has spilled over into the Internet of Things (IoT) sector. Engineers are eager to push a “complete‑stack” LLM into a sensor node, a smart lock, or an industrial controller, assuming that local inference automatically solves latency and privacy concerns. This article flips that narrative: it explains why the practice is fraught with hidden costs, and it walks you through a minimal, safe alternative that keeps the model off the firmware while still offering responsive AI features.

What “Embedding an LLM” Actually Means

In most firmware projects, “embedding” refers to compiling a binary blob of the model into the flash image. The model is then loaded into RAM at boot time and executed by a runtime library (often ONNX Runtime Lite or TensorFlow Lite Micro). The appeal is clear: the device never contacts the cloud for inference, and the developer can ship a single monolithic image.

The hidden internals, however, reveal three major problem areas:

  • Memory footprint: Even the most aggressively quantized 1‑B‑parameter LLM easily exceeds the 256 KB RAM ceiling of many MCUs.
  • Power draw: Continuous tensor operations keep the CPU out of deep‑sleep modes, cutting battery life dramatically.
  • Attack surface: A static model blob can be extracted, reverse‑engineered, and used to craft adversarial prompts that trigger undefined behaviour on the device.

Alternative Architecture: Off‑Device Inference with a Tiny Edge Proxy

The safer pattern separates the heavy lifting from the constrained node. The device streams a short, anonymized request to a nearby edge proxy (a Raspberry Pi, a Jetson Nano, or a small VM). The proxy runs the full LLM, returns the result, and the IoT firmware merely handles the transport layer. This approach retains low latency (local network < 10 ms) while keeping the firmware footprint minimal. 

// Minimal request payload (C struct)
typedef struct {
    uint8_t  magic[4];      // "REQ0"
    uint16_t seq_id;
    uint16_t payload_len;
    char     data[64];      // short, anonymized prompt
} iot_req_t;

// Minimal response payload
typedef struct {
    uint8_t  magic[4];      // "RES0"
    uint16_t seq_id;
    uint16_t payload_len;
    char     result[128];   // truncated answer
} iot_res_t;

The snippet above defines a compact binary protocol that fits comfortably within the typical 128‑byte UART or BLE packet size. The firmware never touches the model; it only packages the user input and parses the answer.

Step‑by‑Step: Building the Edge Proxy

Below is a concise Python script that uses onnxruntime to serve a quantized LLM over a TCP socket. The script is deliberately simple so you can adapt it to any edge hardware you have. 

# proxy_server.py
import socket
import struct
import onnxruntime as ort

HOST = '0.0.0.0'
PORT = 5555

# Load a 2‑bit quantized TinyLlama model (≈ 30 MB on disk)
sess = ort.InferenceSession('tinyllama_q2.onnx', providers=['CPUExecutionProvider'])

def handle_client(conn):
    # Read fixed‑size request header
    header = conn.recv(8)
    if len(header) < 8:
        return
    magic, seq_id, payload_len = struct.unpack('!4sHH', header)
    if magic != b'REQ0':
        return
    data = conn.recv(payload_len)
    prompt = data.decode('utf-8')

    # Run inference (simple greedy decode)
    inputs = {'input_ids': tokenizer.encode(prompt)}
    outputs = sess.run(None, inputs)
    answer = tokenizer.decode(outputs[0][0])

    # Build response
    res_payload = answer.encode('utf-8')[:128]
    res_header = struct.pack('!4sHH', b'RES0', seq_id, len(res_payload))
    conn.sendall(res_header + res_payload)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.bind((HOST, PORT))
    s.listen()
    print(f'Proxy listening on {HOST}:{PORT}')
    while True:
        conn, _ = s.accept()
        with conn:
            handle_client(conn)

The proxy does three things worth noting:

  1. It keeps the model in RAM only on the edge box, not on the MCU.
  2. The network payload is limited to a few dozen bytes, avoiding bandwidth spikes.
  3. Because the proxy runs on a more powerful OS, you can add authentication, rate‑limiting, and logging without blowing the MCU’s resources.

Integrating the Proxy Call into Firmware

On the MCU side, the code below shows how to open a TCP socket, send the request structure, and wait for a response. The example uses the lightweight lwIP stack, which is common on ESP‑32 and similar devices. 

// iot_client.c (simplified)
#include "lwip/sockets.h"
#include "string.h"

#define SERVER_IP   "192.168.1.42"
#define SERVER_PORT 5555

void send_prompt(const char *prompt) {
    int sock = socket(AF_INET, SOCK_STREAM, 0);
    if (sock < 0) return;

    struct sockaddr_in serv_addr = {
        .sin_family = AF_INET,
        .sin_port   = htons(SERVER_PORT),
        .sin_addr.s_addr = inet_addr(SERVER_IP)
    };
    if (connect(sock, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0) {
        close(sock);
        return;
    }

    iot_req_t req;
    memcpy(req.magic, "REQ0", 4);
    static uint16_t seq = 0;
    req.seq_id = htons(seq++);
    req.payload_len = htons(strlen(prompt));
    strncpy(req.data, prompt, sizeof(req.data)-1);
    req.data[sizeof(req.data)-1] = '\\0';

    send(sock, &req, 8 + ntohs(req.payload_len), 0);

    // Receive response header
    iot_res_t res;
    recv(sock, &res, 8, 0);
    if (memcmp(res.magic, "RES0", 4) != 0) {
        close(sock);
        return;
    }
    uint16_t res_len = ntohs(res.payload_len);
    recv(sock, res.result, res_len, 0);
    res.result[res_len] = '\\0';

    // Use the answer (e.g., display, act on it)
    printf("LLM says: %s\\n", res.result);
    close(sock);
}

By keeping the heavy inference off the MCU, the firmware stays under 30 KB of code, well within the limits of most 32‑bit microcontrollers. The device can still benefit from AI‑driven features without the hidden liabilities of a full‑scale model in flash.

Hidden Risks of Full‑Model Firmware Embedding

If you still decide to ship a model inside the firmware, be aware of these often‑overlooked dangers:

  • Unauditable updates: Changing the model requires a full firmware flash, which may be blocked by OTA policies or regulatory certification cycles.
  • Model extraction attacks: Attackers with physical access can read flash, decompress the model, and use it to generate malicious prompts that trigger undefined behaviour.
  • Regulatory compliance: Certain jurisdictions treat embedded AI as a “high‑risk” component, demanding documentation and third‑party evaluation that most firmware teams are not prepared for.

The proxy architecture sidesteps most of these concerns because the model lives on a device that can be patched, audited, and isolated from the production network.

Security and Best Practices

Whether you adopt the proxy pattern or still need to embed a tiny model, follow these guidelines: