Blog | Why On‑Device Generative AI for Rea...

Overview

Adding live captioning to a web‑based video conference sounds appealing: users get instant text, the server stays out of the loop, and the feature can be rolled out with a few lines of JavaScript. This tutorial walks through a minimal implementation using TensorFlow.js and the Web Audio API, then examines why the same approach can expose sensitive audio, drain battery life, and create unpredictable latency spikes.

Setting Up the Development Environment

The stack consists of a plain HTML page, a small npm script to bundle the model, and a service worker that caches the model files. Start by creating a new project folder and installing the required packages:

mkdir caption‑demo
cd caption‑demo
npm init -y
npm install @tensorflow/tfjs @tensorflow-models/speech-commands
npm install --save-dev parcel-bundler

The parcel-bundler command will compile the ES modules into a single bundle.js that the browser can load without a build server.

Loading a Small Speech‑to‑Text Model

TensorFlow.js ships a pre‑trained speech‑commands model that can recognize a limited vocabulary. For a demo we’ll use it to transcribe simple phrases. Create src/index.js with the following code:

import * as tf from '@tensorflow/tfjs';
import * as speechCommands from '@tensorflow-models/speech-commands';

let recognizer;

async function initModel() {
  // Load the model from the CDN; the file size is ~4 MB.
  recognizer = speechCommands.create('BROWSER_FFT');
  await recognizer.ensureModelLoaded();
  console.log('Model loaded');
}

async function startListening() {
  await recognizer.listen(result => {
    const scores = result.scores; // probability of each word
    const words = recognizer.wordLabels();
    const maxScore = Math.max(...scores);
    const maxIndex = scores.indexOf(maxScore);
    const transcript = words[maxIndex];
    displayCaption(transcript);
  }, {
    overlapFactor: 0.5,
    probabilityThreshold: 0.75
  });
}

function displayCaption(text) {
  const el = document.getElementById('caption');
  el.textContent = text;
}

// UI wiring
document.getElementById('startBtn').addEventListener('click', async () => {
  await initModel();
  await startListening();
});

The code above does three things: loads the model, starts a continuous audio stream, and updates a <div id="caption"> element with the most likely word. The model runs entirely in the browser, which means the raw audio never leaves the client.

HTML Boilerplate and Service Worker Registration

Next, create src/index.html. The page includes a button to start captioning and a placeholder for the live text.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Live Caption Demo</title>
  <style>
    body { font-family: Arial, sans-serif; padding: 2rem; }
    #caption { margin-top: 1rem; font-size: 1.5rem; color: #333; }
  </style>
</head>
<body>
  <button id="startBtn">Start Captioning</button>
  <div id="caption">(caption will appear here)</div>

  <script src="bundle.js"></script>
  <script>
    if ('serviceWorker' in navigator) {
      navigator.serviceWorker.register('sw.js')
        .then(() => console.log('Service worker registered'));
    }
  </script>
</body>
</html>

The service worker (src/sw.js) simply caches the model files so that subsequent loads are instant:

self.addEventListener('install', event => {
  event.waitUntil(
    caches.open('caption-demo').then(cache =>
      cache.addAll([
        '/bundle.js',
        'https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@latest/dist/tf.min.js',
        'https://cdn.jsdelivr.net/npm/@tensorflow-models/speech-commands@latest/dist/speech-commands.min.js'
      ])
    )
  );
});

self.addEventListener('fetch', event => {
  event.respondWith(
    caches.match(event.request).then(response =>
      response || fetch(event.request)
    )
  );
});

Build the bundle with Parcel:

npx parcel build src/index.html --out-dir dist --public-url ./

The generated dist folder can be served by any static host. At this point the demo works locally, and the model runs entirely on the client device.

Hidden Risks and Performance Pitfalls

Although the implementation is straightforward, several hidden liabilities surface when this pattern is pushed into production:

Audio Leakage via Side‑Channels: Even though the raw microphone stream never leaves the browser, browsers expose microphone activity to the OS scheduler. Malicious extensions can infer speech content by monitoring CPU usage spikes.
Battery Drain on Mobile Devices: Continuous FFT computation at 16 kHz consumes a noticeable portion of the device’s power budget. Real‑time captioning on a 4‑hour video call can reduce battery life by up to 30 %.
Latency Variability: The model size (~4 MB) must be downloaded and compiled into WebGL shaders. On low‑end hardware the initialization can take several seconds, creating a “cold‑start” gap that users notice.
Model Staleness: The bundled model cannot be updated without forcing a full page reload. In a regulated environment, outdated language models could mis‑recognize legally required terminology.
Privacy Compliance: Some jurisdictions consider on‑device processing of voice data as “personal data processing”. Even without network transmission, the act of capturing audio may require explicit consent under GDPR or CCPA.

The following snippet shows how to monitor CPU usage and abort the recognizer if the process exceeds a threshold, mitigating battery impact:

let cpuOverload = false;
setInterval(async () => {
  const usage = await navigator.performance.now(); // simplified example
  if (usage > 150) { // arbitrary threshold in ms per frame
    cpuOverload = true;
    recognizer.stop();
    console.warn('CPU overload – captioning paused');
  } else if (cpuOverload) {
    cpuOverload = false;
    await recognizer.listen(...);
    console.info('CPU load normal – captioning resumed');
  }
}, 5000);

Security and Best Practices

If you still need on‑device captioning, follow these safeguards:

Request microphone access only when the user explicitly clicks “Start Captioning”. Do not request permission on page load.
Encrypt the model file at rest using the SubtleCrypto API and decrypt it in memory after download.
Implement a watchdog that suspends the recognizer when the page is hidden (document.hidden) or when battery level drops below 20 % (navigator.getBattery()).
Provide a clear opt‑out mechanism that disables the feature and deletes any cached model files.
Log only anonymized statistics (e.g., “captioning enabled”, “model version”) and never store raw audio or transcription results on the server.

“Running a neural network in the browser is a powerful demonstration, but developers must weigh the convenience against privacy, battery, and compliance costs before shipping it at scale.”

Conclusion

The tutorial above proves that on‑device generative

♥ Enjoyed this article? Use the like banner at the top of the page to let us know — you can like it more than once! Each additional like from the same reader carries a little less weight than the first, so our appreciation scores reflect genuine enthusiasm rather than accidental clicks. Your feedback helps us understand which topics resonate most.

Why On‑Device Generative AI for Real‑Time Captioning Is a Hidden Liability