Why Not to Deploy On‑Device Generative AI for Real‑Time Email Drafting in Web Apps

Introduction: The All‑ure of Browser‑Based LLMs

Embedding a language model in a web page promises instant, offline‑ready assistance for composing emails, proposals, or support tickets. The idea is seductive: download a TensorFlow.js model once, run inference locally, and never send user‑generated text to a remote API. However, the convenience comes at a steep price. This article walks through a minimal prototype, then uncovers the hidden technical, privacy, and operational liabilities that make this approach unsuitable for production‑grade email drafting tools.

Step 1: Pulling a Small Pre‑Trained Model

Many developers start by selecting a “tiny” transformer that fits under 50 MB, such as distilGPT‑2 exported for TensorFlow.js. Below is a simple HTML page that loads the model and exposes a generate() function.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>On‑Device Email Draft</title>
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script>
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/gpt2"></script>
</head>
<body>
  <textarea id="prompt" rows="4" cols="60">Dear team,</textarea><br>
  <button id="run">Generate</button>
  <pre id="output"></pre>

  <script>
    let model;
    async function loadModel() {
      model = await tf.loadGraphModel('https://example.com/distilgpt2/model.json');
      console.log('Model loaded');
    }

    async function generate() {
      const prompt = document.getElementById('prompt').value;
      // Tokenize – placeholder, real tokenizers are more complex
      const inputIds = tf.tensor([/* token ids */]);
      const result = await model.executeAsync({input_ids: inputIds});
      const text = decodeTokens(result);
      document.getElementById('output').textContent = text;
    }

    document.getElementById('run').addEventListener('click', generate);
    loadModel();
  </script>
</body>
</html>

The code above works in a controlled sandbox, but already hints at two problems: (1) the model bundle is large enough to affect page load times, and (2) the tokenization step is dramatically oversimplified, leading to incorrect outputs.

Step 2: Measuring Performance on Real Devices

To quantify the impact, we instrument the page with the Performance API. The following snippet records model load time and inference latency on a typical 2022‑era laptop and a low‑end Android phone.

async function loadModel() {
  const t0 = performance.now();
  model = await tf.loadGraphModel('https://example.com/distilgpt2/model.json');
  const t1 = performance.now();
  console.log(`Model load: ${Math.round(t1 - t0)} ms`);
}

async function generate() {
  const t0 = performance.now();
  const result = await model.executeAsync({input_ids: inputIds});
  const t1 = performance.now();
  console.log(`Inference: ${Math.round(t1 - t0)} ms`);
  // …
}

On a desktop Chrome instance, the model loads in ~2 seconds, and a single inference takes ~800 ms. On a mid‑range Android Chrome, load time balloons to >5 seconds and inference exceeds 2 seconds—far beyond the acceptable latency for an interactive email editor. Users will experience jitter, UI freezes, and a perception that the web app is “slow”.

Step 3: Privacy – What You Think Is Private Isn’t

Even though the model runs locally, the surrounding JavaScript ecosystem can unintentionally leak content. For example, browsers automatically send crash reports, telemetry, or even console.log output to remote analytics services if developers enable them. A careless console.log(prompt) call can expose draft text to a third‑party endpoint.

// Bad practice – logs user‑generated text
console.log('User prompt:', prompt);

Moreover, the model weights themselves are copyrighted artifacts. Distributing them via a CDN can violate licensing terms, especially if the model is derived from proprietary data. Companies must perform a legal audit before serving any LLM asset to browsers.

Step 4: Battery Drain and Thermal Throttling

Running matrix multiplications on a device’s CPU (or even GPU via WebGL) consumes noticeable power. Repeated inference during typing (e.g., auto‑completion after each keystroke) can keep the device’s GPU busy, leading to rapid battery depletion and thermal throttling. The following experiment logs battery level before and after a 30‑second typing session.

let startBattery;
navigator.getBattery().then(b => { startBattery = b.level; });

async function autoComplete() {
  // Called after each keystroke
  await generate();
}

setTimeout(async () => {
  const endBattery = (await navigator.getBattery()).level;
  console.log(`Battery dropped ${((startBattery - endBattery) * 100).toFixed(1)}%`);
}, 30000);

On a typical smartphone, the test shows a 12 % battery drop after just half a minute of continuous generation—an unacceptable user experience for any productivity tool.

Step 5: Maintenance Overhead

Keeping the on‑device model up‑to‑date is non‑trivial. A new model release requires republishing a potentially multi‑hundred‑megabyte bundle, invalidating cached assets, and coordinating versioning across browsers. In contrast, a server‑side API can roll out improvements without forcing users to download new assets.

// Naïve cache‑busting strategy – forces full re‑download
const script = document.createElement('script');
script.src = `https://example.com/distilgpt2/model.json?ts=${Date.now()}`;
document.head.appendChild(script);

The above approach defeats the original goal of offline operation and adds bandwidth costs for every user session.

Security and Best Practices

If a team still decides to experiment with client‑side LLMs, the following safeguards are essential:

• Never log raw user input; strip or hash before any console output.
• Employ a Web Worker to isolate heavy computation from the main UI thread.
• Limit inference calls to explicit user actions (e.g., a “Generate” button) instead of per‑keystroke.
• Serve the model over HTTPS with Subresource Integrity (SRI) hashes to prevent tampering.
• Implement a feature‑flag that gracefully falls back to a server‑side API when device resources are insufficient.

“A model that never leaves the device is only as private as the code that runs it.”

Conclusion

While on‑device generative AI sounds like a privacy‑first, latency‑free solution for email drafting, the reality is a complex trade‑off: inflated bundle sizes, unpredictable performance on heterogeneous hardware, hidden telemetry leaks, battery drain, and a heavy maintenance burden. For most enterprise and consumer scenarios, a hybrid approach—lightweight client UI with server‑side inference and strict data handling policies—delivers a far better balance of speed