Client-Side AI Embeddings: Running Transformers.js and ONNX Directly in the Browser
Eliminate server vector costs and guarantee 100% user privacy. Learn how to run quantized embedding models inside Web Workers using Transformers.js, WebAssembly, and WebGPU.
The Cloud API Bottleneck: Privacy, Latency, and Compounding Costs
For the past three years, vector search and semantic retrieval architectures followed a single pattern: whenever a user typed text or searched their notes, the client app sent the string over HTTP to an embedding API (like OpenAI text-embedding-3-small). The server returned a 1536-dimensional float vector, which was stored in a remote vector database.
While functional, this model has three fatal flaws for client-heavy productivity software: first, user data leaves the device, creating severe compliance hurdles for healthcare, legal, and enterprise customers. Second, every keystroke incurs API costs that scale linearly with user activity. Third, offline usage is impossible.
With the maturation of WebAssembly SIMD and WebGPU in modern browsers, we can now run quantized ONNX embedding models directly on the user's laptop or smartphone CPU/GPU using Transformers.js.
In this guide, we architect a high-speed, client-side semantic search engine running 100% in the browser with zero cloud dependencies.
Selecting the Optimal Browser-Grade Embedding Model
Not all embedding models can run in a browser. Standard 768-dimensional models consume 500MB of memory, causing mobile browser tabs to crash.
We recommend three lightweight quantized models:
1. Xenova/all-MiniLM-L6-v2 (q8): 23MB download size, 384-dimensional output. The gold standard for fast in-browser English semantic similarity.
2. Xenova/bge-small-en-v1.5: 33MB download size, outstanding retrieval precision for technical documentation.
3. Xenova/paraphrase-multilingual-MiniLM-L12-v2: 70MB download size, supporting 50+ languages with cross-lingual search capabilities.
Offloading Inference to a Dedicated Web Worker
Running ONNX neural network inference on the browser's main thread blocks UI rendering, freezing button clicks and dropping frame rates. Always execute embedding generation inside an isolated Web Worker.
import { pipeline, env } from "@xenova/transformers";
// Configure cache storage in IndexedDB
env.allowLocalModels = false;
env.useBrowserCache = true;
class EmbeddingPipeline {
static instance: any = null;
static async getInstance(progressCallback?: (p: any) => void) {
if (this.instance === null) {
this.instance = await pipeline(
"feature-extraction",
"Xenova/all-MiniLM-L6-v2",
{ progress_callback: progressCallback }
);
}
return this.instance;
}
}
self.onmessage = async (e: MessageEvent) => {
const { id, text } = e.data;
const extractor = await EmbeddingPipeline.getInstance((progress) => {
self.postMessage({ type: "progress", progress });
});
// Generate 384-dimensional normalized vector
const output = await extractor(text, { pooling: "mean", normalize: true });
const vector = Array.from(output.data);
self.postMessage({ id, type: "result", vector });
};Cosine Similarity Search in Pure TypeScript
Once candidate documents and user queries are embedded into normalized float vectors, calculating their semantic match is a simple dot product: cosine_similarity = dot(A, B) since vectors are unit normalized.
Modern JavaScript engines (V8) optimize typed array loops with SIMD vectorization, computing 50,000 vector similarity checks in under 5 milliseconds.
export function cosineSimilarity(a: Float32Array, b: Float32Array): number {
let dotProduct = 0;
for (let i = 0; i < a.length; i++) {
dotProduct += a[i] * b[i];
}
return dotProduct;
}
export function searchLocalIndex(
queryVector: Float32Array,
documents: { id: string; text: string; vector: Float32Array }[],
topK = 5
) {
const scored = documents.map((doc) => ({
id: doc.id,
text: doc.text,
score: cosineSimilarity(queryVector, doc.vector),
}));
// Sort descending by score
return scored.sort((a, b) => b.score - a.score).slice(0, topK);
}Caching ONNX Model Weights in CacheStorage API
A 23MB neural network model must never be downloaded repeatedly on every page load. Transformers.js automatically caches model weights inside the browser's CacheStorage API.
Subsequent page visits load the model weights instantly from local disk in under 100ms, making your semantic search completely functional even in airplane mode.
To provide a responsive user interface during initial hydration, track download progress using pipeline callbacks and render a progress bar. Once downloaded, cache headers prevent redundant roundtrips.
export async function preloadEmbeddingModel(
onProgress?: (percent: number) => void
): Promise<boolean> {
try {
const { pipeline, env } = await import("@xenova/transformers");
env.useBrowserCache = true;
await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2", {
progress_callback: (info: any) => {
if (info.status === "progress" && onProgress) {
const percent = Math.round((info.loaded / info.total) * 100);
onProgress(percent);
}
},
});
return true;
} catch (error) {
console.error("Failed to preload ONNX model cache:", error);
return false;
}
}WebGPU Acceleration: 10x Throughput on Modern Laptops
When available, Transformers.js can bind to WebGPU (navigator.gpu) rather than WASM CPU fallbacks. WebGPU executes parallel tensor operations across hundreds of GPU shader cores, accelerating multi-document batch embedding throughput by over 1,000%.
WebGPU utilizes 16-bit half-precision floats (fp16), cutting tensor memory bandwidth in half while doubling computational throughput. For browsers lacking WebGPU, fallback gracefully to multi-threaded WebAssembly SIMD by assigning worker thread counts to match navigator.hardwareConcurrency.
- WebGPU Execution: ~12ms per 256-token chunk on modern Apple Silicon and RTX GPUs.
- WASM SIMD (Multi-threaded): ~45ms per chunk using 4 parallel Web Workers.
- WASM Fallback (Single-threaded): ~140ms per chunk for legacy mobile browsers.
Scaling to 50,000+ Vectors: Approximate Nearest Neighbor (ANN) in IndexedDB
While brute-force cosine similarity checks run in less than 5ms for collections under 3,000 documents, linear scans degrade when indexing larger knowledge bases like full email archives or book libraries.
For enterprise-scale client datasets (10,000 to 100,000 items), integrate an in-memory Hierarchical Navigable Small World (HNSW) graph or Vantage Point Tree (VP-tree) serialized directly into IndexedDB.
HNSW constructs multi-layered skip-graphs where top layers navigate wide vector spaces and bottom layers converge on exact neighbors, achieving logarithmic O(log N) search complexity in pure JavaScript.
How does local embedding accuracy compare to OpenAI text-embedding-3-small?
Models like all-MiniLM-L6-v2 achieve over 86% of OpenAI's retrieval score on standard MTEB benchmarks, which is more than sufficient for in-app search, notes, and personal file retrieval.
What happens if a user is on an older phone without WebGPU?
Transformers.js gracefully falls back to WebAssembly (WASM) with SIMD vector instructions, which runs smoothly on virtually all modern smartphones.
Can client-side embeddings be stored in IndexedDB?
Yes! Store the Float32Array directly as binary data in IndexedDB alongside text metadata for instant offline persistence.
How should long documents be chunked in the browser?
Chunk text by paragraph boundaries with a target of 200 to 400 characters and a 50-character sliding overlap to preserve contextual continuity.
Does running AI models in the browser consume noticeable battery power?
Only during active batch indexing. For single search queries, inference takes less than 20ms of CPU time, resulting in imperceptible battery consumption.