Skip to content

WebGPU Token-Classification Runtime

OpenMed's browser package exposes a typed token-classification session for local ONNX artifacts. It probes a real WebGPU adapter, selects the WebGPU execution provider when usable, and falls back to WebAssembly without changing the run(tokens) -> logits contract.

The runtime consumes either the root ONNX/WebGPU export or the ONNX files inside a Transformers.js bundle. It does not download models from a registry.

Install

npm install openmed onnxruntime-web

Serve ONNX Runtime Web's WASM files and the exported model from the same local application bundle. WebGPU requires HTTPS or localhost; threaded WASM also requires cross-origin isolation.

Load local model variants

An OpenMed ONNX export normally contains a float16 WebGPU graph and a float32 fallback graph:

/models/openmed-pii/
  model.webgpu.onnx
  model.onnx
/runtime/onnxruntime/
  ort-wasm-simd-threaded.wasm
  ...

Pass both local paths so fallback never tries to reinterpret a missing or remote artifact:

import { loadWebGpuTokenClassificationSession } from "openmed";

const session = await loadWebGpuTokenClassificationSession({
  modelPath: {
    webgpu: "/models/openmed-pii/model.webgpu.onnx",
    wasm: "/models/openmed-pii/model.onnx",
  },
  assetPath: "/runtime/onnxruntime/",
  labelCount: 17,
  modelName: "OpenMed/local-pii",
  deviceName: "browser",
  tier: "base",
  canonicalTier: "Base",
});

const logits = await session.run({
  inputIds: [101, 2023, 2003, 1037, 3231, 102],
  attentionMask: [1, 1, 1, 1, 1, 1],
  batchSize: 1,
  sequenceLength: 6,
});

console.log(logits.dims); // [1, 6, 17]
await session.dispose();

The input names default to input_ids, attention_mask, and optional token_type_ids. Override inputNames and outputName for an exported graph with different names. Inputs are validated and converted to ONNX int64 tensors; the returned logits are an owned Float32Array with shape [batch, sequence, labels].

Each typed session uses a private ORT cache by default, so dispose() can release its resources safely. Pass an explicit cache only to share an ORT session across wrappers; an individual wrapper will not release a caller-owned shared cache.

For a Transformers.js bundle, point both variants at its local onnx/ directory:

modelPath: {
  webgpu: "/models/openmed-pii/transformersjs/onnx/model.onnx",
  wasm: "/models/openmed-pii/transformersjs/onnx/model_quantized.onnx",
}

Graphs whose weights live in external-data shards pass them through the typed externalData option described in the ONNX Runtime Web loader guide. As with sessionOptions, the same shards are passed to the WebGPU graph and to the WASM fallback graph. Invalid shard locations, oversized shards, and sessionOptions.externalData are rejected before any session is created.

Capability and fallback behavior

probeOrtWebCapabilities() distinguishes API exposure from an adapter that can actually be requested. WebGpuTokenClassificationSession.capabilityProbe records the probe result, while backend reports one of webgpu, wasm-simd-threads, or wasm-basic.

Selection is deterministic:

  1. Use WebGPU only when navigator.gpu.requestAdapter() returns an adapter.
  2. Otherwise use SIMD plus threaded WASM when WebAssembly, SharedArrayBuffer, cross-origin isolation, and at least two logical cores are available.
  3. Otherwise use single-threaded WASM.

If WebGPU session creation fails after a successful probe, the loader retries once with the dedicated WASM model. Set fallbackOnWebGpuError: false only when the application must fail instead. Configure these response headers to enable threaded WASM:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

Content-Security-Policy

A synthetic WebAssembly fixture exercises a strict CSP (script-src 'self' 'wasm-unsafe-eval', connect-src 'self' blob:, worker-src 'self' blob:) plus COOP/COEP. See Content-Security-Policy profile for the full header set and what each directive is for. Removing 'wasm-unsafe-eval' makes WebAssembly compilation fail with a CompileError rather than falling back silently.

Direct WGSL classification head

ONNX Runtime Web's WebGPU provider already uses shader-backed matrix multiplication for a complete token-classification graph. For an encoder that exports hidden states separately, OpenMed also ships the audited web/runtime/kernels/classify.wgsl head and a small wrapper:

import { createWebGpuClassificationHead } from "openmed";

const adapter = await navigator.gpu.requestAdapter();
if (!adapter) throw new Error("WebGPU is unavailable");
const device = await adapter.requestDevice();

const head = await createWebGpuClassificationHead(device, {
  weights, // row-major [hiddenSize, labelCount]
  bias,    // [labelCount]
  hiddenSize: 768,
  labelCount: 17,
});

const logits = await head.run(hiddenStates, batchSize, sequenceLength);
head.dispose();

The kernel dispatches the sequence and label axes in 8-by-8 workgroups and the batch on the third dispatch axis. Weights and bias remain resident on the GPU; per-run hidden-state and logits buffers are destroyed after readback.

Parity and recall gates

certifyWebGpuReference() fails closed unless all of these conditions hold:

  • candidate and Python-reference shapes match;
  • every logit is finite and the maximum absolute delta is within the fixed tolerance;
  • decoded BIO/BILOU token spans match within the configured token-offset tolerance;
  • recall loss stays within maxRecallDelta; and
  • no label listed in criticalLabels is missed.
import { certifyWebGpuReference } from "openmed";

const evidence = certifyWebGpuReference({
  referenceLogits: pythonReference,
  candidateLogits: logits,
  id2label,
  attentionMask,
  tolerance: 1e-3,
  maxRecallDelta: 0,
  criticalLabels: ["PERSON", "EMAIL", "PHONE", "ID_NUM"],
});

Certification evidence contains tensor deltas, token offsets, labels, and aggregate recall only. It never copies note text or identifier surfaces.

Per-device benchmark record

Call benchmark() before unrelated inference when cold-start evidence is required. The session measures model load, first inference, warm p50/p95 and mean latency, and active tokens per second:

const report = await session.benchmark(tokens, {
  warmupIterations: 1,
  iterations: 10,
});

await saveLocally(JSON.stringify(report, null, 2));

The result uses the shared BenchmarkReport envelope:

{
  "suite": "webgpu-token-classification-runtime",
  "model_name": "OpenMed/local-pii",
  "device": "webgpu:browser",
  "fixture_count": 1,
  "generated_at": null,
  "metadata": {
    "runtime": "onnxruntime-web",
    "local_files_only": true,
    "tier": "base",
    "canonical_tier": "Base",
    "warmup_iterations": 1,
    "token_count": 6,
    "fallback_used": false
  },
  "metrics": {
    "devices": {
      "browser": {
        "backend": "webgpu",
        "precision": "float32",
        "batch_size": 1,
        "sequence_length": 6,
        "latency": {
          "cold_load_ms": 25.1,
          "first_inference_ms": 7.4,
          "cold_ms": 32.5,
          "warm_ms": 2.2,
          "p50_ms": 2.1,
          "p95_ms": 2.6,
          "count": 10
        },
        "throughput": { "tokens_per_second": 2727.3 }
      }
    }
  }
}

No benchmark is transmitted automatically. The optional benchmarkSink callback is local application code, and the built-in record omits model paths, tokens, note text, browser user agents, and adapter fingerprints.

Offline and privacy boundary

  • Model and WASM asset paths must be relative, root-relative, Windows-local, or file:// paths. HTTP(S), protocol-relative, data:, and blob: references are rejected before session construction.
  • A browser may make same-origin requests while constructing the session to read those application-local files. Once construction completes, run() performs no fetch and sends no inference data anywhere.
  • The package enables no telemetry, remote model fallback, console logging, or raw-PHI benchmark fields.
  • Model licensing remains the deployer's responsibility. OpenMed bundles no restricted model weights or clinical datasets. See Export Toolchains, Artifact Licenses and Redistribution for runtime library licenses and model-license propagation.

Validation

The committed fixture is synthetic. Node tests cover capability profiles, offline paths, typed tensors, fallback, benchmarking, parity, and recall. A Playwright Chromium gate dispatches the production WGSL kernel on a real headless WebGPU context and compares it with the Python float32 fixture.

cd js/openmedkit-web
npm ci
npm test

cd ../../tests/browser/brand
npm ci
npx playwright test webgpu-token-classification.spec.ts --project chromium

Browser execution lane for exported graphs

The WGSL gate above checks only the classification head. The ONNX Runtime Web lane in tests/browser/ort-lane loads complete exported graphs:

  1. python -m openmed.onnx.browser_lane prepare builds a tiny, randomly initialised BERT token classifier offline (seed 3940, no downloads), exports model.onnx (fp32, with its model.onnx.data sidecar) and model.webgpu.onnx (fp16) with the repository exporter, and derives a negative-control graph that appends Mod(logits, 65504, fmod=1), an identity operator the WebGPU provider does not implement. Python ONNX Runtime computes the fp32 reference logits for synthetic token feeds, and everything lands in output/playwright/ort-lane/fixture/.
  2. A local server serves the graphs, ONNX Runtime Web from the pinned onnxruntime-web package (never a CDN), and the built OpenMed runtime from one loopback origin. Pages that need threads are cross-origin isolated with COOP and COEP.
  3. Headless Chromium runs each case through loadWebGpuTokenClassificationSession() and compares the logits and decoded spans with the reference. Browser context routing aborts and counts any request to another origin; a negative-control test proves that a CDN fetch is blocked and counted.
Case Provider Graph Logit tolerance
wasm-threaded-fp32 threaded WASM, cross-origin isolated fp32 1e-4
wasm-single-fp32 single-thread WASM, not isolated fp32 1e-4
webgpu-fp32 WebGPU with WASM fallback fp32 1e-3
webgpu-fp16 WebGPU with WASM fallback fp16 5e-2
webgpu-fallback-control WebGPU with WASM fallback fp32 + Mod 1e-3

Every case also requires identical decoded spans (zero token-offset tolerance) and no recall loss. The synthetic feeds are redrawn until every active token's top logit leads the runner-up by more than twice the loosest tolerance, so a candidate inside its tolerance cannot flip a label.

The lane captures ONNX Runtime's session log inside the page and keeps only operator types and node counts. A node without a WebGPU kernel (webgpu kernel not found in registries) is fallback evidence: the case reports webgpu_support: "partial" with the operator types, otherwise "full". The negative control must report partial and name Mod. With onnxruntime-web 1.31.1 the exported BERT graphs report partial as well: their int64 shape subgraph, IsNaN and Max run on the WASM CPU provider while the matrix multiplications, attention softmax and layer norms run on WebGPU.

Each case writes output/playwright/ort-lane/records/<case>.json with the provider, browser name and version, ONNX Runtime Web version, SHA-256 digests of the graph, sidecar, feeds and reference, the declared tolerances, and the outcomes: status (passed, failed or unavailable), a controlled reason_code, the support level, fallback operator types, node placement counts, the maximum logit delta, span counts, recall, and request counts. Records never contain logits, token ids, text, URLs, paths or log lines.

A missing WebGPU adapter or a missing shader-f16 feature for the fp16 graph is recorded as unavailable with that reason, never skipped or passed. check gates the records and fails closed on any failed record and on any unavailable reason that was not explicitly accepted:

make ort-browser-lane
# or step by step
uv run --extra dev --extra onnx python -m openmed.onnx.browser_lane prepare
npm --prefix js/openmedkit-web ci && npm --prefix js/openmedkit-web run build
npm --prefix tests/browser/ort-lane ci
npm exec --prefix tests/browser/ort-lane -- playwright install chromium
npm --prefix tests/browser/ort-lane test
uv run python -m openmed.onnx.browser_lane check

check --allow-unavailable <reason> and OPENMED_ORT_LANE_ALLOW_UNAVAILABLE (comma-separated reason codes, for the Playwright run) accept named unavailable reasons; OPENMED_ORT_LANE_CHANNEL=chrome uses an installed Chrome instead of Playwright's Chromium. The Pages workflow runs the lane in the pinned Playwright container, where SwiftShader provides WebGPU without shader-f16, so it accepts only webgpu_shader_f16_unavailable and every other case must pass. prepare --checkpoint <dir> exports a local token-classification checkpoint instead of the synthetic one; it must still meet the margin and span gate on the synthetic feeds.

Training, the demo UI, OCR, and other multimodal workloads remain outside this runtime.