All posts
Mobile Apps
3 min read• 11/22/2026

Zero-Latency On-Device SLM Inference in Swift and Kotlin

Cloud roundtrips for mobile AI features are killing your user retention. Here is how we embed quantized, on-device small language models directly into production mobile apps.

Share X LinkedIn

Tip: use ← / → to browse posts.

Zero-Latency On-Device SLM Inference in Swift and Kotlin
## The Latency Tax of Cloud-Dependent Mobile AI Most teams shipping mobile "AI experiences" are still chaining user interactions to centralized API endpoints. A user taps an action, an HTTP request traverses cellular infrastructure, queues behind an LLM gateway, waits for token generation, and streams back over an unstable radio connection. The end-to-end latency rarely dips below 1,400 milliseconds. For real-time productivity apps, camera utilities, and on-the-fly text summarization, 1.4 seconds is an eternity. It destroys the illusion of software as a physical instrument. Worse, it exposes your application to offline failure states, data residency headaches, and runaway API token costs. Running quantized Small Language Models (SLMs)—specifically sub-3B parameter architectures like SmolLM2, Llama 3.2 1B/3B, and Gemma 2 2B—directly on Apple's Neural Engine (ANE) and Qualcomm/MediaTek NPUs is now entirely viable. Here is the operational blueprint we use at Better Call Hashim to deliver sub-80ms time-to-first-token directly on hardware. ## Hardware Acceleration: CoreML vs. ExecuTorch Cross-platform abstraction wrappers are tempting, but on mobile silicon, you cannot afford generic runtimes. You need hardware-specific compilation. ### iOS: Apple Silicon and the Unified Memory Advantage On iOS, CoreML remains undisputed. Apple’s unified memory architecture allows the GPU and the Neural Engine (ANE) to share weight buffers without redundant memory copies. However, naive conversions from Hugging Face checkpoints to CoreML frequently fall back to CPU execution because unsupported tensor operations break the ANE graph. Key constraints when converting weights: - Keep quantization schemes strictly to 4-bit or mixed 4/8-bit palletization (`linear_symmetric`). - Ensure KV-cache operations utilize stateful tensor APIs introduced in CoreML 7+ rather than passing explicit cache arrays back and forth through the compute graph. - Restrict dynamic context window lengths. Pre-allocate fixed context buckets (e.g., 256, 512, 1024 tokens) to avoid dynamic graph re-compilation. ### Android: ExecuTorch and the Heterogeneous NPU Problem Android fragmentation makes uniform execution difficult. Google’s ExecuTorch runtime paired with the Qualcomm AI Engine Direct (QNN) backend provides the cleanest production pathway. Rather than compiling an arbitrary ONNX runtime build, compile dedicated `.pte` (PyTorch Edge) binaries targeting specific chipset families (Snapdragon 8 Gen 2/3/4 vs. MediaTek Dimensity). ```swift import CoreML import Tokenizers final class LocalInferenceEngine { private let model: MLModel private let tokenizer: Tokenizer init(modelURL: URL, tokenizerURL: URL) throws { let config = MLModelConfiguration() config.computeUnits = .cpuAndNeuralEngine // Block GPU to preserve battery self.model = try MLModel(contentsOf: modelURL, configuration: config) self.tokenizer = try AutoTokenizer.from(modelFolder: tokenizerURL) } func generateStreaming(prompt: String) -> AsyncThrowingStream<String, Error> { AsyncThrowingStream { continuation in Task.detached(priority: .userInitiated) { let inputTokens = self.tokenizer.encode(text: prompt) var currentState = try self.model.makeState() for token in inputTokens { let input = LocalSLMInput(input_ids: token, state: currentState) let output = try self.model.prediction(from: input) let nextToken = output.featureValue(for: "next_token")!.int64Value let decoded = self.tokenizer.decode(tokens: [nextToken]) continuation.yield(decoded) currentState = output.featureValue(for: "state")!.stateValue! } continuation.finish() } } } } ``` ## Practical Optimization Tactics ### 1. Thermal and Battery Throttling Running inference sustained over 30 seconds spikes device temperature, prompting the OS scheduler to aggressively throttle compute clocks. For on-device intelligence, structure your features around micro-bursts: - Limit generative completions to under 128 tokens. - Prefer classification, structured JSON extraction, and contextual ranking over lengthy generative prose. - Disallow continuous background inference. Always run on background QoS priority or decouple jobs using the native background tasks framework. ### 2. Binary Footprint and Dynamic Delivery Shipping a 1.8GB model binary inside your base App Store package destroys download conversions. Never bundle the model directly in the IPA or APK. - Ship the application shell with the tokenizer and base fallback heuristics (under 50MB). - Download quantized model weights on first application boot or as an optional feature download using Apple On-Demand Resources (ODR) or Google Play Feature Delivery. - Store weights in the app's `Application Support` directory, marking them with `.isExcludedFromBackupKey = true` to avoid consuming iCloud/Google Drive backup quotas. ## When to Avoid On-Device Models Do not force edge execution if your feature demands massive world knowledge, real-time web retrieval, or strict factual alignment across thousands of edge cases. In our architecture reviews, we enforce a strict dividing line: UI-blocking interactions, edge-filtering, sensory transformations, and privacy-sensitive local context belong on the phone. Reasoning-heavy synthesis, multi-step agent graphs, and long-tail domain intelligence belong on your private inference clusters. Design for hybrid elasticity or fail at the user experience layer.
ios
android
on-device-ai
swift
coreml
Share X LinkedIn

What clients say

Real reviews from founders and teams we've shipped with.

5.0 · 6 reviews
"Grok integration for real-time X data was the differentiator our product needed. Fast to prototype, faster to ship."
xAI Integration
Grok · Real-time X data
"A newsroom platform that actually scales. Editors love the workflow Hashim built for us."
Marco B.
Editor-in-Chief, TradeView News
"Supabase + TanStack Start setup was production-ready on day one. Auth, RLS, everything."
Marcus D.
CTO, Ledgerly
"Postgres schema and indexes were rebuilt properly. Queries that took seconds now take milliseconds."
Hana Y.
DBA, Datacore
"Stable Diffusion and Stable Audio deployed on our own GPUs — private, fast, and tuned to our brand."
Stability AI Deploy
Stable Diffusion · Stable Audio
"SEO went from invisible to page-one in three months. Technical, content, and schema — all dialed in."
Karan V.
CMO, Brightscale