Replace Managed Kubernetes with Bare-Metal MicroVMs

Managed Kubernetes clusters add massive hypervisor latency and infrastructure cost to real-time AI workloads. Here is our blueprint for migrating to bare-metal MicroVMs.

## The Kubernetes Tax on Heterogeneous Compute Kubernetes was engineered for long-lived, stateless HTTP microservices running on generic multi-tenant virtual machines. It was never designed to schedule high-density, millisecond-scale machine learning inferences or orchestrate fractional GPU allocations across heterogeneous bare metal. Yet enterprise engineering teams continue to spin up managed EKS, GKE, or AKS clusters as their default instinct. When applied to modern generative workloads, this architecture breaks down rapidly: * **Hypervisor Layering**: Running container engines inside virtualization layers on top of managed hyperscaler hosts creates layers of nested I/O indirection. Network throughput degrades, and direct memory access (DMA) transfers between host storage and GPU buffers face CPU interrupt bottlenecks. * **Sluggish Pod Provisioning**: Even with optimized node pre-warming, cold starts for 15GB model containers regularly exceed 45 to 90 seconds. In fluctuating traffic scenarios, you are forced to over-provision static GPU nodes, burning tens of thousands of dollars monthly in idle compute. * **Control Plane Latency**: The Kubernetes reconciliation loop (`etcd` writes, kube-scheduler passes, CNI network setup, CSI volume mounting) is fundamentally too slow when tasks finish in less than 200 milliseconds. At BetterCallHashim, we have eliminated managed Kubernetes for our core inference and high-density computing workloads, replacing it with bare-metal hardware running lightweight MicroVM orchestrators based on Firecracker and custom Rust daemons. ## The Architecture: MicroVM Swarms on Pure Bare Metal Instead of managing complex node pools, we lease unvirtualized bare-metal servers equipped with PCIe Gen 5 buses and direct NVMe arrays. A custom, lightweight daemon written in Rust executes directly on the Linux host kernel, provisioning and destroying ephemeral MicroVM instances in under 8 milliseconds. ```rust // Provisioning an ephemeral MicroVM worker with direct PCIe pass-through pub async fn spawn_inference_microvm(vm_id: &str, model_weights_offset: u64) -> Result<VmHandle> { let mut config = VmConfig::default(); config.vcpus = 4; config.memory_mb = 8192; config.enable_direct_io = true; let vm = FirecrackerRuntime::build() .with_kernel("/boot/vmlinux-6.10-edge") .with_rootfs("/images/rootfs-inference.ext4") .with_pcie_passthrough("0000:01:00.0") // Direct GPU mapping .spawn(vm_id) .await?; vm.mount_shared_memory(model_weights_offset)?; Ok(vm) } ``` ### Key Architectural Pillars * **Shared Read-Only Memory Mapping**: Instead of copying model weights into individual container overlay filesystems, weights reside in host page cache memory. Hundreds of ephemeral MicroVMs mount this memory slice simultaneously via `virtio-fs` with zero duplicate allocations. * **Kernel-Bypassing Network Pipelines**: We replace overlay networking meshes (like Calico or Cilium) with eBPF-driven XDP (eXpress Data Path) routing. Packets are parsed at the network interface card (NIC) layer and dropped straight into the target MicroVM's virtual queue, reducing p99 network latency from 4.2ms to 180 microseconds. * **Single-Tenant Security Isolation**: Unlike standard container runtimes (runc/containerd), which share the host Linux kernel and rely on cgroups and namespaces, MicroVMs utilize hardware-level virtualization (KVM). A compromised tenant cannot escape into the host environment. ## Eliminating Idle GPU Overhead GPU time is the most expensive line item in modern infrastructure engineering. Managed Kubernetes setups typically handle multi-tenancy on GPUs via NVIDIA Multi-Process Service (MPS) or Multi-Instance GPU (MIG). Both require rigid configuration profiles that cannot dynamically adapt to wild traffic swings on the fly. Under our bare-metal MicroVM orchestrator, we use custom PCIe device mediation: 1. A request arrives via a raw WebSocket edge connection. 2. The orchestrator checks GPU core occupancy via direct MMIO register reads. 3. A microVM initializes in ~8ms, locks the needed tensor cores, streams the input tensors, runs inference, returns the output, and immediately terminates. 4. The physical GPU core allocation is reclaimed instantly for the next waiting queue, driving average physical hardware utilization above 88%. ## Cost and Operational Comparisons We executed a side-by-side migration for a client running an automated document vectorization and processing pipeline handling 40 million operations per day. | Metric | Managed EKS Cluster | Bare-Metal MicroVM Platform | |---|---|---| | **Monthly Compute Cost** | $42,300 (G5 Instances + EKS fees) | $11,800 (Leased Bare Metal) | | **Cold Start Latency** | 38 seconds (Pod init + pull) | 9 milliseconds (MicroVM init) | | **P99 Inference Latency** | 412 milliseconds | 64 milliseconds | | **Engineering Hours/Mo** | ~24 hours (Cluster upgrades, CSI bugs) | < 2 hours (Host OS updates) | ## The Trade-Offs Dropping Kubernetes requires your team to take ownership of operational primitives you previously outsourced: * You must build or maintain your own health checking and rolling deployment mechanisms. * Bare-metal hardware failures (e.g., bad DIMMs, dead NVMe drives) cannot be solved by simply clicking "replace node pool"; you need automated server cordon-and-drain scripts. If your organization runs standard enterprise line-of-business applications with modest traffic, stay on managed clusters. But if your compute budget is dominated by GPU workloads and latency-sensitive microservices, abstracting your infrastructure through Kubernetes is an expensive mistake you can no longer afford.

Replacing Managed K8s with Bare-Metal MicroVM Orchestration

What clients say

Replacing Managed K8s with Bare-Metal MicroVM Orchestration

What clients say

Related posts