Post-Quantum TLS: Preventing Latency Regressions

Migrating to ML-KEM and ML-DSA means negotiating cryptographic keys six times larger than RSA. Here is how we prevent connection latency regressions at the ingress proxy.

## The Looming Edge Disaster Nobody Is Measuring Every security team knows about 'harvest now, decrypt later' (HNDL) attacks. Nation-states are hoarding encrypted transit data today, waiting for fault-tolerant quantum hardware to tear through elliptic curve cryptography (ECC) and RSA. NIST has finalised its post-quantum cryptography (PQC) standards, canonising ML-KEM (Kyber) for key encapsulation and ML-DSA (Dilithium) for digital signatures. Most engineering blogs stop there, treating PQC as a simple library swap. Upgrade your OpenSSL version, toggle a compiler flag, and your edge proxy is suddenly quantum-safe. In real-world ingress architectures, reality hits hard. The transition to post-quantum cryptography threatens to undo fifteen years of hard-fought TCP and TLS latency optimisations. The primary culprit is not CPU overhead—it is physical packet size. ## The Jumbo Handshake Problem Traditional elliptic-curve Diffie-Hellman key exchange (X25519) requires exchanging tiny keys: roughly 32 bytes for a public key. The entire ClientHello and ServerHello sequence fits neatly inside a single standard IP packet within the default 1500-byte Maximum Transmission Unit (MTU). Now look at the numbers for NIST's primary key encapsulation mechanism, ML-KEM-768: * **Public Key:** 1,184 bytes * **Ciphertext:** 1,088 bytes * **Signature (ML-DSA-44 or Falcon):** 2,420 to 3,293 bytes When combining these payloads to execute a hybrid TLS 1.3 handshake (X25519 + ML-KEM-768), your initial handshake frames balloon from a few hundred bytes to multiple kilobytes. ### Packet Fragmentation and Dropped Connections The fundamental constraint of the public internet is that most networks enforce an MTU of 1500 bytes (or 1420 bytes on certain mobile or tunnelled networks). When a TLS handshake message exceeds this threshold, the IP layer must fragment the packet. ```text Legacy (X25519): [ ClientHello (300 B) ] --> Fits in 1 IP Packet (No Fragmentation) Hybrid PQC (X25519 + ML-KEM-768): [ ClientHello Part 1 (1420 B) ] --> IP Frag 1 [ ClientHello Part 2 (800 B) ] --> IP Frag 2 (Multi-packet assembly required) ``` On clean enterprise fiber, fragmentation is an annoying efficiency loss. Across public consumer networks, mobile carrier boundaries, and lossy Wi-Fi, fragmented UDP (for HTTP/3 QUIC) or multi-packet TCP segments suffer disproportionately high packet drop rates. When Packet 2 of a fragmented ClientHello drops, the entire handshake stalls, triggering tail latency spikes up to three times the base Round-Trip Time (RTT). ## Engineering the PQC Ingress Pipeline To deploy quantum-resistant connections without tanking your core Web Vitals or mobile API performance, edge infrastructure must be carefully tuned. ### 1. Hybrid KEM Selection Do not jump straight to pure post-quantum algorithms. Standards bodies and security leaders recommend hybrid modes—pairing classical X25519 with ML-KEM-768. This ensures that if a hidden mathematical vulnerability is discovered in the lattice-based mathematics of ML-KEM, the connection remains as secure as classical cryptography. Ensure your edge ingress (Envoy, NGINX, or custom Rust-based proxies using `aws-lc-rs` or `rustls`) negotiates `X25519MLKEM768` as the priority cipher group. ### 2. Offloading Signature Verification via Intermediate Caching While key exchange happens on every handshake, certificate verification does not have to pay the full ML-DSA penalty on every cold start. A standard TLS certificate chain using ML-DSA signatures can easily push the ServerHello well past 7 KB. To survive this: * **Use Classical Signatures with PQC KEM:** Retain ECDSA certificates for server authentication while using ML-KEM for key encapsulation. This protects against HNDL attacks immediately because the transient session keys are protected by quantum-safe encryption, avoiding the need to deploy multi-kilobyte certificates. * **Enforce TLS 1.3 Session Tickets (Resumption):** For returning clients, zero-RTT resumption bypasses the full key encapsulation exchange entirely, reverting to pre-shared symmetric keys derived from previous quantum-safe sessions. ```yaml # Envoy snippet for modern hybrid PQC edge termination tls_certificates: - certificate_chain: filename: "/etc/envoy/certs/ecdsa-leaf.crt" private_key: filename: "/etc/envoy/certs/ecdsa-leaf.key" common_tls_context: tls_params: tls_minimum_protocol_version: TLSv1_3 ecdh_curves: - X25519MLKEM768 - X25519 ``` ### 3. Tuning Initial Congestion Window (initcwnd) On Linux ingress gateways, the default Initial Congestion Window (`initcwnd`) has historically been set to 10 MSS (Maximum Segment Size). When serving massive post-quantum certificate chains, 10 MSS is often exceeded before the connection finishes establishing. Inspect your edge kernel settings and raise `initcwnd` accordingly: * Check your ingress route: `ip route show` * Adjust the congestion window on edge nodes: `ip route change default via <GATEWAY> dev eth0 initcwnd 20` This ensures the edge server transmits the oversized ServerHello, Certificate, and CertificateVerify chain in a single burst without waiting for an intermediate ACK from high-latency edge clients. ## Real-World Verification: What to Monitor Once hybrid PQC is live at your edge, monitor these metrics closely: * **P99 Handshake Latency:** Track the time delta between TCP SYN arrival and TLS Finished receipt, stratified by client operating system and network type. * **Fragmented Frame Drop Rates:** Watch `netstat -s` or eBPF counters for `IP: ReasmFails` and `TCPTimeouts` specifically occurring during the first round-trip. * **Handshake Abandonment Rate:** Keep track of mobile clients that disconnect before sending their first HTTP request payload. Quantum safety is not just a cryptographic challenge; it is a distributed systems problem. If you simply update your libraries without restructuring your packet boundaries and tuning your edge network, you will buy security at the cost of crippling system responsiveness.

Surviving the Jumbo Handshake: Post-Quantum TLS at the Edge

What clients say

Surviving the Jumbo Handshake: Post-Quantum TLS at the Edge

What clients say

Related posts