Performance Engineering

Performance Engineering

The Performance Mindset

Measure first, optimize second. Profile the actual system — the biggest wins come from a small number of hotspots (80/20 rule: 80% of time in 20% of code).

Latency vs throughput: a single-threaded server has low latency (no queuing) but poor throughput. High concurrency increases throughput but can increase P99 latency under load.

Latency Analysis

Always use percentiles, not averages: 99 requests at 10ms + 1 request at 10s = average ~110ms. That average describes neither the typical case nor the worst case.

Track: P50 (typical), P95 (starting to see outliers), P99 (experienced by significant fraction at scale), P999 (0.1%).

Tail latency causes: GC pauses (stop-the-world), lock contention (serialized waits), OS scheduling jitter, CPU power management (C-state transitions), cold caches after deployment.

CPU Profiling

Sampling profiler: every N ms, capture running thread stacks. Builds histogram of where time is spent. Tools: Linux perf, Go pprof, Node.js --prof, Async Profiler (JVM).

Flame graphs (Brendan Gregg): call stacks visualized as rectangles; width = % of time. Widest plateaus = hotspots.

On-CPU: where threads are running (CPU-bound hotspots). Off-CPU: where threads are blocked on I/O or locks.

CPU performance counters (perf stat): cache miss rate > 10% is a red flag.

Memory Performance

Cache-friendly access: CPU prefetches 64-byte cache lines. Sequential array traversal = fast (prefetcher works). Linked list pointer chasing = slow (each node is a cache miss).

Struct of Arrays vs Array of Structs: SoA stores field x contiguously for all objects — scanning x values is cache-friendly. AoS stores whole objects contiguously — scanning x loads unused fields.

Allocation overhead: malloc in hot paths causes system calls and lock contention. Solutions: object pools, arena allocators, slab allocators.

I/O Performance

Sequential I/O (log writes) is fast on all media. Random I/O (database index scans) is slow on HDD (~5ms seek), tolerable on SSD (~100μs).

fsync flushes to durable storage (1-10ms on SSD). Databases call fsync on WAL writes for durability.

io_uring (Linux 5.1+): shared ring buffers between user and kernel space. I/O operations submitted in batches without per-I/O syscalls. Major IOPS improvement for high-throughput storage workloads.

Amdahl's Law in Practice

A web service: 80ms database (parallelizable), 15ms computation (CPU-bound), 5ms serialization (serial). Even infinite parallelism on the database and computation yields max speedup of 100/5 = 20x. The serial bottleneck is the ceiling.

Benchmarking mistakes: coordinated omission (client waits for response before sending next — hides queuing delay), forgetting warmup (cold cache), resource-limited test client (client saturates before server).