Operating System Internals

Operating System Internals

The operating system is the software layer between hardware and application programs. It manages hardware resources (CPU, memory, storage, devices), provides abstractions that make them easier to use, and ensures that multiple programs can run safely and fairly.

Process Management

A process is a running instance of a program. It has:

• Address space: virtual memory the process can use (code, data, heap, stack)
• File descriptors: handles to open files and sockets
• Process ID (PID): a unique integer identifier
• State: running, runnable (ready to run), blocked (waiting for I/O or a lock), zombie (finished but parent hasn't read exit status)

Process lifecycle: fork() creates a child process that is a copy of the parent; exec() replaces the current process's image with a new program; wait() lets the parent collect the child's exit status.

A thread shares the process's address space but has its own stack and registers. Threads within a process communicate through shared memory. Threads are faster to create and context-switch than processes (no address space switch). Downside: a bug in one thread (buffer overflow, use-after-free) can corrupt data for all threads.

Context switching: the CPU can only execute one thread per core at a time. The OS scheduler regularly switches which thread runs. A context switch saves the current thread's registers/stack pointer and restores the next thread's. Context switches are expensive (~microseconds) — this is why too many threads hurts performance.

Scheduling

The scheduler decides which runnable thread gets the CPU. Key objectives:

• Throughput: maximize work completed per time unit
• Latency: minimize time from task arrival to completion
• Fairness: every thread gets CPU time proportional to its priority
• CPU utilization: keep the CPU busy (don't leave it idle while work is waiting)

Round-robin: each thread runs for a time slice (quantum), then yields to the next. Simple and fair but doesn't differentiate priorities.

Priority scheduling: higher-priority threads preempt lower-priority ones. Risk: priority inversion — a low-priority thread holds a lock that a high-priority thread needs; the low-priority thread can't run because of preemption; deadlock.

CFS (Completely Fair Scheduler): Linux's scheduler. Tracks how much CPU time each thread has consumed and runs the thread with the least. Uses a red-black tree for efficient selection.

I/O-bound vs CPU-bound: I/O-bound processes (web servers, databases) spend most time waiting for I/O. CPU-bound processes (compression, ML training) use CPU continuously. Schedulers handle these differently.

Memory Management

Programs use virtual addresses. The OS maps virtual addresses to physical memory frames through the page table. This enables:

• Isolation: each process sees a private address space; process A cannot read process B's memory
• More memory than RAM: pages not recently used can be swapped to disk; loaded back on page fault (at the cost of a disk read — 100,000x slower than RAM)
• Memory-mapped files: map file contents directly to virtual address space; efficient I/O for large files (databases, mmap'd files)

Heap vs Stack:

• Stack: fixed-size, LIFO, automatic allocation/deallocation. Local variables, function call frames. Stack overflow from deep recursion.
• Heap: dynamic, explicit allocation (malloc/free, new/delete, or GC). Objects created at runtime. Memory leaks from allocation without deallocation.

Garbage Collection (Go, Java, Python, JavaScript): automatically frees heap objects with no live references. Eliminates memory leaks and use-after-free but introduces GC pauses and overhead. Real-time systems (game engines, HFT) often avoid GC for predictability.

Memory-mapped I/O: mmap() maps a file (or device) into the virtual address space. Reads become memory accesses (served by the page cache if available; from disk if not). Databases (PostgreSQL, SQLite) use mmap for efficient buffer pool management.

File Systems

A file system organizes data on storage devices:

• Directory tree: hierarchical namespace mapping names to inodes
• Inode: metadata about a file (size, permissions, timestamps, block pointers) without the name; a directory entry maps names to inode numbers
• Blocks: fixed-size units of storage on disk; a file's data is stored in blocks, referenced by the inode

File system types: ext4 (Linux standard), NTFS (Windows), APFS (macOS), ZFS (integrity + snapshots), btrfs. Each makes different tradeoffs on performance, integrity, and features.

Journaling: ext3/ext4, NTFS write a journal before committing changes to the file system. If the system crashes mid-operation, the journal enables recovery without a full fsck. Without journaling, crashes can corrupt file system structures.

File descriptors and I/O modes:

• Buffered I/O: library (stdio) buffers reads/writes; actual syscalls happen in large chunks. More efficient for sequential access.
• Direct I/O: bypasses the kernel page cache; goes straight to disk. Used by databases that manage their own caching.
• Memory-mapped I/O: treat file as memory array; kernel handles caching.
• Async I/O / io_uring (Linux): submit I/O operations that execute without blocking the calling thread.

Signals and Inter-Process Communication

Signals: asynchronous notifications sent to processes. SIGTERM (terminate gracefully), SIGKILL (terminate immediately, uncatchable), SIGHUP (reload config), SIGSEGV (segmentation fault — accessed invalid memory).

IPC mechanisms:

• Pipes: unidirectional byte stream between processes (ls | grep foo — the | is a pipe)
• Unix domain sockets: bidirectional, local-machine communication faster than TCP sockets
• Shared memory: processes map the same physical memory; fastest IPC but requires explicit synchronization
• Message queues: OS-managed FIFO queues for message passing between processes