Inter-Process Communication (IPC)

Definition: Mechanisms provided by the OS allowing independent processes, each with its own isolated address space, to exchange data and synchronize state.

How It Works

Pipes and named pipes (FIFOs): unidirectional byte streams between processes. Anonymous pipes only work between related processes (e.g., parent/child from fork()); named pipes exist as filesystem entries, letting unrelated processes connect by opening the same path.
Shared memory: a memory region mapped directly into the virtual address space of multiple processes (via mmap/shmget), so reads and writes avoid any kernel copy. The fastest IPC mechanism, but it requires explicit synchronization, a Mutex and Semaphore or similar, since the kernel provides no built-in coordination over that memory.
Message queues and Unix domain sockets: structured, discrete message passing managed by the kernel, preserving message boundaries (unlike a raw byte-stream pipe) and often supporting priority ordering.
Signals: an asynchronous, minimal-payload notification mechanism (SIGKILL, SIGTERM, SIGCHLD) that interrupts a target process to notify it of an event, without transferring any structured data beyond the signal number.
Sockets, including loopback TCP/UDP on 127.0.0.1, extend IPC across machine boundaries, letting the same programming model scale from same-host processes to networked services. See Sockets and Socket Programming.

Under the Hood

A pipe is backed by a kernel-managed circular buffer, commonly 64KB by default on Linux (tunable via fcntl(F_SETPIPE_SZ)), with no filesystem storage involved even for named pipes; the FIFO’s directory entry is just a rendezvous point two processes use to get file descriptors pointing at the same kernel buffer. write() on a pipe copies bytes from the writer’s userspace buffer into this kernel buffer, and read() copies out from it into the reader’s, two userspace-kernel copies total, unlike shared memory’s zero-copy path.

Shared memory works because the same physical page frames get mapped into each process’s page table, at possibly different virtual addresses. There’s no kernel involvement in the actual read/write, the CPU’s MMU resolves each access directly to the shared physical page, exactly as it would for any other memory access. That’s what makes it the fastest IPC mechanism and also why it provides zero synchronization on its own: the kernel isn’t in the loop to serialize anything.

Unix domain sockets (AF_UNIX) behave like TCP sockets, SOCK_STREAM for a reliable ordered byte stream, SOCK_DGRAM for message-oriented, but never leave the kernel’s socket buffers to touch a real network stack, no IP headers, no checksums, no loopback interface traversal. This makes them meaningfully faster than 127.0.0.1 TCP for local IPC, which is why databases (PostgreSQL, MySQL) and container runtimes (Docker’s /var/run/docker.sock) default to Unix domain sockets for local control-plane communication.

Unix domain sockets have one more capability none of the other mechanisms here share: passing file descriptors between unrelated processes, via SCM_RIGHTS ancillary messages with sendmsg()/recvmsg(). This is how a privileged process can open a socket or file and hand the already-open descriptor to a lower-privilege sandboxed process, without the receiver ever needing permission to open the resource itself, a pattern used heavily by browser sandbox architectures and systemd socket activation.

Debugging Workflow

Diagnosing a stuck or misbehaving IPC path starts with figuring out which side is actually blocked and on what:

$ ipcs -a                          # list System V shared memory, semaphores, message queues
$ ls -la /dev/shm/                 # POSIX shared memory segments (backed by tmpfs)
$ lsof -p <pid> | grep -E "PIPE|unix|REG"   # what a specific process has open

For a pipe that looks stuck, one process not making progress reading or writing, strace on both ends shows exactly where each is blocked:

$ strace -p <writer_pid>
write(1, "...", 65536)             # hangs here: reader isn't draining, buffer is full

$ strace -p <reader_pid>
read(0, ...)                        # hangs here: nothing has been written yet

Seeing both processes blocked in complementary pipe operations, one in write() waiting for buffer space, the other in read() waiting for data that never comes because the writer is stuck, is the signature of the two-processes-writing-to-full-pipes deadlock described above. ipcs -a run periodically over time is also the standard way to catch leaked System V resources, growing counts of orphaned segments with no owning process left is a leak, not expected steady-state usage.

Why It Matters

Enables modular microservice architectures, multi-process application designs (browsers isolating tabs into separate processes), and composable Unix command-line tool chains.
Choosing the right IPC mechanism is a real performance decision: shared memory can move gigabytes per second with near-zero copy overhead, while a message queue or socket incurs kernel copy and context-switch cost per message. The tradeoff is convenience and safety versus raw throughput.
Container and sandboxing technology relies heavily on controlling which IPC mechanisms a process can use, for example disabling shared memory or restricting Unix domain socket access, as part of the isolation boundary.

Common Pitfalls

Shared memory without synchronization causes race conditions identical to multi-threaded shared-state bugs, except now across process boundaries, where debugging tools have less visibility into what the other side is doing.
Unbuffered or small-buffer pipes cause writer blocking once the OS pipe buffer fills up if the reader isn’t consuming fast enough. This can deadlock two processes that both write to full pipes while waiting to read from each other.
Leaking IPC resources, unlinked shared memory segments, orphaned named pipes, unreleased System V semaphores, that outlive the process, since the OS doesn’t always automatically garbage-collect these the way it reclaims ordinary process memory.
Assuming message ordering is preserved across independent IPC channels. Messages sent on separate pipes or queues can be received out of relative order at the consumer even if each individual channel preserves its own internal order.
Forgetting that fork() duplicates file descriptors, including pipe ends, into the child. A parent that doesn’t close its copy of a pipe’s write end after forking can leave the read end waiting for EOF forever, since the kernel still sees an open writer.

History

Pipes and signals date to the earliest Unix (1970s), reflecting the “small tools connected together” philosophy, | in the shell is a direct user-facing expression of pipe-based IPC.
System V IPC (message queues, semaphores, shared memory, via ipcget/shmget-family calls) was standardized in the 1980s and is still present today, though its identifier-based API (integer keys rather than filesystem paths) is widely considered clunkier than what came after.
POSIX IPC (POSIX message queues, POSIX shared memory via shm_open, POSIX semaphores) arrived later specifically to give these same capabilities a more filesystem-like, consistent API, and is generally preferred in new code over System V IPC today.

FAQ

Is a socket on 127.0.0.1 slower than a Unix domain socket for local IPC? Yes, generally. A loopback TCP socket still goes through the full network stack, IP routing, checksums, and TCP’s flow-control machinery, even though the data never leaves the machine. A Unix domain socket skips all of that.

Do I need IPC between threads of the same process? No. Threads already share the same address space, so ordinary shared variables plus a Mutex and Semaphore are sufficient; IPC mechanisms exist specifically for the case where address spaces are separate.

What happens to IPC resources if a process crashes? It depends on the mechanism. Pipes and sockets are cleaned up automatically by the kernel once the owning process’s file descriptors close. Named pipes, System V shared memory segments, and semaphores can outlive a crashed process and need explicit cleanup, one reason POSIX/System V IPC leaks are a common bug in less careful server code.

Comparison

Mechanism	Speed	Structure	Cross-machine capable	Needs manual sync
Pipe / FIFO	Medium	Byte stream	No	No (kernel serializes)
Shared Memory	Fastest	Raw bytes	No	Yes
Message Queue	Medium	Discrete messages	No	No
Unix Domain Socket	Fast	Stream or datagram	No	No
TCP/UDP Socket	Slowest (of these)	Stream or datagram	Yes	No
Signal	Fast (tiny payload)	None (just a number)	No	No

Kernel Copies Per Transfer

Mechanism	Userspace-kernel copies
Shared Memory	0
Pipe / FIFO	2 (write into kernel buffer, read out of it)
Unix Domain Socket	2 (same pattern as a pipe)
TCP/UDP Socket (loopback)	2, plus full network stack processing on each

Choosing a Mechanism

Need maximum throughput between processes on the same machine and are willing to write your own synchronization: shared memory.
Need simple, ordered byte-stream communication between a parent and its children: an anonymous pipe.
Need unrelated local processes to talk, with message framing preserved: Unix domain sockets or named pipes.
Need the same code to work whether the other end is on the same machine or across a network: TCP/UDP sockets, accepting the extra overhead for that flexibility.
Need a lightweight, low-latency “something happened” notification with no payload: signals.

Example

cat access.log | grep 404 passes the stdout of cat to the stdin of grep via an OS pipe:

process cat:  writes bytes -> [kernel pipe buffer, 64KB] -> process grep: reads bytes

If grep processes slower than cat produces output, the pipe buffer fills and cat’s write() call blocks until grep drains it, the kernel handles this backpressure automatically without either process needing custom flow-control logic. ipcs on Linux lists active System V shared memory segments, message queues, and semaphores system-wide, useful for spotting leaked IPC resources from crashed processes.

Referenced by

Dig deeper

Spot an error in this note?