How Daemons Work

Synthesis. The full lifecycle from "user runs a CLI command" to "daemon is running and serving requests" to "daemon shuts down cleanly when no longer needed."

The lifecycle in pictures

   First CLI invocation
        │
        ▼
   ┌──────────────────────────────────────────────────────┐
   │  Probe socket — daemon up?                           │
   │  ├─ yes → connect, send request, exit                │
   │  └─ no  → continue                                   │
   └────────────────────┬─────────────────────────────────┘
                        │
                        ▼
   ┌──────────────────────────────────────────────────────┐
   │  Read PID file — process alive?                      │
   │  ├─ yes (but socket missing) → race / crash recovery │
   │  └─ no  → fork new daemon                            │
   └────────────────────┬─────────────────────────────────┘
                        │
                        ▼
   ┌──────────────────────────────────────────────────────┐
   │  Spawn daemon (re-exec self with daemon subcommand)  │
   │  ├─ detach from controlling terminal (setsid)        │
   │  ├─ chdir("/") to release the parent's cwd           │
   │  ├─ close stdin/stdout/stderr; reopen to /dev/null   │
   │  ├─ atomic-write PID file                            │
   │  ├─ flock the PID file (single-instance enforcement) │
   │  ├─ bind Unix socket (or named pipe on Windows)      │
   │  ├─ start tracing-subscriber → logs to file          │
   │  ├─ trap SIGTERM / SIGINT → graceful shutdown        │
   │  └─ enter accept loop                                │
   └────────────────────┬─────────────────────────────────┘
                        │
                        ▼
   ┌──────────────────────────────────────────────────────┐
   │  Client side: poll for socket to appear (~100ms × N) │
   │  ├─ found → connect                                  │
   │  └─ timeout (2s) → error: daemon failed to start     │
   └────────────────────┬─────────────────────────────────┘
                        │
                        ▼
                 Send request, get response, exit

Subsequent CLI invocations (microseconds later, hours later, the next day): probe → connect → request → response → exit. The daemon is just there.

What the daemon is doing while idle

Even when no client is connected, the daemon may be:

• Running background tasks: sync loops (mxr's IMAP / Gmail polling), watchpoint timers (lazydap's snooze countdown), file watchers (config reload).
• Holding open connections: IMAP IDLE, DAP adapter sessions, persistent HTTP connections to APIs.
• Caching expensive state: parsed config, decoded auth tokens, prepared database statements, source files.
• Listening on the IPC socket for new clients.

Idle daemons consume RAM (often the biggest cost) and minimal CPU (a few epoll/kqueue syscalls per second). The trade is "always-warm responsiveness" against "always-resident process."

Graceful shutdown

The full ritual when the daemon receives SIGTERM (or mxr shutdown):

1. Stop accepting new IPC connections (close listener)
2. Cancel new background tasks (set shutdown flag, watch::Sender::send(true))
3. Wait for in-flight work to complete or hit deadline:
   - Finish active IPC requests (each has a timeout)
   - Drain pending writes to durable storage (SQLite/TOML)
   - Disconnect from external services (close DAP adapter with 1s timeout)
4. Close children: for each spawned subprocess (codelldb, smtp connection),
   send SIGTERM, wait briefly, send SIGKILL if needed
5. Remove PID file
6. Exit with status 0

The deadline matters. Init systems wait ~30 seconds after SIGTERM, then send SIGKILL. Your shutdown handler must complete within that window or get killed mid-flight, leaving stale state.

Crash recovery

The daemon will crash. Patterns that survive:

• Durable state lives outside the daemon. SQLite for mxr; TOML for lazydap. The daemon caches it in RAM but the disk has the truth.
• Live session state is ephemeral by design. When the daemon dies, in-flight DAP sessions die. The next launch is a fresh session. Persistence is for what should outlive sessions (breakpoints, watches, drafts), not for the session itself.
• PID files locked via flock. A crashed daemon's lock is released by the kernel; the next attempt can lock and start.
• Clients auto-respawn. If the socket vanishes, the client's next probe forks a new daemon. The user sees a millisecond of latency, not a "daemon dead" error.

Per-instance scoping

A single binary may run multiple daemon instances. mxr and lazydap both use this:

$LAZYDAP_INSTANCE=project-a → /tmp/lazydap/instance-a.sock + instance-a.pid
$LAZYDAP_INSTANCE=project-b → /tmp/lazydap/instance-b.sock + instance-b.pid

The instance key (often hashed from project root) ensures different projects get different daemons. Two cd into different repos = two daemons.

This is how lazydap supports multiple debug sessions concurrently across projects without sharing state.

Logging

Daemons can't write to stdout — they're detached. Options:

• File logs in a known location ({data_dir}/daemon.log). Rotated by the daemon (max size + max files) or by an external logrotate.
• syslog via libc::syslog or tracing-syslog — system-wide log aggregation. Less common on user-level daemons.
• Structured (JSON) logs for machine consumption. Both mxr and lazydap use tracing with a JSON layer when running in background mode, plain-text when foreground.

Log levels via env: RUST_LOG=info,lazydap=debug.

What a daemon CAN'T do

• Be reliably restarted on system boot without an init manager. systemd / launchd handles this; auto-spawning daemons rely on the user invoking the CLI.
• Run before the user logs in. Per-user daemons run inside the user's session.
• Hold privileged resources (port 80, raw sockets) without setuid or capabilities.
• Survive a kernel panic. State on disk does; the daemon doesn't.

How mxr and lazydap embody this

Both:

• Auto-spawn on first command via re-exec of the binary's daemon --foreground subcommand
• PID file at {data_dir}/daemon.pid, atomic write + flock
• Unix socket at {runtime_dir}/{instance}.sock, mode 0700
• tracing with JSON to file when daemonised, text to stderr when --foreground
• SIGTERM/SIGINT graceful shutdown with deadlines
• kill_on_drop(true) on all spawned children
• Clients probe + retry; on socket missing, fork fresh daemon
• Optional idle-shutdown (off by default)

The daemons are nearly identical at the orchestration level. The differences are entirely in what they do — mxr syncs email, lazydap drives DAP adapters. The daemon machinery is interchangeable.

See also (atomic notes)

• Daemons — the concept
• PID Files — process tracking
• Signal Handling — graceful shutdown
• The Local Daemon Pattern — auto-spawning specifically
• Headless Core + Multiple Clients — the architectural shape that makes daemons useful
• Unix Domain Sockets — what daemons typically listen on
• Mxr · Lazydap — concrete implementations

Other syntheses (sibling MOCs in the vault)

The other "How X actually works" synthesis notes — each is the entry point to its own cluster:

• How Debuggers Actually Work — DAP / adapters / ptrace / DWARF / breakpoints
• How Email Actually Works — SMTP / IMAP / MIME / threading / OAuth / internal model
• How Processes Talk to Each Other — IPC, Unix sockets, framing, JSON-RPC (this is what daemons typically talk over)
• The Elm Architecture (TEA) — Model / Update / View / Cmd / Reducers
• Client-Agnostic Cores — headless core + many clients, the broader architectural pattern daemons enable

Further reading

• man 7 daemon (systemd) — modern daemon conventions
• W. Richard Stevens, Advanced Programming in the UNIX Environment — Chapter 13. The classic reference.
• man 7 sysvipc, man 7 unix — IPC mechanisms daemons use