Linux System Administration: A Practical Guide from Kernel to Production

The Linux kernel manages CPU scheduling, virtual memory, devices, networking, and filesystem access. Applications normally execute in user space and request privileged operations through system calls such as openat(), read(), and fork(). Linux uses a monolithic kernel architecture with loadable modules: major subsystems share kernel address space, while many drivers can be loaded as needed. A distribution adds libraries, utilities, package management, and service configuration around that kernel. POSIX defines portable operating-system interfaces and utility behavior; it is not another name for Linux, and Linux-specific features extend beyond its scope.

These boundaries explain everyday diagnostic evidence. A program reporting permission denied may have reached the kernel successfully but failed an access check. A slow file read may involve filesystem code, page cache behavior, and device latency rather than application computation. Start an inventory with uname -r for the kernel release, cat /etc/os-release for distribution identity, and lsblk for block devices. Use free -h to inspect memory, paying attention to available memory rather than assuming all cache is wasted. Kernel logs, available through journalctl -k on journald systems, can reveal device errors and out-of-memory events, subject to access permissions.

Linux process management starts with the distinction between a program and a running process. Each process has a process ID, credentials, open file descriptors, an environment, and a virtual address space. A common creation pattern is fork followed by exec: the first creates a child, while the second replaces its program image. Threads share many process resources but are independently scheduled. A terminated child remains a zombie until its parent collects its exit status; killing the zombie itself cannot solve that condition. Administrators also need to distinguish runnable tasks from sleeping tasks and tasks waiting uninterruptibly, often for I/O.

For a worked investigation, run ps -eo pid,ppid,user,stat,%cpu,%mem,comm --sort=-%cpu to identify busy processes, then inspect a chosen process with ps -p 1234 -o pid,ppid,etime,stat,args, replacing 1234 appropriately. The /proc/1234 directory exposes additional kernel-maintained information, subject to permissions. If a managed application needs stopping, prefer its service manager so shutdown follows the configured lifecycle. For an unmanaged process, kill -TERM 1234 requests graceful termination; kill -KILL 1234 prevents application cleanup and should be a last resort. Load average includes runnable and uninterruptible tasks, so high load does not prove CPU saturation. Combine process evidence with memory and I/O observations.

Linux file permissions combine ownership with read, write, and execute bits for the owner, group, and others. Their meaning changes for directories: read permits listing names, execute permits traversal, and write generally permits changing directory entries when combined with traversal. Deleting a file therefore depends primarily on its parent directory’s permissions, not whether the file itself is writable. Access control lists can grant more specific permissions, while SELinux or AppArmor can impose additional restrictions beyond traditional ownership checks. Effective troubleshooting must consider all applicable layers rather than repeatedly broadening mode bits.

Suppose a team needs a shared workspace. After creating a project group and adding authorized users, run sudo install -d -o root -g project -m 2770 /srv/project. The leading 2 sets the directory’s set-group-ID bit, making new entries inherit its group. It does not guarantee group-write permission: application creation modes and the user’s umask still matter. Where filesystem ACL support and the relevant tools are available, sudo setfacl -m d:u::rwx,d:g::rwx,d:m::rwx,d:o::--- /srv/project establishes default ACL entries, although requested creation modes still constrain resulting permissions. Verify with id and getfacl /srv/project. Group membership changes generally require a new login session. Grant administrative access through narrowly scoped sudo rules, validated with visudo.

Linux storage management involves several distinct layers: a physical or virtual disk, optional partitions, optional software RAID or logical volumes, a filesystem, and a mount point. Logical Volume Manager separates logical-volume allocation from underlying device boundaries, making some capacity changes easier. RAID can improve availability for particular device failures, but it does not replace backups against deletion, corruption, or ransomware. Filesystem behavior also matters: ext4 and XFS have different maintenance and resizing capabilities. Never assume that a block device can be shrunk merely because a logical volume or filesystem can grow.

Consider diagnosing a full application volume. Run findmnt /srv/app to identify its backing filesystem, df -h /srv/app for block usage, and df -i /srv/app for inode usage. Then use sudo du -xhd1 /srv/app to locate large directories without crossing into other filesystems. If df reports much more usage than du, inspect open deleted files with sudo lsof +L1 when lsof is installed; their blocks remain allocated until the final open reference closes. Persistent mounts belong in /etc/fstab, commonly identified by filesystem UUID. After editing, validate with findmnt --verify and test the intended mount in a maintenance window. Formatting commands destroy existing filesystem structures, so verify device identity and backups before executing them.

On many enterprise distributions, systemd acts as the system service manager and coordinates boot through units and dependencies. A service unit describes how to start an application, which identity it uses, and what should happen after failure. Enabling a service arranges activation through its installation relationships; starting it runs it now. These are separate actions, though systemctl enable --now combines them. Dependency ordering also has limits: After=network.target does not guarantee usable connectivity. Applications should tolerate transient network failures, and services that genuinely need configured networking may require network-online.target plus an appropriate wait-online implementation.

For a foreground application, a minimal custom unit can specify Type=simple, User=appsvc, WorkingDirectory=/opt/app, ExecStart=/opt/app/bin/server, and Restart=on-failure under [Service]. The account and executable must already exist, and the account needs access to required files. Place the unit in /etc/systemd/system/app.service, include [Install] with WantedBy=multi-user.target if boot activation is desired, then run sudo systemctl daemon-reload and sudo systemctl enable --now app.service. Inspect systemctl status app.service and journalctl -u app.service -b. Add hardening such as NoNewPrivileges=true and filesystem restrictions incrementally, testing application behavior. Automatic restarts improve recovery from some failures but do not fix defective software or replace health monitoring.

Linux networking troubleshooting works best by separating name resolution, addressing, routing, transport, and application behavior. Start with ip address and ip route to establish local configuration. Use getent hosts example.com to test hostname lookup through the host’s configured name-service mechanisms. Inspect listening TCP sockets with sudo ss -ltnp, and test an HTTP endpoint with curl -v, using the actual service URL. A connection refusal often indicates no listener or an active rejection; a timeout can indicate filtering, routing problems, or an unresponsive destination. Ping alone cannot establish application health, and failed ping does not prove the host is unavailable.

For example, if an application answers curl http://127.0.0.1:8080 locally but not remotely, inspect its bind address before changing firewall rules. A listener on 127.0.0.1 is intentionally loopback-only. Then check host firewall policy, upstream controls, and routing. For SSH security, prefer individual accounts and protected keys, restrict privileged access, and disable unused authentication methods only after confirming an alternative works. OpenSSH configuration may involve included files and Match blocks, so inspect effective settings and validate syntax with sudo sshd -t. Keep an existing session open while testing a second login before closing access. Use the distribution-supported firewall management approach rather than mixing competing tools.

Reliable administration uses a change loop: observe the current state, formulate a hypothesis, make a limited change, verify the result, and record what happened. Linux troubleshooting becomes faster when evidence is timestamped and correlated across application logs, journal entries, kernel messages, and resource measurements. Establish normal workload behavior before choosing alert thresholds. Monitor user-visible outcomes alongside capacity indicators: a running process is not proof that requests succeed. Keep configuration under version control where appropriate, excluding secrets, and use automation that converges toward a declared state rather than accumulating undocumented manual edits.

Routine maintenance should include supported package updates, planned reboots when needed, restore exercises, and review of privileged access. Package managers resolve dependencies, but safe upgrades still require release-note review, representative testing, and a rollback or recovery plan. Back up both data and the configuration needed to use it; databases may need coordinated snapshots or application-aware backup tools for consistency. A practical exercise is to restore an application onto a fresh virtual machine, recreate its account and service unit, restore data, and verify requests from another host. Record recovery time and missing dependencies. That test demonstrates operational competence more convincingly than merely showing that a backup job completed.