Linux fundamentals explained

Kernel, userspace, and distributions

The Linux kernel is the core program that talks to hardware: CPU scheduling, memory paging, block and network I/O, device drivers, and security modules. Everything else — shells, editors, web servers, databases — runs in userspace and asks the kernel for services through system calls (read, write, fork, exec, socket, and hundreds more).

A distribution (Ubuntu, Debian, Fedora, Rocky Linux) packages the kernel with a curated userspace: GNU coreutils, a package manager, default services, and release cadence. For cloud servers, Ubuntu LTS and Debian stable dominate because documentation, security updates, and third-party repos target them first. Pick one distro per environment and standardize — mixing CentOS habits on Ubuntu wastes time on package name differences.

What you actually interact with

• Shell — bash or zsh interprets commands; pipes and redirects connect programs.
• Init system — systemd on virtually all modern server distros starts services at boot and supervises them.
• Package manager — apt on Debian/Ubuntu, dnf on Fedora/Rocky; installs signed binaries and tracks dependencies.
• Configuration files — plain text under /etc/; changes take effect after service reload or reboot depending on the daemon.

Filesystem hierarchy and paths

Linux follows the Filesystem Hierarchy Standard (FHS). Knowing where things live saves hours when logs are full or a service cannot find its config.

• / — root of the tree; everything hangs here.
• /home/ — per-user home directories (~ expands to yours).
• /etc/ — system and service configuration (nginx, sshd, systemd units).
• /var/ — variable data: logs (/var/log), databases, mail spools, web roots on some setups.
• /tmp/ — world-writable temp; cleared on reboot (use /var/tmp for longer-lived scratch).
• /usr/bin/ — user commands; /usr/sbin/ — admin binaries.
• /proc/ and /sys/ — virtual filesystems exposing kernel and process state (read-only for humans, invaluable for debugging).

Paths are case-sensitive. . is the current directory; .. is the parent. Symlinks (ln -s) are common — always ls -la when a path “does not exist” but something similar does.

Processes, signals, and resource limits

Every running program is a process with a PID. View them with ps aux, top, or htop. Parent processes spawn children via fork; daemons often double-fork to detach from the terminal. systemd replaces manual daemonization for services you manage.

Signals you must know

• SIGTERM (15) — polite shutdown request; services should flush state and exit.
• SIGKILL (9) — immediate kill; cannot be caught; use only when SIGTERM fails.
• SIGHUP (1) — historically “hang up”; many daemons reload config on SIGHUP.

Send signals with kill -TERM <pid> or systemctl stop <service> (which sends SIGTERM under the hood). Zombie processes (exited children not reaped) usually mean a buggy parent — restart the parent service rather than killing zombies individually.

cgroups and namespaces (exposed via systemd and containers) limit CPU, memory, and I/O per service. On a shared VPS, set MemoryMax= in unit files so one leaky Node process cannot OOM the entire box. Pair process awareness with graceful shutdown patterns in your applications.

Users, groups, and permissions

Linux is multi-user. The root user (UID 0) bypasses permission checks — use it sparingly. Daily work should run as an unprivileged user with sudo for elevated commands. Each file has an owner, a group, and permission bits for user/group/others: read (r), write (w), execute (x).

ls -l /var/www/app/index.html
-rw-r--r-- 1 www-data www-data 4096 Jun  8 12:00 index.html

Octal chmod 644 sets rw-r--r--. Directories need execute bit to be traversable (chmod 755 for web roots). chown user:group file changes ownership; umask masks default bits on new files.

Security habits

• Never run internet-facing daemons as root when a dedicated user exists (www-data, postgres).
• Restrict private keys: chmod 600 ~/.ssh/id_ed25519.
• Use groups for shared write access instead of world-writable directories (chmod 777 is a red flag).
• Enable secrets management for API keys — not world-readable files in git.

Package management with apt

On Debian/Ubuntu, apt installs signed packages from configured repositories. Typical workflow:

sudo apt update          # refresh package index
sudo apt upgrade -y      # install security patches
sudo apt install nginx   # add a package

apt search finds packages; apt show prints metadata; dpkg -l lists installed versions. Pin production servers to a release (e.g. Ubuntu 24.04 LTS) and automate unattended security upgrades for the -security pocket only until you test major version bumps in staging.

For language-specific deps, prefer official repos or vendor packages over random curl-to-bash scripts. When you need newer versions than the distro ships, use upstream PPAs or containers — not a mixed Frankenstein of manual /usr/local installs nobody can reproduce.

systemd: services, timers, and journals

systemd is PID 1 on modern Linux. It starts units: services (.service), mounts, sockets, and timers (.timer for cron-like jobs with journal integration).

sudo systemctl status nginx
sudo systemctl restart nginx
sudo systemctl enable nginx   # start on boot

Unit files live in /etc/systemd/system/ (overrides) or /lib/systemd/system/ (packages). After editing, run systemctl daemon-reload. Use ExecStart, Restart=on-failure, User=, and EnvironmentFile= for production services — not nohup in /etc/rc.local.

Logs go to the journal: journalctl -u nginx -n 100 --no-pager tails a unit; journalctl -f follows live. Persistent journals survive reboot when /var/log/journal exists. Forward critical logs to centralized storage per your observability plan — disks fill when nobody rotates logs.

Networking essentials

Every interface has an IP address. ip addr and ip route replace deprecated ifconfig. List listening ports with ss -tlnp (TCP listen, numeric, process). Outbound connectivity tests: curl -v https://example.com or nc -zv host 443.

Firewall and DNS

ufw (Ubuntu) or firewalld wraps nftables/iptables. Default-deny inbound except SSH (22), HTTP (80), and HTTPS (443) is a sane VPS baseline. DNS resolves names to IPs — see our DNS guide for records, TTL, and debugging dig / nslookup.

Place TLS termination on nginx or a load balancer; bind application servers to 127.0.0.1 when they do not need direct internet exposure.

SSH: remote access done right

SSH encrypts remote shells and file copies (scp, rsync, sftp). Authenticate with Ed25519 key pairs — disable password login in /etc/ssh/sshd_config (PasswordAuthentication no) once keys work.

• Copy your public key to ~/.ssh/authorized_keys on the server.
• Use ssh -i ~/.ssh/key user@host or ~/.ssh/config aliases.
• Install fail2ban or cloud firewall rules to throttle brute force.
• Forward agents cautiously (AllowAgentForwarding) — a compromised remote host can abuse your local keys.

For automation, dedicated deploy users with forced commands or CI OIDC beat shared root keys checked into pipelines.

Worked example: Harbor API on a fresh Ubuntu VPS

A team deploys a small Node API and static frontend on a single VPS (similar to how many indie products start before Kubernetes).

1. Provision — Ubuntu 24.04, create sudo user deploy, SSH key only, enable UFW (22/80/443).
2. Install stack — apt install nginx certbot python3-certbot-nginx; install Node via NodeSource or nvm under deploy.
3. App layout — code in /opt/harbor-api, owned by deploy; environment in /etc/harbor/api.env mode 600.
4. systemd unit — harbor-api.service with User=deploy, EnvironmentFile=, Restart=on-failure, binds 127.0.0.1:3000.
5. nginx site — reverse proxy to 127.0.0.1:3000, serve static files from /var/www/harbor, gzip on.
6. TLS — certbot --nginx -d api.harbor.example; verify auto-renew timer with systemctl list-timers.
7. Verify — curl -I https://api.harbor.example/healthz; check journalctl -u harbor-api on failures.

Result: one predictable server, reproducible from a runbook, ready to migrate into Docker or k8s later without relearning where logs and configs live.

Hosting approach decision table

Common pitfalls

• Running everything as root — one RCE owns the box; use service accounts.
• No firewall — databases and Redis bound to 0.0.0.0 get scanned within hours.
• Disk full on /var — unrotated journals or Docker layers; monitor free space and set log limits.
• Manual edits without backups — snapshot before changing sshd_config or nginx; keep a serial console path.
• curl | bash installers — opaque supply chain; prefer signed packages.
• Ignoring time sync — TLS and distributed logs break when NTP drifts; enable systemd-timesyncd or chrony.
• chmod 777 debugging — fixes symptoms, creates security debt; fix ownership instead.
• Skipping updates — unpatched OpenSSH and kernel CVEs are the lowest-hanging fruit for attackers.

Production checklist

• Non-root sudo user with SSH keys; password and root SSH login disabled.
• Automatic security updates or monthly patch window documented.
• UFW or cloud firewall: default deny inbound; only required ports open.
• Services run as dedicated users with systemd restart policies.
• Application binds to localhost unless it must face the internet directly.
• TLS certificates with auto-renewal verified (certbot renew --dry-run).
• Disk, CPU, and memory alerts; journal size capped or forwarded off-box.
• Time synchronization enabled and monitored.
• Runbook: how to restart each service, where configs and logs live.
• Tested backup/restore for /etc, app data, and database dumps.

Key takeaways

• Linux separates kernel responsibilities from userspace tools you operate daily.
• FHS paths (/etc, /var, /home) are the map for configs, logs, and data.
• Permissions and users contain blast radius — root is for emergencies, not routine deploys.
• systemd + journalctl are how modern servers start services and tell you what went wrong.
• SSH keys, firewalls, and patches are non-negotiable baselines before any application code ships.

Related reading

• nginx fundamentals explained — reverse proxy and TLS termination on Linux
• Docker fundamentals explained — containers sharing the host Linux kernel
• DNS explained — name resolution from browser to your VPS IP
• TLS and HTTPS explained — certificates, handshakes, and termination patterns