Shell Filesystem Semantics: Hard Links, Symlinks, Mount Namespaces, fsync Discipline & Atomic-Rename Guarantees

Why Filesystem Semantics Matter: Five Bugs You Don’t See Until Production

Your script does:

echo "$content" > /etc/myapp/config.json
systemctl reload myapp

It works. It works thousands of times. Then one night the box loses power between line 1 and line 2, you reboot, and /etc/myapp/config.json is empty — not the old config, not the new config, empty. Your service crashes on startup, and you spend the night learning that > is not atomic, that the kernel buffered the write, that fsync exists, and that “the file is on disk” is much harder than it looks.

This lesson is the five rules of filesystem semantics that every script which writes to disk needs to know:

By the end of this lesson, you’ll have a lib/fs.sh with safe_write, safe_rename, safe_symlink, and is_same_filesystem — primitives that survive the failure modes most shell scripts trip on.

Ground Concepts: Inodes, Dentries, and the (filesystem, inode) Pair

Before we talk about hard links and atomicity, let’s anchor the model:

        ┌─────────────────────────────────────────────┐
        │                Filesystem                   │
        │                                             │
        │  ┌───────────────┐     ┌─────────────────┐  │
        │  │   Directory    │     │     Inode       │  │
        │  │   entries      │     │   (metadata)    │  │
        │  │               │     │                 │  │
        │  │  /etc/passwd  ├────►│ #5012           │  │
        │  │  /etc/shadow  ├────►│ #5013           │  │
        │  │  /etc/group   ├────►│ #5014           │  │
        │  │  /etc/aliases ├──┐  │                 │  │
        │  └───────────────┘  │  │  permissions    │  │
        │                     └─►│  owner/group    │  │
        │                        │  size, mtime    │  │
        │                        │  link count: 2  │  │
        │                        │  data blocks    │  │
        │                        └─────────────────┘  │
        └─────────────────────────────────────────────┘

A file is identified by the pair (filesystem, inode). The path you type (/etc/passwd) is just a directory entry (dentry) that points to an inode. Multiple dentries can point to the same inode — that’s a hard link. The file’s metadata (owner, mode, size, mtime, link count, data block list) lives in the inode, not the dentry.

Three operations and what they do to this picture:

• cp src dst — reads inode A’s data, allocates inode B, writes the data to B, creates a dentry dst pointing to B. New inode, new data.
• ln src dst — creates a dentry dst pointing to inode A. No new inode. Link count of A increments to 2. This is a hard link. src and dst are equally “the file” — there’s no concept of “the original.”
• ln -s src dst — creates a dentry dst pointing to a new tiny inode (the symlink), whose data is the string “src”. Following the symlink is a string-resolution operation done at every access. Dangling symlinks point to nothing valid.

Why this matters for shell scripts:

# Tale of two backups.
ln file.txt copy.txt        # hard link: 1 inode, 2 dentries.  rm file.txt → copy.txt still valid.
ln -s file.txt link.txt     # symlink: 2 inodes (1 file + 1 link). rm file.txt → link.txt is dangling.

stat reveals it:

$ stat -c '%i %h %s %F' file.txt copy.txt link.txt
5012 2 100 regular file
5012 2 100 regular file
5018 1   8 symbolic link

%i is the inode number; %h is the hard-link count; %s is the size. The hard link shares everything. The symlink is its own file whose contents are 8 bytes (the path string file.txt).

Hard Links: Two Names, One Inode

Hard links are powerful and surprising. Every file you create is technically already a “hard link” — the filename in the directory is the first link to the inode. ln src dst adds a second link.

What hard links can do that copies can’t

# Atomic content swap by manipulating directory entries (no copy).
ln /etc/myapp/config.json.new /etc/myapp/config.json.tmp
mv  /etc/myapp/config.json.tmp /etc/myapp/config.json   # atomic on same FS

# Snapshot-style backups (rsync --link-dest): unchanged files share data blocks.
rsync -aH --link-dest=/backup/yesterday /data/ /backup/today/
# /backup/yesterday/file and /backup/today/file are TWO dentries pointing to ONE inode.
# Storage cost: ~size of changed files only.

The 7 hard-link rules every script must respect

1. Hard links cannot cross filesystems. Try it: ln /tmp/a /mnt/usb/a → EXDEV: Invalid cross-device link. Inodes are per-filesystem; you cannot reference an inode on another filesystem from a directory on this one. Most cross-mount errors in shell scripts are this.

2. Hard links to directories are forbidden (on Linux/most Unix; macOS HFS+ allows them but it’s a footgun — directory cycles break find). Symlinks for dirs only.

3. rm decrements the link count. The inode is freed when count hits 0 and no process has it open. Hence:

   exec 3>>/var/log/myapp.log     # process holds inode open via fd 3
   rm /var/log/myapp.log           # link count → 0, but inode persists for fd 3 holders
   # df shows space NOT freed until fd 3 is closed (process exits or fd closes)
   

   This is why du and df disagree after deleting open log files. Use lsof +L1 to find unlinked-but-open inodes.

4. mv of a hard link doesn’t break the link. Both dentries still point to the inode; mv src newname just renames one dentry. The other dentry (if any) is unaffected.

5. Editors break hard links via “save-as-new.” vim, emacs, and many config-management tools default to write a new file and rename over (the safe-write pattern). The new file is a new inode. Your hard-linked twin still points at the old, unchanged inode. Use :set backupcopy=yes in vim to preserve hard links.

6. Permissions live on the inode, not the dentry. Change perms via any link and all links see the new perms — there’s only one inode.

7. stat -c %h tells you how many links exist. find / -inum NUMBER finds them all (slow; scans the whole filesystem).

Recipe: hard-link snapshot backup

# Backup that costs ~delta size, not full size, by hard-linking unchanged files.
backup_dir() {
  local src="$1" dest_root="$2"
  local today yesterday
  today="$dest_root/$(date -u +%F)"
  yesterday=$(find "$dest_root" -maxdepth 1 -type d -name '20*' | sort | tail -n1)

  if [[ -n "$yesterday" && -d "$yesterday" ]]; then
    rsync -aH --delete --link-dest="$yesterday" "$src/" "$today/"
  else
    rsync -a "$src/" "$today/"
  fi
}
backup_dir /var/lib/myapp /backup

du -sh /backup/2025-01-08 /backup/2025-01-09 shows each “full backup” — but du -sh /backup shows the actual disk used (much smaller). Hard links make this storage trick possible without filesystem-level snapshots.

Symlinks: Strings That Resolve at Access Time

A symlink’s contents are the path it points to. Resolution happens at every access (open, stat, readdir of a directory containing it), not when the symlink is created.

The -L vs -P distinction (the biggest stat footgun)

Most stat-like tools have two modes: follow the symlink (-L, “logical”) or operate on the symlink itself (-P, “physical”). Shell scripts that don’t choose explicitly inherit the default — and the default differs across tools:

This produces classic bugs:

# Bug: report says "missing" but the link is there, just dangling.
[[ -e /etc/myapp/config.json ]] || echo missing
# This returns "missing" if config.json is a SYMLINK pointing to a deleted file.
# Fix: explicit choice.
[[ -L /etc/myapp/config.json || -e /etc/myapp/config.json ]] || echo missing

# Bug: "size" of a 4 GB file shows as 27 bytes because we statted the symlink.
size=$(stat -c %s /etc/myapp/data.bin)
# Fix:
size=$(stat -L -c %s /etc/myapp/data.bin)   # follow
# or, if you really want symlink size:
size=$(stat -c %s /etc/myapp/data.bin)

# Bug: backup misses files in symlinked dirs.
find /etc -type f -name '*.conf'    # default -P; doesn't descend symlinked dirs
# Fix:
find -L /etc -type f -name '*.conf'  # logical; follows symlinks
# Caveat: -L can loop on cyclic symlinks; find detects but it's slow.

ln -sf vs ln -sfn — the directory-link footgun

# You think this updates the symlink:
ln -sf /opt/app/v2 /opt/app/current

# What actually happens if /opt/app/current is a SYMLINK pointing to /opt/app/v1:
# - ln -sf "follows" the existing link
# - It treats /opt/app/current as the directory /opt/app/v1
# - It creates /opt/app/v1/v2 as a symlink (NOT what you wanted)

# The fix: -n means "don't dereference an existing symlink-to-dir; treat it as a file"
ln -sfn /opt/app/v2 /opt/app/current

Always ln -sfn when updating a symlink that may already exist and may point at a directory. This is the canonical “blue-green deploy” pattern: current is a symlink, releases live in releases/v1/, releases/v2/, and you flip with ln -sfn.

Atomic deploy via symlink swap

deploy_symlink_swap() {
  local release="$1"   # e.g. /opt/app/releases/v2
  local current=/opt/app/current
  local tmp_link=/opt/app/current.new

  [[ -d "$release" ]] || { echo "release missing: $release"; return 1; }

  ln -sfn "$release" "$tmp_link"   # create new symlink at sibling path
  mv -T  "$tmp_link" "$current"     # rename: atomic on same FS
  # Now /opt/app/current points at $release.
}

mv -T (GNU; --no-target-directory) ensures mv replaces the link rather than moving the new link into the dir. On BSD/macOS, use mv -fn carefully or stage in the same parent and rename. The key property: a reader doing open("/opt/app/current/binary", O_RDONLY) either sees v1’s binary (resolved before the swap) or v2’s (resolved after) — never a half-state.

Atomic Renames: When mv Is Atomic and When It Is Not

The atomic-rename pattern is the foundation of every “safe write” in Unix:

tmp=$(mktemp /etc/myapp/config.json.XXXXXX)
printf '%s' "$content" >"$tmp"
mv "$tmp" /etc/myapp/config.json

This works because rename(2) (the syscall behind mv for same-FS targets) is atomic: from any observer’s perspective, the destination is either the old file or the new file, never both, never partial.

But there are three caveats most scripts miss:

Caveat 1: rename(2) is atomic only on the same filesystem

mktemp /tmp/foo.XXXXXX           # /tmp may be tmpfs, separate FS from /etc
mv /tmp/foo.XXXXXX /etc/myapp/config.json
# Under the hood: mv calls rename(); rename returns EXDEV (cross-device);
# mv FALLS BACK TO copy + unlink, which is NOT atomic.

If /tmp is tmpfs (RAM) and /etc is on disk, mv will copy then unlink. A crash mid-copy leaves the destination half-written. The atomic-rename guarantee is gone, silently.

The fix: always create the temp file in the same directory as the destination.

dest=/etc/myapp/config.json
tmp=$(mktemp "${dest}.XXXXXX")    # same directory → same FS → rename(2) is atomic
printf '%s' "$content" >"$tmp"
chmod 644 "$tmp"
mv "$tmp" "$dest"

mktemp defaults to /tmp if you don’t give it a path template. Never use bare mktemp for files you’ll move into specific directories.

Detect cross-FS dynamically:

is_same_filesystem() {
  local a="$1" b="$2"
  local fa fb
  fa=$(stat -c %d "$a" 2>/dev/null || stat -f %d "$a")
  fb=$(stat -c %d "$b" 2>/dev/null || stat -f %d "$b")
  [[ "$fa" == "$fb" ]]
}

if ! is_same_filesystem "$tmp" "$(dirname "$dest")"; then
  echo "warning: cross-FS rename will not be atomic" >&2
fi

Caveat 2: rename(2) is atomic for the directory entry, not the data

Atomicity means “from a directory-listing perspective, dest is old or new, never partial.” It does not mean “the new file’s data is safely on disk.” After rename, the dentry points to the new inode, but the inode’s data blocks may still be in the kernel’s writeback cache. If the power dies before the kernel flushes, you can have:

• dest exists, points to a new inode
• The new inode has zeros or partial data because the writeback never happened
• Reboot: dest is corrupt

The fix: fsync the data file before the rename, and fsync the directory after.

# Bash doesn't have fsync directly; use python or sync.
fsync_file() {
  python3 -c "import os, sys; fd=os.open(sys.argv[1], os.O_RDONLY); os.fsync(fd)" "$1"
}

tmp=$(mktemp "${dest}.XXXXXX")
printf '%s' "$content" >"$tmp"
fsync_file "$tmp"          # ensure data is on disk
mv "$tmp" "$dest"          # atomic rename (same FS)
fsync_file "$(dirname "$dest")"   # ensure dentry change is on disk

sync (the command) flushes all dirty data on the system, which is a heavy hammer. Per-file fsync is cheaper. The trade-off: fsync adds latency (often 5–50 ms on rotational disks, <1 ms on NVMe). For configs and credentials, that latency is worth it.

fdatasync is a related call that flushes data but skips inode metadata that doesn’t affect data (like atime). For data-only durability it’s faster; for “the file is the right size and has the right perms,” use fsync.

Caveat 3: Some tools work around rename atomicity unsafely

cp -f on most platforms unlinks the target then writes. Window of badness: a reader between unlink and the new file’s first byte sees ENOENT. install -m is the same — it copies bytes then sets perms; if the script crashes between, you have a perms-wrong file. rsync --inplace actively rewrites the destination in place, breaking atomicity. truncate followed by writes is never atomic.

Use mv of a sibling temp file unless you have a specific reason otherwise.

fsync, fdatasync, sync: What Actually Goes to Disk

The Linux/Unix write path:

        write(fd, buf, n)                fsync(fd)              power off
            │                                │                       │
            ▼                                ▼                       ▼
    ┌─────────────┐    ┌──────────────┐   ┌─────────────┐    ┌─────────────┐
    │ user buffer │ →  │ page cache   │ → │ disk cache  │ →  │   platter   │
    │  (libc)     │    │ (kernel RAM) │   │ (drive RAM) │    │   (durable) │
    └─────────────┘    └──────────────┘   └─────────────┘    └─────────────┘
                              │                  │                 │
                              │                  │                 │
                       fsync flushes      drive's "write barrier" │
                       page→disk cache    flushes cache→platter   │
                       (with FUA flag,    (depends on OS+drive    │
                       both in one go)    cooperation)            │

write(2) only puts data in the kernel page cache. The kernel decides when to write it down. fsync(2) forces the page cache → disk path and a write barrier to the drive. Without fsync, you can power-cycle and find that the last 5–30 seconds of writes never made it.

The 4 fsync facts shell scripts get wrong

1. sync (command) returns when the request is sent, not when data is durable on slow media. On many older Linux versions, sync returned after queuing; modern kernels block until durable, but rotational drives with their own caches can still lie. For specific files, prefer fsync over global sync.

2. You must fsync the directory after creating, renaming, or deleting files. The directory’s contents (dentries) are themselves data that the kernel caches. Renaming tmp → dest modifies the directory inode; that change is in cache until you fsync the directory. Without it: power loss can leave the data on disk but the rename undone.

3. NFS and some network filesystems weaken fsync. O_SYNC and fsync are forwarded to the server, but the client cache may still hold writes. commit semantics (NFSv4) help, but if you’re writing scripts that need durability over NFS, design for “best effort” and use checksums + reads to verify.

4. fsync is per-file-descriptor. fsync(fd_a) does not flush data written via fd_b to a different file. Each file you care about needs its own fsync. (Some kernels with dirty_writeback_centisecs tweaks fsync everything together, but you can’t rely on it.)

A shell-callable durable-write helper

# Requires python3 (almost universally available); usable on all distros.

durable_write() {
  local dest="$1"
  local content
  content=$(cat)            # read stdin

  local dir tmp
  dir=$(dirname "$dest")
  tmp=$(mktemp "${dest}.XXXXXX")

  trap 'rm -f "$tmp"' EXIT

  # Write data.
  printf '%s' "$content" >"$tmp"

  # fsync the data file.
  python3 - "$tmp" <<'PY'
import os, sys
fd = os.open(sys.argv[1], os.O_RDONLY)
try:
    os.fsync(fd)
finally:
    os.close(fd)
PY

  # Atomic rename (same FS).
  mv "$tmp" "$dest"
  trap - EXIT

  # fsync the directory (dentry change).
  python3 - "$dir" <<'PY'
import os, sys
fd = os.open(sys.argv[1], os.O_RDONLY)
try:
    os.fsync(fd)
finally:
    os.close(fd)
PY
}

# Usage:
echo '{"port":8080}' | durable_write /etc/myapp/config.json

After this returns, config.json exists, has the right content, and the dentry change is on disk. A power loss now leaves either the old config.json or the new one — never an empty or partial file, never a missing file.

Mount Namespaces, Bind Mounts, and “Why Doesn’t My Backup See This?”

A mount stitches one filesystem (e.g. /dev/sda1) into the directory tree at a specific path (/var/lib/postgresql). A bind mount stitches an existing directory into another path: mount --bind /var/lib/postgresql /backup-source/postgresql. Now both paths show the same files; both paths are equally “real.”

This becomes a footgun when:

The “hidden under the mount” trap

# Original state.
ls /opt/app
# config.json  data/

# Someone mounts a new filesystem on /opt/app/data.
mount /dev/sdb1 /opt/app/data

# Now /opt/app/data shows the contents of /dev/sdb1.
# The old /opt/app/data/* contents are STILL THERE on the underlying filesystem,
# but you cannot see them through this path.

# Backups of /opt/app see:
#  - /opt/app/config.json (from underlying FS)
#  - /opt/app/data/* (from /dev/sdb1)
# But NOT the old /opt/app/data/* files (hidden under the mount).

# Disaster: someone unmounts /dev/sdb1, the old data reappears, the new data is "gone."

Detection: mount | grep /opt/app lists active mounts. findmnt /opt/app is GNU-only but readable. df shows mount-point-to-device mappings.

Defense:

• Don’t bind-mount or mount over a directory that is part of a backup root.
• If you do, document and check with findmnt --target <path> in your backup script.

Detecting filesystem boundaries with stat

# Two paths on the same filesystem return the same %d (device number).
stat -c %d /etc /etc/myapp     # same → 64769
stat -c %d /etc /tmp           # different → 64769 vs 25 (tmpfs)

# Use this to decide whether `mv` will be atomic:
is_same_fs() { [[ "$(stat -c %d "$1")" == "$(stat -c %d "$2")" ]]; }

is_same_fs /etc /tmp           # exit 1 (different)
is_same_fs /etc /etc/myapp     # exit 0 (same)

Many scripts ship with mv /tmp/foo /etc/foo and the user only finds out it was non-atomic when a crash leaves /etc/foo half-written. is_same_fs makes this checkable.

Putting It Together: lib/fs.sh Drop-In

# lib/fs.sh — durable, atomic-aware filesystem helpers.

# ─── Detection ─────────────────────────────────────────────────────────────
fs_dev() {
  if stat -c %d "$1" 2>/dev/null; then return; fi  # GNU
  stat -f %d "$1"                                    # BSD
}

is_same_fs() { [[ "$(fs_dev "$1")" == "$(fs_dev "$2")" ]]; }

# ─── fsync via python (portable) ───────────────────────────────────────────
fs_fsync() {
  if command -v python3 &>/dev/null; then
    python3 - "$1" <<'PY'
import os, sys
fd = os.open(sys.argv[1], os.O_RDONLY)
try: os.fsync(fd)
finally: os.close(fd)
PY
  elif command -v python &>/dev/null; then
    python - "$1" <<'PY'
import os, sys
fd = os.open(sys.argv[1], os.O_RDONLY)
try: os.fsync(fd)
finally: os.close(fd)
PY
  else
    sync   # heavy fallback: flushes everything
  fi
}

# ─── Safe write: atomic + durable ──────────────────────────────────────────
fs_safe_write() {
  local dest="$1" mode="${2:-0644}"
  local dir tmp
  dir=$(dirname "$dest")
  tmp=$(mktemp "${dest}.XXXXXX")
  trap 'rm -f "$tmp"' EXIT

  cat >"$tmp"                      # stdin → tmp
  chmod "$mode" "$tmp"
  fs_fsync "$tmp"                  # data on disk
  mv "$tmp" "$dest"                # atomic rename (same FS)
  fs_fsync "$dir"                  # dentry change on disk

  trap - EXIT
}

# ─── Safe symlink: atomic, no-clobber-non-link ─────────────────────────────
fs_safe_symlink() {
  local target="$1" link="$2"
  if [[ -L "$link" ]]; then
    [[ "$(readlink "$link")" == "$target" ]] && return 0
  elif [[ -e "$link" ]]; then
    echo "fs_safe_symlink: $link exists and is not a symlink; refusing" >&2
    return 1
  fi
  local tmp="${link}.tmp.$$"
  ln -sfn "$target" "$tmp"
  mv -T "$tmp" "$link" 2>/dev/null || mv "$tmp" "$link"
  fs_fsync "$(dirname "$link")"
}

# ─── Safe rename across possibly-different filesystems ────────────────────
fs_safe_rename() {
  local src="$1" dest="$2"
  if is_same_fs "$src" "$(dirname "$dest")"; then
    mv "$src" "$dest"
    fs_fsync "$(dirname "$dest")"
  else
    # Cross-FS: copy + verify + delete + fsync.
    local tmp; tmp=$(mktemp "${dest}.XXXXXX")
    cp "$src" "$tmp"
    fs_fsync "$tmp"
    mv "$tmp" "$dest"          # atomic within dest's FS
    fs_fsync "$(dirname "$dest")"
    rm -f "$src"
  fi
}

# ─── Hard-link to dest, fall back to cp on EXDEV ──────────────────────────
fs_link_or_copy() {
  local src="$1" dest="$2"
  if ln "$src" "$dest" 2>/dev/null; then return 0; fi
  cp -p "$src" "$dest"
  fs_fsync "$dest"
}

# ─── Inode + link-count diagnostics ───────────────────────────────────────
fs_inode_info() {
  local p="$1"
  printf 'path:  %s\n' "$p"
  printf 'inode: %s\n' "$(stat -c %i "$p" 2>/dev/null || stat -f %i "$p")"
  printf 'links: %s\n' "$(stat -c %h "$p" 2>/dev/null || stat -f %l "$p")"
  printf 'fs id: %s\n' "$(fs_dev "$p")"
  if [[ -L "$p" ]]; then
    printf 'symlink target: %s\n' "$(readlink "$p")"
  fi
}

Real-World Recipes

Recipe 1: Safe config update with rollback on validation failure

. /opt/myapp/lib/fs.sh

update_config() {
  local dest="$1" validator="$2"     # validator: command that exits 0 if config is valid
  local content
  content=$(cat)

  # Stage in same dir for atomic rename.
  local tmp
  tmp=$(mktemp "${dest}.XXXXXX")
  printf '%s' "$content" >"$tmp"

  # Validate the staged file BEFORE swapping in.
  if ! "$validator" "$tmp"; then
    echo "config validation failed; not deploying" >&2
    rm -f "$tmp"
    return 1
  fi

  # Backup current.
  if [[ -f "$dest" ]]; then
    fs_link_or_copy "$dest" "${dest}.prev"
  fi

  fs_fsync "$tmp"
  mv "$tmp" "$dest"
  fs_fsync "$(dirname "$dest")"
}

# Usage:
echo "$new_nginx_config" | update_config /etc/nginx/nginx.conf "nginx -t -c"
# nginx -t exits non-zero on syntax error; we never write a broken config.

Recipe 2: Detect cross-mount before mv

deploy_artifact() {
  local src="$1" dest="$2"
  if ! is_same_fs "$src" "$(dirname "$dest")"; then
    echo "warning: $src and $dest are on different filesystems" >&2
    echo "         atomic rename is not possible; using copy+verify" >&2
  fi
  fs_safe_rename "$src" "$dest"
}

deploy_artifact /tmp/build/myapp.tar /opt/myapp/releases/myapp.tar
# Detects /tmp = tmpfs vs /opt = root FS; falls back to copy+fsync+rename.

Recipe 3: Find and clean up dangling symlinks

# Find symlinks whose target doesn't exist.
find /etc -type l -exec sh -c '[ ! -e "$1" ] && echo "$1"' _ {} \;
# A more efficient single-pass version using -xtype:
find /etc -xtype l   # GNU only: matches links to non-existent targets

# Clean them up safely (with audit log):
find /etc -xtype l -print -exec rm {} \; >/var/log/dangling-cleanup.log

-xtype l resolves the symlink and matches if it still resolves to a symlink — which only happens when the chain ends in a missing target. It’s the cleanest way to find dangling links on GNU find.

Recipe 4: Audit hard-link counts to find shared inodes

# Find files with multiple hard links — useful for detecting accidental sharing.
find /etc -type f -links +1 -exec stat -c '%i  %h  %n' {} \; | sort

# Output example:
# 5012  2  /etc/passwd
# 5012  2  /etc/passwd-
# (these two share an inode; editor backups did this)

If you discover this in /etc/passwd, it’s because vipw or some legacy tool used ln for backups. A future useradd that opens /etc/passwd for write may write through to passwd- too if the editor doesn’t break the link. Hence the rule: editors should write a temp file and rename, never modify in place.

Footgun List

1. mv cross-filesystem silently degrades to copy+unlink. Always create temp files in the same directory as the destination.
2. > is not atomic. A crash mid-write truncates the file. Use temp + rename.
3. fsync of the file isn’t enough. Also fsync the parent directory after rename, or the dentry change can be lost.
4. ln -sf follows existing dir symlinks. Use ln -sfn to replace a symlink instead of writing into it.
5. stat follows symlinks by default; [ -e ] follows; [ -L ] doesn’t. Choose explicitly.
6. find defaults to physical (-P). Add -L to follow symlinks, but be aware of cycles.
7. cp follows symlinks by default; cp -P doesn’t. Backups can balloon if you copy through symlinks unintentionally.
8. rm of an open file doesn’t free space until the fd closes. Common with running services that hold log files; restart or close the fd to actually free.
9. Bind mounts hide files under the mount point. Backups and audits must check mount output.
10. NFS, S3FS, FUSE filesystems may not honor fsync strictly. Verify durability claims with the specific FS before assuming the rename pattern works.
11. tmpfs is RAM-backed; everything in /tmp is gone on reboot on most distros. Don’t put state markers, build artifacts you need post-reboot, or anything not transient there.
12. Hard links in different directories make file ownership ambiguous. “Whose file is this?” is unanswerable; both paths own it equally. Document carefully or avoid for shared files.

Quick-Reference Card

┌─ HARD LINK vs SYMLINK ────────────────────────────────────────────────┐
│  ln src dst           hard link (same inode, link count + 1)         │
│  ln -s src dst        symlink (new inode whose data is the path)     │
│  ln -sfn src link     replace existing symlink (idempotent)          │
│  hard links: same FS only, same inode, perms shared                  │
│  symlinks: cross-FS ok, dangling possible, resolved at access        │
└────────────────────────────────────────────────────────────────────────┘

┌─ ATOMIC WRITE PATTERN ────────────────────────────────────────────────┐
│  tmp=$(mktemp "${dest}.XXXXXX")    # SAME DIRECTORY as dest!         │
│  cat >"$tmp"                                                          │
│  fsync "$tmp"                       # data durable                   │
│  mv "$tmp" "$dest"                  # atomic on same FS               │
│  fsync "$(dirname "$dest")"         # dentry change durable          │
└────────────────────────────────────────────────────────────────────────┘

┌─ STAT / TEST FOLLOW BEHAVIOR ─────────────────────────────────────────┐
│  [ -e PATH ]      follows symlinks (no -L variant)                    │
│  [ -L PATH ]      true iff PATH is a symlink                         │
│  stat -c %s       follows                                            │
│  stat -L -c %s    explicit follow (some BSD systems flip default)    │
│  ls               doesn't follow (lists symlink type)                │
│  cp -P            doesn't follow (preserves symlinks)                │
│  find -L          follows                                            │
│  find -xtype l    GNU-only: dangling symlinks                        │
└────────────────────────────────────────────────────────────────────────┘

┌─ FSYNC SEMANTICS ─────────────────────────────────────────────────────┐
│  fsync(fd)        flushes data + metadata of THIS file               │
│  fdatasync(fd)    flushes data + size-relevant metadata only         │
│  sync             flushes everything (heavy)                         │
│  Always fsync the directory after rename                             │
│  NFS / S3FS / FUSE may not honor fsync strictly                      │
└────────────────────────────────────────────────────────────────────────┘

┌─ DETECTION COMMANDS ──────────────────────────────────────────────────┐
│  stat -c %d PATH                  filesystem id (same FS = same #)   │
│  findmnt --target PATH            mount info for path (GNU)          │
│  lsof +L1                         unlinked-but-open inodes           │
│  find / -inum N                   all dentries pointing to inode N   │
│  readlink -f PATH                 resolve symlinks fully             │
│  realpath PATH                    canonical absolute path            │
└────────────────────────────────────────────────────────────────────────┘

What’s Next

Filesystem semantics give you durable, atomic writes. The next layer down is the kernel itself: /proc, /sys, and sysctl — the live introspection and tuning interface that turns a shell script from “user-space code” into “control-plane operator.” The next lesson, /proc, /sys & sysctl: Kernel Introspection and Tuning From a Shell Script, walks through reading process state from /proc/$pid/, tuning runtime kernel parameters via /proc/sys and sysctl, and writing safe sysctl-management scripts that survive reboots.