Scheduling: cron, systemd Timers & anacron — Choosing the Right Tool, Idempotency, Lockfiles & Drift-Free Recurring Jobs

Every team’s first scheduled job goes into cron. Eventually one of these things happens:

• The job runs at 03:00 daily, but the server reboots overnight and the job is silently skipped — nobody notices for weeks.
• A deployment causes the job to take 90 minutes instead of 5; cron starts a second copy at 03:00 the next day, and the two trample each other.
• A laptop or developer VM that’s only on during work hours keeps “missing” its weekly task.
• The script writes to a file, the user it runs as has different PATH than yours, and you spend an afternoon figuring out why node isn’t found.
• The cron job logs to /var/log/syslog, which gets rotated, and now you can’t find when the job last failed.
• You move from VMs to systemd-managed instances, and want first-class status, restart, and dependency tracking.

This lesson covers the three production-grade schedulers in Linux:

• cron — the classic, everywhere, simple, but minimal.
• systemd timers — modern, integrated with the service manager, far more capable.
• anacron — for machines that aren’t always on (laptops, dev VMs, edge devices).

Plus the non-negotiable patterns every scheduled job must follow: idempotency, concurrency locks, explicit PATH/TZ, output capture.

By the end, you’ll know which scheduler to choose, how to write jobs that don’t double-run, and how to operate them without surprises.

---

1. The three schedulers — a 30-second overview

Rule of thumb: on a modern systemd-based Linux server, prefer systemd timers for anything non-trivial. Use cron for legacy compatibility, simple short jobs, or where systemd isn’t available. Use anacron if uptime is unreliable.

---

2. cron — the everywhere scheduler

2.1 Crontab syntax

*  *  *  *  *  command-to-run
│  │  │  │  │
│  │  │  │  └─── day of week (0-7, Sun=0 or 7)
│  │  │  └────── month (1-12)
│  │  └───────── day of month (1-31)
│  └──────────── hour (0-23)
└─────────────── minute (0-59)

Common patterns

# Every minute
* * * * * /usr/local/bin/heartbeat

# Every 5 minutes
*/5 * * * * /usr/local/bin/poll

# Every hour at :15
15 * * * * /usr/local/bin/sync-feeds

# Every day at 03:00
0 3 * * * /usr/local/bin/nightly-backup

# Every Monday at 04:30
30 4 * * 1 /usr/local/bin/weekly-report

# 1st of every month at midnight
0 0 1 * * /usr/local/bin/monthly-rollup

# Every weekday at 09:00 (Mon-Fri)
0 9 * * 1-5 /usr/local/bin/biz-hour-task

# Every 15 minutes between 08:00 and 18:00
*/15 8-18 * * * /usr/local/bin/intraday

Special strings (most cron implementations)

@reboot   /usr/local/bin/on-startup
@hourly   /usr/local/bin/hourly        # = 0 * * * *
@daily    /usr/local/bin/daily         # = 0 0 * * *
@weekly   /usr/local/bin/weekly        # = 0 0 * * 0
@monthly  /usr/local/bin/monthly       # = 0 0 1 * *
@yearly   /usr/local/bin/yearly        # = 0 0 1 1 *

2.2 Where cron jobs live

Three places, increasing in scope:

crontab -e                              # Per-user crontab. ${USER}'s view.
crontab -l                              # List current user's crontab.
sudo crontab -e -u alice                # Edit alice's crontab as root.

# System-wide:
/etc/crontab                            # Has an extra "user" field
/etc/cron.d/*                           # Drop-in files, same format as /etc/crontab
/etc/cron.{hourly,daily,weekly,monthly}/*  # Scripts run by run-parts

The /etc/cron.d/* drop-in is the right place for system jobs. Each file looks like:

# /etc/cron.d/myapp-backup
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=ops@example.com

0 2 * * *  myapp  /usr/local/bin/myapp-backup

Note the 6-field format in /etc/cron.d and /etc/crontab: minute hour DoM month DoW user command. Per-user crontabs (from crontab -e) don’t have the user field.

2.3 The cron environment — the #1 source of “works on my machine” bugs

Cron runs jobs in a minimal environment:

• HOME, LOGNAME, SHELL are set.
• PATH is set, but to a minimal value (often /usr/bin:/bin).
• Most other variables are not inherited.
• The working directory is $HOME.

This is why node or aws or kubectl runs fine for you in an ssh shell but errors with “command not found” in cron — the binaries are installed somewhere not in cron’s PATH.

The fix: set environment explicitly at the top of the cron file or inside the script.

# At the top of /etc/cron.d/myapp:
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TZ=UTC
LC_ALL=C

0 3 * * * myapp /usr/local/bin/myapp-job

Or — and this is the more robust pattern — set them in the script itself. The script should not depend on the cron environment.

#!/usr/bin/env bash
set -Eeuo pipefail
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export TZ=UTC LC_ALL=C
# ... rest of script

That way, the script runs the same when you invoke it manually for testing as it does from cron.

2.4 Output and MAILTO

cron captures stdout and stderr from each job and mails it to the user (or to MAILTO=) on every run that produces output. If no MTA is configured, that mail vanishes — and so does any error message your script printed.

Two approaches:

1. Suppress all output (silent success, mail-on-error only):

   0 3 * * *  myuser  /usr/local/bin/job >/dev/null 2>&1
   

   Bad — you’ve thrown away all errors. Don’t do this without logging inside the script.

2. Redirect to a log file (recommended):

   0 3 * * *  myuser  /usr/local/bin/job >> /var/log/myapp/job.log 2>&1
   

   Now you have a file to grep. Pair this with logrotate (covered later in this lesson).

3. Use MAILTO= for actual alerts:

   MAILTO=ops@example.com
   0 3 * * *  myuser  /usr/local/bin/job
   

   Output → email. If your script runs cleanly with no output, no email. Print on errors and you get notified. Requires a working local MTA (postfix, ssmtp, etc.).

The best pattern is all of them: log to file inside the script (so you have an authoritative record), keep stderr quiet on success, and use MAILTO= for the rare error that escapes.

2.5 cron operators by location

# View running cron service:
systemctl status cron       # Debian/Ubuntu
systemctl status crond      # Red Hat / CentOS / Rocky / Alma

# Logs (where cron itself logs):
journalctl -u cron -f       # Live tail
grep CRON /var/log/syslog   # Debian
grep CRON /var/log/cron     # Red Hat

cron’s own logs tell you when jobs started and ended, but not what they did. Always combine with a per-job log file inside the script.

---

3. systemd timers — the modern alternative

systemd timers are far more powerful than cron. The trade-off is two unit files instead of one cron line, but you get logs, dependencies, sandboxing, and recovery for free.

3.1 The two-file pattern

A timer needs a .service (what to run) and a .timer (when to run).

# /etc/systemd/system/myapp-backup.service
[Unit]
Description=Daily backup of myapp data
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=myapp
ExecStart=/usr/local/bin/myapp-backup
StandardOutput=journal
StandardError=journal
Environment=TZ=UTC LC_ALL=C

# /etc/systemd/system/myapp-backup.timer
[Unit]
Description=Run myapp-backup daily at 03:00 UTC

[Timer]
OnCalendar=*-*-* 03:00:00 UTC
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Enable + start the timer (not the service — the timer will trigger the service):

sudo systemctl daemon-reload
sudo systemctl enable --now myapp-backup.timer

3.2 The key timer directives

OnCalendar= — calendar-based scheduling

OnCalendar=*-*-* 03:00:00 UTC          # Daily at 03:00 UTC
OnCalendar=Mon..Fri 09:00 UTC          # Weekdays at 09:00
OnCalendar=*-*-01 00:00 UTC            # 1st of every month
OnCalendar=hourly                       # Every hour at :00
OnCalendar=*:0/15                       # Every 15 minutes
OnCalendar=Mon *-*-* 04:30 UTC          # Every Monday at 04:30
OnCalendar=*-*-* *:00:30                # 30 seconds past every minute

The grammar is WEEKDAY YEAR-MONTH-DAY HOUR:MINUTE:SECOND TZ. * is a wildcard. Test syntax with:

systemd-analyze calendar 'Mon..Fri 09:00 UTC'
# Original form: Mon..Fri 09:00 UTC
# Normalized form: Mon..Fri *-*-* 09:00:00 UTC
# Next elapse: Mon 2024-03-11 09:00:00 UTC
# (in UTC): Mon 2024-03-11 09:00:00 UTC
# From now: 18h left

systemd-analyze calendar is invaluable when learning the syntax — it tells you exactly when a given expression will next fire.

OnBootSec= / OnUnitActiveSec= — relative scheduling

OnBootSec=15min                         # 15 min after boot
OnUnitActiveSec=1h                      # 1 hour after last activation

Useful for periodic-but-not-clock-aligned tasks (cleanup loops, telemetry pings).

Persistent=true — catch up missed runs

[Timer]
OnCalendar=daily
Persistent=true

If the machine was off when the timer should have fired, it fires immediately on next boot. This is the killer feature vs cron, which just silently skips missed runs.

Persistent=true is on by default for OnCalendar=daily/hourly/etc shortcuts but must be explicitly set for custom OnCalendar= expressions if you want this behaviour. Set it explicitly to avoid surprise.

RandomizedDelaySec= — avoid the thundering herd

[Timer]
OnCalendar=*-*-* 03:00 UTC
RandomizedDelaySec=300

The job will fire at a random time within 300 seconds of 03:00. If you have 100 hosts that all back up to the same S3 bucket, this prevents all of them from hitting the bucket at exactly the same instant.

3.3 The service unit — what to run

[Service]
Type=oneshot                            # One-shot job (vs Type=simple for daemons)
User=myapp                              # Run as this user (no need for sudo crontab)
Group=myapp
ExecStart=/usr/local/bin/myapp-backup
WorkingDirectory=/opt/myapp
Environment=TZ=UTC LC_ALL=C
EnvironmentFile=/etc/myapp/env

# Logging:
StandardOutput=journal                  # → journalctl -u myapp-backup.service
StandardError=journal

# Resource limits:
MemoryMax=2G
CPUQuota=50%
TimeoutStartSec=30min                   # Kill if it runs longer than this

# Sandboxing:
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp /var/log/myapp

Type=oneshot is the right type for scheduled scripts: systemd considers the unit “active” only while the script is running, then the unit goes “inactive” again. It’s exactly the model for one-off scheduled work.

3.4 Operating systemd timers

# List all active timers, sorted by next run:
systemctl list-timers

# List all timers including inactive ones:
systemctl list-timers --all

# Status of one timer:
systemctl status myapp-backup.timer
systemctl status myapp-backup.service   # Last run results

# Force a run now:
systemctl start myapp-backup.service

# Disable / enable:
systemctl disable --now myapp-backup.timer
systemctl enable --now myapp-backup.timer

# Logs (last 100 lines):
journalctl -u myapp-backup.service -n 100

# Logs since last hour:
journalctl -u myapp-backup.service --since '1 hour ago'

# Follow live:
journalctl -u myapp-backup.service -f

Compare with cron, where there’s no equivalent of list-timers (you have to read crontabs from multiple locations) and no equivalent of journalctl -u (logs are interleaved in syslog).

3.5 User timers

You can run timers as a non-root user without root privileges:

# As your user:
mkdir -p ~/.config/systemd/user
$EDITOR ~/.config/systemd/user/personal-backup.service
$EDITOR ~/.config/systemd/user/personal-backup.timer

systemctl --user daemon-reload
systemctl --user enable --now personal-backup.timer
systemctl --user list-timers

User timers run only while the user is logged in, unless you enable lingering:

sudo loginctl enable-linger $USER       # User services start at boot, run forever.

Useful for personal cron-style tasks without needing root.

3.6 cron syntax → systemd timer cheat sheet

---

4. anacron — for machines that aren’t always on

Cron and systemd timers (without Persistent=true) assume the machine is up when the schedule fires. For laptops, edge devices, or developer VMs, this assumption breaks.

anacron solves this by tracking when each job last ran, in /var/spool/anacron/<jobname>, and running any job whose interval has elapsed when anacron itself runs.

4.1 The anacrontab format

# /etc/anacrontab
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# period (days)  delay (min)  job-id          command
1               5             daily.backup    /usr/local/bin/daily-backup
7               25            weekly.report   /usr/local/bin/weekly-report
30              45            monthly.rollup  /usr/local/bin/monthly-rollup

• period — how often (in days). 1 = daily, 7 = weekly.
• delay — wait this many minutes after anacron starts before running, to avoid load spike.
• job-id — unique name; the timestamp file /var/spool/anacron/job-id tracks last run.
• command — what to run.

4.2 How anacron is triggered

anacron is not a daemon. It runs once and exits. It needs to be triggered:

• Most distros wire it via cron: /etc/cron.daily/anacron triggers anacron -s.
• Or via systemd: anacron.timer is shipped on many distros and fires hourly.

The flow:

1. Machine boots up.
2. anacron runs (via cron-launched script or its own systemd timer).
3. For each job in /etc/anacrontab:
   • Check /var/spool/anacron/<job-id> — is it older than period days?
   • If yes, sleep delay minutes, then run.
   • If no, skip.
4. Update timestamp on success.

So your laptop powered off for 4 days will, on next boot, run all daily and weekly jobs — once each.

4.3 When anacron is the right choice

• Developer VMs that are off overnight.
• Laptops that may be closed during scheduled times.
• Edge devices on intermittent power.
• Systems where you can’t predict uptime.

For 24/7 servers, prefer systemd timers with Persistent=true — anacron’s day-granularity is too coarse for most server workloads.

---

5. The non-negotiable patterns for scheduled jobs

Whichever scheduler you pick, every scheduled job must satisfy these properties:

5.1 Idempotency — running twice ≡ running once

If a job is interrupted, retried, or accidentally double-scheduled, its second run must not break things. Concrete examples:

Bad (not idempotent — accumulates duplicates):

psql -c "INSERT INTO daily_summary (date, total) VALUES (CURRENT_DATE, $total)"

Good (idempotent — UPSERT):

psql -c "INSERT INTO daily_summary (date, total) VALUES (CURRENT_DATE, $total)
         ON CONFLICT (date) DO UPDATE SET total = EXCLUDED.total"

Bad (not idempotent — appends to file):

echo "$DATE: $count records" >> /var/log/daily.log

Good — appending is fine if duplication is harmless, but for accounting/billing data, you need a way to detect “already processed”:

marker="/var/lib/myapp/processed/$DATE"
if [[ -f $marker ]]; then
  log_info "Already processed $DATE — skipping"
  exit 0
fi
process_data
touch "$marker"

The marker pattern is universal: at the start of any job, check whether this iteration has already completed. If yes, skip.

5.2 Concurrency control — two copies must not run simultaneously

The classic failure: a job that normally takes 5 minutes today takes 2 hours. Cron starts a second instance an hour into the first. Now you have two scripts both writing to the same database / files / S3 keys.

The fix is flock (covered in detail in L16). One-line invocation:

0 3 * * *  myuser  /usr/bin/flock -n /var/run/myjob.lock /usr/local/bin/myjob

Or inside the script (more flexible, lets you log “skipped”):

#!/usr/bin/env bash
set -Eeuo pipefail

LOCK=/var/run/myjob.lock
exec 9>"$LOCK"
if ! flock -n 9; then
  echo "Another instance is running, exiting." >&2
  exit 0
fi

# ... real work ...

flock -n = non-blocking; if locked, exit immediately. Use flock -w 60 to wait up to 60 seconds for the lock instead.

In systemd, you can declare a service Restart=no with Type=oneshot and let RemainAfterExit=no handle the “this unit is currently running” status — but flock is still the safer belt-and-braces approach if the same script can be invoked manually.

5.3 Explicit environment — PATH, TZ, LC_ALL

Every scheduled job script should start with:

#!/usr/bin/env bash
set -Eeuo pipefail
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export TZ=UTC
export LC_ALL=C

Don’t trust the scheduler’s environment. Don’t trust which version of node or python is in PATH. Use absolute paths for any binary not in the standard locations:

/opt/node/bin/node /opt/myapp/index.js
/usr/local/bin/aws s3 cp file s3://bucket/file

5.4 Log everything — to a file the script controls

LOG=/var/log/myapp/job-$(date -u +%Y-%m-%dT%H:%M:%SZ).log
exec >>"$LOG" 2>&1
echo "=== run $(date -u +%FT%TZ) ==="

exec >>"$LOG" 2>&1 redirects all subsequent stdout/stderr to the log file. Now even if the scheduler eats the output, you have it.

For systemd, StandardOutput=journal goes to journald, queryable per-unit — usually you don’t need an extra log file. For cron, you almost always do.

5.5 Exit code discipline

Cron and systemd both interpret the exit code:

• 0 = success (silent in cron, “active (exited)” in systemd).
• non-zero = failure (cron emails it; systemd marks the unit failed).

Every error path in your script should exit 1 (or another non-zero value). The set -e from strict mode handles many cases, but:

if ! curl -fsS "$URL" -o "$file"; then
  echo "Failed to fetch $URL" >&2
  exit 1
fi

Don’t print “ERROR: …” and continue. The scheduler can’t see your console; it only sees the exit code.

5.6 Timeouts — don’t wedge forever

A job that hangs forever will block the next scheduled run (with flock) or pile up parallel instances (without). Add an outer timeout:

# In the script:
timeout --signal=TERM --kill-after=30s 1h /usr/local/bin/inner-task

# Or systemd:
[Service]
TimeoutStartSec=1h

After the timeout, the job is killed. Combined with retries (next section), this turns “infinite hang” into “logged failure, retry next run.”

---

6. Drift-free patterns

6.1 The “drift” problem

Suppose you run a job hourly that takes ~6 minutes and posts to https://api/events:

• Hour 1: starts at 12:00:00, ends at 12:06:00.
• Hour 2: starts at 13:00:00, ends at 13:06:00.

That’s fine. But consider a job that runs every 6 minutes (*/6 * * * *) and itself takes 7 minutes occasionally:

• 12:00 starts, ends 12:07.
• 12:06 wants to start, but flock blocks → skipped.
• 12:12 starts.

You missed an iteration. With OnUnitActiveSec=6min in systemd, the timer is rearmed after the previous run finishes, not on a fixed schedule, so you never overlap or drift.

[Timer]
OnBootSec=1min                          # First run, 1 min after boot
OnUnitActiveSec=6min                    # Then 6 min after each finish

This pattern is rarely available in cron (you’d have to fake it with a self-rescheduling script). For “every N minutes after the previous one finished”, use systemd timers.

6.2 Calendar drift on long-running tasks

If your “daily 03:00” job takes 90 minutes, today it starts at 03:00 and ends at 04:30. Tomorrow, it starts at 03:00 again — fine. But what if you wanted to chain it with another job (“the report runs after the backup”)?

In cron: hard-coded delay (0 5 * * *), hope the backup is done. Brittle.

In systemd: dependency:

# myapp-report.service
[Unit]
Description=Daily report
After=myapp-backup.service
Requires=myapp-backup.service          # Pull in backup if it isn't running

[Service]
Type=oneshot
ExecStart=/usr/local/bin/myapp-report

# myapp-report.timer
[Timer]
OnCalendar=*-*-* 03:30 UTC             # Schedule report at 03:30
Persistent=true

After=myapp-backup.service makes systemd wait for the backup unit to finish before starting the report. No hard-coded delay; works correctly even if the backup runs long.

Note: this only works if the backup is also a systemd unit (e.g. triggered by its own timer, or by the report’s Requires=). cron jobs aren’t visible to systemd dependency tracking.

6.3 Retries with exponential backoff

A job that needs to fetch from a flaky API should retry. This is inside the script, not at the scheduler level:

fetch_with_retry() {
  local url=$1 out=$2
  local attempt=0 max=5 delay=10
  while (( attempt < max )); do
    if curl -fsS --max-time 60 "$url" -o "$out"; then
      return 0
    fi
    attempt=$((attempt + 1))
    if (( attempt < max )); then
      log_warn "Fetch failed, retry $attempt/$max in ${delay}s"
      sleep "$delay"
      delay=$((delay * 2))
    fi
  done
  log_error "Failed after $max attempts: $url"
  return 1
}

set -e will then propagate the non-zero exit and the scheduler will alert.

6.4 Jitter on multi-host fleets

If 100 hosts run 0 3 * * * against a shared service, all 100 hit at exactly 03:00:00. Spread the load:

In cron — randomise the delay:

# At start of script:
JITTER=$(( RANDOM % 300 ))             # 0..299 seconds
sleep "$JITTER"

In systemd — RandomizedDelaySec=:

[Timer]
OnCalendar=*-*-* 03:00 UTC
RandomizedDelaySec=300

systemd’s randomisation is deterministic per host (seeded by the hostname) but spread across the fleet. Better than RANDOM because it doesn’t change between runs on the same host.

---

7. Logging and rotation

7.1 Logrotate for cron-managed log files

If your script writes to /var/log/myapp/job.log daily, you need to rotate it or it grows forever.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        # If the app holds the file open, signal it to reopen.
        # For a per-cron-run script, this is usually empty.
    endscript
}

logrotate itself runs daily via a cron job (/etc/cron.daily/logrotate). The settings:

• daily — rotate every day.
• rotate 14 — keep 14 old versions, then delete.
• compress — gzip old versions.
• delaycompress — don’t compress yesterday’s (so it’s still grep-able).
• missingok — don’t error if the log doesn’t exist yet.
• notifempty — don’t rotate empty files.
• create 0640 myapp myapp — create new log file with these perms.

7.2 systemd journald — usually no rotation needed

journald has built-in retention based on size and age, configured in /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=2G
SystemMaxFileSize=200M
MaxRetentionSec=2week

For systemd-timer-managed jobs that log to journal, you don’t need logrotate.

7.3 Structured queries via journalctl

# All logs for one timer's service:
journalctl -u myapp-backup.service

# Last 7 days:
journalctl -u myapp-backup.service --since '7 days ago'

# Just errors:
journalctl -u myapp-backup.service -p err

# JSON output (for piping into jq):
journalctl -u myapp-backup.service -o json | jq 'select(.PRIORITY <= "3")'

Far more queryable than grepping files. This is one of the strongest arguments for systemd timers.

---

8. Choosing between cron, systemd timer, and anacron

A decision tree:

Is the host always-on (server)?
├── No → anacron (laptops, intermittent hosts)
└── Yes
    ├── Is it a systemd-managed Linux?
    │   ├── No → cron (BSD, alpine without OpenRC-systemd, busybox)
    │   └── Yes
    │       ├── Is the job trivial (1 line, no deps, no monitoring needs)?
    │       │   ├── Yes → cron (faster to set up)
    │       │   └── No → systemd timer (better in every other dimension)

Specific cases that lean systemd:

• Need to chain jobs (After= / Requires=).
• Need resource limits (MemoryMax=, CPUQuota=).
• Need sandboxing (PrivateTmp=, NoNewPrivileges=).
• Need queryable per-job logs (journalctl -u).
• Need to catch up missed runs (Persistent=true).
• Need fine-grained intervals or “after previous finishes” semantics.

Specific cases that lean cron:

• Single-line, well-known job on a system where everyone knows where to look.
• Container with no systemd (alpine, distroless).
• BSD or Solaris.
• Trivially short job that has no operational complexity.

Specific cases that mandate anacron:

• Laptops and developer VMs.
• IoT/edge devices on unreliable power.

8.1 Heterogeneous fleets

If you have a mix of systemd hosts and non-systemd containers, don’t try to use systemd timers on the containers — write the job for cron, run it identically across all hosts. Operational consistency > scheduler features.

---

9. End-to-end example: nightly backup with both schedulers

The same script (/usr/local/bin/nightly-backup) — written once, scheduled differently:

#!/usr/bin/env bash
# /usr/local/bin/nightly-backup
# Idempotent, locked, UTC-only nightly backup.

set -Eeuo pipefail
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export TZ=UTC LC_ALL=C

SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
source /usr/local/lib/myapp/lib/log.sh
source /usr/local/lib/myapp/lib/time.sh

LOCK=/var/run/nightly-backup.lock
exec 9>"$LOCK"
if ! flock -n 9; then
  log_warn "Another instance running, exiting."
  exit 0
fi

DATE=$(today_utc)
DEST="/srv/backups/$DATE"
MARKER="$DEST/.complete"

if [[ -f $MARKER ]]; then
  log_info "Backup for $DATE already complete — skipping"
  exit 0
fi

log_info "Starting backup → $DEST"
mkdir -p "$DEST"

# ... actual backup work ...
rsync -aP /data/ "$DEST/data/"

touch "$MARKER"
log_info "Backup for $DATE complete"

Schedule it via cron:

# /etc/cron.d/nightly-backup
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=ops@example.com

5 3 * * *  myapp  /usr/local/bin/nightly-backup >> /var/log/myapp/backup.log 2>&1

Or schedule it via systemd:

# /etc/systemd/system/nightly-backup.service
[Unit]
Description=Nightly backup of /data
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=myapp
Group=myapp
ExecStart=/usr/local/bin/nightly-backup
StandardOutput=journal
StandardError=journal
Environment=TZ=UTC LC_ALL=C
TimeoutStartSec=2h

# Sandboxing:
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/srv/backups /var/log/myapp /var/run

# /etc/systemd/system/nightly-backup.timer
[Unit]
Description=Run nightly backup at 03:05 UTC

[Timer]
OnCalendar=*-*-* 03:05:00 UTC
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now nightly-backup.timer

The script is identical. Idempotent (marker file). Locked (flock). UTC. Strict mode. Logs to its own log via the cron approach, or to journald via the systemd approach. Either way, the operational behaviour is the same.

---

10. Quick reference card

cron — the must-knows

*  *  *  *  *  command      # min hour day month dow
@daily, @hourly, @reboot     # special strings

# Top of crontab/cron.d:
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin
TZ=UTC
MAILTO=ops@example.com

crontab -l                   # List
crontab -e                   # Edit
sudo crontab -e -u alice     # Edit alice's
systemctl status cron        # Service health
journalctl -u cron           # Cron daemon logs

systemd timers — the must-knows

systemctl list-timers              # All active
systemctl list-timers --all        # Including stopped
systemctl status myjob.timer       # Timer status
systemctl status myjob.service     # Last run
systemctl start myjob.service      # Run now
journalctl -u myjob.service -f     # Live logs
systemd-analyze calendar 'Mon..Fri 09:00 UTC'   # Test cron expr

# .timer
[Timer]
OnCalendar=*-*-* 03:00 UTC
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Every scheduled script’s mandatory preamble

#!/usr/bin/env bash
set -Eeuo pipefail
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export TZ=UTC LC_ALL=C

LOCK=/var/run/$(basename "$0").lock
exec 9>"$LOCK"
flock -n 9 || { echo "Already running"; exit 0; }

The 7 commandments of scheduled jobs

1. Idempotent: running twice is identical to running once.
2. Locked: only one instance at a time (flock).
3. Pinned environment: PATH, TZ=UTC, LC_ALL=C set explicitly.
4. Logged: to a file or journald — never relying on cron’s mail.
5. Bounded runtime: outer timeout so a hang doesn’t block forever.
6. Exit-coded: every error path sets a non-zero exit.
7. Jittered: RandomizedDelaySec= or sleep $((RANDOM % N)) on fleets.

---

11. Wrap-up

Cron has been the default scheduler for 40 years; it’s still fine for simple cases. But for production workloads on systemd-managed Linux, systemd timers are a clear upgrade: catch-up after downtime, journal-integrated logs, dependency tracking, sandboxing, resource limits, and fleet-friendly jitter — all without needing extra glue.

For laptops and intermittent hosts, anacron fills the gap that neither cron nor non-persistent systemd timers cover.

Whichever scheduler you pick, the seven commandments are non-negotiable: idempotency, locking, explicit environment, logging, timeouts, exit codes, jitter. Those discipline a job into something you can leave running and trust to behave.

Next: L21 — testing shell scripts with bats-core / shunit2. We’ll cover how to test the very functions we’ve been building (logging, retries, atomic writes, time helpers, scheduler-friendly wrappers), how to mock external commands, and how to wire test runs into CI so you find regressions before production does.