Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

rsync, scp, and sftp: Incremental Sync, Delete Flags, and Bandwidth Limits

Why the trailing slash decides whether rsync merges or mirrors, how --delete behaves with filters, safe resume with --partial, and where scp and sftp still beat a delta transfer.

~5 min readLinux
#Linux#SSH#Backup#Networking
3D illustration of two servers linked by an encrypted SSH tunnel carrying incremental delta blocks

Quick Summary

rsync, scp, and sftp all move files over SSH, and they are not interchangeable. rsync computes a delta: it compares the destination with the source using size and modification time by default (or a full checksum with -c), then transfers only what differs, with optional compression, deletion of stale destination files, and a bandwidth cap. scp copies whole files with almost no options and no resume. sftp is an interactive or batch file-transfer protocol with get, put, reget, and chmod, useful when you need a few files and no synchronization logic.

The single most consequential detail in rsync is the trailing slash: rsync -a src/ dest/ copies the contents of src into dest, while rsync -a src dest/ creates dest/src. That one character decides whether a --delete run merges a tree or mirrors it — and --delete is the flag that has erased more production directories than any other option in this list.

Bandwidth limits are --bwlimit=RATE in kilobytes per second (rsync 3 accepts K, M, G suffixes), applied per connection, so a mirror over a shared link stays polite. This guide covers the option semantics that matter (-a versus -aHAX, --delete variants, --partial, --link-dest), the scp/scp-versus-sftp differences in modern OpenSSH, safe dry-run practice, and the exit codes that tell you whether a transfer actually converged.

Basic Syntax & Core Signatures

bash
# Local copy: archive mode, verbose, human-readable rsync -av src/ dst/ # Remote pull / push (SSH transport is the default when a host is given) rsync -av user@host:/srv/app/ ./app/ rsync -av --delete ./app/ user@host:/srv/app/ # Custom port and a non-standard remote binary rsync -avz -e 'ssh -p 2222 -i ~/.ssh/deploy' ./dist/ user@host:/var/www/dist/ rsync -av --rsync-path='sudo rsync' ./etc/ user@host:/etc/ # Progress and resumability rsync -avh --progress --partial src/ dst/ rsync -a --info=progress2 --stats src/ dst/ # Bandwidth limit: 5000 KB/s (~5 MB/s), per connection rsync -av --bwlimit=5000 big/ user@host:/backup/big/ rsync -av --bwlimit=2M big/ user@host:/backup/big/ # Dry run first: prints what would change, transfers nothing rsync -avn --delete --itemize-changes ./app/ user@host:/srv/app/ # Excludes and includes (evaluated in order) rsync -av --exclude='node_modules/' --exclude='*.log' \ --include='logs/' --include='logs/*.json' --exclude='logs/*' \ ./app/ /backup/app/ rsync -av --exclude-from=.rsyncignore --files-from=filelist.txt ./ /backup/
bash
# scp: whole-file copy, port is -P, -p preserves modes and times scp file.txt user@host:/tmp/ scp -r ./dir user@host:/srv/ scp -P 2222 -p user@host:/var/log/app.log ./ scp -C user@host:/data/dump.sql ./dump.sql # on-the-fly compression # sftp: batch mode, scripted sftp -b - user@host <<'EOF' put -r ./release /var/www/release reget /var/log/app.log ./app.log chmod 640 /var/www/release/config.yml EOF sftp -P 2222 user@host # interactive: ls, cd, get, put, reget, df, lls

Exit codes from rsync that change how a script must react:

text
0 success 1 syntax or usage error 2 protocol incompatibility 3 file selection error 5 client error starting the protocol 10 socket I/O error 11 file I/O error (disk full, permissions) 12 protocol data stream error 13 errors with diagnostics 20 SIGUSR1 or SIGINT received 23 partial transfer (some files were not copied) 24 partial transfer due to vanished source files 30 timeout in data send/receive 35 timeout waiting for the daemon connection

Key Comparison Table

Aspectrsyncscpsftp
Transfer modelDelta (size+mtime, or -c checksum)Whole fileWhole file, resumable with reget
Resume a broken transfer--partial --append-verifyNoreget / reput
Delete destination extras--deleteNorm manually
Bandwidth limit--bwlimittrickle externally-l limits throughput
Compression-z (skip for pre-compressed data)-C-C
Preserve owner/group-o -g as root, --numeric-ids-p (modes/times only)chmod after
ACLs / xattrs-A -X (-aHAX)NoNo
Excludes / filtersRich filter rulesNoNo
Symlink handling-l (as link), -LFollows by defaultFollows
Incremental snapshots--link-destNoNo
Scriptable progress--info=progress2, --statsNoneProgress in batch output
Port flag-e 'ssh -p 2222'-P 2222-P 2222
Protocol in OpenSSH 9+SSHSFTP by default (-O restores legacy)SFTP
OptionWhat it actually does
-a-r -l -p -t -g -o -D; archives but does not include ACLs or xattrs
-zCompresses the wire; wastes CPU on local copies and on .gz/.zst sources
--deleteRemoves destination files absent from the source, during the transfer by default
--delete-delayDecides deletions at the end; lower peak deletion risk on full disks
--delete-excludedAlso deletes destination files matching --exclude — rarely what you want
-cCompares by checksum instead of size+mtime; slow, catches same-size edits
-uSkips files that are newer on the receiver
--inplaceWrites into the destination file instead of a temp file; no atomic replace
--partial-dir=.rsync-partialKeeps partial files next to the target for later resume
--numeric-idsTransfers UID/GID numbers, not names; correct for cross-host restores

Practical Edge Cases & Deep Dives

1. The trailing slash decides whether --delete merges or mirrors

bash
# Mirrors: dst becomes an exact copy of the contents of src, extra files deleted rsync -av --delete /srv/app/ /backup/app/ # Copies the directory itself: /backup/app/app/... is created rsync -av --delete /srv/app /backup/app/

The second form with --delete deletes nothing extra inside /backup/app — it adds a nested directory and leaves stale files alone, which is how "the backup is not actually pruning" incidents happen. Write both paths in the same style (both with, or both without a trailing slash) and dry run with -n --itemize-changes before trusting a new command line:

bash
rsync -an --delete --itemize-changes /srv/app/ /backup/app/ # *deleting old/file.txt # >f+++++++++ new/file.txt

--delete never runs while the transfer is failing for another reason unless you pass --ignore-errors, and it is refused entirely if the source path has a wildcard that matched nothing. Note also that --delete interacts with filters: an excludes rule protects nothing on the destination unless --delete-excluded is absent (it is absent by default) — so excluded files survive, which is usually what protects caches and .env files.

2. Why an unchanged file transfers again: mtime, size, and checksums

bash
# Default quick check: same size AND same mtime -> skipped rsync -av src/ dst/ # Same size, different content, same mtime: skipped. Use checksums to catch it. rsync -avc src/ dst/ # Restores that must not trust timestamps at all rsync -av --size-only src/ dst/

Plain rsync compares size and modification time with one-second granularity. Files generated by a build process often keep the same size and a copied timestamp, so a corrupt or stale file survives every sync. -c fixes correctness at the cost of reading every file on both sides. Windows/SMB destinations, filesystems with coarse timestamps (FAT: two seconds), and containers that copy files with --no-preserve=timestamps are the common sources of a mismatch in the other direction: everything transfers every time. --modify-window=2 absorbs coarse-granularity differences.

3. Resuming interrupted transfers safely

bash
# Option A: keep partial files, resume by re-running the same command rsync -av --partial --partial-dir=.rsync-partial big.iso user@host:/data/ # Option B: append-only for a growing file (log tail, append-only dump) rsync -av --append-verify bigfile user@host:/data/ # Never combine --append with --inplace on a file that can be rewritten

--partial keeps the partially transferred file at the destination, and the next run reuses it. --append tells rsync the destination is a strict prefix of the source — true for an append-only log, false for a database dump that was rewritten. --append-verify (rsync 3.0+) re-checks the existing bytes with a checksum before appending, which is the safer choice; plain --append trusts them. For scp there is no resume at all: a 4 GB copy that dies at 90% starts over, which is the practical reason to reach for rsync on anything large or flaky.

bash
# 2 MB/s ceiling, including for the SSH transport rsync -av --bwlimit=2M --timeout=300 /data/ user@host:/backup/data/ # scp is throttled externally trickle -s -d 2048 -u 2048 scp -r ./data user@host:/backup/ # sftp has a per-command throughput limit in Kbit/s sftp -l 16000 user@host

--bwlimit applies per rsync process; -P/parallel rsync runs multiply the ceiling. --timeout is an I/O inactivity timeout, not a total runtime limit, so a slow-but-alive link does not trip it. On long-haul copies, -z usually helps text and hurts already-compressed archives; measure once with --stats rather than assuming.

bash
BASE=/backup/app DATE=$(date +%F) rsync -aHAX --delete --numeric-ids \ --link-dest="$BASE/current" \ /srv/app/ "$BASE/$DATE/" rm -f "$BASE/current" && ln -s "$DATE" "$BASE/current"

--link-dest hard-links unchanged files from the reference directory, so each daily snapshot costs only the changed bytes while looking like a full tree. Three conditions make it work: same filesystem for the snapshots (hard links cannot cross one), --delete so removed files disappear, and never modifying a completed snapshot in place. Adding -H (hard links) for the source's own hard-linked files and -A -X for ACLs and xattrs avoids losing the capabilities some binaries need.

bash
# Verify a mirror actually matches before declaring success rsync -an --delete --itemize-changes /srv/app/ /backup/app/ | grep -v '^\.d' || echo "in sync"

Common Runtime & Configuration Mistakes

SymptomCauseFix
Destination has src/src/...Trailing slash missing on the sourcesrc/ to copy contents
--delete removed more than expectedSource was a wrong/empty path or a mounted filesystemAlways -n first, check the source exists
Files re-transfer every runmtime granularity or a filesystem that does not keep times--modify-window=2, --size-only, -c
Changed file not transferredSame size and mtime-c
rsync: connection unexpectedly closedRemote binary missing or wrong --rsync-pathInstall rsync on both ends
Permission denied on readRunning as the wrong user, or -o -g without rootMatch the file owners; sudo via --rsync-path
Owner became nobodyName lookup differs across hosts--numeric-ids and root on the receiver
Interrupted copy restarts from zero--partial missingAdd --partial --partial-dir=...
Exit code 24 in a script treated as failureSource changed during the runTreat 24 as success, fail on 23+
Transfer saturates the linkNo --bwlimit, or many parallel runs--bwlimit, serialise, --timeout
scp fails with "subsystem request failed"Server has no sftp subsystem, OpenSSH 9+ uses SFTPscp -O for the legacy protocol
Restored archive loses xattrs/ACLs-a does not include them-aHAX

The pattern behind nearly all of these is that rsync's defaults are fast, not strict: quick size-and-time checks, filters that keep excluded files on the receiver, and deletion that trusts your source path completely. A transfer is trustworthy when the source path is spelled exactly as intended, the first run was a dry run with --itemize-changes, and the exit code was read rather than ignored.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.