Continue interrupted uploads via dd skip
While falling into the routing engine rabbit hole planing my next hiking trip via nextcloud maps, I needed to upload a big file to my server. Upload bandwidth sucked as it mostly does in Germany1. Well scp is my friend I guess:
$ scp jakobsweg.tar.gz root@my.server:~/
lost connection
Oops, after ten minutes with slow progress, the upload suddenly stopped due to a connection error.
Problem
Unfortunately scp can’t continue an interrupted upload. There are some StackOverflow solutions about using rsync but this solution using dd with its skip option got my attention.
Upload what’s missing
The solution suggest to just skip the already written bytes and append the rest to the destination file. Since this command is from the server perspective and my local machine is in a NAT’ed network which isn’t accessible from the www, I did the steps manually:
# checkout sizes of the source and aborted destination file
server $ ls jakobsweg.tar.gz
-rw-r--r-- 1 root root 490766336 Jan 14 14:26 jakobsweg.tar.gz
local $ ls jakobsweg.tar.gz
-rw-r--r-- 1 raphael raphael 1102126303 Jan 14 14:18 jakobsweg.tar.gz
local $ UPLOADED=490766336
local $ REMAINING=$((1102126303 - $UPLOADED))
# copy source file locally but skip the already transmitted bytes, divide remaining by something to raise the blocksize
# while copying with a blocksize takes ages, better fill up the filesystem alignment
local $ dd if=jakobsweg.tar.gz of=jakobsweg.tar.gz.part2 skip=$UPLOADED count=$(($REMAINING/1024) bs=1024 status=progress
# upload the remaining data as usual
local $ scp jakobsweg.tar.gz.part2 root@my.server:~/
# append the remaining data to the aborted file
remote $ cat jakobsweg.tar.gz.part2 >> jakobsweg.tar.gz
# compare checksum -> tada
local $ shasum jakobsweg.tar.gz
a8b00b334319932b5794595f3eafc1c5bedfaa35 jakobsweg.tar.gz
remote $ shasum jakobsweg.tar.gz
a8b00b334319932b5794595f3eafc1c5bedfaa35 jakobsweg.tar.gz
- citation needed