Mon 07 October 2013

Filed under OS

Tags Awesome Linux

Compression Problems

Have legacy iSCSI target files in /targets/hosts/fname[0-9].img. I need to relocate them to /backup/Legacy/hosts/fname[0-9].img.gz

I first tried this:

for x in */*.img;
 DIR=$(basename $(dirname ${x}))
 FILE=$(basename ${x})
 mkdir -p /path/to/backup/${DIR}
 gzip -9c ${x} > /path/to/backup/${x}.gz
done

Practical Yeild (via iotop): 2MB/s.

I next tried this:

for x in */*.img;
 DIR=$(basename $(dirname ${x}))
 FILE=$(basename ${x})
 mkdir -p /path/to/backup/${DIR}
 dd if=${x} bs=16384 | gzip -9c > /path/to/backup/${x}.gz
done

Practical Yeild (via iotop): 2MB/s.

I was growing throughouly disgusted with the entire problem. I download pigz and I set about to fix the problem:

for x in */*.img;
 DIR=$(basename $(dirname ${x}))
 FILE=$(basename ${x})
 mkdir -p /path/to/backup/${DIR}
 dd if=${x} bs=16384 | pigz -9c| dd of=/path/to/backup/${x}.gz bz=4096
done

Practical yeild: Metal Speed. 80-110MB/sec

Comment

Up To Something © Joshua M Schmidlkofer Powered by Pelican and Twitter Bootstrap. Icons by Font Awesome and Font Awesome More