User Tools

Site Tools


Moving Recordings to a New Backend

My current MythTV machine has been expanded as far as it will go, so its time to upgrade. :-) I've built a more powerful server and I want to transition everything to it. I need to move all my data at once to the new server because I don't have enough space to run two servers permanently. And I have 1TB of data, so efficient transport will be an important factor.

Data Transfer

Initially, I attempted to move the TV recordings with a graphical SFTP client. But after several minutes, I realized my transfers were maxing out at about 2.2MBps. That's less than 20% of the capacity of my 100MBps network. I don't have anything else running, so what's limiting my download speed? I decided to investigate. Here are the results of my tests:

Transfer Method Example Command Speed
SFTP with hostname sftp . 2.2MBps
SCP with hostname scp . 2.2MBps
SFTP with IP address sftp me@ . 11.2MBps
SCP with IP address scp me@ . 11.1MBps
SCP with IP & blowfish scp -c blowfish me@ . 11.1MBps
SCP with IP & compression scp -C me@ . 4.4MBps

I read somewhere that using a less CPU-intensive encryption algorithm like blowfish would make my SCP transfers faster, but it didn't help. And turning on SSH compression (the -C) was reported to improve download speeds on slow connections. I tried it on my fast network and it actually made things worse.

So here's what I learned: using a hostname rather than an IP address made a big performance difference. It probably has something to do with my router configuration. I'll ignore that for now…the point is that I got an 11.1MBps transfer rate, which is 91% of the theoretical max of 12.5MBps. The difference is due to protocol overhead.

Data Synch

My MythTV backend is constantly recording new shows. Hence, the set of files I copied to my new server become stale within hours. I need a way to synchronize my new machine with the existing one until I can complete my conversion. Enter rsync.

cd /my/video/folder
rsync -avz .

I'm forced to enter my password with this command because I specified my user ID, but at least this got my files transferred.

/home/cfreyer/public_html/data/pages/technology/mythtv/moving_recordings_to_a_new_backend.txt · Last modified: 2010/02/21 21:59 by Chris Freyer