Today, I had to reinstall my Ubuntu server where I regularly test different applications and softwares. My Ubuntu server has over 200 GB data, and I don’t want to lose it. I can transfer those data via scp, or setup NFS, FTP to copy files, but I am pretty sure it would take hours to transfer such big files. While searching for an alternative method, I came across the following method. We can quickly transfer large files between between two systems over Network using netcat, tar, and pv commands in Linux and Unix operating systems. Unlike other methods, I find it very fast and quick. For those who don’t know, Netcat is a simple Unix utility which reads and writes data across network connections, using TCP or UDP protocol. tar is a commandline archiving tool, and pv, pipe viewer, is used to monitor the progress of data.
Now, let us go ahead and see how to transfer large files quickly between two Linux systems. It’s not that difficult.
Make sure you have installed netcat and pv on your systems. If they are not installed already, you can install them as shown below. The tar command is available by default on most Linux systems, so you don’t have to install it separately.
On Arch Linux and its derivatives:
sudo pacman -S netcat pv
On RHEL, CentOS, Fedora:
sudo yum install epel-release
sudo yum install nc pv
sudo dnf install nc pv
On Debian, Ubuntu, Linux Mint:
sudo apt-get install netcat pv
Now, to copy a large file, we do the following.
On the receiving node (destination system), run the following command as root user:
netcat -l -p 7000 | pv | tar x
On the sending node (source system), run this command as root user:
tar cf - * | pv | netcat 192.168.1.105 7000
Here, 192.168.1.105 is my destination system. tar cf – * will copy everything in the current working directory to the destination system, and the files will be extracted at the other end.
Note: On RHEL, CentOS systems, use “nc” instead of “netcat” as shown below. And, you need to add the port “7000” to the iptables / firewall-cmd on the target system.
After adding the port on target system, you can transfer the larger files as shown below.
On destination system:
nc -l -p 7000 | pv | tar x
On source system:
tar cf - * | pv | nc 192.168.1.105 7000
Also, You can specify a particular file like below.
tar cf - /home/sk/test.file | pv | netcat 192.168.1.105 7000
Please be mindful that both system should have netcat installed. Now, grab a cup of coffee. You’ll see that the files will be copied very quickly than the traditional methods like scp.
Also, you will not see any sign of the file transfer completion on both sides. These commands will keep running until you manually stop them. You need manually check the file sizes on both systems using “du -h <filename>” command. If the file size in destination system is same as in source system, then file transfer process is completed. Afterwords, you can quit the command by pressing CTRL+C.
Quickly Transfer Large Files Between Two Systems In Unix
On Unix operating systems, netcat is called as nc. So, to copy large files between systems over Ethernet, the command would be:
On destination system:
nc -l 7000 | pv | tar -xpf -
On source system:
tar -cf - * | pv | nc 192.168.1.105 7000
Again, these commands should be run as root user. And, both source and destination systems should have netcat and pv installed. Transferring large files over LAN using netcat and tar can indeed save a lot of time.
Disclaimer: There is no security in this method. Because, as you see in the above example, there is no authentication either side. All you need to know the destination system’s IP address. Transferring files using netcat is recommended only in protected networks. If you concern about security, I strongly suggest you to use scp command.
That’s it. Do you know any other way to copy large files quickly? please share it in the comment section below. I am all ears!
I will be soon here with another interesting article. Until then, stay happy!
Thanks for stopping