How To Run Multiple Commands In Parallel on Linux

Sarath Pillai's picture
Parallal Execution of Commands in Linux

Traditionally computers can only do one single thing at a time. It generally does not do multitasking. Well, in most cases (with single core computers - meaning computers with one single CPU), the computer gives you an illusion that multiple things are happening simultaneously.  

 

You might be running multiple things at the same time on a computer, but the computer will always execute one single task from your list of tasks at a time. But it will quickly switch to the next task, and then the next task and then the next (so basically in a way you can say that multiple tasks are progressing one by one).

 

This switching between tasks happens so fast that it is very difficult for us to notice. A typical computer will do 100s of switching between tasks in a single second. You can now imagine why we are under the illusion that multiple tasks are being executed at the same time.

 

This switching is generally called as context switching. When you have too many number of tasks waiting in line for CPU, then we say that the “machine is under load”. In this article, we are going to discuss the methods available to execute multiple processes in parallel on a Linux system.

 

The best example is to execute a command across 10s of servers from a machine. If you go one by one, it will consume a lot of time. But if we have a method to run the command simultaneously across all the servers in parallel, then that saves a lot of time. Again, from a computer/CPU standpoint it mainly deals with one task at a time, but keeps on switching between tasks, which happens too fast, and we are perfectly fine as far as multiple tasks are progressing simultaneously.

 

Let us dig through the different methods that are available in Linux, to execute commands simultaneously(in Parallel). The very first one is using the bash shell job control mechanism. We simply execute the command and then ask the shell to place that in background and proceed with the next command(while the first command is already being executed in the background), and then the next and so on. 

 

To demonstrate this, I have few junk files sitting in an object storage. Downloading all these files to a Linux machine can be done simultaneously to test and see how this parallel thing works.

 

Running Commands in Parallel using Bash Shell

 

I basically need to execute 3 wget commands and make them run in parallel. The best method is to put all the wget commands in one script, and execute the script. The only thing to note here is to put all these wget commands in background (shell background). See our simple script file below.

 

root@instance-10:/tmp# cat test_parallel.sh
#!/bin/bash
wget https://storage.googleapis.com/test-bucket-sarath/junkfile1 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile2 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile3 &

 

 

Notice the & towards the end of each command. This will put the command in background, and execute the next (put this in background) and proceed to the next and so on.

 

You can confirm all these commands are being executed simultaneously using another shell, and see the process list (in our case it should show 3 wget commands, with three different processes).

 

root@instance-10:~# ps aux|grep wget
root      4985 16.2  0.1  27732  5080 pts/0    D+   07:05   0:00 wget https://storage.googleapis.com/test-bucket-sarath/junkfile1
root      4986 15.8  0.1  27732  5164 pts/0    D+   07:05   0:00 wget https://storage.googleapis.com/test-bucket-sarath/junkfile2
root      4987 16.6  0.1  27732  5228 pts/0    D+   07:05   0:00 wget https://storage.googleapis.com/test-bucket-sarath/junkfile3
root      4994  0.0  0.0  10480  2140 pts/1    S+   07:05   0:00 grep --color=auto wget

 

 

We can clearly see from the above output that our three wget commands are running in parallel.

 

In case you need to execute several processes in batches, or in chunks, you can use the shell builtin command called "wait". See below.

 

#!/bin/bash
wget https://storage.googleapis.com/test-bucket-sarath/junkfile1 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile2 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile3 &
wait
wget https://storage.googleapis.com/test-bucket-sarath/junkfile1 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile2 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile3 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile1 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile2 &
wget https://storage.googleapis.com/test-bucket-sarath/junkfile3 &
wait

 

The first three commands wget commands will be executed in parallel. "wait" will make the script wait till those 3 gets finished. Once it is finished, the script will simultaneously run the next 6 commands, and wait till it completes and so on.

 

We can modify our script and make it a bit more generic as shown below.

 

root@instance-10:/tmp# cat test_parallel.sh
#!/bin/bash
for task in "$@"; do {
  $task &
} done

 

Now you can run as many commands as you like by using the script as shown below.

 

./test_parallel.sh "cmd1 arg1 arg2 arg3" "cmd2 arg1" "cmd3 arg1 arg2"

 

 

How to Run Multiple Processes Simultaneously using Xargs?

 

The next method that we can use to run processes in parallel is our regular xargs command. Xargs supports an option to specify the number of processes that you want to run simultaneously. See below.

 

seq 1 3 | xargs -I{} -n 1 -P 3 wget https://storage.googleapis.com/test-bucket-sarath/junkfile{}

 

seq command will simply give 1, 2, and 3 as output in three lines. Which is then passed to xargs as input using the standard Linux pipe. We have used  the option -I in xargs to remove the space character that generally gets added towards the end of our command.

Without -I{} and the last junkfile{}, xargs would construct our command with the file name of "junkfile 1, junkfile 2 etc" (we needed to remove the space), rather than junkfile1, junkfile2 and so on.

You can quickly confirm that 3 processes are running in parallel (as we passed -P 3), using another terminal and counting the number of wget processes as we did earlier. See below.

 

root@instance-10:/tmp# ps aux|grep wget
root      2197  0.0  0.0   6376   772 pts/0    S+   04:42   0:00 xargs -I{} -n 1 -P 3 wget https://storage.googleapis.com/test-bucket-sarath/junkfile{}
root      2198 11.0  0.1  27732  5160 pts/0    D+   04:42   0:01 wget https://storage.googleapis.com/test-bucket-sarath/junkfile1
root      2199  7.6  0.1  27732  5232 pts/0    D+   04:42   0:01 wget https://storage.googleapis.com/test-bucket-sarath/junkfile2
root      2200  6.4  0.1  27732  5040 pts/0    D+   04:42   0:00 wget https://storage.googleapis.com/test-bucket-sarath/junkfile3
root      2209  0.0  0.0  10480  2180 pts/1    S+   04:42   0:00 grep --color=auto wget

 

wget is just an example that we are using here for our tutorial. One reason for using wget with those three junkfile is to keep the process alive for few minutes, so that we can confirm the parallel execution (as the files are quite big, these processes will couple of few minutes to finish). You can replace wget with whatever is applicable in your use case.

 

How to Use GNU Parallel to Run commands simultaneously?

 

Apart from this, there is a tool from GNU, which is designed to execute jobs in parallel. Its called GNU Parallel. It can be installed by the below commands (depending upon your Linux distribution).

For RedHat Based Systems:

yum install parallel

 

For Debian Based Systems(Ubuntu):

apt-get install parallel

 

You can use GNU Parallel for some of the below use cases. These use cases nicely covers regular shell based system admin activities.

  • A list of files that can be passed as input to parallel command, to do some operations in parallel on all of them
  • You can give a list of IP addresses/hostnames, on which you need to fire up a command in parallel
  • List of links/URLs (similar to our wget example we saw with xargs and shell above)

 

GNU Parallel was designed by keeping xargs in mind, so majority of the command line options and parameters might match with xargs command. Let's first execute the wget example that we saw using GNU parallel.

 

seq 1 3 | parallel -j 5 -I{} wget https://storage.googleapis.com/test-bucket-sarath/junkfile{}

 

If you see the above command, parallel is using -j option (very similar to xargs, to specify the number of processes to run in parallel), and -I option (to remove the default space character). And without a doubt, it will run all our 3 wget commands simultaneously.

 

Do you want to compress all files in the current directory (in parallel and simultaneously)? We can do that very easily with parallel. Below shown is an example to achieve just that. 

 

ls | parallel -j 10 gzip

In the above example, a maximum of 10 compression will happen together. Similarly to uncompress/gunzip all the files simultaneously, run the below.

ls | parallel -j 10 gunzip

 

You can gzip all the files in the current directory using the below method as well (in parallel). In our below example, we have limited the number of jobs to 10, that will run in parallel.

 

parallel -j 10 gzip ::: *

 

Parallel has the option to pass a file as an argument, so that it can run command against entries in the file. For example, you can have a file with a list of URLs to download.

root@instance-10:/tmp# cat list-of-urls.txt
https://storage.googleapis.com/test-bucket-sarath/junkfile1
https://storage.googleapis.com/test-bucket-sarath/junkfile2
https://storage.googleapis.com/test-bucket-sarath/junkfile3

 

parallel -j 10 -a list-of-urls.txt wget

The above should download the URLs listed in the file "list-of-urls.txt" in parallel.

 

Parallel can also execute a series of commands specified in a test file. See an example below. Lets first create a test file with few sleep commands in there.

cat job.txt
sleep 100; echo "first"
sleep 100; echo "second"
sleep 100; echo "third"

 

Now let us ask parallel to execute all the commands in that file simultaneously. This can be done as shown below.

 

parallel -j 10 :::: job.txt

 

How to Manage Output in GNU Parallel?

A common issue while executing multiple commands in parallel is output. The output of different commands should not get mixed up. If you use the very first method that we saw in this article (ie: using the shell job control mechanism), there is actually no guarantee of the order of the output. For example, let us try ping command towards multiple hosts using the shell method of &.

 

root@testserver:~# ping -c 3 10.12.2.40 & ping -c 3 172.17.0.1 & ping -c 3 127.0.0.1 &
PING 10.12.2.40 (10.12.2.40) 56(84) bytes of data.
64 bytes from 10.12.2.40: icmp_seq=1 ttl=64 time=0.027 ms
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.018 ms
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.020 ms
64 bytes from 10.12.2.40: icmp_seq=2 ttl=64 time=0.029 ms

 

You can clearly see the output is completely messed up(outputs of those three pings are mixed up). Now let's try with parallel, and prevent the output from getting messed up. See below.

 

root@testserver:~# parallel -j 4 ping -c 3 ::: 10.12.2.40 172.17.0.1 127.0.0.1
PING 10.12.2.40 (10.12.2.40) 56(84) bytes of data.
64 bytes from 10.12.2.40: icmp_seq=1 ttl=64 time=0.026 ms
64 bytes from 10.12.2.40: icmp_seq=2 ttl=64 time=0.041 ms
64 bytes from 10.12.2.40: icmp_seq=3 ttl=64 time=0.025 ms

--- 10.12.2.40 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.025/0.030/0.041/0.009 ms
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.021 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=0.018 ms
64 bytes from 172.17.0.1: icmp_seq=3 ttl=64 time=0.019 ms

--- 172.17.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.018/0.019/0.021/0.003 ms
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.016 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.019 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.023 ms

--- 127.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.016/0.019/0.023/0.004 ms

 

Basically parallel will show the complete output of one process, only when it completes.  Does the order of the output matter to you? If you want the output also to be in the same order as your input, then you can use -k option in parallel as shown below.

 

parallel -k -j 4 ping -c 3 ::: 10.12.2.23 172.17.0.13 127.0.0.1

 

Please keep the fact in mind that GNU Parallel will find out the number of CPU cores available in the system, and it will run only one job per core. We used -j option in our examples to override this default behavior. If you have 100 commands to execute using GNU parallel, the jobs will be executed in smaller chunks. The chunk size is again determined by -j option.

 

If you want to quickly terminate GNU parallel, you can run fire up the below command. On receiving this signal, GNU parallel will finish the currently executing chunk and exit.

 

killall -TERM parallel

 

Parallel Execution of Commands on a list of Remote Machines

To be honest, I have not found GNU Parallel that user friendly, when it comes to remote command execution on a list of servers simultaneously. For this purpose, there are tools like clustershell and pdsh (agreed that GNU parallel has parameters like sshlogin and sshloginfile), but i did not find that straight forward. In my test cases, some of these commands were not stable enough to recommend to execute against number of servers in parallel.

Am going to start this with clustershell and then pdsh.

Clustershell can be easily installed using the command applicable to your platform.

 

RedHat Based Systems:

yum install clustershell

 

Ubuntu Based Systems:

apt-get install clustershell

 

As the name indicates, it is used for administering a cluster of servers. Basically execute something/fetch information from all machines in a cluster. Similar to other Linux utilities, the configuration file for clustershell is located at /etc/clustershell/clush.conf. I have the below inside that file.

 

#cat /etc/clustershell/clush.conf
[Main]
fanout: 64
connect_timeout: 15
command_timeout: 0
color: auto
fd_max: 16384
history_size: 100
node_count: yes
verbosity: 1
ssh_user: ubuntu
ssh_path: /usr/bin/ssh
ssh_options: -oStrictHostKeyChecking=no

You can execute a command of your interest across a coma separated list of servers using clustershell as below.

 

#clush -l ubuntu -w 10.12.2.40,10.12.2.114 uname -r
10.12.2.114: 4.4.0-1041-aws
10.12.2.40: 3.13.0-48-generic

 

Please note the fact in mind that clush uses the SSH private key inside /home/$user/.ssh/id_rsa file. For example, if am executing this command as "ubuntu" user, then the private key used will be /home/ubuntu/.ssh/id_rsa. The corresponding public key is expected to be present on all the servers where you are executing the command using clustershell.

 

Related:  SSH Working Explained

 

You can use shortcuts, and regular expressions, if you have servers in the format of node1.example.com or something like that. See below.

 

#clush -l ubuntu -w node[1-2].example.com uname -r
node2.example.com: 4.4.0-1041-aws
node1.example.com: 3.13.0-48-generic

 

You can copy files in parallel to multiple servers using clsutershell using the --copy option.

 

clush -w node[1-2].example.com --copy /home/ubuntu/testfile

The above command will copy the file /home/ubuntu/testfile to the same location on all servers

 

You can also creates a grouping of servers using the file /etc/clustershell/groups (if the file does not exist, then create it). An example groups file is shown below.

# cat /etc/clustershell/groups
web: node[1,2].example.com
db: node[3-4].example.com

You can now execute commands against these groups by calling the group name (web and db in our case).

 

#clush -l ubuntu -w @web uname -r
node2.example.com: 4.4.0-1041-aws
node1.example.com: 3.13.0-48-generic

 

# clush -l ubuntu -w @db uname -r
node4.example.com: 4.4.0-1041-aws
node3.example.com: 4.4.0-1041-aws

 

Clustershell supports an interactive mode for executing commands across multiple machines. It is quite interesting. We simply pass the -b option to clush command against one of our group and we can interactively fire commands on these group. An example is below.

 

# clush -l ubuntu -w @web -b
Enter 'quit' to leave this interactive mode
Working with nodes: node[1-2].example.com
clush> uptime
---------------
node1.example.com
---------------
 00:09:09 up 709 days, 18:19,  1 user,  load average: 0.11, 0.05, 0.05
---------------
node2.example.com
---------------
 00:09:09 up 47 days, 21:18,  0 users,  load average: 0.00, 0.00, 0.00
clush> quit

 

Similar to clustershell is another utility named pdsh. The idea for pdsh is similar to the utility rsh, which can be used to execute something on one remote host. However, pdsh can be used to execute commands in parallel on multiple hosts. 

Similar to clustershell, the installation is quite straightforward (a single apt-get or yum command depending upon your distribution).

RedHat Based Systems:

yum install pdsh

 

Debian/Ubuntu Based Systems:

apt-get install pdsh

 

The very first thing to do is to tell pdsh that we would like to use SSH for remote connections. This can be done using an environment variable.

export PDSH_RCMD_TYPE=ssh

 

To make it permanent, you can add it inside the user's .bashrc file as well. After executing the below command, you can logout and login to confirm that the environment variable is set and available for the user.

 

echo "export PDSH_RCMD_TYPE=ssh" >> ~/.bashrc

 

Let us fire up a command against our node[1-2].example.com using pdsh. See below.

 

# pdsh -l ubuntu -w node[1-2].example.com uname -r
node2: 4.4.0-1041-aws
node1: 3.13.0-48-generic

Without specifying the RCMD environment variable, you can also run commands like the one shown below.

 

root@jenkins:~# pdsh -w ssh:ubuntu@node[1-2].example.com uptime
node2:  00:32:34 up 47 days, 21:42,  0 users,  load average: 0.00, 0.00, 0.00
node1:  00:32:34 up 709 days, 18:43,  1 user,  load average: 0.08, 0.07, 0.05

 

Another interesting environment variable that we can use with pdsh is WCOLL. This will specify a file which contains a list of servers.

cat nodelist
node1.example.com
node2.example.com

 

export WCOLL=/root/nodelist

 

Similar to our previous example, you can add this one to .bashrc to make it permanent.

 

# pdsh -l ubuntu uptime
node2:  00:36:46 up 47 days, 21:46,  0 users,  load average: 0.00, 0.00, 0.00
node1:  00:36:47 up 709 days, 18:47,  1 user,  load average: 0.09, 0.06, 0.05

 

Rate this article: 
Average: 3.8 (601 votes)

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.