How to run ssh connection in parallel while running multiple commands Linux

Question

I am trying to understand best way to run multiple command on a remote host with ssh in a parallel way and Display the output on the screen while squeezing the warning and errors on the screen.

This post is mainly in connection with previous post where KamilCuk answered the POST with some beautiful tricks however i have some more open questions.

Below code works for the given commands, however few of my systems really old and do not understand the nproc command to get the number of processor on the system hence i need to use grep -c processor /proc/cpuinfo which is not working if i do so in the current solution like tmp=$(ssh "$server" bash -c 'grep -c "^processor" /proc/cpuinfo; free -g').

also would be look to incorporate conditions like if nproc is there then use the file /proc/cpuinfo to count the CPU, this is Just one thing similar to many other. So, what i am looking forward if there are more ways to do it if you have different subsets of command where you need to get values from commands and files.

 #!/bin/bash

read -rsp $'Please Enter password below: ' SSHPASS
export SSHPASS

work() {
   server=$1
   # ONE connection
   tmp=$(ssh "$server" bash -c 'nproc; free -g')
   # parsing later
   cpu_info=$(<<<"$tmp" awk 'NR==1')
   mem_info=$(<<<"$tmp" awk '/Mem:/{printf $2}')
   swap_info=$(<<<"$tmp" awk '/Swap:/{printf $2}')
   # outputting
   printf "%-40s %5s %5s %5s\n" "$server" "$cpu_info" "$mem_info" "$swap_info"
}
export -f work
< /home/user1/mem xargs -P0 -n1 -d'\n' bash -c 'work "$@"' _

I tried Below where the idea is borrowed from But it just prints the SEREVR name and nothing else:

#!/bin/bash
read -rsp $'Please Enter password below: ' SSHPASS
export SSHPASS

for SERVER in '$(cat /home/user1/mem)'
do
sshpass -e ssh -q -t -oStrictHostKeyChecking=no $SERVER << EOF
        cpu_info=$(grep processor /proc/cpuinfo | awk 'NF==3{count++} END {printf count}')
        mem_info=$(free -g | awk /Mem:/{printf' '})
        swap_info=$(free -g | awk /Swap:/{printf' '})
        
        printf "%-40s %5s %5s %5s\n" "$server" "$cpu_info" "$mem_info" "$swap_info"

EOF
done | tee  2>/dev/null

I may have multiple mistakes please excuse me for the same just trying to learn.

EDIT Note:

I Know there will be a definite questions why not use ssh keys or already existed tooling like ansible , salt etc, the reason is only , due to company security policies somewhat we can not use any of them else there is no way re-inventing the wheel, however we are using Ansible and other tooling heavily in the normal environment.

This should work `tmp=$(ssh "$server" bash -c '"grep -c ^processor /proc/cpuinfo; free -g"')` Also in your last script, you should use `'EOF'` to avoid expansion be done locally. — Philippe, Mar 06 '21 at 20:55
Have you thought about [sharing SSH keys](https://stackoverflow.com/questions/4520578/sharing-ssh-keys) - so you don't have to do anything with passwords - just run what you want where? — Mr R, Mar 09 '21 at 05:10
What do you mean "in parallel" - your example doesn't seem to try anything in parallel?? OR are you just trying to say run multiple commands remotely ... e.g `ssh user@IP 'command1 args1 && command2 args2' `[I put the quotes to force eval at the remote machine, otherwise you have to be very careful with quoting] — Mr R, Mar 09 '21 at 05:15
@MrR, sorry for leading the confusion, however, in my previous post i mentioned as due to some company policy reasons i am not able to utilize `keys` & `tools such as ansible`. However, `in Parallel` , as you see the First code provided executes the `prallel threads` for ssh connections with `xargs` . — user2023, Mar 09 '21 at 05:56
Ah yes `xargs -PN` - note to self "read more carefully" - haven't done that before (cute parallelization).. -P0 you possible might want to nice/renice things so your interactive process still has control of the box. ACTUALLY you do want to control how many concurrent outbound connections you have [corporate firewall might get upset otherwise] - so do -P N where N>0 — Mr R, Mar 09 '21 at 06:18
Can I ask how you intend to track "failures"? i.e. couldn't connect to machine? [look for connect failure]? And how long is too long - shgould be a failure (SSH connect options). — Mr R, Mar 09 '21 at 06:31
Thats true, i would like condition to be placed `if then elif ese` to track the failures. — user2023, Mar 09 '21 at 07:19
Can you create `shell script` on remote hosts? Do you have permission? — Shakiba Moshiri, Mar 16 '21 at 08:00
I can create it as a user not as a root however as a sudo can be done. — user2023, Mar 16 '21 at 10:49

Ivan · Accepted Answer · 2021-03-10T07:25:22.753

3

First, this loop won't work:

for SERVER in '$(cat /home/user1/mem)'
do
...
done

because of '', remove '' or better read hosts list in var or an array first:

var=$(cat  /home/user1/mem)
arr=($(cat /home/user1/mem))

and loop over this var(arr):

for SERVER in   $var     ; { ...; }
for SERVER in "${arr[@]}"; { ...; }

Parallel ssh connections can be done like this:

cmd="
    cpu_info=\$(grep processor /proc/cpuinfo | awk 'NF==3{count++} END {printf count}')
    mem_info=\$(free -g | awk /Mem:/{printf' '})
    swap_info=\$(free -g | awk /Swap:/{printf' '})

    printf \"%-40s %5s %5s %5s\n\" \"\$HOSTNAME\" \"\$cpu_info\" \"\$mem_info\" \"\$swap_info\"
"

for SERVER in "${arr[@]}"; {
    sshpass -p "$SSHPASS" ssh -q -t -oStrictHostKeyChecking=no "$SERVER" "$cmd" &
}

edited Mar 10 '21 at 07:25

answered Mar 10 '21 at 05:51

Ivan

6,188
1
16
23

Thanks for the answering +1 for the same, however why we are using `\` everywhere, secondly how this cmd will be printed on the screen from where we are running the script. your solution is just logging in to the systems one by one and asking to exit manually, do you think the `sshpass -e ssh -q -t` should come before the `cmd`? – user2023 Mar 10 '21 at 07:11
Updated my answer. – Ivan Mar 10 '21 at 07:53

sinkmanu · Answer 2 · 2021-03-10T14:21:02.697

Another solution to get these values you need could be:

$ ssh user@foobar "bash -c \"grep -c '^processor' /proc/cpuinfo; free -g\"" | awk '{if (NR==1) {printf $1 " "} if ($1=="Mem:") { for (i=1; i<=NF; i++) printf $i " "}; if ($1=="Swap:") { for (i=1; i<=NF; i++) printf $i " "};}'
user@foobar's password:
4 Mem: 15 2 2 0 10 11 Swap: 15 0 15

It could be good if you tell us what are the command or the tools available in these "old" systems. I have only used to use the commands you have in your script (grep and awk).

For the multiple SSH connections, just introduce the command line into the for loop.

UPDATE: To handle the problem that if the machine does not have command nproc just add an if condition:

if ! command -v nproc &> /dev/null
then
        ssh user@foobar "bash -c \"grep -c '^processor' /proc/cpuinfo; free -g\"" | awk '{if (NR==1) {printf $1 " "} if ($1=="Mem:") { for (i=1; i<=NF; i++) printf $i " "}; if ($1=="Swap:") { for (i=1; i<=NF; i++) printf $i " "};}'

else
        ssh user@foobar "bash -c \"nproc ; free -g\"" | awk '{if (NR==1) {printf $1 " "} if ($1=="Mem:") { for (i=1; i<=NF; i++) printf $i " "}; if ($1=="Swap:") { for (i=1; i<=NF; i++) printf $i " "};}'
fi

Thanks for your answer sinkmanu, however the solution 1 in my post is a perfect working example as in parallel ssh connection. the only challenge i see to place condition like if `nproc` is not found then use `grep -c '^processor' /proc/cpuinfo`. there are no tools available that's the reason reinventing this way. — user2023, Mar 10 '21 at 08:32
Then you can handle that problem just with an IF condition. I have updated the answer — sinkmanu, Mar 10 '21 at 14:21
@ sinkmanu, thanks for the answer, however i would like the first approach as i stated in my POST with Parallelism with conditions and i don not want to open multiple ssh connection for single server to fetch the commands +1 for throwing another ideas. — user2023, Mar 10 '21 at 15:25

How to run ssh connection in parallel while running multiple commands Linux

2 Answers2