Bash: wait with timeout

Question

In a Bash script, I would like to do something like:

app1 &
pidApp1=$!
app2 &
pidApp2=$1

timeout 60 wait $pidApp1 $pidApp2
kill -9 $pidApp1 $pidApp2

I.e., launch two applications in the background, and give them 60 seconds to complete their work. Then, if they don't finish within that interval, kill them.

Unfortunately, the above does not work, since timeout is an executable, while wait is a shell command. I tried changing it to:

timeout 60 bash -c wait $pidApp1 $pidApp2

But this still does not work, since wait can only be called on a PID launched within the same shell.

Any ideas?

Answer 1

Both your example and the accepted answer are overly complicated, why do you not only use timeout since that is exactly its use case? The timeout command even has an inbuilt option (-k) to send SIGKILL after sending the initial signal to terminate the command (SIGTERM by default) if the command is still running after sending the initial signal (see man timeout).

If the script doesn't necessarily require to wait and resume control flow after waiting it's simply a matter of

timeout -k 60s 60s app1 &
timeout -k 60s 60s app2 &
# [...]

If it does, however, that's just as easy by saving the timeout PIDs instead:

pids=()
timeout -k 60s 60s app1 &
pids+=($!)
timeout -k 60s 60s app2 &
pids+=($!)
wait "${pids[@]}"
# [...]

E.g.

$ cat t.sh
#!/bin/bash

echo "$(date +%H:%M:%S): start"
pids=()
timeout 10 bash -c 'sleep 5; echo "$(date +%H:%M:%S): job 1 terminated successfully"' &
pids+=($!)
timeout 2 bash -c 'sleep 5; echo "$(date +%H:%M:%S): job 2 terminated successfully"' &
pids+=($!)
wait "${pids[@]}"
echo "$(date +%H:%M:%S): done waiting. both jobs terminated on their own or via timeout; resuming script"

.

$ ./t.sh
08:59:42: start
08:59:47: job 1 terminated successfully
08:59:47: done waiting. both jobs terminated on their own or via timeout; resuming script

Answer 2

Write the PIDs to files and start the apps like this:

pidFile=...
( app ; rm $pidFile ; ) &
pid=$!
echo $pid > $pidFile
( sleep 60 ; if [[ -e $pidFile ]]; then killChildrenOf $pid ; fi ; ) &
killerPid=$!

wait $pid
kill $killerPid

That would create another process that sleeps for the timeout and kills the process if it hasn't completed so far.

If the process completes faster, the PID file is deleted and the killer process is terminated.

killChildrenOf is a script that fetches all processes and kills all children of a certain PID. See the answers of this question for different ways to implement this functionality: Best way to kill all child processes

If you want to step outside of BASH, you could write PIDs and timeouts into a directory and watch that directory. Every minute or so, read the entries and check which processes are still around and whether they have timed out.

EDIT If you want to know whether the process has died successfully, you can use kill -0 $pid

EDIT2 Or you can try process groups. kevinarpe said: To get PGID for a PID(146322):

ps -fjww -p 146322 | tail -n 1 | awk '{ print $4 }'

In my case: 145974. Then PGID can be used with a special option of kill to terminate all processes in a group: kill -- -145974

Answer 3

Here's a simplified version of Aaron Digulla's answer, which uses the kill -0 trick that Aaron Digulla leaves in a comment:

app &
pidApp=$!
( sleep 60 ; echo 'timeout'; kill $pidApp ) &
killerPid=$!

wait $pidApp
kill -0 $killerPid && kill $killerPid

In my case, I wanted to be both set -e -x safe and return the status code, so I used:

set -e -x
app &
pidApp=$!
( sleep 45 ; echo 'timeout'; kill $pidApp ) &
killerPid=$!

wait $pidApp
status=$?
(kill -0 $killerPid && kill $killerPid) || true

exit $status

An exit status of 143 indicates SIGTERM, almost certainly from our timeout.

Answer 4

I wrote a bash function that will wait until PIDs finished or until timeout, that return non zero if timeout exceeded and print all the PIDs not finisheds.

function wait_timeout {
  local limit=${@:1:1}
  local pids=${@:2}
  local count=0
  while true
  do
    local have_to_wait=false
    for pid in ${pids}; do
      if kill -0 ${pid} &>/dev/null; then
        have_to_wait=true
      else
        pids=`echo ${pids} | sed -e "s/${pid}//g"`
      fi
    done
    if ${have_to_wait} && (( $count < $limit )); then
      count=$(( count + 1 ))
      sleep 1
    else
      echo ${pids}
      return 1
    fi
  done   
  return 0
}

To use this is just wait_timeout $timeout $PID1 $PID2 ...

Answer 5

To put in my 2c, we can boild down Teixeira's solution to:

try_wait() {
    # Usage: [PID]...
    for ((i = 0; i < $#; i += 1)); do
        kill -0 $@ && sleep 0.001 || return 0
    done
    return 1 # timeout or no PIDs
} &>/dev/null

Bash's sleep accepts fractional seconds, and 0.001s = 1 ms = 1 KHz = plenty of time. However, UNIX has no loopholes when it comes to files and processes. try_wait accomplishes very little.

$ cat &
[1] 16574
$ try_wait %1 && echo 'exited' || echo 'timeout'
timeout
$ kill %1
$ try_wait %1 && echo 'exited' || echo 'timeout'
exited

We have to answer some hard questions to get further.

Why has wait no timeout parameter? Maybe because the timeout, kill -0, wait and wait -n commands can tell the machine more precisely what we want.

Why is wait builtin to Bash in the first place, so that timeout wait PID is not working? Maybe only so Bash can implement proper signal handling.

Consider:

$ timeout 30s cat &
[1] 6680
$ jobs
[1]+    Running   timeout 30s cat &
$ kill -0 %1 && echo 'running'
running
$ # now meditate a bit and then...
$ kill -0 %1 && echo 'running' || echo 'vanished'
bash: kill: (NNN) - No such process
vanished

Whether in the material world or in machines, as we require some ground on which to run, we require some ground on which to wait too.

• When kill fails you hardly know why. Unless you wrote the process, or its manual names the circumstances, there is no way to determine a reasonable timeout value.

• When you have written the process, you can implement a proper TERM handler or even respond to "Auf Wiedersehen!" send to it through a named pipe. Then you have some ground even for a spell like try_wait :-)

Answer 6

You could use the timeout of the 'read' internal command.

The following will kill unterminated jobs and display the names of the completed jobs after at most 60 seconds:

( (job1; echo -n "job1 ")& (job2; echo -n "job2 ")&) | (read -t 60 -a jobarr; echo ${jobarr[*]} ${#jobarr[*]} )

It works by making a sub shell containing all the background jobs. The output of this sub shell is read into a bash array variable, which can be used as desired (in this example by printing the array + element count).

Be sure to reference the ${jobarr} in the same sub shell as the read command (hence the parenthesis), otherwise ${jobarr} will be empty.

All sub shells will automatically be muted (not killed) after the read command terminates. You have to kill them you self.

Answer 7

app1 &
app2 &
sleep 60 &

wait -n

Answer 8

Yet another timeout bash's script

Running many subprocess with an overall timeout. Using recent bash features, I wrote this:

#!/bin/bash
maxTime=5.0 jobs=() pids=() cnt=1 Started=${EPOCHREALTIME/.}
if [[ $1 == -m ]] ;then maxTime=$2; shift 2; fi

for cmd ;do  # $cmd is unquoted in order to use strings as command + args
    $cmd &
    jobs[$!]=$cnt pids[cnt++]=$!
done

printf -v endTime %.6f $maxTime
endTime=$(( Started + 10#${endTime/.} ))
exec {pio}<> <(:) # Pseudo FD for "builtin sleep" by using "read -t" 
while ((${#jobs[@]})) && (( ${EPOCHREALTIME/.} < endTime ));do
    for cnt in ${jobs[@]};do
        if ! jobs $cnt &>/dev/null;then
            Elap=00000$(( ${EPOCHREALTIME/.} - Started ))
            printf 'Job %d (%d) ended after %.4f secs.\n' \
                   $cnt ${pids[cnt]} ${Elap::-6}.${Elap: -6}
            unset jobs[${pids[cnt]}] pids[cnt]
        fi
    done
    read -ru $pio -t .02 _
done
if ((${#jobs[@]})) ;then
    Elap=00000$(( ${EPOCHREALTIME/.} - Started ))
    for cnt in ${jobs[@]};do
        printf 'Job %d (%d) killed after %.4f secs.\n' \
               $cnt ${pids[cnt]} ${Elap::-6}.${Elap: -6}
    done
    kill ${pids[@]}
fi

Sample run:

• Commands with argument could be submited as strings
• -m switch let you choose a float as max time in seconds.

$ ./execTimeout.sh -m 2.3 "sleep 1" 'sleep 2' sleep\ {3,4}  'cat /dev/tty'
Job 1 (460668) ended after 1.0223 secs.
Job 2 (460669) ended after 2.0424 secs.
Job 3 (460670) killed after 2.3100 secs.
Job 4 (460671) killed after 2.3100 secs.
Job 5 (460672) killed after 2.3100 secs.

For testing this, I wrote this script that

• choose random duratiopn between 1.0000 and 9.9999 seconds
• for output random number of line between 0 and 8. (they could not ouptut anything).
• lines output contain process id ($$), number of line left to print and total duration in seconds.

#!/bin/bash

tslp=$RANDOM lnes=${RANDOM: -1}
printf -v tslp %.6f ${tslp::1}.${tslp:1}
slp=00$((${tslp/.}/($lnes?$lnes:1)))
printf -v slp %.6f ${slp::-6}.${slp: -6}
# echo >&2 Slp $lnes x $slp == $tslp
exec {dummy}<> <(: -O)
while read -rt $slp -u $dummy; ((--lnes>0)); do
    echo $$ $lnes $tslp
done

Running this script 5 times in once, with a timeout of 5.0 seconds:

$ ./execTimeout.sh -m 5.0 ./tstscript.sh{,,,,}
2869814 6 2.416700
2869815 5 3.645000
2869814 5 2.416700
2869814 4 2.416700
2869815 4 3.645000
2869814 3 2.416700
2869813 5 8.414000
2869812 1 3.408000
2869814 2 2.416700
2869815 3 3.645000
2869814 1 2.416700
2869815 2 3.645000
Job 3 (2869814) ended after 2.4511 secs.
2869813 4 8.414000
2869815 1 3.645000
Job 1 (2869812) ended after 3.4518 secs.
Job 4 (2869815) ended after 3.6757 secs.
2869813 3 8.414000
Job 2 (2869813) killed after 5.0159 secs.
Job 5 (2869816) killed after 5.0159 secs.

Answer 9

There are some processes that do not work well when invoked from timeout. I came into an issue where needed to put a timeout catch around a qemu instance and if you invoke

timeout 900 qemu 

it will always hang.

My solution

./qemu_cmd &
qemuPid=$!
timeout 900 tail --pid=$qemuPid -f /dev/null
ret=$?
if [ "$ret" != "0" ]; then
   allpids=()
   descendent_pids $tracePid
   for pids in ${allpids[@]};do
      kill -9 $pids
   done
fi

descendent_pids(){
   allpids=("${allpids[@]}" $1)
   pids=$(pgrep -P $1)
   for pid in $pids; do
      descendent_pids $pid
   done
}

Also of note that timeout will not always kill descendant processes depending on how sophisticated the cmd that you are spawning from the timeout.

Answer 10

Better late than never, a solution using wait without polling (still a loop though), yet still stops as soon as possible.

app1 &
pidApp1=$!
app2 &
pidApp2=$!

# timeout 60 wait $pidApp1 $pidApp2
declare -A pidApps=( [$pidApp1]=running [$pidApp2]=running )
{ sleep 60; echo "stop"; } | read &
pidTmout=$!
while [[ ${#pidApps[@]} -gt 0 ]]; do
    wait -np pidStop
    [[ $pidStop == $pidTmout ]] && break
    unset pidApps[$pidStop]
done
[[ ${pidApps[$pidApp1]} == running ]] && kill -9 $pidApp1
[[ ${pidApps[$pidApp2]} == running ]] && kill -9 $pidApp2