If most of the time is spent in computing with the CPU (in other words, if the 10 seconds in ./startmerge are not spent on IO) then you want to limit the CPU time, so use the setrlimit(2) syscall with RLIMIT_CPU. In bash, you want to use its ulimit builtin (calling setrlimit syscall), see this answer, like
ulimit -t 10
before running startmerge.
If your startmerge is not CPU intensive, you want to kill it after 10 seconds, perhaps (see this)
startmerge & ; pidof_startmerge=$!
sleep 10; kill $pidof_startmerge
Better yet, make that a loop (e.g. loop ten times around a sleep 1)
If you have access to the C source code of startmerge and want to improve it, read first time(7) and signal(7). As Duck commented, consider alarm(2) or setitimer(2) or timer_create(2) and set up a signal handler -e.g. for SIGALRM with sigaction(2) (but don't forget that signal handlers are limited to calling async-signal-safe functions only; you could set a global volatile sig_atomic_t variable in them, and test that variable outside). If your startmerge has some event loop based upon some multiplexing syscall like poll(2) (or the old select(2)) it should even be simpler, and the Linux specific timerfd_create(2) could be very helpful.
You could trap SIGALRM in bash, see this.
Advanced Linux Programming has some chapters related to your question. Advanced Bash Programming Guide is also useful to read.