10

I know how to loop through all the files in a directory, for example:

for i in *
do
  <some command>
done

But I would like to go through all the files in a directory, including (particularly!) all the ones in the subdirectories. Is there a simple way of doing this?

Sebastian
  • 101
  • 1
  • 1
  • 5

3 Answers3

14

The find command is very useful for that kind of thing, provided you don't have white space or other special characters in the file names:

For example:

for i in $(find . -type f -print)
do
    stuff
done

The command generates path names relative from the start of the search (the first parameter).

As pointed out, this will fail if your filenames contain spaces or some other characters.

You can also use the -exec option which avoids the problem with spaces in file names. It executes the given command for each file found. The braces are a placeholder for the filename:

find . -type f -exec command {} \;
rghome
  • 8,529
  • 8
  • 43
  • 62
  • 7
    `find` is right, but iterating over its output like this fails for file names containing whitespace, newlines, or shell metacharacters. – chepner Aug 02 '18 at 13:36
6

find and xargs are great tools for recursively processing the contents of directories and sub-directories. For example

find . -type f -print0 | xargs -0 command

will run command on batches of files from the current directory and its sub-directories. The -print0 and -0 arguments avoid the usual problems with filenames that contain spaces, quotes or other metacharacters.

If command just takes one argument, you can limit the number of files passed to it with -L1.

find . -type f -print0 | xargs -0 -L1 command

And as suggested by alexgirao, xargs can also name arguments, using -I, which gives some flexibility if command takes options. -I implies -L1.

find . -type f -print0 | xargs -0 -Iarg command arg --option
Jon
  • 3,573
  • 2
  • 17
  • 24
  • 1
    nice answer, maybe you should also exemplify the use of `-I` flag, which implies `-L1` and allows for argument position, as in this example: `seq 10 | xargs -Iarg echo arg --more` – alexgirao Aug 02 '18 at 14:44
  • find also has its own `-exec` option which works pretty well. But it has a few foibles, like you have to remember to terminate the command with `\;` , but xargs is also useful in the way shown to safely run the tasks in parallel, and for detecting that any command has failed. – Gem Taylor Aug 02 '18 at 16:50
  • I'm a bit confused by this answer. Though helpful, it seems to involve some unecessary steps / parts. I've written this snippet of code (using this and other answers) which would appear to do what I want: `find . -type f | while read line; do echo $line # Or some other command for each file encountered; done`. Is there something I'm missing / potential pitfalls from doing this? – Sebastian Aug 03 '18 at 11:00
  • @Sebastian Yes, indeed, there are problems with using a `while read` loop. The worst one is that it won't work for filenames with, say, spaces or quotes in them. All that `-print0` stuff may look unnecessary, but over the years I've frequently been bitten by leaving it out. You would have thought I'd have learned by now :-) Also, if you can use `xargs` without `-L1` everything runs a lot quicker. – Jon Aug 03 '18 at 11:52
0
recurse() {
  path=$1
  If [ -d "$path" ] ; then
     for i in "$path/"*
     do
        recurse "$i"
     done
  elif [ -f "$path" ] ; then
     do-something
  fi
}

Call recurse and pass first positional parameter as directory path from where you want to start.

Ex: recurse /path

tripleee
  • 175,061
  • 34
  • 275
  • 318
Sumit
  • 852
  • 5
  • 6
  • The lack of quoting thoroughly breaks this otherwise reasonable reimplementation of `find`. See https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable – tripleee Aug 03 '18 at 04:55
  • Oh there's a dollar sign missing too. – tripleee Aug 03 '18 at 04:55