1

I have several different inputs to a Python script, which I invoke in the form:

./myscript.py myfile1
./myscript.py myfile2
./myscript.py myfile3
...

I can profile the code on a per-function basis for any one input file using python -m cProfile -s ./myscript.py myfile1. Unfortunately, depending on the input file, the time is spent in completely different parts of the code.

The rough ideas I have for profiling the code for all inputs is to either (1) quick and dirty, write a bash script to call python -m cProfile -s ./myscript.py myfile for each myfile, and parse the output or (2) parse the cProfile results within python itself.

How can I profile myscript.py for all of my input files, and average over the results, so I know where the hotspots are in general?

Douglas B. Staple
  • 10,510
  • 8
  • 31
  • 58
  • I think it's better not to look in general, but to look in specific. Like each input might be spending 90% of its time doing something different from the other inputs. Mushing them together makes none of them stand out, but optimizing them separately makes you concentrate on each one. [*This is the method I use.*](http://stackoverflow.com/a/4299378/23771) – Mike Dunlavey Jul 22 '14 at 18:41
  • By in general, I meant for all the different inputs. For sure line-by-line profiling is also necessary. And using stack traces isn't a bad idea. Maybe I can get poor man's profiler to do it... – Douglas B. Staple Jul 22 '14 at 18:47

1 Answers1

1

This was a long time ago, but the solution I ended up taking was to write quick and very-dirty scripts using a combination of python cProfile, awk, grep, and bash. The first (main) script indexed the files and called the other scripts, a second script ran python cProfile on (one) input file and formatted the output for easy parsing, and a third script combined the results.

Douglas B. Staple
  • 10,510
  • 8
  • 31
  • 58
  • Your question said "so I know where the hotspots are in general". [*Here is*](http://stackoverflow.com/a/4299378/23771) the method many people use. It is more concerned with finding worthwhile things to eliminate than with measuring (which doesn't tell you what to remove, only the result of doing so). – Mike Dunlavey Jun 30 '16 at 16:05