Iterate through linux sort output in python

Question

I am having trouble finding a solution to utilize linux sort command as an input to my python script.

For example I would like to iterate through the result of sort -mk1 <(cat file1.txt) <(cat file2.txt))

Normally I would use Popen and iterate through it using next and stdout.readline(), something like:

import os
import subprocess

class Reader():
    def __init__(self):
        self.proc = subprocess.Popen(['sort -mk1', '<(', 'cat file1.txt', ')', '<(', 'cat file2.txt', ')'], stdout=subprocess.PIPE)

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            line = self.proc.stdout.readline()
            if not line:
                raise StopIteration
            return line


p = Reader()
for line in p:
    # only print certain lines based on some filter 

With the above, I would get an error: No such file or directory: 'sort -mk1'

After doing some research, I guess I cant use Popen, and have to use os.execl to utilize bin/bash

So now I try below:

import os
import subprocess

class Reader():
    def __init__(self):
        self.proc = os.execl('/bin/bash', '/bin/bash', '-c', 'set -o pipefail; sort -mk1 <(cat file1.txt) <(cat file2.txt)')

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            line = self.proc.stdout.readline()
            if not line:
                raise StopIteration
            return line


p = Reader()
for line in p:
    # only print certain lines based on some filter 

The problem with this is that it actually prints all the lines right away. I guess one solution is to just pipe its results to a file, then in python I iterate through that file. But I dont really want to save it to a file then filter it, seems unneccessary. Yes I can use other linux commands such as awk, but I would like to use python for further processing.

So questions are:

1. Is there a way to make solution one with Popen to work?
2. How can I iterate through the output of sort using the second solution?

Answer 1

If you want to use shell features, you have to use shell=True. If you want to use Bash features, you have to make sure the shell you run is Bash.

        self.proc = subprocess.Popen(
            'sort -mk1 <(cat file1.txt) <(cat file2.txt)',
            stdout=subprocess.PIPE,
            shell=True,
            executable='/bin/bash')

Notice how with shell=True the first argument to Popen and friends is a single string (and vice versa; if you don't have shell=True you have to parse the command line into tokens yourself).

Of course, the cats are useless but if you replace them with something which the shell performs easily and elegantly and which you cannot easily replace with native Python code, this is probably the way to go.

In brief, <(command) is a Bash process substitution; the shell will run command in a subprocess, and replace the argument with the device name of the open file handle where the process generates its output. So sort will see something like

sort -mk /dev/fd/63 /dev/fd/64

where /dev/fd/63 is a pipe where the first command's output is available, and /dev/fd/64 is the read end of the other command's standard output.

Answer 2

Quite a lot of problem in your scripts.

First, your Popen won't work because of several reasons:

1. First argument is supposed to be the command to run, and you passed sort -mk and there is no such file. You should simply pass sort, and pass -mk as arguments.
2. Process substituion <( command ) is something handled by the shell, for which it is doing something like running a command, create a FIFO and substitute it as the name of the FIFO. Passing these directly to sort is not going to work. sort will probably just treat <( as a filename.

Your second way using os.exec* won't work either because os.exec* is going to replace your current process. Hence it will never continue to next statement in your Python script.

In your case, there seems no reason using process substitution. Why can't you simply do somethng like subprocess.Popen(['sort', '-mk', 'filename1', 'filename2']) ?

Answer 3

I do not understand why you are doing sort -mk1 $(cat file), sort can operate on file. look at check_output. That will make your life simple

output=subprocess.check_output('ls')
for line in output:
    print(line)

you will, of course, have to deal with the exceptions, the man page has the details