How to use a custom file-like object as subprocess stdout/stderr?

Question

Consider this code, where a subprocess.Popen is spawned. I'd like writes to the subprocess' stdout and stderr to go to my custom file-object's .write() method, however this isn't the case.

import subprocess

class Printer:

    def __init__(self):
        pass

    def write(self, chunk):
        print('Writing:', chunk)

    def fileno(self):
        return 0

    def close(self):
        return

proc = subprocess.Popen(['bash', '-c', 'echo Testing'], 
                        stdout=Printer(),
                        stderr=subprocess.STDOUT)
proc.wait()

Why is the .write() method not used, and what is the use of specifying a stdout= parameter in this case?

Here's an example where a custom file-like object is used as subprocess stdout/stderr: [Python subprocess get children's output to file and terminal?](http://stackoverflow.com/q/4984428/4279) — jfs, Jan 10 '14 at 21:54

score 4 · Accepted Answer · answered Jan 10 '14 at 02:25

4

According to the documentation:

stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None.

Using subprocess.PIPE:

proc = subprocess.Popen(['bash', '-c', 'echo Testing'], 
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT)
print('Writing:', proc.stdout.read())
# OR  print('Writing:', proc.stdout.read().decode())

answered Jan 10 '14 at 02:25

falsetru

357,413
63
732
636

That works, however I need the `stdout` and `stderr` separate (should've been in question), and also in realtime. Also, why does my example not work, isn't it a file-like object? – kiri Jan 10 '14 at 02:33
@minerz029, As I quoted the documentation, the subprocess.Popen does not accept file-like object, but only `PIPE`, `DEVNULL`, a file descriptor, a file object, `None`. – falsetru Jan 10 '14 at 02:34
@minerz029, `for line in proc.stdout: print('Writing', line.decode())` – falsetru Jan 10 '14 at 02:41
@minerz029, If you meant to write to stdout, the return value of the `fileno` should be `1`. Even if you changed to `1`, `write` method is not called. It is just written to stdout of current process. – falsetru Jan 10 '14 at 02:51

score 0 · Answer 2 · answered Nov 13 '20 at 22:16

Not directly. Maybe some future version of Python will support converting file-like objects into some sort of auto-populating pipe, but the crux of it is that the subprocess needs to have access to a file handle it can read without calling Python code somehow. This means it needs to be something that works at an OS level, meaning it is limited to a few possibilities:

Inheriting stdin from the parent process (which happens when stdin=None)
Taking input from a file (which happens when stdin is a file or an integer file handle)
Taking input from a pipe (which happens when stdin=subprocess.PIPE)

Using a file-like object means you're going to have to read the data yourself and then feed it through a pipe.

For example:

proc = subprocess.Popen(['sha256sum', '-'], stdin=subprocess.PIPE)
while True:
    chunk = filelike.read(BLOCK_SIZE)
    proc.stdin.write(chunk)
    if len(chunk) < BLOCK_SIZE:
        break
proc.wait()

How to use a custom file-like object as subprocess stdout/stderr?

2 Answers2

Linked

Related