Linux FIFO not returning EOF when I expect it to

Question

Let us consider the following Python code, to be executed by cpython on a Linux system (warning: it will try to create or overwrite files in /tmp/first, /tmp/second and /tmp/third).

#!/usr/bin/env python2
# -*- coding: utf-8 -*-

import subprocess
import os
import sys
import threading

class ThreadizedPopen(threading.Thread):

    def __init__(self, command, stdin_name, stdout_name):
        super(ThreadizedPopen, self).__init__()
        self.command = command
        self.stdin_name = stdin_name
        self.stdout_name = stdout_name
        self.returncode = None

    def run(self):
        with open(self.stdin_name, 'rb') as fin:
            with open(self.stdout_name, 'wb') as fout:
                popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None)
                popen.communicate()
                self.returncode = popen.returncode

def main():
    os.system('mkfifo /tmp/first')
    os.system('mkfifo /tmp/second')
    os.system('mkfifo /tmp/third')

    popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
    popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
    popen1.start()
    popen2.start()
    with open('/tmp/third') as fin:
        print fin.read()
    popen1.join()
    popen2.join()

if __name__ == '__main__':
    main()

I execute it then, on another shell, I write something in /tmp/first (say with echo test > /tmp/first). I would expect the Python program to quickly exit and print the same thing I fed to the first FIFO.

In theory it should happen that the string I wrote in /tmp/first gets copied over by the two cat processes spawned by my program to the other two FIFOs and then picked up by the main Python program to be wrote on its stdout. As soon as every cat process finished, it should close its end of the writing FIFO, making the corresponding reading end return EOF and triggering the termination of the following cat process. Looking at the program with strace reveals that the test string is copied correctly through all the three FIFOs and is read by the main Python program. The first FIFO is also correctly closed (and the first cat process exits, together with its manager Python thread). However the second cat process is stuck in a read() call, expecting data from its reading FIFO.

I do not understand why this happens. From the pipe(t) man page (which, I understand, covers also this kind of FIFOs) it seems that a read on a FIFO is returned EOF as soon as the writing end (and all its duplicates) are closed. According to strace this appears to be the trace (in particular, the cat process is dead, thus all its file descriptors are closed; its managing thread has closed its descriptors as well, I can see it in the strace output).

Can you suggest me why that happens? I can post the strace output if it can be useful.

jDo · Accepted Answer · 2016-03-11T16:00:54.837

I found this question and simply added close_fds=True to your subprocess call. Your code now reads:

#!/usr/bin/env python2
# -*- coding: utf-8 -*-

import subprocess
import os
import sys
import threading

class ThreadizedPopen(threading.Thread):

    def __init__(self, command, stdin_name, stdout_name):
        super(ThreadizedPopen, self).__init__()
        self.command = command
        self.stdin_name = stdin_name
        self.stdout_name = stdout_name
        self.returncode = None

    def run(self):
        with open(self.stdin_name, 'rb') as fin:
            with open(self.stdout_name, 'wb') as fout:
                popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None, close_fds=True)
                popen.communicate()
                self.returncode = popen.returncode

def main():
    os.system('mkfifo /tmp/first')
    os.system('mkfifo /tmp/second')
    os.system('mkfifo /tmp/third')

    popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
    popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
    popen1.start()
    popen2.start()
    with open('/tmp/third') as fin:
        print fin.read()
    popen1.join()
    popen2.join()

if __name__ == '__main__':
    main()

I placed your code in a script called fifo_issue.py and ran it in a terminal. The script was idling as you'd expect (ignore mkfifo: cannot create fifo):

$ python fifo_issue.py 
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists

Then, in a second terminal, I typed:

$ echo "I was echoed to /tmp/first!" > /tmp/first

Back to the first terminal that's still running your idling threads:

$ python fifo_issue.py 
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists
I was echoed to /tmp/first!

After which python exited correctly

Thanks, it works. I do not really understand why my previous code did not work, so I will accept the answer only if in some reasonable time nobody is able to give more insight on what is actually happening. — Giovanni Mascellani, Mar 12 '16 at 09:06
@giomasce Yeah, do that. I'm just providing the full, working example for people to copy/paste if they're interested - I'm not adding much information. I should have commented instead, I guess. Anyway, we need a Unix ninja or a very patient doc reader to elaborate on what's going on under the hood. — jDo, Mar 12 '16 at 20:08

Linux FIFO not returning EOF when I expect it to

1 Answers1