As the man page says,
The open command opens a file (or a directory or URL), just as if you had double-clicked the file's icon.
Ultimately, the application gets started by LaunchServices, but that's not important—what's important is that it's not a child of your shell, or Python script.
Also, the whole point of open is to open the app itself, so you don't have to dig into it and find the Unix executable file. If you already have that, and want to run it as a Unix executable… just run it:
xfoil = sp.Popen(['/Applications/Xfoil.app/Contents/MacOS/Xfoil'], stdin=sp.PIPE, stdout=sp.PIPE)
As it turns out, in this case, MacOS/Xfoil isn't even the right program; it's apparently some kind of wrapper around Resources/xfoil, which is the actual equivalent to what you get as /usr/local/bin/xfoil on linux. So you want to do this:
xfoil = sp.Popen(['/Applications/Xfoil.app/Contents/Resouces/xfoil'], stdin=sp.PIPE, stdout=sp.PIPE)
(Also, technically, your command line shouldn't even work at all; the -a specifies an application, not a Unix executable, and you're supposed to pass at least one file to open. But because LaunchServices can launch Unix executables as if they were applications, and open doesn't check that the arguments are valid, open -a /Applications/Xfoil.app/Contents/MacOS/Xfoil ends up doing effectively the same thing as open /Applications/Xfoil.app/Contents/MacOS/Xfoil.)
For the benefit of future readers, I'll include this information from the comments:
If you just write a line to stdin and then return from the function/fall off the end of the main script/etc., the Popen object will get garbage collected, closing both of its pipes. If xfoil hasn't finished running yet, it will get an error the next time it tries to write any output, and apparently it handles this by printing Fortran runtime error: end of file (to stderr?) and bailing. You need to call xfoil.wait() (or something else that implicitly waits) to prevent this from happening.