python - subprocess's Popen closes stdout/stderr filedescriptors used in another thread when Popen errors -

an internal library heavily uses subprocess.popen() started failing automated tests when upgraded python 2.7.3 python 2.7.5. library used in threaded environment. after debugging issue, able create short python script demonstrates error seen in failing tests.

this script (called "threadedsubprocess.py"):

import time import threading import subprocess  def subprocesscall():     p = subprocess.popen(         ['ls', '-l'],         stdin=subprocess.pipe,         stdout=subprocess.pipe,         stderr=subprocess.pipe,         )     time.sleep(2) # simulate popen call takes time complete.     out, err = p.communicate()     print 'succeeding command in thread:', threading.current_thread().ident  def failingsubprocesscall():     try:         p = subprocess.popen(             ['thiscommandsurelydoesnotexist'],             stdin=subprocess.pipe,             stdout=subprocess.pipe,             stderr=subprocess.pipe,             )     except exception e:         print 'failing command:', e, 'in thread:', threading.current_thread().ident  print 'main thread is:', threading.current_thread().ident  subprocesscall_thread = threading.thread(target=subprocesscall) subprocesscall_thread.start() failingsubprocesscall() subprocesscall_thread.join() 

note: script not exit ioerror when ran python 2.7.3. fail @ least 50% of times when ran python 2.7.5 (both on same ubuntu 12.04 64-bit vm).

the error raised on python 2.7.5 this:

/opt/python/2.7.5/bin/python ./threadedsubprocess.py  main thread is: 139899583563520 failing command: [errno 2] no such file or directory 139899583563520 exception in thread thread-1: traceback (most recent call last):   file "/opt/python/2.7.5/lib/python2.7/threading.py", line 808, in __bootstrap_inner     self.run()   file "/opt/python/2.7.5/lib/python2.7/threading.py", line 761, in run     self.__target(*self.__args, **self.__kwargs)   file "./threadedsubprocess.py", line 13, in subprocesscall     out, err = p.communicate()   file "/opt/python/2.7.5/lib/python2.7/subprocess.py", line 806, in communicate     return self._communicate(input)   file "/opt/python/2.7.5/lib/python2.7/subprocess.py", line 1379, in _communicate     self.stdin.close() ioerror: [errno 9] bad file descriptor  close failed in file object destructor: ioerror: [errno 9] bad file descriptor 

when comparing subprocess module python 2.7.3 python 2.7.5 see popen()'s __init__() call indeed explicitly closes stdin, stdout , stderr file descriptors in case executing command somehow fails. seems intended fix applied in python 2.7.4 prevent leaking file descriptors (http://hg.python.org/cpython/file/ab05e7dd2788/misc/news#l629).

the diff between python 2.7.3 , python 2.7.5 seems relevant issue in popen __init__():

@@ -671,12 +702,33 @@           c2pread, c2pwrite,           errread, errwrite) = self._get_handles(stdin, stdout, stderr)  -        self._execute_child(args, executable, preexec_fn, close_fds, -                            cwd, env, universal_newlines, -                            startupinfo, creationflags, shell, -                            p2cread, p2cwrite, -                            c2pread, c2pwrite, -                            errread, errwrite) +        try: +            self._execute_child(args, executable, preexec_fn, close_fds, +                                cwd, env, universal_newlines, +                                startupinfo, creationflags, shell, +                                p2cread, p2cwrite, +                                c2pread, c2pwrite, +                                errread, errwrite) +        except exception: +            # preserve original exception in case os.close raises. +            exc_type, exc_value, exc_trace = sys.exc_info() + +            to_close = [] +            # close pipes created. +            if stdin == pipe: +                to_close.extend((p2cread, p2cwrite)) +            if stdout == pipe: +                to_close.extend((c2pread, c2pwrite)) +            if stderr == pipe: +                to_close.extend((errread, errwrite)) + +            fd in to_close: +                try: +                    os.close(fd) +                except environmenterror: +                    pass + +            raise exc_type, exc_value, exc_trace 

i think have 3 questions:

1) true should principally possible use subprocess.popen, pipe stdin, stdout , stderr, in threaded environment?

2) how prevent file descriptors stdin, stdout , stderr closed when popen() fails in 1 of threads?

3) doing wrong here?

i answer questions with:

1. yes.
2. you shouldn't have to.
3. no.

the error occurs indeed in python 2.7.4 well.

i think bug in library code. if add lock in program , make sure 2 calls subprocess.popen executed atomically, error not occur.

@@ -1,32 +1,40 @@  import time  import threading  import subprocess  +lock = threading.lock() +  def subprocesscall(): +    lock.acquire()      p = subprocess.popen(          ['ls', '-l'],          stdin=subprocess.pipe,          stdout=subprocess.pipe,          stderr=subprocess.pipe,          ) +    lock.release()      time.sleep(2) # simulate popen call takes time complete.      out, err = p.communicate()      print 'succeeding command in thread:', threading.current_thread().ident   def failingsubprocesscall():      try: +        lock.acquire()          p = subprocess.popen(              ['thiscommandsurelydoesnotexist'],              stdin=subprocess.pipe,              stdout=subprocess.pipe,              stderr=subprocess.pipe,              )      except exception e:          print 'failing command:', e, 'in thread:', threading.current_thread().ident +    finally: +        lock.release() +   print 'main thread is:', threading.current_thread().ident   subprocesscall_thread = threading.thread(target=subprocesscall)  subprocesscall_thread.start()  failingsubprocesscall()  subprocesscall_thread.join() 

this means due data race in implementation of popen. risk guess: bug may in implementation of pipe_cloexec, called _get_handles, (in 2.7.4) is:

def pipe_cloexec(self):     """create pipe fds set cloexec."""     # pipes' fds set cloexec default because don't want them     # inherited other subprocesses: cloexec flag removed     # child's fds _dup2(), between fork() , exec().     # not atomic: need pipe2() syscall that.     r, w = os.pipe()     self._set_cloexec_flag(r)     self._set_cloexec_flag(w)     return r, w 

and comment warns explicitly not being atomic... causes data race but, without experimentation, don't know if it's causes problem.