annotate MoinMoin/util/SubProcess.py @ 6110:4e911b751b5b

SubProcess: remove methods found in py27 base class
author Thomas Waldmann <tw AT waldmann-edv DOT de>
date Tue, 06 Sep 2016 03:20:27 +0200
parents 109f7b6471af
children 1fdd537e9d83
rev   line source
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
1 """
6110
4e911b751b5b SubProcess: remove methods found in py27 base class
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5732
diff changeset
2 Enhanced subprocess.Popen subclass, supporting .communicate() with timeout.
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
3
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
4 Sample usage:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
5 out, err = Popen(...).communicate(input, timeout=300)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
6 """
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
7
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
8 import os
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
9 import subprocess
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
10 import threading
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
11
6110
4e911b751b5b SubProcess: remove methods found in py27 base class
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5732
diff changeset
12 if not subprocess.mswindows:
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
13 import select
5406
0dd02729b3b5 SubProcess: fix imports, cosmetic other changes
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5404
diff changeset
14 import errno
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
15
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
16
6110
4e911b751b5b SubProcess: remove methods found in py27 base class
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5732
diff changeset
17 class Popen(subprocess.Popen):
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
18 def communicate(self, input=None, timeout=None):
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
19 """Interact with process: Send data to stdin. Read data from
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
20 stdout and stderr, until end-of-file is reached. Wait for
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
21 process to terminate. The optional input argument should be a
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
22 string to be sent to the child process, or None, if no data
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
23 should be sent to the child.
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
24
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
25 communicate() returns a tuple (stdout, stderr)."""
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
26
5465
3e6b8234861c partially revert 0dd02729b3b5 (self.timeout related changes caused AttributeError)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5406
diff changeset
27 self.timeout = timeout
3e6b8234861c partially revert 0dd02729b3b5 (self.timeout related changes caused AttributeError)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5406
diff changeset
28
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
29 # Optimization: If we are only using one pipe, or no pipe at
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
30 # all, using select() or threads is unnecessary.
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
31 if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
32 stdout = None
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
33 stderr = None
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
34 if self.stdin:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
35 if input:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
36 self._fo_write_no_intr(self.stdin, input)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
37 self.stdin.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
38 elif self.stdout:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
39 stdout = self._fo_read_no_intr(self.stdout)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
40 self.stdout.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
41 elif self.stderr:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
42 stderr = self._fo_read_no_intr(self.stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
43 self.stderr.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
44 self.wait()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
45 return (stdout, stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
46
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
47 return self._communicate(input)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
48
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
49 if subprocess.mswindows:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
50 def _communicate(self, input):
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
51 stdout = None # Return
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
52 stderr = None # Return
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
53
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
54 if self.stdout:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
55 stdout = []
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
56 stdout_thread = threading.Thread(target=self._readerthread,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
57 args=(self.stdout, stdout))
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
58 stdout_thread.setDaemon(True)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
59 stdout_thread.start()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
60 if self.stderr:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
61 stderr = []
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
62 stderr_thread = threading.Thread(target=self._readerthread,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
63 args=(self.stderr, stderr))
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
64 stderr_thread.setDaemon(True)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
65 stderr_thread.start()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
66
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
67 if self.stdin:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
68 if input is not None:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
69 self.stdin.write(input)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
70 self.stdin.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
71
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
72 if self.stdout:
5465
3e6b8234861c partially revert 0dd02729b3b5 (self.timeout related changes caused AttributeError)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5406
diff changeset
73 stdout_thread.join(self.timeout)
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
74 if self.stderr:
5465
3e6b8234861c partially revert 0dd02729b3b5 (self.timeout related changes caused AttributeError)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5406
diff changeset
75 stderr_thread.join(self.timeout)
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
76
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
77 # if the threads are still alive, that means the thread join timed out
5404
bf05a6ae61ea win32-specific part of SubProcess: use correct attributes (fixes calling of external xapian index filters)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5212
diff changeset
78 timed_out = (self.stdout and stdout_thread.isAlive() or
bf05a6ae61ea win32-specific part of SubProcess: use correct attributes (fixes calling of external xapian index filters)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5212
diff changeset
79 self.stderr and stderr_thread.isAlive())
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
80 if timed_out:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
81 self.kill()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
82 else:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
83 self.wait()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
84
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
85 # All data exchanged. Translate lists into strings.
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
86 if stdout is not None:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
87 stdout = stdout[0]
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
88 if stderr is not None:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
89 stderr = stderr[0]
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
90
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
91 # Translate newlines, if requested. We cannot let the file
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
92 # object do the translation: It is based on stdio, which is
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
93 # impossible to combine with select (unless forcing no
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
94 # buffering).
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
95 if self.universal_newlines and hasattr(file, 'newlines'):
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
96 if stdout:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
97 stdout = self._translate_newlines(stdout)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
98 if stderr:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
99 stderr = self._translate_newlines(stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
100
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
101 return (stdout, stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
102
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
103 else: # POSIX
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
104 def _communicate(self, input):
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
105 timed_out = False
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
106 read_set = []
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
107 write_set = []
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
108 stdout = None # Return
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
109 stderr = None # Return
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
110
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
111 if self.stdin:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
112 # Flush stdio buffer. This might block, if the user has
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
113 # been writing to .stdin in an uncontrolled fashion.
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
114 self.stdin.flush()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
115 if input:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
116 write_set.append(self.stdin)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
117 else:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
118 self.stdin.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
119 if self.stdout:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
120 read_set.append(self.stdout)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
121 stdout = []
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
122 if self.stderr:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
123 read_set.append(self.stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
124 stderr = []
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
125
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
126 input_offset = 0
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
127 while read_set or write_set:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
128 try:
5465
3e6b8234861c partially revert 0dd02729b3b5 (self.timeout related changes caused AttributeError)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5406
diff changeset
129 rlist, wlist, xlist = select.select(read_set, write_set, [], self.timeout)
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
130 except select.error, e:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
131 if e.args[0] == errno.EINTR:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
132 continue
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
133 raise
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
134
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
135 timed_out = not (rlist or wlist or xlist)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
136 if timed_out:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
137 break
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
138
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
139 if self.stdin in wlist:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
140 # When select has indicated that the file is writable,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
141 # we can write up to PIPE_BUF bytes without risk
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
142 # blocking. POSIX defines PIPE_BUF >= 512
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
143 chunk = input[input_offset:input_offset + 512]
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
144 bytes_written = os.write(self.stdin.fileno(), chunk)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
145 input_offset += bytes_written
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
146 if input_offset >= len(input):
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
147 self.stdin.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
148 write_set.remove(self.stdin)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
149
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
150 if self.stdout in rlist:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
151 data = os.read(self.stdout.fileno(), 1024)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
152 if data == "":
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
153 self.stdout.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
154 read_set.remove(self.stdout)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
155 stdout.append(data)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
156
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
157 if self.stderr in rlist:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
158 data = os.read(self.stderr.fileno(), 1024)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
159 if data == "":
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
160 self.stderr.close()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
161 read_set.remove(self.stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
162 stderr.append(data)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
163
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
164 # All data exchanged. Translate lists into strings.
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
165 if stdout is not None:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
166 stdout = ''.join(stdout)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
167 if stderr is not None:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
168 stderr = ''.join(stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
169
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
170 # Translate newlines, if requested. We cannot let the file
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
171 # object do the translation: It is based on stdio, which is
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
172 # impossible to combine with select (unless forcing no
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
173 # buffering).
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
174 if self.universal_newlines and hasattr(file, 'newlines'):
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
175 if stdout:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
176 stdout = self._translate_newlines(stdout)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
177 if stderr:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
178 stderr = self._translate_newlines(stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
179
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
180 if timed_out:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
181 self.kill()
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
182 else:
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
183 self.wait()
5732
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
184
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
185 # make sure all files are closed:
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
186 for f in [self.stdin, self.stdout, self.stderr]:
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
187 try:
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
188 f.close()
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
189 except:
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
190 pass
109f7b6471af SubProcess: make sure files are closed at the end (experimental change)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 5465
diff changeset
191
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
192 return (stdout, stderr)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
193
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
194
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
195 def exec_cmd(cmd, input=None, timeout=None):
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
196 p = Popen(cmd, shell=True,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
197 close_fds=not subprocess.mswindows,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
198 bufsize=1024,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
199 stdin=subprocess.PIPE,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
200 stdout=subprocess.PIPE,
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
201 stderr=subprocess.PIPE)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
202 data, errors = p.communicate(input, timeout=timeout)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
203 return data, errors, p.returncode
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
204
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
205
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
206 if __name__ == '__main__':
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
207 print exec_cmd("python", "import time ; time.sleep(20) ; print 'never!' ;", timeout=10)
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
208 print exec_cmd("python", "import time ; time.sleep(20) ; print '20s gone' ;")