annotate MoinMoin/util/SubProcess.py @ 6111:1fdd537e9d83

SubProcess: reimplement exec_cmd subclassing Popen and overriding some methods isn't pretty. the code we have was written for py 2.4 or so and the py 2.7 Popen looked quite different. this way with the timer should be less problematic.
author Thomas Waldmann <tw AT waldmann-edv DOT de>
date Tue, 06 Sep 2016 04:39:28 +0200
parents 4e911b751b5b
children
rev   line source
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
1 """
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
2 Execute a shell command with timeout.
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
3
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
4 @copyright: 2016 Thomas Waldmann <tw@waldmann-edv.de>
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
5 @license: GNU GPL, see COPYING for details.
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
6 """
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
7
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
8 import os
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
9 import signal
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
10 import subprocess
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
11 from threading import Timer
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
12
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
13
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
14 def exec_cmd(cmd, stdin=None, timeout=None):
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
15 """
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
16 Execute a shell <cmd>, send <stdin> to it, kill it after <timeout> if it
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
17 is still running. Return stdout, stderr, rc.
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
18 """
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
19 def preexec_fn():
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
20 if not subprocess.mswindows:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
21 os.setsid() # start a new session
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
22
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
23 def kill_it(p):
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
24 if not subprocess.mswindows:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
25 # kills all the processes of the session,
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
26 # includes the shell + process started by shell
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
27 os.killpg(p.pid, signal.SIGKILL)
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
28 else:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
29 p.kill()
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
30
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
31 p = subprocess.Popen(cmd, shell=True,
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
32 close_fds=not subprocess.mswindows,
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
33 bufsize=1024,
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
34 preexec_fn=preexec_fn,
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
35 stdin=subprocess.PIPE,
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
36 stdout=subprocess.PIPE,
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
37 stderr=subprocess.PIPE)
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
38 if timeout is None:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
39 stdout, stderr = p.communicate(stdin)
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
40 else:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
41 timer = Timer(timeout, kill_it, [p, ])
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
42 try:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
43 timer.start()
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
44 stdout, stderr = p.communicate(stdin)
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
45 finally:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
46 timer.cancel()
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
47 return stdout, stderr, p.returncode
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
48
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
49
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
50 if __name__ == '__main__':
6111
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
51 # expected output:
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
52 # ('', '', -9) --> no stdout, stderr output, killed by SIGKILL (signal 9)
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
53 # ('20s gone\n', '', 0) --> some output on stdout, no stderr, rc = 0 (did not get killed)
1fdd537e9d83 SubProcess: reimplement exec_cmd
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 6110
diff changeset
54 print exec_cmd("python", "import time ; time.sleep(20) ; print 'timeout does not work!' ;", timeout=10)
5212
ce70252a3e90 Xapian indexing: fix deadlocks, new MoinMoin.util.SubProcess module (see below)
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents:
diff changeset
55 print exec_cmd("python", "import time ; time.sleep(20) ; print '20s gone' ;")