diff options
Diffstat (limited to 'build/blog/2015-05-05-graceful-handling-of-sigint-when-using-pythons-multiprocessingprocess.html')
| -rw-r--r-- | build/blog/2015-05-05-graceful-handling-of-sigint-when-using-pythons-multiprocessingprocess.html | 106 |
1 files changed, 106 insertions, 0 deletions
diff --git a/build/blog/2015-05-05-graceful-handling-of-sigint-when-using-pythons-multiprocessingprocess.html b/build/blog/2015-05-05-graceful-handling-of-sigint-when-using-pythons-multiprocessingprocess.html new file mode 100644 index 00000000..f5e2a076 --- /dev/null +++ b/build/blog/2015-05-05-graceful-handling-of-sigint-when-using-pythons-multiprocessingprocess.html @@ -0,0 +1,106 @@ +<!DOCTYPE html> +<html> +<head> +<meta charset="utf-8"/> +<meta content="pandoc" name="generator"/> +<meta content="Zhiming Wang" name="author"/> +<meta content="2015-05-05T22:03:39-07:00" name="date"/> +<title>Graceful handling of SIGINT when using Python's multiprocessing.Process</title> +<link href="/img/apple-touch-icon-152.png" rel="apple-touch-icon-precomposed"/> +<meta content="#FFFFFF" name="msapplication-TileColor"/> +<meta content="/img/favicon-144.png" name="msapplication-TileImage"/> +<meta content="width=device-width, initial-scale=1" name="viewport"/> +<link href="/css/normalize.min.css" media="all" rel="stylesheet" type="text/css"/> +<link href="/css/theme.css" media="all" rel="stylesheet" type="text/css"/> +<link href="/css/highlight.css" media="all" rel="stylesheet" type="text/css"/> +</head> +<body> +<div id="archival-notice">This blog has been archived.<br/>Visit my home page at <a href="https://zhimingwang.org">zhimingwang.org</a>.</div> +<nav class="nav"> +<a class="nav-icon" href="/" title="Home"><!--blog icon--></a> +<a class="nav-title" href="/"><!--blog title--></a> +<a class="nav-author" href="https://github.com/zmwangx" target="_blank"><!--blog author--></a> +</nav> +<article class="content"> +<header class="article-header"> +<h1 class="article-title">Graceful handling of <code>SIGINT</code> when using Python's <code>multiprocessing.Process</code></h1> +<div class="article-metadata"> +<time class="article-timestamp" datetime="2015-05-05T22:03:39-07:00">May 5, 2015</time> +</div> +</header> +<p>Today I learned something about Python's (at least CPython's) multiprocessing and signal handling, and I would like to share it here. Basically my situation was such (when developing <a href="https://github.com/zmwangx/zmwangx.github.io/blob/source/pyblog"><code>pydoc</code></a> that powers this blog):</p> +<ul> +<li>I would like to serve the blog with an HTTP server while auto-regenerating for changes;</li> +<li>The auto-regeneration is handled in the main process with a while loop, whereas the HTTP server (requiring little human intervention) is put in a <code>multiprocessing.Process</code> and launched with <code>http.server.HTTPServer.serve_forever()</code>;</li> +<li>Upon sending <code>SIGINT</code>, both processes need to clean up and quit; in particular, the server needs to exit its <code>serve_forever()</code> loop (which can be done via <code>shutdown()</code>, but how to invoke the method is a problem, since <code>serve_forever()</code> blocks);</li> +<li>Handling of <code>SIGINT</code> must be graceful in the main process — there might be an ongoing build that must not be interrupted until finishing.</li> +</ul> +<p>Given this context, I learned the following two critical concepts (at least true in the current version of CPython) through trial and error:</p> +<ol style="list-style-type: decimal"> +<li><strong>A user-triggered <code>SIGINT</code> is sent to both processes</strong> — the main process and the <code>multiprocessing.Process</code> instance;</li> +<li><strong>Except for the defined interfaces, a <code>multiprocessing.Process</code> instance is almost completely separated from the main process, sharing as little resources as possible</strong>; by "defined interfaces" I mean the defined attributes and methods of a <code>Process</code> instance, as well as defined communication channels like <code>multiprocessing.Pipe</code> or <code>multiprocessing.Queue</code>. And to expand on resource sharing: yes, the two processes have their own copies of global variables, so using global variables as state registers is a no-go.</li> +</ol> +<p>Both concepts can be used to one's benefit or detriment. Below is how I solved my problem, using the two concepts. Observe that without a custom handler, Python translates a <code>SIGINT</code> to a <code>KeyboardInterrupt</code> exception; therefore, I use the default <code>KeyboardInterrupt</code> to interrupt the HTTP server in its own process (through handling the exception and calling <code>shutdown()</code>), but instead install a custom <code>SIGINT</code> handler in the main process that translates <code>SIGINT</code> to setting a <code>sigint_raised</code> flag that can be picked up by the while loop once the current build (if any) is finished. The proof of concept script is as follows (the production code is <a href="https://github.com/zmwangx/zmwangx.github.io/blob/a7a0b2073f30b1d0214c3152998d95e40a39b438/pyblog#L567-L635">here</a>):</p> +<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="co">#!/usr/bin/env python3</span> + +<span class="im">import</span> http.server +<span class="im">import</span> multiprocessing +<span class="im">import</span> signal +<span class="im">import</span> sys +<span class="im">import</span> time + +<span class="kw">class</span> HTTPServerProcess(multiprocessing.Process): + <span class="kw">def</span> run(<span class="va">self</span>): + httpd <span class="op">=</span> http.server.HTTPServer( + (<span class="st">""</span>, <span class="dv">8000</span>), http.server.SimpleHTTPRequestHandler) + <span class="cf">try</span>: + httpd.serve_forever() + <span class="cf">except</span> <span class="pp">KeyboardInterrupt</span>: + httpd.shutdown() + +<span class="kw">def</span> do_things(): + <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">10</span>): + sys.stderr.write(<span class="st">"."</span>) + sys.stderr.flush() + time.sleep(<span class="dv">1</span>) + sys.stderr.write(<span class="st">"</span><span class="ch">\n</span><span class="st">"</span>) + +<span class="kw">def</span> main(): + server_process <span class="op">=</span> HTTPServerProcess() + server_process.start() + + <span class="co"># define and install custom SIGINT handler</span> + sigint_raised <span class="op">=</span> <span class="va">False</span> + + <span class="kw">def</span> sigint_mitigator(signum, frame): + <span class="kw">nonlocal</span> sigint_raised + sigint_raised <span class="op">=</span> <span class="va">True</span> + + signal.signal(signal.SIGINT, sigint_mitigator) + + <span class="cf">while</span> <span class="kw">not</span> sigint_raised: + do_things() + + server_process.join() + +<span class="cf">if</span> <span class="va">__name__</span> <span class="op">==</span> <span class="st">"__main__"</span>: + main()</code></pre></div> +<p>Beware that with this solution, if there are external programs or OS level operations happening in the main process, then the operation at the time of <code>SIGINT</code> will still be interrupted<a class="footnoteRef" href="#fn1" id="fnref1"><sup>1</sup></a> (for example, in the script above, the <code>time.sleep(1)</code> at the exact point of <code>SIGINT</code> is still interrupted, but otherwise <code>do_things</code> is carried on to its completion). I'm not sure how to explain this — maybe the handler isn't capturing the signal fast enough?<a class="footnoteRef" href="#fn2" id="fnref2"><sup>2</sup></a> Anyway, one single early interruption is at least more acceptable than a completely corrupted build<a class="footnoteRef" href="#fn3" id="fnref3"><sup>3</sup></a>, and certainly more graceful.</p> +<div class="footnotes"> +<hr/> +<ol> +<li id="fn1"><p><a href="https://hg.python.org/cpython/file/1320ec1b24af/Modules/_multiprocessing">CPython's <code>multiprocessing</code> is written in C</a>, so the behavior might depend on the OS. I'm talking about OS X here. I haven't inspected and won't inspect the C source code.<a class="footnotes-backlink" href="#fnref1">↩︎</a></p></li> +<li id="fn2"><p>That's awfully naive and layman-sounding, I know, but I <em>am</em> almost a layman when it comes to system-level programming.<a class="footnotes-backlink" href="#fnref2">↩︎</a></p></li> +<li id="fn3"><p>That's assuming your build isn't interdependent in which any single failure corrupts everything. In that case, what can we do? I honestly see no way of injecting signal handling in <code>subprocess.Popen</code>.<a class="footnotes-backlink" href="#fnref3">↩︎</a></p></li> +</ol> +</div> +</article> +<hr class="content-separator"/> +<footer class="footer"> +<span class="rfooter"> +<a class="rss-icon" href="/rss.xml" target="_blank" title="RSS feed"><!--RSS feed icon--></a><a class="atom-icon" href="/atom.xml" target="_blank" title="Atom feed"><!--Atom feed icon--></a><a class="cc-icon" href="https://creativecommons.org/licenses/by/4.0/" target="_blank" title="Released under the Creative Commons Attribution 4.0 International license."><!--CC icon--></a> +<a href="https://github.com/zmwangx" target="_blank">Zhiming Wang</a> +</span> +</footer> +</body> +</html> |