1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
|
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<meta content="pandoc" name="generator"/>
<meta content="Zhiming Wang" name="author"/>
<meta content="2016-01-01T22:21:14-08:00" name="date"/>
<title>Virtualenvs for everyone</title>
<link href="/img/apple-touch-icon-152.png" rel="apple-touch-icon-precomposed"/>
<meta content="#FFFFFF" name="msapplication-TileColor"/>
<meta content="/img/favicon-144.png" name="msapplication-TileImage"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="/css/normalize.min.css" media="all" rel="stylesheet" type="text/css"/>
<link href="/css/theme.css" media="all" rel="stylesheet" type="text/css"/>
<link href="/css/highlight.css" media="all" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="archival-notice">This blog has been archived.<br/>Visit my home page at <a href="https://zhimingwang.org">zhimingwang.org</a>.</div>
<nav class="nav">
<a class="nav-icon" href="/" title="Home"><!--blog icon--></a>
<a class="nav-title" href="/"><!--blog title--></a>
<a class="nav-author" href="https://github.com/zmwangx" target="_blank"><!--blog author--></a>
</nav>
<article class="content">
<header class="article-header">
<h1 class="article-title">Virtualenvs for everyone</h1>
<div class="article-metadata">
<time class="article-timestamp" datetime="2016-01-01T22:21:14-08:00">January 1, 2016</time>
</div>
</header>
<p>Python distutils for the most part is rather pleasant to work with. That is, pleasant until you've accumulated so many packages that you eventually run into a clash of namespace, or a dependency conflict (or dependency hell as most would affectionately call it).<a class="footnoteRef" href="#fn1" id="fnref1"><sup>1</sup></a> In contrast, npm's approach to dependencies shuts out dependency hell completely, but it is so paranoid and costs so much duplication that I find it hard to appreciate unless necessary. Somewhere in between there's the virtualenv approach which I find most appealing for smallish projects — keep a single copy of each package in the dependency tree in a contained environment specific to the project at hand. This is how we debug Python projects, and it certainly also should be <em>the</em> way we run command line tools written in Python.</p>
<p>There's another reason I like virtualenvs. There are tons of problems associated with choosing between Python 2 and 3 — some projects are Python 2 only, some are instead Python 3, some claim to be compatible with both but actually present subtle problems when you use one instead of the other. However, without virtualenvs, there's only one <code>bin</code> — <code>/usr/local/bin</code> — and everything's competing for it. Most programs (especially ones with a typical <code>setup.py</code>) don't install a soft/hardlink with a helpful <code>2</code> or <code>3</code> suffix when installing executables, let alone detailed suffixes like <code>2.7</code> or <code>3.5</code>, so without probing into the shebangs you're never sure which version of Python you're running your program with, and as a result Python 2/3 (or even a point release)-specific bugs occur randomly. Virtualenvs solve the problem by allowing you to have as many bins (and includes, and libs) as you like.</p>
<p>Hence the title "virtualenvs for everyone". I would like to install each command line program written in Python into a separate virtualenv. The only issue is that apparently I don't want too many bins in my <code>$PATH</code>; to solve this issue, the executable bits of each project should be linked to a central place, for which I choose <code>$HOME/bin</code>. There could be as many symlinks as we like, so now we can have multiple links with increasing detailed version suffixes, e.g., <code>3</code>, <code>3.5</code>, <code>3.5.1</code>. Very nice.</p>
<p>This task could clearly be automated; the only slightly tricky bit is to programmatically figure out which scripts a project installs to <code>bin</code>. Luckily, for projects using <code>setuptools.setup</code>, we can simply spoof that function. Here's my <code>setuptools/__init__.py</code>:</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="co">#!/usr/bin/env python3</span>
<span class="co">"""setuptools stubs.</span>
<span class="co">Here we only stubbed the symbols in setuptools.__all__. Hopefully that's</span>
<span class="co">enough (actually I can't remember seeing any setup.py using more than</span>
<span class="co">setup and find_packages).</span>
<span class="co">setup has been spoofed to print the names of scripts, console_scripts</span>
<span class="co">and gui_scripts defined in the arguments to setup. Some user-friendly</span>
<span class="co">messages are also printed to stderr.</span>
<span class="co">"""</span>
<span class="im">from</span> __future__ <span class="im">import</span> print_function
<span class="im">import</span> re
<span class="im">import</span> sys
<span class="im">import</span> os
__all__ <span class="op">=</span> [
<span class="st">'setup'</span>, <span class="st">'Distribution'</span>, <span class="st">'Feature'</span>, <span class="st">'Command'</span>, <span class="st">'Extension'</span>, <span class="st">'Require'</span>,
<span class="st">'find_packages'</span>
]
<span class="kw">def</span> setup(<span class="op">**</span>kwargs):
scripts <span class="op">=</span> [os.path.basename(script_path)
<span class="cf">for</span> script_path <span class="kw">in</span> kwargs.pop(<span class="st">'scripts'</span>, [])]
<span class="cf">if</span> scripts:
<span class="bu">print</span>(<span class="st">'scripts:</span><span class="ch">\n</span><span class="st"> - </span><span class="sc">%s</span><span class="st">'</span> <span class="op">%</span> <span class="st">'</span><span class="ch">\n</span><span class="st"> - '</span>.join(scripts), <span class="bu">file</span><span class="op">=</span>sys.stderr)
entry_points <span class="op">=</span> kwargs.pop(<span class="st">'entry_points'</span>, {})
<span class="cf">for</span> entry_point <span class="kw">in</span> [<span class="st">'console_scripts'</span>, <span class="st">'gui_scripts'</span>]:
extra_scripts <span class="op">=</span> [re.split(<span class="st">'(\s|=)'</span>, spec.strip())[<span class="dv">0</span>]
<span class="cf">for</span> spec <span class="kw">in</span> entry_points.pop(entry_point, [])]
<span class="cf">if</span> extra_scripts:
<span class="bu">print</span>(<span class="st">'</span><span class="sc">%s</span><span class="st">:</span><span class="ch">\n</span><span class="st"> - </span><span class="sc">%s</span><span class="st">'</span> <span class="op">%</span> (entry_point, <span class="st">'</span><span class="ch">\n</span><span class="st"> - '</span>.join(extra_scripts)),
<span class="bu">file</span><span class="op">=</span>sys.stderr)
scripts.extend(extra_scripts)
<span class="bu">print</span>(<span class="st">'</span><span class="ch">\n</span><span class="st">'</span>.join(<span class="bu">sorted</span>(scripts)))
<span class="kw">class</span> Distribution(<span class="bu">object</span>): <span class="cf">pass</span>
<span class="kw">class</span> Feature(<span class="bu">object</span>): <span class="cf">pass</span>
<span class="kw">class</span> Command(<span class="bu">object</span>): <span class="cf">pass</span>
<span class="kw">class</span> Extension(<span class="bu">object</span>): <span class="cf">pass</span>
<span class="kw">class</span> Require(<span class="bu">object</span>): <span class="cf">pass</span>
<span class="kw">def</span> find_packages(<span class="op">**</span>kwargs): <span class="cf">pass</span></code></pre></div>
<p>Now, let <code>$HERE</code> be the directory containing our fake <code>setuptools/</code>, and <code>$PROJECT_ROOT</code> be the project root directory containing <code>setup.py</code>. Run</p>
<div class="sourceCode"><pre class="sourceCode zsh"><code class="sourceCode zsh"><span class="ot">PYTHONPATH=$HERE</span>:<span class="ot">$PYTHONPATH</span> python <span class="ot">$PROJECT_ROOT</span>/setup.py</code></pre></div>
<p>and bam! We get the names of all scripts on stdout.</p>
<p>My full automation scripts, including the Zsh main function <code>virtual-install</code>, can be found in <a href="https://github.com/zmwangx/prezto/tree/master/modules/python/functions"><code>modules/python/functions</code> in zmwangx/prezto</a>. I'm not including it here because it uses some custom helper, and it's just too long (200+ lines, but not very sophisticated). Happy virtualenving!</p>
<div class="footnotes">
<hr/>
<ol>
<li id="fn1"><p>In rare cases, even installing a single package could land you in trouble. The classical example is installing the <code>readme</code> package on a case-insensitive filesystem (e.g., the default mode of HFS+). "Unfortunately" <a href="https://bugs.python.org/issue24633">this</a> has been fixed.<a class="footnotes-backlink" href="#fnref1">↩︎</a></p></li>
</ol>
</div>
</article>
<hr class="content-separator"/>
<footer class="footer">
<span class="rfooter">
<a class="rss-icon" href="/rss.xml" target="_blank" title="RSS feed"><!--RSS feed icon--></a><a class="atom-icon" href="/atom.xml" target="_blank" title="Atom feed"><!--Atom feed icon--></a><a class="cc-icon" href="https://creativecommons.org/licenses/by/4.0/" target="_blank" title="Released under the Creative Commons Attribution 4.0 International license."><!--CC icon--></a>
<a href="https://github.com/zmwangx" target="_blank">Zhiming Wang</a>
</span>
</footer>
</body>
</html>
|
