1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
|
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<meta content="pandoc" name="generator"/>
<meta content="Zhiming Wang" name="author"/>
<meta content="2015-12-27T16:47:05-08:00" name="date"/>
<title>Lesson on magic method access of Python new-style classes (from my failed Python3 port of Tomorrow)</title>
<link href="/img/apple-touch-icon-152.png" rel="apple-touch-icon-precomposed"/>
<meta content="#FFFFFF" name="msapplication-TileColor"/>
<meta content="/img/favicon-144.png" name="msapplication-TileImage"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="/css/normalize.min.css" media="all" rel="stylesheet" type="text/css"/>
<link href="/css/theme.css" media="all" rel="stylesheet" type="text/css"/>
<link href="/css/highlight.css" media="all" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="archival-notice">This blog has been archived.<br/>Visit my home page at <a href="https://zhimingwang.org">zhimingwang.org</a>.</div>
<nav class="nav">
<a class="nav-icon" href="/" title="Home"><!--blog icon--></a>
<a class="nav-title" href="/"><!--blog title--></a>
<a class="nav-author" href="https://github.com/zmwangx" target="_blank"><!--blog author--></a>
</nav>
<article class="content">
<header class="article-header">
<h1 class="article-title">Lesson on magic method access of Python new-style classes (from my failed Python3 port of Tomorrow)</h1>
<div class="article-metadata">
<time class="article-timestamp" datetime="2015-12-27T16:47:05-08:00">December 27, 2015</time>
</div>
</header>
<p>I know the title is formidably long, but I can't find something more accurate (and my homegrown mini CMS doesn't support subtitle), so please bear with me.</p>
<p>So, I have <a href="https://github.com/madisonmay/Tomorrow">madisonmay/Tomorrow</a> — "magic decorator syntax for asynchronous code in Python 2.7" — bookmarked for a long time<a class="footnoteRef" href="#fn1" id="fnref1"><sup>1</sup></a> without ever trying it, because I simply don't write Python 2 code any more (except when I try to maintain compatibililty). I felt kind of strange that a ~50-line project with ~1000 stars on GitHub hasn't been ported to Python 3 already, so I gave it a shot just now.</p>
<p>I thought it would be easy:</p>
<ol style="list-style-type: decimal">
<li>Modernize the old-style class <code>Tomorrow</code>;</li>
<li>Replace <code>__getattr__</code> with <code>__getattribute__</code> for unconditional attribute routing, then make a few exceptions to prevent infinite recursion;</li>
<li><code>2to3</code> test cases;</li>
<li>Make meta changes, like removing the <code>futures</code> dependency.</li>
</ol>
<p>However, after doing 1–3, I ran the tests, and out of the five test cases, three failed and one errored. I tried to isolate the problem, and ended up with the following piece of proof-of-concept:</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> PassThrough(<span class="bu">object</span>):
<span class="kw">def</span> <span class="fu">__init__</span>(<span class="va">self</span>, obj):
<span class="va">self</span>._obj <span class="op">=</span> obj
<span class="kw">def</span> <span class="fu">__getattribute__</span>(<span class="va">self</span>, name):
<span class="cf">if</span> name <span class="op">==</span> <span class="st">"_obj"</span>:
<span class="cf">return</span> <span class="bu">object</span>.<span class="fu">__getattribute__</span>(<span class="va">self</span>, name)
<span class="bu">print</span>(<span class="st">"Accessing '</span><span class="sc">%s</span><span class="st">'"</span> <span class="op">%</span> name)
<span class="cf">return</span> <span class="va">self</span>._obj.<span class="fu">__getattribute__</span>(name)</code></pre></div>
<p>This snippet is valid in both Python 2.7 and Python 3, but here's the surprise:</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="op">>>></span> g <span class="op">=</span> PassThrough(<span class="dv">0</span>)
<span class="op">>>></span> <span class="bu">print</span>(g)
<span class="op"><</span>__main__.PassThrough <span class="bu">object</span> at <span class="bn">0x10c662e48</span><span class="op">></span>
<span class="op">>>></span> <span class="bu">str</span>(g)
<span class="co">'<__main__.PassThrough object at 0x10c662e48>'</span>
<span class="op">>>></span> <span class="bu">hasattr</span>(g, <span class="st">'__str__'</span>)
Accessing <span class="st">'__str__'</span>
<span class="va">True</span>
<span class="op">>>></span> g.<span class="fu">__str__</span>()
Accessing <span class="st">'__str__'</span>
<span class="co">'0'</span></code></pre></div>
<p>In addition, here's what happens if you try to "pass through" a function:</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="op">>>></span> <span class="kw">def</span> f(): <span class="cf">return</span> <span class="va">True</span>
<span class="op">>>></span> g <span class="op">=</span> PassThrough(f)
<span class="op">>>></span> g()
Accessing <span class="st">'__class__'</span>
Accessing <span class="st">'__class__'</span>
Traceback (most recent call last):
File <span class="st">"<ipython-input-6-d65ffd94a45c>"</span>, line <span class="dv">1</span>, <span class="kw">in</span> <span class="op"><</span>module<span class="op">></span>
g()
<span class="pp">TypeError</span>: <span class="st">'PassThrough'</span> <span class="bu">object</span> <span class="kw">is</span> <span class="kw">not</span> <span class="bu">callable</span>
<span class="op">>>></span> <span class="bu">callable</span>(g)
<span class="va">False</span>
<span class="op">>>></span> <span class="bu">hasattr</span>(g, <span class="st">'__call__'</span>)
Accessing <span class="st">'__call__'</span>
<span class="va">True</span>
<span class="op">>>></span> g.<span class="fu">__call__</span>()
Accessing <span class="st">'__call__'</span>
<span class="va">True</span></code></pre></div>
<p>As you can tell, although <code>__str__</code> or <code>__call__</code> may have been implemented through <code>__getattribute__</code>, and <code>hasattr</code> (which in turn depends on <code>getattr</code>) has no trouble finding them, they are not picked up by <code>str</code> or function call <code>(...)</code>. At this point, one would suspect that this is due to <code>str</code> or function call only looking at the class instance's <code>__dict__</code>. Compare this to the behavior of an old-style class:</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> PassThrough():
<span class="kw">def</span> <span class="fu">__init__</span>(<span class="va">self</span>, obj):
<span class="va">self</span>._obj <span class="op">=</span> obj
<span class="kw">def</span> <span class="fu">__getattr__</span>(<span class="va">self</span>, name):
<span class="bu">print</span>(<span class="st">"Acessing '</span><span class="sc">%s</span><span class="st">'"</span> <span class="op">%</span> name)
<span class="cf">return</span> <span class="va">self</span>._obj.<span class="fu">__getattribute__</span>(name)</code></pre></div>
<p>Now:</p>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="op">>>></span> g <span class="op">=</span> PassThrough(<span class="dv">0</span>)
<span class="op">>>></span> <span class="bu">print</span>(g)
Acessing <span class="st">'__str__'</span>
<span class="dv">0</span>
<span class="op">>>></span> <span class="kw">def</span> f(): <span class="cf">return</span> <span class="va">True</span>
<span class="op">>>></span> g <span class="op">=</span> PassThrough(f)
<span class="op">>>></span> g()
Acessing <span class="st">'__call__'</span>
<span class="va">True</span></code></pre></div>
<p>Note that magic method access is always routed through <code>__getattr__</code>.</p>
<p>After some digging, my suspicion was confirmed: indeed, for new-style classes, rather than invoking <code>__getattribute__</code>, the Python interpreter only looks for magic methods in <code>__dict__</code>. But is there a workaround for implementing something like the <code>PassThrough</code> class above? There's a <a href="http://stackoverflow.com/a/9059858/">nice answer</a> on StackOverflow that uses a metaclass to "automatically add proxies for magic methods at the time of class creation", to quote the author. However, the thing about Tomorrow is that <em>we don't have the result and don't know whatever magic methods it might have at class creation</em> — after all, Python isn't a statically typed language. It is possible for programmers to offer hints, but then Tomorrow won't be as elegant and magical anymore. Therefore, unfortunately enough, Tomorrow isn't portable to Python 3 — at least not without a substantial hack that's beyond my knowledge, or a complete overhaul of its logic (haven't thought about that).</p>
<div class="footnotes">
<hr/>
<ol>
<li id="fn1"><p>Pretty much since the beginning, I believe (the <a href="https://github.com/madisonmay/Tomorrow/commit/22a53dfbcf9b516ecd1770eeca9fcf1720271240">initial commit</a> was from July 24 of this year). I don't remember how I came accross it though.<a class="footnotes-backlink" href="#fnref1">↩︎</a></p></li>
</ol>
</div>
</article>
<hr class="content-separator"/>
<footer class="footer">
<span class="rfooter">
<a class="rss-icon" href="/rss.xml" target="_blank" title="RSS feed"><!--RSS feed icon--></a><a class="atom-icon" href="/atom.xml" target="_blank" title="Atom feed"><!--Atom feed icon--></a><a class="cc-icon" href="https://creativecommons.org/licenses/by/4.0/" target="_blank" title="Released under the Creative Commons Attribution 4.0 International license."><!--CC icon--></a>
<a href="https://github.com/zmwangx" target="_blank">Zhiming Wang</a>
</span>
</footer>
</body>
</html>
|
