dl? cmplnts?

pyenv: compiling Python with SQLite in nonstandard location

zmwangx@gmail.com (Zhiming Wang) — Wed, 26 Oct 2016 16:16:22 GMT

- -

This is a quick post sharing a workaround that I needed just now.

I was trying to compile Pythons with pyenv on a RHEL 6.8 cluster. Unfortunately sqlite-devel is not installed and I doubt I can convince my sysadmin to install a package for me. The lack of SQLite headers resulted in Pythons without _sqlite3 which is essential for me. Hinting at SQLite headers from Linuxbrew with CPATH did not help either.

Digging into CPython source code, turns out that CPython only looks into a fixed set of paths:

sqlite_inc_paths = [ '/usr/include',
-                     '/usr/include/sqlite',
-                     '/usr/include/sqlite3',
-                     '/usr/local/include',
-                     '/usr/local/include/sqlite',
-                     '/usr/local/include/sqlite3',
-                     ]
-if cross_compiling:
-    sqlite_inc_paths = []

Well that's unfortunate. Luckily pyenv makes it really easy to patch Python source code; take a look at plugins/python-build/share/python-build/patches and you'll get the idea. Therefore, in the case of Linuxbrew'ed pyenv and SQLite, say we want to build Python 3.5.2 with SQLite support, we simply put the following patch at ~/.linuxbrew/opt/pyenv/plugins/python-build/share/python-build/patches/3.5.2/Python-3.5.2/linuxbrew-sqlite3.patch:

diff --git a/setup.py b/setup.py
-index 174ce72..774fd65 100644
---- a/setup.py
-+++ b/setup.py
-@@ -1108,6 +1108,7 @@ class PyBuildExt(build_ext):
-                              '/usr/local/include',
-                              '/usr/local/include/sqlite',
-                              '/usr/local/include/sqlite3',
-+                             os.path.expanduser('~/.linuxbrew/opt/sqlite/include/'),
-                              ]
-         if cross_compiling:
-             sqlite_inc_paths = []

That's it. Now

$ pyenv install 3.5.2

and enjoy.

This blog is now behind CloudFlare

zmwangx@gmail.com (Zhiming Wang) — Thu, 01 Sep 2016 12:11:00 GMT

- -

Back in July I registered the domain zhimingwang.org and pointed this GitHub Pages-powered blog at it. Since then I have lost the HTTPS badge due to GitHub Pages not supporting HTTPS on custom domains (see isaacs/github#156).

There have been a lot of discussions on isaacs/github#156 (and stupid +1's too). Among the proposed solutions is putting the website behind CloudFlare. I carefully investigated this option and read almost all the arguments against it. I fully understand CloudFlare's SSL models (summarized in the image below), and I do realize most if not all of the limitations of CloudFlare, including CloudFlare being a huge MITM (which is inevitable for a CDN anyway), as well as most if not all of its annoyances, including CAPTCHAs which I myself would occasionally run into when I'm browsing with PIA VPN, and JavaScript-based browser checks.

- -

CloudFlare's SSL modes. I use the Full SSL mode so that both ends of the connection are encrypted. Again, I know CloudFlare is a big MITM and could be a high profile target. Credit: CloudFlare.

After careful evaluation, I decided that CloudFlare's SSL model is good enough for me. After all, this is just a damn blog, with nothing sensitive. TLS is still nice because it guards against prying eyes and unethical ad-injecting ISPs or Wi-Fi hotspots, but other than that, it isn't necessary.

End result: this blog is now behind CloudFlare. Readers should now see that green HTTPS badge again (note that I'm enforcing HTTPS — without HSTS though). As for CAPTCHAs, I have adjusted the firewall settings on CloudFlare's dashboard — "Security Level" to "Essentially Off" and "Challenge Passage" to 1 year, so hopefully it won't be too annoying.¹

09/01/2016 Update. I just realized that CloudFlare supports whitelisting Tor traffic. Did that.

I don't use Tor, and don't intend to raise Big Brother's suspicion by using it, so I have no idea of the actual Tor experience.↩︎

It's 2016, and Microsoft is the only legit player who spams me without unsubscribe links

zmwangx@gmail.com (Zhiming Wang) — Thu, 23 Jun 2016 21:40:01 GMT

- -

I'm so tired of Microsoft spam. Microsoft is known for being intrusive accross the board, and their newsletters are no different: I literally can't name one legit company in this day and age who doesn't put unsubscribe links in newsletters.¹ I get "Azure pricing and services updates" newsletters all the time, as well as "exciting news" from Windows Insider Program (which doesn't excite me at all) every once in a short while.² I still occasionally receive random Chinese language promotions of Windows, presumably because I used a Windows Phone as my secondary phone for three months back home in the summer of 2013 (which was a horrible experience). Why Microsoft hasn't been regulated for spam yet, I do not know.

To be fair, Amazon used to force promotional email on student members, but (if memory serves) I haven't seen one in ages.↩︎
"If you wish to stop receiving Windows Insider Program emails, you will need to leave the program." I guess that's the price of downloading a couple of Windows 10 insider builds.↩︎

Chrome is screwing with our extensions... Again

zmwangx@gmail.com (Zhiming Wang) — Sun, 08 May 2016 01:49:26 GMT

- -

Chrome is growing more and more hostile by the day. See Google Chrome keeps getting uglier for an earlier take. What I didn't report in the earlier post is that not only can't you show/hide extension buttons as easily as before, you can't even control which buttons appear in the toolbar anymore — they come and go as they wish.

As if screwing the app icon, extension buttons and the overall design is not enough, now they have upped their game again. I'm running Chrome 50.0.2661.94 from April 28, and I just rebooted my machine only to be greeted with a fleeting "unsupported extensions" (or something like that) message as I launched Chrome. I digged into the extensions page, and guess what, all sideloaded extensions (except unpacked ones) have been disabled.¹ This in itself may not be too surprising (Google took away sideloaded extensions from non-developers last year), except that the "enable" button, which used to work at least in developer mode, doesn't function anymore. The message is very stupid:

-
This extension is not listed in the Chrome Web Store and may have been added without your knowledge.
-

Bold by me. Okay, so what if they have been added with my knowledge? No way to enable legit extensions (some written by none other than myself) just because of a "may"? Here's the only migration path they offer, by the way:

-
If you need to use a disabled extension, you can contact the extension's developer and ask them to upload their extension to the Chrome Web Store.
-

Seriously? Do they honestly think Chrome Web Store serves everyone's needs? First, they have every right to refuse or take down any extension in their store. This is dangerous. What if one day they conclude that Adblock Plus is hurting their ad revenue too much and decide to take it down? Secondly, people may not want to make every extension publicly available. For instance, I have some personal extensions that I have developed on my dev machine, packaged into .crx, and installed on other machines. Some of these are publicly available (on GitHub), and others are not. It's not hard to conclude that other people may have private extensions too, and there may be extensions that are only available in some private circles. Now, people have to load unpacked extensions, which is much easier to screw up for regular folks, or they're out of luck.

To add insult to injury, every time I launch Chrome now, I'm greeted by this "Disable Developer Mode Extensions" message:

-
Extensions running in developer mode can harm your computer. If you're not a developer, you should disable these extensions running in developer mode to stay safe.
-

As if there're not enough malicious extensions in the Chrome Web Store, let alone crap. May I tell Chrome that I am a developer and ask it to shut up? Apparently no.

With the current trend in Chrome, I might want to switch to Opera again.² The only thing preventing me from doing so right now is their new horrendous-looking fat icon. However, Chrome has also destroyed their icon and I need to replace it after every update anyway, so I might as well do the same thing for Opera. We'll see.

I'm not sure why they weren't disabled upon first launch after the update, but given the randomness of extension buttons in my toolbar with hardly any action on my part, I won't be surprised if I were told that they had messed up the extension system completely.↩︎
Modern day Safari is also pretty nice, and the team is showing great attitude lately, with Safari Technology Preview and tweets like this one, for instance. However, the lack of extensions is a big road block, and the fact that the used-to-be-free Safari Developer Program has been incorporated into the $99/yr Apple Developer Program certainly doesn't help. (I used to be a member. Now I've been kicked out.)
-
Note that Safari is more locked down in a sense compared to OS X and iOS. On OS X you can apparently run unsigned software; on iOS 9 and later you can create personal provisioning profiles with just an Apple ID. Neither is true for Safari extensions, which still require a signing key from developer program membership. I wonder if Apple will introduce free keys for personal use on Safari, too.↩︎

Emacs's got a redesigned website!

zmwangx@gmail.com (Zhiming Wang) — Sun, 10 Apr 2016 10:04:19 GMT

- -

I've been very busy lately, so I haven't posted anything for a month. As a result, I have many topic notes sitting in Notes.app — including the kik-left-pad-npm drama, the Text Expander outcry, and such — waiting to be organized and written up (I'll probably never write them up in the end, since these days it's very weird to write an opinion piece about an event whose attention span has already lapsed).

Anyway, this will be a short post about Emacs's redesigned website. See screenshot at the end. Apparently this was last week's news, but there's little interest in Emacs in general, so the news only reached me two days ago, sort of by chance.

According my impression, Emacs has been the underdog for quite some time. First, when you compare to vi/Vim, hard stats shed light on popularity: there are way more VimL repos than Elisp ones on GitHub.¹ Also, there are more exciting (or at least exciting-sounding or excitement-inducing) things happening in the Vim realm, e.g. Neovim, but not so much in the Emacs kingdom (excuse me if I missed something big). I'm actually very curious how Vim sold itself to so many people. I, for one, can't tolerate the Esc key at all (yes, I know basic editing in Vim, and I know the various workarounds to Esc, some of them reasonable and some not). I can't understand how people could laugh at Escape Meta Alt Control Shift — oh, I never used Esc in Emacs once, by the way — when the single most awkward Esc key serves a fundamental purpose by default in their own beloved editor. The Esc key is of course not my only gripe with Vim, nor the biggest; I'll however stop here to avoid turing this post into a complaint about Vim. Apart from Vim, Emacs is also being sidelined by more modern GUI-based text editors like Atom,² or various IDEs. Atom recently reached one million monthly active users. I actually like certain parts of Atom a lot, e.g. the project navigation sidebar,³ but I simply can't give up my good ol' tty.

Personal preferences aside, I think Emacs does need a bit more publicity to draw a few more users. Whether redesigning the website will help at all I don't know; maybe the effect will be statistically indistinguishable from zero, but the bottom line is that people like pretty websites, so why not. The redesigned homepage is a bit more graphics-heavy, but it currently weighs a total of 521.33KB — within the tolerable range.

The most interesting thing I found on the redesigned homepage is the link to emacsrocks.com. I aimlessly clicked on the last episode — episode 15 — just to see what it was like, and ended up astonished. The episode is about restclient.el, which turned out to be wicked cool. In the real world it's probably a little bit too geeky to my liking, and I use the more mundane (and more powerful) Paw as my REST client, but I can't stop admiring the beauty of restclient-mode. I'll definitely find time to watch all episodes of Emacs Rocks, and you probably should, too.

- -

A scaled down screenshot of the redesigned gnu.org/software/emacs. Full screenshot on my 2880x1800 MBP is here. Actually I lied a bit — the screenshots were taken with pageres, so I could have specified any resolution.

According to GitHut, in 2014 Q4, there were 22,450 VimL and 9,978 Elisp repositories on GitHub, respectively. And according to a real time search I did just now, the VimL number has risen to 82,519 and the Elisp number to 30,320. The ratio has risen from 2.25:1 to 2.72:1. To add insult to injury, on GitHub's advanced search page, GitHub lists VimL in the "Popular" language section and Elisp in "Everything else". Hurt feelings anyone?↩︎
Of course Emacs can operate in standalone GUI mode (or more precisely, window system mode), and more can be done in GUI mode (both in terms of customizability and functinality). However, in my early days with Emacs I found the GUI look like crap — the default always does, even to this day. I can never bring myself to use anything crappy-looking, unless I've got no choice, so I went with the TUI. Later I learned how to make the GUI habitable (still not as nice as the uniformity I find in tty, though), but by that time I'm already totally in love with tty mode and probably will never switch.↩︎
In Emacs I have ido, fiplr and sr-speedbar to help with navigation, but this is one area where a graphical sidebar really shines.↩︎

Google Chrome keeps getting uglier

zmwangx@gmail.com (Zhiming Wang) — Sun, 06 Mar 2016 22:59:45 GMT

- -

I hate to say this, but the Google Chrome team keeps making poor design decisions to make it more and more ugly. I still remember the sad day when the kind of cool wrench button gave way to the utterly boring hamburger one. I also remember the sad day when the omnibox dropdown pointlessly went full width (after more than two years, I still fail to see how it makes any sense, although my eyes have long grown used to it).¹ And I'm sure there are other stupid changes that I can't name at the moment.

Unfortunately, they just won't stop. Four days ago stable 49.0.2623.75 came out with a flurry of horrible visual changes.

The icon. For whatever reason I have the impression that I might have seen this a while ago, but let's just pretend it's brand new. The new app icon is the most outrageously flat icon I've even seen. Compare it to that of 48.0.2564.103²:
-
- -
Old and new app icons side by side. To the left, the 48.0.2564.103 icon; to the right, the 49.0.2623.75 icon.
-
-
And let's see them in action:
-
- -
Both icons in the dock, old one the left and new one on the right.
-
-
Apart from flatness (lack of any gloss found in almost all Apple icons, however flattened they are), see how the new icon is notably larger than the old one, and any other circular icons for that matter. Apparently, consistency and guidelines mean nothing to them.
-
I wonder why they made this change. Maybe for material design? I certainly don't want to see my Mac infested by material design, thank you. And maybe to keep the icon in line with their new Google branding? Indeed, just like the new Google logo which did away with serifs, this one has no depth at all and is very childish.
-
It's a shame I can't just throw this icon out of my dock. Looks like in addition to iTunes now I have yet another icon to replace following each update, except this one updates much more often, and almost silently.
Downloads. I almost thought I was hacked when I opened the Downloads tab and saw
-
- -
Downloads in 49.0.2623.75.
-
-
instead of a nice and clean
-
- -
Downloads in 48.0.2564.103.
-
-
Materail design infestation, apparently. Funny how they managed to convey less info in a LOT more space, and look horrible at the same time. At least they can choose a pleasant color palette if they want to use color (which is totally unnecessary as seen from the old design)? No, they can't.
Incognito mode. There's a reason why books are printed on light-colored paper, and there's a reason why the web is predominantly light-backgrounded, including user agent default style sheets. The old incognito follows the light background rule, plus a non-intrusive notice in the middle and a reasonably shaded tab bar to indicate incognito status:
-
- -
Incognito window in 48.0.2564.103.
-
-
But not anymore. Since those of you using Incognito mode must be conducting shady business, why not highlight that with a black background:
-
- -
Incognito window in 49.0.2623.75. Even more shocking if you maximize your browser windows.
-
-
Oh. My. God. Now I hesitate whenever I want to press ⇧⌘N; it's just too great a cultural shock for me to handle.

Those are just three changes I've discovered so far. Hopefully there are no more lurking surprises.

Conclusion? Sigh.

03/09/2016 update. They also broke showing/hiding extension buttons (from toolbar) recently, probably in the same update. We used to be able to reshow a hidden button from chrome://extensions; that's no longer possible. Now we need to click on the hamburger (great), right click on one of the hidden buttons — which temporarily promotes the button to the toolbar and display the context menu, and while the context menu is still on, click on "Keep in Toolbar". So intuitive, your average computer users are definitely going to figure that out by themselves. Very nice.

Dev left a comment when marking that issue as wont fix:
-
-
... The current look is a precursor to a family of related work to make the Omnibox better, so expect to see more investment in this space to come. ...
-
-
A family of related work? After 2.5 years the omnibox looks almost exactly the same as the screenshot in the issue. More to come my ass. Maybe I should be grateful, at least it didn't get worse.↩︎
I realized the last stable was 48.0.2564.109 instead of 48.0.2564.103 only after taking the screenshots. Doesn't matter anyway.↩︎

Dropbox, Noteworthy, and damned skeuomorphism

zmwangx@gmail.com (Zhiming Wang) — Tue, 26 Jan 2016 20:18:36 GMT

- -

I just opened a note in a PDF within Dropbox's iOS app (never done that before), and instead of readable text what I saw was basically spaghetti:

- -

A PDF note in Dropbox iOS. Noteworthy (scream). I know there's a typo, by the way.

That font is unmistakably Noteworthy, the default font in Apple's Notes app in Mountain Lion, when Apple was still practicing the damned skeuomorphism. (In case you can't recall how it looked like, let me point you to the John Siracusa review for screenshots.) Just like your coworker's average handwritten notes, it is hardly legible and takes tremendous effort just to decode, especially when clustered in a paragraph rather than a short one-liner. Compare that to the same note, legibly rendered in Helvetica in PDF Expert:

- -

The same note (typo corrected) in PDF Expert Mac.

This is an example of sacrificing usability for design aesthetics (an old-fashioned one for that matter, and an abonimable one if you ask for my opinion). Hard to believe we can still see it in 2016, from an otherwise great developer that is Dropbox.

Antivirus app on MAS top chart?

zmwangx@gmail.com (Zhiming Wang) — Mon, 25 Jan 2016 02:43:28 GMT

- -

Today for whatever reason I clicked on MAS's "Top Charts" page, and was immediately in for a surprise. Next to our great friend 1Password is an app called "AntiVirus Sentinel Pro", which sells for $9.99:

- -

AntiVirus Sentinel "Pro". These days many people like to end their apps' names with "Pro", even when there's nothing pro about them. This "Pro" app, for instance, comes from a developer whose three out of four apps matches the regex ^([A-Z](a-z)+ )+Pro$, and despite their names they are definitely geared towards uninformed newbies.

The first rule of using MAS (or any kind of app store, for that matter) is that you research MAS apps outside the MAS. So let's Google "AntiVirus Sentinel Pro"... First result: Apple Support forum thread from late 2014, "Is AntiVirus Sentinel Pro legit? If not, how can I delete it?" Good question about any AV product. However, you'll immediately find it hilarious when you read on:

-
I have purchased and downloaded AntiVirus Sentinel Pro for Macintosh with Yosemite OS. I have a bad feeling this application is useless and maybe even harmful. Anyone knows if it is safe to use it?
-

Okay, so why did you purchase it in the first place? I guess clueless users like this one are in every MAS developer's wet dreams.

Let's continue with the Google search results. Second one is the iTunes preview link. Third one is a YouTube video (also linked from MAS) which seems to be the only online documentation this app's got. Judging from the video the interface seems to be done in Java or something... Never mind that. Fourth result is a rather recent thread (August 2015) from Mac Forums. Not this again:

-
I am a new Mac user. Can somebody answer these questions? Somehow it appears I have installed AntiVirus Sentinel Pro. What is this? Is it a real software? Should I keep it or try to uninstall it?
-

The fifth and sixth results are general OS X AV product reviews that don't even mention this app. The next three results are from MAS aggregation sites. The last result on the first page is the app's product page on MacUpdate (now apparently abandoned), with a shiny 0.5/5 stars (note the zero point) badge.

Now that we've finished the first page (and the results are not a bit reassuring), the question comes: where the heck is this app's home page? It's also not on the second page, actually. There's something interesting on the third (still no home page), that is this tweet:

-
$10 "AntiVirus Sentinel Pro" got top2 in US Mac App Store and top1 in 48 countries--but it's just ClamAV+AdwareMedic signs+3 bullshit signs.
-

Hmm. You might want to read Thomas Reed's (known for The Safe Mac) responses from that thread.

Anyway, we were sidetracked. Back to the home page, actually this app does have one, but it's just a single page, which simply states some marketing bullshit and directs to MAS (where the same bullshit is repeated). Seriously? That's the best you can do for your "pro" app, especially a security-related one?

Back to MAS, the reviews are kind of jokes, too:

- -

Stupid reviews for stupid app.

The first one begins with

-
Just switched to Apple from Windows ... and researched anti-virus software.
-

Don't really need to read further.

Second one, speaking of customer service:

-
... it only took one email to him explaining my problem... had a patch up and ready on the app store to download within hours.
-

This review is from October 2015. Since when was MAS so efficient? Why do I (and everyone else keeping tabs on Apple stuff) keep hearing stories like bug fix updates waiting for review after 59 days?

The pattern goes on. By the way, how does this app keep track of all disk and network activity when itself is running in a sandbox? No idea (maybe I'm misunderstanding sandboxing).

In summary, even as an AV product, this one seems untrustworthy. Not to mention AV products on the Mac are generally superfluous if not harmful.¹ What people really need to learn is to practice safe browsing habits and to properly use content blockers, which AV product vendors and (intrusive-)ad-supported websites (that is most, commercial websites today) won't tell you because they would go out of business if they do.

And how this app got onto the top charts, that is a real mystery.

Disclaimer: I personally have used ClamXav and AdwareMedic (which has since been bought by Malwarebytes) before to help my social-engineered friend (tech support scam, in case you ask). Just to scan their documents though. It is my belief that once you're pwned (even slightly), clean system reinstall is the only way to go, despite what AV products might tell you. In addition, I don't use AV products myself (except Microsoft Security Essentials on Windows); as a programmer I've had enough bad experience with AV blocking my programs back in the Windows days.↩︎

Me-too comments on GitHub

zmwangx@gmail.com (Zhiming Wang) — Tue, 19 Jan 2016 00:36:40 GMT

- -

I frequently subscribe to issues on GitHub, be it bugs I want to see fixed or features I would like to see implemented. Then every once in a short while I get an email notification about one of those obnoxious "me too" or "+1" comments, by which I mean terse comments with little to no content other than "me too" or "+1" or some other variant bearing the same meaning.

Me-too comments under bug reports are the most untolerable. If you have more details regarding the issue (e.g., a more reliable reproducer) or insights into what's really going on, then by any means post them. On the other hand, if you can't provide anything helpful, then just keep your mouth shut, and quietly press "subscribe" if you would like to be kept posted. Posting a me-too comment adds nothing to the discussion, does not expedite the resolution a tiny bit, and only serves to annoy all parties involved.¹ As always, submit a patch if you're dissatisfied with the progress. Keep in mind that no one is obligated to fix bugs for you in FOSS.²

Me-too comments under feature requests are more understandable, though I genuinely doubt that two or three people requesting a feature instead of one would make a big difference. After all, the issue tracker is not a feature voting platform; most folks understand this and behave themselves, so "me-too demand" isn't even remotely accurate at reflecting demand.

Me-too folks: please stop being childish. If you have nothing to add, don't add anything (unless otherwise requested).

01/20/2015 Update. I came accross dear-githuub/dear-github just now, which was started a mere six days ago, and the open letter of which also places +1 comments on its list of biggest problems on GitHub.

03/10/2015 Update. GitHub is finally reacting. See Add Reactions to Pull Requests, Issues, and Comments.

There are exceptional cases.↩︎
Here we're talking about the subset of FOSS that is also free as in beer.↩︎

The dirtiest mistakes of OS X

zmwangx@gmail.com (Zhiming Wang) — Thu, 14 Jan 2016 09:02:52 GMT

- -

I must have written about this elsewhere, but here are my top three:

.DS_Store. Finder litters faster than one could clean up.
HFS+ NFD*.¹ Heard of the cursed encoding UTF8-MAC? Pure Evil. Culprit of tons of garbled text issues (especially cross platform ones) and probably most length miscalculation issues. Even Apple's Terminal.app can't do NFD right. I wonder how Korean users navigate their filesystems in terminal.
Plist XML. It's XML, but even worse.

NFD with an asterisk, i.e., not even NFD. According to Apple in an old Technical Q&A,
-
-
The terms used in this Q&A, precomposed and decomposed, roughly correspond to Unicode Normal Forms C and D, respectively. However, most volume formats do not follow the exact specification for these normal forms. For example, HFS Plus (Mac OS Extended) uses a variant of Normal Form D in which U+2000 through U+2FFF, U+F900 through U+FAFF, and U+2F800 through U+2FAFF are not decomposed (this avoids problems with round trip conversions from old Mac text encodings). It's likely that your volume format has similar oddities.
-
-
They are conscious enough to call these oddities.↩︎

Virtualenvs for everyone

zmwangx@gmail.com (Zhiming Wang) — Sat, 02 Jan 2016 06:21:14 GMT

- -

Python distutils for the most part is rather pleasant to work with. That is, pleasant until you've accumulated so many packages that you eventually run into a clash of namespace, or a dependency conflict (or dependency hell as most would affectionately call it).¹ In contrast, npm's approach to dependencies shuts out dependency hell completely, but it is so paranoid and costs so much duplication that I find it hard to appreciate unless necessary. Somewhere in between there's the virtualenv approach which I find most appealing for smallish projects — keep a single copy of each package in the dependency tree in a contained environment specific to the project at hand. This is how we debug Python projects, and it certainly also should be the way we run command line tools written in Python.

There's another reason I like virtualenvs. There are tons of problems associated with choosing between Python 2 and 3 — some projects are Python 2 only, some are instead Python 3, some claim to be compatible with both but actually present subtle problems when you use one instead of the other. However, without virtualenvs, there's only one bin — /usr/local/bin — and everything's competing for it. Most programs (especially ones with a typical setup.py) don't install a soft/hardlink with a helpful 2 or 3 suffix when installing executables, let alone detailed suffixes like 2.7 or 3.5, so without probing into the shebangs you're never sure which version of Python you're running your program with, and as a result Python 2/3 (or even a point release)-specific bugs occur randomly. Virtualenvs solve the problem by allowing you to have as many bins (and includes, and libs) as you like.

Hence the title "virtualenvs for everyone". I would like to install each command line program written in Python into a separate virtualenv. The only issue is that apparently I don't want too many bins in my $PATH; to solve this issue, the executable bits of each project should be linked to a central place, for which I choose $HOME/bin. There could be as many symlinks as we like, so now we can have multiple links with increasing detailed version suffixes, e.g., 3, 3.5, 3.5.1. Very nice.

This task could clearly be automated; the only slightly tricky bit is to programmatically figure out which scripts a project installs to bin. Luckily, for projects using setuptools.setup, we can simply spoof that function. Here's my setuptools/__init__.py:

#!/usr/bin/env python3
-
-"""setuptools stubs.
-
-Here we only stubbed the symbols in setuptools.__all__. Hopefully that's
-enough (actually I can't remember seeing any setup.py using more than
-setup and find_packages).
-
-setup has been spoofed to print the names of scripts, console_scripts
-and gui_scripts defined in the arguments to setup. Some user-friendly
-messages are also printed to stderr.
-
-"""
-
-from __future__ import print_function
-
-import re
-import sys
-import os
-
-__all__ = [
-    'setup', 'Distribution', 'Feature', 'Command', 'Extension', 'Require',
-    'find_packages'
-]
-
-def setup(**kwargs):
-    scripts = [os.path.basename(script_path)
-               for script_path in kwargs.pop('scripts', [])]
-    if scripts:
-        print('scripts:\n  - %s' % '\n  - '.join(scripts), file=sys.stderr)
-    entry_points = kwargs.pop('entry_points', {})
-    for entry_point in ['console_scripts', 'gui_scripts']:
-        extra_scripts = [re.split('(\s|=)', spec.strip())[0]
-                         for spec in entry_points.pop(entry_point, [])]
-        if extra_scripts:
-            print('%s:\n  - %s' % (entry_point, '\n  - '.join(extra_scripts)),
-                  file=sys.stderr)
-        scripts.extend(extra_scripts)
-    print('\n'.join(sorted(scripts)))
-
-class Distribution(object): pass
-class Feature(object): pass
-class Command(object): pass
-class Extension(object): pass
-class Require(object): pass
-def find_packages(**kwargs): pass

Now, let $HERE be the directory containing our fake setuptools/, and $PROJECT_ROOT be the project root directory containing setup.py. Run

PYTHONPATH=$HERE:$PYTHONPATH python $PROJECT_ROOT/setup.py

and bam! We get the names of all scripts on stdout.

My full automation scripts, including the Zsh main function virtual-install, can be found in modules/python/functions in zmwangx/prezto. I'm not including it here because it uses some custom helper, and it's just too long (200+ lines, but not very sophisticated). Happy virtualenving!

In rare cases, even installing a single package could land you in trouble. The classical example is installing the readme package on a case-insensitive filesystem (e.g., the default mode of HFS+). "Unfortunately" this has been fixed.↩︎

Catches when installing Windows 7 with Boot Camp

zmwangx@gmail.com (Zhiming Wang) — Tue, 29 Dec 2015 23:09:16 GMT

- -

I was looking for a use for my retired Mid-2012 Non-Retina MacBook Pro 13''¹, and unsurprisingly I figured that I would turn it into a OS X-Windows dual boot for some occasional gaming. I'm a CnC fan (not hardcore, but still), mainly for RA2/YR and TW/KW, and playing these inside Fusion is really a subsubpar experience. Due to the age of these games and their compatibility problems on Windows 8 and higher², I chose to shoot for a Windows 7 install.

Apple has a pretty thorough walkthrough in the support article Install Windows 7 and earlier on your Mac using Boot Camp. There are, however, some catches that I would like to collect and share in this post.

Win 7 ISO isn't available for download in the appropriate language (given your product key). This one sounds incredibly stupid... But it is a real problem at least for me and several others (just Google). I have a valid Win 7 Ultimate license from my institution, so I went to https://www.microsoft.com/en-us/software-download/windows7 to grab my ISO (just for fun; I already have the image). However, after verifying my product key, here's the list of languages that I'm asked to choose from, where English is apparently missing (!!!):
-
- -
da !@#$?
-
-
I don't know the solution to this problem. In my case I've archived English Win 7 Ultimate SP1 images (both x86 and x64) before, so I just proceeded with my old image.
FileVault. It is my belief that FileVault needs to turned off before partitioning the drive with Boot Camp.³
An error occured while partitioning the disk. That's the unhelpful message from Boot Camp. If you try to manually partition the drive with Disk Utility, you'll probably get a much more helpful message like Partition failed with the error: couldn't modify partition map because file system verification failed. Now the problem is obvious, and the solution is simple. Boot to single user mode and repair the filesystem with /sbin/fsck -fy, or safer, /sbin/fsck -f which might require interaction.
During Windows installation you'll obviously be prompted to choose a system partition at some point, and due to Boot Camp only formatting to FAT32, you'll get the message Windows cannot be installed to this hard disk space. Windows must be installed to a partition formatted as NTFS. This one is easy, just click "Drive options (advanced)" then "Format", which automatically formats the partition to NTFS. This is actually documented in Apple's walkthrough, but mortals do panic in face of error messages, so let's also note it here.
Even after formatting the Boot Camp partition, it is still possible to get the error Setup was unable to create a new system partition or locate an existing system partition. It this happens, check if you have any USB drives (other than the installation media) plugged in. In my case my Time Capsule was plugged in, and rebooting with it unplugged fixed the problem. The exact cause of the problem is unclear to me. Some say it's due to Master Boot Record limiting the number of partitions to four, but why the heck is my external drive counted towards that limitation? I'd go for Win 7 installer is just confused. Anyway, just unplug anything that's not needed during Windows installation.

Hopefully you're good after solving the aforementioned problems. If you followed Apple's walkthrough correctly, Boot Camp's setup.exe will be invoked automatically immediately after Windows finishes installation, and after a certain number of reboots your drivers will be up and running. Now you're ready to take control of your Windows. Install Chrome⁴ and Microsoft Security Essentials immediately, then hop right into the Windows Update hell to patch your four-year-old system. Of course, Windows Update being Windows Update won't be smooth — servers will be crowded as ever and just checking for updates will likely take forever, let alone downloads. After a semi-infinite amount of time you'll get your estimates (I got 212 updates). Click update and let Windows Update grind for hours. And wish yourself a good luck (that no update errors will occur — luckily I didn't get any).

By the way, the otherwise great Apple trackpad is almost unusable on Boot Camp Windows under any setting. I'm forced to use a mouse.

2.9 GHz i7 + Intel HD Graphics 4000 + 16 GB RAM + frigging slow 750 GB 5400-rpm spinning disk I've yet to replace.↩︎
RA2/YR used to have problems even on Windows 7, at least inside Fusion, so I used to play them in XP SP3 VMs; I've yet to try them with Windows 7 running on bare metal.↩︎
I'm not completely sure that this is necessary. I was greeted with partitioning errors initially which I thought was due to FileVault, so I switched it off (the actual process is much longer than "switching it off", since the whole disk has to be decrypted and rewritten), but as you'll see later, the partitioning errors were at least partly due to a slightly corrupt filesystem.↩︎
You can't even browse Microsoft's own websites with stock IE8. And IE11 is locked behind a hell lot of Windows Updates (even then it is crap). Doing Windows Update is like building up a tech tree.↩︎

Why I want lossless music on iTunes Music Store

zmwangx@gmail.com (Zhiming Wang) — Mon, 28 Dec 2015 11:15:45 GMT

- -

This is an impulse post after reading "Apple again rumored to be working on high-resolution audio".¹

To be clear, I'm no audiophile. I can't tell the difference between 256kbps AAC and lossless (maybe not even the difference between 128k and 256k), and my midrange to lower midrange equipments probably won't let me tell anyway. I'm certainly not a consumer of snake oil.

However, I still prefer to get everything in lossless, simply because "good enough" today is almost never good enough tomorrow. Fifty years later I'm most likely still wandering this planet, I and my music collection. I would be extremely regretful if I didn't archive the highest quality versions of my favorite tracks today, only to find them inferior-sounding fifty years later, which is a pretty realistic possibility given how fast technology advances.² Even today's lossless could be inferior-sounding in the future, but there would be no regret.

To be extra clear, I'm talking about lossless for archival purposes, so what I want to see is a lossless download option in ITMS.³ Streaming can be done in whatever good enough® sampling frequency and bitrate that's currently in use, since it's a one-off thing with no effects on tomorrow (and I don't give a shit about streaming and subscription anyway). Offering lossless downloads likely won't put much burden on Apple's infrastructure, since they already deliver much more bandwidth-demanding movies on the same channel. Moreover, albums on ITMS aren't much cheaper than physical CDs, while the cost is apparently lower than CD production, the audience apparently wider, and the chances of impulse purchases (especially of single tracks) much higher, so I would suppose such a move (delivering lossless on ITMS) won't considerably hurt record labels' profits either. After all, if they don't make it easy for consumers, many consumers will just pirate — it's way too easy to pirate music.

And I did see the MacRumors article a week ago. I even registered a MacRumors account, which I never bothered to do, just to comment on that article... It just didn't occur to me to write a blog post at that time.↩︎
You might be skeptical of my hearing when I'm in my seventies... But I could well be showing my favorites to someone with perfect hearing, say my grandchildren.↩︎
I know there are many online music stores that sell lossless music, but ITMS has the largest catalog in the world, and for many titles I care about, ITMS is still the only place in this country where I can make legal digital purchases.↩︎

Lesson on magic method access of Python new-style classes (from my failed Python3 port of Tomorrow)

zmwangx@gmail.com (Zhiming Wang) — Mon, 28 Dec 2015 00:47:05 GMT

- -

I know the title is formidably long, but I can't find something more accurate (and my homegrown mini CMS doesn't support subtitle), so please bear with me.

So, I have madisonmay/Tomorrow — "magic decorator syntax for asynchronous code in Python 2.7" — bookmarked for a long time¹ without ever trying it, because I simply don't write Python 2 code any more (except when I try to maintain compatibililty). I felt kind of strange that a ~50-line project with ~1000 stars on GitHub hasn't been ported to Python 3 already, so I gave it a shot just now.

I thought it would be easy:

Modernize the old-style class Tomorrow;
Replace __getattr__ with __getattribute__ for unconditional attribute routing, then make a few exceptions to prevent infinite recursion;
2to3 test cases;
Make meta changes, like removing the futures dependency.

However, after doing 1–3, I ran the tests, and out of the five test cases, three failed and one errored. I tried to isolate the problem, and ended up with the following piece of proof-of-concept:

class PassThrough(object):
-
-    def __init__(self, obj):
-        self._obj = obj
-
-    def __getattribute__(self, name):
-        if name == "_obj":
-            return object.__getattribute__(self, name)
-        print("Accessing '%s'" % name)
-        return self._obj.__getattribute__(name)

This snippet is valid in both Python 2.7 and Python 3, but here's the surprise:

>>> g = PassThrough(0)
->>> print(g)
-<__main__.PassThrough object at 0x10c662e48>
->>> str(g)
-'<__main__.PassThrough object at 0x10c662e48>'
->>> hasattr(g, '__str__')
-Accessing '__str__'
-True
->>> g.__str__()
-Accessing '__str__'
-'0'

In addition, here's what happens if you try to "pass through" a function:

>>> def f(): return True
->>> g = PassThrough(f)
->>> g()
-Accessing '__class__'
-Accessing '__class__'
-Traceback (most recent call last):
-  File "", line 1, in <module>
-    g()
-TypeError: 'PassThrough' object is not callable
-
->>> callable(g)
-False
->>> hasattr(g, '__call__')
-Accessing '__call__'
-True
->>> g.__call__()
-Accessing '__call__'
-True

As you can tell, although __str__ or __call__ may have been implemented through __getattribute__, and hasattr (which in turn depends on getattr) has no trouble finding them, they are not picked up by str or function call (...). At this point, one would suspect that this is due to str or function call only looking at the class instance's __dict__. Compare this to the behavior of an old-style class:

class PassThrough():
-
-    def __init__(self, obj):
-        self._obj = obj
-
-    def __getattr__(self, name):
-        print("Acessing '%s'" % name)
-        return self._obj.__getattribute__(name)

Now:

>>> g = PassThrough(0)
->>> print(g)
-Acessing '__str__'
-0
->>> def f(): return True
->>> g = PassThrough(f)
->>> g()
-Acessing '__call__'
-True

Note that magic method access is always routed through __getattr__.

After some digging, my suspicion was confirmed: indeed, for new-style classes, rather than invoking __getattribute__, the Python interpreter only looks for magic methods in __dict__. But is there a workaround for implementing something like the PassThrough class above? There's a nice answer on StackOverflow that uses a metaclass to "automatically add proxies for magic methods at the time of class creation", to quote the author. However, the thing about Tomorrow is that we don't have the result and don't know whatever magic methods it might have at class creation — after all, Python isn't a statically typed language. It is possible for programmers to offer hints, but then Tomorrow won't be as elegant and magical anymore. Therefore, unfortunately enough, Tomorrow isn't portable to Python 3 — at least not without a substantial hack that's beyond my knowledge, or a complete overhaul of its logic (haven't thought about that).

Pretty much since the beginning, I believe (the initial commit was from July 24 of this year). I don't remember how I came accross it though.↩︎

autoenv with auto cleanup

zmwangx@gmail.com (Zhiming Wang) — Sat, 26 Dec 2015 08:15:48 GMT

- -

I heard about kennethreitz/autoenv a long time ago. The idea of autoloading project-specific environment modifications is nice, but no auto cleanup after leaving a project was a showstopper for me.

Today, I took matters into my own hands and wrote a fresh Zsh implementation¹ with auto cleanup support. Check it out: https://github.com/zmwangx/prezto/tree/master/modules/autoenv.

As a quick promotion, let me show you two common examples.

First, inserting some local bin directory into the search path. This is easily done by a one-line .env, say,

autoenv-insert-paths bin libexec

This way $PWD/bin and $PWD/libexec are inserted to the beginning of the search path, which will persist until you leave the directory tree. That is to say, the inserted paths will still be available when you descend into subdirectories (and more specific .env's can even be stacked as you descend), but they will be purged as soon as you leave the tree. Clever, isn't it?

Secondly, exporting project-specific environment variables. The .env would look like

export HOMEBREW_DEVELOPER=not-for-the-faint-hearted
-
-autoenv-purge () unset HOMEBREW_DEVELOPER

where the body of autoenv-purge will be executed when you leave the directory tree. No more junk floating around.

Again, for more info, including detailed usage and customization instructions, please visit modules/autoenv in zmwangx/prezto.

This is not a re-implementation in the common sense. My little Zsh module is inspired by kennethreitz/autoenv and reminiscent of that older project, but I took nothing from there (in fact I didn't even read their source code). I also don't claim to support their entire feature set. For instance, kennethreitz/autoenv claims to be Foreman compatible, which includes turning on ALL_EXPORT. However, I don't think ALL_EXPORT by default is a good idea, so with my autoenv, if you want ALL_EXPORT you have to set it explicitly.↩︎

Regex flavor hell

zmwangx@gmail.com (Zhiming Wang) — Mon, 21 Dec 2015 00:03:03 GMT

- -

I write a lot of shell scripts, which means dealing with common *ix utilities a lot. I typically want my scripts to work on both OS X and Linux (or OS X + GNU utilities, which is my personal setup), which means writing commands that are understood in both GNU/Linux and BSD worlds. Unfortunately that's not so simple, because to do that I usually have to give up readily available functionalities (especially the vast collection of useful options typical of GNU utilities) and am constantly thrown back to the stone age that is POSIX, or a little bit more than POSIX.

Working with regular expressions is especially painful. Almost every implementation of every utility (with regex support) has its own flavor of regex. Most notably the big three: grep, sed and awk. GNU utilities of course come with GNU extensions, but they are nothing when aiming for compatibility. Ignoring GNU extensions, there's a way to turn on standard POSIX extensions (ERE) on sed, but unfortunately GNU and BSD use different flags: -r for GNU sed and -E for BSD sed. The two implementations of grep thankfully use the same flag -E to turn on ERE, but GNU grep, being a GNU utility and having to distinguish itself from its mundane counterpart, further implements -P,--perl-regexp — regexers' dream. It's there but I can't use it, except in an interactive shell. awk has more than two implementations and will be left out of this discussion.

Anyway, despite all these flavor issues, I can usually get away with BRE, although it's verbose and unreadable as hell (quantifiers in particular) and doesn't support alternation. I would be thankful if BRE is the end of the story, but it is not. There are more tools lurking around trying to sabotage scripters. find is a perfect example. BSD find, unsurprisingly, uses BRE by default with -regex and -iregex, and ERE may be turned on with the -E flag. GNU findutils find, however, tries to be helpful and future-proof by having a -regextype option:

-
Changes the regular expression syntax understood by -regex and -iregex tests which occur later on the command line. Currently-implemented types are emacs (this is the default), posix-awk, posix-basic, posix-egrep and posix-extended.
-

The Emacs flavor? You mean Elisp regexp? Okay fine, BRE — with few features other than grouping ($...$), quantifiers (* or \{n,m\}), bracket expressions and character classes — should still be pretty much compatible with Elisp regexp. However, the "Emacs flavor" isn't even the Elisp flavor. It's a stripped version specifically for findutils. In particular, there are *, + and ? but no curly braces quantifiers, so gone is the dream of writing even just mildly complex regexps that are compatible with both BSD find and GNU findutils find. By the way, in case you wonder, the POSIX find doesn't even have a -regex primary/operator...

What a cruelly realistic world we live in.

Spoiled by Retina, in less than a day

zmwangx@gmail.com (Zhiming Wang) — Thu, 17 Dec 2015 05:10:08 GMT

- -

I finally got a 15'' Retina MacBook Pro this morning to replace my 13'' mid-2012 non-Retina MacBook Pro, whose spinning disk has been getting increasingly slower (or so I felt).¹ Apparently this is a pretty significant landmark in my personal computing history, since I'm saying goodbye to both spinning disk and non-Retina display on my primary computing device.

The transition was initially smooth except for a few things. First, as a tap-to-click wizard I immediately turned on tap-to-click, but I had a hard time dragging things because it was too easy to trigger a force touch instead on the medium setting, and under the firm setting I could hardly force touch at all; in the end I just turned off force touch altogether, and haven't had any problem since. By the way, I was initially worried about the keyboard too but it worked surprisingly well for me, so no complaints there. Secondly, 10pt non-anti-aliased Monaco looks weird on Retina since it's no longer the beloved bitmap version. I turned on antialiasing and now it's no longer weird, but it felt totally different and I'm not sure if I like it (definitely not as much as the 10pt bitmap Monaco anyway). It's okay right now but I'll probably need to spend some time trying out different fonts. Obviously there are like-minded folks out there. Sad story.

So much for first impressions. Apart from Monaco, everything felt great, until I returned home (I was doing setup away from home to get a less shitty connection) and connected my 27'' external monitor. Holy crap, I couldn't believe my eyes. The dock icons — the first things I saw before launching anything — looked so blurry I couldn't stare at them for more than a few seconds. That was after staring at the Retina display for less than five hours. Not to mention PDFs; they look ultra crisp on the Retina display and ultra crappy on non-Retina — especially in Preview, which is a problem I've been aware of since Yosemite.² Moreover, the terminal font is more problematic than initially estimated — now I have a retina display and a non-retina one side-by-side, yet I can only set one font for my default profile, which will never satisfy both!³ This is so awkward I can't think of a solution. One obvious approach is to ditch the blurry 27'' and only work from the Retina 15'', but should I really let the large canvas sit idle? No idea. Or should I get a 4K external display? First, a 4K display at 27'' still can't rival the pixel density of 2880x1800 at 15.4'' (Apple ships 5K at 27'' for a reason). Secondly and more importantly, I don't have the budget for such a thing after throwing money at an expensive 15'' rMBP (with 512 GB SSD)...

Transition periods are always awkward, I guess.

12/17/2015 Update. After more than a full day's use, I actually quite love 10pt Monaco on a Retina display. I tried various fonts, including Menlo, Consolas and so on, but none of them has that whimsical feeling of Monaco. Hopefully the font is stuck now.

12/28/2015 Update. A dozen days later, I can hardly look at 10pt Monaco on a non-Retina screen anymore, antialiased or not, especially not in bold. Mind blown.

I haven't got the nerve to replace the hard drive myself, since it looks so much more complicated than upgrading the memory.↩︎
PDFs looked so horrible in Preview (and TeXShop, my LaTeX previewer, which only serves a niche) that I often viewed them in browsers (!!), where text at least looks reasonable (on par with slightly blurry text elsewhere). PDF Expert came along and kind of made the situation better for non-Retina.↩︎
Provided that I'll religiously stick to 10pt non-anti-aliased Monaco on non-Retina.↩︎

Safeguarding git repos against accidental rm

zmwangx@gmail.com (Zhiming Wang) — Tue, 08 Dec 2015 08:17:39 GMT

- -

Everyone who has spent a sizable portion of their life in terminals has experienced that "oh shit" moment: you realize what you've done immediately after you've hit enter, but it's already too late. And needlessly to say, many of those are associated to accidental rms.

I just had one of those moments. I was going to delete a subdirectory of ~/.config, but hit return prematurely, and the command line ended up being rm -r ~/.config. Imagine the horror one second later. Fortunately I was saved by the read-only objects in .git, which triggered prompts; however, damage was already done, to some extent. I had to reinit the repo and do a hard reset, and a corrupted submodule was in my way (it blocked my attempt of git reset --hard) which I eventually had to completely remove and re-add. In the end everything was recovered (hopefully) and back to normal, but this episode was definitely not great for heart health, which led me to rethink rm.

I've tried several safer rm solutions before. The first and obvious is to alias rm to rm -i, but having to answer dozens of prompts a day (or more) is agonizing and unproductive. I've also tried trashing, but a nonempty trash can makes me sick, so not for me either. I also used safe-rm for a couple of months, but without supplying my own blacklist (I have none to be blacklisted), I've never hit the default blacklist; apparently I'm not stupid enough to mess in system locations, so this won't really help much. Fortunately though, this time I might have found a very good solution for myself.

The idea is to protect all git repos. Git repos¹ are among the most valuable assets of programmers, and they have the nice property of not being completely removable without -f or --force (the work tree of a submodule, where .git is a regular file containing the relative path of the git dir, can be removed without --force, but we don't want to damage submodules anyway, so let's not single them out). It's unlikely that we would intend to remove a repo directory without specifying -f or --force, so let's just reject all such rm calls.

The wrapper is very easy to write. Here's one implementation for Zsh with support for both GNU coreutils and BSD rm.

rm () {
-    setopt localoptions noshwordsplit noksharrays
-    local args_backup force node
-    set -A args_backup $@
-    while :; do
-        case $1 in
-            --force|-*f*) force=1 && shift;;
-            --) shift && break;;
-            -*) shift;;
-            *) break;;
-        esac
-    done
-    for node; do
-        # -f, --force hasn't been specified && node is a git repo
-        [[ -z $force && -e $node/.git ]] && {
-            printf "\e[31m'%s' is a git repo -- won't remove without the -f or --force option\e[0m\n" $node
-            return 1
-        }
-    done
-    command rm $args_backup
-}

Personally, I stick it into a Prezto module available from my fork. Hopefully it will serve me well this time round.

In this article, "repo" stands for the work tree of a repo, unless otherwise noted; the actual repo with git objects is referred to as "git dir".↩︎

Bash function exporting fiasco

zmwangx@gmail.com (Zhiming Wang) — Wed, 25 Nov 2015 23:38:13 GMT

- -

Bash is the only major shell (and the only shell that I know of) that implements function exporting. By now everyone should have heard of this feature, I suppose, after the publicity of Shellshock last year. I was personally introduced to it while writing parallel processing scripts with GNU Parallel (long before Shellshock), and it seemed useful and clever at that time. Back then I often wondered why it didn't make its way into Z shell. However, now that I'm much more seasoned in shell scripting, I can see why and how this feature is troubled and of debatable value.

Two problems lie at the heart of function exporting:

As always, everything clever comes at a cost;
Code execution from untrusted source.

Regarding the first problem, the cost of function exporting is to mess with the environment, in a very hackish way. The environment was designed to hold data, not code, and we're not in the utopia of Lisp; but bash forced its way through. Pre-shellshock, exported func was stored as func=() {... in env; post-shellshock, it was first BASH_FUNC_func()=() {... (which didn't entirely fix the issue), and then BASH_FUNC_func%%=() {....

The second problem doesn't need much explanation — shellshock it was. It has been extensively documented elsewhere, so I'll just succinctly comment that to load exported functions into a subshell, function definitions have to be retrieved from the environment and executed (again because we're not in the utopia of Lisp¹), and loading is done passively from the subshell user's point of view, hence the code execution bug(s). The bug(s) has(have) allegedly been fixed, but code execution (presumably with the appropriate safeguards now) still can't be avoided altogether, so just like a sanitized eval, it would still wake you up at night.

Well, if that's all I have to say, I wouldn't have started this post today. The thing that's bugging me is another issue I've found recently that's entirely avoidable, yet upon which we'll probably never see light ever after due to a combination of factors.

It started with this question on SO. While troubleshooting I quickly noticed that a Bash-emulated sh imports those BASH_FUNCs from the environment:

> bash -c 'func () { echo "exported function loaded"; } && export -f func && ln -sf /bin/bash sh && ./sh -c func'
-exported function loaded

It gets worse when the function isn't Bourne shell compatible (e.g., when it uses process substitution):

> bash -c 'func () { cat <(echo hello); } && export -f func && ln -sf /bin/bash sh && ./sh -c func'
-cat: <(echo hello): No such file or directory

That's surprising but not scary enough, because if you're not a fool you won't call func in sh anyway. However, if you're unfortunate enough to be dealing with /bin/sh on OS X (bash 3.2 under the hood, modified by Apple or not I'm not sure), then all hell break loose:

> bash -c 'func () { cat <(echo hello); } && export -f func && /bin/sh -c :'  # OS X only
-/bin/sh: func: line 0: syntax error near unexpected token `('
-/bin/sh: func: line 0: `func () {  cat <(echo hello)'
-/bin/sh: error importing function definition for `func'

Note that we're actively doing nothing in sh, yet we get all these syntax errors from loading func. This happens to every invocation of sh, and as you might expect, there are no shortage of programs that are either sh scripts (e.g., fasd) or have internal sh calls (e.g., GNU Parallel²). A single export of a Bourn shell incompatible function will haunt you through the entire session. Oops.

As I said, I don't know if the displayed error messages are due to Apple's modifications (anyone willing to look at the source code?), since a symlink named sh to /bin/bash doesn't print error messages, but instead load the wrong function, which is almost as bad but less annoying to innocent users. At any rate, it's not even worth reporting, either to GNU or Apple, because we're stuck with bash 3.2 for /bin/sh forever (thank you GPLv3), and it takes a hell of a vulnerability like shellshock to get a small update out of Apple's hands. We can install newer shells to /usr/local as much as we'd like to, but /bin/sh is simply the final word for many tasks involving the shell. Yet it's stained by this troubled bash-specific feature, and it's not going anywhere. So sad.

I'm not commenting on the security of Lisp.↩︎
04/14/2015 Update. GNU Parallel is no longer haunted by this issue since 3d919c6.↩︎

We need a programming keyboard on iOS

zmwangx@gmail.com (Zhiming Wang) — Sun, 15 Nov 2015 10:17:05 GMT

- -

We do. If you ever tried to say something on GitHub (web) or StackOverflow (web or app) on iOS, you'll probably agree with me. The stock keyboard (or any third party keyboard that I've heard of) is simply awful at this. Typing on iOS software keyboard is unpleasant enough to begin with, but behold:

Auto"correct" messes up everything as fast as you can type, which isn't really fast anyway; might as well call it autorot.
The backtick is a click plus a loooong click (on the single quote key) plus another click away. Good luck typing code in Markdown,¹ especially if you use GFM fenced code block like all of us do.
Brackets, curly braces, the underscore, the pound, etc. are all three clicks away.

The solution is pretty obvious actually. I don't know about smaller phones, but the software keyboard on a landscape iPhone 6 Plus has four rows, which takes up about 40% of vertical screen estate, and it has fourteen keys in the top row. With a little bit of effort it can be made into a five-row, full-sized keyboard (without arrow keys perhaps) without taking up a ridiculous amount of space. Since the horizontal 6 Plus could handle it, any iPad should be able to handle it too; definitely shouldn't be an iPad Pro-only luxury. Turn off autocorrect on top of that, and you get a decent programming (or better put, programmer-oriented) keyboard.

This is merely a rant, but it would awesome if anyone sets out to make one.

To be fair, typing BBCode is even worse. Unfortunately that's what Ars Technica use, and I've given up on commenting there.↩︎