Installing Python 3 Alongside Python 2 on OS X (Yosemite)

This was actually pretty easy. Here’s what I did:

  1. Opened a new Terminal window that wasn’t in a virtualenv.
  2. Ran brew update
  3. Ran brew install python3
  4. cd ~/.virtualenvs (the folder where my virtualenvs are)
  5. virtualenv -p `which python3` data3 to create a new virtualenv using Python 3
  6. workon data3 (this is from virtualenvwrapper)
  7. pip install "ipython[all]"

Update: I've been trying out pyenv, which is forked from my much-beloved rbenv. It seems to work fine with virtualenv via pyenv-virtualenv, but I could not get pyenv-virtualenvwrapper to work. I'm using this now to run an updated version of Python 2 without messing with my system Python, and it seems to work fine.

IPython and Jupyter Notebooks: Automatically Export .py and .html

[Updated 2016-03-04 to support Jupyter 4 notebooks – see below.]

IPython notebooks are stored in a format that is not particularly human-readable and doesn’t work well in version control.

One way to solve this problem is to automatically export the code from IPython notebooks into a vanilla Python file after each save.

It’s also useful to automatically generate a HTML file of the notebook on each save. This can be done manually in Jupyter (File > Download as > HTML), but if you always want this, doing it automatically is much easier.

Use the following code to automatically save a .py and a .html file when you save a notebook in Jupyter. These two files will be saved in the same folder as the parent .ipynb file.

First, run ipython locate profile default, which will give you the path to save the following code in.

Save the code below in this folder as

Now, run ipython notebook. You will see an error message in the terminal if there are any syntax or runtime errors with If everything looks good, go to your web browser, open a notebook, and click the Save/Checkpoint button (it’s the floppy disk icon in the Jupyter toolbar). You should see a .py and a .html file appear alongside your .ipynb file.

Update for Jupyter 4 notebooks

After the big split between IPython and Jupyter, and the accompanying update to Jupyter 4.x, the config file above no longer works. Instead, put the following in a file saved at ~/.jupyter/ to achieve the same thing:

This code could be DRYer, but it does work.

Hide Code When Sharing IPython Notebooks

When sharing IPython notebooks with non-programmers, I don't want to cause confusion by including a bunch of code.

To this end, I wrote a snippet of Python and JavaScript that can be put in a code cell at the top of a notebook. When a notebook with this code is exported as HTML or viewed with, code blocks will be hidden by default. A "Toggle code" button is also added in case you want to see the hidden code.

Here is the code snippet to put in the cell at the top of the notebook:

And here is an example of how this works in

(The source code for this example and the accompanying HTML output are available here.)

LauchBar 6 Action to Post to iDoneThis

This is not for the faint of heart because it's difficult to manage Ruby dependencies (i.e. deal with bundler) in LaunchBar actions.

You will need to run bundle install --standalone for this to work. If you don't know what that is, you should probably stop here. If there is demand, I can probably create a dependency-free version of this that you can install with one click (tell me on Twitter if you want this).

In any case, once you get this action set up, you can activate it in LaunchBar, type some text, hit enter, and it will post it to iDoneThis. After it's posted, hitting enter again in LaunchBar will open up the "done" you just posted in your web browser, or you can hit escape to hide LaunchBar.

Below is the code and the settings you need in the LaunchBar Action editor. Again, this can't be packaged up as something you can install directly because you need to install the gems in the Gemfile and set the interpreter in the hashbang correctly for your system. If you can do all this easily, it should only take you a few minutes to set up.

JavaScript for Automation: getting the clipboard

In AppleScript, it is possible to get the system clipboard with something like the following:

set s to the clipboard as text

If you run this in Script Editor with “clip” on the clipboard, you should see this:

In JavaScript for Automation (JXA), you can achieve the same thing with the following code:

app = Application('System Events');  
app.includeStandardAdditions = true;  
s = app.theClipboard();

You can also use:

app = Application.currentApplication();  
app.includeStandardAdditions = true;  
s = app.theClipboard();

Beginner GIS: spatial querying to find Census Blocks inside urban areas

I know literally nothing about GIS, but I need to figure it out because I need to do some spatial querying. Specifically, I need to find all the Census Blocks that are in a given urban area. This is a  I'm documenting it here for anyone else who needs to get into GIS and doesn't know where to start.

QGIS is the application of choice here. It's like open source ArcGIS. ArcGIS is the Microsoft Office of GIS.

Setup on OS X

You need matplotlib from this page.

Download QGIS (open source ArcGIS). You need the GDAL and NumPy installer from this page as well.

Opening Census shapefiles

I want to work with this shape file from the Census for urban areas ( Download, unzip. The in QGIS go to Layer > Add Layer > Add Vector Layer...

Looks like it works.

Adding another layer

I want to look at Census Blocks in a specific urban area. For example, Abbeville, LA (UACE=00037) as defined here.

Same deal as before for adding the layer ( Then do this to see them overlaid:

Finding Census Blocks in the urban area

First, find Abeville using the Query Builder (cmd-f):

This will hide all the other urban areas from the map. Go back in to the Query Builder and delete the query to get everything back.

Select the urban area of interest using the "Select Features" toolbar button (yellow square with a mouse pointer).

Now, to find the blocks in the urban area. This is done with the Spatial Query plugin. Enable it by going in the Plugin Manager in QGIS, searching for "spatial" and toggling the checkbox. You should have this icon in the sidebar now:

Then run this query by clicking that button and selecting the appropriate items in the dropdown (in the screenshot, the results are shown too):

Create a new layer based on the spatial query:

Then right click on that layer created from the spatial query, choose "Save As..." and save it as a CSV. This will export the attribute table for the layer, which is essentially a list of all the Census Blocks in the specified urban area!

Now, on to automating this for all the urban areas in the US.