Phil Elson - Software | Science | Python

Introducing multistage-build: a proxy build-backend with customisable pre- and post-build hooks

2026-09-22T06:00:00+01:00

Sometimes you want to customise your Python project's build with a pre-processing or post-processing step that isn't built-in to the PEP-517 build backend that you happen to be using. In my day-to-day work I have multiple such needs, including:

The ability to run tasks before the actual build takes place, including things like code generation (I have one case where I generate a synchronous module from an equivalent async module, meaning I can maintain a consistent API for both approaches).
The ability to inject metadata into a wheel after build (I was investigating the Requires-External core metadata).

In the past, you would have used custom commands in setuptools/distutils. With pyproject.toml, build backends choose the functionality they wish to support, but in general there are no standardised hooks that you can reliably use across all backends: a hook that exists for hatchling won't help you if you happen to be using setuptools or flit. Building a whole new build backend just to trigger a simple pre- or post-processing step doesn't make any sense, so you are left with continuing to use setup.py alongside pyproject.toml - this has its own downsides, and is also coupled tightly to the build backend you are using. This is where multistage-build can help.

multistage-build: a PEP-517 build backend which proxies your real build backend

Inspired by tools such as setuptools-ext, as well as having implemented another specialised build backend in the past, I figured that this could be generalised, and that building a generic multi-stage backend was possible thanks to PEP-517 exposing the backend as ordinary Python functions.

In multistage-build you declare your build backend to be multistage-build, and then configure multistage-build to use another (real) build backend to do the hard build work. However, you can also declare that for particular phases, some other code should be run on either the input for the PEP-517 build hook in question, or the output of the hook. This means that you can, for example, have a post-build job that modifies the wheel before it is returned to whoever called for the build:

[build-system]
requires = ["multistage-build", "setuptools"]
build-backend = "multistage_build:backend"

[tool.multistage-build]
build-backend = "setuptools.build_meta"
post-build-wheel = [
    {hook-function = "my_hooks:add_requires_external", hook-path = "."},
]

[project]
name = "some-project"
version = "0.1.0"

The my_hooks module lives in the same directory as pyproject.toml. The hook-path mechanism is analogous to the PEP-517 backend-path mechanism, and allows code local to our project to be run. It only needs to be a plain Python module:

# my_hooks.py
def add_requires_external(wheel_path):
    # open the wheel, patch its METADATA file, write it back
    ...

When pip, uv or build ask the backend to build a wheel, multistage-build delegates to the configured build backend (in this example, setuptools), and then calls add_requires_external with the resulting wheel path before returning the result to the build frontend. From the frontend's point of view a normal wheel has been produced; multistage-build just handed the wheel to your hook first.

Pre-hooks also exist, and they run before the real backend is called, receiving the arguments the backend was about to see. This is handy for things like code generation, or anything else that has to happen to the source tree before the real build backend does its thing.

A useful side-effect of this design is that the hook itself is not tied to any particular backend. If you choose to move from setuptools to hatchling, or to some other build backend, you simply change the [tool.multistage-build] build-backend line, and the hook stays in place.

Respecting the PEP-517 contract

Multistage-build only wraps the real backend, and does not enforce any consistency between the hooks you register. That is entirely the responsibility of the hook creator. The most obvious invariant to preserve is metadata consistency - if a post-build-wheel hook adds a field to the wheel's METADATA, then a frontend that later calls prepare_metadata_for_build_wheel on the same project should get the same field back. That means also registering a matching hook on post-prepare-metadata-for-build-wheel (and, for editable builds, on post-prepare-metadata-for-build-editable), so that whatever a frontend can see up-front lines up with whatever eventually ends up in the wheel.

It is conceivable that another layer, built on top of multistage-build, could manage metadata modifications for you in a standards-compliant way. That is out of scope for the multistage-build backend itself, whose focus is purely on being a simple PEP-517 function proxy.

Hooks as plugins

There is also an entry-points-based mode, which is closer to a proper plugin system. Rather than declaring a hook in each project's pyproject.toml, a helper tool can advertise itself:

# in the helper tool's own pyproject.toml
[project.entry-points.multistage_build]
post-build-wheel = "my_mod:the_hook"

Any project that lists that helper tool in its [build-system].requires, and uses multistage_build:backend, will then pick up the hook automatically. No additional configuration is required in the consuming project's pyproject.toml, and it lets a hook be installable as a build dependency, rather than having to be declared per-project.

As a hypothetical example, a multistage-mypyc library could register a pre-build-wheel entry point that runs mypyc over the source tree and drops the resulting extension modules where the underlying backend will pick them up. Any project wanting mypyc compilation would then add multistage-mypyc as a build requirement, choose multistage_build:backend, and get the same behaviour with any of the mainstream backends. hatch-mypyc already does this for hatchling. At the time of writing I am not aware of an equivalent that works across build backends.

One criticism I have of this approach is that it can be a bit opaque as to the build processes that are enabled for a build. Another is that there is no control of the order of these entrypoint registered hooks. I don't yet know if I think this is a deal-breaker, or if the convenience of simply adding a build-time requirement to automatically trigger new behaviour is a price worth paying. I've been playing with this approach a bit, and plan to roll it out to a number of my projects to get a feeling for how it plays out. Feedback welcome!

Next steps

The build backend is working robustly, and I am very happy with the outcome. I think this unlocks a lot of potential for hooking into the build phase of a project, and I'm using it in multiple places already. If this were to become a popular thing, it is conceivable that this could be a built-in concept in the PEP-517 mechanism, rather than requiring a dedicated proxy backend, but I think that is a long-shot.

I'd be interested to know your thoughts on this - is this against the grain of the PEP-517 spec? Or is this a useful step which gives us more atomic build capabilities and moves us towards looser coupling to specific build backends?

Updating this blog for 2026

2026-08-18T12:00:00+01:00

Since my last post more than seven years ago I have changed job, moved to a different country, and started a family. My interests have evolved, and my work capacity is also different: less time (kids), but making more of what there is (more experienced, and AI coding assistants helping with prototyping). What hasn't changed over those years is the spark for new ideas which push forward my personal and professional interests, as well as my desire to share those ideas (normally in the form of prototypes).

I have a backlog of things that I have been meaning to get around to writing about, and so I intend to bring this blog back to life and reframe it a little to reflect my current reality.

The pre-2026 scope of this blog

Before refreshing this blog in 2026, I had the following blurb on the main page:

As a scientific technical lead at a leading natural sciences research institute, I'm fortunate to contribute to a body of tools that empower researchers and data scientists across a broad spectrum of fields.

I'm driven by the idea that the modern scientific method is now almost entirely dependent upon software, and that, in order to facilitate transparency and reproducibility of results, scientific software must increasingly become open source.

As well as deep knowledge of the rapidly changing technology landscape, development of useful scientific tools depends upon a solid understanding of the underlying scientific problem space. For these reasons you will find these pages cover the broad spectrum of topics from technology through to scientific.

Today, even though I still believe in what I wrote, I find it a bit highbrow and lofty, especially for a blog where I just want to share ideas and prototypes.

What are my goals for refreshing my blog?

I have reached a point in my career where I no longer feel the need to project my successes in the way that my old blog framed it. This is a liberating (and, I fully acknowledge, privileged) position, and it means I can focus more on the fun that I have with technology, rather than always having to aspire to a level that was on-brand.

First and foremost, I want this blog to be a place where I can share ideas and prototypes on all topics that interest me. In the past I had gathered my writing into Announcements, Field notes, Articles and Hints & Tips. I want to simplify this, perhaps keeping a distinction between "stuff that I wrote and want to share with my readers" and "stuff that I found useful for myself, and might want to reference it again in the future".

Where I am now

Since 2019, I have been leading the introduction of Python for accelerator control at CERN. There was already Python in use before I arrived, but it was mostly ad-hoc and inconsistent. I formalised the support and designed much of the core infrastructure and controls-system libraries that many of the high-level controls applications used in operations now rely upon.

Delivering technical solutions in a scientific domain, and building communities around them, has always featured heavily in my career. CERN is a perfect fit for me, and I love the combination of working with a broad range of people who use and appreciate the tools I provide, on genuinely exciting scientific endeavours.

Day to day, I write less numeric/scientific code than I used to, and spend more time enabling others (physicists, operators, hardware experts) to use Python effectively. Beyond the usual data analysis infrastructure, the primary use of Python is for PyQt-based operational applications in the CERN control room.

Technology wise there is a lot going on, but at a high level it breaks down into two groups.

Core Python infrastructure:

CPython distributions that are the basis for long-term operational use. The accelerator schedule drives us to maintain Python versions for the duration of a ~5 year run, and in 2026 we have Python versions ranging from 3.7 through to 3.14.
A Python package repository (released under the simple-repository GitHub org) that works inside an internet-less operational network.
Tooling for deploying applications so that they run robustly in the control room.

Controls-system libraries:

Talking to lower-level services, and to networked hardware devices, with extensive use of asyncio and an increased reliance on typing and static analysis.
Bridging Python to Java: JPype for in-house APIs, Py4J for the Spark stack.
Bridging Python to C++ via PyBind11.
REST client libraries for the higher-level controls services.

Increasingly Python is being seen as the primary user-facing high-level controls language. As of today, there is no operational support for server-side Python, though there is already a fair amount of FastAPI (and even some Flask) in use. Over time, I assume this will grow, and formal support would follow. A clear trend is towards web dashboards backed by data served via FastAPI, though I don't see more comprehensive applications moving from PyQt to web in the near future.

Outside of work

Away from work I have been doing home automation and small hardware projects for many years now, mostly around home-assistant and ESP8266/ESP32. Whilst the scale and the needs are quite different, there are some interesting echoes between that world and my professional one that I like to draw on.

AI coding assistants

Finally, the arrival of AI coding assistants (Claude in particular) has been genuinely exciting and a game-changer for me. It opens up ideas that previously were not viable, simply because of the up-front investment they would have needed. I have always been a prototype-making sort of person, and AI supports that mode of working very well. There are some interesting social challenges around AI usage, but I think not using AI is hard to justify in 2026. Finding the sweet spot between AI and human effort is something we are all still figuring out: for prototypes, I think vibe coding works out quite well, but for real code that you want to maintain, I don't believe that fully automated agentic use of AI produces a satisfactory outcome, neither in terms of quality, nor in human understanding.

In the same vein, I will use AI to help me author this blog, but always in a collaborative (human + AI) manner, so that the posts here remain production-quality with me taking full ownership of what is written.

What's next

Thus, with the slightly adapted scope and focus, and a long-overdue personal update off my chest, I hope that this refresh produces some insightful articles that can be enjoyed by readers old and new, and I look forward to sharing the long list of ideas that I have on the backlog.

cf-units get (La)TeX unit repr... and a UDUNITS-2 parser to boot

2019-02-15T00:00:00+00:00

I recently worked on a feature to allow cf-units to produce a TeX representation of a unit so that it can be quickly copy-and-pasted into a LaTeX document, or simply visualised nicely with matplotlib's LaTeX writer.

It turns out that "doing it right" was quite involved, and has ultimately led to some really interesting opportunities for those who use UDUNITS-2. This article explains what was involved, and briefly discusses some of those opportunities.

tl;dr

If you came here to find out how to turn your UDUNITS-2 compatible unit into a (La)TeX string, you need python 3 and cf-units (version >=2.1):

>>> from cf_units.tex import tex
>>> print(tex('microW per s'))
'\frac{{\mu}W}{s}'

Read on if you want to find out about the journey to this functionality...

Motivation

The cf-units implementation goes out of its way to ensure common Earth Sciences units are preserved as much as possible when interfacing with UDUNITS-2. For example, specific humidity is the density of water vapour (kg/m3) divided by the density of all air, and is commonly expressed in "grams per kilogram". It seems that expressing the unit as "grams per kilogram" rather than a dimensionless unit helps to avoid doubt about whether the quantity is a mass ratio or a volume ratio, and so I've had a number of people state that expressing the unit as "g/kg" is preferred.

Unfortunately, when parsing the unit "g km-1" UDUNITS-2 does the reasonable thing of simplifying it to its dimensionless form. There is no way for UDUNITS-2 to hold on to this information - the parsing is tightly bound to the units-system in operation. In order to turn "g/kg" into a TeX representation, we will therefore first need to parse the string ourselves.

UDUNITS-2 parsing

The UDUNITS-2 includes a Backus–Naur form of its unit grammar, so you might think that we can use this directly in a parser generator such as bison. Unfortunately, the documented grammar doesn't actually reflect what is implemented in UDUNITS-2 (issue) as the actual rules are much more sensitive to state than they first appear.

For example, in UDUNITS-2 m2.3 is equivalent to the unit 3*m^2 whereas 2.3m is 2.3 * m, and so we can't tokenize floats unless we know the context of what we have already seen.

Writing a parser often involves two independent steps: tokenizing the input with a lexer, and then parsing the tokens using a grammar (for example, if parsing a spoken language the lexer would turn characters into words, and the parser would turn words into grammatically correct sentences). In the UDUNITS-2 case it would simplify the grammar rules if we can encapsulate some of the necessary state into the lexer's rules. It is for this reason (combined with ease of use and desired target language for the generated parser) that I choose to use ANTLR as my parser generator.

ANTLR

ANTRL is "ANother Tool for Language Recognition" and is "a powerful parser generator for reading, processing, executing, or translating structured text or binary files". Whilst it is written in Java, it can generate parsers for a number of desired languages, including Python.

It took me several days to become familiar enough with the tool to use ANTLR to its full potential (coming from a place of having never written a formal grammar before), but I ultimately was able to take the documented UDUNITS-2 grammar and translate it to a ANTLR (v4) compatible form.

The most important thing that I did whilst developing the grammar was to have a comprehensive test suite of cases - I ended up with several hundred forms of unit supported by UDUNITS-2. Not only did this help me quickly learn the ANTLR ropes, but I was also able to spot performance regressions when handling left-recursive cases (for example, meters second-1 is actually two separate units joined together by the rule unit: base_unit | (unit whitespace+ base_unit)) (actual rule).

One issue I did encounter with ANTLR and its handling of state (known as lexer modes) is that there was a lot of duplication that would result in a grammar that was very easy to forget to update in all places. For this reason, I ended up templating the lexer rules using Jinja2 - allowing me to define the concepts once and apply them in several places (example and jinja code). It seems that this might become a core part of the grammar language in the future, but for now the Jinja2 approach worked well for me.

Parse tree into an abstract representation

Once I'd produced the ANTLR rules, it is easy to turn this into a parser in any of the supported languages. In my case I chose the Python 3 target.

For the lexer:

java -jar <antlr-jarfile> -Dlanguage=Python3 udunits2Lexer.g4 -o ./parser/

And I chose to implement the parser itself using the visitor pattern rather than the listener pattern, this choice made it a little bit easier for me to create a context-free abstract representation later on:

java -jar <antlr-jarfile> -Dlanguage=Python3 udunits2Parser.g4 -no-listener -visitor -o ./parser/

With all of this in place, I went about producing a clean abstract representation of the unit that I could produce based on the parse tree. I implemented basic nodes for an expression graph, and a Visitor to construct the graph.

Finally, I had a simple unit representation that could be used for any purpose - including converting to (La)TeX. As an illustrative example, the unit kg m / s^2 is shown below:

Abstract graph to (La)TeX

Once I had the graph in the form illustrated above, converting it to TeX is relatively straightforward. Out trusty friend the Visitor pattern makes light work of it with so few node types.

As an exercise for the reader, you may like to have a go yourself... Take the following expression produced with cf-units expression nodes:

import cf_units._udunits2_parser.graph as g

expr = g.Divide(lhs=g.Terminal(content='m'), 
                rhs=g.Terminal(content='s'))

Using the knowledge that Terminal nodes have a ".content" attribute, whereas binary nodes have a .lhs and a .rhs attribute, convert the expression into the following string:

\frac{m}{s}

You may wish to use the Visitor base class provided (available as g.Visitor in the above code).

If you want more interesting cases, you can generate arbitrary expression graphs from UDUNITS-2 compatible strings:

>>> import cf_units._udunits2_parser as p
>>> p.parse('kg.m/s-2')
Divide(lhs=Multiply(lhs=Identifier(content='kg'), rhs=Identifier(content='m')), rhs=Raise(lhs=Identifier(content='s'), rhs=Number(content=-2)))

Solution: The actual implementation to turn an expression graph into TeX form is available here.

Opportunities of a working UDUNITS-2 grammar

CF conventions

The Climate-Forecast (CF) conventions are designed to promote the processing and sharing of files created using the NetCDF API. The conventions are the common denominator of much of the open source Earth Sciences stack, so if you've ever used any Earth Sciences NetCDF data or tools, the chances are you've used the CF conventions.

It has always struck me as odd that, when referring to metadata regarding the units of measure, the CF conventions defer to an implementation (UDUNITS-2) rather than citing a specification or formal grammar.

Perhaps the grammar produced for this work could be adopted by the CF conventions, and UDUNITS-2 move to becoming the reference implementation rather than the standard itself?

Online unit validation

Once an ANTLR grammar has been defined, it is relatively easy to generate a parser for any of ANTLR's runtime languages. This includes Python, C++, and JavaScript targets. We could therefore quickly produce a browser based tool to validate UDUNITS-2 units on the client-side.

I put together a quick proof of concept that takes UDUNITS-2 compatible unit specifications and parses them entirely client-side. In theory this could be the basis for implementing a UDUNITS-2 like system natively in the browser. If you'd like to learn more tweet a request (@pypelson) for a write-up.

Alternative simplification and factoring

UDUNITS-2 comes with unit simplification and factorisation, but there are tools out there that are even more powerful for this purpose. Once such tool is SymPy. Given the ease of traversing the expression graph we now have for our unit, constructing a SymPy expression is relatively simple. I'm confident that we could combine this with the XML files that are shipped with UDUNITS-2 to entirely replace the UDUNITS-2 implementation, should we need to do such a thing.

A less controversial option is to improve the handling of offsetted units. For example, degrees Celsius (°C) is defined by UDUNITS to be Kelvin + 273.15, yet offset units such as this lose their offset when squared - if we were to use a tool such as SymPy to preserve the quantities we'd be able to benefit from its strong mathematical heritage and preserve these quantities.

Python 3 only

This is the first feature that has been added to cf-units which is Python 3 only. The decision was pragmatic as cf-units isn't quite ready to drop legacy Python (though it is coming) but it was more work to get ANTLR to produce a Python 2 compatible parser.

Having syntactically invalid Python 2 code in a repository that supports Python 2 comes with a few challenges. We must ensure that the Python 3 code is never imported (else you get a SyntaxError, not an ImportError). This involves some care when handling packaging and test discovery. In the latter case it was necessary to define a conftest to prevent pytest from importing the Python 3 files in order to discover tests it can run.

Conclusion

It has been an interesting journey solving this problem. Having never written a full-blown parser before the ANTLR experience has been a real journey of discovery - I started out writing more and more parser rules to cover the corner cases, until I realised that a refactor of the lexer rules would make most of them redundant (a bit like how the more you know of the SciPy stack, the shorter your code gets). I believe the UDUNITS-2 library, and the Python wrapper cf-units, are great choices for the CF conventions, but that it would also be healthy for the conventions to reference a specific grammar and base unit set, and for the conventions to cite UDUNITS-2 as a reference implementation rather than the specification itself.

Have a go with cf-units and its (La)TeX generation - there are definitely still cases that the grammar doesn't support, but on the whole it has excellent coverage. If you have other ideas about how we can make use of this new capability, let me know (comments or via twitter).

Investigating containment testing on LFRic's cubedsphere

2018-05-29T00:00:00+01:00

UGRID and LFRic: Polygon mesh intersections¶

I'm aiming to investigate whether it is possible to use the existing python tools to compute intersections of cells definied on the Met Office's LFRic cubesphere. A (possibly out of date but still enlightening) presentation about the LFRic datamodel can also be found here.

I'll start in this notebook with baby steps by using a c12 mesh (6 faces, 12x12 on each face), which is just 864 faces. Naturally, this is several orders of magnitude fewer than the expected mesh of c1024, which is ~6,300,000 faces, but you have to start somewhere, right?

For reference, the LFRic mesh is a cubed-sphere with quadrilateral, approximately equal area, cells.

from IPython.display import Image
Image(filename="cubesphere_ortho.png")

The go-to tool for geometry based predicates in Python is Shapely, which I'll combine with Cartopy to add a quasi-spherical geometry treatment. I'll not yet move on to real spherical treatment until I know the limitations of these tools for this data.

My ultimate aim is to develop a familiarity with UGRID data so that I can design an Iris based data-model that can be used for easily interacting with the LFRic output.

Here goes...

To start with, I've been given a NetCDF file that uses the UGRID conventions for mesh definition:

!ncdump -h cubedsphere.12x12.fixed.nc

netcdf cubedsphere.12x12.fixed {
dimensions:
	nMesh2_node = 866 ;
	nMesh2_edge = 1728 ;
	nMesh2_face = 864 ;
	Two = 2 ;
	Four = 4 ;
variables:
	int Mesh2 ;
		Mesh2:cf_role = "mesh_topology" ;
		Mesh2:mesh_class = "sphere" ;
		Mesh2:generator_inputs = "edge_cells=12;smooth_passes=0" ;
		Mesh2:long_name = "Topology data of 2D unstructured mesh" ;
		Mesh2:topology_dimension = 2 ;
		Mesh2:node_coordinates = "Mesh2_node_x Mesh2_node_y" ;
		Mesh2:face_node_connectivity = "Mesh2_face_nodes" ;
		Mesh2:edge_node_connectivity = "Mesh2_edge_nodes" ;
		Mesh2:face_edge_connectivity = "Mesh2_face_edges" ;
		Mesh2:face_face_connectivity = "Mesh2_face_links" ;
		Mesh2:face_coordinates = "face_lons face_lats" ;
	int Mesh2_face_nodes(nMesh2_face, Four) ;
		Mesh2_face_nodes:cf_role = "face_node_connectivity" ;
		Mesh2_face_nodes:long_name = "Maps every quadrilateral face to its four corner nodes." ;
		Mesh2_face_nodes:start_index = 1 ;
	int Mesh2_edge_nodes(nMesh2_edge, Two) ;
		Mesh2_edge_nodes:cf_role = "edge_node_connectivity" ;
		Mesh2_edge_nodes:long_name = "Maps every edge to the two nodes that it connects." ;
		Mesh2_edge_nodes:start_index = 1 ;
	int Mesh2_face_edges(nMesh2_face, Four) ;
		Mesh2_face_edges:cf_role = "face_edge_connectivity" ;
		Mesh2_face_edges:long_name = "Maps every quadrilateral face to its four edges." ;
		Mesh2_face_edges:start_index = 1 ;
	int Mesh2_face_links(nMesh2_face, Four) ;
		Mesh2_face_links:cf_role = "face_face_connectivity" ;
		Mesh2_face_links:long_name = "Indicates which other faces neighbour each face." ;
		Mesh2_face_links:start_index = 1 ;
		Mesh2_face_links:flag_values = -1 ;
		Mesh2_face_links:flag_meanings = "out_of_mesh" ;
	double Mesh2_node_x(nMesh2_node) ;
		Mesh2_node_x:standard_name = "longitude" ;
		Mesh2_node_x:long_name = "longitude of 2D mesh nodes." ;
		Mesh2_node_x:units = "radians" ;
	double Mesh2_node_y(nMesh2_node) ;
		Mesh2_node_y:standard_name = "latitude" ;
		Mesh2_node_y:long_name = "latitude of 2D mesh nodes." ;
		Mesh2_node_y:units = "radians" ;
	double face_lats(nMesh2_face) ;
		face_lats:units = "radians" ;
		face_lats:long_name = "Latitudes of face centres" ;
	double face_lons(nMesh2_face) ;
		face_lons:units = "radians" ;
		face_lons:long_name = "Longitudes of face centres" ;
}

Whilst I'm doing this work to ultimately develop a data-model for Iris, interacting with a NetCDF file directly - without any data-model or interpretation - is more easily achieved with XArray.

There seems to be some confusion about what Iris and XArray are. My very simplistic take on the two:

Use XArray if you want to interact with a Python object that looks just like a NetCDF file
Use Iris if you want to interact with a Python object that abstracts the file into standard conventions (CF-conventions)

The key difference being that XArray is very familiar to those comfortable with NetCDF but the lack of standards-based abstraction makes it is easy to write code for one file that does not work with another (for example, when a variable name changes).

There are definitely pros and cons to both, and I hope that one-day we will be able to bridge the gap between the two. For now though, it is fairly easy to convert from one to the other using XArray's xarray.DataArray.to_iris function.

import xarray as xr
ds = xr.open_dataset('cubedsphere.12x12.fixed.nc')
ds

<xarray.Dataset>
Dimensions:           (Four: 4, Two: 2, nMesh2_edge: 1728, nMesh2_face: 864, nMesh2_node: 866)
Dimensions without coordinates: Four, Two, nMesh2_edge, nMesh2_face, nMesh2_node
Data variables:
    Mesh2             int32 ...
    Mesh2_face_nodes  (nMesh2_face, Four) int32 ...
    Mesh2_edge_nodes  (nMesh2_edge, Two) int32 ...
    Mesh2_face_edges  (nMesh2_face, Four) int32 ...
    Mesh2_face_links  (nMesh2_face, Four) int32 ...
    Mesh2_node_x      (nMesh2_node) float64 ...
    Mesh2_node_y      (nMesh2_node) float64 ...
    face_lats         (nMesh2_face) float64 ...
    face_lons         (nMesh2_face) float64 ...

import numpy as np

xs, ys = np.rad2deg(ds['Mesh2_node_x']), np.rad2deg(ds['Mesh2_node_y'])

Now, let's get those faces as geometries:

import shapely.geometry as sgeom

def lfric_geoms(ds):
    faces = ds['Mesh2_face_nodes']
    offset = faces.attrs.get('start_index', 0)

    if offset:
        faces -= offset
        faces.attrs['start_index'] = 0

    for face in faces:
        geom = sgeom.Polygon([(xs[i], ys[i]) for i in face])
        yield geom

mesh_geoms = list(lfric_geoms(ds))
len(mesh_geoms)

So for those 864 faces, how long is it taking us to construct them?

%timeit list(lfric_geoms(ds))

3.84 s ± 881 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Let's dive straight in and use cartopy to get a land geometry. There are so many ways of doing this (looking at you GeoPandas), but here is one I wrote a few years ago that still holds up:

import cartopy.io.shapereader as shpreader

shpfilename = shpreader.natural_earth(
    resolution='110m', category='cultural', name='admin_0_countries')
countries = list(shpreader.Reader(shpfilename).records())

[Australia] = [country for country in countries
               if country.attributes['NAME'] == 'Australia']
AU_geom = Australia.geometry

So I intend to check the intersection of this Australia geometry with the mesh. Whenever you do repeated intersection calculations with shapely, you should consider "preparing" it to factorise out some of the initialisation

from shapely.prepared import prep

AU_prep = prep(AU_geom)

%timeit [AU_prep.intersects(cell) for cell in mesh_geoms]

11.9 ms ± 1.24 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

AU_geoms = [cell for cell in mesh_geoms
            if AU_prep.intersects(cell)
            # Pick out only small-ish cells for now. More detail later on...
            if cell.area < 2000]
len(AU_geoms)

So let's take a look at these matching geometries... cartopy is the go-to tool here.

I'll also get hold of matplotlib and use the interactive backend so that I can interact with the figure while developing in the notebook.

import cartopy.crs as ccrs
import matplotlib as mpl
import matplotlib.pyplot as plt

mpl.rcParams['figure.figsize'] = (4, 4)

%matplotlib notebook

This little utility function is a god-send. I should probably get it into cartopy...

def set_extent_from_geom(ax, geom):
    x0, y0, x1, y1 = geom.bounds
    ax.set_extent([x0, x1, y0, y1], ccrs.Geodetic())

OK, all ducks in a row, now plot the mesh cells that we found to intersect with Australia.

plt.figure()
pc_180 = ccrs.PlateCarree(central_longitude=180)
ax = plt.axes(projection=pc_180)
ax.add_geometries(AU_geoms, ccrs.PlateCarree(),
                  facecolor='blue', edgecolor='k', alpha=0.3)
set_extent_from_geom(ax, AU_geom.buffer(30))
ax.coastlines();

This is a good start. Let's do this on a global scale and produce a land-sea mask.

from shapely.ops import unary_union

shpfilename = shpreader.natural_earth(resolution='110m',
                                      category='physical',
                                      name='land')

land = unary_union(list(shpreader.Reader(shpfilename).geometries()))
# There is a problem with this geometry, so the old "buffer it by 0"
# trick should fix it up for us. 
land = land.buffer(0)
land

So how many of these mesh cells intersect with land?

land_prep = prep(land)
land_cells = [cell for cell in mesh_geoms
              if land_prep.intersects(cell)]
print(len(land_cells))

This seems a little low. Let's take a look:

plt.figure()
ax = plt.axes(projection=ccrs.PlateCarree())

plt.title('Land geometries')
ax.add_geometries(land_cells, ccrs.PlateCarree(),
                  facecolor='blue', edgecolor='k', alpha=0.3)
ax.coastlines();

The problem is that some geometries don't back project correctly to a projected space such as plate carree. Until we have spherical geometries, this is always going to be painfull.

Let's leave the painfull cells out for now..

def drop_large(geoms, area):
    """Generator of geometries that have an area below the given value."""
    for geom in geoms:
        if geom.area < area:
            yield geom

plt.figure()
ax = plt.axes(projection=ccrs.PlateCarree())
plt.title('Land geometries with an area <1000 $degrees^2$ (in PlateCarree)')
ax.add_geometries(drop_large(land_cells, 1000),
                  ccrs.PlateCarree(),
                  facecolor='blue', edgecolor='k', alpha=0.3)
ax.coastlines();

Clearly we have had a bit of sucess here, but we must be suffering from a value range of [0,360] and [-180,180]. Our land geometry lies within [-180, 180], but our mesh doesn't:

def multipoly_geom_xlimit(multipoly):
    if not multipoly.geoms:
        raise ValueError('Not a multipolygon')
    
    xmin = min([min(geom.exterior.coords)[0] for geom in multipoly.geoms])
    xmax = max([max(geom.exterior.coords)[0] for geom in multipoly.geoms])
    return xmin, xmax

print('Land min: {}; max: {}'.format(*multipoly_geom_xlimit(land)))
print('Cells min: {}; max: {}'.format(float(xs.min()), float(xs.max())))

Land min: -180.0; max: 180.00000000000014
Cells min: -0.0; max: 352.5000000475477

In projected space these ranges are critical, and require a reprojection of one or the other geometry. Let's re-project the land geometry, as there are fewer of those. In truth though, I usually try to normalise to [-180, 180].

The trick is to convert the geometry into a PlateCarree with a central longitude at the antimeridian, and then to add 180 to the values to get us back to longitude and latitudes.

pc_180 = ccrs.PlateCarree(central_longitude=180)

# Instead of the geometry in lon/lat, let's project it to lon+180/lat.
land_180 = pc_180.project_geometry(land, ccrs.PlateCarree())

# Same trick as before - try to make this a valid geometry.
land_180 = land_180.buffer(0)

# Finally, translate the projected lon+180/lat geometry back to lon/lat.
import shapely.affinity

land_180 = shapely.affinity.translate(land_180, xoff=180)

land_180

Let's double check this geometry is as expected:

print('Land_180 min: {}; max: {}'.format(*multipoly_geom_xlimit(land_180)))

Land_180 min: 0.0; max: 360.0

So using the original mesh cells in the range [0, 360] and our projected land in the range [0, 360], let's compute the intersections!

land_180_prep = prep(land_180)
land_180_geoms = [cell for cell in mesh_geoms
                  if land_180_prep.intersects(cell)]
print(len(land_180_geoms))

Great! We have more matching cells than we did before!

def draw_mesh(ax, geoms):
    # Note: Writing functions that take the axes as an argument is a
    # recommended paragdim. It allows easy subplot creation ;)
    ax.coastlines()
    ax.add_geometries(
        geoms,
        ccrs.PlateCarree(),
        facecolor='blue', edgecolor='k', alpha=0.2)

plt.figure()
ax = plt.axes(projection=ccrs.PlateCarree())
plt.title('Land geometries with an area <1000 $degrees^2$ (in PlateCarree)')

draw_mesh(ax, drop_large(land_180_geoms, 1000));

Nope, that wasn't it. Must be an issue with the faces on the western portion of the projection.

Turns out, cartopy doesn't handle the PlateCarree geometries that are greater than 180. This does surprise me, which itself is a surprise given I wrote cartopy. For now, let's just move those cells we've identified back by 360 degrees if they are too far over.

def fixup_wraparound_geoms(geoms):
    # If a cell is over 180, move it back 360.
    for geom in geoms:
        if max(geom.exterior.coords)[0] > 180:
            yield shapely.affinity.translate(geom, xoff=-360)
        else:
            yield geom

plt.figure()
ax = plt.axes(projection=ccrs.PlateCarree())
plt.title('Land cells with an area $<1000 degrees^2$ in PlateCarree')

draw_mesh(ax,
          fixup_wraparound_geoms(drop_large(land_180_geoms, 1000)));

More progress. Notice that we are identifying corner cells here, so no worries there!

Now let's look at the antimeridian issue with the stripe through Europe and Africa missing.

I guess at the dateline there are geometries that have a wraparound which has impacted their geometry orientation (that was probably why we were getting those really wide cells in our first plot, and why filtering large area geometries has been an effective way of removing them). This is probably a bug with the data, as it does state that the geometry should be CCW (on the sphere).

Observation: all UGRIDs are implicitly spherical if they are global and nodes are shared between geometries that cross the antimeridian.

This is getting a bit manual. Instead of playing whack-a-mole with these mesh geometries, let's try projecting the mesh the cells into plate carree from a geodetic coordinate system using cartopy.

That will make some of the cells have many more than 4 nodes, but it will be much more representative of their shape in PlateCarree. It seems highly unlikely that this approach will scale beyond the c12 mesh though...

def project_from_geodetic(geoms):
    pc = ccrs.PlateCarree()
    geod = ccrs.Geodetic()
    for geom in geoms:
        yield pc.project_geometry(geom, geod)

mesh_geoms_prj = list(project_from_geodetic(lfric_geoms(ds)))

How long did that take? Remember that the original lfric_geoms call was ~3s:

%timeit mesh_geoms_prj = list(project_from_geodetic(lfric_geoms(ds)))

6.2 s ± 899 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

~4.5s is far better than I thought it would be, if I'm honest.

OK, let's use these nicely projected geometries and plot them all:

def draw_mesh(ax, geoms):
    # Note: Writing functions that take the axes as an argument is a
    # recommended paragdim. It allows easy subplot creation ;)
    ax.coastlines()
    ax.add_geometries(
        geoms,
        ccrs.PlateCarree(),
        facecolor='blue', edgecolor='k', alpha=0.2)

plt.figure()
draw_mesh(plt.axes(projection=ccrs.PlateCarree()),
          mesh_geoms_prj)
plt.title('Land cells with an area $<1000 degrees^2$ in PlateCarree');

Wowzers, that seems to have done the trick nicely. Let's look at this in a few different ways:

plt.figure()
draw_mesh(plt.axes(projection=ccrs.Orthographic(-30, 30)),
          mesh_geoms_prj);

plt.figure(figsize=[10, 5])
ax1 = plt.subplot(1, 2, 1, projection=ccrs.NorthPolarStereo())
ax2 = plt.subplot(1, 2, 2, projection=ccrs.SouthPolarStereo())

for ax in [ax1, ax2]:
    draw_mesh(ax, mesh_geoms_prj)
    lim = 1e7
    ax.set_extent([-lim, lim, -lim, lim], ccrs.SouthPolarStereo())

And finally, do these nicely projected mesh cells actually function as we expect when it comes to land intersection?

land_geoms = [cell for cell in mesh_geoms_prj
              if land_prep.intersects(cell)]
print(len(land_geoms))

plt.figure()
draw_mesh(plt.axes(projection=ccrs.PlateCarree()),
          land_geoms)
plt.title('All land geoms in PlateCarree');

Success! The indices of these cells can be back computed fairly easily:

land_geoms_ids = [id(geom) for geom in land_geoms]
land_mask = np.array([id(geom) in land_geoms_ids for geom in mesh_geoms_prj])

print('n-faces: {}, n-land-faces: {}'.format(land_mask.size, land_mask.sum()))

ds['face_lats'][land_mask]

n-faces: 864, n-land-faces: 423

<xarray.DataArray 'face_lats' (nMesh2_face: 423)>
array([ 0.709339,  0.717838,  0.717838, ..., -0.779728, -0.813979, -0.836692])
Dimensions without coordinates: nMesh2_face
Attributes:
    units:      radians
    long_name:  Latitudes of face centres

What do the land cells look like on a South polar stereo plot?

plt.figure()
draw_mesh(plt.axes(projection=ccrs.SouthPolarStereo()),
          land_geoms)
plt.title('All land geoms in South Polar Stereographic')

lim = 1.6e7
plt.gca().set_extent([-lim, lim, -lim, lim], ccrs.SouthPolarStereo());

Looks rather splendid. Remember, the containment testing happened in PlateCarree space (poles, wraparounds and all), yet we seem to have got away with this working even for Antarctica.

Conclusion¶

The approach of projecting the spherical/geodetic geometries into PlateCarree with cartopy works.

I do not expect this approach to scale to millions of cells (as is the case for c1024 - a 1024x1024x6 mesh of faces), but I will put metrics on that in a follow-up notebook. The scaling issue comes from the fact that Shapely is operating at the python level, and creating that many objects of any kind is going to be slow. This is doubly true if we are having to project those shapely geometries through cartopy (another Python layer).

What's more, it seems unlikely that we really want to visualise the results in this way for a c1024 mesh. There are more efficient rasterisation techniques (e.g. AGG and datashader) that will work very effectively and provide equivalent fidelity to a figure (but still not render all ~6,200,00 faces in the figure itself).

The wraparound and poles are going to consistently prove to be problematic unless we can handle truly spherical geometries. Cartopy gets us close, but there is no good non-spherical representation of geometries that can be done with GEOS/Shapely.

Is S2 the solution to this perhaps? Can that be made to scale? Does ESMF have a trick up its sleve for containment testing on the sphere?

Analysing walks on the South West Coast Path

2018-05-26T00:00:00+01:00

I'm fortunate to live near to one of the best continuous coastal walking/hiking routes in the world. The UK's South West Coast Path is a scenic, cultural and culinary delight spanning 630 miles from Minehead to Poole. It is steeped with history, and apparently was originally created by the coastguard who used it to patrol the south west peninsula looking out for smugglers.

I don't consider myself to be a "serious" hiker, but I do enjoy going for a walk and being by the sea. When I remember to do so, I also use a GPS tracker so that I can see how far/high/fast I walked. This is somewhat of a defensive strategy, as I've a bit of a reputation with friends and family for under-estimating exactly how far it is I intend to drag them along for :).

Over the last few walks I've been thinking about how much of the 630 miles I've actually done. I've also noticed that I get into a bit of a rut with regards to going to new sections of the path. To address both of these, I figured I needed to visualise the data and to do some calculations to figure out how much of it I've walked (whilst using a GPS tracker). This might spur me on to do new sections, as well as giving me some more motivation for remembering to track my walks in the future.

Working with the coast path geometry itself¶

First up, I need to get hold of the South West Coast Path geometry. Luckily, the excellent National Trail website has a GPX file that I can download. Let's do that.

from pathlib import Path
from urllib.request import urlretrieve

# From the fantastic https://nationaltrail.co.uk website.
URL = "https://www.nationaltrail.co.uk/sites/default/files/south_west_coast_path.gpx"

coast_path_gpx = Path('south_west_coast_path.gpx')

if not coast_path_gpx.exists():
    urlretrieve(URL, coast_path_gpx)

GPX in-hand, Fiona and Shapely are my go-to tools for loading and manipulating vector data like this. Luckily they don't disappoint:

import fiona
import shapely.geometry as sgeom

layers = fiona.listlayers(str(coast_path_gpx))
print('Layers available: {}'.format(', '.join(layers)))

with fiona.open(str(coast_path_gpx), "r", layer='tracks') as records:
    coast_path_tracks = [
        sgeom.shape(record['geometry']) for record in records]

Layers available: waypoints, routes, tracks, route_points, track_points

This has created a bunch of Shapely geometries. We really only need one mega-geometry, so let's take their union.

import shapely.ops

coast_path = shapely.ops.unary_union(coast_path_tracks)

Let's figure out how many coordinate points that geometry is:

def n_coords(geometry):
    """
    A function to figure out the number of coordinate
    points are in the given geometry.

    """
    if hasattr(geometry, 'geoms'):
        return sum(n_coords(geom) for geom in geometry.geoms)
    else:
        return len(geometry.coords)

    
print(f'N coords in the coast path geometry: { n_coords(coast_path) }')

N coords in the coast path geometry: 45066

That's quite a lot of coordinate points. We are going to want to do things like intersections and distances from this geometry, and can happily accept a reduction in fidelity for an increase in performance, so let's also generate a "simplified" version of this geometry.

SIMPLIFICATION_FACTOR = 0.0005

coast_path_simplified = coast_path.simplify(SIMPLIFICATION_FACTOR)

print(f'N coords in the simplified geometry: { n_coords(coast_path_simplified) }')

coast_path_simplified

N coords in the simplified geometry: 3287

Let's take a look at this simplified geometry in detail. Some interactivity would be nice and I don't need any fancy projection stuff at this scale, so Folium is a good choice.

import folium 


def coast_path_map():
    """Setup a Folium map with the simplified coast path geometry"""
    m = folium.Map()
    
    map_set_bounds_from_geometry(m, coast_path_simplified)

    style_function = lambda x: {'color': "#808000",
                                'line_opacity': 0.5}
    m.add_child(
        folium.GeoJson(
            coast_path_simplified.__geo_interface__,
            name='South West Coast Path',
            style_function=style_function,
        )
    )
    return m


def map_set_bounds_from_geometry(m, geom):
    """
    Given a folium map an a geometry, set the map's
    bounds to show it
    
    """
    geom_bounds = geom.bounds
    # Folium is looking for [[y0, x0], [y1, x1]]
    bounds = [geom_bounds[:2][::-1],
              geom_bounds[2:][::-1]]
    m.fit_bounds(bounds)


coast_path_map()

Even at this level of simplification, it is clear that there is enough detail allow us to continue the analysis.

Calculating the distance of the coast path¶

Earlier, I quoted a distance of 630 miles for the coast path, but how accurate is that in reality? Geodetic distances can be well approximated with Vincenty's formula, but even better accuracy can be achieved with Charles Karney's algorithm and its associated implementation found in geographiclib. This is now a standard part of the proj.4 library, and can be used with tools such as pyproj and cartopy. Here I'll use Cartopy, but this isn't a documented interface, and I think the API is likely to be integrated with Cartopy's Globe in the near future:

from cartopy.geodesic import Geodesic
import numpy as np

geod = Geodesic()

def distance(pt1, pt2):
    """
    Return the distance, in meters, between the given points.
    Note, this function is vectorized, and can accept multiple points.

    """
    result = np.array(
        geod.inverse(np.asanyarray(pt1),
                     np.asanyarray(pt2)))
    # Inverse returns [[distance, azim1, azim2], ...].
    # We just want to return the distance
    return result[:, 0]

Let's take that function for a spin with some random locations:

result = distance([40.6, -73.8], [49.1, 2.5])
print(f'Geodetic distance between two random points: ~{ result[0]//1000 }km')

Geodetic distance between two random points: ~8489.0km

The coast path geometry is made up of a bunch of individual geometries, so let's work out their constituent distances:

def linestring_distance(geom):
    if hasattr(geom, 'geoms'):
        return sum(linestring_distance(subgeom) for subgeom in geom.geoms)
    else:
        points = np.array(geom.coords)
        return distance(points[:-1, :2],
                        points[1:, :2]).sum()

    
coast_path_distance_km = linestring_distance(coast_path) // 1000

print(f'Coast path distance {coast_path_distance_km} km '
      f'({0.621371 * coast_path_distance_km//1} miles)')

Coast path distance 1073.0 km (666.0 miles)

Great to see that my calculation is so close to the advertised distance for the coast path!

Getting hold of my recorded tracks¶

Even though it is now deprecated, I've used Google's "My Tracks" since the beginning of time (or at least, since I got my first Android device). It has/had this really neat feature of backing up my tracks to my Google Drive. I also automatically sync my Google Drive with my laptop, so I don't need to do any work to get my tracks as they are already on my machine in KMZ format.

Whilst I paint a rosy picture of the backup mechanics, I have had some difficulties syncing from "My Tracks" to my Google Drive since it was deprecated, and suspect it may have stopped working altogether. In that situation, I simply used "My Tracks" export all functionality on my phone, and put that into my Google Drive directory.

Let's try opening this with Fiona:

my_tracks_dir = Path('/Users/pelson/Documents/GDrive pelson/My Tracks')

example_file = str(my_tracks_dir / 'Newborough.kmz')

with fiona.open(example_file, "r") as records:
    track = [
        sgeom.shape(record['geometry']) for record in records]

---------------------------------------------------------------------------
DriverError                               Traceback (most recent call last)
<ipython-input-10-d510555a9088> in <module>()
      3 example_file = str(my_tracks_dir / 'Newborough.kmz')
      4 
----> 5 with fiona.open(example_file, "r") as records:
      6     track = [
      7         sgeom.shape(record['geometry']) for record in records]

~/miniconda/envs/gis/lib/python3.6/site-packages/fiona/__init__.py in open(path, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt)
    163         c = Collection(path, mode, driver=driver, encoding=encoding,
    164                        layer=layer, vsi=vsi, archive=archive,
--> 165                        enabled_drivers=enabled_drivers)
    166     elif mode == 'w':
    167         if schema:

~/miniconda/envs/gis/lib/python3.6/site-packages/fiona/collection.py in __init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, **kwargs)
    160 
    161         if self.session is not None:
--> 162             self.guard_driver_mode()
    163             if not self.encoding:
    164                 self.encoding = self.session.get_fileencoding().lower()

~/miniconda/envs/gis/lib/python3.6/site-packages/fiona/collection.py in guard_driver_mode(self)
    174         driver = self.session.get_driver()
    175         if driver not in supported_drivers:
--> 176             raise DriverError("unsupported driver: %r" % driver)
    177         if self.mode not in supported_drivers[driver]:
    178             raise DriverError("unsupported mode: %r" % self.mode)

DriverError: unsupported driver: 'LIBKML'

Oh dear. KML&KMZ are not supported by Fiona it seems. :(

How about something like fastkml? From a brief inspection, it looks like fastkml wants to deal with strings, so we need to unzip the kmz file and read the contents of doc.kml ourselves.

from fastkml import kml
from zipfile import ZipFile

with ZipFile(example_file, 'r') as kmz:
    with kmz.open('doc.kml', 'r') as  kml_fh:
        kml_doc = kml_fh.read()

tracks_kml = kml.KML()
tracks_kml.from_string(kml_doc)

No geometries found

Turns out this KML isn't understood by fastkml either.

Perhaps I shouldn't have been so quick to dismiss Fiona here. There is some secret-source to enable the LIBKML driver within Fiona:

# There was a helpful hint for geopandas to enable the functionality within fiona:
fiona.drvsupport.supported_drivers['LIBKML'] = 'r'

with fiona.open(example_file, "r") as records:
    geometries = [
        sgeom.shape(record['geometry']) for record in records]

This gives me back 3 geometries: the start point, the track, and the end point. I think the start and end are redundant, as they are also in the track itself, so let's take a look at the middle one of these geometries:

geometries[1]

And looking at this geometry on a Folium map:

m = coast_path_map()

m.add_child(
    folium.GeoJson(
        geometries[1].__geo_interface__,
        name='Example track'
    )
)
map_set_bounds_from_geometry(m, geometries[1].buffer(0.01))

m

Looks great! Unfortunately the example file I've chosen isn't actually on the south west coast path - it is a lovely walk that I did up in North Wales a few years ago to an island associated with Wales' patron saint of lovers, Saint Dwynwen.

Let's go through all of my walks, and just pull out the ones that are somewhat close to the coast path. I'll also drop KMZ that don't look like the one I've already processed, and get rid of those that are unrealistically long (>0.6 degrees):

tracks_kmz = sorted(my_tracks_dir.glob('*.kmz'))

records = {}
for track_kmz in tracks_kmz:
    with fiona.open(str(track_kmz), "r") as kml_records:
        try:
            start, record, end = kml_records
        except ValueError:
            # Not the My Tracks format we were expecting...
            print('Dropping (due to error): {}'.format(track_kmz.parts[-1]))
            continue

        name = record['properties'].get('Name', str(track_kmz)).strip()
        geom = sgeom.shape(record['geometry'])
        
        record['geometry'] = geom

        distance_from_path = coast_path.distance(sgeom.shape(start['geometry']))
        
        # If the distance is less than a small number of degrees, let's include it
        if distance_from_path < 0.6 and geom.length < 0.8:
            records[name] = record
        else:
            print('Dropping (due to distance): {}'.format(track_kmz.parts[-1]))

            
print(f'\nFound {len(records)} walks')

Dropping (due to distance): 12_11_2017 10_31.kmz
Dropping (due to error): 25_07_2015 09:30.kmz
Dropping (due to error): 27_09_2014 14:58.kmz
Dropping (due to distance): 29_07_2014 14:14.kmz
Dropping (due to distance): Aber Falls.kmz
Dropping (due to distance): Adelaide Run.kmz
Dropping (due to distance): Amalfi Coast.kmz
Dropping (due to distance): Boar's Hill.kmz
Dropping (due to distance): Bryher.kmz
Dropping (due to error): Budleigh To Sidmouth .kmz
Dropping (due to distance): Geirionydd .kmz
Dropping (due to distance): Half Scilly.kmz
Dropping (due to distance): Koala Walk.kmz
Dropping (due to distance): Newborough.kmz
Dropping (due to distance): Obelisk.kmz
Dropping (due to distance): Sault .kmz
Dropping (due to distance): Snowdon.kmz
Dropping (due to distance): St Mary's.kmz
Dropping (due to distance): Tresco.kmz
Dropping (due to distance): Un Mapped Coast Path.kmz
Dropping (due to distance): Unnamed Road.kmz
Dropping (due to distance): Via Pennino.kmz
Dropping (due to distance): Vicar Of Dibley .kmz

Found 45 walks

In the interests of being a little bit discrete about exactly where I live in the South West of England, I'll mask out a few locations from the tracks:

filter_locations = [sgeom.Point(-3.4792, 50.7179).buffer(0.025),
                    sgeom.Point(-3.4910, 50.7020).buffer(0.025),
                    sgeom.Point(-3.3333, 50.6333).buffer(0.015)]

filter_locations = shapely.ops.unary_union(filter_locations)

# Update tracks to remove those locations, and add a simplified geometry to the record
for record in records:
    geom = records[record]['geometry'].difference(filter_locations)
    records[record]['geometry'] = geom
    records[record]['simplified_geometry'] = geom.simplify(SIMPLIFICATION_FACTOR)

Let's take a look at these tracks:

def add_records(m, records):
    """
    Draw the simplified geometries of the given records.

    """
    for name, record in records.items():
        m.add_child(
            folium.GeoJson(
                record['simplified_geometry'].__geo_interface__,
                name='Example track'
            )
        )
    return m

m = coast_path_map()
add_records(m, records)

That is a lovely record of some of the walks I've done in the South West, but some of those are definitely not coastal.

Let's just get the tracks that are really on the coast-path:

coast_path_records = {
    name: record for name, record in records.items()
    if coast_path_simplified.distance(record['simplified_geometry']) < 0.01}

print(f'Number of tracks: { len(records) }')
print(f'Number of coast path tracks: { len(coast_path_records) }')

Number of tracks: 45
Number of coast path tracks: 22

Let's take a look at those coast path tracks:

m = coast_path_map()
add_records(m, coast_path_records)
m

There is clearly at least one geometry that could still be tidied up (I suspect I left the tracker on whilst driving home!), but I can still use what I have to answer the question "How much of the coast path have I walked/tracked?".

First, let's combine together all of those walks into a single geometry.

coast_path_walks_union = shapely.ops.unary_union(
    [record['simplified_geometry'] for record in coast_path_records.values()])

Next, let's buffer around the coast path walks (by 0.001 degrees). This is going to be fairly slow:

coast_path_walks_buffered = coast_path_walks_union.buffer(0.001)

Finally, take the buffered region away from the coast path track.

Taking the difference from the coast path line-string ensures that the result result remains a line-string. The alternative approach of taking a union or intersection would not have this effect, and would result in a polygon instead.

unwalked_coast_path = coast_path.difference(coast_path_walks_buffered)

# The walked section is therefore the coast path minus the unwalked coast path:
walked_coast_path = coast_path.difference(unwalked_coast_path)

Let's look at how much coast path there is left to walk:

remaining_km = linestring_distance(unwalked_coast_path) // 1000

print(f'Distance of South West Coast Path unwalked: { remaining_km }km')

Distance of South West Coast Path unwalked: 954.0km

It turns out that there is quite a lot of coast path left for me to do...

full_length_km = linestring_distance(coast_path) // 1000
completed_km = full_length_km - remaining_km

print(f'Total coast path distance: { full_length_km } km')
print(f'Distance walked: { completed_km } km')
print(f'Percentage of coast path complete: { np.round(completed_km / full_length_km * 100) }%')

Total coast path distance: 1073.0 km
Distance walked: 119.0 km
Percentage of coast path complete: 11.0%

Finally, let's look at those sections that I have done:

m = folium.Map()
map_set_bounds_from_geometry(m, coast_path_simplified)

m.add_child(
    folium.GeoJson(
        walked_coast_path.__geo_interface__,
        name='Example track'
    )
)
print("Sections of the South West Coast Path I've walked (and tracked via GPS):")
m

Sections of the South West Coast Path I've walked (and tracked via GPS):

Conclusion¶

I got hold of some GPX data from the UK's National Trails website, and combined this with my own GPS logger data (KMZ/KML via My Tracks) to compute how much of the South West Coast Path I've walked (and tracked).

I was able to use OGR/Fiona to read in the GPS logger data (in KML/KMZ), but it wasn't able to keep track of the timestamp of each coordinate. For that, I would need to roll my own KML parser. This will be necessary if I want a more sophisticated means of filtering those tracks that include both walking and driving (i.e. when I forget to turn off the tracker). I will want to do this in the future in order to answer my remaining question "How far have I walked in total whilst walking the South West Coast Path?", but I will have to keep that for a future blog post.

Folium was a great tool to make use of for visualising this data. My day job often deals with geometries on a global scale, at which point one really needs Cartopy's clever treatment of the antimeridian and poles, but when the problem lies in a space that Cartesian coordinates are appropriate, the existing tools (Shapely, Fiona, GeoJSON, Folium) really do shine.

So it turns out that I've walked just 11%, or 120km, of the South West Coast Path with my GPS tracker on. I've probably completed quite a significant amount more, but clearly failed to track it. Perhaps my next step should be to dig into my Google location data to see how much detail they have on me. Either that, or maybe I should just go for a walk...

Recursive search and replace

2018-05-21T12:00:00+01:00

I had a need to recursively search and replace for a particular string within a git repository. A combination of find and sed is all that is needed, but one must be careful to avoid modifying the contents of the ".git" directory (I accidentally did this, and effectively destroyed my clone).

The command necessary on my OSX, in all its glory:

LC_ALL=C find . \
    -path ./.git -prune \
    -o -type f \
    -exec sh -c 'sed -i "" -- "s|old|new|g" {};' \;

Unfortunately, this has the effect of adding newlines at the end of each file that doesn't have them. This can be seen in a number of StackOverflow questions, such as this one.

Instead of sed therefore, perl can be used instead:

LC_ALL=C find . \
    -path ./.git -prune \
    -o -type f \
    -exec sh -c 'perl -pi -e "s|old|new|g" {};' \;

Creating an application for a website on OSX

2017-09-20T12:00:00+01:00

I wanted to get a desktop application for a website that doesn't provide one, and had success with https://github.com/eladnava/applicationize. This worked a treat for my local Home Assistant instance, so now I don't need to open a tab in my web-browser.

Using nmap to find all devices on network listening on a particular port

2017-08-20T12:00:00+01:00

Finding all IPs listening on a particular port using nmap:

nmap -p 80 --open -sV 192.168.1.0/24

Refs: * http://thoughtsbyclayg.blogspot.co.uk/2008/06/use-nmap-to-scan-for-ssh-servers-on.html * https://www.digitalocean.com/community/tutorials/how-to-use-nmap-to-scan-for-open-ports-on-your-vps

Creating a continuous integration service to check CLAs on GitHub

2017-08-13T00:00:00+01:00

Contributor license agreements are a thing. As much as I dislike the bureaucracy of them, they do provide some additional cover to the owners of an open source project, and some companies insist on having them. I find myself in such a situation in the SciTools organisation, for which I am one of the lead developers. We have a CLA form, a signing process, and a list of signatories - but the really painful part is having to remember to cross check that list of signatories each time we want to merge a pull request.

This is the story of how (and to some extent why) I went about automating that process.

In early August 2017, I ran a 2 day tutorial for my team to improve knowledge and give practical experience of using and creating "webapps, webhooks and SaaS". I picked off a few useful technologies to play with, and presented a few hours of material to them. In particular, I focused on tornado, the GitHub API, and Heroku.

Next, we took the knowledge I presented and came up with a bunch of ideas to hack on. I decided to focus on the pesky human-in-the-loop process of cross checking our CLA signatories list, and see if I could turn it into a fully fledged CI service in less than 1.5 days (in practice, since I led the tutorial and helped others whenever they got stuck, I only got a fraction of that time actually working at my machine).

My basic plan was as follows:

Write a CLI tool that gets our CLA signatories list and prints it
Write a CLI tool that gets the authors of a GitHub PR (there may be more than one - it is the commits we actually care about)
Write a CLI tool that updates a PR's status and labels based on the status of a CLA check
Write a webapp that listens for GitHub webhooks, and applies the tool developed each time a PR is created/updated

Since I knew I was targeting a tornado webapp at the end, I ensured all of the key components were written as asynchronous coroutines, and only at the very top level of the CLI tool would I call the coroutines synchronously. It is important to note that writing coroutines generally requires making use of other coroutines - it is somewhat invasive, and much easier to start off in the knowledge that you want your code to be asynchronous.

Calling (asynchronous) coroutines from a (synchronous) CLI tool

The pattern that I followed to make my tools asynchronous, but my CLI synchronous was simply to use tornado's run_sync method on an IOLoop instance:

@tornado.gen.coroutine
def my_coroutine(arg1, arg2):
    result1 = yield another_coroutine_1(arg1) 
    result2 = yield another_coroutine_2(arg2)
    return result1, result2


def main():
    ...
    # argparse stuff
    ...


    # Finally, call the coroutine with the argparse arguments.
    result1, result2 = tornado.ioloop.IOLoop.current().run_sync(
        lambda: my_coroutine(args.arg1, args.arg2))

Using the GitHub API for labels and statuses

A big part of the challenge was making use of the GitHub API (v3) to query information about the PR in question, and to update things like the commit status and PR labels.

The first thing to note is the GitHub data model. It is a strange quirk that pull requests are actually issues (though not all issues are pull requests), and pull request statuses are actually statuses on commits. Updating the labels of a pull request therefore involves using the POST /repos/:owner/:repo/issues/:number/labels API. Updating the commit status first involves finding the HEAD commit of the pull request, and then using the POST /repos/:owner/:repo/statuses/:sha.

Authentication of GitHub API

Because I was writing this tool for a very niche set of repositories, I didn't need anything fancy in terms of authentication. I simply created a personal access token with the necessary scopes, and updated all GitHub API calls to include the extra Authorization: token ${TOKEN} http header. When it comes to deploying this, there is an option to set secure environment variables on Heroku (under settings) which is where the token is put under the environment variable name TOKEN.

If I did need a more sophisticated authentication scheme there are two options available. Both have reasonably good solutions for use with tornado:

OAuth - useful for interactive (mostly browser based) services that act at the time the user visits the service
GitHub apps - useful for non-interactive services, where scheduled tasks or other event triggers require authenticated work to occur (e.g. a status updater when a PR is modified)

It is worth noting that, despite my description of GitHub apps, tools like Travis-CI could happily make use of personal access tokens as a means of setting commit statuses because they are happy to report as the @travis-ci user, not the user who registered a repo's travis-ci integration.

An example of GitHub's OAuth with tornado can be found on GitHub at jkeylu/torngithub, and an example of creating a GitHub app with python can be found in a gist I created at https://gist.github.com/pelson/47c0c89a3522ed8da5cc305afc2562b0.

Writing a webhook

GitHub webhooks are simply a POST request to a pre-determined URL. Of course, you need to have a webapp listening at that URL in order to do anything with it. In this instance, I simply pushed my webapp up to the cloud (Heroku in this case). The docs for webhooks are pretty good, but the single most important piece of advice for when creating one is to take a look at the "manage webhooks" page. You can find this in your webhooks' settings page, and you get some invaluable info about the deliveries that have been made, as well as having a button to get the hook redelivered:

Preventing man-in-the-middle webhooks

The default webhook essentially requires a POST request of the right form sent to the right address. With this information anybody could submit their own POST request and get your service to do something. If you want to have confidence that the POST request does originate from GitHub, and hasn't been tampered with, you need to define a secret that GitHub will use to sign the POST content. The secret can be anything, and will be used to create a HMAC digest of the request being sent. You simply need to use the same secret to create a digest on the webapp side, and confirm that they match. If they don't match, the message hasn't come from GitHub.

The docs on "Securing your webhooks" are pretty extensive. They show an example in ruby, but almost exactly the same calls are required to validate the HMAC signature in Python:

import hmac
import hashlib


hmac_digest = headers.get('X-Hub-Signature', None)
webhook_secret = os.environ['WEBHOOK_SECRET'].encode()

# Compute the payload's hmac digest.
expected_hmac = hmac.new(
        webhook_secret, self.request.body, hashlib.sha1).hexdigest()
expected_digest = 'sha1={}'.format(expected_hmac)

if hmac_digest != expected_digest:
    logging.warning('HMAC FAIL: expected: {}; got: {};'
                    ''.format(expected_digest, hmac_digest))
    self.set_status(403)

Tornado async user agent

Towards the end of the 2 day session, we were all frantically trying to deploy our own developments, and as the leader of the tutorial it was inevitable that I would be called up to help each group finalise their work. I too was frantically trying to get my CLA checker deployed, and was running up against the most peculiar of issues...

Tornado's AsyncHttpClient was returning a 403 (Authentication failure) on the GitHub API on my Heroku deployment, but not locally. In addition, I was able to ssh onto my Heroku instance and verify that the exact same call using the exact same token worked like a charm using requests and curl.

Had I not been all over the place trying to help others out, and perhaps had a shorter iteration cycle (I was having to deploy to Heroku to debug), I may have solved the problem a little quicker. It took a good hour to realise that the problem was that I was getting 403 because the User-Agent wasn't being set by AsyncHttpClient, and GitHub have a strict enforcement policy on it needing to exist. Simply adding the user agent when calling fetch solved this issue:

yield http_client.fetch(URL, method='GET',
                        user_agent='MY-CLI-CHECKER',
                        headers=headers)

Round-up

So there you have it... I put together a CLA checking service in less than a couple of days, and in the process was able to help my team learn about webapp, Heroku and the GitHub API. A screenshot of the kind of behaviour I have created shows both the status and labels being set when a CLA is missing:

Although I've actually written a CI type tool on top of the GitHub API before, there were a bunch of things I learnt along the way. Hopefully this shows you that you really can have the integrations you've always dreamed of on GitHub with just a small amount of development effort.

You can see the code for the SciTools CLA checker I produced at SciTools-incubator/scitools-cla-checker.

Converting rasters to SVG, and creating a rudimentary font with font-forge - Part 4 of an XKCD font saga

2017-04-21T00:00:00+01:00

In part three of my XKCD font saga I generated several hundred glyphs as PPM images, and classified them with their associated character(s). In this instalment, I will convert the raster glyphs into vector form (SVG) and then generate a rudimentary font using FontForge.

If you'd like to follow along, the input files for this article may be found at https://gist.github.com/pelson/1d6460289f06acabb650797b88c15ae0, while the code (in notebook form) and output may be found at https://gist.github.com/pelson/18434e3bd37dcde8dd28a5a24def0060.

I make no apologies for the amount of code here - I've tidied it up a little, but it is representative of the journey I have taken to produce a fully functional XKCD font. If you just want to see the resulting font you can scroll about three-quarters of the way down the page to play with the final font in your browser.

Raster to Vector conversion (PPM to SVG)¶

The glyph images that need SVG vectorisation are simple black and white images, so this falls well into the realm of potrace and autotrace.

I notice that bioconda has a potrace binary, so I use conda to install it. There are python bindings for potrace, but really we just want to convert one filename into another filename, so I'll just call out to the system.

import subprocess

def potrace(input_fname, output_fname):
    subprocess.check_call(['potrace', '-s', input_fname, '-o', output_fname])

import os
import glob

for fname in glob.glob('../xkcd_font_stroke_classification/char_*.ppm'):
    new_name = os.path.splitext(os.path.basename(fname))[0] + '.svg'
    potrace(fname, new_name)

Just to confirm this is as we expected, I'll take a look at one of the newly vectorised SVGs. The slight complexity here is that IPython.display doesn't support the ppm format, so I need to load the ppm image with PIL/Pillow, and convert it to PNG (without creating a temporary file) before doing the standard IPython image display trick.

import PIL.Image
from io import BytesIO
from IPython.display import SVG, Image, display, Markdown

ppm_sample = '../xkcd_font_stroke_classification/char_L2_P2_x378_y1471_x766_y1734_a-s_0x61-0x73.ppm'
svg_sample = 'char_L2_P2_x378_y1471_x766_y1734_a-s_0x61-0x73.svg' 

im = PIL.Image.open(ppm_sample)
png_img = BytesIO()
im.save(png_img, 'PNG')
png_img.seek(0)

display(Markdown('#### Raster version (PPM, original pixel resolution)'))
display(Image(data=png_img.read()))

display(Markdown('#### Vector version converted from PPM to SVG with potrace'))
display(SVG(filename=svg_sample))

Raster version (PPM, original pixel resolution)¶

Vector version converted from PPM to SVG with potrace¶

Naturally, the vectorisation has smoothed some of the edges - I just used the default settings and I think the results are pretty pleasing. We now have glyphs that scale extremely well (much better than the rasterised form), and in this instance just so happen to be around 75x smaller in filesize!

!du -h $ppm_sample $svg_sample

300K	../xkcd_font_stroke_classification/char_L2_P2_x378_y1471_x766_y1734_a-s_0x61-0x73.ppm
4.0K	char_L2_P2_x378_y1471_x766_y1734_a-s_0x61-0x73.svg

Creating a font from SVGs using fontforge¶

Now that I have all of the glyphs in vector form, I'm going use fontforge to create a font with glyphs imported from the appropriate SVG shapes.

Finding a build of fontforge with python bindings that could be used in my environment proved to be a challenge, so I ended up compiling fontforge myself. On OSX the build followed a fairly standard pattern:

    export PREFIX=/Users/pelson/miniconda/envs/fontforge

    git clone git@github.com:fontforge/fontforge.git
    cd fontforge && git checkout 20161012

    # Install some of font-forge's dependencies (I'm confident that there are others that are already
    # on my system, so better isolation would be needed to turn this into a proper conda-forge recipe).
    conda create -c conda-forge python=2.7 pango glib freetype libxml2 pkg-config -p ${PREFIX}
    source activate ${PREFIX}
    
    # Bootstrap the compilation. This eventually calls auto(re)conf.
    ./bootstrap --force

    # Tell fontforge where it can find PANGO.
    export PANGO_CFLAGS=-i${PREFIX}/include/pango-1.0
    export PANGO_LIBS=-lpango-1.0
    
    # Turn off the X11 interface - we just want the python package.
    ./configure --prefix=${PREFIX} --without-x --without-cairo
    make
    make install

You may notice that I forced the version back to python 2.7. It was a shame to have to use a legacy python version, but I was having issues with FontForge's creation of lookuptables with python 3.5+.

Unfortunately, this does mean I'm going to have to manually deal with "wide" unicode characters. At the time of writing conda only provides narrow python 2 build, which means it doesn't deal well with characters outside of unicode's "basic multilingual plane". Python 3.5+ completely did away with the "narrow" vs "wide" python concept and therefore the need to recompile python in order to natively handle some unicode characters properly!

Some more detail on wide vs narrow can be found at https://www.python.org/dev/peps/pep-0261/ and http://wordaligned.org/articles/narrow-python.

The best place I can find the python 3.5+ improvement documented is in Nick Coghlan's blog about why python 3 is better than python 2 out of the box.

In particular, with a narrow python build (i.e. a version before python 3.5 that has been compiled with the default settings) the following will occur:

$ python2
Python 2.7.12 | packaged by conda-forge | (default, Sep  8 2016, 14:41:48) 

>>> print(u'🎂')
🎂
>>> print(ord(u'🎂'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ord() expected a character, but string of length 2 found

To repeat myself, the fix for this kind of issue is one of the following:

recompile python in "wide" mode
upgrade to python 3.5+
deal with the special case in your code

In my situation, the first option would be a burden, the second option was not possible due to the fontforge bug mentioned above, so I'm forced with option 3: deal with the special case in my code.

With the above complete, I can now import fontforge and continue the saga...

import fontforge

fontforge.__version__

'2.0 git:fe1d84b7d05178885a36ba0adce0559a82a493d5'

I start by getting hold of the metadata from my SVG files.

import os
import glob
import parse

fnames = glob.glob('char_*.svg')

characters = []
for fname in fnames:
    # Sample filename: char_L2_P2_x378_y1471_x766_y1734_a-s_0x61-0x73.svg
    
    pattern = 'char_L{line:d}_P{position:d}_x{x0:d}_y{y0:d}_x{x1:d}_y{y1:d}_{char}_{hex_repr}.svg'
    result = parse.parse(pattern, os.path.basename(fname))
    
    # Split the hex representation by the "-" separator and convert to unicode ordinals.
    ords = [int(hex_char, 16) for hex_char in result['hex_repr'].split('-')]
    
    # Now, where possible, generate character glyphs.
    # In python 3.5+ this would be so much more pleasant.
    chars = tuple(unichr(ordinal) if ordinal < 65536 else ordinal
                  for ordinal in ords)

    # Let's deal with that second I variant that has the two crosses - typically used for I as a
    # pronoun.
    if len(chars) > 1 and ''.join(chars) == 'I-pronoun':
        chars = tuple(' I ')

    bbox = (result['x0'], result['y0'], result['x1'], result['y1'])
    characters.append([result['line'], result['position'], bbox, fname, chars])

First attempt¶

Because the handwriting sample included a few paragraphs, we have several characters which are repeated - eventually it would be great to use all of the information as single alternates withing the font, so that I get variation on glyphs (I haven't checked that this is a real thing, but it sounds like it should be).

For now though, just use the first instance of each character. I'll also use the first instance of each ligature (those glyphs, such as TT, that I couldn't separate in the previous installments).

First, set up the font globals:

def basic_font():
    font = fontforge.font()
    font.familyname = font.fontname = 'XKCD'
    font.encoding = "UnicodeFull"

    font.version = '1.0';
    font.weight = 'Regular';
    font.fontname = 'XKCD'
    font.familyname = 'XKCD'
    font.fullname = 'XKCD'
    font.em = 1024;
    font.ascent = 768;
    font.descent = 256;

    # We create a ligature lookup table.
    font.addLookup('ligatures', 'gsub_ligature', (), [[b'liga',
                                                       [[b'latn',
                                                         [b'dflt']]]]])
    font.addLookupSubtable('ligatures', 'liga')
    
    return font

Now I want to create each of the glyphs from our SVG images. This requires a call such as:

    c = font.createMappedChar(unicode_ordinal)
    c.importOutlines(svg_filename)

Unfortunately, FontForge can't currently deal with unicode filenames for its import (issue 3058), so I make a context manager that creates temporary non-unicode filenames that are just symlinks to the original file.

from contextlib import contextmanager
import tempfile
import shutil
import os


@contextmanager
def tmp_symlink(fname):
    """
    Create a temporary symlink to a file, so that applications that can't handle
    unicode filenames don't barf (I'm looking at you font-forge)
    
    """
    target = tempfile.mktemp(suffix=os.path.splitext(fname)[1])
    fname = os.path.normpath(os.path.abspath(fname))
    try:
        os.symlink(fname, target)
        yield target
    finally:
        if os.path.exists(target):
            os.remove(target)

Finally, I'm ready to start adding glyphs to my font. I track which ones I've added to avoid adding multiple versions of glyphs that have already been seen.

def create_char(font, chars, fname):
    if len(chars) == 1:
        # A single unicode character, so I create a character in the font for it.

        # Because I'm using an old & narrow python (<3.5) I need to handle the unicode
        # characters that couldn't be converted to ordinals (so I kept them as integers). 
        if isinstance(chars[0], int):
            c = font.createMappedChar(chars[0])
        else:
            c = font.createMappedChar(ord(chars[0]))
    else:
        # Multiple characters - this is a ligature. We need to register this in the
        # ligature lookup table we created. Not all font-handling libraries will do anything
        # with ligatures (e.g. matplotlib doesn't at vn <=2)

        ligature_name = '_'.join(chars)
        ligature_tuple = tuple([character.encode('ascii') for character in chars])
        
        c = font.createChar(-1, ligature_name)

        c.addPosSub('liga', ligature_tuple)

    c.clear()
        
    # Use the workaround to have non-unicode filenames.
    with tmp_symlink(fname) as tmp_fname:
        # At last, bring in the SVG image as an outline for this glyph.
        c.importOutlines(tmp_fname)

    return c


font = basic_font()

visited = []
for line, position, bbox, fname, chars in characters:
    if chars in visited:
        # We have already seen this set of chars. For now, I do nothing
        # with duplicates.
        continue
    visited.append(chars)
    
    c = create_char(font, chars, fname)

So it seems I have now created the appropriate FontForge font. I want to test this in a tool that can display fonts well. As it happens, I'm using jupyter notebook, which is browser based. Browsers display fonts well. Let's target that using the Web Open Font Format.

Creating a woff with FontForge is as easy as:

font.generate('xkcd.woff')

<fontforge.font at 0x0x1149e42f0 "XKCD">

Now I need to do some magic to use the font in the notebook...

I use IPython's display functionality to embed HTML directly in the notebook:

The HTML will essentially be the appropriate CSS directives to make use of the woff I just created.
In order to ensure the woff can be enjoyed by everybody, I embed the font as a base64 object directly in the CSS (the same trick that Jupyter notebooks use for images).
I'm going to iterate a few times on this font, so I get a unique handle to this font within my CSS by adding a uuid component to the IDs.
I'll do some fancy things to allow you, the reader, to type whatever you like and see the results.

import uuid
import base64
import textwrap
from IPython.display import display, HTML


def preview_font(fname, content, unique_id=None):
    if unique_id is None:
        unique_id = uuid.uuid4()
    
    with open(fname, 'rb') as fh:
        woff_base64 = encoded_string = base64.b64encode(fh.read())

    html = u"""
    <style>
    @font-face {{
        font-family: xkcd-{uid};
        src: url('data:font/opentype;base64,{woff_base64}') format('woff');
    }}

    .XKCD-{uid} {{
        font-family: xkcd-{uid};
        font-size: 2em;
        color: black;
        line-height: 150%;
        white-space: pre;
    }}
    </style>
    <a name="font-{uid}"></a>
    <textarea id="input-{uid}" style="width: 60%; min-width: 20em; height: 6em;">{content}</textarea>
    <div id="content-{uid}" class="XKCD-{uid}"></div>

    <script>
    function update_content() {{
        document.getElementById("content-{uid}").textContent = document.getElementById("input-{uid}").value;
    }}
    document.getElementById("input-{uid}").addEventListener('input', update_content);
    update_content();
    </script>""".format(uid=unique_id, woff_base64=woff_base64, content=content)

    font_preview = HTML(html)
    return font_preview


font_fname = 'xkcd.woff'
display(preview_font(font_fname, content="Hello world - this is my first attempt.\nLigature: BETTER"))

For a first attempt, it's not terrible (well, I can read it)... though there are clearly some pretty fundamental issues here.

If you haven't already done so, have a play with the text area to get a feel for the font that has been created.

The biggest issue is with the scaling of the glyphs (as highlighted by the comical period). It seems that when FontForge imports the SVGs, it scales the image to the size of the glyph being created - this isn't really the behaviour I want, as my images are already scaled appropriately. I'm going to need to get hold of the glyphs bounding box and scale the resulting geometries using fontforge's inbuilt affine matrix package psMat.

Some documentation on the python interface, particularly the Glyph class, can be found at https://fontforge.github.io/python.html#Glyph. It isn't the most comprehensive documentation, and in reality, I will probably need to experiment somewhat.

In this instance it was convenient to create a web open font format directly, but it is equally possible to write directly to the OTF or TTF formats with:

font.generate('xkcd.otf')
font.generate('xkcd.ttf')

<fontforge.font at 0x0x1149e42f0 "XKCD">

Scaling the glyphs¶

In order to scale the glyphs I will need to be able to state how much of the EM square the glyph should consume. To answer this question, I choose some characters that give me good approximations for baseline and cap-height - I need coverage on all lines of the original image, so am forced to include lower-case, numerical and symbolic characters as well the standard capitals.

baseline_chars = ['a' 'e', 'm', 'A', 'E', 'M', '&', '@', '.', u'≪', u'É']
caps_chars = ['M', 'A', 'E', 'k', 't', 'l', 'b', 'd', '1', '2', u'3', u'≪', '?', '!']

Using these, I keep track of the y positions for pertinent baselines and cap-heights for a given line:

line_stats = {}
for line, position, bbox, fname, chars in characters:
    if len(chars) == 1:
        this_line = line_stats.setdefault(line, {})
        char = chars[0]
        if char in baseline_chars:
            this_line.setdefault('baseline', []).append(bbox[3])
        if char in caps_chars:
            this_line.setdefault('cap-height', []).append(bbox[1])

The following functions define the scaling and translation of the glyphs according to their relative position in the original handwriting sample. The functions are somewhat dense, and you'd be forgiven for not actually reading them...

from __future__ import division

import numpy as np
import psMat

def scale_glyph(char, char_bbox, baseline, cap_height):
    # TODO: The code in this function is convoluted - it can be hugely simplified.
    # Essentially, all this function does is figure out how much
    # space a normal glyph takes, then looks at how much space *this* glyph takes.
    # With that magic ratio in hand, I now look at how much space the glyph *currently*
    # takes, and scale it to the full EM. On second thoughts, this function really does
    # need to be convoluted, so maybe the code isn't *that* bad...
    
    font = char.font
    
    # Get hold of the bounding box information for the imported glyph.
    import_bbox = c.boundingBox()
    import_width, import_height = import_bbox[2] - import_bbox[0], import_bbox[3] - import_bbox[1]
    
    # Note that timportOutlines doesn't guarantee glyphs will be put in any particular location,
    # so translate to the bottom and middle.
    
    target_baseline = char.font.descent
    top = char.font.ascent
    top_ratio = top / (top + target_baseline)
    
    y_base_delta_baseline = char_bbox[3] - baseline
    
    width, height = char_bbox[2] - char_bbox[0], char_bbox[3] - char_bbox[1]

    # This is the scale factor that font forge will have used for normal glyphs...
    scale_factor = (top + target_baseline) / (cap_height - baseline)
    glyph_ratio = (cap_height - baseline) / height
    
    # A nice glyph size, in pixels. NOTE: In pixel space, cap_height is smaller than baseline, so make it positive.
    full_glyph_size = -(cap_height - baseline) / top_ratio
    
    to_canvas_coord_from_px = full_glyph_size / font.em
    
    anchor_ratio = (top + target_baseline) / height
    
    # pixel scale factor
    px_sf = (top + target_baseline) / font.em
    
    frac_of_full_size = (height / full_glyph_size)
    import_frac_1000 = font.em / import_height
    
    t = psMat.scale(frac_of_full_size * import_frac_1000)
    c.transform(t)


def translate_glyph(c, char_bbox, cap_height, baseline):
    # Put the glyph in the middle, and move it relative to the baseline.

    # Compute the proportion of the full EM that cap_height - baseline should consume.
    top_ratio = c.font.ascent / (c.font.ascent + c.font.descent)
    
    # In the original pixel coordinate space, compute how big a nice full sized glyph
    # should be.
    full_glyph_size = -(cap_height - baseline) / top_ratio
    
    # We know that the scale of the glyph is now good. But it is probably way off in terms of x & y, so we
    # need to fix up its position.
    glyph_bbox = c.boundingBox()
    # No matter where it is currently, take the glyph to x=0 and a y based on its positioning in
    # the original handwriting sample.
    t = psMat.translate(-glyph_bbox[0], -glyph_bbox[1] + ((baseline - char_bbox[3]) * c.font.em / full_glyph_size))
    c.transform(t)

    # Put horizontal padding around the glyph. I choose a number here that looks reasonable,
    # there are far more sophisticated means of doing this (like looking at the original image,
    # and calculating how much space there should be).
    space = 20
    scaled_width = glyph_bbox[2] - glyph_bbox[0] 
    c.width = scaled_width + 2 * space
    t = psMat.translate(space, 0)
    c.transform(t)

font = basic_font()
font.ascent = 600;
    
for line, position, bbox, fname, chars in characters:
    c = create_char(font, chars, fname)

    # Get the linestats for this character.
    line_features = line_stats[line]

    scale_glyph(
        c, bbox,
        baseline=np.mean(line_features['baseline']),
        cap_height=np.mean(line_features['cap-height']))
    
    translate_glyph(
        c, bbox,
        baseline=np.mean(line_features['baseline']),
        cap_height=np.mean(line_features['cap-height']))

    # Simplify, then put the vertices on rounded coordinate positions.
    c.simplify()
    c.round()


# Take a look at this refined font.
font.generate(font_fname)

preview_content = u"""\
Hello world!! Testing some ligatures: BETTER;
XKCD IS OFTEN IN *CAPITALS*, SO IT'S (SEMI-) IMPORTANT TO GET IT "RIGHT"...!?!
Can one really have their 🎂 AND eat it, I wonder. Hmmm caaake! 🎂 Cool.
C:\\\\ 12345 (6+8)/2=7 </html> @ http://fooj.com/under_score/dash-dot."""

display(preview_font(font_fname, content=preview_content))

Really not too shabby... some attention to detail is needed now though. There are a number of issues with this font that I think are readily addressable:

I really dislike the capital C and G - they come from the prose part of the sample handwriting dataset, and I'd rather use the glyphs from the alphabet line above for these particular characters
The spacing on the ligatures is better, but not perfect. The point of the ligatures is to make the characters flow smoothly, and there is simply too much space between the "TT" and "ER" of "BETTER".
The spacing on the lower case "g" and "j" could do with some work.
The kerning on the capital "T" and lower case "r" (when followed by "e") need some work.
The slashes (both forward and back) when paired could do with being closer together.

The finished article¶

The biggest issue I had with the previous attempt was related to glyph spacing. For that reason, I'm going to focus on kerning, which is essentially a place for special case spacing between character pairs. All the advice I've read about kerning is that it is necessary, but should be used sparing (in favour of improved glyph horizontal space).

In order to add kerning to the font, I need to add a kerning lookup table, and a subtable to hold the kern pairs. I can then ask fontforge to determine sensible kerning pairs based on the space that I'd like to have between the character groups.

def autokern(font):
    # Let fontforge do some magic and figure out automatic kerning for groups of glyphs.

    all_glyphs = [glyph.glyphname for glyph in font.glyphs()
                  if not glyph.glyphname.startswith(' ')]    

    ligatures = [name for name in all_glyphs if '_' in name]
    upper_ligatures = [ligature for ligature in ligatures if ligature.upper() == ligature]
    lower_ligatures = [ligature for ligature in ligatures if ligature.lower() == ligature]
    
    caps = list('ABCDEFGHIJKLMNOPQRSTUVWXYZ') + upper_ligatures
    lower = list('abcdefghijklmnopqrstuvwxyz') + lower_ligatures
    all_chars = caps + lower

    # Add a kerning lookup table.
    font.addLookup('kerning', 'gpos_pair', (), [[b'kern', [[b'latn', [b'dflt']]]]])
    font.addLookupSubtable('kerning', 'kern')
    
    # Everyone knows that two slashes together need kerning... (even if they didn't realise it)
    font.autoKern('kern', 150, ['slash', 'backslash'], ['slash', 'backslash'])

    font.autoKern('kern', 60, ['r', 's'], lower, minKern=50)
    font.autoKern('kern', 100, ['f'], lower, minKern=50)
    font.autoKern('kern', 180, all_chars, ['j'], minKern=35)
    
    font.autoKern('kern', 150, ['T', 'F'], all_chars)
    font.autoKern('kern', 30, ['C'], all_chars)

font = basic_font()
font.ascent = 600;

# Pick out particular glyphs that are more pleasant than their latter alternatives.
special_choices = {('C', ): dict(line=4),
                   ('G',): dict(line=4),
                   # Get rid of the "as" ligature - it's not very good.
                   ('a', 's'): dict(line=None),
                   # A nice tall I.
                   ('I', ): dict(line=4)}

for line, position, bbox, fname, chars in characters:
    if chars in special_choices:
        spec = special_choices[chars]
        spec_line = spec.get('line', any)
        if spec_line is not any and spec_line != line:
            continue
        
    c = create_char(font, chars, fname)

    # Get the linestats for this character.
    line_features = line_stats[line]

    scale_glyph(
        c, bbox,
        baseline=np.mean(line_features['baseline']),
        cap_height=np.mean(line_features['cap-height']))
    
    translate_glyph(
        c, bbox,
        baseline=np.mean(line_features['baseline']),
        cap_height=np.mean(line_features['cap-height']))

    # Simplify, then put the vertices on rounded coordinate positions.
    c.simplify()
    c.round()

autokern(font)

font.generate(font_fname)
display(preview_font(font_fname, content=preview_content, unique_id='final'))

What's missing?¶

First up, I want to remind myself what the current version of the XKCD font looks like (taken from SHA 5a662e1 of ipython/xkcd-font):

original_font_fname = os.path.expanduser('~/Downloads/xkcd-Regular.otf')

content = "This is the original font, not the one I've created:\n" + preview_content
display(preview_font(original_font_fname, content=content))

This font is far more regular in its vertical alignment, but in reality the original handwriting sample that my work is based upon is far from regular.

Obviously there are some issues with the font weight from this version - it is particularly stark for caps vs lower case characters, but even within these groups there are obvious issues. In the URL example, the lowercase "j" seems to be scaled poorly, and the lowercase "u", "n" and "d" are very different in appearance (scale and thickness).

Having taken a look at this, I'm very pleased with how my work has turned out. My remaining question is whether there are any glyphs in the original font that aren't in mine?

original_font = fontforge.open(original_font_fname)
original_glyphs = set([glyph.glyphname for glyph in original_font.glyphs()])
new_glyphs = set([glyph.glyphname for glyph in font.glyphs()])

missing_glyphs = sorted(original_glyphs - new_glyphs)
print('Missing glyphs in new font: {}'.format(', '.join(missing_glyphs)))

Missing glyphs in new font: .notdef, bar, bullet, circumflex, copyright, degree, ellipsis, emdash, endash, minus, multiply, paragraph, questiondown, registered, section, space, trademark

missing_chars = []
for glyph_name in missing_glyphs:
    glyph = original_font[glyph_name]
    if glyph.unicode > 0:
       missing_chars.append(unichr(glyph.unicode))

display(
    preview_font(
        original_font_fname,
        content='Missing chars: ' + ' '.join(missing_chars)))

It seems that some of these may be useful, but I don't currently know how to easily copy a glyph from one font file to another (it should be achievable by exporting the contours, and then re-importing them, but that sounds like hard work for not a huge amount of benefit). If you know how to do it easily, you might consider answering https://superuser.com/q/334971/452816 and letting me know in the comments.

Wrap-up¶

So that is the end of my XKCD font saga (for now at least). I've learnt a huge amount along the way and strongly encourage this kind of exploration as a means of learning new (and unexpected!) things. On this journey I've learnt:

How to use scipy & scikit-image's image labeling functionality to classify distinct features in an image, and to extract them as separate entities for further processing
How to convert raster images (PPM) into SVGs using potrace
How to compile FontForge on OSX
How to create a basic font with FontForge's scripting tools
How to embed a web font as base64 object in a webpage (in this case, the Jupyter Notebook interface)
A whole heap about Fonts, including the EM square, ligatures and kerning
Patience - because finding out most of this stuff has been trial and error!

I'm really pleased with the outcome of this investigation, and will be turning all of this work into a PR on the ipython/xkcd-font repo. I'm hopeful that my improvements to the font will be beneficial to all the XKCD lovers out there - I'm particularly excited about using it in my matplotlib figures (who am I kidding, I'm excited about using it everywhere!).

Enjoy!!! :)

Classifying segmented strokes as characters - Part 3 of an XKCD font saga

2017-04-01T00:00:00+01:00

In part two of my XKCD font saga I was able to separate strokes from the XKCD handwriting dataset into many smaller images. I also handled the easier cases of merging some of the strokes back together - I particularly focused on "dotty" or "liney" type glyphs, such as i, !, % and =.

Now I want to attribute a Unicode character to my segmented images, so that I can subsequently generate a font-file. We are well and truly in the domain of optical character recognition (OCR) here, but because I want absolute control of the results (and 100% accuracy) I'm going to take the simple approach of mapping glyph positions to characters myself.

If you'd like to follow along, the input files for this article may be found at https://gist.github.com/pelson/b80e3b3ab9edbda9ac4304f742cf292b, the notebook and output may be found https://gist.github.com/pelson/1d6460289f06acabb650797b88c15ae0.

As a reminder, here is a down-sampled version the XKCD handwriting file:

First, I want to read in the files that we produced previously. These can be found at https://gist.github.com/pelson/b80e3b3ab9edbda9ac4304f742cf292b.

I get metadata from the filename using the excellent parse library.

import glob
import os

import matplotlib.pyplot as plt
import numpy as np
import parse


pattern = 'stroke_x{x0:d}_y{y0:d}_x{x1:d}_y{y1:d}.png'
strokes_by_bbox = {}
for fname in glob.glob('../b80e3b3ab9edbda9ac4304f742cf292b/stroke*.png'):
    result = parse.parse(pattern, os.path.basename(fname))
    bbox = (result['x0'], result['y0'], result['x1'], result['y1'])
    img = (plt.imread(fname) * 255).astype(np.uint8)
    strokes_by_bbox[bbox] = img

In order to map strokes to glyphs, I'm going to want to sort them first by line, then from left to right.

At this point, I could either hard code the approximate y location of each baseline or attempt to cluster them. I opt for the latter, using a k-means implementation from scipy.cluster.vq.kmeans.

To confirm I have the correct information and to visualise the data before clustering, I plot the baseline point (maximum y pixel location) of the stroke against its left-most x position:

import matplotlib.pyplot as plt
import numpy as np

xs, ys = zip(*[[bbox[0], bbox[3]]
               for bbox, img in strokes_by_bbox.items()])
xs = np.array(xs, dtype=np.float)
ys = np.array(ys, dtype=np.float)
plt.scatter(xs, ys, s=50)
plt.gca().invert_yaxis()
plt.autoscale(tight=True)
plt.show()

There appears to be enough structure to group the glyphs into lines... let's go for it.

import scipy.cluster.vq

n_lines = 11

lines, _ = scipy.cluster.vq.kmeans(ys, n_lines, iter=1000)
lines = np.array(sorted(lines))

print('Approx baseline pixel of lines:', repr(lines.astype(np.int16)))

# Plot the strokes, and color by their nearest line.
plt.scatter(xs, ys, s=50,
            c=np.argmin(np.abs(ys - lines[:, np.newaxis]), axis=0), cmap='Paired')
plt.gca().invert_yaxis()
plt.autoscale(tight=True)

# Draw the line positions
plt.hlines(lines, xmin=0, xmax=xs.max())
plt.show()

Approx baseline pixel of lines: array([ 447, 1103, 1708, 2316, 2932, 3380, 3830, 4276, 4713, 5584, 6045], dtype=int16)

Using the baseline position, group the strokes into their respective lines, and then sort each line from left to right.

glyphs_by_line = [[] for _ in range(n_lines)]
for bbox, img  in list(strokes_by_bbox.items()):
    nearest_line = np.argmin(np.abs(bbox[3] - lines))
    glyphs_by_line[nearest_line].append([bbox, img])

# Put the glyphs in order from left-to-right.
for glyph_line in glyphs_by_line:
    glyph_line.sort(key=lambda args: args[0][0])

Let's take a look using the same notebook trick we did in part 1...

from IPython.display import display, HTML
from io import BytesIO
import PIL
import base64


def html_float_image_array(img, downscale=5, style="display: inline; margin-right:20px"):
    """
    Generates a base64 encoded image (scaled down) that is displayed inline.
    Great for showing multiple images in notebook output.

    """
    im = PIL.Image.fromarray(img)
    bio = BytesIO()
    width, height = img.shape[:2]
    if downscale:
        im = im.resize([height // downscale, width // downscale],
                       PIL.Image.ANTIALIAS)
    im.save(bio, format='png')
    encoded_string = base64.b64encode(bio.getvalue())
    html = ('<img src="data:image/png;base64,{}" style="{}"/>'
            ''.format(encoded_string.decode('utf-8'), style))
    return html

To keep this somewhat short, I'll just show a few lines:

for glyph_line in glyphs_by_line[::4]:
    display(HTML(''.join(html_float_image_array(img, downscale=10)
                         for bbox, img in glyph_line)))

Now the tedious part - mapping glyph positions to characters (and ligatures)...

paragraph = r"""
a b c d e f g h i j k l m n o p q r s t u v w x y z
u n a u t h o r i t a t i v e n e s s   l e a t h e r b a r k   i n t r a co l i c   m i c r o c h e i l i a   o f f s i d e r
g l as s w e e d   r o t t o l o   a l b e r t i  t e   h e r m a t o r r h a c h i s   o r g a n o m e t a l l i c
s e g r e g a t i o n i s t   u n e v a n g e l i c   c a m p s t oo l
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z I-pronoun
U N A U T H O R I T A T I V E N E S S   L EA T H E R B A R K   I N T R A CO L I C   M I CR OCH E L I A
O F F S I D ER   G LA S S W EE D   R O TT O L O   A LB E R T I T E   H ER M A T O RR H A C H I S
O R G A N O M E T A LL I C   S E G R E G A T I ON I S T   U N E V A N G E L I C   CA M PS TO O L
+ - x * ! ? # @ $ % ¦ & ^ _ - - - ( ) [ ] { } / \ < > ÷ ± √ Σ
1 2 3 4 5 6 7 8 9 0 ∫ = ≈ ≠ ~ ≤ ≥ |> <| 🎂 . , ; : “ H I ” ’ ‘ C A N ' T ' "
É Ò Å Ü ≪ ≫ ‽ Ē Ő “ ”
""".strip()
paragraphs = [[char for char in line.replace('   ', ' ').split(' ') if char]
              for line in paragraph.split('\n')]

As I mentioned in the previous post, joining together all strokes into constituent glyphs wasn't easily achievable. We therefore have four glyphs that each require two strokes to be merged before we can map them to the appropriate unicode character(s). The glyphs in question are |>, <|, ≪ and ≫.

glyphs_needing_two_strokes = ['≪', '≫', '|>', '<|']

We need the function from part 2 which merges together two images:

def merge(img1, img1_bbox, img2, img2_bbox):
    bbox = (min([img1_bbox[0], img2_bbox[0]]),
            min([img1_bbox[1], img2_bbox[1]]),
            max([img1_bbox[2], img2_bbox[2]]),
            max([img1_bbox[3], img2_bbox[3]]),
            )
    shape = bbox[3] - bbox[1], bbox[2] - bbox[0], 3
    img1_slice = [slice(img1_bbox[1] - bbox[1], img1_bbox[3] - bbox[1]),
                  slice(img1_bbox[0] - bbox[0], img1_bbox[2] - bbox[0])]
    
    img2_slice = [slice(img2_bbox[1] - bbox[1], img2_bbox[3] - bbox[1]),
                  slice(img2_bbox[0] - bbox[0], img2_bbox[2] - bbox[0])]

    merged_image = np.zeros(shape, dtype=np.uint8)
    merged_image.fill(255)
    merged_image[img1_slice] = img1
    merged_image[img2_slice] = np.where(img2 != 255, img2, merged_image[img2_slice])
    return merged_image, bbox

characters_by_line = []

for line_no, (character_line, glyph_line) in enumerate(zip(paragraphs, glyphs_by_line)):
    glyph_iter = iter(glyph_line)
    characters_this_line = []
    characters_by_line.append(characters_this_line)

    for char_no, character in enumerate(character_line):
        bbox, img = next(glyph_iter)
        if character in glyphs_needing_two_strokes:
            other_bbox, other_img = next(glyph_iter)
            img, bbox = merge(img, bbox, other_img, other_bbox)
        characters_this_line.append([character, bbox, img])

import html
for line in characters_by_line[-3:]:
    display(HTML(''.join(html.escape(char) + html_float_image_array(img)
                         for char, bbox, img in line)))

+-x*!?#@$%¦&^_---()[]{}/\<>÷±√Σ

1234567890∫=≈≠~≤≥|><|🎂.,;:“HI”’‘CAN'T'"

ÉÒÅÜ≪≫‽ĒŐ“”

Finally, save out each glyph as a PPM.

import skimage.io


replacements = {'/': 'forward-slash', '_': 'underscore'}

for line_no, line in enumerate(characters_by_line):
    for char_no, (char, bbox, img) in enumerate(line):
        char_repr = '-'.join(replacements.get(c, c) for c in char)
        hex_repr = '-'.join(str(hex(ord(c))) for c in char)
        
        fname = ('char_L{}_P{}_x{}_y{}_x{}_y{}_{char_repr}_{hex_repr}.ppm'
                 ''.format(line_no, char_no, *bbox, char_repr=char_repr, hex_repr=hex_repr))
        skimage.io.imsave(fname, img)

Conclusion

In this edition, I merged the few remaining strokes together to produce the finished glyphs, and classified each of the glyphs with associated characters (mostly Unicode). I then saved these rasters out to a semantic filename in the PPM format. Next up, convert the rasters to vector SVGs so that we can import them into our font tool programmatically.

The next article in this series is: Converting PPM to SVG, and creating a font with FontForge.

Segment, extract, and combine features of an image with SciPy and scikit-image - Part 2 of an XKCD font saga

2017-03-20T00:00:00+00:00

In part one of XKCD font saga I gave some background on the XKCD handwriting dataset, and took an initial look at image segmentation in order to extract the individual strokes from the scanned image. In this instalment, I will apply the technique from part 1, as well as attempting to merge together strokes to form (some of) the glyphs desired.

I'm going to pay particular attention to "dotted" glyphs, such as "i", "j", ";" and "?". I will need to do future work to merge together non-dotted glyphs such as the two arrows from "≫", as these are indistinguishable from two characters that happen to be close to one another.

If you'd like to follow along, this notebook and the handwriting file may be found at https://gist.github.com/pelson/b80e3b3ab9edbda9ac4304f742cf292b.

We start by using the technique from part one to segment the images into individual strokes.

import matplotlib.pyplot as plt
from skimage.color import rgb2gray
import scipy.ndimage.measurements
from skimage import measure
import numpy as np
from scipy import ndimage as ndi


handwriting_img = plt.imread('handwriting_minimal.png')
handwriting_img_gray = rgb2gray(handwriting_img)

labels, _ = ndi.label(handwriting_img_gray < 1)

stroke_locations = measure.regionprops(labels)

Using this information, we create an image array that isolates the label, and use the sub-image's bounding box as a lookup:

bbox_to_stroke_img = {}

for stroke in stroke_locations: 
    # [miny, minx, maxy, maxx] (I don't know why its that way around...)
    bbox = stroke.bbox

    # Construct a slice that can be used to pick out this bounding box from
    # the full image.
    full_index = [slice(bbox[0], bbox[2]), slice(bbox[1], bbox[3])]
    
    # Pick out the sub-image, and take a copy so that we can modify it without
    # modifying the original.
    stroke_img = handwriting_img[full_index + [Ellipsis]].copy()
    
    # Using the "labels" array, produce a binary mask that is True for every
    # pixel that is marked as this label, and False otherwise. 
    stroke_mask = labels[full_index] == stroke.label

    # For each color channel, use the mask to maintain the full image pixels that
    # are part of this stroke. Where a pixel remains that is not part of this stroke,
    # replace it with 1 (ultimately making it white).
    for channel in range(3):
        stroke_img[:, :, channel] = np.where(stroke_mask, stroke_img[:, :, channel], 1)
        
    # Convert the image to an RGB byte array.
    stroke_img = (stroke_img * 255).astype(np.uint8)

    bbox_to_stroke_img[bbox] = stroke_img

Pick off potentially wide, but not high, strokes - these are our dots and lines that we may want to merge together (though not in all cases - we do want some punctuation glyphs to remain!).

stroke_merge_contenders = {}

for bbox, img in bbox_to_stroke_img.items():
    height = bbox[2] - bbox[0]
    width = bbox[3] - bbox[1]
    if width < 450 and height < 150:
        stroke_merge_contenders[bbox] = img

Great. Let's do some work to take a look at these images in the notebook. Essentially, the easiest approach is to convert the images to base64 PNGs, and display them in raw HTML. I also want the images to display inline so that I can show many images at once.

from IPython.display import display, HTML
from io import BytesIO
import PIL
import base64


def html_float_image_array(img, downscale=5, style="display: inline; margin-right:20px"):
    """
    Generates a base64 encoded image (scaled down) that is displayed inline.
    Great for showing multiple images in notebook output.

    """
    im = PIL.Image.fromarray(img)
    bio = BytesIO()
    width, height = img.shape[:2]
    if downscale and (width > downscale ** 2 and height > downscale ** 2):
        im = im.resize([height // downscale, width // downscale],
                       PIL.Image.ANTIALIAS)
    im.save(bio, format='png')
    encoded_string = base64.b64encode(bio.getvalue())
    html = ('<img src="data:image/png;base64,{}" style="{}"/>'
            ''.format(encoded_string.decode('utf-8'), style))
    return html

Using this new function on our stroke_merge_contenders:

display(HTML(''.join(html_float_image_array(img)
                     for bbox, img in stroke_merge_contenders.items())))

Now, let's take a look at the bounding boxes of these marks, so that we can get some intuition about what we have picked out:

import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

plt.figure(figsize=(10, 10))
ax = plt.axes()
ax.imshow(handwriting_img)
for bbox, img in stroke_merge_contenders.items():
    height = bbox[2] - bbox[0]
    width = bbox[3] - bbox[1]
    rect = mpatches.Rectangle([bbox[1], bbox[0]], width=width, height=height,
                                facecolor='none', edgecolor='blue')
    ax.add_patch(rect)
plt.show()

This looks promising. Let's try to merge the chosen images to other, suitably close, strokes. In all situations, we want to allow a reasonable y distance, and a small x distance (e.g. consider the strokes for a speech mark ").

First, define the function that can actually do the merging of two images for us. We will have two image arrays and two bounding boxes, and we want to come up with a single image array and bounding box.

def merge_images(img1, img1_bbox, img2, img2_bbox):
    """
    A function to merge together two images with different bounding boxes.

    """
    # The new image bounding box.
    bbox = (min([img1_bbox[0], img2_bbox[0]]),
            min([img1_bbox[1], img2_bbox[1]]),
            max([img1_bbox[2], img2_bbox[2]]),
            max([img1_bbox[3], img2_bbox[3]]))
    
    # The new image shape.
    shape = (bbox[2] - bbox[0], bbox[3] - bbox[1], 3)
    
    # The slice for image 1 inside of the new image array.
    img1_slice = [slice(img1_bbox[0] - bbox[0], img1_bbox[2] - bbox[0]),
                  slice(img1_bbox[1] - bbox[1], img1_bbox[3] - bbox[1])]
    
    # The slice for image 2 inside of the new image array.
    img2_slice = [slice(img2_bbox[0] - bbox[0], img2_bbox[2] - bbox[0]),
                  slice(img2_bbox[1] - bbox[1], img2_bbox[3] - bbox[1])]

    # Construct the new image, and fill it with white.
    merged_image = np.empty(shape, dtype=np.uint8)
    merged_image.fill(255)
    
    # Use all of image 1 and just drop it into the correct location within the new image.
    merged_image[img1_slice] = img1
    
    # We can't use the same approach for image 2, as it potentially overlaps with image 1.
    # Instead we use the parts of image 2 that aren't at the maximum of each color channel. 
    merged_image[img2_slice] = np.where(img2 != 255, img2,
                                        merged_image[img2_slice])
    
    return merged_image, bbox

This function probably also exists within skimage. Now that we have the ability to merge together images, let's define some functions that will help us measure the distance between potential images.

def min_interval_distance(interval_1, interval_2):
    """
    Calculate the distance between two intervals.
    
       >>> min_interval_distance([0, 1], [2, 3])
       1
       >>> min_interval_distance([0, 1], [0.5, 3])
       0
       >>> min_interval_distance([10, 11], [5, 8])
       2
       >>> min_interval_distance([10, 11], [8, 5])
       2

    There is so much room for more elegance here, but hey-ho...

    """
    interval_1_sorted = sorted(interval_1)
    interval_2_sorted = sorted(interval_2)

    min_distance = min([np.abs(i_1 - i_2) for i_1 in interval_1 for i_2 in interval_2])
    within_1 = any(interval_1_sorted[0] <= i_2 <= interval_1_sorted[1] for i_2 in interval_2)    
    within_2 = any(interval_2_sorted[0] <= i_1 <= interval_2_sorted[1] for i_1 in interval_1)

    if within_1 or within_2:
        min_distance = 0
    return min_distance


def max_interval_distance(interval_1, interval_2):
    """
    Calculate the distance between two intervals.
    
       >>> max_interval_distance([0, 1], [2, 3])
       3
       >>> max_interval_distance([0, 1], [0.5, 3])
       3
       >>> max_interval_distance([10, 11], [5, 8])
       6
       >>> max_interval_distance([10, 11], [8, 5])
       6

    There is so much room for more elegance here, but hey-ho...

    """
    max_distance = max([np.abs(i_1 - i_2) for i_1 in interval_1 for i_2 in interval_2])
    return max_distance


def contains(interval_1, interval_2):
    """
    Whether one bounding box entirely contains another
    
       >>> contains([0, 4], [2, 3])
       True
       >>> contains([0, 1], [0.5, 3])
       False
       >>> contains([2, 3], [4, 0])
       True
    """
    interval_1_sorted = sorted(interval_1)
    interval_2_sorted = sorted(interval_2)

    within_1 = all(interval_1_sorted[0] <= i_2 <= interval_1_sorted[1] for i_2 in interval_2)    
    within_2 = all(interval_2_sorted[0] <= i_1 <= interval_2_sorted[1] for i_1 in interval_1)
    return within_1 or within_2


# Check the function actually works...
import doctest; doctest.testmod()

TestResults(failed=0, attempted=11)

A neat trick from the functions above is that I did all of my development in the notebook by starting with function stubs, docstrings and doctests. Only then did I begin to gradually fill in the functionality. I don't often do development in the notebook, but in situations where it makes sense, the doctest technique I've used is an approach that allows you to write testable code in the notebook fairly easily.

Now for the hard work - we want to look at each of the stroke_merge_contenders and find potential pairs of images that may be merged. No doubt the metrics themselves to determine "suitable" pairs will require some development iteration.

from collections import OrderedDict, namedtuple

Metric = namedtuple('Metric', ['maxx', 'maxy', 'minx', 'miny', 'withinx', 'withiny'])


# Track the images that ultimately got merged.
images_that_were_mergers = []

# Track the images that didn't find a merge buddy.
images_that_didnt_get_merged = []

# Take a copy of our existing images - we will remove merged strokes from here, and replace
# with the composite.
merged_bbox_to_stroke_img = bbox_to_stroke_img.copy()

# We could use an RTree for efficient nearest neighbour lookup, but just use a greedy search for now.
reduced_stroke_merge_contenders = stroke_merge_contenders.copy()


while reduced_stroke_merge_contenders:
    bbox, img = reduced_stroke_merge_contenders.popitem()
    
    # If we have already processed this file (as the right hand pair of a merge) then skip it.
    if bbox not in merged_bbox_to_stroke_img:
        continue
    
    # Remove this image from the final glyph set.
    merged_bbox_to_stroke_img.pop(bbox, None)

    candidates = []
    for other_bbox, other_img in list(merged_bbox_to_stroke_img.items()):
        if other_bbox == bbox:
            # We don't want to merge with ourselves.
            continue
        
        x_range, x_range_other = [other_bbox[1], other_bbox[3]], [bbox[1], bbox[3]]
        minx = min_interval_distance(x_range, x_range_other)
        miny = min_interval_distance([other_bbox[0], other_bbox[2]], [bbox[0], bbox[2]])
        
        maxx = max_interval_distance([other_bbox[1], other_bbox[3]], [bbox[1], bbox[3]])
        maxy = max_interval_distance([other_bbox[0], other_bbox[2]], [bbox[0], bbox[2]])
        
        withinx = contains([other_bbox[1], other_bbox[3]], [bbox[1], bbox[3]])
        withiny = contains([other_bbox[0], other_bbox[2]], [bbox[0], bbox[2]])
        
        metric = Metric(minx=minx, miny=miny, maxx=maxx, maxy=maxy, withinx=withinx, withiny=withiny)
        
        # The condition for which a pair of glyphs will be consiered for merging.
        # This took a significant amount of iteration to get a good performance.
        if (withinx and withiny) or \
                (((maxx < 250 or minx == 0) and minx < 10) and miny < 50) or \
                (other_bbox in reduced_stroke_merge_contenders and minx == 0 and maxy < 350):    
            candidates.append([metric, other_bbox, other_img])

    if not candidates:
        images_that_didnt_get_merged.append([img, bbox])
        # Put the stroke back into the new_bboxes so that we don't loose it.
        merged_bbox_to_stroke_img[bbox] = img
    else:
        # Prefer the candidates with a small maximum x over anything else.
        candidates.sort(key=lambda candidate: (candidate[0].maxx, candidate))

        metric, other_bbox, other_img = candidates[0]
        merged_image, merged_bbox = merge_images(img, bbox, other_img, other_bbox)

        # We're done with the other image too - remove it from the results.
        merged_bbox_to_stroke_img.pop(other_bbox)

        # If the resulting image is *tiny* then consider putting it back
        # into the pool of items that may be merged. This really is only of
        # value to the sprinkles within the cupcake. 
        height = merged_bbox[2] - merged_bbox[0]
        width = merged_bbox[3] - merged_bbox[1]
        if width < 100 and height < 100:
            reduced_stroke_merge_contenders[merged_bbox] = merged_image

        images_that_were_mergers.append(merged_bbox)
        merged_bbox_to_stroke_img[merged_bbox] = merged_image

print("Images that didn't get merged")
display(HTML(''.join(html_float_image_array(img) for img, bbox in images_that_didnt_get_merged)))
print('-' * 60)
print('Images that got merged')
display(HTML(''.join(html_float_image_array(img)
                     for bbox, img in merged_bbox_to_stroke_img.items()
                     if bbox in images_that_were_mergers)))

Images that didn't get merged

------------------------------------------------------------
Images that got merged

We need to double check what didn't get merged against the original image:

import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

plt.figure(figsize=(10, 10))
ax = plt.axes()
ax.imshow(handwriting_img)
for img, bbox in images_that_didnt_get_merged:
    height = bbox[2] - bbox[0]
    width = bbox[3] - bbox[1]
    rect = mpatches.Rectangle([bbox[1], bbox[0]], width=width, height=height,
                                facecolor='none', edgecolor='blue')
    ax.add_patch(rect)
plt.show()

Finally, let's take a look at all of the characters that we have produced.

display(HTML(''.join(html_float_image_array(img, downscale=6)
                     for bbox, img in merged_bbox_to_stroke_img.items())))

Awesome. You may notice that there are a few character pairs (and even one triplet) - rather than attempt to separate these, I will instead keep them as a form of natural variation and attempt to use them as "ligatures" within the final font.

Let's write these strokes to disk so that we can bank what we have got and then move on to attributing the strokes to glyphs.

import skimage.io

for bbox, img_array in merged_bbox_to_stroke_img.items():
    fname = 'stroke_x{1}_y{0}_x{3}_y{2}.png'.format(*bbox)
    skimage.io.imsave(fname, img_array)

Conclusion

In summary, I first used scipy.ndimage.label to pick out the individual features of the handwriting image. I then extracted the labels from the image, before blending together suitably small images (mostly the dots and short+wide strokes) into slightly larger composite-strokes. This was an effective technique for bringing back together things like the dots on the glyphs i, j and !, as well as joining together things like =, % and ±.

Inputs, code, and output can all be found at https://gist.github.com/pelson/b80e3b3ab9edbda9ac4304f742cf292b.

The next article in this series is: Classifying segmented strokes as characters.

Playing with Randall Munroe's XKCD handwriting

2017-03-16T00:00:00+00:00

The XKCD font (as used by matplotlib et al.) recently got an update to include lower-case characters. For some time now I have been aware of a handwriting sample produced by Randall Munroe (XKCD's creator) that I was interested in exploring. The ultimate aim is to automatically produce a font-file using open source tools, and to learn a few things along the way.

The thing about fonts is that there is actually a lot going on:

spacing - the whitespace around a character
kerning - special case whitespace adjustments (e.g. notice the space between "Aw" is much closer than between "As")
hinting - techniques for improved rasterization at low resolution
ligatures - special pairs/groups of characters. The traditional example is ffi, but for hand-writing, any character combination is plausible.

The raw material I'm going to use is a scan produced by XKCD author Randall Munroe containing many characters, as well as some spacing and ligature information. Importantly, all of the glyphs have been written at the same scale and using the same pen. These two details are important, as they will allow us to derive appropriate spacing and kerning information, and should result in a font that is well balanced.

(full image available on GitHub at https://github.com/ipython/xkcd-font/issues/9#issuecomment-127412261)

Notice some interesting features of this sample, including ligatures (notice that "LB" is a single mark) and kerning (see the spacing of "VA"):

There are a few useful articles already out there on this topic, particularly one from 2010 regarding the creation of fonts from a hand-written sample. Notably though, there isn't much that is automated - this is a rub for me, as without automation it is challenging for others to contribute to the font in an open and diffable way.

Let's get stuck in by separating each of the glyphs from the image into their own image.

We start by getting hold of the image as a numpy array. Original image cropped from https://github.com/ipython/xkcd-font/issues/9#issuecomment-127412261:

import matplotlib.pyplot as plt
hand = plt.imread('../../images/xkcd-font/handwriting_minimal.png')

This is a pretty beefy image, so we will want to cut it down a little whilst doing some development. Once we are happy with the process, we can then use the full image:

print(hand.shape)

(6099, 11429, 3)

plt.imshow(hand)
plt.show()

Before we even start, notice that some of the characters overlap, and even touch one another. This isn't going to be a clean bounding box type problem:

small_image = hand[3000:4500, 4000:7000]
plt.imshow(small_image)
plt.show()

from skimage.color import rgb2gray
from skimage.color.colorlabel import label2rgb
from scipy import ndimage as ndi
import numpy as np

Convert to grayscale, and label the image by distinct marks.

small_image_gray = rgb2gray(small_image)
labels, _ = ndi.label(small_image_gray < 1)

Now convert the labels to RGB so that we can see our different segmentations:

labels = np.ma.masked_equal(labels, 0)
label_img = label2rgb(labels.data, colors=['red', 'blue', 'green', 'black', 'yellow'],
                      bg_label=0, bg_color=(1, 1, 1))
plt.imshow(label_img)
plt.show()

Our first attempt to separate the letters has had some success. Let's compute each glyph's bounding box, and add that to the image. Initially, I tried using scipy.ndimage.measurements.extrema to give me each label's bounding box, but ended up needing to brute-force it using np.where.

Note from the future: * skimage.measure.regionprops is the best tool for the job.*

glyph_locations = {}

for i in range(1, labels.max()):
    ys, xs = np.where(labels == i)
    bbox = [xs.min(), ys.min(), xs.max(), ys.max()]
    glyph_locations[i] = bbox

Now that we have the locations of the labels, let's draw a rectangle around each distinct label.

import matplotlib.patches as mpatches

ax = plt.axes()
plt.imshow(label_img)
 
for glyph_i, bbox in glyph_locations.items():
    height, width = bbox[3] - bbox[1], bbox[2] - bbox[0]
    rect = mpatches.Rectangle([bbox[0], bbox[1]], width, height,
                             edgecolor='blue', facecolor='none', transform=ax.transData)
    ax.patches.append(rect)
    
plt.show()

Conclusion

In this initial phase, we haven't done anything particularly clever - we've simply loaded in the image, taken a subset, and used scipy's image labelling capabilities to understand what the labelling process looks like. Originally I had planned to try to separate some of the glyphs that were obviously fused together - I'd even gone as far as prototyping using a filter to dissolve the outline (so that I could separate the labels) and then subsequently growing the labels again back to their original form (but keeping the separate labels). This technique worked, but it produced shapes that weren't perfect, and the complexities (e.g. handling of bits that disappeared, such as dots on the letter "i") weren't worth it.

In the next phase, we will use the technique shown here to generate individual image files. In addition, we will apply some heuristics merge back together glyphs such as the dot and comma of a semi-colon.

The next article in this series is: Segment, extract, and combine features of an image with SciPy and scikit-image.

Follow up items (some not yet written):

Blending together obvious glyphs
Converting to vector
Generating the font with font-forge

Mounting a FUSE filesystem in Heroku

2017-02-06T00:00:00+00:00

This evening I'm going to take a different approach to how I would normally blog.

Rather than reporting the results of a technical investigation or highlighting a new/shiny package, I wanted to paint a realistic picture of the technical exploration process.

As it happens, this particular investigation consumed a couple of hours and appears to have drawn an unsuccessful result. Despite this, the learnings are invaluable as they will be directly and immediately applicable to other areas of my work.

The problem

I run a number of services on Heroku. It's an amazing platform for rapid prototyping of (mostly web) applications. The beauty of Heroku is the ease of deployment as well as its availability, scalability and cost.

I have one particular application on Heroku that draws statistics from a moderately sized, slow paced, filestore. Unfortunately, since all storage on Heroku is ephemeral, that means I must fetch the data at least once every 24 hours (when Heroku restarts the container) to recompute my statistics (which are cached for application performance). Fetching the data is a costly operation, and I'd rather avoid it if possible.

Instead of fetching the data on a daily basis, I'd like to have a third-party filestore that can be used from within the Heroku application for its statistics generation. I'd also like to use the store for the cache (which itself can be mem-cached/cached to the ephemeral Heroku disk). In order to invalidate the cache I will want an efficient means of getting a checksum or pertinent fstat info. The detail here isn't important, suffice to say I'd just like some persistence my source data for an otherwise stateless application.

The proposal

Having read a little in the past about libfuse, it seemed that using FUSE to mount a networked resource would be a great match for the problem. Because I have a few GB lying around on Dropbox, and there appeared to be a python based Dropbox-fuse client (ff4d), I decided to investigate the feasibility of mounting my Dropbox directory inside my Heroku application. Other options would have worked just as a well, including an S3 bucket (with s3fs-fuse), or a direct ssh connection (with sshfs).

I had also read a little bit about Heroku's new Docker container registry and runtime environment (https://devcenter.heroku.com/articles/container-registry-and-runtime) and thought this would be a great opportunity to shorten the deployment cycle.

The findings

Mounting a Dropbox directory through ff4d

The first step is to get FUSE up-and-running on my own machine (OSX). libfuse is the first thing in a while that I've not been able to get hold of through conda, so I ended up dusting off my homebrew installation and going through a pretty hefty update. Once complete I tried:

brew install libfuse

With no success. Turns out that osxfuse is a thing, and so I:

brew install osxfuse

Again, no luck, but homebrew does point me in the direction of a cask version (pre-compiled, rather than building it from source on my machine). I'm pretty keen to get going with this, so I go ahead and install the cask version:

brew install Caskroom/cask/osxfuse

After a short wait, the result is positive, and it looks like I have a local FUSE installation.

Next, I want to try it out. Rather than choose to do something simple, I go straight for the jugular and try to get a Dropbox FUSE mount working. I clone https://github.com/realriot/ff4d and get myself set up with a legacy python installation (in a clean environment named "ff4d_py2":

conda create -n ff4d_py2 python=2 pip
source activate ff4d_py2
pip install dropbox

I take a note of the dependency on a legacy version of Python, and vow to submit a merge request making the codebase python (3) compatible should I want to turn this proof-of-concept into something more.

Next, I create a "Dropbox API" -> "Full Dropbox" application on https://www.dropbox.com/developers/apps/create and run:

python getDropboxAccessToken.py

I follow on the on-screen prompts and am eventually rewarded with an OAuth token that I store securely. With my token in hand:

mkdir foobar
python ff4d.py  ./foobar -ap <my_token>

Gives me a directory called foobar containing my Dropbox content, and reminds me that I can delete nearly a GB of images that were shared with colleagues on my Dropbox account. As I delete the files (rm -rf ./foobar/my_image_directory) I'm aware that there are a number of 404 error type messages being logged by ff4d.py - I take a note that there is something that needs deeper investigation here.

So there we have it, a locally mounted Dropbox folder sitting on my OSX machine, thanks to ff4d. Now, I want to create a webapp that can show the contents of my directory (as a proof-of-concept), and to replicate this setup in a docker container, and then ultimately in a Heroku web app deployment.

Creating the webapp

I'm a big fan of tornado, and since the application that could benefit from access to my Dropbox mount is also using tornado I put together a quick web-app to browse the directory.

I'd assumed that creating a http handler that allows directory listing would be built-in to tornado, but it appears not, so I ended up re-using code from https://github.com/imom0/SimpleTornadoServer/blob/master/SimpleTornadoServer.py (BSD-3) to allow me to navigate my directories from within the webapp.

In order for my webapp and ff4d mount to be run on the same process within Heroku (not the only option - it is easy enough to create new processes on Heroku, I just like the ability to control them all from a single process) I need to get the webapp to mount the Dropbox directory itself. Since mounting through ff4d is blocking, I am going to need to run one 2 IOLoops, one on the main thread (for tornado) and the other in a ThreadPoolExecutor managed thread (for ff4d).

Getting another thread to run a blocking script whilst still running a responsive tornado main IOLoop thread is something I have done a few times now. In other situations I have wanted to communicate through Kafka ((example)[http://stackoverflow.com/a/40602866/741316]) in my tornado application, and in another application I wanted the ability to (optionally) spawn a Dask scheduler, workers and client. Truth be told, in most of these situations processes are a better choice, but I digress.

Getting a tornado webapp to run a blocking process in another thread is surprisingly easy. We need a ThreadPool, and an asynchronous function that can run on the main thread:

@tornado.gen.coroutine
def async(executor, function, *args, **kwargs):
    yield executor.submit(function, *args, **kwargs)

thread_pool = ThreadPoolExecutor(1)
tornado.ioloop.IOLoop.current().spawn_callback(async, thread_pool, start_mount, mount_dir=mount_dir)

The function itself is fairly trivial, and simply spawns a sub-process which mounts my Dropbox directory:

def start_mount(mount_dir):
    dropbox_token = os.environ.get('DROPBOX_TOKEN')
    if not os.path.exists(mount_dir):
        os.mkdir(mount_dir)
    subprocess.check_call([sys.executable, 'ff4d.py', mount_dir, '-ap', dropbox_token])

With all of this in place, I'm able to fire up my tornado webapp, and navigate my Dropbox content from within the browser.

The complete code can be found at https://github.com/pelson/heroku-with-dropbox-mount.

Build & deploy with Docker

Groundwork complete, we now want to put some scaffolding around our proof-of-concept so that we can easily test and deploy the application on Heroku.

As I mentioned at the beginning, I recently read-up about the new docker based Heroku deployment option, and want to give it a shot. Since I'm on OSX, I fire up docker-machine (needed an update) and get started with writing my Dockerfile.

I was aware of Continuum's Docker images, and so reached for that as the base image, before extending to my requirements. There is nothing earth-shattering about the Dockerfile I produced (available at https://github.com/pelson/heroku-with-dropbox-mount), and my build -> test workflow looks like:

docker build --tag=docker_webapp_test

and

docker run -p 5000:5000 -e PORT=5000 -e DROPBOX_TOKEN=<my_token> -t -i docker_webapp_test

As mentioned, I'm using docker-machine to run docker - it essentially manages a VirtualBox machine to run a suitable host OS for docker. This means that even though the the port was forwarded in my docker run call, it won't be visible to me on localhost. To find out what IP my machine is running on, we can call docker-machine ip <machine_name>. It is this address (+":5000") that I use to see the webapp in my browser.

Things are starting to come together nicely, except the mount of my device was failing with messages similar to:

Starting FUSE...
fuse: failed to open /dev/fuse: Operation not permitted
[ERROR] Failed to start FUSE... (Traceback (most recent call last):

It turns out that we need elevated privileges to mount a FUSE device within docker. Adding --cap-add SYS_ADMIN and --device /dev/fuse to docker should be enough (though it appears there was once a bug in docker that meant the container needed to be run with full privileges). I make a note of this as a potential problem for our Heroku deployment.

Finally, I'm able to launch my docker image and navigate my Dropbox content through my web app. The final step is to push this image to Heroku:

heroku container:push web

After a considerable amount of time waiting for all of the image layers to upload I get an error in my heroku logs. Simply changing CMD to something that should obviously work (ls -ltr) I get a similar error:

2017-02-07T11:13:48.755363+00:00 heroku[web.1]: State changed from crashed to starting
2017-02-07T11:13:56.607038+00:00 heroku[web.1]: Starting process with command `/bin/sh -c \"ls\ -ltr\"`
2017-02-07T11:13:59.043845+00:00 app[web.1]: Error: No such file or directory

It may be a red-herring, but the command escaping looks a little off. I iterate further with just ls as the CMD, but get the same error. Iterating takes about 30s, so a trial and error approach to solving the problem is proving tedious - I would love to be able to reproduce the issue locally to shorten the loop. I try removing the quotes within the Dockerfile's CMD section and I ensure that PATH is correctly set - still nothing.

I look back at the docs learn about docker-compose - looks like an interesting tool for managing processes in a similar way to the Proc file within Heroku - definitely something to note for future exploration.

With frustration setting in, I roll back to the Dockerfile provided in the documentation. This uses alpine as is base and takes some time to upload, but eventually I'm able to confirm that I can at least run CMD on Heroku with that Dockerfile. Whilst I'd love to understand what is wrong different between Continuum's image and the alpine image (other than the whole OS), my focus on getting my proof-of-concept up and running on Heroku, so I change tactic and update my Dockerfile derive FROM the alpine base image.

After a little more iteration, I eventually manage to use alpine as the base for my webapp's image (including installing ca-certificates to prevent urllib from raising a CERTIFICATE_VERIFY_FAILED exception). I push the image to heroku, and define the DROPBOX_TOKEN environment variable that my code expects in the heroku web portal, but alas there is a problem with the FUSE mount:

2017-02-07T14:36:06.997268+00:00 app[web.1]: Starting FUSE...
2017-02-07T14:36:06.997269+00:00 app[web.1]: [ERROR] Failed to start FUSE... (Traceback (most recent call last):
2017-02-07T14:36:06.997270+00:00 app[web.1]:   File "ff4d/ff4d.py", line 812, in <module>
2017-02-07T14:36:06.997271+00:00 app[web.1]:     FUSE(Dropbox(ar), mountpoint, foreground=args.background, debug=debug_fuse, sync_read=True, allow_other=allow_other, allow_root=allow_root)
2017-02-07T14:36:06.997271+00:00 app[web.1]:   File "/opt/webapp/ff4d/fuse.py", line 405, in __init__
2017-02-07T14:36:06.997272+00:00 app[web.1]:     raise RuntimeError(err)
2017-02-07T14:36:06.997273+00:00 app[web.1]: RuntimeError: 1

In addition, there is a message in the log along the lines of:

fuse: device not found, try 'modprobe fuse' first

It may be as I feared: Heroku doesn't currently support FUSE mounts.

In order to get one final datapoint, I put together the equivalent Dockerfile for Ubuntu rather than alpine. Unfortunately the results are the same.

Building a matrix of conda distributions with conda-build-all

2015-12-09T12:00:00+00:00

Introducing conda-build-all, a tool which extends conda-build to provide powerful build matrix capabilities.

Repositories such as conda-forge/stages-recipes, SciTools/conda-recipes-scitools and ioos/conda-recipes exist to provide a set of conda recipes, and ultimately, channels from which users can access the product of conda-build-ing those recipes. The build phase of all of these repositories looks very similar: a tool (ObviousCI) computes the build matrix, builds those distributions which haven't already been built, and then uploads them to their respective channels. The functionality is tried and tested, and has been powering these repositories for over a year with huge success, however, I recently had need to use this functionality without wanting to upload the built distributions to conda.anaconda.org and found the tool didn't quite fit the bill. Additionally, having originally cobbled together ObviousCI with string and sticky-tape to prove the concept of a continuously integrated repo of recipes, I didn't have huge confidence in its ability to function between python/conda/conda-build upgrades.

As a result, I have re-factored the build part of ObviousCI into a general purpose library which can now be used for the original "conda recipe repository" use-case as much as it can for the ad-hoc "just build this" use-case. Critically, the most significant part of this re-factoring was adding a huge array of unit and integration tests which can be used to ensure expected behaviour is unchanged through future dependency version upgrades.

The new CLI is conda-build-all (BSD license) and is developed at SciTools/conda-build-all.

The build matrix

So what does a build matrix actually look like? Let's jump in at the deep-end and look at a package which has a NumPy C-API dependency. Whilst NumPy's ABI is (intended to be) forward-compatible, in practice it is safer to compile against a specific version and "pin" the distribution to that version. Essentially, that means we need to build our recipe N times, where N is the number of NumPy versions we wish to support. Of course, the same is true for Python itself, leading to a permutation problem of up to NxM builds (N: number of supported NumPy versions; M: number of supported Python versions).

The current conda recipe form for such a package looks like:

package:
    name: my_library
    version: 1.0
requirements:
    build:
        - python
        - numpy x.x
    run:
        - python
        - numpy x.x

Whilst I believe there is room for improvement in the recipe definition, it is still pretty easy to define a complex set of build- and run-time dependencies.

With the existing conda-build tool, should we want to build this for Python 2.7, 3.4 and 3.5, and against NumPy 1.9 and 1.10 (the latest versions of these libraries at the time of writing), things can get a little tedious:

CONDA_PY=27 CONDA_NPY=19 conda build my_library
CONDA_PY=34 CONDA_NPY=19 conda build my_library
CONDA_PY=35 CONDA_NPY=19 conda build my_library
CONDA_PY=27 CONDA_NPY=110 conda build my_library
CONDA_PY=34 CONDA_NPY=110 conda build my_library
CONDA_PY=35 CONDA_NPY=110 conda build my_library

With conda-build-all the special environment variables are taken care of for you (and importantly there is future scope to generalise beyond Python & NumPy) and a build matrix is computed:

$ conda-build-all my_library
Resolving distributions from 1 recipes... 
Computed that there are 7 distributions from the 1 recipes:
Resolved dependencies, will be built in the following order: 
    my_library-1.0-np19py26_0 (will be built: True)
    my_library-1.0-np110py27_0 (will be built: True)
    my_library-1.0-np19py27_0 (will be built: True)
    my_library-1.0-np110py34_0 (will be built: True)
    my_library-1.0-np19py34_0 (will be built: True)
    my_library-1.0-np110py35_0 (will be built: True)
    my_library-1.0-np19py35_0 (will be built: True)

Notice how this command is not conceptually equivalent to the original conda-build calls as I have not asked for particular versions to build against. conda-build-all has chosen the top two major versions and within those, the top two minor versions of the packages which require "pinning". Unfortunately, that included Python 2.6, which I didn't really want - to resolve that, we can add extra conditions to our build:

$ conda-build-all my_library --matrix-conditions "python >=2.7"
Fetching package metadata: ........
Resolving distributions from 1 recipes... 
Computed that there are 6 distributions from the 1 recipes:
Resolved dependencies, will be built in the following order: 
    my_library-1.0-np110py27_0 (will be built: True)
    my_library-1.0-np19py27_0 (will be built: True)
    my_library-1.0-np110py34_0 (will be built: True)
    my_library-1.0-np19py34_0 (will be built: True)
    my_library-1.0-np110py35_0 (will be built: True)
    my_library-1.0-np19py35_0 (will be built: True)

We now have functionally equivalent behaviour that will move forwards as new Python and NumPy versions become available.

Building multiple recipes in a single call

conda-build-all knows what a conda recipe looks like, and will traverse the directories you give it to look for things to build.

Supposing we have a directory of recipes which we wish to build, such as the following:

$ find * -name meta.yaml -exec sh -c "echo RECIPE: {}; cat {}; echo" \;
RECIPE: my_recipes_directory/my_library/meta.yaml
package:
    name: my_library
    version: 1.0
requirements:
    build:
        - python
        - numpy x.x
    run:
        - python
        - numpy x.x

RECIPE: my_recipes_directory/my_other_library/meta.yaml
package:
    name: my_other_library
    version: 1.0
requirements:
    build:
        - python
        - numpy x.x
    run:
        - python
        - numpy x.x

We can simply call conda-build-all on the directory of recipes to have them built appropriately:

$ conda-build-all my_recipes_directory --matrix-conditions "python 2.7.*|3.5.*"
Fetching package metadata: ........
Resolving distributions from 2 recipes... 
Computed that there are 8 distributions from the 2 recipes:
Resolved dependencies, will be built in the following order: 
    my_library-1.0-np110py27_0 (will be built: True)
    my_library-1.0-np19py27_0 (will be built: True)
    my_library-1.0-np110py35_0 (will be built: True)
    my_library-1.0-np19py35_0 (will be built: True)
    my_other_library-1.0-np110py27_0 (will be built: True)
    my_other_library-1.0-np19py27_0 (will be built: True)
    my_other_library-1.0-np110py35_0 (will be built: True)
    my_other_library-1.0-np19py35_0 (will be built: True)

This functionality becomes invaluable when we wish to build many packages, such is the case for the conda-recipes repositories mentioned earlier.

Only building the missing distributions

The build matrix is supremely useful, but it does come at the cost of the extra time needed to build the many distributions. With repositories full of recipes, it is easy to come to hundreds of build matrix items. If we want to be able to run conda-build-all on a regular basis, we can't reasonably expect to build each of those items each time. Therefore, conda-build-all has the ability to inspect various locations to determine if a distribution has already been built. In fact, the default behaviour is to inspect the local conda-build directory to determine if a distribution has already been built locally. Other options include the ability to inspect conda channels as well as arbitrary local directories. Supposing we wanted the pelson/channel/testing channel to have all of the built distributions from my_recipes_directory, we can use conda-build-all to good effect:

conda-build-all my_recipes_directory/ --matrix-conditions "python 2.7.*|3.5.*" \
    --inspect-channels "pelson/channel/testing" \
    --upload-channels "pelson/channel/testing" \
    --no-inspect-conda-bld-directory

Summary

conda-build-all is a tool which builds on top of conda-build to give powerful build-matrix options when building conda distributions. It has come from ObviousCI, whose primary objective was to simplify the build and upload of many recipes in a Continuous Integration environment. In migrating the code base from ObviousCI several new test strategies have been developed - making conda-build-all easier to maintain, and giving rise to the possibility of improving the conda and conda-build test suites themselves.

Vim search and replace across many files

2015-12-03T12:00:00+00:00

A powerful combination of commands for search and replace across multiple files with Vim.

http://vim.wikia.com/wiki/Opening_multiple_files_from_a_single_command-line http://vim.wikia.com/wiki/Search_and_replace_in_multiple_buffers

:arg **/*.cpp

:argdo %s/pattern/replace/ge | update

Running scripts in temporary conda environments with conda execute

2015-10-03T12:00:00+01:00

Update, August 2026

I no longer maintain conda execute, and haven't for some time. If you're looking for the same idea today, running a script in a throwaway environment described by inline metadata, the well-supported options are pixi exec (the closest conda-native successor, from the prefix-dev team) and, for pure-Python scripts, uv run or pipx run using PEP 723 inline script metadata. The rest of the article is preserved unchanged for historical interest.

Conda is awesome - it is a simple package manager which allows me to create isolated software environments much like virtualenv. Unlike virtualenv though it can handle any package type, not just python ones.

The more I use it, the more I want to make use of conda's dependency tracking for my own simple scripts to ensure they were being executed in a suitable environment with the expected dependencies already installed.

conda build is an excellent tool for building your own distributions and sharing them on anaconda.org, but creating a distribution is tiresome if all you have is a single script, rather than a fully-fledged software package. That is where conda execute comes in.

conda execute allows you to run a script of any kind in a temporary environment defined by metadata in the script itself.

For example, take the following Python script:

#!/usr/bin/env python

import numpy as np

print(np.random.poisson(size=5))

By adding appropriate conda execute metadata to our script, we can describe the kind of environment we would need to be able to run this code:

$ cat my_script.py

#!/usr/bin/env python

# conda execute
# env:
#  - python >=3
#  - numpy

import numpy as np

print(np.random.poisson(size=5))

conda execute can now be used to run this script:

$ conda execute -v my_script.py

Using specification: 
env: [python >=3, numpy]
run_with: [/usr/bin/env, python]

Prefix: /Users/pelson/miniconda/tmp_envs/ea977067a8fbeb21a594

[1 2 0 1 0]

As you can see, the special comment in the script has been read, and an appropriate temporary environment has been created.

Temporary environments

In order to provision suitable environments for the executed scripts, conda-execute implements a temporary environment concept. Rather than adding a new environment in your conda environments directory (and thus filling up the available environments listed in conda env list), a new "tmp_envs" environments directory has been created, within which conda-execute's temporary environments are created (this location is configurable with the conda-execute/env-dir conda config item). As you may have noticed in the previous example, where the environment created was named ea977067a8fbeb21a594, these temporary environments are named by a hashing algorithm (SHA 256, truncated to 20 characters). The hash is taken from the conda-execute metadata of your script, which means that you can re-run a script many times and only need one environment to be created. Additionally, it has the advantage that multiple scripts can share the same environment if their conda-execute metadata is the same.

Each time a temporary environment is run with conda-execute a log entry is added, allowing it to keep track of which environments are still in use. Once an environment has been unused for 25 hours any subsequent conda-execute call will trigger it to be garbage collected, thus preventing your disk filling up with unneeded temporary environments.

Configurability

conda-execute builds on top of conda's configuration to allow some customisation in behaviour. The following condarc shows the configuration options that are available:

conda-execute:
    # The directory to use to hold the temporary environments.
    env-dir: "{config.envs_dirs[0]}/../tmp_envs"

    # The number of hours that an environment should be unused for to be
    # considered for garbage collection.
    remove-if-unused-for: 25

Reproducibility of scripts with `conda-execute`

There are a few really interesting use-cases which I'm keen to explore with conda-execute.

I'm already making use of conda-execute as a form of Makefile for this blog. My make.py is simply a command line wrapper to the appropriate pelican sub-command:

$> ./make.py --help
usage: Help [-h] {html,publish,reload} ...

positional arguments:
  {html,publish,reload}
    html                Make the html
    publish             Make publishable html, and put it in the
                        output_branch.
    reload              Make the html, and watch the folder for any changes.

optional arguments:
  -h, --help            show this help message and exit

The concept of creating reproducible scripts goes far wider than trivial Makefiles though - with conda-execute, because the metadata in the script is the definition of the execution environment, important information about its dependencies and how it is run are all embedded into the script itself.

I'm particularly keen to explore the reproducibility angle that conda-execute brings, particularly for scientific applications.

conda-execute can be found at github.com/pelson/conda-execute, and installed with conda install conda-execute --channel conda-forge.

Interactive matplotlib figures in the IPython notebook - they've landed!

2014-06-03T12:00:00+01:00

After what feels like years chipping away at the problem, not least from some awesome developers including the whole of the IPython development team, Michael Droetboom and Jason Grout, I recently closed the development loop and added a new matplotlib backend providing interactive figures in the IPython notebook environment.

The "nbagg" backend makes use of the familiar anti-grain geometry rendering engine, so your figures will render as they always have with other agg based backends, with the key difference that instead of an interactive GUI upon show, an inline interactive figure is displayed, complete with the "pan/zoom" and "zoom to rectangle" buttons, as can be seen in the following screenshot:

The backend itself is still very much a work in progress, but it is functional, and well worth trying out for yourself. Turning the backend on with v1.4+ of matplotlib (not yet released at time of writing) is as simple as:

import matplotlib 

matplotlib.use('nbagg')

Now that we've changed the backend, we can go ahead and import pyplot and produce a figure as normal:

import matplotlib.pyplot as plt
import numpy as np

plt.plot(np.sin(np.linspace(0, 2*np.pi)))
plt.show()

To avoid re-writing the matplotlib rendering engine in a browser compatible language (i.e. JavaScript), the nbagg backend requires a live IPython session for interactive figures. Notice however that in this static rendering of the notebook, we have a static image of the figure, akin to the existing IPython "inline" backend. As with the "inline" backend, it is only the snapshot of the figure which is preserved, rather than the figure instance itself (the ability to pickle a figure does make that a possibility though) - essentially, what this means is that we have an opportunity to interact with out data before the figure is cast in stone as an image.

Features¶

The "nbagg" backend has some pretty unique features which have the potential to be built upon in the future.

One interesting feature of a nbagg figure is that it can be displayed multiple times, with all views on the figure being updated simultaneously (even accross different browsers), so there is a real potential for some collaborative figure investigation when the concept of shared users becomes a reality in the notebook.

I'm keen to make this backend easy to extend such that its easy to add things such as an "export to mpld3" button, some snapshotting functionality so that notebook viewers can see multiple views of the same figure in a slideshow and I'm sure there is a whole host of other cool functionality that could be added.

Let me know what you think? Got any ideas on how to make this backend even better? Get in touch in the comments section.

Dealing with arrays which are bigger than memory - an introduction to biggus

2013-09-25T12:00:00+01:00

Update, August 2026

Biggus is no longer developed. For lazy, chunked, out-of-core array computation in Python today, use Dask. Biggus was one of several early efforts in this space and had some influence on Dask's design. The rest of this article is preserved unchanged for historical interest.

I often deal with huge gridded datasets which either stretch or indeed are beyond the limits of my computer's memory. In the past I've implemented a couple of workarounds to help me handle this data to extract meaningful analyses from them. One of the most intuitive ways of reducing gridded datasets is through indexing/slicing and in this regard netcdf4-python's excellent ability to slice subsets of a larger NetCDF file is invaluable. The problem with the netcdf4-python implementation is that this capability is only available if you have NetCDF files, and doing any analysis on the data involves loading all of the data into memory in the form of a numpy array.

That is where biggus steps in.

import biggus

In this article I will cover some of the basic biggus classes and demonstrate its use in visualising part of an array which is an order of magnitude larger than my machine's memory.

A very convenient and extremely large dataset is the map tile images made available via the folks over at mapquest.com. As some very rudimentary background, the map tiles are split into distinct zoom levels with each zoom level z having 2**z by 2**z images of 256x256 pixels each. So at zoom level 0, there is 1 image to represent the whole globe, with a resolution of 256x256 pixels. At zoom level 1 that turns into 2x2 images, with a combined resolution of 512x512. In the case of these images there is also an RGB channel, which means at zoom level 9 we have a combined resolution image of shape (131072, 131072, 3).

Ok, lets start by implementing a class which represents a given x, y, z map tile. We will also add a __getitem__ method which actually retrieves the data from the web server.

import io
import urllib

import numpy as np
import PIL.Image


class TileLoader(object):
    URL_TEMPLATE = 'http://otile1.mqcdn.com/tiles/1.0.0/sat/{z}/{x}/{y}.jpg'
    
    def __init__(self, x, y, z):
        self.xyz = x, y, z
        #: A cached version of the array that is downloaded.
        self._cached_array = None
        
        # The required attributes of a "concrete" array, according
        # to biggus, are 'ndim', 'dtype', and 'shape'.
        self.ndim = 3
        self.dtype = np.uint8
        self.shape = (256, 256, 3)
    
    def __getitem__(self, slices): 
        # Implement the getitem/slicing interface which is the interface needed for biggus to create numpy arrays.
        if self._cached_array is None:
            print 'Downloading tile x={} y={} z={}'.format(*self.xyz)
            url = self.URL_TEMPLATE.format(x=self.xyz[0], y=self.xyz[1], z=self.xyz[2])
            tile_fh = urllib.urlopen(url)
            image_file = io.BytesIO(tile_fh.read())
            self._cached_array = np.array(PIL.Image.open(image_file))
        return self._cached_array[slices]

A TileLoader instance can now be created by passing just the x, y, z coordinates of the tile which is of interest. Notice how the data isn't actually loaded until we ask for it with the index notation.

defered_img = TileLoader(x=1, y=2, z=3)

print defered_img[...].shape
print defered_img[::5, 10:35, 0].shape

Downloading tile x=1 y=2 z=3
(256, 256, 3)

(52, 25)

So far this hasn't required any Biggus specific code, but using this new TileLoder class we can go ahead and create a biggus.ArrayAdapter instance.

import biggus
img = biggus.ArrayAdapter(TileLoader(x=15, y=10, z=5))

print img
sub_img = img[::5, 10:35, 0]
print sub_img

<biggus.Array type (ArrayAdapter) shape=(256, 256, 3) dtype=<type 'numpy.uint8'>>
<biggus.Array type (ArrayAdapter) shape=(52, 25) dtype=<type 'numpy.uint8'>>

Notice how no "Downloading tile..." message has been printed, yet the correct shape is calculated for the indexed sub-image. This is because biggus helps us to defer the actual data loading step until the data is required and the Array.ndarray() method is called.

sub_img_array = sub_img.ndarray()
print type(sub_img_array), sub_img_array.shape

Downloading tile x=15 y=10 z=5
<type 'numpy.ndarray'>

 (52, 25)

So we can now arbitrarily index any sized array and the data is not realized until we actually request the data. This is generally the paradigm in Biggus and it allows one to build up complex operations on arrays without needing access to all of the data at once.

Whilst in this example we've still had to load the full 256x256 image before indexing it (see the __getitem__ implementation above), we could implement some clever image loader which picks off the appropriate data from disk. In truth an image of 256x256 is so small that we probably don't need to worry about implementing such a tricky low level data reader in this case, but if you are using netcdf4-python with massive variables then being able to read only the sections of data which are requested becomes invaluable.

Now that we can create biggus.Array instances (biggus.ArrayAdapter is a subclass of biggus.Array) we can use some of its more interesting features. One of the really interesting classes in biggus is LinearMosaic (which again is a biggus.Array subclass) - this class allows one to join together a list of homogeneous biggus.Array instances to form a bigger array.

Lets create a biggus.LinearMosaic to represent a column of map tiles at zoom level 5, just to the west of the Greenwich meridian (x=15 at z=5):

arrays = [biggus.ArrayAdapter(TileLoader(x=15, y=y, z=5)) for y in xrange(2 ** 5)]
col = biggus.LinearMosaic(arrays, axis=0)

print col.shape

(8192, 256, 3)

We can go ahead and slice a subset of the big column of images and visualise the resulting ndarray. Notice how only the necessary images from the 32 image mosaic are actually downloaded.

import matplotlib.pyplot as plt

smaller_column = col[2000:3500]
plt.figure(figsize=(3, 10))
plt.imshow(smaller_column.ndarray())

Downloading tile x=15 y=7 z=5
Downloading tile x=15 y=8 z=5

Downloading tile x=15 y=9 z=5

Downloading tile x=15 y=10 z=5

Downloading tile x=15 y=11 z=5

Downloading tile x=15 y=12 z=5

Downloading tile x=15 y=13 z=5

<matplotlib.image.AxesImage at 0x10c650810>

The biggus.LinearMosaic will take any biggus.Array instances in its mosaic, including other biggus.LinearMosaic instances.

Lets go ahead an make a horizontal mosaic of column mosaics to represent the whole z=9 dataset (that's 2^18 tiles at a resolution of 256x256 each).

def tile_array(z):
    columns = []
    for x in xrange(2 ** z):
        column = [biggus.ArrayAdapter(TileLoader(x=x, y=y, z=z)) for y in xrange(2 ** z)]
        columns.append(biggus.LinearMosaic(column, axis=0))
    return biggus.LinearMosaic(columns, axis=1)

z9 = tile_array(z=9)

print z9.shape

(131072, 131072, 3)

Just in case you were wondering whether you could actually create a numpy array of this size in memory, the answer is only if you are on a machine with at least 48GB of contiguous memory:

def sizeof_fmt(num):
    for x in ['bytes','KB','MB','GB']:
        if num < 1024.0:
            return "%3.1f%s" % (num, x)
        num /= 1024.0
    return "%3.1f%s" % (num, 'TB')

def sizeof_array(a):
    return sizeof_fmt(a.dtype().itemsize * np.prod(a.shape))

print sizeof_array(z9)

48.0GB

Again, we can easily visualise a subset of this large array by using the numpy indexing notation.

smaller_img = z9[54600:55000, 59300:59850]
plt.imshow(smaller_img.ndarray())

Downloading tile x=231 y=213 z=9
Downloading tile x=231 y=214 z=9

Downloading tile x=232 y=213 z=9

Downloading tile x=232 y=214 z=9

Downloading tile x=233 y=213 z=9

Downloading tile x=233 y=214 z=9

<matplotlib.image.AxesImage at 0x1107d1990>

Note, this is not a tutorial on spatial extraction of data - there are better ways to extract specific regions of big images than using the pixel index, but that is beyond the scope of this notebook.

So far, we've had to index the biggus.Array in order to pass the data through to matplotlib for visualisation, but wouldn't it be better if we made matplotlib index the array as appropriate for the viewing pane? That way we could pan and zoom on this huge array whilst under the hood, just the parts of the data that need loading actually are.

Fortunately, with a bit of jiggling of the underling matplotlib code this is relatively easy to achieve. The following code block will modify some of the matplotlib internals so that we can go ahead and add the z9 image array as an image to matplotlib and ultimately pan and zoom on that data. Feel free to skip the details of the next code block:

import matplotlib.image as mimg
import matplotlib._image as _image


class AxesBiggusImage(mimg.AxesImage):
    """Subclasses AxesImage to support biggus.Array instances."""
    def set_data(self, A):
        self._A = A

        if (self._A.dtype != np.uint8 and
            not np.can_cast(self._A.dtype, np.float)):
            raise TypeError("Image data can not convert to float")
  
        if (self._A.ndim not in (2, 3) or
            (self._A.ndim == 3 and self._A.shape[-1] not in (3, 4))):
            raise TypeError("Invalid dimensions for image data")
  
        self._imcache = None
        self._rgbacache = None
        self._oldxslice = None
        self._oldyslice = None


def frombyte(image, *args, **kwargs):
    """Override frombyte to handle biggus.Array instances."""
    if isinstance(image, biggus.Array):
        image = image.ndarray()
    return _image.frombyte_orig(image, *args, **kwargs)

_image.frombyte_orig = _image.frombyte
_image.frombyte = frombyte

Now for the magic. If you are running this notebook with an interactive matplotlib backend (which I encourage you to try) the following code will produce a standard matplotlib figure with an image of our zoom level 9 data (centered at Tenerife in the Canary Islands) which you can interactively pan and zoom in the same way as you can for any other matplotlib figure - the key difference is that the image that you are viewing is far larger than most machines could possibly hold in memory and therefore only the tiles which are currently being displayed are actually being downloaded from the server. [Note: This has been tested with v1.3.0 of matplotlib]

import matplotlib.pyplot as plt

    
ax = plt.axes(aspect=1)
im = AxesBiggusImage(ax, origin='upper')
im.set_data(z9)
ax.images.append(im)
 
plt.xlim(59300, 59850)
plt.ylim(55000, 54600)
 
plt.show()

In summary, I've shown Biggus' easy to implement biggus.ArrayAdapter interface which gives us a familiar numpy-like interface to any data fit for a numpy array, with the added benefit that combined with off the shelf tools such as the python4-netcdf Variable objects, can quickly make deferred data analysis workflows a reality. The biggus.LinearMosaic allows you to easily build up concatenations of biggus.Arrays (including other biggus.LinearMosaic) instances to form arrays far bigger than could be held in RAM, and allows one to index the data arrays to get hold of smaller, more manageable subsets of the super array. Finally I demonstrated the prospect that matplotlib could one day handle biggus.Array instances without ever needing to read in data that it doesn't ultimately visualise.

Whilst biggus is still very much in its infancy (there is no docs and just a handfull of tests at this point), there is even more to biggus that I haven't discussed here - one very exciting prospect is its ability to do whole dataset operations (such as mean, stddev etc.) without ever needing to allocate all of the memory that the larger array represents - with the added prospect that these operations will become multi-threaded and potentially quicker than the standard numpy operations. In my next post on http://pelson.github.io I will try to cover some of these operations on massive datasets and compare the performance with other tools currently available.

Do you deal with arrays of data that are bigger than your machine can deal with? How do you get around the obstacle of running out of memory? Let me know what you think in the comments section.

Working with colours in matplotlib

2013-06-03T12:00:00+01:00

When dealing with colours in scientific visualisations some people like to have a colourmap which can be indexed into to pick specific colours. Whilst this isn't necessarily the best way of handling colours in matplotlib, it certainly adds a degree of familiarity to users who have come over from other visualisation tools, such as IDL.

In this article I'll cover one approach to using the colour-by-index paradigm in matplotlib.

I've created a text file containing byte values for each of the RGB channels. I actually already had this as a numpy array, so wrote it out with:

np.savetxt('custom_colors.txt', colors_array, fmt='%03i', delimiter=', ')

with open('custom_colors.txt') as fh:
    print ''.join(fh.readlines())

# R,   G,   B
255, 255, 255
000, 000, 000
250, 060, 060
000, 220, 000
030, 060, 255
000, 200, 200
240, 000, 130
230, 220, 050
240, 130, 040
160, 000, 200
160, 230, 050
000, 160, 255
230, 175, 045
# This is a comment...
000, 173, 208
135, 136, 000
185, 219, 014
215, 169, 000
119, 074, 057
156, 162, 153
237, 041, 057
255, 255, 255

import numpy as np

colors = np.loadtxt('custom_colors.txt', delimiter=', ')

# convert to RGB values in the [0, 1] interval
colors = colors / 255.

print colors.shape

(21, 3)

import matplotlib.colors as mcolors

my_cmap = mcolors.ListedColormap(colors)

plt.figure(figsize=(20, 0.5))
plt.title('My own color map*')
plt.pcolormesh(np.arange(my_cmap.N).reshape(1, -1), cmap=my_cmap)
plt.gca().yaxis.set_visible(False)
plt.gca().set_xlim(0, my_cmap.N)
plt.show()

* Actually, I borrowed this colormap from Bernd Becker (https://groups.google.com/forum/?fromgroups#!topic/scitools-iris/hsb6ExNgSss).

Now that I have a cmap instance I can access specific colors by their index, using the call notation on the colormap:

plt.fill([0, 0, 0.5, 0.5], [0, 1, 1, 0], color=my_cmap(2))
plt.fill([0.5, 0.5, 1, 1], [0, 1, 1, 0], color=my_cmap(3))
plt.show()

It is possible to get all of the RGB values themselves fairly easily:

my_cmap(np.arange(my_cmap.N))

array([[ 1.        ,  1.        ,  1.        ,  1.        ],
       [ 0.        ,  0.        ,  0.        ,  1.        ],
       [ 0.98039216,  0.23529412,  0.23529412,  1.        ],
       [ 0.        ,  0.8627451 ,  0.        ,  1.        ],
       [ 0.11764706,  0.23529412,  1.        ,  1.        ],
       [ 0.        ,  0.78431373,  0.78431373,  1.        ],
       [ 0.94117647,  0.        ,  0.50980392,  1.        ],
       [ 0.90196078,  0.8627451 ,  0.19607843,  1.        ],
       [ 0.94117647,  0.50980392,  0.15686275,  1.        ],
       [ 0.62745098,  0.        ,  0.78431373,  1.        ],
       [ 0.62745098,  0.90196078,  0.19607843,  1.        ],
       [ 0.        ,  0.62745098,  1.        ,  1.        ],
       [ 0.90196078,  0.68627451,  0.17647059,  1.        ],
       [ 0.        ,  0.67843137,  0.81568627,  1.        ],
       [ 0.52941176,  0.53333333,  0.        ,  1.        ],
       [ 0.7254902 ,  0.85882353,  0.05490196,  1.        ],
       [ 0.84313725,  0.6627451 ,  0.        ,  1.        ],
       [ 0.46666667,  0.29019608,  0.22352941,  1.        ],
       [ 0.61176471,  0.63529412,  0.6       ,  1.        ],
       [ 0.92941176,  0.16078431,  0.22352941,  1.        ],
       [ 1.        ,  1.        ,  1.        ,  1.        ]])

To visualize specific colors using pcolormesh, we need to avoid using a normalization of the data by using matplotlib.colors.NoNorm() and need a 2d array of colors

colors = [4, 11, 0, 2, 17]
plt.pcolormesh(np.array(colors, ndmin=2), norm=mcolors.NoNorm(), cmap=my_cmap)
plt.show()

Using a recent change applied to matplotlib (available in matplotlib v1.3 after https://github.com/matplotlib/matplotlib/pull/2002) it is possible to pick specific colours to go with levels when contouring with an extend argument:

import cartopy.crs as ccrs
import iris
import iris.quickplot as qplt


fname = iris.sample_data_path('air_temp.pp')
temperature = iris.load_cube(fname)

colors = [4, 11, 0, 2, 17]
levels = [263, 273, 288, 298]

plt.figure(figsize=(15, 10))
ax = plt.axes(projection=ccrs.PlateCarree(central_longitude=-180))

cs = qplt.contourf(temperature, colors=my_cmap(colors), levels=levels, extend='both')

# Get hold of the colorbar instance and update its extendfrac length
cbar = cs.colorbar[0]
cbar.extendfrac = 0.15
cs.changed()

ax.coastlines()
plt.show()

This article certainly shows a way of handling the colour-by-index paradigm, though it must be said that handling colours like this in matplotlib is not necessarily the best approach - I'll leave that to a future article.

Find this useful? How do you handle colours in your matplotlib figures? Is there a killer feature you think matplotlib is missing out on? Let me know via the comments section.

Drawing a pseudo-colour blockplot (pcolormesh) in matplotlib with levels and specific colours

2013-05-03T12:00:00+01:00

I recently added a new function to matplotlib to make it easier to draw pseudo-colour plots given specific levels and colours, in exactly the same way as you can with contour and contourf.

import numpy as np
import matplotlib.pyplot as plt

from matplotlib.colors import from_levels_and_colors

cmap, norm = from_levels_and_colors([2, 5, 7, 10], ['red', 'blue', 'green'])
data = np.arange(12).reshape(3, 4)

plt.pcolormesh(data, cmap=cmap, norm=norm)
plt.colorbar()
plt.show()

It's also possible to easily add "extend" like capabilities, similar to that of contourf

import numpy as np
import matplotlib.pyplot as plt

from matplotlib.colors import from_levels_and_colors

cmap, norm = from_levels_and_colors([2, 5, 7], ['red', 'blue', 'green'], extend='max')
data = np.arange(12).reshape(3, 4)

plt.pcolormesh(data, cmap=cmap, norm=norm)
plt.colorbar()
plt.show()

If you already have a colormap that you want to use as the source of your colours it is possible to call the colormap with the index of the colour.

I'm going to be using Iris' nice descrete Brewer colour palettes

import iris.palette
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.colors import NoNorm

nice_cmap = plt.get_cmap('brewer_RdYlBu_11')

plt.figure(figsize=(20, 0.5))
plt.title('The "brewer_RdYlBu_11" cmap')
plt.pcolormesh(np.arange(nice_cmap.N).reshape(1, -1), norm=NoNorm(), cmap=nice_cmap)
ax = plt.gca()
ax.yaxis.set_visible(False)
ax.autoscale_view(tight=True)
plt.show()

Lets go ahead and start using the new from_levels_and_colors function.

import iris
import iris.palette
import iris.quickplot as qplt
from matplotlib.colors import from_levels_and_colors
import matplotlib.pyplot as plt
import numpy as np

fname = iris.sample_data_path('air_temp.pp')
temperature = iris.load_cube(fname)

nice_cmap = plt.get_cmap('brewer_RdYlBu_11')
# Pick some of the nicer colors from the palette...
colors = nice_cmap([9, 8, 6, 3, 2, 0])
levels = [250, 265, 275, 285, 300]

cmap, norm = from_levels_and_colors(levels, colors, extend='both')

plt.figure(figsize=(16, 10))
qplt.pcolormesh(temperature, cmap=cmap, norm=norm)
plt.gca().coastlines()
plt.show()

For the record, the PR also adds a new method on a colormap instance to quickly see what it looks like (EDIT: Sadly, this was removed from the pull request):

nice_cmap.peek()

Phil Elson - Software | Science | Python

Introducing multistage-build: a proxy build-backend with customisable pre- and post-build hooks

multistage-build: a PEP-517 build backend which proxies your real build backend

Respecting the PEP-517 contract

Hooks as plugins

Next steps

Updating this blog for 2026

The pre-2026 scope of this blog

What are my goals for refreshing my blog?

Where I am now

Outside of work

AI coding assistants

What's next

cf-units get (La)TeX unit repr... and a UDUNITS-2 parser to boot

tl;dr

Motivation

UDUNITS-2 parsing

ANTLR

Parse tree into an abstract representation

Abstract graph to (La)TeX

Opportunities of a working UDUNITS-2 grammar

CF conventions

Online unit validation

Alternative simplification and factoring

Python 3 only

Conclusion

Investigating containment testing on LFRic's cubedsphere

UGRID and LFRic: Polygon mesh intersections¶

Conclusion¶

Analysing walks on the South West Coast Path

Working with the coast path geometry itself¶

Calculating the distance of the coast path¶

Getting hold of my recorded tracks¶

Conclusion¶

Recursive search and replace

Creating an application for a website on OSX

Using nmap to find all devices on network listening on a particular port

Creating a continuous integration service to check CLAs on GitHub

Calling (asynchronous) coroutines from a (synchronous) CLI tool

Using the GitHub API for labels and statuses

Authentication of GitHub API

Writing a webhook

Preventing man-in-the-middle webhooks

Tornado async user agent

Round-up

Converting rasters to SVG, and creating a rudimentary font with font-forge - Part 4 of an XKCD font saga

Raster to Vector conversion (PPM to SVG)¶

Raster version (PPM, original pixel resolution)¶

Vector version converted from PPM to SVG with potrace¶

Creating a font from SVGs using fontforge¶

First attempt¶

Scaling the glyphs¶

The finished article¶

What's missing?¶

Wrap-up¶

Classifying segmented strokes as characters - Part 3 of an XKCD font saga

Conclusion

Segment, extract, and combine features of an image with SciPy and scikit-image - Part 2 of an XKCD font saga

Conclusion

Playing with Randall Munroe's XKCD handwriting

Conclusion

Mounting a FUSE filesystem in Heroku

The problem

The proposal

The findings

Mounting a Dropbox directory through ff4d

Creating the webapp

Build & deploy with Docker

Building a matrix of conda distributions with conda-build-all

The build matrix

Building multiple recipes in a single call

Only building the missing distributions

Summary

Vim search and replace across many files

Running scripts in temporary conda environments with conda execute

Temporary environments

Configurability

Reproducibility of scripts with conda-execute

Interactive matplotlib figures in the IPython notebook - they've landed!

Features¶

Reproducibility of scripts with `conda-execute`