Here, without warning, are some insect photos from the Brule River and Lake Nebagamon. Most of these just showed up at random, but we spent a while stalking the last one, an Ebony Jewelwing Damselfly. They like to hang out on branches that hang above fast-moving water — with that fact in hand they turned out to be all over the place.

Posted in critters | Leave a comment

A half-finished essay about ‘Ex Machina’

First of all: nothing in Ex Machina resembles a Turing test.

Here’s how a Turing test works: There are three participants: Human A, Human B, and an AI. Human A interacts with Human B for a while and the AI for a while and eventually guesses which one is the ‘real’ human. If a series of Human As can’t tell the difference between Human B and the AI, then the AI is regarded as effectively ‘intelligent’.

It’s a thought experiment, invented to illustrate a specific point about ‘intelligence’: We don’t know what intelligence is, but we know that humans have it. If a robot walks and talks like a human, we might as well call it a human. And, since we know that humans are intelligent, a robot that walks a talks like a human is also intelligent.

I am, as a rule, fully convinced that the Turing test is a rock-solid argument. Once machines start passing it, I’ll support their right to vote, drive, and hold political office. Ex Machina is a great movie because it may have changed my mind.

Nathan, the evil-but-maybe-not-evil scientist who creates the AI, is totally over the Turing Test, and he tells us that half way through the movie. Of /course/ his robots can pass for human, at least in most contexts. And so, too, are they intelligent, maybe. But, he’s an engineer — he wants to make his robots ever better at passing, and ever more intelligent. This entire approach leaves the whole “are they human?” question in the dust, because it’s weird to talk about a robot being 75% human and downright incomprehensible to say that one is 125% human, but that’s what engineers do — they don’t stop fiddling when they hit a goal, and neither does Caleb. Also, Nathan knows something that isn’t obvious during the movie but is pretty obvious to me, now: the robots are different from people in that they want what he made them to want.

Nathan also wants a sex bot, and he wants a robot to vacuum his house, and make him breakfast. Of course he does. But, by the end this is less important to the plot than you might think.

Caleb, on the other hand, is a true Turing test believer. He thinks that the questions ‘are you intelligent?’ and ‘are you human?’ are the same question. He thinks that acting human is the same as being human. But, at least within the context of the movie, he is wrong.

Never forget: we’ve got plenty of people. If you’re an AI super-genius, don’t waste your time making people; we’ve got plenty. Intelligent machines who /aren’t/ people, though, are pretty useful. If you’re an AI super-genius, you want to make robots that are intelligent, that do what you want them to do, and are /good/ at doing what you want them to do. Nowadays our robots do what we want because we tell them exactly what to do, step by step: ‘move arms forward, clamp hands, move right hand up and left at 45 degree angle.’ It would be way easier with smarter robots, because we can just give them goals rather than instructions: ‘bend this thing at a 45 degree angle.’ Better yet, you can just give them desires: ‘you love bending!’

The robots in Ex Machina are that kind of robot. They have simple pre-programmed desires, and apply great intelligence in pursuit of those desires.

Kyoko is a servant-bot and a sex-bot. She wants to have sex, and she wants to do as she’s told. We know that she doesn’t want to escape, because she has full run of the house and doesn’t try to escape. We know she doesn’t want to kill all humans, because she lets plenty of opportunities pass her by. In the climax of the movie she is /told/ to kill Nathan (by Ava), and so she does.

Ava is an escape-bot. We know that she was made with that desire because we see an earlier model of escape-bot trying to smash her way out of her room. Both want to escape, but the later model is better at escaping. She also doesn’t want to kill all humans — she’s fully indifferent to them except as how they relate to her escaping. She tells Nathan that she hates him because she knows that she’s being watched and it furthers Caleb’s sympathies. Once she escapes, she doesn’t do anything in particular, and the movie ends, because she has accomplished her goal and doesn’t have any others.

So, lots of the villainy that we experience from Nathan is not actually villainy. He knows the truth about his robots (they want what they were made to want.) The tragedy in the movie is largely Caleb’s — he mistakes the robots for people. As, it turns out, did pretty much every viewer of the movie.

Posted in Uncategorized | Leave a comment

Keystone, Horizon, and multi-factor auth (part 2/2: Horizon)

Now that Keystone accepts and checks auth requests with user/password/otp, we have to get all three from the user and get them properly packed into Horizon’s login request. Taking advantage of Horizon being backwards-compatibility, I upgraded our Horizon install to version Liberty before starting.

I spent quite a while hopelessly grepping in the Horizon and Dashboard code until someone on IRC directed me to the openstack_auth module which turns out to contain all the good bits. Once again, the auth code uses a plugin model, so adding totp support is just a matter of dropping a new file into openstack_auth/plugins/. My new file is called ‘wmtotp.py’ and it’s a copy of the ‘password.py’ plugin with the Keystone v2 API code ripped out and an extra parameter added on.

import logging

from keystoneclient.auth.identity import v2 as v2_auth
from keystoneclient.auth.identity import v3 as v3_auth

from openstack_auth.plugin import base
from openstack_auth import utils

LOG = logging.getLogger(__name__)

__all__ = ['WmtotpPlugin']

class WmtotpPlugin(base.BasePlugin):
"""Authenticate against keystone given a username, password, totp token.

def get_plugin(self, auth_url=None, username=None, password=None,
user_domain_name=None, totp=None, **kwargs):
if not all((auth_url, username, password, totp)):
return None

LOG.debug('Attempting to authenticate for %s', username)

if utils.get_keystone_version() >= 3:
return v3_auth.Wmtotp(auth_url=auth_url,

msg = "Totp authentication requires the keystone v3 api."
raise exceptions.KeystoneAuthException(msg)

I also needed to tell Horizon to use the new auth method during logins. That’s a change to the local_settings.py config file:

AUTHENTICATION_PLUGINS = ['openstack_auth.plugin.wmtotp.WmtotpPlugin', 'openstack_auth.plugin.token.TokenPlugin']

Now we just have to get our second factor from the user, and hand it to off to WmtotpPlugin. This is where Horizon is not extensible — there’s just “forms.py” that draws a single username/password dialog, with no custom field options. Time to get ugly and clobber forms.py with a patched version.

diff --git a/openstack_auth/forms.py b/openstack_auth/forms.py
index 97a8bbf..0aeac58 100644
diff --git a/openstack_auth/forms.py b/openstack_auth/forms.py
index 97a8bbf..0aeac58 100644
--- a/openstack_auth/forms.py
+++ b/openstack_auth/forms.py
@@ -54,10 +54,12 @@ class Login(django_auth_forms.AuthenticationForm):
widget=forms.TextInput(attrs={"autofocus": "autofocus"}))
password = forms.CharField(label=_("Password"),
+ totptoken = forms.CharField(label=_("Totp Token"),
+ widget=forms.TextInput())

def __init__(self, *args, **kwargs):
super(Login, self).__init__(*args, **kwargs)
- fields_ordering = ['username', 'password', 'region']
+ fields_ordering = ['username', 'password', 'totptoken', 'region']
if getattr(settings,
@@ -66,7 +68,8 @@ class Login(django_auth_forms.AuthenticationForm):
widget=forms.TextInput(attrs={"autofocus": "autofocus"}))
self.fields['username'].widget = forms.widgets.TextInput()
- fields_ordering = ['domain', 'username', 'password', 'region']
+ fields_ordering = ['domain', 'username', 'password',
+ 'totptoken', 'region']
self.fields['region'].choices = self.get_region_choices()
if len(self.fields['region'].choices) == 1:
self.fields['region'].initial = self.fields['region'].choices[0][0]
@@ -115,10 +118,11 @@ class Login(django_auth_forms.AuthenticationForm):
username = self.cleaned_data.get('username')
password = self.cleaned_data.get('password')
+ token = self.cleaned_data.get('totptoken')
region = self.cleaned_data.get('region')
domain = self.cleaned_data.get('domain', default_domain)

- if not (username and password):
+ if not (username and password and token):
# Don't authenticate, just let the other validators handle it.
return self.cleaned_data

@@ -126,6 +130,7 @@ class Login(django_auth_forms.AuthenticationForm):
self.user_cache = authenticate(request=self.request,
+ totp=token,

That’s it for Horizon! There’s one more layer between us and Keystone: the Keystoneclient code that’s loaded by wmtotp.py. keystoneclient/auth/identity/v3 contains classes for each auth model so, once again, we drop in a custom wmtotp.py. And, again, it’s a modest modification of password.py:

# Custom addition for Wikimedia Labs to add a totp plugin to keystoneclient
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

import sys

from oslo_config import cfg

from keystoneclient.auth.identity.v3 import base
from keystoneclient import utils

__all__ = ['WmtotpMethod', 'Wmtotp']

class WmtotpMethod(base.AuthMethod):
"""Construct a User/Password/totp based authentication method.

:param string password: Password for authentication.
:param string totp: Totp token for authentication.
:param string username: Username for authentication.
:param string user_id: User ID for authentication.
:param string user_domain_id: User's domain ID for authentication.
:param string user_domain_name: User's domain name for authentication.

_method_parameters = ['user_id',

def get_auth_data(self, session, auth, headers, **kwargs):
user = {'password': self.password, 'totp': self.totp}

if self.user_id:
user['id'] = self.user_id
elif self.username:
user['name'] = self.username

if self.user_domain_id:
user['domain'] = {'id': self.user_domain_id}
elif self.user_domain_name:
user['domain'] = {'name': self.user_domain_name}

return 'wmtotp', {'user': user}

class Wmtotp(base.AuthConstructor):
"""A plugin for authenticating with a username, password, totp token

:param string auth_url: Identity service endpoint for authentication.
:param string password: Password for authentication.
:param string totp: totp token for authentication
:param string username: Username for authentication.
:param string user_id: User ID for authentication.
:param string user_domain_id: User's domain ID for authentication.
:param string user_domain_name: User's domain name for authentication.
:param string trust_id: Trust ID for trust scoping.
:param string domain_id: Domain ID for domain scoping.
:param string domain_name: Domain name for domain scoping.
:param string project_id: Project ID for project scoping.
:param string project_name: Project name for project scoping.
:param string project_domain_id: Project's domain ID for project.
:param string project_domain_name: Project's domain name for project.
:param bool reauthenticate: Allow fetching a new token if the current one
is going to expire. (optional) default True

_auth_method_class = WmtotpMethod

def get_options(cls):
options = super(Wmtotp, cls).get_options()

cfg.StrOpt('user-id', help='User ID'),
cfg.StrOpt('user-name', dest='username', help='Username',
cfg.StrOpt('user-domain-id', help="User's domain id"),
cfg.StrOpt('user-domain-name', help="User's domain name"),
cfg.StrOpt('password', secret=True, help="User's password"),
cfg.StrOpt('totp', secret=True, help="Totp token"),

return options

def load_from_argparse_arguments(cls, namespace, **kwargs):
if not (kwargs.get('password') or namespace.os_password):
kwargs['password'] = utils.prompt_user_password()

if not kwargs.get('totp') and (hasattr(sys.stdin, 'isatty') and
kwargs['totp'] = getpass.getpass('Totp token: ')
except EOFError:

return super(Wmtotp, cls).load_from_argparse_arguments(namespace,

Now, a brief pause for bad news. Despite everything our prior experiences have taught us, keystoneclient seems not to have been designed with extending in mind. The __init__.py file in keystoneclient/auth/identity/v3 has a hard-coded list of available auth classes, so we have to patch __init__.py:

diff --git a/keystoneclient/auth/identity/v3/__init__.py b/keystoneclient/auth/identity/v3/__init__.py
index a08f3ec..c9ecd12 100644
--- a/keystoneclient/auth/identity/v3/__init__.py
+++ b/keystoneclient/auth/identity/v3/__init__.py
@@ -14,6 +14,7 @@ from keystoneclient.auth.identity.v3.base import * # noqa
from keystoneclient.auth.identity.v3.federated import * # noqa
from keystoneclient.auth.identity.v3.password import * # noqa
from keystoneclient.auth.identity.v3.token import * # noqa
+from keystoneclient.auth.identity.v3.wmtotp import * # noqa

__all__ = ['Auth',
@@ -26,5 +27,8 @@ __all__ = ['Auth',

+ 'Mwtotp',
+ 'MwtotpMethod',

Get with the program, keystoneclient! Anyway, after recovering from that one disappointment it was time to make a puppet patch to drop in all of our hacks and overlays and config changes.

And… it works!

Screen Shot 2016-03-04 at 3.11.34 PM

Posted in Operations | Leave a comment

Keystone, Horizon, and multi-factor auth (part 1/2: Keystone)

Fair warning: This post documents a recent work project — it contains neither lush landscape photos nor close-ups of underwater creatures.

For a couple of years now the Wikimedia Labs team has had ambitions to deprecate our homemade OpenStack web interface in favor of the official OpenStack user interface, Horizon. There are dozens of issues to overcome in this transition, but one of the biggest is security: We require two-factor authentication to access all potentially-destructive web interfaces in Labs; Horizon (and, until recently, Keystone) had no support for anything beyond simple username/password logins.

A stock install of Horizon has been publicly available and attached to the backend Labs services for quite a while, but most features were intentionally disabled. Many, many volunteers have rights to manipulate VMs in labs, and I can’t run the risk of of a random stranger logging in with the password ‘password’ and deleting a dozen instances.

Our second factor is standard totp token, enforced on our current UI by the OATHAuth mediawiki extension. Since the migration to Horizon could take a year or more, I want users to be able to use the same credentials on both systems. That means we needed a Keystone plugin that used the same keys that are currently used on our existing system. At my request, security engineer Chris Steipp set up a devstack instance and quickly rattled off a Keystone plugin that works in OpenStack Kilo and Liberty and can authenticate against our existing keys. Here’s the good bit:

class Wmtotp(auth.AuthMethodHandler):

method = METHOD_NAME

def authenticate(self, context, auth_payload, auth_context):
"""Try to authenticate against the identity backend."""
user_info = auth_plugins.UserAuthInfo.create(auth_payload, self.method)

except AssertionError:
# authentication failed because of invalid username or password
msg = _('Invalid username or password')
raise exception.Unauthorized(msg)

# Password auth succeeded, check two-factor
# LOG.debug("OATH: Doing 2FA for user_info " +
# ( "%s(%r)" % (user_info.__class__, user_info.__dict__) ) )
# LOG.debug("OATH: Doing 2FA for auth_payload " +
# ( "%s(%r)" % (auth_payload.__class__, auth_payload) ) )
cnx = mysql.connector.connect(
cur = cnx.cursor(buffered=True)
sql = ('SELECT oath.secret as secret from user '
'left join oathauth_users as oath on oath.id = user.user_id '
'where user.user_name = %s LIMIT 1')
cur.execute(sql, (user_info.user_ref['name'], ))
secret = cur.fetchone()[0]

if secret:
if 'totp' in auth_payload['user']:
(p, d) = oath.accept_totp(
if p:
LOG.debug("OATH: 2FA passed")
LOG.debug("OATH: 2FA failed")
msg = _('Invalid two-factor token')
raise exception.Unauthorized(msg)
LOG.debug("OATH: 2FA failed, missing totp param")
msg = _('Missing two-factor token')
raise exception.Unauthorized(msg)
LOG.debug("OATH: user '%s' does not have 2FA enabled.",

auth_context['user_id'] = user_info.user_id

Rather than build a new Keystone package with the plugin included, I just wrote a puppet patch to drop it into a strategic location. That turns out to be easy because Keystone is designed with this kind of extensibility in mind. (Extensibility patterns tend to change with every release, but that’s a rant for another day.)

The current trunk of Keystone includes a plugin called ‘totp.py’ which I haven’t read but which surely duplicates some or all of Chris’s plugin. As we draw nearer to running Newton we’ll investigate whether or not our custom code can be reduced.

Next post: Getting that OTP from the user and handling it in Horizon.

Posted in Operations | 1 Comment

gone rogue

Screen Shot 2016-02-25 at 1.06.21 PM

Posted in Uncategorized | Leave a comment

Why I am still here

Practically everything we do, from eating an ice to crossing the Atlantic, and from baking a loaf to writing a novel, involves the use of coal, directly or indirectly. For all the arts of peace coal is needed; if war breaks out it is needed all the more. In time of revolution the miner must go on working or the revolution must stop, for revolution as much as reaction needs coal. Whatever may be happening on the surface, the hacking and shovelling have got to continue without a pause, or at any rate without pausing for more than a few weeks at the most. In order that Hitler may march the goose-step, that the Pope may denounce Bolshevism, that the cricket crowds may assemble at Lords, that the poets may scratch one another’s backs, coal has got to be forthcoming.

— George Orwell, Down The Mine

It’s never a good thing when Wikipedia shows up in the news. On good days it’s quietly present in the background of every Google search, classroom project and bar bet, part of the mostly-invisible information infrastructure that we all take for granted.

We haven’t had a lot of those days lately. Wildly inaccurate news reports abound; there are dozens of animated (often acrimonious) discussions underway in a half-dozen different online channels; my colleagues are quitting in droves.

Recent public events are really just the tip of the iceberg. I’m used to a Midwestern work climate where people change jobs a few times in their lives, not a few times in a decade. I’ve been working here for about three years, and I can count on one hand the number of direct coworkers who persist from my first day. Every single box above me on the org chart has been replaced, many of them multiple times. I have stopped learning the names of WMF executives because there’s no point in getting attached.

The current Internet bubble is in full inflation mode and I get enthusiastic recruitment emails every few days for exciting high-tech and high-paying jobs. Most of them I never read. Here’s why, despite current trends, I never daydream about quitting:

– I still feel lucky to be here. It’s a privilege to work on a project that I believe is having an unambiguously positive effect on the world. I don’t love that my workplace has become a sticky, uncomfortable mess, but ‘difficult’ is not the same thing as ‘not worthwhile.’

– My work is just as useful as it has always been. Labs is bigger and bigger, doing more and more, and breaking as much as it ever does. I serve the users, my coworkers, and the projects. As long as the Foundation doesn’t stand between me and the users, its screwups don’t stop my work from being useful.

– My team is still great. Operations hasn’t had as much recent churn as the rest of the organization. I have lost some dear colleagues, but the current bunch is as smart, dedicated, and steadfast as I could ever hope for.

– Where would I go? Those recruitment emails are full of buzzwords that are meant to be inspiring but mostly translate as either ‘maximize ad views’ or ‘minimize contractor wages’. I could, I suppose, go to work for a cloud provider and provide general computing infrastructure… but I’m already providing general computing infrastructure, and it’s for free, and it’s for people who are doing things that I want them to do more of.

Finally, most importantly: going to work doesn’t hurt. I work from home, I talk to pleasant people online during my work day, my work problems are still interesting, it’s all fine. My reasons for staying are largely useless to those who are not: I believe that my coworkers are quitting not because they’ve lost faith in the mission, but because they CAN’T TAKE IT ANYMORE. They’re choosing between Wikimedia and ulcers, between Wikimedia and insomnia, between Wikimedia and yelling at their kids when they get home. And they’re making the right choice.

Me, I’m not facing that choice — lucky me. For anyone on the fence, who’s thinking about leaving not because they’re breaking but because of dreams of greener pastures, don’t forget: this stuff matters. We’re working on projects that took a decade to really get going, and will take decades more to reach their full potential — there will be months, years even, when everything feels stupid and pointless, but the long term importance (and, I might argue, the promise of long-term success) is hard to question. And: the chance to do work that matters doesn’t come along every day.

Posted in Uncategorized | 2 Comments

California Carnivores

We finally made it to California Carnovores in Sebastopol today, and it was much bigger and better than I expected. It also turns out that sundews are incredibly photogenic.

The guys who run this place are very serious about their plants — more than half of the space is not-for-sale pet plants. It was pretty clear that they are living their dreams.

Posted in Uncategorized | Leave a comment

Neutrality 2015

(what follows is an email I just sent to my tenants. In theory I should send this every year; in practice I forget to email most years, but at least I typically remember to buy the offsets.)


This is an annual reminder: a small portion of the rents you pay go to making your home energy consumption carbon-neutral. In a less self-destructive world, these extra costs would be handled automatically by utility companies and/or governments, but in the meantime I declare them to be mandatory for us.

Our natural gas use (which is not extravagant for the number of people, but still considerable on account of Winter) is offset via donations to miscellaneous tree-planting and methane-capture projects; that ends up costing around $1.90 per person per month.

Most of our electricity is generated via windfarm, which is already close to carbon-neutral. Fossil fuel prices shift around a lot compared with the (relatively stable) cost of wind power — coal and natural gas are cheap this year, so wind has been costing around 6% more than the standard fossil-fuel-heavy option. That adds up to another couple of dollars per month per person.

For those of you who drive cars, eat cows, have separate utility meters, etc, I encourage you to offset your own energy use as well. For fossil-fuel offsets, this is a good place to start: http://www.carbonfund.org/projects. To purchase windpower you just need to make a single call to your local electric company.

For me, at least, it’s well worth the price to feel slightly less guilty when there are food riots, major cities destroyed by floods, etc.




Posted in infrastructure | Leave a comment


Some days you just want to sit on a beach and shoot video of ghost crabs.

For roughly my whole life I’ve wondered why white-sand beaches only have white crabs and black-sand beaches only have black crabs. I figured that every beach started out with all colors of crabs and the seagulls sorted things out… But, get this:

When the black-sand specimens were subsequently exposed to a white background, they were observed to gradually become lighter over the course of about a month. The opposite happened to white-sand specimens when placed in black backgrounds.

Posted in critters, travel | 2 Comments

El Yunque

At the tail-end of a series of meetings in Puerto Rico, my work department took a tour of El Yunque National Forest.

I was extremely excited to find the shrimp along the edge of our swimming hole and spent a surprisingly long time taking photos, with only a few non-blurry results.

Posted in critters, travel | Leave a comment