Keystone, Horizon, and multi-factor auth (part 2/2: Horizon)

Now that Keystone accepts and checks auth requests with user/password/otp, we have to get all three from the user and get them properly packed into Horizon’s login request. Taking advantage of Horizon being backwards-compatibility, I upgraded our Horizon install to version Liberty before starting.

I spent quite a while hopelessly grepping in the Horizon and Dashboard code until someone on IRC directed me to the openstack_auth module which turns out to contain all the good bits. Once again, the auth code uses a plugin model, so adding totp support is just a matter of dropping a new file into openstack_auth/plugins/. My new file is called ‘’ and it’s a copy of the ‘’ plugin with the Keystone v2 API code ripped out and an extra parameter added on.

import logging

from keystoneclient.auth.identity import v2 as v2_auth
from keystoneclient.auth.identity import v3 as v3_auth

from openstack_auth.plugin import base
from openstack_auth import utils

LOG = logging.getLogger(__name__)

__all__ = ['WmtotpPlugin']

class WmtotpPlugin(base.BasePlugin):
"""Authenticate against keystone given a username, password, totp token.

def get_plugin(self, auth_url=None, username=None, password=None,
user_domain_name=None, totp=None, **kwargs):
if not all((auth_url, username, password, totp)):
return None

LOG.debug('Attempting to authenticate for %s', username)

if utils.get_keystone_version() >= 3:
return v3_auth.Wmtotp(auth_url=auth_url,

msg = "Totp authentication requires the keystone v3 api."
raise exceptions.KeystoneAuthException(msg)

I also needed to tell Horizon to use the new auth method during logins. That’s a change to the config file:

AUTHENTICATION_PLUGINS = ['openstack_auth.plugin.wmtotp.WmtotpPlugin', 'openstack_auth.plugin.token.TokenPlugin']

Now we just have to get our second factor from the user, and hand it to off to WmtotpPlugin. This is where Horizon is not extensible — there’s just “” that draws a single username/password dialog, with no custom field options. Time to get ugly and clobber with a patched version.

diff --git a/openstack_auth/ b/openstack_auth/
index 97a8bbf..0aeac58 100644
diff --git a/openstack_auth/ b/openstack_auth/
index 97a8bbf..0aeac58 100644
--- a/openstack_auth/
+++ b/openstack_auth/
@@ -54,10 +54,12 @@ class Login(django_auth_forms.AuthenticationForm):
widget=forms.TextInput(attrs={"autofocus": "autofocus"}))
password = forms.CharField(label=_("Password"),
+ totptoken = forms.CharField(label=_("Totp Token"),
+ widget=forms.TextInput())

def __init__(self, *args, **kwargs):
super(Login, self).__init__(*args, **kwargs)
- fields_ordering = ['username', 'password', 'region']
+ fields_ordering = ['username', 'password', 'totptoken', 'region']
if getattr(settings,
@@ -66,7 +68,8 @@ class Login(django_auth_forms.AuthenticationForm):
widget=forms.TextInput(attrs={"autofocus": "autofocus"}))
self.fields['username'].widget = forms.widgets.TextInput()
- fields_ordering = ['domain', 'username', 'password', 'region']
+ fields_ordering = ['domain', 'username', 'password',
+ 'totptoken', 'region']
self.fields['region'].choices = self.get_region_choices()
if len(self.fields['region'].choices) == 1:
self.fields['region'].initial = self.fields['region'].choices[0][0]
@@ -115,10 +118,11 @@ class Login(django_auth_forms.AuthenticationForm):
username = self.cleaned_data.get('username')
password = self.cleaned_data.get('password')
+ token = self.cleaned_data.get('totptoken')
region = self.cleaned_data.get('region')
domain = self.cleaned_data.get('domain', default_domain)

- if not (username and password):
+ if not (username and password and token):
# Don't authenticate, just let the other validators handle it.
return self.cleaned_data

@@ -126,6 +130,7 @@ class Login(django_auth_forms.AuthenticationForm):
self.user_cache = authenticate(request=self.request,
+ totp=token,

That’s it for Horizon! There’s one more layer between us and Keystone: the Keystoneclient code that’s loaded by keystoneclient/auth/identity/v3 contains classes for each auth model so, once again, we drop in a custom And, again, it’s a modest modification of

# Custom addition for Wikimedia Labs to add a totp plugin to keystoneclient
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

import sys

from oslo_config import cfg

from keystoneclient.auth.identity.v3 import base
from keystoneclient import utils

__all__ = ['WmtotpMethod', 'Wmtotp']

class WmtotpMethod(base.AuthMethod):
"""Construct a User/Password/totp based authentication method.

:param string password: Password for authentication.
:param string totp: Totp token for authentication.
:param string username: Username for authentication.
:param string user_id: User ID for authentication.
:param string user_domain_id: User's domain ID for authentication.
:param string user_domain_name: User's domain name for authentication.

_method_parameters = ['user_id',

def get_auth_data(self, session, auth, headers, **kwargs):
user = {'password': self.password, 'totp': self.totp}

if self.user_id:
user['id'] = self.user_id
elif self.username:
user['name'] = self.username

if self.user_domain_id:
user['domain'] = {'id': self.user_domain_id}
elif self.user_domain_name:
user['domain'] = {'name': self.user_domain_name}

return 'wmtotp', {'user': user}

class Wmtotp(base.AuthConstructor):
"""A plugin for authenticating with a username, password, totp token

:param string auth_url: Identity service endpoint for authentication.
:param string password: Password for authentication.
:param string totp: totp token for authentication
:param string username: Username for authentication.
:param string user_id: User ID for authentication.
:param string user_domain_id: User's domain ID for authentication.
:param string user_domain_name: User's domain name for authentication.
:param string trust_id: Trust ID for trust scoping.
:param string domain_id: Domain ID for domain scoping.
:param string domain_name: Domain name for domain scoping.
:param string project_id: Project ID for project scoping.
:param string project_name: Project name for project scoping.
:param string project_domain_id: Project's domain ID for project.
:param string project_domain_name: Project's domain name for project.
:param bool reauthenticate: Allow fetching a new token if the current one
is going to expire. (optional) default True

_auth_method_class = WmtotpMethod

def get_options(cls):
options = super(Wmtotp, cls).get_options()

cfg.StrOpt('user-id', help='User ID'),
cfg.StrOpt('user-name', dest='username', help='Username',
cfg.StrOpt('user-domain-id', help="User's domain id"),
cfg.StrOpt('user-domain-name', help="User's domain name"),
cfg.StrOpt('password', secret=True, help="User's password"),
cfg.StrOpt('totp', secret=True, help="Totp token"),

return options

def load_from_argparse_arguments(cls, namespace, **kwargs):
if not (kwargs.get('password') or namespace.os_password):
kwargs['password'] = utils.prompt_user_password()

if not kwargs.get('totp') and (hasattr(sys.stdin, 'isatty') and
kwargs['totp'] = getpass.getpass('Totp token: ')
except EOFError:

return super(Wmtotp, cls).load_from_argparse_arguments(namespace,

Now, a brief pause for bad news. Despite everything our prior experiences have taught us, keystoneclient seems not to have been designed with extending in mind. The file in keystoneclient/auth/identity/v3 has a hard-coded list of available auth classes, so we have to patch

diff --git a/keystoneclient/auth/identity/v3/ b/keystoneclient/auth/identity/v3/
index a08f3ec..c9ecd12 100644
--- a/keystoneclient/auth/identity/v3/
+++ b/keystoneclient/auth/identity/v3/
@@ -14,6 +14,7 @@ from keystoneclient.auth.identity.v3.base import * # noqa
from keystoneclient.auth.identity.v3.federated import * # noqa
from keystoneclient.auth.identity.v3.password import * # noqa
from keystoneclient.auth.identity.v3.token import * # noqa
+from keystoneclient.auth.identity.v3.wmtotp import * # noqa

__all__ = ['Auth',
@@ -26,5 +27,8 @@ __all__ = ['Auth',

+ 'Mwtotp',
+ 'MwtotpMethod',

Get with the program, keystoneclient! Anyway, after recovering from that one disappointment it was time to make a puppet patch to drop in all of our hacks and overlays and config changes.

And… it works!

Screen Shot 2016-03-04 at 3.11.34 PM

Posted in Operations | Leave a comment

Keystone, Horizon, and multi-factor auth (part 1/2: Keystone)

Fair warning: This post documents a recent work project — it contains neither lush landscape photos nor close-ups of underwater creatures.

For a couple of years now the Wikimedia Labs team has had ambitions to deprecate our homemade OpenStack web interface in favor of the official OpenStack user interface, Horizon. There are dozens of issues to overcome in this transition, but one of the biggest is security: We require two-factor authentication to access all potentially-destructive web interfaces in Labs; Horizon (and, until recently, Keystone) had no support for anything beyond simple username/password logins.

A stock install of Horizon has been publicly available and attached to the backend Labs services for quite a while, but most features were intentionally disabled. Many, many volunteers have rights to manipulate VMs in labs, and I can’t run the risk of of a random stranger logging in with the password ‘password’ and deleting a dozen instances.

Our second factor is standard totp token, enforced on our current UI by the OATHAuth mediawiki extension. Since the migration to Horizon could take a year or more, I want users to be able to use the same credentials on both systems. That means we needed a Keystone plugin that used the same keys that are currently used on our existing system. At my request, security engineer Chris Steipp set up a devstack instance and quickly rattled off a Keystone plugin that works in OpenStack Kilo and Liberty and can authenticate against our existing keys. Here’s the good bit:

class Wmtotp(auth.AuthMethodHandler):

method = METHOD_NAME

def authenticate(self, context, auth_payload, auth_context):
"""Try to authenticate against the identity backend."""
user_info = auth_plugins.UserAuthInfo.create(auth_payload, self.method)

except AssertionError:
# authentication failed because of invalid username or password
msg = _('Invalid username or password')
raise exception.Unauthorized(msg)

# Password auth succeeded, check two-factor
# LOG.debug("OATH: Doing 2FA for user_info " +
# ( "%s(%r)" % (user_info.__class__, user_info.__dict__) ) )
# LOG.debug("OATH: Doing 2FA for auth_payload " +
# ( "%s(%r)" % (auth_payload.__class__, auth_payload) ) )
cnx = mysql.connector.connect(
cur = cnx.cursor(buffered=True)
sql = ('SELECT oath.secret as secret from user '
'left join oathauth_users as oath on = user.user_id '
'where user.user_name = %s LIMIT 1')
cur.execute(sql, (user_info.user_ref['name'], ))
secret = cur.fetchone()[0]

if secret:
if 'totp' in auth_payload['user']:
(p, d) = oath.accept_totp(
if p:
LOG.debug("OATH: 2FA passed")
LOG.debug("OATH: 2FA failed")
msg = _('Invalid two-factor token')
raise exception.Unauthorized(msg)
LOG.debug("OATH: 2FA failed, missing totp param")
msg = _('Missing two-factor token')
raise exception.Unauthorized(msg)
LOG.debug("OATH: user '%s' does not have 2FA enabled.",

auth_context['user_id'] = user_info.user_id

Rather than build a new Keystone package with the plugin included, I just wrote a puppet patch to drop it into a strategic location. That turns out to be easy because Keystone is designed with this kind of extensibility in mind. (Extensibility patterns tend to change with every release, but that’s a rant for another day.)

The current trunk of Keystone includes a plugin called ‘’ which I haven’t read but which surely duplicates some or all of Chris’s plugin. As we draw nearer to running Newton we’ll investigate whether or not our custom code can be reduced.

Next post: Getting that OTP from the user and handling it in Horizon.

Posted in Operations | 6 Comments

gone rogue

Screen Shot 2016-02-25 at 1.06.21 PM

Posted in Uncategorized | Leave a comment

Why I am still here

Practically everything we do, from eating an ice to crossing the Atlantic, and from baking a loaf to writing a novel, involves the use of coal, directly or indirectly. For all the arts of peace coal is needed; if war breaks out it is needed all the more. In time of revolution the miner must go on working or the revolution must stop, for revolution as much as reaction needs coal. Whatever may be happening on the surface, the hacking and shovelling have got to continue without a pause, or at any rate without pausing for more than a few weeks at the most. In order that Hitler may march the goose-step, that the Pope may denounce Bolshevism, that the cricket crowds may assemble at Lords, that the poets may scratch one another’s backs, coal has got to be forthcoming.

— George Orwell, Down The Mine

It’s never a good thing when Wikipedia shows up in the news. On good days it’s quietly present in the background of every Google search, classroom project and bar bet, part of the mostly-invisible information infrastructure that we all take for granted.

We haven’t had a lot of those days lately. Wildly inaccurate news reports abound; there are dozens of animated (often acrimonious) discussions underway in a half-dozen different online channels; my colleagues are quitting in droves.

Recent public events are really just the tip of the iceberg. I’m used to a Midwestern work climate where people change jobs a few times in their lives, not a few times in a decade. I’ve been working here for about three years, and I can count on one hand the number of direct coworkers who persist from my first day. Every single box above me on the org chart has been replaced, many of them multiple times. I have stopped learning the names of WMF executives because there’s no point in getting attached.

The current Internet bubble is in full inflation mode and I get enthusiastic recruitment emails every few days for exciting high-tech and high-paying jobs. Most of them I never read. Here’s why, despite current trends, I never daydream about quitting:

– I still feel lucky to be here. It’s a privilege to work on a project that I believe is having an unambiguously positive effect on the world. I don’t love that my workplace has become a sticky, uncomfortable mess, but ‘difficult’ is not the same thing as ‘not worthwhile.’

– My work is just as useful as it has always been. Labs is bigger and bigger, doing more and more, and breaking as much as it ever does. I serve the users, my coworkers, and the projects. As long as the Foundation doesn’t stand between me and the users, its screwups don’t stop my work from being useful.

– My team is still great. Operations hasn’t had as much recent churn as the rest of the organization. I have lost some dear colleagues, but the current bunch is as smart, dedicated, and steadfast as I could ever hope for.

– Where would I go? Those recruitment emails are full of buzzwords that are meant to be inspiring but mostly translate as either ‘maximize ad views’ or ‘minimize contractor wages’. I could, I suppose, go to work for a cloud provider and provide general computing infrastructure… but I’m already providing general computing infrastructure, and it’s for free, and it’s for people who are doing things that I want them to do more of.

Finally, most importantly: going to work doesn’t hurt. I work from home, I talk to pleasant people online during my work day, my work problems are still interesting, it’s all fine. My reasons for staying are largely useless to those who are not: I believe that my coworkers are quitting not because they’ve lost faith in the mission, but because they CAN’T TAKE IT ANYMORE. They’re choosing between Wikimedia and ulcers, between Wikimedia and insomnia, between Wikimedia and yelling at their kids when they get home. And they’re making the right choice.

Me, I’m not facing that choice — lucky me. For anyone on the fence, who’s thinking about leaving not because they’re breaking but because of dreams of greener pastures, don’t forget: this stuff matters. We’re working on projects that took a decade to really get going, and will take decades more to reach their full potential — there will be months, years even, when everything feels stupid and pointless, but the long term importance (and, I might argue, the promise of long-term success) is hard to question. And: the chance to do work that matters doesn’t come along every day.

Posted in Uncategorized | 2 Comments

California Carnivores

We finally made it to California Carnovores in Sebastopol today, and it was much bigger and better than I expected. It also turns out that sundews are incredibly photogenic.

The guys who run this place are very serious about their plants — more than half of the space is not-for-sale pet plants. It was pretty clear that they are living their dreams.

Posted in Uncategorized | Leave a comment

Neutrality 2015

(what follows is an email I just sent to my tenants. In theory I should send this every year; in practice I forget to email most years, but at least I typically remember to buy the offsets.)


This is an annual reminder: a small portion of the rents you pay go to making your home energy consumption carbon-neutral. In a less self-destructive world, these extra costs would be handled automatically by utility companies and/or governments, but in the meantime I declare them to be mandatory for us.

Our natural gas use (which is not extravagant for the number of people, but still considerable on account of Winter) is offset via donations to miscellaneous tree-planting and methane-capture projects; that ends up costing around $1.90 per person per month.

Most of our electricity is generated via windfarm, which is already close to carbon-neutral. Fossil fuel prices shift around a lot compared with the (relatively stable) cost of wind power — coal and natural gas are cheap this year, so wind has been costing around 6% more than the standard fossil-fuel-heavy option. That adds up to another couple of dollars per month per person.

For those of you who drive cars, eat cows, have separate utility meters, etc, I encourage you to offset your own energy use as well. For fossil-fuel offsets, this is a good place to start: To purchase windpower you just need to make a single call to your local electric company.

For me, at least, it’s well worth the price to feel slightly less guilty when there are food riots, major cities destroyed by floods, etc.




Posted in infrastructure | Leave a comment


Some days you just want to sit on a beach and shoot video of ghost crabs.

For roughly my whole life I’ve wondered why white-sand beaches only have white crabs and black-sand beaches only have black crabs. I figured that every beach started out with all colors of crabs and the seagulls sorted things out… But, get this:

When the black-sand specimens were subsequently exposed to a white background, they were observed to gradually become lighter over the course of about a month. The opposite happened to white-sand specimens when placed in black backgrounds.

Posted in critters, travel | 2 Comments

El Yunque

At the tail-end of a series of meetings in Puerto Rico, my work department took a tour of El Yunque National Forest.

I was extremely excited to find the shrimp along the edge of our swimming hole and spent a surprisingly long time taking photos, with only a few non-blurry results.

Posted in critters, travel | Leave a comment

Lysmata, 30 days

A month later all 7 little Lysmata shrimp are doing fine. Three of them had a netbox to themselves, and two of them were paired up. I didn’t have any trouble with cannibalism, and the shrimp grow slowly but visibly. Food was plankton-based flake food, Wardley shrimp pellets, and frozen krill.


Meanwhile, the parents have gone through another cycle of laying and hatching. I’ve collected a few dozen zoeae which I’m raising in the same manner as before. If they make it to the post-larva I’m planning to try a more free-range approach to raising them and see how they do in larger numbers.


Posted in critters | Leave a comment

Lysmata wurdemanni

Peppermint shrimp with eggs

After several failed attempts, I’ve just now succeeded in raising my first peppermint shrimp from egg to post-larva. I took a bunch of photos and videos, and they’re all terrible.

This species has a reputation for being fairly easy to raise, although googling turns up more tales of failure than success. The process I’m using now is dead simple, and I’ve met with success on my first try with these particular parents, so I suspect that the variation in success with captive rearing is a result of a bunch of different species being imported with the same ‘peppermint shrimp’ label. Are my shrimp really L. wurdemanni? I’ll probably never know.

I started out with a paltry number of larva — around a dozen. I have an air-powered plankton collector in the tank with the adults, but also a protein skimmer which I’m sure is snagging most of the hatchlings. I moved half the larva into a 1-liter box with blacked out sides (as per googled suggestions). Those didn’t last long. The other half went into an extremely low-tech, improvised kreisel:

planktonbowl - 1

That’s a 2-gallon drum bowl with a bit of bubbling airline in the corner. This worked surprisingly well — shrimp larvae and their brine shrimp food remained suspended in the water column at all times, unlike the larvae in the smaller box who tended to bump around on the bottom.

Here are close-ups of newly-hatched Lysmata, along with a high-speed video of their weird bendy pleopods:

L_wurdmanni_larva - 2

L_wurdmanni_larva - 1

After five days, not much had changed.

A week later, they had changed shape completely. I mistook the longer legs for pincers, but closer inspection shows that they’re back legs rather than forelegs — it seems like they’re used for stability when floating rather than for hunting.

larva_11_days - 1

At day 25, the larvae are much bigger, but the body form remains much the same. It’s hard to tell scale from the photos, but they’re in the neighborhood of 10-15mm at this point. They have the same constantly-waving pleopods that they had on day one; the front four pairs of legs look like legs, and the back pair are still those crazy, giant oar-like shapes.

larva_21_days - 1

Finally on day 30, one of the larvae changed in a flash into a shrimp. The crazy oars are gone, the pleopods are tucked under the body, and it’s almost entirely adult-shaped. Rather than drifting aimlessly in the bowl, it’s sitting on the back or the sides, occasionally zooming with purpose to a new roost.

For scale, this photo includes a US quarter dollar. The video is completely boring, since now the shrimp is capable of resting.

L_wurdmanni_postlarva - 1

I still have six more late-stage larvae floating in the bowl. Their sizes vary quite a bit; I’ve no idea how they decide when to metamorphose and settle.

Care is pretty simple. I feed the larvae newly-hatched artemia onec per day, and leave the brine shrimp to swim amongst the Lysmata for as long as it takes to be eaten. Every three days or so I also feed 3-day-old SELCO-enriched artemia. I’ve no idea if the SELCO is essential or not — it’s certainly a lot more trouble. At no point did I attempt to feed larger or different foods.

Every few days I scooped 1/4 of the water out of the bowl and exchanged it with water in the 40-gallon tank that houses the adults. I didn’t attempt to vaccuum the bottom of the bowl — some debris accumulated but it doesn’t seem to do any harm. A light shining on the bottom of the bowl keeps the larvae down low to reduce the chances of them getting scooped out during water changes.

There’s no direct light over the bowl. Other reports complain of larva getting trapped in biofilm along the edges of the container; I didn’t see anything like this, possibly because there wasn’t enough light for biofilm or possibly because the circular current prevented larvae from resting on the glass.

None of the regimen was very compulsive. I missed days of feeding here and there, and two or three times a breaker tripped and the bowl was without aeration for half a day at a time.

The one post-larva is now inhabiting a much-too-big net breeder in the parent’s tank. I’m hoping to keep the young adults separate from one another to avoid cannibalism during molting… we will see how many more I get, and if raising them after metamorphosis is harder than rearing them before.

netbreeder - 1

UPDATE, 2015-09-05: Now six of the seven larvae have settled. All the tiny shrimp are in a long row of tiny net breeders, and all are still alive.

Posted in critters | 5 Comments