Gunnar Hellekson has many awesome-sounding job titles.

He's the chief technology strategist for Red Hat's US Public Sector group, where he works with government departments to show them how open source can meet their needs, and with systems integrators to show them what they can do to provide the government with what it needs.

He's co-chair of Open Source for America, which campaigns for software that has been funded by the tax-payer to be open sourced, so that all Americans can benefit from it. He's also on the boards of the Military Open Source Working Group, Civic Commons and the SIIA Software Division.

He's a clever chap with the ear of some pretty influential people, so we sar down with him for a chat.

Linux Format: First thing: Is the US government in favour of open source, or does it see it as stealing food from Microsoft's children?

Gunnar Hellekson: The government was actually an early user of open source, going back to 1978. You had the US government funding the development of things like the BSD TCP/IP stack; the ping tool was developed by the army research lab, and at some point in the 90s people started to wonder more, look more carefully at open source; and the government started passing rules like the Clinger-Cohen Act of 1996, which formalised the rules around IP acquisition. Suddenly, there were rules, which meant that there were concerns about whether people were following the rules or not.

When we had no rules, open source made sense. When the rules came in, people started to ask: "Wait, can we do open source?" And it wasn't until about 2003–4, when the Department of Defense [DOD] and the Office of Management and Budget said: "Actually, open source is fine. Don't worry about it, open source is just like any commercial software licence."

The irony of this is that while the slow gears of policy were moving, the Department of Energy and DOD, and the NSA [the National Security Agency] were all releasing source code out to the public, uninterrupted. So to say that the government has one position or another on open source is not only inaccurate, but it's impossible to describe, because the government is 12 million people. Some of them are great open source advocates and some of them aren't.

LXF: Didn't the NSA come up with SELinux, which is in the kernel now?

GH: Security Enhanced Linux, in 2001. And the reason why they did that is the classic story, right? They did it for a number of reasons. First, they wanted to relieve themselves of the technical debt of having developed the technology. If they had developed it and kept it to themselves, only they could have maintained it, and that's expensive.

So by putting it out to the open source community, into the Linux kernel, they could get some help, which was nice for them.

More importantly, the part of the mission that everyone forgets is that the NSA is also responsible for protecting the country's information infrastructure and making commercial products more secure. And so by making SELinux highly available – it's in every copy of Linux – it's actually improved the overall security of the country. So there were a bunch of reasons to do it.

LXF: What does your role with Red Hat entail? Are you trying to push this agenda to various government departments?

GH: In part, that's what it is. The best way to describe my job is telling the government what's happening in open source, and telling open source communities what the government is after.

LXF: Right, kind of like a community manager for those 12 million people who work in government?

GH: More like a hostage negotiator.

LXF: Have you seen any of the IT projects that are going on in the UK? Like the NHS IT project, for instance. That's an enormous project, and a black hole for tax-payers' money. Do you think there's anything inherent in a government that makes them think along such large lines?

GH: Well that's interesting, because in the United States we're heading in the opposite direction. The federal CIO has declared an end to large procurements, so rather than having one $500 million contract, we have 100 $5 million contracts. And it was the reason for the change, not just because it's more efficient and because there's less risk, but also because the current procurement system can literally not keep up with advances in technology.

At the DOD, the lead-in time for a top-level program takes 48 months to get from initiation to requirements. You haven't even put a bid out, you haven't even made a tender yet, but you've spent four years developing requirements. And in those four years, the entire world has changed. And so it's just not practical to run an IT project like that.

Open government

And so, in 2012 in the Appropriations Bill for the Defence Department, Congress ordered the DOD to come up with alternative acquisition strategies specifically for IT to fix this problem. They were asking for things like continual involvement of the user, an iterative, evolutionary approach... and what they were describing was that they wanted an agile IT project.

And so, subsequently, we've seen this model all over the government, with a more iterative approach and projects broken down into tiny chunks.

LXF: Our secretary of state has said that should happen, that you should break down contracts in to small chunks, but as yet, that's all that has happened: that he has said that it should happen.

GH: Well, from what I understand, since the early 2000s the UK Government put out a number of very large contracts with very long performance terms, like 10-year engagements.

I'm thinking specifically of the MOD, which visibly took most of its IT organisation and threw it up for ransom to a consortium of five companies... what were they called? Fujistu Siemens, those kinds of people. And you're getting exactly what you paid for, right?

Not only was it a huge amount of cash up-front, but also the government has no negotiating position, because any change they want to make translates into more money that you have to pay the consortium.

And so that's what agile IT combats: not only is it more iterative, but there's more competition for each iteration.

LXF: Do you see that agile, more responsive development... do you think that's a key advantage of open source in big government projects, as compared with open file formats, for example.

GH: Yeah, so, what's more important? Open source or open standards, right? I think they both solve a different set of problems. When you have an open standard, you're creating a market. You're creating the opportunity for many people to perform the same task.

So if I'm using a standard like, say, IMAP for email, then I can ask any number of IMAP servers, and I don't have to change clients every time I change my server, because if I'm on IMAP I can compete all my IMAP servers against each other, which will drive down the cost.

With open source, what I'm giving myself is a vendor of first resort or of last resort, and I always have that option. So that even if... you can have an open standard and if only one company implements it you're just as locked-in as you were before.

But with open source, you always have an alternative. I can use the code unsupported, or I can find a clever open source hacker who can support it for me.

LXF: What about the argument that free software is crowding out commercially-made software, and stopping companies from making money?

GH: The government has always been concerned about... the Clinger-Cohen act, passed in the 90s, created a preference for commercial software in government, and what they meant by that was software that the government didn't make. What they wanted to do was to make sure that the government didn't end up doing itself something that could be done more effectively by the private sector.

So the rules say that before you go and build something, you have to look outside and make sure that nobody has already built one. So when government starts writing software, there's an immediate gut reaction that it's duplicative.

But, of course, there are cases where it makes sense for the government to be writing its own software, and Accumulo is a great example because Accumulo has features that didn't exist in any other project at the time.

The way they do charting, the way they do document storage, the way they do cell-level security, so I can determine for each individual piece of data who's allowed to talk to it or not. These features were unique to the Accumulo project, and the government did the right thing, because they open-sourced it.

And when you open-source it, it becomes a commercial item, because it's released under a commercial licence, which is the Apache licence. So it's under the Apache licence, which means that it's a commercial product. The senate is rightly concerned about crowding stuff out, but we have a case where it's not government-owned software anymore.

LXF: And rather than crowding out private enterprise, they've actually created a market for support services.

GH: There's a company called SQRRL, which closed its first round of funding a few months ago, wanting to be the Red Hat for Accumulo. So, here's an example of a government technology transfer that works. So I think that the senate concerns are valid, but in this case they're conflating government software with government-produced open source software, which are two very different things.

LXF: The other thing that we're always pushing as an advantage of open source is that it costs less, because you're not paying a licence fee. Is that an important factor, or is it irrelevant at government level? Because I imagine that the number of hackers you'd need to employ would be pretty expensive.

GH: Writing software is expensive, and it's even more expensive to maintain software. We've spent a lot of money writing code. Cost is often a factor in open source software for all the reasons you mentioned. It's often cheaper.

But I always caution people against saying that open source is always cheaper, or always more expensive, because although there are a number of advantages to the open source process it's always possible that a project is going to be very expensive to run.

Or bring it to your IT shop. So when we talk about saving money, it's important to look at what purpose you're using it for. So the economic value of open source is going to be very specific to which software project we're talking about.

All that said, there are a number of second-order effects to using open source that are definitely advantages.

And it's stuff like: you can always compete for maintenance; you can always fix it if it breaks; and the most important thing for me is that it gives you access to a whole bunch of innovation that would not be available to you otherwise.

LXF: Which you don't get if you're locked in to a ten-year contract with Capgemini.

GH: The way we look at it is this: take your favourite software vendor and draw a circle around all those developers. In the world, are there more smart people inside or outside that circle? And that's true of any organisation.

If you're buying from a proprietary vendor, your software is only as good as how many smart people they can hire, which doesn't seem like a very good risk proposition to me. You want to use software and software vendors that have access to as many smart people as possible.

Open government

LXF: How's Red Hat doing?

GH: Well great, you just saw the press today. I heard the number three and I heard the number 12, but we're one of the few pure software companies to go over $1bn revenue.

One thing that's exciting for me about Red Hat right now, I love working for Red Hat, I love my job, I've been here for seven years. And I've never been as excited about anything as I am about OpenShift.

I know it's not on your beat, but having an open-source platform for a server is a complete game changer for my customers to the way they plan on procuring software. Giving them a way to control all of their technology going forward from that point of view is to start looking at platforms as a server.

And I'm going to be blogging a bunch about that. It's a huge deal, and it's really exciting.

LXF: Is that related to OpenStack?

GH: It lies on top of OpenStack. So OpenStack will give you a VM; OpenShift will let you say, "give me a Python environment, give me a PHP environment and put WordPress on it".

And then it will automatically create these things we call cartridges; cartridges for Mongo, PHP, Perl, Ruby, Java... so all the building blocks have already been laid out, already secure. And you're not even looking at the virtualisation layer.

In fact, you don't even know where the stuff is running. As a developer, all you're working with is Git. You write your code, then you do a Git push, and once you've pushed it, it's running inside the environment. It's really cool.

LXF: It sounds kind of like Ubuntu's Juju thingamibob, the cloud as a service product they launched earlier this year.

GH: I sat in front of a Juju session two days ago, and I'm still trying to figure out what it is. But Juju is coming at a similar problem from a different angle. OpenShift has a bunch of other stuff. Juju is a way of helping sysadmins; what OpenShift does is, it lets you create these Linux containers so you can confine applications inside. One container, one jail, and nobody can talk to anybody else.

We've got it running on EC2; it'll spin up VMs, then spin up containers within those VMs, so we can get 400 customers on one box, which is awesome. OpenShift is a way to manage who owns what.

The way we make money is that you can spin up three cartridges for free, but if you want extra stuff, if you want management tools and all the rest of it, then you sign up for the service and you pay for additional space. The navy are looking at it, the air force...

LXF: Did you see that Linux is being used in one of the US navy drones now?

GH: Uh-huh. The Firescout. We actually made fun of that article. If you look at the timeline of government use of open source, you see that there are all these data points going back to 1978 all the way up to 2012, there's this mass of articles all calling for advocacy and adoption of open source projects.

Then you get this one $26 million contract to put Linux on a Firescout, which is like an insignificant dot, right? The thing we were talking about in the session was, "when do we get to stop talking about open source in government?"

What I said was: "Every time we talk about open source being a big deal in government, that's us not winning". We want open source to be totally unremarkable. It should just be part of the infrastructure.

And so when someone is surprised that the US government is using Linux... you know, governments have been using Linux for a very long time. The government has been contributing to the Linux kernel since at least 2000. It's kind of funny to see people say "Ooh, Linux is in the Firescout". It's as if it's going to force cataclysmic changes in the GPL. No! It's not a cataclysmic event, it's a contract.

LXF: Are there any particularly awesome Red Hat adoptions in the US government?

GH: Sure. The FAA, their traffic flow management system, since 2001 has been running on Linux. So, every time you take a civilian flight in the United States, there's a Linux workstation managing it. US Census is a Red Hat user.

Every week, the government puts out employment numbers. If they are even five minutes late announcing it that is an apocalyptic event on Wall Street. They're running Linux.

The Patent and Trademark office, what else... the national weather service, every weather forecast comes off a Linux box.

Oh, the FBI, this is kind of fun. Every time you see a fingerprint check or a background check on somebody, that is all going back to a system that is running on... I think they're using every Red Hat product that there is.

They have 16 million records, and they run every background check, every fingerprint, all the biometric data, and when you crossed the border they took a fingerprint, right? That fingerprint went back to a data centre running all that stuff and came back in under 15 seconds to make sure that you weren't flagged. That's all Red Hat.