Yesterday one of our team posted a link in our work chat channel to a post by Philip Greenspun on how Accenture recently landed a $90 million dollar per year contract to take care of healthcare.gov, joking that we should have bid on it.
I don't think anyone can dispute that the original contractors bungled the job severely. [1] However, before I rant on why this contract disgusts me let's get a few things out of the way:
- We all know that working inside a big bureaucracy slows everything down and basically sucks. Design by committee. A dozen or so project managers. Pre-meetings to get ready for the real meeting, etc.
- While I've only had a few small dealings with HIPAA I'm sure it was a big pain, for better or worse.[2]
- The process required coordinating data, feeds, and integrations with several different government and private health insurance systems to one degree or another. Legacy systems, speaking legacy protocols, on legacy hardware. Not an easy task, but as someone who has presided over dozens and dozens of weird systems integrations not exactly Nobel Prize worthy stuff.
- They had to deal with ancient IE browsers and a bunch of other crap those of us working with more modern tech don't bother with any longer. Grandma isn't likely running the latest Chrome.
- It was a project with a high public profile and was assured to get traffic that by comparison makes most large startup launches look like my personal blog. And no matter how awesome it might have gone, it was going to get some level of complaints.
Some cost comparisons
All that aside, I'm confident $90 million is easily more than double what this should cost. Let's look at some quick comparisons:
We sent two robots to Mars for $820 million dollars. When they performed better than expected we extended their missions over 5 years for about $100 million. Apparently it costs roughly twice as much to travel to and explore Mars with freaking robots controlled from earth per year as it does to signup a few million people on a website.
While on the topic of Space and not to sound like an Elon Musk fan boy, but a SpaceX Falcon 9 rocket only costs $57 million.
So let's break it down a bit in actual costs for something like this.
- Let's assume they need a lot more hardware than I think, let's call it $10 million a year in hardware costs[3].
- Toss in another $10 million a year to cover bureaucracy, lawyers, lobbyists, etc.
- Add in another $10 million a year in consultants and anything else I'm forgetting for good measure.
- Average US programmer salary in 2011 was $72k, so let's call it $250k/engineer/year to round up because fuck it why not. So another $40 million of the budget would get you 160 extremely highly paid engineers[4].
And we're still at only $70 million with these inflated estimates.
So what's wrong?
When I read Greenspun's post it reminded me of a conversation I had a few years ago with a colleague. He's tangentially involved in state level healthcare policy and mentioned there were a few contracts out for bid that he thought RevSys might be interested in. The exact details escape me, but one involved a few web forms for doctors and hospitals in the state to check on the enrollment of patients in some Medicare program. It would have involved integrating a few legacy state systems, but otherwise pretty straight forward. He asked how much something like this would cost and I ball parked it around $200-300k assuming the integrations would be painful and that the project would eat up tons of time in meetings. His response floored me, he laughed and said "Oh you'd need to bid at least a few million or they won't take you seriously."
The whole process of bidding and then winning/running a project with the government is a feat unto itself. I've been loosely involved with a few proposals, usually as the "tech expert" and not the party actually putting together the full proposal materials, and I can tell you it eats up an inordinate amount of time and energy just to bid on the project.
The process almost assures that the company best at the process wins rather than the one that might deliver the best or most cost effective technological solution. Luckily I've never been a part of a winning proposal as I imagine the day-to-day to be equally grueling and professionally unfulfilling.
When the government announced they were bringing in "tech's best and brightest" I rolled my eyes, assuming they would fail miserably at getting decent talent. However they brought in Red Hat, which was a pleasant surprise. But as many others have commented on, it was too little and way too late.
Most projects like this aren't sexy. They certainly aren't pushing the envelope tech challenges. So perhaps they can't attract even the Top 10% of tech talent either because the problem is boring or they're too busy trying to strike it rich with a startup. But what about the next 10-20% below them on the tech totem pole?
Is $90 million the best we can do? Should I be happy it wasn't $120 million? Does it really all boil down to boring problems and the burden of the process? We all know it's broken. The question is can it be fixed and how do we go about it?
Until then I'm sure our company and many companies like us will, as Jeff Hammerbacher was quoted as saying continue "thinking about how to make people click ads" [5].
If you have ideas on how to fix this mess you can find me on Twitter @fwiles or via email frank@revsys.com.
Footnotes
[1] Besides the widely reported outages and bugs the site has experienced it STILL gets a low C rating on simple performance tools such as YSlow. Doesn't take a lot of imagination to think how badly the internals were constructed in terms of performance.
[2] What I mean here is that with most laws and even corporate policies the original intention is usually good. The spirit if you will, but you quickly get mired in details and wording and CYA behavior.
[3] I'd ball park it in the $2-3 million dollar range and I consider that an inflated estimate. $10 million would work out to 475 c3.8xlarge On Demand AWS instances running for the entire year. Which would be about 51k ECUs, 27 TBs of RAM, and 148 TBs of mirrored SSD instance storage! Even if we shift some of those monies around to S3, EBS, and traffic charges it's still a metric ton of horsepower. This works out to be roughly 2 Pinterests or about 4 circa 2011 Instagrams in terms of hardware.
[4] From US News. If we used the actual average you could have a tech staff of 555 people FYI.
[5] The full quote is "The best minds of my generation are thinking about how to make people click ads. That sucks." from a Business Week article. [ CYA]: Cover Your Ass [ECU]: Elastic Compute Units
