Various news articles are reporting that at least 55 contractors were involved in building healthcare.gov (example from USA Today). That raises the question of why anyone thought that all of the interfaces among 55 different components were going to work upon launch. But I wonder whether, before hiring 55 different companies, someone didn’t ask “Could we just take the source code for the up-and-running Massachusetts exchange and extend it?”
What does healthcare.gov do that the Massachusetts exchange, which has been running for at least five years, does not do? Wouldn’t that have been the lower risk approach? Add a “state” column to every database table and form, soft launch with 10,000 consumers in each state, and then release to the general public?
Obviously it is easier to say in hindsight what should have been done, but given the existence of a working exchange and the fact that federal workers themselves did not do any programming, why did the Feds fall victim to “Not Invented Here” syndrome?
…and it’s not too late to do that now!
I’ve built software for the government — specifically, for CMS — before and my experience was that the bid, specification, and build processes were more about the government trying to manage their relationships with the contractors they like to use (all of whom bid against each other for the same work) than it was about making solid decisions about stable and scalable software.
That’s because apparently nobody on this project read the Mythical Man-Month nor it seems any text on best practices within the last decade. You can see this with the dumb “tech surge” idea. Putting more people on a late project will only make it later.
Why does anybody fall in the NIH syndrome? Because mediocre coders don’t like to read and learn other people’s code, thinking they can do it better themselves.
From the government’s point of view, software is a gas, and expands to fill the budget given to it. Some campaign contributor said it would cost X, that got written into the bill, and here we are. When dealing with government, which has, essentially, limitless funds, why in the world would it EVER be a question that a firm would NOT just build it from scratch?
When Hilary was trying to “do” health care, I told everyone not to worry, because no one could build a computer system big enough to run it. Between this, and the UK’s recently-scuttled £2B system, I’m still not wrong. But, just like over there, we’ll keep the same contracting firm, and give them even MORE money to “fix” it, and it will drag on forever.
Phil, Phil, Phil. So naïve. Everyone who’s anyone in IT knows you need a requirements analysis + architectural process before you write a single line of code. How on earth will you know you’ve built the right thing without pages and pages of perfectly worded specification, reviewed by top IT management and government officials? These people didn’t become who they are because they want to take home a pay cheque, Phil. They’re the real smarts in every government IT project. Open your eyes!
_Your_ approach runs the risk of lowly _engineers_ making decisions about how the system should work. And anyone with half an eyeball knows you need to manage those guys, otherwise they’ll start hanging all sorts of technical baubles on your healthcare tree, putting in their preferred toy technology, all the while missing the big picture. What makes you think they’re interested in making a _working_ system? They’re only in it for the free lunches and Counterstrike parties on the office LAN. They don’t understand the gravity of the problem we face here. That’s _why_ you need senior executives, otherwise you’ll end up with some unproductive hippie commune like Xerox PARC. Nobody bought their stuff anyway. Be realistic!
Anyway, a good architectural team can design for thousands of requirements and integrate hundreds of disparate components, _even_ if they don’t conform to the same interface yet. You can integrate anything if you use another layer of indirection. It’s the adaptor pattern, stupid — design patterns 101. And if you have 55 different components, you can scale up development 55x faster. It’s the industrialization of software, Phil; it’s called scientific management. You don’t want _unscientific_ management, like those weird systems theory guys? They’re just academics. Get with it!
Besides, everyone knows that relational databases are dead. Do you really expect to build a nation-scale system on those 1970s technologies? They are obsolete. They don’t scale linearly or run in the cloud. Not to mention their total obsession with consistency: everyone knows that the truth has terrible performance characteristics, Phil. That logical stuff is Ancient Greek technology. Pay attention!
Maybe in your day a whole system could be understandable by one guy and serve hundreds of thousands of people. But those weren’t _infrastructure_, Phil. They were toys like shopping websites, operating systems, network protocols… this is _healthcare_. Do you want all those people to die because of your short-sighted SQL schema? And what makes you think that only one guy should be able to build it anyway? What if that guy gets hit by a bus? It’s fundamentally _good_ that modern computing technologies require more people to use them. It’s called job creation. Wake up!
I must say I am really disappointed to hear all this common sense mumbo-jumbo from you. I think all that time being hyper-productive on those boondoggle Lisp machines has made you a selfish lone-wolf type, and blinded you to the reality of the complex world we live in. All this glorious extra work is keeping the nation busy and out of trouble, and keeping the economy ticking over. Why would you want people _not_ to work?
All this work will set us free, Phil. Arbeit macht frei.
I don’t know, but I’m guessing that managing health care for millions of Americans is going to be even more challenging than building a website. Are we sure this is the group we want to entrust that to?
It seems to me the common thread in all the project management infrastructure disasters the U.S. government is involved in is undermanned or incompetent government staff. You can’t outsource everything – at some point you need a government employee who is in charge of writing the checks.
Despite the “any idiot can do this” attitude many people seem to share, this is a difficult job requiring smart people with ample staff support. But given our begrudging attitude of supporting people who do this work we get underqualified and unsupported people on the government end.
But what smart person would want such a job? You become a $130K a year civil service employee while your peers make far more in the private sector and you have to put up with people who expect that nobody in the government should make over $100k. The revolving door to private contracting begins to look very attractive.
We need more government administrators who make 300K a year. And if they are good at their jobs we need to thank them rather than castigate them.
Brian: Your idea that the “peers” of government workers are paid more in the private sector is the opposite conclusion to the analyses referenced in
http://www.downsizinggovernment.org/overpaid-federal-workers
What data do you have to support your idea that government workers are paid less than peers in the private sector?
(Some of the top administrators, e.g., cabinet secretaries, are plainly paid less in the short term but they tend to make millions of dollars from speeches and lobbying after leaving the government so if you smooth out their compensation they actually are paid very handsomely for their time in government.)
Its the lack of talent among top administrators that seems to be the problem. What financial or cultural changes would be required to make the door revolve the other way? You get training, experience, and money in the private sector then join the government to do something of consequence for the greater good. What would it take to get somebody like you, talented, successful, technically competent, and semi-retired, in charge of a government program?
In the matter of being completely unprepared – two interesting related stories to note, last night I had dinner with a reporter friend from the NY TImes – who made two comments, first healthcare.gov had come to them to questions some of the Times’ “language / programming” which seemed odd to them and then second point is the testing before launch was apparently less than a month. Little scary if that’s accurate – whole US healthcare website tested and launched in 30 days… Well, we’ll see what comes of it – still effectively contractor “nepotism”?
Brian: I’m flattered that you think I would have done better, but really I don’t think this is about the money. Google is able to hire much better programmers than average and they often pay less than the market rates. Productive people enjoy working in a productive organization.
There have been groups of programmers within the government who have produced high quality software. I think that it is more about ownership and a commitment to users than about the money. You can’t pay someone to care about other people (i.e., users).
Incredibly well said: “You can’t pay someone to care about other people (i.e., users).” – besides “ownership” of the organization / site – I’m curious to think if there’s some relationship in the quality of work – ie. working for google an entrepreneurial for profit organization versus relatively nameless faceless govt. Thinking about it another way google (and most for profit entities) create a corporate culture and a public image which gives employees pride of ownership for work done what is the “corporate culture” of govt?
My own musing / food for thought
AndyD: I’ve seen this with students at MIT in the software engineering project course that Hal Abelson and I designed. The student teams that did the best work were the ones that were most passionate about the users, not the ones with the most software engineering experience. Nearly all MIT students (except for me!) are smart and capable but some teams produced spectacularly bad systems and just didn’t understand why their applications needed to be as usable as Google, Yahoo!, or Amazon, albeit not as feature-rich.
“The student teams that did the best work were the ones that were most passionate about the users, not the ones with the most software engineering experience.”
This should be engraved above the entrance to the Stata Center so that everyone entering the building – students and faculty alike – can ponder what they’re really there for.
Such a thought could have been helpful in recent years – not just around healthcare.gov but in Silicon Valley – for example, saving Google the expense and embarrassment that was Google Wave. (More on that, if I may beat my drum, in “Customers Included,” the book I just wrote on the very topic of why it’s important not to completely ignore one’s users.)
Phil, I had to jump in on this one… I have been involved with several large, name brand sites over the years. The larger the organization, the harder it is to get anything done, an in several of these companies I literally had to break rules that *could* have gotten me fired to actually launch sections that were useful to people as well as search engines and indexes. The government should have looked at the MA site, but from a “political” power POV, that would have shifted from getting the job done ASAP with a positive ROI away from “protecting my little castle.” I may have a more grim view on it, but, to me, it looked like a bunch of people with no clue directing how to build it while keeping as many other political camps/rivals out of the potential glory.
The government should have looked at the MA site? Really?
Well, I’ve been using it for the past 2 years, and it is poor quality at best. The thing that irked me is a login page with two enter buttons. I had to go to the help desk to reset my password about four times before I realized the enter key was sending the key press to the other enter button. That defect was never fixed. It’s very presence showed serious project management problems–any defect at all could be hiding in a site that somehow goes live with a form having two enter buttons–because two enter buttons proves it is idiots in charge.
Anyway, the new MA site is so bad that as of today (11/12/13) almost no one in MA (where 160,000 Romneycare subscribers–including me–are poised to lose their insurance) has succeeded is signing up for the new insurance which has a steep jump in monthly rate, coinsurance, and deductible.
http://bostonherald.com/news_opinion/local_politics/2013/11/just_1_percent_of_bay_staters_facing_canceled_health_plans_have
zz: That’s the best answer by far! That Herald article was interesting. I didn’t realize that the Massachusetts site had needed to be redone for Obamacare or that it had been built by the same contractor as the federal site ($62 million extra, which is a lot less than California paid but on the other hand this “CGI Group” in Canada might end up being one of the largest companies by revenue in that country, running purely on American incompetence!).
http://science.slashdot.org/story/13/11/12/1339215/how-3-young-coders-built-a-better-portal-to-healthcaregov