Today, President Barack Obama is expected to address the numerous technical issues surrounding the technology-challenged healthcare.gov website which has captured headlines stemming from its continued significant failures since launching almost three weeks ago. Reuters reports that the president plans to call out these “glitches” as “unacceptable” in remarks planned for today. He will also outline how people can sign up for the insurance as the systems are being repaired.

Bringing in the Best and Brightest To Fix All This

On Sunday, the HHS put out a 600 word blog post acknowledging these continued problems, and noted that they have pulled in the “Best and Brightest” from throughout the government and from private industry into the operation to help fix the issues.

“Our team is bringing in some of the best and brightest from both inside and outside government to scrub in with the team and help improve HealthCare.gov.

“We’re also putting in place tools and processes to aggressively monitor and identify parts of HealthCare.gov where individuals are encountering errors or having difficulty using the site, so we can prioritize and fix them.

“We are also defining new test processes to prevent new issues from cropping up as we improve the overall service and deploying fixes to the site during off-peak hours on a regular basis.”

It is an interesting admission of the existence of obvious technical flaws that they spent the better part of the last three weeks pretty much dismissing, attributing the issues to massive traffic at first, then ‘glitches’.

HealthCare.gov Sounds Like It’s Getting Machine Data Analysis

No word has been put out yet on who these people might be on this new team, or what companies they may come from.  The HHS post was also rather lax on details on what repairs have started, but it did state that steps have been made to help improve the functionality of the site including several updates being made through new code that includes bug fixes. These tools and processes they are implementing appear to reference machine data logging type of technologies made famous by companies such as Splunk that will aid them in monitoring and identifying where the largest errors exist and aid them through changes and updates as they strive to improve the operation.

There are a number of flexible solutions in this space and some major use cases including security, application monitoring, and more in countless environments:

“By adopting the Sumo Logic solution, Netflix gains the ability to collect, manage, and derive IT and business insights from all of its machine data, whether it comes from their data centers or from their cloud infrastructure. From emerging technology companies to today’s enterprise leaders, Sumo Logic’s cloud-based service is rapidly becoming the solution of choice to unlock the value of machine-generated Big Data,”

“Considering the national debate around privacy, clearly there is a need for Big Data technology that at once provides advanced analytic functionality and safeguards sensitive data,” commented Wikibon Principal Analyst Jeff Kelly. “Sqrrl Analytics, which leverages Apache Accumulo and Apache Hadoop, makes powerful Big Data analytics more accessible to more end-users thanks to its cell-level security capabilities combined with analytic application development tools.”

Ten-Year Old Technology – and a Minimum 3-Week Shutdown

This sure would have been much better to have in place in advance.  As we discussed a couple of weeks ago, the launch has been a disaster, filled with massive missteps from the beginning.  It seems that the truth is that this is far beyond glitch status as there is so much that has gone wrong with the site.  In USA Today it was reported that:

“the federal health care exchange was built using 10-year-old technology that may require constant fixes and updates for the next six months and the eventual overhaul of the entire system.“

Some tech experts are saying the site will require at least a three week shutdown so that these massive problems can be fixed.  If you look at it, we’re in crunch time with a Dec 15th deadline coming up as written into the law. If you thought the costs so far were expensive, imagine having to gather “the best and brightest” into the fold and in a hurry, with bonuses, long hours and overtime.

Will It Be Fixed in Time?

The site has already been taken down a number of times for maintenance and reports say that the efforts to fix the troubled system have been running 24 by 7.  Leaked emails and reports have hinted at massive software and hardware issues deep within the system.  Officially, there have been no publicly made predictions on how long it will take to repair the system, but be prepared to read about this effort for months.

Tested Less Than One Week Before Launch

In the meantime, a congressional committee is starting to question how this project went so off-track.  No telling where that will end up, but there is so much that has gone wrong, and unfortunately for the American tax payer, not only is this costly, but the program is rife with things to question.  Testing of the site was an early obvious deficiency.  The Washington Examiner reported last week that the site wasn’t tested until the week before it was launched.

“full testing of the site was delayed until four to six days before the fateful Oct. 1 launch of the health care exchanges…

Normally a system this size would need 4-6 months of testing and performance tuning, not 4-6 days,”

Whopper – and not the flame-broiled variety.

CGI Federal – How Did They Get This Contract Anyway?

There are also many questions about the architects.  The Washington Post described how CGI Federal, the federal contractor that helped build the site was pretty new to US government contract work. The company had also reportedly failed to deliver on a number of Canadian healthcare projects and were thus fired within the last year.

Mistakes happen, contractors sometimes don’t live up to expectations, right?  We can just learn from that and just go with another contractor next time.  Well, those things are usually true but it seems that evidence shows the contract to build the site was awarded to CGI Federal without any competitive bids at all. This information was found on the site USAspending.gov, which officially tracks federal spending.  CGI is listed as the only bidder on the design award, and this is also reinforced on the FFATA Sub-award Reporting System – a site that specifically tracks these types of contracts.  Yikes.  One bidder?

Rather than open the contracting process to a competitive public solicitation with multiple bidders, officials in the Department of Health and Human Services’ Centers for Medicare and Medicaid accepted a sole bidder, CGI Federal, the U.S. subsidiary of a Canadian company with an uneven record of IT pricing and contract performance.

CMS officials are tight-lipped about why CGI was chosen or how it happened. They also refuse to say if other firms competed with CGI, or if there was ever a public solicitation for building Healthcare.gov, the backbone of Obamacare’s problem-plagued web portal.

Instead, it appears they used what amounts to a federal procurement system loophole to award the work to the Canadian firm.

This last item, well it doesn’t seem quite right.  It turns out that CGI Federal, the only bidder on a multi-million dollar website contract had a unique level of access at the White House before the contract was awarded.

Prior to the official award, senior CGI executives met with top White House officials and attended a number of invitation-only addresses by President Obama.

According to the logs, CGI Federal President Donna Ryan visited the White House six times prior to her company being selected to do the IT design work behind the high-profile website.

Perhaps that is just a coincidence or even a series glorified sales meetings.  We may never know, but the conditions under which so much was handed over to this non-US, under-performing, and lightly experienced contractor has to be analyzed.  Not to pick on CGI Federal, but there was a history here.  I have been on projects that have gone wrong, sometimes the problem is management, sometimes it’s the technology, and sometimes the whole thing is just put together wrong.  That’s another thing that the public may never find out, but it has left the rest of the country to pick up the pieces.  This should have been done better.

It Would Be a Significant IT Event IF They Can Turn This Site Around

The task to patch up, fix, and overhaul the system is a big one – and expensive without a doubt. Add to that the special challenges of having the site running while this operation is being executed. Then throw a deadline on top of that. The circumstance is pretty major and to be looking at repairing these issues three weeks in, well there’s not a lot of time to turn this around and there is a great threat to the fiscal and operational health of the insurance initiative itself. If indeed the yet-to-be identified “best and brightest” do indeed turn this operation around it would be a remarkable comeback. We would then be witnessing the most rapid ROI and turnaround case study in the history of the web for such technologies such as log analysis, storage technologies, cloud, and that list goes on and on with a system that is undoubtedly so complex, large, and troubled. Thus far, we have no official word on what technical problems are actually the worst, it took them this long to even publicly and softly admit there was an actual technical problem. We may never see that case study, especially if the turnaround doesn’t succeed in time.