When Noel Catrambone joined insurance holding company Kingsway Financial Services Inc.  as CIO in 2012, he faced a rebuilding situation. Originally a Canadian-based company and still traded on the Toronto as well as the New York Stock Exchanges, Kingsway had brought in new senior management after the stock collapsed in 2008. Kingsway moved its center of operations to Itasca, a Chicago suburb, and trimmed a number of unprofitable businesses from its portfolio.

Through all this IT was neglected. Catrambone soon discovered that the last real infrastructure refresh had happened around 2005. Kingsway’s main platforms were eight years old and out of warranty, and spare parts were hard to find. “They literally had a pile of stuff in a corner, and if something died they would go pick through the rubbish to see if they could find a replacement part,” he said. “You can’t run a business like that.”

Catrambone learned that replacement hardware just for Kingsway’s two largest subsidiaries, Mendota Insurance Co. and Assigned Risk Solutions, would cost $800,000 to $1 million. To make matters worse, the existing hardware was scattered in offices across the country. Getting Kingsway’s infrastructure into shape would be a five-year project, “and at the end of five years I’ve got an $800,000 five-year-old infrastructure. That just didn’t make any sense to me,” he said.

The alternative was to move to a public infrastructure-as-a-service (IaaS) platform. Kingsway needed a provider that was large enough to be financially and technically stable, but small enough that it would value Kingsway as an important customer. Catrambone also preferred a company in the midwest, close to Kingsway’s new headquarters.

“A lot of my peers said, ‘Why don’t you just push it all up to [Amazon Web Services] S3?’ I’m not going to do that. If we have a major problem, I want to know that I can contact someone way up the ladder and make sure we are getting the attention we need,” he said.

He chose Chicago-based SingleHop LLC, which filled his requirements perfectly. “They have an established reputation in the marketplace. I need to know our business is extremely important to the people I’m working with, and that’s the definite feeling I get with SingleHop.” And it offered a good price.

Planning the move

Kingsway signed with SingleHop in October, 2013 and spent the next seven months developing a migration strategy. Then they devoted two full months to the networking challenges alone.

Kingsway writes policies in 26 states, each with an insurance commission that needs to be sure that customer data is protected and all operations meet regulatory specifications. It also takes credit card payments online, so the systems have to meet Payment Card Industry (PCI) requirements as well. “We get audited all the time,” Catrambone said. He had to be sure the new systems met all those requirements in full.

The migration was also complicated by the level of integration required among multiple core systems. For example, the claims and underwriting system is integrated with the general ledger and the analytics engines that look for evidence of fraud. There are also connections to external data providers like LexisNexis and Ivans Insurance Solutions and to payment processors like Authorize.Net to be managed. Ensuring that all those interconnections were maintained as systems were moved to SingleHop meant identifying each specific line of code where an IP address had to be changed before the move happened. As always, the devil was in the details. And many of those systems ran on physical servers, so the team had to do a lot of physical-to-virtual conversions at the same time.

The migration also involved moving large quantities of data, too much to send across the Internet. In the end the migration required several multi-terabyte USB drives. They scheduled migrations on weekends. Every Friday night for about eight months they would shut down whatever system they were migrating, copy the VMs and data to the USB drives and either ship them from Kingsway’s Miami or Minnesota offices or drive them from its Elk Grove Village, Ill., office to SingleHop.

On Saturday, the SingleHop team would copy everything to the new environment and test. Then the Kingsway team would change the IP addresses on their end, run their tests, and shut down the old systems.

“I have to say, it’s gone pretty flawlessly,” Catrambone said. The one issue has been connecting to the company’s IBM AS/400, which is the central repository for policy information and other central data. The connection over the network has too much lag time and moving the data is a challenge because of the AS/400’s proprietary format. SingleHop VP of Products Jordan Jacobs said the service provider will probably collocate the physical AS/400 and run it for Kingsway to get it close to the rest of the infrastructure. “We aren’t a colo,” he said, “but we will do that for a client in special circumstances.”

Advice

With the end of this tunnel in sight, Catrambone says planning is the key to a successful migration. “However long the planning stage of your project plan is right now, double it and spend as much time as you can up front really digging into all those very specific details of whatever it is you’re looking to move,” he said. Things like identifying all the interconnections ahead of time so the team knows what IP addresses need to be changed can head off a lot of problems. “We thought we really understood the way things were laid out, but then we moved into the build phase and realized we weren’t ready, so we had to put it on hold and go back to planning.”

Allow plenty of time for testing, too. The network connection is particularly vital, and redundancy is important. “That was the one thing that really made me somewhat nervous,” Catrambone said. “It’s one thing for the Internet connection to go down and have people in the office screaming, but if all your data and systems are off-site and your LAN goes down, people are basically twiddling their thumbs until you get it back.” Automatic fail-over and recovery between the primary and backup network is vital.

In the end, however, the project was worth the effort. Kingsway will be consolidated in one data center with one service contract, which will greatly simplify operations. It no longer has to maintain detailed hardware skills, and the days of searching for parts for obsolete hardware are over.

“Our team can concentrate on supporting the business, aligning the services we provide to the needs of the business,” Catrambone said. “It freed us up to be responsive, which is vital because we do go through quite a few acquisitions each year.” As a holding company, the company’s growth primarily comes from acquiring or start new companies. It even acquired two small insurers during the migration. One has already been moved the SingleHop and the other is in the process.

The IT team also does quite a bit of development and proof-of-concept work, which now can run on virtual machines at SingleHop. That means that instead of buying and installing new hardware, they can spin up some VMs, and when the project is done shut them down.

“Another thing I like is the predictability of budgeting,” Catrambone says. “The IT budget was kind of a disaster when I got here.” Now it is under control, and the CIO’s job has become much more predictable. That’s the way he likes it.

Photo via Pixabay