Real-time streaming data applications may get all the press, but the reality is that much of the business world still runs in batch mode. As companies grow, so does the volume of management reports that must be run overnight. At Raymond James Financial Inc., that growth threatened to swamp the batch processing systems and delay needed management information.

Founded in 1962, RJF now manages over $500 billion of assets and employs 6,500 financial advisors serving 2.7 million client accounts in more than 2,600 locations. As the company has grown organically and through acquisition, the batch processing load has increased as well. “I’ve been here 16 years and over the last five to seven we’ve been growing faster than I’ve ever seen,” said Chris Haynes, RJF’s manager of workload engineering.

RJF’s batch environment had grown complex, thanks to the variety of Linux and Oracle-based platforms that it been added onto its principal batch platform, which is based on systems from the Tandem Computer division of Hewlett-Packard Enterprise Co. The task of processing more than 800,000 batch of each month was taxing the ability of the information technology staff to keep up.

“Our service level agreements continue to get earlier and earlier,” Haynes said. “We’ve had to continue to press the pedal to the metal to finish earlier.”

Until recently, job scheduling was handled by a hand-coded distributed messaging system that assigned and released batch jobs. That not only added points of failure but increased the need for large file transfers between systems. RJF chose the Control-M workload scheduling system from BMC Software Inc. to streamline the process. The software has not only enabled the IT organization to grow its batch load nearly fourfold to 3 million jobs per month, but also improved software testing and enabled it to embrace DevOps development.

A whole new view

Control-M is now the workload engineering team’s primary tool for identifying bottlenecks and fixing problems that can delay batch processing. Control-M manages jobs across a complex web of processors and hundreds of applications that access the company’s data warehouse and consolidated data store.

Lifting and shifting the entire workload management process to a new tool wasn’t a simple process, Haynes said. Testing took six weeks and a BMC partner to help. “It took about a year to get everything moved over because of the risk of running two different schedulers,” he said.

The result has been not only improved efficiency but better insight into dependencies and problem areas. “We’re now able to pick out a job in a stream that sticks out like a sore thumb, whereas before it would have taken hours to identify,” Haynes said. “We not only understand if there is an issue, but we can report it out to the business. Instead of making our SLA [service-level agreeement] deadlines by 10 minutes, we’re ahead by hours.” Audit reports that previously took up to three weeks to prepare are now done in a few hours.

As business users become more savvy, some have used Control-M’s self-service features to track and even schedule their own jobs. Developers use Control-M to schedule their own test runs as they move to a DevOps agility development model. Control-M’s visual interface presents the job stream as a flow diagram, which is a more natural way to view inter-dependencies that the old list format. It also enables the scheduling team to better anticipate congestion during busy batch runs. “The business would rather know something’s going to be a little late ahead of time than find out that morning,” Haynes said.

Adopting workload automation involves a big learning curve. Surprises are inevitable as the automation process exposes workarounds that were buried in existing systems. “Understanding your environment is key,” Haynes said. “We were fortunate in that we had a lot of tribal knowledge. Our people understood the ins and outs of our batch processing.”

Image: Pixabay