Patch Management for Clustered Servers
The Company
The company provides financial services and runs multiple trading applications on 35 discrete clusters. They need to ensure that each cluster is patched to the correct level and each server has identical versions of the trading applications.
The Problem
Their server clusters are intended to provide failover coverage. Typically in pairs, the inactive or offline server is patched in advance of the active server supporting the business. The company has experienced application outages caused by software and infrastructure differences in the cluster.
There are two fundamental challenges faced by the customer. One is identifying which patch versions are deployed on which servers and the second is ensuring the remaining software on the clustered servers is identical.
Patch Deployment
Patches are immediately tested upon release and then are promoted to the inactive and active servers at different times pending successful acceptance. Typically the entire process takes 4-6 weeks, and often means that multiple patches are in process. Because of the process complexity and the large number of server clusters, it is often difficult to know what patches have been applied where. A manual process is required to verify the state of any given machine prior to taking any actions.
Cluster Software Synchronization
There are fewer application and middleware releases than patches, however the company has had difficulty ensuring the correct versions are applied consistently across each cluster. Software updates use the same process described above meaning software is deployed to servers at different times and can get out of synch.
The Solution
The company decided to use Enterprise Trust Server™ (ETS) to provide better visibility into what software is deployed where across the clusters and improving their control.
Since the ETS is populated with historical patches and has access to newly released patches, it can be used to identify what patches are deployed where across the entire set of clusters. The SignaCert client was rolled out to each server in the cluster. The client is a lightweight non-persistent application that scans the files on disk, computes a cryptographic hash and sends an XML document to the ETS appliance for comparison with software signatures stored in the ETS. Scans are performed at the completion of any software deployment process to determine if the correct patch was deployed completely. If any deviations are found, the results identify any added or removed files and what product they belong to or if any of the files have been modified. This helps them understand explicitly what they need to correct.
To prove that the remaining software in the cluster is identical, the customer creates a snapshot of one server in the cluster. The snapshot is captured using the SignaCert client and published to the ETS appliance for comparison with the other servers. This snapshot includes all software except patches which are verified using the processes above. The other servers in the cluster are scanned and compared with the snapshot for the first server to identify any difference. If there are deviations, the results will provide a detailed list of what is different for remediation.
Results
The company now has complete assurance that their clusters are always patched correctly and that the paired devices have identical applications. This has resulted in the following benefits:
- Process Assurance—Automated verification processes allow them to know that their IT processed produced the expected results.
- Faster diagnoses—When the processes break, verification identifies what is different requiring remediation.
- Reliability—Improved uptime and availability for critical applications.