IT Infra and Virtualization Services for Financial Services Industry

Client Information

The client is a global leader in facilitating e-commerce across 6.5 million locations and over 3,750 financial institutions in more than 90 countries. The client and its associates are specialized in servicing businesses of all sizes including start-ups, and amount to more than USD 2 million transactions per year.

Client Requirements

As per the client’s requirement, Veritis team of experts offered managed IT Infrastructure and Virtualization consulting services.

This project involved building, security, patching and updating of countless LINUX, AIX and attached subsystems. These subsystems support existing systems and are responsible for leading technical aspects of the project in terms of building new systems and also provide technical review for the go-live checklists across UNIX servers.

The client’s UNIX server team required the assistance of personnel with special skill-set in UNIX server administration, automation scripting skills in Shell or Python languages and experience in handling the project, which would prove critical and sensitive to financial transactions provided by our experts.

The project prospects involved:

• Ensuring adherence to established technology standards and consulting with senior leadership to deploy technology solutions
• Supporting existing systems and being responsible for leading technical aspects of the project
• Installing and supporting Red Hat, CentOS, Oracle Linux and/or IBM AIX operating systems
• Performing additional job duties as required
• Leading complex projects and issues within the functional technical area and cross-functional area
• Working independently with IT functional units and business units
• Recommending strategies for technical aspects of projects as well as broad system improvements, with a focus on AIX and Linux Systems

Challenges Faced by the Client

1) Manual-server Configuration Validation

The client’s server administration team manually validated server configurations to ensure compliance with standards, which involved checking over 40,000 servers. This labor-intensive approach was inefficient and prone to errors, leading to operations delays.

2) Complexity in Managing a Multi-platform Environment

The client’s infrastructure included a mix of Red Hat, CentOS, Oracle Linux, and IBM AIX servers in physical and virtual environments. Managing this complex multi-platform setup required specialized expertise and a streamlined approach to ensure smooth operations and minimize downtime.

3) Patching and System Maintenance

The client faced challenges with timely patching, installing updates, and maintaining hardware and software across their infrastructure. These manual processes often caused production issues, impacting system availability and performance.

4) Data Security and System Monitoring

Ensuring optimal system performance and data security across a global infrastructure was a critical challenge. The client required continuous monitoring, quick resolution of hardware and software issues, and proactive measures to secure sensitive financial data.

Solutions Provided by Veritis

1) Automation of Server Configuration Validation

Veritis automated the server configuration validation process using bash and KSH scripting, eliminating the need for manual intervention. This automation ensured all 40,000 servers were regularly validated for compliance, reducing errors and streamlining operations.

2) Efficient Management of Multi-Platform Infrastructure

Veritis provided comprehensive IT infrastructure support, managing Red Hat, CentOS, Oracle Linux, and IBM AIX servers across physical and virtual environments. The team utilized tools like VMware vSphere, Saltstack, and GIT to ensure seamless operation and cross-platform efficiency.

3) Automated Patching and System Updates

Veritis automated the patching and system maintenance process using advanced configuration management tools. The team developed new procedures and standards for server builds, patching, and documentation, significantly reducing production issues and improving system performance.

4) Enhanced Security and System Monitoring

Veritis implemented robust monitoring tools like Nagios and proactive security measures to ensure optimal performance of the client’s UNIX systems. The team performed regular health checks, system backups, and security audits to prevent outages and protect critical financial data from potential threats.

Veritis Approach

Veritis team of experts assisted the client with IT Infrastructure and Virtualization services meeting the right technology standards. The team eliminated the traditional manual practice of validating server configurations by automating the process and created new standards, procedures and processes for builds and documents among other infrastructure-related tasks.

Key Responsibilities Held

• Installed, configured and performed day-to-day administration to solve system-related issues and user assistance activities on Red Hat Linux 4.x, 5.x, 6.x, 7.x, Open SuSE Linux 9.x, 10.x, 11.x. 12.x, CentOS Linux 6.x, 7.x, Oracle Linux 6.x, 7.x, IBM AIX 6.1/7.1 servers in both physical and virtual system environments
• Created new standards, processes and procedures for builds and documents. Provided documentation for new builds to be created in the environment
• Performed administration tasks like setup, maintenance and management of user accounts and groups, and allocated quotas to users as well as system startup and shutdown
• Installed patches, packages related to hardware, vendor software and installed third-party software for system customization
• Monitored the performance of the UNIX systems using PS, prstat, vmstat, netstat, iostat, and RPCinfo to check the CPU utilization, bottlenecks of I/O devices, memory usage and network traffic
• Provided drill down reporting for application teams to be used in monitoring client’s application resource usage/performance
• Worked closely with database administration staff to ensure optimal performance of databases and maintained development applications and databases
• Monitored system resources, logs, disk usage, scheduled and restored backups using Nagios and VERITAS Netbackup
• Diagnosed, troubleshoot and resolved hardware, software and system problems. Quickly arranged for repair in occasion of hardware failure
• Configured High Availability (HA) solutions to include multiple cluster configurations and support
• Implemented network security measures to protect data, software and hardware
• Worked closely with Hardware vendor (IBM, HP, Dell, and Cisco) support to detect and resolve any issues related to the environment
• Performed day-to-day health check and resolved any errors during the health check
• Maintained logs related to system functions, as well as maintained and repaired records
• Assisted in critical outage situations to isolate the root cause, put a temporary fix in place, and recommended/implemented long-term corrective action
• Network troubleshoot using NDD, traceroute, netstat, ifconfig and snoop, among others
• Administered, maintained and troubleshoot EMC Clariion arrays by creating volumes, LUNs and implementing different RAID levels
• Implemented enterprise storage solutions SAN (created VGs and LVs on SAN disks), NAS (mounted and shared NAS devices for servers) and Logical Volume Management (LVM)
• Configured and scheduled various automated jobs for backups, database, and proprietary jobs using CRON, Autosys and BMC Control-M and troubleshoot daily job problems
• Performed system backup and recovery from tapes using ufsdump, ufsrestore, tar and CPIO
• Performed restores from the tape backups and recovered database from hot and cold backups
• Wrote UNIX/Linux shell scripts to automate tasks (SH, bash, KSH, AWK and SED)
• Performed UNIX server build CLIP’s, work build and production queue IR’s and provided technical review for the go-live checklists across UNIX servers
• Held regular meetings to review usage as well as create, revise, and report any new measurements required to manage mainframe or distributed environments
• Recommends strategies for technical aspects of projects as well as broad system improvements, with a focus on AIX and Linux Systems

Environments Used

Redhat LINUX, CentOS LINUX, Oracle LINUX, IBM AIX, HP Server Automation, VMware vSphere, Saltstack, GIT, BlueCat IP Address Management, BMC Remedy, Cisco switches.

Value Delivered

The client was planning to replace all the Red-Hat Enterprise Linux servers with Oracle Enterprise Linus OS and were going to approach external consultants for the same. Veritis team informed the client that they have previous experience for the same and chalked out a plan with the client’s staff for execution. The installation was successfully completed and the client spent USD 55-60k less than what they would spend on external consultants.