#133 Lessons in HPC Oil & Gas
Subscribe to get the latest
on Wed Apr 12 2023 17:00:00 GMT-0700 (Pacific Daylight Time)
with Darren W Pulsipher, Keith Gray,
In this episode, Darren interviews Keith Gray, a former director of high-performance computing at British Petroleum. With over 30 years of managing HPC centers, Keith gives great insight into the challenges, best practices, and the future of high-performance computing.
Keywords
#compute #hpc #people #innovation #cloud
Keith had worked in the industry for his entire career and joined Intel last July. He had led Consolidated Resources’ High-Performance Computing group since 1999, where they started with only one-tenth of a teraflop of computing power. However, Keith successfully consolidated multiple sites and established their team as integral to BP’s oil and gas exploration, development, and production operations. Over the last three decades of working in oil and gas high-performance computing, Keith has had three factors influencing the direction of HPC: cloud computing, a shortage of human talent, and the balance between collaboration and first-mover advantage.
Cloud or On-Prem
Keith shared his experience building an HPC (High-Performance Computing) facility for BP. The construction took almost nine years, with the first seven spent on patching the existing facility and building a business case for a dedicated one. Despite the rise of cloud technology, they found running their facility cost-effective because of control over resources and bespoke architecture. Being in the oil and gas industry presented unique challenges in dealing with massive amounts of data, but having a dedicated facility allowed them to focus on BP’s research needs. These factors are also relevant for other industries dealing with HPC clusters.
The big question for anyone running an HPC cluster is whether or not HPC workloads can be run successfully on general cloud services. That depends on the business’s specific needs and the workload scale in question. While some companies may find it practical to collaborate with the cloud for spikes in demand, others require a large, fully utilized cluster to meet their needs. Factors like predictive performance, data gravity, data size, and cost of operations are essential factors in HPC workload placement.
Fighting the Resource Shortage
Acquiring and retaining talent in high-performance computing is challenging, particularly as the baby boomer generation retires. Keith suggests that building relationships with universities and identifying talented individuals early is critical to developing the necessary skills within an organization. Organizations must think long-term about creating successful HPC teams and building through mentorships. In these heterogeneous groups, scientists, system engineers, mathematicians, and computer engineers work together for years instead of months.
Building long-term commitments by investing in internships, pipelines, and fellowships is critical to creating a career-oriented work environment. Keith shares a challenge they faced in creating a shared vision for their team’s future but found success in building a network of contacts outside of BP to help them achieve their goals.
First Mover Advantage, Sharing
Another critical aspect of HPC in Oil in Gas is to have the first-mover advantage. Finding a new technique to find new energy sources can save the company billions of dollars. However, sharing information with competitors can also lead to discoveries that help the industry. This is an important balance that HPC professionals need to understand to get the most out of their work.
Keith shares a fun story of how their friends helped develop innovative technology to understand better the Gulf of Mexico subsurface geologic structures to find oil and gas. Despite the competition watching and questioning their methods, BP’s experimentation proved successful. Keith explains that everyone is watching when you do something new in the field.
For example, they often contracted with companies like PGS to acquire data and the innovative solution of using helicopters and hard drives to deliver newly acquired data for processing. The results were impressive, showing better results than traditional methods. When asked about the future of HPC in the industry, Keith believes that collaboration will continue to be necessary, whether through shared clusters or collaborating on foundational science benchmarks. Despite the challenges, the HPC industry is exciting, and there are great opportunities for people to bring their computational skills and willingness to solve challenging problems.