< Back to Data Examiner

How To Attain "The Prize" with Big Data Management

Article Written By Andy Reinert, Stonebridge Consulting

Like oil, data is a commodity with tangible value. Yet the digital oilfield is like the wild west where energy professionals spend an inordinate portion of their day on data wrangling because of a chaotic mix of structured and unstructured data, multiple versions of the truth, and lack of data governance. Every department in the energy enterprise is contending with increasing volumes, variety, and velocity of data, leading to analysis paralysis, delayed decision-making, and unnecessary rework (e.g., prior period adjustments). This “big data” dilemma often leaves important questions difficult to answer quickly, including:
  • Do geoscience, drilling, and completion teams have the most accurate datasets to optimize well placement and performance?
  • Are mineral and royalty owners getting paid based on accurate volumes and interest decimals?
  • Do field and production teams have clear situational awareness to effectively manage assets following a merger or acquisition?
  • Is compliance providing the most accurate production and environmental reporting to state and federal agencies?
  • Are land, production operations, accounting, and regulatory making decisions with the current/correct well status?

For organizations who do it well, data management provides a competitive edge in an increasingly digital oilfield that accelerates business performance and sets every department up for success. But teams are all too often so busy managing all the moving parts of data management (oh, and not to mention the business of finding, extracting, and moving hydrocarbons) that they take their eye off of “the prize,” i.e., the payoff after you have put everything into place to sustain successful data management.

In terms of attaining the prize, there are many business goals data management should achieve for workflows across the energy enterprise, including land, drilling and completions, geoscience, production management, field operations, and even human resources. Think about the impact of poor data management on an E&Ps cash register, from allocations and well tests to sales and financial statements. No matter the price of WTI or Henry Hub, even a little bad accounting data or rounding error can hurt the bottom line.

For example, inconsistent use of data standards introduces the possibility of missing an obligation to interest owners if the land department uses API-12 while the drilling department uses API-14, preventing accurate tracking of a wellbore trajectory or new sidetrack on a lease. Or consider a midstream example where fixed asset accounting depends on high quality location data for an interstate pipeline. Lat/long for preliminary right of way, permitted, and as-built designs can vary and with hundreds of miles to account for across dozens of city, county, and state lines, teams struggle to stay compliant. These examples share a similar risk where even a small discrepancy to asset location can have massive financial implications, underpayment to interest owners in the first and underpayment to state and municipal taxing authorities in the latter, all hinging on effective data management.

Then there are new prizes (or burdens, depending on how you look at it) as regulatory and compliance obligations expand. Companies must manage an increasingly complex variety of environmental data for accurate HSE reporting (e.g., audio, visual, olfactory inspection of oilfield assets/AVO). The “E” in ESG is only making the need for effective environmental data management more urgent as greenhouse gas reporting increases in scope and complexity. Part of the recent Inflation Reduction Act, oil & gas companies now face a methane fee of $900 per metric ton starting in 2024 (increasing to $1,500 through 2026), making it absolutely imperative to master regulatory data management to avoid over- or underpaying the government.

Prizes of the future include predictive analytics, machine learning, and artificial intelligence. The oil & gas industry has already seen some success in areas like artificial lift optimization and drilling automation, but many PA, ML, and AI initiatives stall out or plateau because they need big data to thrive. Gartner defines big data like this:

Big data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.


The digital oilfield of the future runs on big data, underscoring the need for effective “big data management.” It’s a 2 part problem. First, you need robust tech and energy industry-specific capabilities. And second, your tech stack must be able to accommodate accelerating volumes, variety, and velocity of oilfield data now streaming from smart connected devices, SCADA, and the Internet of Things (IoT). It’s about solving big data management as well as big data consumption, bringing digital oilfield data sources together, orchestrating it, then pushing big data sets to machine learning and AI applications.

Data management maturity, technology, and standards must keep pace, expanding the scope to include the data coming in from SCADA and IoT systems to achieve the nirvana of AI/ML. and automation. Big data management unlocks a new world of data science. The implications are staggering, from reservoir simulation, well planning, marrying completions to the rock, economic forecasts, enhanced oil recovery, and HSE. The benefits of predictive analytics and ML also extend into midstream, bringing a new level of capability to optimize operations and minimize costs, power transactions, and maximize spreads.

In the digital oilfield, data management is the most important skillset that can make or break an organization. The strategies you choose, technology you deploy, and data managers you hire are the determining factors that will hold organizations back or set them up to succeed and grow.