The OSDU™ data platform has garnered lots of attention as the new standardized data platform of the energy industry. Much of this attention has focused on the benefits that a standardized platform can bring, such as interoperability and increased data-sharing through breaking down data silos, but it’s important to expand beyond the idea of the OSDU data platform as a platform designed for data managers and IT professionals.
While the OSDU data platform can indeed reduce the friction of interoperability and make data easier to find, adopters should look beyond data platform capabilities and shine their light on the power of the OSDU data platform on AWS and how a data platform in the AWS cloud ecosystem creates new opportunities to focus on the end users, including geoscientists, seismologists, geophysicists, and more.
Overcoming OSDU challenges
An OSDU data platform implementation holds many benefits, but it also presents challenges. In particular, the organization’s application portfolio will likely require adaptation to leverage the OSDU data platform, and data ingress and egress procedures will pose difficulties due to both the sheer quantities of data and the varying degrees of cloud adoption maturity in user companies. Migration from existing data platforms can also present complications, and implementation of the OSDU data platform will have positive impacts that weave through an organization’s security, taxonomy, governance, partitioning, and data sovereignty and residency needs. However, as a part of exploring the OSDU data platform on AWS, you can also explore the Well-Architected Framework to guide you through your AWS cloud journey.
For the most part, however, the above hurdles are near-term challenges often found in the initial setup of a platform and the obligatory period where members of an organization get acquainted with its use and capabilities. What will likely prove more difficult (and more relevant) to the OSDU data platform’s utility is the need to approach a technological solution from the end instead of the beginning — that is, by identifying the intended business outcomes from the outset and letting those value-adds use cases to drive the implementation process. AWS calls this process “working backward.”
Focusing on business outcomes ahead of technical capabilities often requires a fundamental shift in an organization and the IT and data management departments, as exciting potential is ignored in favor of user needs. While it may prove difficult, however, such a shift in the implementation strategy offers the following three compelling benefits.
Force a right-to-left approach
By implementing the OSDU data platform in the context of end users’ needs in partnership with the implementation envisioned by IT and the data management organization, the platform meets end-user requirements, enhances their workflows, and produces business value. You must resist the temptation to see the data platform as a solution whose implementation details are encapsulated away from the end users. Engage your end users to understand their workflows and business needs, letting those desired outcomes drive how you implement the OSDU data platform within your organization. As a part of this exercise, engage your end users in the art of the possible: What is it that they can’t do today that they wish they could? This exercise will also let you map how the many AWS services, specifically in data science (e.g., SageMaker, Textract, and Comprehend) can change the relationship between the end users and the data that will live in the OSDU data platform on AWS.
Prevent wasted resources
Without business outcomes in mind, it’s easy to give in to the temptation of workstreams that increase the data robustness of an OSDU data platform without a real purpose. These instances ultimately waste both time and money by creating unnecessary work to produce unnecessary data. As stated in Benefit 1, work backward from the business outcomes to create the data products your OSDU customers need. Those data products then drive how you build your ingestion pipelines and implement your ELT strategy (noting that OSDU best practices favor Extract, Load, and then Transform versus Extract, Transform, and Load; the why is better left for another discussion). One of the key pillars of an AWS Well-Architected solution is cost optimization. Don’t just think about cost optimization in terms of services leveraged or storage consumed. Apply these same AWS principles to how users will interact with the data and ensure the AWS OSDU data platform is serving up the data products end users need most to efficiently and effectively perform their function.
Deepen the partnership between IT and the end user
Business users often have new technological innovations thrust upon them, which can be disruptive and put adoption at risk. For the OSDU data platform to be successful, users must want to use it, which means their needs have to drive every aspect of the implementation. This is an age-old problem, but one we often forget when pursuing a large-scale IT or Data Management project. While many end users may only interact with the OSDU data platform through the applications they use every day, engaging them helps to create the partnership that enables Benefit 1. Leverage this partnership as a continuous feedback loop to ensure the “art of the possible” explored in Benefit 1 becomes a never-ending loop of learning.
Starting at the end
It’s important that we never lose sight of the OSDU data platform’s intended purpose — to make data easier for end users. Instead of getting caught up in all the exciting capabilities of the platform and forgetting the customers, we must look at it through their eyes: What are their primary pain points? How are their workflows evolving? How do we position the OSDU data platform within the AWS cloud ecosystem to best meet the needs of the business? What follows are two anecdotes and an approach I recommend in order to engage in right-to-left thinking to ensure you start with the end user’s needs.
I recently worked with a global energy company that wanted to move the tedious process of document collaboration to a solution on AWS. When we started with their thorough standards and process documentation and built an initial workflow, we noticed that many of the complexities no longer made sense in the new digital context. Upon going back to the team and digging deeper into what they were trying to achieve, we were able to dramatically reduce the number of steps required. Rather than simply lift and shift the process to a digital context, we started with the needs of the business and redesigned the process to better exist in an AWS digital world with innovative services and capabilities. Taking the time to rethink the job to be done allowed us to simplify their workflow and reduce the time it took to complete their tasks.
A short note on jobs to be done: Over roughly the past 20 years, American economist Clayton Christensen has introduced what’s now known as the jobs-to-be-done concept, which theorizes that the purpose of a product is to help a customer complete a certain job. The better it does — and the more jobs that particular product can help a customer accomplish — the more valuable that product is. Using this framework enabled us to identify areas to automate, allowing the end users to focus on higher-value tasks.
I worked with another client whose IT organization spent a decade building a data management platform for the business — a platform that was seen as a role model for other IT and data management teams in the industry. The company was exploring commercial offerings to replace the platform and hired me to evaluate it. As I met with the many business groups who were the intended customers, I learned most either didn’t know it existed or no longer trusted it due to stale data caused by twice-weekly data refresh times (a problem easily resolved if the IT group had spoken to the end users sooner). At the end of the day, the company had spent millions on a top-notch, in-house developed platform that never identified the job to be done, and the business never used it.
Ultimately, instead of requiring users to “pull” the necessary data out of the system, we should combine the OSDU data platform with AWS’ vast cloud ecosystem of services that makes it a proactive data-serving solution. When we design systems that anticipate user needs and “push” newly available data to the user, we’ve produced a far more useful and powerful implementation that leaves a deeper impact on the organization taking advantage of it. This is the kind of utility AWS and the OSDU data platform together can create for energy companies — provided they follow an approach designed to produce it.
