What is a Data Platform? It is a central integrated persistence data store designed to distribute data required by strategic business initiatives. A Data Platform is used to standardize fragmented data across multiple data sources, provide capability to track a data element’s behavior across all channels and to enable intelligent data driven decisions based on facts. A good Data Platform is no longer a nice to have; it is a necessity. By centralizing the data required for business initiatives, organizations eliminate point-to-point integrations that unnecessarily run up IT costs and cause misinformation. While the case for a Data Platform is well acknowledged and adopted by organizations, successful Data Platforms are hard to find.
This diagram captures the crucial pieces of a Data Platform. When all the components in the diagram are stood up with the appropriate process and methodology, data can be leveraged as an asset to get insights and to deliver business initiatives. However, after investing millions into building a Data Platform, many organizations find the return on investment is not there. In this article, we will examine some key factors causing Data Platforms to fail.
Lack of accessible metadata
A data catalog is an organized inventory of data assets in the organization. It uses technical and business metadata to help organizations manage their data. It helps business, IT and data professionals to get answers to these questions which, in return, enables data governance:
-
What data is available?
-
Which systems have the data required?
-
How does the data flow through the enterprise?
-
How to get access to the data?
-
What transformations are applied to the data between source and common data model?
Accessible and reliable metadata is the gateway to the Data Platform because it showcases all the data assets that are available to be leveraged by business projects, analytics and reports. Unfortunately, most Data Platform implementations put priority on technical data ingestion and metadata becomes a lower priority item to be done later. The result is data gets stored in the Platform without a glossary, classifications, a data dictionary, source to target mapping documents and service level agreements. Over time, as key resources involved in the initial build out of the platform leave the project, it becomes harder to explain the data because answers to questions listed above become time consuming meetings paired with lengthy data analysis exercises.
The Data Platform owners often feel it is possible to reverse engineer the metadata after the initial phase of the project, however crawling through source code to document transformation logic, creating data definitions and establishing classifications can be a project in itself. And, getting funding and resources to document metadata after-the-fact is often unsuccessful or only partially successful.
Without reliable metadata, establishing trust in the data becomes impossible. Getting business and IT data owners outside the Data Management or Data Office teams to own their data on the Data Platform becomes onerous because these owners see the Data Platform as a “black box” that is hard to understand or support. Without strong collaboration between IT, business and Data Management teams, it is not possible for Data Platforms to succeed.
Lack of data quality
The Data Management Body of Knowledge (DMBOK) describes Data Quality (DQ) as “the planning, implementation, and control of activities that apply quality management techniques to data, in order to assure it is fit for consumption and meet the needs of data consumers.” In order to achieve accurate quality, the data stored in the Data Platform should be measured against these dimensions:
-
Accuracy
-
Completeness
-
Consistency
-
Integrity
-
Reasonability
-
Timeliness
-
Uniqueness/Deduplication
-
Validity
-
Accessibility
Aligning to these data quality dimensions is best achieved when the checks are built into the project or data source onboarding process. A lot of data quality issues can be avoided if the data quality dimensions listed above are considered and addressed at the onset of data source onboarding. The aim is to ensure data quality is addressed before data is productionized in the Data Platform.
In most Data Platform implementations, data quality initiatives are often deprioritized as something that can be done later once all the critical data sources are onboarded. However, once the data is in production, securing resources and funding to address the data quality issues is very difficult and if there are consumers using that data in production, then the impact and cost becomes a larger exercise that may not be feasible to undertake.
If the data is not fit for purpose, it is not an asset that can be leveraged to deliver projects or to do analytics. A Data Platform with poor data degrades over time and eventually gets replaced by a newer version of the Platform.
Consumer specific data
The format or data model in the Data Platform needs to empower the “publish once, use many times” data policy. This means the data on the Data Platform needs to be reusable by multiple downstream consumers. Successful implementation of this strategy means data producers are motivated to publish their most granular reusable data to the Data Platform so they can eliminate the numerous point-to-point integration pipelines they are maintaining for various projects. This reduces their workload so they can focus more on running their systems to support the business rather than integration work.
Often the initial creation of the Data Platform is sponsored by one or two critical stake holders who want their needs met by the data providers. And, the Data Platform owners feel obliged or pressured to satisfy all their requirements, including the creation of customized data sets. Essentially, this translates to a Data Platform that is consumer specific with special calculations and transformations applied to the source data. The unfortunate outcome of this practice is data becomes unusable by multiple downstream projects. For example, if the data is transformed to reflect the Finance view, then that data cannot be used by the Operations team.
When a Data Platform becomes consumer specific, it is hard to drive adoption at the enterprise level.
Complexity
Data Platforms need to be flexible, easy to use and maintain in order to guarantee long-term success. This is often simpler said than done. As the Data Platform grows, keeping it simple becomes a continuous challenge. The established patterns may not be enough to satisfy the growing needs of the Data Platform so new patterns have to be introduced to accommodate the emerging needs. As new patterns are introduced, the Platform may become increasingly complex to understand. To minimize the complexity, the evolution of the Data Platform needs to be handled to always follow well thought-out minimal set of patterns that are not overly complex.
Too much bureaucratic oversight is another reason why Data Platforms become complex. The increased amount of documentation and governance checkpoints means adding one data element can take up to three months to complete. In a fast-paced world where instant decision making is necessary, Data Platforms that are slow to deliver struggle to add business value.
In summary, a successful Data Platform needs accessible metadata, good data quality, granular data that can be reused and simple design that can evolve to support business needs. Central Data Platforms play an important role in Data Management practice and provide an effective way to harness the power of data while keeping costs down. With a strong vision and pragmatic strategy, it is possible to have a successful long running Data Platform.
To learn more about the benefits of a Data Platform read- Why do you need a Data Platform.
About the Author:
Jane Estrada is the founder of Euryka Data Management Inc. (www.eurykadm.com), a data catalog software solution provider. Jane is a seasoned Data Management Professional with over 15 years of experience in Data Architecture, Data Governance, Data Modelling, Metadata, Master Data and Big Data while working for organizations such as CPPIB, RBC, TD Bank, TELUS and Scotiabank. Jane’s strong background in all aspects of data management uniquely positions her to help organizations of all sizes and complexities successfully implement data governance and other information management initiatives.