Does SMB need Metadata?
The simple answer is Yes. Any organization with business-critical data in multiple systems and files needs metadata to make sense of the data. Metadata is data about data. It is a set of data that describes and gives information about other data. When it’s possible to self-serve answers to data related questions, an organization reaches a state of maturity level that enables them to harness the power of data.
-
What data is available?
-
Which systems have the data required?
-
How does the data flow through the enterprise?
-
How to get access to the data?
-
Who can help me answer questions about the data I need?
When an organization is metadata driven, it becomes data driven. Which, in return, enables good decision making to benefit the business. Dilbert cartoons like this one is no longer a relatable joke.
Dibert cartoon
Just as papyrus transformed humanity, metadata transforms data management. The invention of paper allowed people to document their thoughts and share it with others in a way that was not possible before. The ability to create documentation that can be passed on from generation to generation is one of the transformational achievements in human civilization. Reliable accessible metadata (aka Data Catalog) can have the same impact in data management practice.
Ancient papyrus
Small and Medium Businesses
Gartner describes small businesses as organizations with fewer than 100 employees and midsize enterprises as those organizations with 100 to 999 employees. These organizations not only need metadata management function but they also need other data management pillars like data quality, analytics, data governance and etc. When considering data management practice, the size of an organization does not matter rather, consideration should be given to the nature of business. If the business activities are data centric then it makes sense to focus on data management. For example, business revolves around data at insurance and wealth management organizations so it makes sense to govern the data to reduce risk and increase profits.
https://www.gartner.com/en/information-technology/glossary/smbs-small-and-midsize-businesses
What is Metadata Management?
Metadata Management within the DAMA Body of Knowledge (DMBoK®) is defined as “the planning, implementation, and control activities to enable access to high quality, integrated metadata.” The tools and processes to support metadata management practice needs to support the following different categories of metadata:
Business metadata
-
Glossary
-
Data Quality Rules
-
Regulatory constraints
-
Data policies
Technical metadata
-
Physical data models
-
Mappings and transformations
-
Data lineage
-
Data producing systems
-
Data consuming systems
Data ownership
-
Data steward roles
-
Data custodian roles
Process metadata
- Data processing times
Metadata for Digital Transformation
Organizations of all sizes are focusing on digital transformation. Digital transformation is the implementation of new technologies and processes to improve business operations and improve customer experience. In order transform existing technologies, it is important to make sense of the data in them and to understand how the data is integrated between the various systems. How do you go about collecting this information? Metadata provides this information. Software engineers, product managers, data scientist and data architects need metadata to roll out new digital products and services while enhancing iterative development and shortening time to market.
Unfortunately, most CIOs and CTOs put priority on technology improvements and metadata becomes a lower priority item to be done later or never. The result is data gets stored in the systems without definitions, classifications and data flow documentations. Over time, as key resources involved in projects rollout, it becomes harder to explain the data because answers to questions listed above become time consuming meetings paired with lengthy data analysis exercises. To avoid these pit falls, technology and data leaders should guide their teams to get started on metadata management.
Getting Started
Metadata management practice doesn’t have to be a multi million-dollar project. In most cases, there is already plenty of metadata collected in Excel spreadsheet, SharePoint sites and in Confluence pages. These artefacts provide a good starting point. Assigning some resources to consolidate the metadata into a single artefact is a good starting point. Often Excel is a good tool for starters but it is not recommended to use it for long-term because it is hard to link various metadata together and enabling collaboration in Excel is error prone.
There are very expensive and complex Data Catalog tools in the market today. These tools are often designed by technical experts and the user interface tends to be too technical for business users such as product owners to understand without costly training. There are also tools that promise to automatically crawl and discover metadata in various systems. These connectors are beneficial to minimize the manual data entry but it is important to understand that metadata does require human resources to keep it updated and accurate; it cannot be 100% automated. Additionally, it is not possible to auto discover business metadata such as glossary since these are not stored inside any systems.
When choosing metadata tools, look for tools that can be implemented fast, without rigid framework. Easy to understand screens that can be adopted by business users with minimal or no training should be a key consideration because collaboration with the business users is key to a successful metadata management program. A tool that allows customization is great because metadata is unique in each enterprise. Lastly, API should be available for integration so metadata can be integrated with other systems such as Data Quality tools and crawling to auto discover and auto populate metadata is possible.
Once a tool selected and resources are assigned to support the Data Catalog, start populating the metadata by leveraging existing artefacts. After the initial metadata population, operationalizing the ongoing maintenance of metadata is the most critical step. This is best accomplished as follows:
-
Decentralize the responsibility to the operational, business and technical teams that need to contribute to the data catalog. This spreads the responsibility and promotes a collaborative approach which ensures long term success.
-
Combine automatic metadata discovery with manual data entry as part of SDLC (System delivery life cycle) to keep the metadata up-to-date and relevant.
-
Couple of dedicated resources can focus on governance activities to ensure the various teams have the access and information required to carry out their tasks
Large organizations often struggle to successfully initiate and implement a metadata management practice. There are too many stake holders so consensus on a coherent solution is hard to achieve; it is common to see multiple Data Catalogs in different business units that are sparsely populated and not accessible to users who need the information. So, the SMB enterprises definitely have an advantage with fewer decision makers allowing for technically advanced solutions that are business friendly.
Data is increasing in volume every day. As more data gets used more data is created. Data can be a valuable “asset” and “resource” if it can be understood and leveraged to create product and services to satisfy customer experience. Metadata is fundamental to understanding the data. Unfortunately, the CIOs, CTOs and CDOs de-prioritize metadata thinking it is documentation that can be done later. This is a mistake that can impact digital transformation and the organization’s ability to lower risk and increase profit.