In the Field with our Experts

Metadata is Key

By Stephen Putman, Senior Consultant

One of the most promising developments in data management over the last ten years is the rise of semantic processing, commonly referred to as the "Semantic Web." Briefly described, semantic processing creates a "web of data" complimenting the "web of documents" of the World Wide Web. The benefits of such an array of linked data are many, but the main benefit could the ability for machines to mine for needed data to enhance searches, recommendations, and the like, where humans do this now.

Unfortunately, the growth of the semantic data industry has been slower than anticipated, mainly due to a "chicken and egg" problem - the systems needs descriptive metadata to be added to existing structures to function efficiently, but major data management companies are reluctant to invest a great deal into creating tools to do this until an appropriate return on investment is demonstrated. I feel that there is an even more basic issue with the adoption of semantics that has nothing to do with tools or investment - we need the implementers and managers of data systems to change their thinking about how they do their jobs; to make metadata production central to the systems they produce.

The interoperability and discoverability of data is becoming an increasingly important requirements for organizations of all types - the financial industry is keenly aware of the requirements of reporting systems that are XBRL-enabled, for example. If we leave external requirements to the side, the same requirements can benefit the internal reporting of the organization as well. Reporting systems go through extended periods of design and implementation, with their contents and design a seemingly well-guarded secret. Consequently, effort is required for departments not originally included in the system design to discover and use appropriate data for their operations.

The organization and publication of metadata about these reporting systems can mitigate the cost of this discovery and use by the entire organization. Here is a sample of the metadata produced by every database system, either formally or informally:

• System-schema-table-column
• Frequency of update
• Input source(s)
• Ownership-stewardship
• Security level

The collection and publication of such metadata in standard forms will prepare your organization for the coming “semantic wave," even if you do not have a specific application that can utilize this data at the present time. This will give your organization an advantage over those companies that wait for these requirements to be implemented and will need to play catch-up. You will also gain the advantage of your staff thinking in terms of metadata capture and dissemination, which will help your company become more efficient in its data management functions.

photo by ~Brenda-Starr~ via Flickr (Creative Commons License)

---

Stephen Putman has over 20 years experience supporting client/server and internet-based operations from small offices to major corporations.  He has extensive experience in a variety of front-end development tools, as well as relational database design and administration, and is extremely effective in project management and leadership roles. He is the co-author of The Data Governance eBook, available at information-management.com.

The Data Governance eBook: Maps, Mechanics and Morals

Recently, we launched our newest e-book, The Data Quality eBook. We ran an excerpt from that book last week. But did you know we have another e-book out there? Earlier this year, we published The Data Governance eBook to rousing success. If you haven't had a chance to download and read this book by Kimberly Nevala and Stephen Putman, don't worry—it's still available online.

Here's a description:

When the fictitious business team at the SpectraDynamo corporation launches a new data governance initiative, the trials and tribulations begin. The Data Governance eBook, authored by the experts at Baseline Consulting, is an entertaining business parable with a message aimed at helping you launch data governance the right way. This bold new look at data governance is more than just a cautionary tale. It's what Baseline partner Jill Dyché calls "a prescription for getting it right." The Data Governance eBook includes all the nuts and bolts necessary to transcend the requisite decision-by-committee and role confusion so endemic to companies’ information management efforts.

Here's a brief excerpt that showcases what you'll find in our first e-book:

The moral of the story: data governance can be tough. By definition, innovation is disruptive and antithetical to established paradigms. Applying rigor in the form of governance and oversight to information is no exception to this rule. Mix the laundry list of competing information needs and issues in the average enterprise with the inevitable confusion regarding ‘what does it mean’ and the data governance proposition quickly becomes overwhelming. The multitude of data governance definitions and methodologies parlayed by both industry analysts and vendors—is it: data quality? information policy? a dashboard? a council?—certainly doesn’t help.

The fact is: there is no standard playbook for data governance. The structure of your program is dependent on the specific problems governance needs to address and the organization’s incumbent organization and culture.  Some companies—like SpectroDynamo—find a bottom-up approach centered around core data management functions provides the greatest initial leverage. Organizations that are more hierarchical or consensus driven in nature may require formal enlistment of executive sponsorship and deliberate assignment of decision rights for governance to take root. And yes, you may start somewhere in the middle...

What is consistent? The fundamental roles and responsibilities found in functioning data governance programs. In this section, we define those core functions: executive sponsorship, data governance (policy making and decision rights), data management and data stewardship. These functions align with and support everything from strategy alignment and organizational buy-in to tactical enablement and operations.

When evaluating these capabilities, remember the single most important success factor for entrenching data governance as a core operating principle: continuous improvement. Rather than boiling the ocean, identify key business pain points and the core capabilities required to address them. Prioritize and pick a pilot project. Finally, prototype data governance-related processes and don’t be afraid to refine them as your capabilities and reach grow.

Interested in knowing more? Download The Data Governance eBook today!

Responsible [Data] Stewardship

By Caryn Maresic, Senior Consultant

Contribute to society and human well-being.  Avoid harm to others.  Be honest and trustworthy.  Be fair and take action not to discriminate.  Those are the first four items in the ACM Code of Ethics.  The ACM, for those who may not be familiar, is the Association for Computing Machinery, whose mission is to advance computing as a science and a profession.

In the course of a recent assignment with a major insurance carrier our team was asked to create various target lists for sales and marketing based on certain selection criteria.  While it is likely that all of the things they asked for were legal and ethical, we never questioned it.  As good Data Stewards, what should we have done in this case?  Should we be asking the business to justify their selection criteria?  Should we be checking to make sure there are no legal or ethical violations inherent in the rules?  A little research on the topic turned up this presentation 
which is very interesting and thought provoking.  That being said, it focuses more on the hot-topic issues like privacy and identity theft than it does the ethical dilemmas of sales and marketing.

This article tells the story of an “Agent Profile System” set up by an insurer in Texas to rate its agents.  Agents who didn’t score well were punished by not getting any new business.  The agents filed suit contending this was illegal as it compelled them to drop clients with low credit ratings, low income, and/or those who lived in undesirable locations in order to boost their own score.  Is the IT team that built the Agent Profile System responsible, at least in part, for discrimination?

When we are dealing with situations where lives are in danger the ethical answer is clear.  For example, no reasonable person would deny that engineers working on Space Shuttle software have a duty to report concerns regarding possible malfunction.  In the BI community our issues are not always so clear cut.  Sometimes discrimination is good for the business’ bottom line, yet still unethical and possibly illegal.  If we go back to the statements “Avoid harm to others” and “Be fair” and “take action not to discriminate” it appears that we should take serious our responsibility to be involved in how the business uses data.  In fact, I would argue that we should make ethical considerations part of our data governance program.

photo by Robert S. Donovan via Flickr (Creative Commons License)

---

Caryn has over 20 years experience in providing high-quality data solutions to clients in the areas of Business Intelligence, Data Warehousing and System Integration.  Caryn has expertise in across industries with an emphasis in Pharmaceutical, Manufacturing, and Insurance.  Prior to joining to Baseline, she ran her own consulting company.

Bringing Data Governance to the Table

By Frank Dravis, Senior Consultant

It’s counter-intuitive how something as simple as a lookup table can drive a data governance initiative. At one client site I worked at, they had a reference table called Customer Level Code. In that table, there were nine records containing values such as Premier, Platinum, Invalid, etc. The table included a description column for the tier and a corresponding lookup code, but that was it. For years, this little table sat on the legacy data warehouse with no true source system from which to track its origins or authorized editors. From time to time, a code was added, such as Not Required. The person who told the mainframe DBA to add the value was an IT manager who worked with the business and saw the need.

Fast forward to the time of building a much better data warehouse. The architects, in their efforts to satisfy the business requirements, accumulated a list of over 50 such reference tables. These tables spanned the enterprise. They were referenced by many applications, and encompassed everything from employee pay grades to merchandise categories. In order to do things right, the new data warehouse needed a defined source for each data domain and table it would load, and for a large number of the reference tables, there was either no single source or defined owner or even an electronic repository to hold the original values. They were written in a notebook.

To wrap our arms around all of the reference tables, a solution was implemented where the contents of all 50 tables was loaded into one table in the landing area for the new data warehouse. This new table included columns such as change date, who authorized the change, previous value, etc.; in other words, all the rich metadata to describe how the code values were managed. Better still, this single table was published as the source of record for those tables that had no previous source system. The owners of specific records were identified, and standard extracts were written to pull data from those source systems that did exist. Taken in isolation, maintaining the Customer Level Code table was a small, informal activity below the data management radar. But when the cumulative effect of all 50 tables was considered with their errors, latency, and lack of linkage, a substantial benefit was gained by managing them as a corporate resource. That single table housing all code values aided the implementation of a data governance initiative by:  identifying gaps in ownership; establishing a central repository for management, common access routines, standardized interfaces; and better yet, uniform change management for policies and permissions.

So the next time you are working with a code table, ask yourself how many more are there like it, and wouldn’t it be nice to have one central place to go for every reference field you ever needed?

photo by p_a_h via Flickr (Creative Commons License)

---

Frank Dravis is a senior consultant with 21 years of experience in enterprise information management (EIM) and data quality solutions design, implementation, and consulting. He specializes in data integration, data quality, and data governance solutions, advising key clients and industry vendors.

Did You Forget Something? Gaps in Execution in Data Governance Programs

By Carol Newcomb, Senior Consultant

Somewhere along the way to understanding the whole data governance thing, someone forgot to mention that data management really is central.  Did I say Mention?  I really meant Emphasize.  Urgently Emphasize.  Data management processes are critical to the establishment and maintenance of a data governance program.

The thing is, data governance is quite impossible without a fundamental data management process that provides tactical direction. For example, if you have a bunch of people running around acting as data stewards, asking people questions about metadata, source systems, definitions, use of data and implications of sharing it, what happens when they have to write a policy?  And when data stewards meet as a group, how can they describe the magnitude of issues to discuss with their users?  How do issues get prioritized?  How can anybody decide who is accountable if due diligence hasn’t been paid to the fundamental source data issues, and how can decisions about governance enforcement be made?

Data governance should never be undertaken as a project.  It is a long term program that needs to start with a foothold in core data problems, like the inability to use or share data.  To get that foothold, stewards need to identify data quality and consistency issues that resonate with end-users.  Find some tangible pain.  This requires a sound set of data quality profiling activities to prove where conflicts or inconsistencies are most problematic, and a quantifiable statement of what the impact is.  The problem needs to be quantifiable to get people’s attention, as well as to compare one set of problems against another.  The data management program should have the capacity to routinely analyze and reveal data quality issues,and keep bringing them back to end-users to build a sense of urgency towards taking restorative and preventive action. 

Data governance is all about People, Processes, Decision Rights and Controls.  But before you can even begin, you will need to have identified the problems through hardcore data management practices.  This involves understanding the data architecture, examining privacy and compliance mandates around data visibility and sharing, exploring metadata definitions and having the ability to make changes through sound data administration practices.  Without these systems and roles in place, data governance is just a hollow shell.

photo by limaoscarjuliet (via Flickr)

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.