In the Field with our Experts

’Tis the Season for a Data Governance Carol (Part Two)

By Rob Paller, Consultant

 

The writing was on the wall, and Ebenezer was beginning to comprehend that while the mergers and acquisitions looked good on Wall Street, the company was imperiled by the aftermath of poorly integrated systems, an absence of proper data management, and a gaping void in the governance of the company’s data. Furthermore, Ebenezer knew that one more ghost had yet to call upon him. Never before had the fear of uncertainty enveloped Ebenezer, but tonight was different.

The Ghost of Data Governance Future

No sooner did Ebenezer begin to calm his nerves, when from the shadows of Ebenezer’s office, the Ghost of Data Governance Future appeared. Ebenezer’s pulse quickened as he did not want to acknowledge the ghost’s presence or what the ghost would ultimately share with him. It was with great reluctance that Ebenezer agreed to go with the final ghost.

Much to Ebenezer’s surprise, their travels did not take them very far into the upcoming decade before trouble surfaced. Poor Bob had suffered through another year and a half of cobbling reports together before reaching his saturation point and submitting his resignation. The remaining analysts scrambled to glean what they could from the frazzled, empty shell that was once the brightest among them. Surely a lone analyst would not beckon the fall of a Fortune 500 company, but it would begin the unraveling that followed in the months to come.

After Bob left the company, the remaining analysts fought to produce the fiscal reports on time. The data fixes that Bob orchestrated in the various systems to reconcile the reports were not being done. Some analysts were entertaining the idea of bringing in an outside firm to try and help address the problems, but IT was adamant they could shore up the problems on their end. Slowly the company’s competitive advantage began to erode and confidence on Wall Street went with it. It wasn’t long before rumors of bankruptcy began circulating. By then, Ebenezer had seen enough.

He couldn’t stand to see the company he so proudly guided through the first decade of the new century crumble before his eyes. He asked the Ghost of Data Governance Future if it was too late to turn things around. And if he promised to acknowledge all the data management, quality, integration, and governance issues that had been brought to light, would the company be able to avoid an inevitable collapse? But the ghost made no promises and offered no reassurances that the company could be saved. He simply returned Ebenezer to his office and quietly retreated back into the shadows of the office.

Ebenezer Wakes Up

Ebenezer, startled awake by the sensation of falling, quickly realized that he was still seated in his office chair. However, the relief was short-lived as he wiped the sleep from his eyes, only to see that the once-organized stacks of paper on his desk were strewn across the floor. Dream or no dream, the gravity of the situation was beginning to weigh on him:  The single most vital asset neglected for the past decade would be responsible for saving or crippling Ebenezer’s company.

There would be no quick solution. In order to build the momentum necessary to stay the course, Ebenezer needed to identify projects that would make an impact with the right sponsors who would, in turn, spread the word.

Slowly, over the course of several quarters, Ebenezer began to see the results of his mission to properly manage his company’s data as an asset. There were policies in place that supported the data governance initiative, and there were regular meetings to review the company’s progress with regards to an industry-approved maturity model. Data quality was being driven back to the source systems whenever possible.

There were formalized processes in place to deal with exceptions. There were master data hubs that dealt with integrating the various sources of master data across the company.  Data was beginning to move with the precision of a fine time piece.

---

Rob Paller is an expert at business analytics and database administration. Since joining Baseline, Rob has been responsible for developing a case analysis system to streamline the oversight of food assistance benefits, implementing a common citizen data model, and assisting in the rollout of a new public assistance data model integrating data from over 10 years of legacy with a new benefit eligibility determination system. He also believes in Christmas, and he'll gladly give you a shilling to buy the prize turkey for Christmas  day.

’Tis the Season for a Data Governance Carol

By Rob Paller, Consultant

Charles Dickens wrote the classic story A Christmas Carol over 165 years ago, and since then, there have been countless adaptations from the stage to the silver screen. While watching Disney’s latest adaptation with my family, it dawned on me not long after Scrooge declared Marley “dead as a doornail” that there was one adaptation of this classic that isn’t nearly as dead as Marley. It was when Marley came back to visit Ebenezer that this adaptation started to solidify.

In this adaptation, Ebenezer is played by the CEO of a Fortune 500 company, Marley is the CIO, and Bob Cratchit represents one of the many white collar workers at this company.

This tale begins with the present-day departure of Marley from the company. The past decade had presented this company with the usual challenges: Y2K, September 11, the market meltdown, and lingering economic woes. The company had weathered most of these storms but faced an uncertain future—even after completing several successful mergers and acquisitions. Ebenezer was a frugal CEO, but even he knew that the current economic climate would be hard on the company. Even with careful planning.

Ebenezer was working late one night in December trying to plan for the upcoming year as the year-end figures were coming in. He realized that with several companies still reporting their information independently, it was going to be another long night. As each minute passed spent digesting spreadsheet after spreadsheet, Ebenezer fought to stay awake—but it wasn’t long before Ebenezer drifted fast asleep. He would soon be visited by Marley in his dreams.

Marley was shackled in chains, carrying the burden of every system introduced by a past merger and acquisition—only to be poorly integrated. Marley warned Ebenezer that if these systems were not properly integrated, if the data was not properly managed and mastered, and if they didn’t get a grip on their data governance, he could either risk being ousted by the board or losing the company under the weight of these mergers and acquisitions. To make sure Ebenezer understood the gravity of the situation, Marley told him he would be visited by three more apparitions.

The Ghost of Data Governance Past

The Ghost of Data Governance Past was the first to visit. This ghost would bring Ebenezer back to see how the company operated long before the mergers and acquisitions. While things were far from perfect, the issues surrounding data quality, compliance, security, and master data management were easier to address. This was a time long before many of these issues were on the radar of many CIOs. Back then, data warehouse teams did their due diligence to mitigate the risks associated with the lack of formalized processes and technology solutions to address these problems

As a result, there was a semblance of a single version of the truth and trust placed in the reports produced. But as the decade went by, the company grew organically, and through each merger and acquisition, the ability to integrate information became impossible. Ultimately, the company was left supporting federated systems that fed multiple warehouses and relied on multiple reporting systems.

Upon Ebenezer’s return to the lobby of his office building, he rushed to the elevators in a panic. Now that he had seen the 30,000-foot view of the path his company had traveled, he was scrambling to figure out how and where to start solving his company’s massive data disaster. Unfortunately, for Ebenezer, he would be visited by the next ghost before he could jot down a few scant ideas.

The Ghost of Data Governance Present

The Ghost of Data Governance Present was seated at Ebenezer’s desk and offered Ebenezer a seat in one of his own guest chairs. The ghost laughed while looking over the stack of reports that Ebenezer had left behind. With a swipe of the ghost’s arm, the stack of papers flew through the air, landing haphazardly across the floor. The ghost went on to explain that it didn’t matter whether the reports were collated or not—they couldn’t be trusted any longer. The ghost only needed to bring Ebenezer back a few weeks to show him all the trouble that Bob Cratchit had gone through to finalize the numbers for these reports.

Bob scoured over the numbers—only to be forced to dive deep into the tables in the countless systems to validate the numbers. When Bob discovered what he thought was a data quality problem, there was no process in place to fix the data. The subject matter experts could only commiserate with Bob about the problems and had little recourse in solving them. When all was said and done, Bob’s effort required a 60-hour week every fiscal period. Before he had a single iota of confidence in the results. Before he could send them on to Ebenezer.

The ghost reminded Ebenezer that he had only witnessed a single day of Bob preparing these reports, and that Bob had been suffering through this for years. He was the lone analyst capable of producing these reports since he had been around the longest and understood the intricacies that each new merger and acquisition introduced. But Bob could only be the unsung hero for so long. Eventually he would burn out and move on. The thought that Bob could leave the company was a burning question that Ebenezer tried to get answered, but the ghost of Data Governance Present offered no affirmation of Bob’s actions. The ghost returned Ebenezer to his office to ponder an uncertain future.

To be continued...

---

Rob Paller is an expert at business analytics and database administration. Since joining Baseline, Rob has been responsible for developing a case analysis system to streamline the oversight of food assistance benefits, implementing a common citizen data model, and assisting in the rollout of a new public assistance data model integrating data from over 10 years of legacy with a new benefit eligibility determination system.

Real-Time Data Quality Through MDM

By Bob Wall, Senior Consultant

I have found that one of the biggest challenges for data migration, data integration and data warehousing projects that I’ve worked on, is the requirement for clean data. Typically, one of a plethora of data quality/cleansing tools is deployed. The tool employs sophisticated heuristic, probabilistic, deterministic, phonetic, linguistic and empirical methods and algorithms to perform data quality analysis. For example, for customer data being integrated into a data warehouse, we want the tool to reconcile Easthartford, Hartford East, Hartford and East Hartford to the same physical address. The data quality process is usually run in a batch process, first for the initial load and then continually after that.

Today, operational MDM hubs present an even bigger challenge. We have to ensure synchronization across multiple source systems and all data must consistently be correct. The data quality checks must be applied at various stages within the master data lifecycle, and must support federated sharing across and between systems, databases and applications. In a sense operational MDM requires real-time data quality.

Digressing for a moment, I have also worked on projects that employed various search engine technologies. On one in particular, a client wanted to have an automated way to categorize structured and non-structured web data (emails, documents, images, etc.); search for certain conditions and do it in an automated fashion. We investigated some technology that used what was referred to as the semantic web technology, which incorporated advanced semantic and linguistic analysis with classification schemes (using Hyper Text Markup Language (HTML), eXtensible Markup Language (XML), Resource Description Framework (RDF), and Web Ontology Language (OWL)) to render web content machine-readable and make it capable of being searched in an automated fashion.

Software tools that support inline SOA data quality services combined with advanced semantic and linguistic analysis/machine learning capabilities are beginning to evolve. Microsoft’s purchase last year of Zoomix is a testimonial to the strategic value of these types of products. The convergence of data quality and semantic web technologies may provide operational MDM projects with the ability to automate ongoing classification, matching, and standardization of master data records. I think it is definitely worth keeping an eye on to see if it leads to real-time data quality capabilities embedded in MDM tools.

photo by Blude (via Flickr)

---

Bob Wall is a senior consultant with Baseline Consulting. He is an information technology specialist with 30 years experience in all areas of data warehouse administration, data architecture, data resource management, training, and applications systems development, as well as in corporate management.

Disruptive Technologies: Governance in Action

By Carol Newcomb, Senior Consultant

photo by roland

I recently consulted on a Customer Data Governance program implementation project.  I must say, most companies get to the point where they design to death what governance should look like, how it should operate, who should be on which committees and who should have specific decision rights.  But how often have you ever seen data governance work well?  What does a governance policy even LOOK like?
Yeah, yeah, governance is the “4 pillars of People, Processes, Decisions Rights and Controls”, or it’s the “5 boxes of Alignment, Oversight, Measurability, Visibility, and Transparency”, but come on—what does it FEEL like?  How does it really WORK??  What makes it happen??  How can people understand it when they are told it is so important?  Who does what?

Face it.  Governance is disruptive.  We speak of disruptive technologies, like the cell phone or FaceBook, the Hybrid automobile, or the laptop computer.  These were all disruptive in their day.  They changed the way we think about social networks, distances between people and getting around.  Governance is disruptive because it seizes control from the hands (silos) of data analysts, IT staff and report writers, and puts it squarely into the laps of business people who (unknowingly) need the data to make accurate and timely business decisions.  Oh, and did we mention—share the data with each other?  Just think of the power of sharing trusted data and how much more efficient and effective the business would become with strong dependencies instead of weak, redundant independencies.

“Don’t put those decisions in MY LAP.”  We hear this all the time.  Why not?  It’s your bread and butter.  Stand up and be accountable for the fact that you don’t trust the data, but you still report it up the chain of command.  Show that you’re willing to expose who made what data decision, when, and how broad the impacts on actually are.  Be proud of the data that you use to run your portion of the business.    This is uncomfortable stuff.  Trust?  You want me to TRUST the data you lob over the fence? 
Well, let me think about that and I’ll get back to you.

Governance is disruptive because it breaks down the weaknesses we have that permeate most corporations, no matter what industry you’re in.  It forces people to talk to each other.  It forces people whom we call Data Stewards, to beat the bushes, get people to admit what they’ve been doing and how they’ve been doing it.  It wrests the decisions out of the IT cubicle and puts it squarely on the Director’s desk.  Data quality?  Let me show you our data quality.  Ouch.  Metadata?  How do you spell that?

So, get used to the fact that all change is disruptive.  But, the changes that are REALLY disruptive are those that break down the social and cultural barriers (aka personal barriers) that corporations have allowed to take control of their data, weakened its use as a decision-making tool, and contributed to massive cash outlays in data correction, cleaning and rework just to manipulate data so that it can serve its ultimate purpose: to measure the effectiveness of decisions made and to guide more profitable decisions in the future.  It’s that simple.

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

Defining Data Requirements

By Fernando Martinez Campos, Senior Consultant

photo by clayirving

In my last blog post I described a way to define BI metrics through business user interview techniques. Now it’s   time to start investigating what data is needed to answer the business questions. This post discusses data requirements, a subset of business requirements gathering.

A fundamental step in defining data requirements is to build a logical data model (LDM) which reflects the natural way data relates to the business. Aside from interviewing users and building an LDM, another input involves analyzing the sources of this data: operational applications and departmental databases.

Data requirements should be a precursor to data source analysis and profiling. A mistake many analysts make is to assume that the source is good enough to run BI Analytics. Once they load the source, much to their chagrin they find “data surprises” that will make the analysis unreliable. So it is extremely wise to being data source analysis by interviewing the SMEs and ask these fundamental questions:

• Does the data contain duplicates?
• Is there occasional missing data in some of the fields?
• Is there a better source for this data?
• Is the data content recorded accurately and reflect the business event?
• Are the business rules defined when adding or changing the data?
• Is there a data owner responsible for fixing it?

I am always surprised how much users know about errors in their data. They are invaluable in pinpointing where the errors are in the sources. Your initial efforts in the data quality investigation will uncover these “data hot spots” from them. When you start profiling the data to obtain more accurate results on exactly where all the problems exist, you’ll have a great headstart With the data profiling results you will be prepared to proactively handle any unexpected problems with the data before you load the data.

Besides creating the LDM, you will also need to create a Business Requirements Document (BRD) that will have a data requirements section. These are the activities needed to build this section:

• Define the data with a precise meaning and content, in other words, the metadata
• Determine how often the data needs to be loaded
• Determine the cutoff time for the data
• Analyze audit and compliance considerations
• Define the performance requirements and response times in retrieving the data
• Determine if the data has special security and privacy needs
• Define the length of history needed for the data

Creating a BRD streamlines the design phase with these benefits. You can uncover early security, audit and performance considerations that will need to be done in the physical implementation. And data profiling helps you proactively minimize any data quality problems during the load process. Finally, creating an LDMprovides the foundation for the physical data model (PDM) which becomes the de-facto data structure for the tables on your data warehouse.

---

Fernando Martinez-Campos is a senior consultant with Baseline Consulting and expert in data architecture, data interchange standards, legacy coexistence strategies, and reference architectures templates for infrastructure, applications and databases.

Does Data Governance Equal Data Quality?

By Frank Dravis, Senior Consultant

I was working with a client collecting stakeholder inputs in preparation for a proposed data governance program. One of the questions I asked the interviewees was “How do you expect data governance to help you?” The responses varied by the person’s role in the organization, but universally, all of the managers, be they in field marketing, global marketing, sales ops, or other areas answered the question in roughly the same way: they expected data governance to improve data quality.

Yes, data governance will indeed improve data quality over time through the implementation of policies and better processes, but the respondents expected data governance to directly affect data cleansing operations.

Suffice it to say there is a whole chain of activities that need to take place between the formation of a data governance program and the point where a data analyst connects a data quality solution to the data and begins the standardizing and de-duping. However, it was encouraging for me to see the value proposition of the two concepts, governance and quality, inextricably intertwined, no attempts to separate or isolate them. Our client’s business people fervently believed that taken together data governance and data quality were the solution to the myriad of problems exacerbated by silo-ed, fractured, and unreliable data.

My guess is that this client’s adoption curve for data governance will be shorter and can piggy-back off the successes of data quality. I’ll let you know if I’m right.

---

Frank Dravis is a senior consultant with 21 years of experience in enterprise information management (EIM) and data quality solutions design, implementation, and consulting. He specializes in data integration, data quality, and data governance solutions, advising key clients and industry vendors.

The Color of Data Quality

By Frank Dravis, Senior Consultant

Is green. I don’t know why I am surprised about this, but having worked in data quality for two decades, it was really never an issue. Until now. In a webcast that was picked up by the Inspired Economist, I discussed how data quality can improve the effectiveness of marketing, specifically in matching diverse customer data.

During the webcast, I cited a case where a client saved $12 million in annual printing and postage fees from a 50% reduction in distribution of catalogs to customers. Fully 50% of this company’s catalogs were being sent to incorrect addresses, to the wrong people, and in many cases, multiple copies were being sent to the same person. The ROI on the original $100,000 data quality investment was impressive. The Inspired Economist referenced the financial ROI, but the publication also emphasized the green, eco-friendly benefits of data quality.

In this day and age of burgeoning landfills, ground water contamination, and greenhouse gas creation from power consumption, a reduction in printing six million catalogs is a substantial benefit. Fewer trees are cut, less ink is sprayed, and less coal is burned. Any time you can be more efficient and effective at what you do—while being eco-minded—is a win-win. And in this case, it comes from better data management far upstream from the actual manufacturing process.

---

Frank Dravis is a senior consultant with 21 years of experience in enterprise information management (EIM) and data quality solutions design, implementation, and consulting. He specializes in data integration, data quality, and data governance solutions, advising key clients and industry vendors.

Does Your Data Warehouse Project Need a “House” Call?

By Bob Wall, Senior Consultant

  For the last several years I have been an avid fan of the television show “House”, an American medical drama.  Each week the episode begins with an unsuspecting patient falling prey to some unknown and mysterious disease. They typically fail to receive an initial correct diagnosis, rendering their case as rare and very complex.  Dr. House and his team of specialists then spend the remainder of the episode acting like the medical equivalents of Sherlock Holmes, trying to solve the root cause of the disease and discover a life saving cure. Throughout the episode, Dr. House is rude, obnoxious, condescending, and addicted to pain killers, and the team performs numerous ethically dubious procedures. For instance, one or more of the doctors often makes a visit to the patient’s home in search of clues that might suggest certain pathology. They enter the home with or without permission, sometimes breaking the law, to try to discover any underlying evidence.

Miraculously by the end of the hour show, the cure is almost always found; the patient lives, and House and his team are ready to solve the next big mystery the following week.

Wondering what this has to do with your data warehouse project? Well, imagine if you will, some slight changes in the show’s format and cast and maybe you will see the similarity.

In this episode a business manager mysteriously falls prey to a sick data warehouse environment and all indications are that it has something to do with bad data.  No one can figure out the underlying cause and in fact, the manager has received several misdiagnoses. Just as the manager is ready to give up, Dr. House and his team of diagnosticians step in to assist.  In this case House’s specialty is Data Management and the other doctors are specialists in Data Governance, Data Architecture, Data Modeling, Data Analysis/Profiling, and Data Quality.

The Data team first identifies the overall problem and then the symptoms:

Overall Problem

• Inability to understand source data and uncover the bad data disease, imperiling the data warehouse

Symptoms

• Data quality issues
• Multiple sources generating the same data
• Non-existent documentation of source systems
• Data models and file structures not understood
• Lack of knowledge about core data elements
• Mismatch between source system data and developers’ definition
• Analyzing multiple data sources for a high profile IT project

Next the team interviews all of the patient stakeholders (SMEs, Data Stewards, Business Analysts, IT, etc.) and makes a visit to the corporate home to gather  evidence that might help in the diagnosis (business plans, data models, system documentation, ETL specs, etc.).

Now the team concurs on several possible reasons for the problem and runs several diagnostic tests, the most prominent being:

• Data Source Analysis – the purpose is to attain an understanding of what data is available and where it resides within the patient’s myriad of systems and applications.
• Data Profiling – the purpose is to attain an understanding of an organizations data validating data types, domain values, etc.

Finally, the Data team is able to understand the patient’s disease and dispense a cure for the data warehouse problem by addressing the data quality problems. This cure includes offering short term data corrective actions, ongoing incorporation of data assessment—a data profiling tool and analysis process—as a permanent part of their source system infrastructures, and a long term common recommendation for the creation of a new role, Source Data Steward, for systems that are either highly complex or in demand by other business applications.

So, just like the television show, a “House” call for a sick Data Warehouse Project from the Data team can be seen as analogous to the House television episode. However, there are several noteworthy differences: The lead “Data Doctor” in this case is never rude, obnoxious or addicted to drugs! The team never breaks into a building to gather documents and evidence, but always asks permission first!  And, most importantly, there is no certainty that the cure will be found in only an hour episode!

---

Bob Wall is a senior consultant with Baseline Consulting. He is an information technology specialist with 30 years experience in all areas of data warehouse administration, data architecture, data resource management, training, and applications systems development, as well as in corporate management.

Dirty Little (Data) Secret: Baseline in the Field

Inasmuch as Baseline’s partners and senior consultants are in the industry and on the speaking and publishing circuit, our bread-and-butter business is working with Global 2000 businesses to deploy business information. This business information is cross-functional. It’s strategic and operational. And often, it originates from cryptic legacy systems that don’t talk to one another, much less well-meaning programmers. The data is dirty, to say the least. The BI tools are ready, but in the course of the data deployment they may have gotten a bad rap. And clients often don’t know what to do next.

We help them with all these things. In the course of doing this work we’ve become acutely aware that with every client engagement comes another lesson learned. Or two. Or three. The mission of our In the Field blog is to share those lessons. Many of them are hard-won. Many of them we see on almost every engagement. And many of them are likely in your future. So let’s talk about them here. We look forward to seeing you here—and reading your comments. Come back often to see what’s happening in the field!