In the Field with our Experts

A Data Governance Primer (Part 3 of 3)

By Carol Newcomb, Senior Consultant

Diamond in the Rough: Data Quality

The third part of my summertime primer addresses Data Quality Analysis.  Don’t even start a data quality analysis until you have completed the first two steps of your Root Cause Analysis--investigate & prioritize any potential causative factors, and start your metadata assessment.  Otherwise, you may be misled by your findings.

Data quality is defined as complete and accurate data that is ready for business consumption.  Sources of poor data quality may include lack of data entry rules, unclear data element definitions, inconsistent metadata definitions for field type, format or intent, or breakdowns in data transformation processes as data flow between systems or applications.  Poor data quality results in bad business decisions; it contributes to major problems in using data effectively, and costs companies millions of dollars/year in terms of rework and inefficiency.  Data quality, in combination with robust metadata definitions, is part of the foundation of good data governance.

Data Quality Analysis

A Data Quality Management process should be designed to enable an area to start with a simple approach and over time to mature to one that is more proactive and comprehensive.  Initially, investigation may be focused on single data elements or events.  As patterns, data commonalities and other relationships appear, the data quality management process will grow to support complete business processes.   A mature data quality management process will not just resolve individual issues; it will also track relationships between data elements, ensure that business rules are consistent and generate statistical analyses to monitor previously addressed issues to ensure that data quality is stable and that an early warning system is in place as part of the data governance program.  The goal is to design a data quality management lifecycle, as shown in this diagram:

Initial Data Quality Analysis Process

I. Define data scope

• Determine data elements that are associated with or are direct results of the reported issue
• Check that all metadata definitions are present and current
• Enlist the involvement of the Data SME or Data Stewards
• Identify all source systems where the data originates, is  entered or derived

II. Extract and profile the data

• Extract the relevant data from all key source systems.
• Design the profile.  A profile will consist, at a minimum, of total record counts, min/max values, frequency of unique values, and frequency of invalid values (if defined) for each data element profiled. 
• Profile the data to determine key characteristics that are contributing to the issue, such as:
• 1. Wrong values
  2. Missing values
  3. Corrupt transformation processes
  4. Incorrect business rules
  5. Incorrect usage rules

III. Analyze Data Profile Results

• Summarize the key findings from the profile detail
• Determine what key drivers are contributing to the impact
• Determine accountability for the data quality issue
• Involve other Data Stewards in troubleshooting and designing the data quality solution

IV. Design the Corrective Action Plan

Two types of plans should be developed to address known data quality issues: a corrective action plan to fix the immediate source of the problem identified, and an ongoing monitoring plan, where thresholds have been determined and metrics are routinely collected and reported to data stakeholders.  This monitoring process should be scalable based on the number of data elements being tracked.

1. Corrective Action Plan
2. • Does scope of problem warrant change in metadata definitions, business practices or data entry rules?
   • Does scope of problem warrant a data governance standard?
   • Does the corrective action plan include details on how to fix the source of the problem as well as ways to correct historical data in the system?
3. Preventive Action Plan
4. • This plan will be designed to minimize the probability of data quality issues from recurring
   • Determine ‘early warning triggers’ based on designated thresholds.  These thresholds should reflect the business tolerance for inaccurate data (is 95% acceptable?)
   • If data latency is the source of a data quality issue, then latency thresholds should be included in the monitoring plan
   • Determine how frequently results of the monitoring plan will be reported to data stakeholders or governance oversight committees

So, now that summer is officially here, this wraps up my Data Governance Primer series.  Time for some iced tea and my favorite beach towel.  Come August, these little refreshers might be just the thing!

photo by Swamibu via Flickr (Creative Commons License)

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

To Infinity and Beyond! — Healthcare in the Cloud (Part 2)

By Carol Newcomb, Senior Consultant

This is the second installment in a 3-part blog series, discussing the opportunities that cloud computing offers in healthcare. I present futuristic scenarios from each healthcare contingent’s vantage point:  patients, providers and payers.   A myriad of technologies exist today.  It will be up to healthcare organizations of all types to get their data ready to meet the demands for data integration, security, portability, transparency and accountability in this brave new world.  Mature data governance systems and enterprise-wide data integration will be critical in this endeavor.

II:  Providers

It’s 11:30pm on a Sunday night, and I’m working a double shift in the Emergency Room.  It’s been a little slow tonight, so I’ve been online researching some reasons why we’ve been seeing more Somali children here recently.  The hospital keeps the latest demographic figures on the intranet.  Turns out that at first just a few families settled here, but more relatives are joining them and many of the children are orphaned.  When they came through immigration, though, they were given medical checkups and those records are available for each of the kids I have treated recently, since they were each given a universal medical record number.  I can see the results of any tests they had, and whether they were given any medications or treatment.  There’s also a public health database through CDCP that tracks the incidence of infectious diseases, and some primary prevention steps we should prescribe at the time we treat these kids.  It’s very helpful stuff.  I’m going to share what I learn through the local Health Information Exchange (HIE) with the social workers in the neighborhood clinics.

Whoa!  Suddenly things are picking up!  There’s a little old lady with severe dementia who fell and knocked out her tooth, and a teenager who got into a brawl with a black eye.  Where are his parents!?  First I try talking to the lady, but she’s confused.  At least she is wearing an ID tag, so I look up her name (Dorothy Xu) in the HIE portal.  She’s never been to this hospital, but I see that she has spent some time in a local nursing home, and was recently discharged to the care of Dr. Jones.  We start a new hospital record for her, instantly downloading the data from the nursing home and Dr. Jones’ office.  I also see that her daughter lives nearby and that her husband died less than a year ago, so I get her daughter’s telephone number and give her a call.  I’m surprised that the ambulance techs didn’t do this already, but they were making sure she didn’t have any other injuries.  When her daughter arrives, she has brought a list of all of Dorothy’s medications, and one of her favorite nightgowns.  I send an email requesting an urgent appointment with one of the dentists in her insurance network first thing in the morning, and then make sure she has a prescription for some pain medications to get her through the night.  Poor thing.  Also, I talk to her daughter about whether Dorothy has seemed depressed since her husband’s death.  I give her daughter a printout of some good psychiatrists in in the local area, and then she takes her Mom home.

So what about this black eye?  Jason is 15 and it looks like he already has a police record.  He refuses to talk to me, so we get his fingerprint and match it to his medical record.  He doesn’t want the police to know he was in another fight, but I’m going to have to call his parents to pick him up.  He seems concerned, but refuses to say anything more.  I scan through his chart and his Mom’s, and I see that she has also been treated for a fractured wrist and a black eye about 4 weeks ago.  Either this family is really clumsy, or something is going on here!  Apparently, a social worker at school also noted a few things in his record, and the pieces are beginning to suggest some family abuse problems.

When his Mom arrives, we talk privately for a few minutes.  I tell her I am concerned, but don’t really have very much information.   As we talk, she becomes visibly upset.  Eventually, she admits that her husband, who was recently laid off, has been taking his anger out first at her son, and also at her.  I ask if she needs any other assistance or resources, but she refuses.  So I treat her son’s eye and send them on their way.  I also make note in the husband’s record that he may have some history of abusive behavior, and I notice that he was recently in the hospital as a result of a car wreck.  At least this information will be available to the nurses in case they are treated here again.

---

Offloading disparate datasets to integrated Cloud servicers that will also perform basic analytics can greatly empower healthcare providers with ready access to data.  Remotely hosted Health Information Exchanges will empower social workers, schools and hospitals, even law enforcement authorities, to work together in assisting their local communities.   This is the profile of the new healthcare provider as a result of Cloud technology.

photo by dawvon via Flickr (Creative Commons license)

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

Data Profiling with SQL is Hazardous to Your Company’s Health

By Stephen Putman, Senior Consultant

Data Profiling is the act of extracting descriptive  and statistical data about what resides in a database in order to facilitate data quality and design efforts, among other things. Tools for performing profiling range from open-source software (DataCleaner, Talend) to database system tools (Microsoft SQL Server, Oracle) to specific and purpose-built products (vendors Trillium and DataFlux both offer data profiling solutions). All of these tools automate and standardize the data profiling task, and integrate with other areas of the design effort to varying degrees. However, most organizations do not use them, relying instead on SQL queries, run manually. Why does this happen?

The main reason is that data profiling is not seen as a high-priority activity. Therefore, the investment in licenses, hardware resources, and training for a profiling system is seen to be an unnecessary expense. Consequently, data and business analysts are forced to use existing tools such as SQL query clients to perform basic profiling tasks.

Many times, this is due to a basic misunderstanding of the power of a data profiling tool, and the data it analyzes. Profiling data is useful in logical and physical data modeling, functional system requirement production, and test script generation. Profiling gives the overall system scope and context, and ensures that the system produced satisfies the requirements of the business by characterizing the values present in source systems.

Manual SQL data profiling is fraught with risk. It relies on the skill and knowledge of the analysts producing the queries for accuracy and effectiveness, and there is no guarantee of consistency of results or standardization.. Modern profiling tools have optimized data gathering technology that is faster than manual SQL and scenario repositories that can be shared between analysts and projects to promote reuse.

Implementing a robust data profiling system is an essential part of an effective data management environment. The cost of putting such a system into place is more than offset by the benefits of deep analysis of data sources in both the transactional and analytic spaces. The essential activity is to discover just how powerful such a system can be in your organization.

Photo by mandolux via Flickr (Creative Commons license)

Healthcare Data’s Baby Steps: Meaningful Use, Pay for Performance, Quality Improvement (Part 2)

By Carol Newcomb, Senior Consultant

In my last post, I bemoaned the carrot-or-stick approach that has resulted in immature, disintegrated reporting solutions.  In this post, I challenge the notion that business intelligence, as opposed to formal data governance, is a fix to healthcare organizations’ fragmented data. 

Collecting data is easy.  Most electronic health records (EHRs) and personal health records (PHRs) are front-end data acquisition and transfer tools, with sophisticated built-in rules and customized algorithms.  Storing data, metadata, and even “data exhaust” gets cheaper by the hour.  The trick is turning all this data into meaningful information, and using that information to make smarter decisions.  How hard is it to turn data into reports using your current data warehousing technology?  How frequently do you get reports that just don’t jibe?  Can someone confidently explain how a number on a report got there?

IBM is running a catchy ad that resonates deeply in our data-saturated world.  “If you can’t believe the data, how can you believe the analytics?”  (Healthcare CFO)  Right on!  Business intelligence is NOT the place to tackle healthcare integrity issues.  Let me pose some fundamental BI questions:

• What’s your headcount of 3rd year medical residents today?  In each department.
• How many RNs and LPNs do you have in any single department?  Last Sunday?
• What is the average patient wait time in your GI lab?  What’s the variability?
• How many repeat lab tests did you have for patients who are being diagnosed with AIDS?  How many are positive?
• What’s the dollar amount for lawsuits filed for events that happened on the pediatrics unit?  Were those lawsuits for ‘Never Events’?
• What is the failure rate of certain brands of hip prostheses used in hip replacement surgery?  How long before they failed?
• What’s the infection rate on each day of the week in the ICU?  Does it change at shift-change?

These are straightforward administrative and clinical quality questions that most healthcare organizations are hard-pressed to answer without scraping around in spreadsheets, disconnected data systems or standard reports.  Why?  There is a fundamental lack of recognition that data governance and data integration are critical infrastructure elements that continually get no attention in a strategic IT budget, and get zero dollars as a result.  Healthcare data management is stuck in its infancy, crawling on hands and feet.

It is important for healthcare organizations to use the government’s incentive dollars and retool for the 21st century.  However, while incentives are there to facilitate the technology transition from paper to electronic data, there is huge risk in installing yet another front-end system and further complicating your back-end IT infrastructure.   Organizations that have spent years consolidating on SAP, Oracle and other ERP systems have needed to return to fundamental governance practices to resolve the confusion that usually results from inconsistent definitions and non-existent data usage rules even after years of effort. 

The failure to design data governance at the start of an integration project is expensive, and wastes the best features of even the most sophisticated business intelligence tools.  

It is sad to compare healthcare to retail, manufacturing, banking, and just about any other industry in terms of customer-centric capabilities.  Healthcare organizations have failed to take a sustainable approach to data infrastructure because they have ignored the need for data governance.  This churn will only continue until healthcare leaders finally step back and insist that solid infrastructure--including a data governance plan, an MDM plan, a data integration roadmap--are budgeted, staffed and successful.  Otherwise, they had better hope that attestation will continue to work.

photo by Yoni Lerner via Flickr (Creative Commons License)

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

Healthcare Data’s Baby Steps: Meaningful Use, Pay for Performance, Quality Improvement (Part 1)

By Carol Newcomb, Senior Consultant

When I think about all the time and energy that has been spent and dollars invested in collecting and analyzing healthcare data over the past 30 years, it makes me crazy!!  From fee-for-service to HMOs to PPOs to HDHPs, billions of patient records exist that could inform policy on quality, safety, outcomes, effectiveness and efficiency, but as with our little Christmas-day Bomber incident, “We had the data, we just couldn’t put it all together.”  In the time that it has taken banks, retailers, even government to retool their operations to provide real-time, web-enabled, customer-focused applications, healthcare has gone nowhere.  The government is literally bribing healthcare organizations to convert from paper-based systems (or from stand-alone off the shelf-applications) to integrated electronic systems.  “Meaningful Use” is being touted as the newest data reporting incentive to show that your EHR system is up and at least partially running.  The first phase of evidence will include attestations that providers are using certain features of the system.  Not data, attestations!! 

And this isn’t the first attempt to get healthcare organizations to willingly embrace their data.  For the past two decades, outfits like the Joint Commission, AHRQ, NCQA, NQF and many others, have developed reporting metrics, hoping to drive down not just the cost of care, but the inefficient, wasteful and sometimes harmful practices that exist in our provider organizations.  Tying these measures to accreditation, CMS reimbursements, Pay for Performance and other carrot-or-stick mechanisms hasn’t been enough for hospital CIOs to seriously undertake more than just one-off fixes to their data infrastructure.  What they need is a back-to-basics roadmap for enterprise data management to integrate and govern data as it is gathered and aggregated over time.  Data governance is virtually non-existent in healthcare as a formal program.  Without governance, an organization cannot stand on its own data.

Healthcare is swimming in data.  Some would argue that what healthcare needs is better business intelligence.  Sure enough, it’s pretty hard to analyze a lot of loosely-coupled data that resides under any one healthcare organization’s roof, much less tie it to any one patient or doctor.  What good are analytics if the data haven’t been through rigorous quality controls?  What if the data aren’t stable or even accurate?   How many people contribute diagnosis and procedure and billing codes and modifiers to any single patient record?  How many people change it before a claim is submitted to a payer?  How many different sets of rules are there and who enforces them?  A few months ago, I went for an elective outpatient surgical procedure, but due to complications they were unable to operate.  Two days later, Blue Cross notified me of a claim for the full procedure.  When I called to contest it, they told me they couldn’t second-guess the doctor’s office.  Huh??  They paid for the un-performed procedure in full.  Wither our scarce healthcare dollars? 

(To be continued……)

photo by marisag via Flickr (Creative Common License)

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

’Tis the Season for a Data Governance Carol (Part Two)

By Rob Paller, Consultant

 

The writing was on the wall, and Ebenezer was beginning to comprehend that while the mergers and acquisitions looked good on Wall Street, the company was imperiled by the aftermath of poorly integrated systems, an absence of proper data management, and a gaping void in the governance of the company’s data. Furthermore, Ebenezer knew that one more ghost had yet to call upon him. Never before had the fear of uncertainty enveloped Ebenezer, but tonight was different.

The Ghost of Data Governance Future

No sooner did Ebenezer begin to calm his nerves, when from the shadows of Ebenezer’s office, the Ghost of Data Governance Future appeared. Ebenezer’s pulse quickened as he did not want to acknowledge the ghost’s presence or what the ghost would ultimately share with him. It was with great reluctance that Ebenezer agreed to go with the final ghost.

Much to Ebenezer’s surprise, their travels did not take them very far into the upcoming decade before trouble surfaced. Poor Bob had suffered through another year and a half of cobbling reports together before reaching his saturation point and submitting his resignation. The remaining analysts scrambled to glean what they could from the frazzled, empty shell that was once the brightest among them. Surely a lone analyst would not beckon the fall of a Fortune 500 company, but it would begin the unraveling that followed in the months to come.

After Bob left the company, the remaining analysts fought to produce the fiscal reports on time. The data fixes that Bob orchestrated in the various systems to reconcile the reports were not being done. Some analysts were entertaining the idea of bringing in an outside firm to try and help address the problems, but IT was adamant they could shore up the problems on their end. Slowly the company’s competitive advantage began to erode and confidence on Wall Street went with it. It wasn’t long before rumors of bankruptcy began circulating. By then, Ebenezer had seen enough.

He couldn’t stand to see the company he so proudly guided through the first decade of the new century crumble before his eyes. He asked the Ghost of Data Governance Future if it was too late to turn things around. And if he promised to acknowledge all the data management, quality, integration, and governance issues that had been brought to light, would the company be able to avoid an inevitable collapse? But the ghost made no promises and offered no reassurances that the company could be saved. He simply returned Ebenezer to his office and quietly retreated back into the shadows of the office.

Ebenezer Wakes Up

Ebenezer, startled awake by the sensation of falling, quickly realized that he was still seated in his office chair. However, the relief was short-lived as he wiped the sleep from his eyes, only to see that the once-organized stacks of paper on his desk were strewn across the floor. Dream or no dream, the gravity of the situation was beginning to weigh on him:  The single most vital asset neglected for the past decade would be responsible for saving or crippling Ebenezer’s company.

There would be no quick solution. In order to build the momentum necessary to stay the course, Ebenezer needed to identify projects that would make an impact with the right sponsors who would, in turn, spread the word.

Slowly, over the course of several quarters, Ebenezer began to see the results of his mission to properly manage his company’s data as an asset. There were policies in place that supported the data governance initiative, and there were regular meetings to review the company’s progress with regards to an industry-approved maturity model. Data quality was being driven back to the source systems whenever possible.

There were formalized processes in place to deal with exceptions. There were master data hubs that dealt with integrating the various sources of master data across the company.  Data was beginning to move with the precision of a fine time piece.

---

Rob Paller is an expert at business analytics and database administration. Since joining Baseline, Rob has been responsible for developing a case analysis system to streamline the oversight of food assistance benefits, implementing a common citizen data model, and assisting in the rollout of a new public assistance data model integrating data from over 10 years of legacy with a new benefit eligibility determination system. He also believes in Christmas, and he'll gladly give you a shilling to buy the prize turkey for Christmas  day.

’Tis the Season for a Data Governance Carol

By Rob Paller, Consultant

Charles Dickens wrote the classic story A Christmas Carol over 165 years ago, and since then, there have been countless adaptations from the stage to the silver screen. While watching Disney’s latest adaptation with my family, it dawned on me not long after Scrooge declared Marley “dead as a doornail” that there was one adaptation of this classic that isn’t nearly as dead as Marley. It was when Marley came back to visit Ebenezer that this adaptation started to solidify.

In this adaptation, Ebenezer is played by the CEO of a Fortune 500 company, Marley is the CIO, and Bob Cratchit represents one of the many white collar workers at this company.

This tale begins with the present-day departure of Marley from the company. The past decade had presented this company with the usual challenges: Y2K, September 11, the market meltdown, and lingering economic woes. The company had weathered most of these storms but faced an uncertain future—even after completing several successful mergers and acquisitions. Ebenezer was a frugal CEO, but even he knew that the current economic climate would be hard on the company. Even with careful planning.

Ebenezer was working late one night in December trying to plan for the upcoming year as the year-end figures were coming in. He realized that with several companies still reporting their information independently, it was going to be another long night. As each minute passed spent digesting spreadsheet after spreadsheet, Ebenezer fought to stay awake—but it wasn’t long before Ebenezer drifted fast asleep. He would soon be visited by Marley in his dreams.

Marley was shackled in chains, carrying the burden of every system introduced by a past merger and acquisition—only to be poorly integrated. Marley warned Ebenezer that if these systems were not properly integrated, if the data was not properly managed and mastered, and if they didn’t get a grip on their data governance, he could either risk being ousted by the board or losing the company under the weight of these mergers and acquisitions. To make sure Ebenezer understood the gravity of the situation, Marley told him he would be visited by three more apparitions.

The Ghost of Data Governance Past

The Ghost of Data Governance Past was the first to visit. This ghost would bring Ebenezer back to see how the company operated long before the mergers and acquisitions. While things were far from perfect, the issues surrounding data quality, compliance, security, and master data management were easier to address. This was a time long before many of these issues were on the radar of many CIOs. Back then, data warehouse teams did their due diligence to mitigate the risks associated with the lack of formalized processes and technology solutions to address these problems

As a result, there was a semblance of a single version of the truth and trust placed in the reports produced. But as the decade went by, the company grew organically, and through each merger and acquisition, the ability to integrate information became impossible. Ultimately, the company was left supporting federated systems that fed multiple warehouses and relied on multiple reporting systems.

Upon Ebenezer’s return to the lobby of his office building, he rushed to the elevators in a panic. Now that he had seen the 30,000-foot view of the path his company had traveled, he was scrambling to figure out how and where to start solving his company’s massive data disaster. Unfortunately, for Ebenezer, he would be visited by the next ghost before he could jot down a few scant ideas.

The Ghost of Data Governance Present

The Ghost of Data Governance Present was seated at Ebenezer’s desk and offered Ebenezer a seat in one of his own guest chairs. The ghost laughed while looking over the stack of reports that Ebenezer had left behind. With a swipe of the ghost’s arm, the stack of papers flew through the air, landing haphazardly across the floor. The ghost went on to explain that it didn’t matter whether the reports were collated or not—they couldn’t be trusted any longer. The ghost only needed to bring Ebenezer back a few weeks to show him all the trouble that Bob Cratchit had gone through to finalize the numbers for these reports.

Bob scoured over the numbers—only to be forced to dive deep into the tables in the countless systems to validate the numbers. When Bob discovered what he thought was a data quality problem, there was no process in place to fix the data. The subject matter experts could only commiserate with Bob about the problems and had little recourse in solving them. When all was said and done, Bob’s effort required a 60-hour week every fiscal period. Before he had a single iota of confidence in the results. Before he could send them on to Ebenezer.

The ghost reminded Ebenezer that he had only witnessed a single day of Bob preparing these reports, and that Bob had been suffering through this for years. He was the lone analyst capable of producing these reports since he had been around the longest and understood the intricacies that each new merger and acquisition introduced. But Bob could only be the unsung hero for so long. Eventually he would burn out and move on. The thought that Bob could leave the company was a burning question that Ebenezer tried to get answered, but the ghost of Data Governance Present offered no affirmation of Bob’s actions. The ghost returned Ebenezer to his office to ponder an uncertain future.

To be continued...

---

Rob Paller is an expert at business analytics and database administration. Since joining Baseline, Rob has been responsible for developing a case analysis system to streamline the oversight of food assistance benefits, implementing a common citizen data model, and assisting in the rollout of a new public assistance data model integrating data from over 10 years of legacy with a new benefit eligibility determination system.

Real-Time Data Quality Through MDM

By Bob Wall, Senior Consultant

I have found that one of the biggest challenges for data migration, data integration and data warehousing projects that I’ve worked on, is the requirement for clean data. Typically, one of a plethora of data quality/cleansing tools is deployed. The tool employs sophisticated heuristic, probabilistic, deterministic, phonetic, linguistic and empirical methods and algorithms to perform data quality analysis. For example, for customer data being integrated into a data warehouse, we want the tool to reconcile Easthartford, Hartford East, Hartford and East Hartford to the same physical address. The data quality process is usually run in a batch process, first for the initial load and then continually after that.

Today, operational MDM hubs present an even bigger challenge. We have to ensure synchronization across multiple source systems and all data must consistently be correct. The data quality checks must be applied at various stages within the master data lifecycle, and must support federated sharing across and between systems, databases and applications. In a sense operational MDM requires real-time data quality.

Digressing for a moment, I have also worked on projects that employed various search engine technologies. On one in particular, a client wanted to have an automated way to categorize structured and non-structured web data (emails, documents, images, etc.); search for certain conditions and do it in an automated fashion. We investigated some technology that used what was referred to as the semantic web technology, which incorporated advanced semantic and linguistic analysis with classification schemes (using Hyper Text Markup Language (HTML), eXtensible Markup Language (XML), Resource Description Framework (RDF), and Web Ontology Language (OWL)) to render web content machine-readable and make it capable of being searched in an automated fashion.

Software tools that support inline SOA data quality services combined with advanced semantic and linguistic analysis/machine learning capabilities are beginning to evolve. Microsoft’s purchase last year of Zoomix is a testimonial to the strategic value of these types of products. The convergence of data quality and semantic web technologies may provide operational MDM projects with the ability to automate ongoing classification, matching, and standardization of master data records. I think it is definitely worth keeping an eye on to see if it leads to real-time data quality capabilities embedded in MDM tools.

photo by Blude (via Flickr)

---

Bob Wall is a senior consultant with Baseline Consulting. He is an information technology specialist with 30 years experience in all areas of data warehouse administration, data architecture, data resource management, training, and applications systems development, as well as in corporate management.

Disruptive Technologies: Governance in Action

By Carol Newcomb, Senior Consultant

photo by roland

I recently consulted on a Customer Data Governance program implementation project.  I must say, most companies get to the point where they design to death what governance should look like, how it should operate, who should be on which committees and who should have specific decision rights.  But how often have you ever seen data governance work well?  What does a governance policy even LOOK like?
Yeah, yeah, governance is the “4 pillars of People, Processes, Decisions Rights and Controls”, or it’s the “5 boxes of Alignment, Oversight, Measurability, Visibility, and Transparency”, but come on—what does it FEEL like?  How does it really WORK??  What makes it happen??  How can people understand it when they are told it is so important?  Who does what?

Face it.  Governance is disruptive.  We speak of disruptive technologies, like the cell phone or FaceBook, the Hybrid automobile, or the laptop computer.  These were all disruptive in their day.  They changed the way we think about social networks, distances between people and getting around.  Governance is disruptive because it seizes control from the hands (silos) of data analysts, IT staff and report writers, and puts it squarely into the laps of business people who (unknowingly) need the data to make accurate and timely business decisions.  Oh, and did we mention—share the data with each other?  Just think of the power of sharing trusted data and how much more efficient and effective the business would become with strong dependencies instead of weak, redundant independencies.

“Don’t put those decisions in MY LAP.”  We hear this all the time.  Why not?  It’s your bread and butter.  Stand up and be accountable for the fact that you don’t trust the data, but you still report it up the chain of command.  Show that you’re willing to expose who made what data decision, when, and how broad the impacts on actually are.  Be proud of the data that you use to run your portion of the business.    This is uncomfortable stuff.  Trust?  You want me to TRUST the data you lob over the fence? 
Well, let me think about that and I’ll get back to you.

Governance is disruptive because it breaks down the weaknesses we have that permeate most corporations, no matter what industry you’re in.  It forces people to talk to each other.  It forces people whom we call Data Stewards, to beat the bushes, get people to admit what they’ve been doing and how they’ve been doing it.  It wrests the decisions out of the IT cubicle and puts it squarely on the Director’s desk.  Data quality?  Let me show you our data quality.  Ouch.  Metadata?  How do you spell that?

So, get used to the fact that all change is disruptive.  But, the changes that are REALLY disruptive are those that break down the social and cultural barriers (aka personal barriers) that corporations have allowed to take control of their data, weakened its use as a decision-making tool, and contributed to massive cash outlays in data correction, cleaning and rework just to manipulate data so that it can serve its ultimate purpose: to measure the effectiveness of decisions made and to guide more profitable decisions in the future.  It’s that simple.

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

Defining Data Requirements

By Fernando Martinez Campos, Senior Consultant

photo by clayirving

In my last blog post I described a way to define BI metrics through business user interview techniques. Now it’s   time to start investigating what data is needed to answer the business questions. This post discusses data requirements, a subset of business requirements gathering.

A fundamental step in defining data requirements is to build a logical data model (LDM) which reflects the natural way data relates to the business. Aside from interviewing users and building an LDM, another input involves analyzing the sources of this data: operational applications and departmental databases.

Data requirements should be a precursor to data source analysis and profiling. A mistake many analysts make is to assume that the source is good enough to run BI Analytics. Once they load the source, much to their chagrin they find “data surprises” that will make the analysis unreliable. So it is extremely wise to being data source analysis by interviewing the SMEs and ask these fundamental questions:

• Does the data contain duplicates?
• Is there occasional missing data in some of the fields?
• Is there a better source for this data?
• Is the data content recorded accurately and reflect the business event?
• Are the business rules defined when adding or changing the data?
• Is there a data owner responsible for fixing it?

I am always surprised how much users know about errors in their data. They are invaluable in pinpointing where the errors are in the sources. Your initial efforts in the data quality investigation will uncover these “data hot spots” from them. When you start profiling the data to obtain more accurate results on exactly where all the problems exist, you’ll have a great headstart With the data profiling results you will be prepared to proactively handle any unexpected problems with the data before you load the data.

Besides creating the LDM, you will also need to create a Business Requirements Document (BRD) that will have a data requirements section. These are the activities needed to build this section:

• Define the data with a precise meaning and content, in other words, the metadata
• Determine how often the data needs to be loaded
• Determine the cutoff time for the data
• Analyze audit and compliance considerations
• Define the performance requirements and response times in retrieving the data
• Determine if the data has special security and privacy needs
• Define the length of history needed for the data

Creating a BRD streamlines the design phase with these benefits. You can uncover early security, audit and performance considerations that will need to be done in the physical implementation. And data profiling helps you proactively minimize any data quality problems during the load process. Finally, creating an LDMprovides the foundation for the physical data model (PDM) which becomes the de-facto data structure for the tables on your data warehouse.

---

Fernando Martinez-Campos is a senior consultant with Baseline Consulting and expert in data architecture, data interchange standards, legacy coexistence strategies, and reference architectures templates for infrastructure, applications and databases.