In the Field with our Experts

Data Architect – Who is right for the job?

by Caryn Maresic, Senior Consultant

The Data Architect is the core of any BI team.  It is important to choose a person with the right skill set.  As I tried to put together a list of skills I looked to IT Toolbox and Database Answers for help, but my mind wandered a bit.  System Construction. Data Architect. Data Warehouse. Software Factory.  We like to portray what we do in terms of construction and/or manufacturing.  A recent client bemoaned her departments inability to move from “building custom cars” to “an assembly line”.  Comparing ourselves to these burly industries might make us feel strong, but it does it accurately represent what we aspire to be?

What is a Data Architect?  What should they know how to do?   I borrowed the following description from this article. Before you click, read on and see if you can guess what this is really describing.  I think it is a great description for a Data Architect:

A Data Architect is qualified by education, experience, and imagination to enhance the function and quality of systems. The purpose of this pursuit is to improve the quality of life, increase productivity, and protect the health, security, and welfare of the business.

The best Data Architects are capable of analyzing a client's needs, goals, safety and business requirements and integrating this information into a design that is both pleasing to the eye and functional. They will work with the client closely to develop preliminary design concepts that meet their aesthetic, functional, and economic needs while maintaining adherence to standards.

In essence, the best Data Architects are part detective, part artist, and part psychologist and they use these skill sets to create systems that fit a client's tastes and needs with their budget in mind.

Doesn’t that sound like a great job?  Sign me up!  What this is actually describing is an interior designer.  While I doubt that HGTV has any plans to showcase the next dashboard you build, we are indeed closer to Designing Women than Rosie the Riveter!  Stay tuned for future posts on the talents of a good Data Design Star.

photo by Annahape Gallery via Flickr (Creative Commons License)

---

Caryn has over 20 years experience in providing high-quality data solutions to clients in the areas of Business Intelligence, Data Warehousing and System Integration.  Caryn has expertise in across industries with an emphasis in Pharmaceutical, Manufacturing, and Insurance.  Prior to joining to Baseline, she ran her own consulting company.

I’m All Ears

By Caryn Maresic, Senior Consultant

Most Data Warehouse designs include constructs for Address, Phone, and/or Email for Customers.  Len Silverston came up with what he calls a Universal Data Model that does a very good job of abstracting address, email and phone number data.  I have seen clients use the Contact Point portion of his model as-is and with a few simplifications with great success.  That being said, in the area of Marketing and Sales, the manner in which we reach out to our customers and prospects gets more diverse every day.  Disneyland has just partnered with Verizon so that park guests can get real time information about the park and play Disney games on their phones….and, of course, Disney gets access to more information about its customers!

How does this new and ever changing world of communication change the way we think about and model contact points?  What would my “address” look like if I were near the Haunted Mansion looking for a lunch spot?  Would it be different than if I were at Downtown Disney looking for a cup of coffee?  On Main Street looking for Winnie the Pooh?  In all instances I would be using the same phone, possibly the same IP address, but I would be in different locations which would be important to the marketeers at Disney.

As time goes by (and cell phone GPS systems become more accurate) I suspect that the way we run marketing campaigns to smart phones will be similar to the way in which we use billboards today.  Where the customer is physically located at any given time will be as important as the phone number and/or IP address, thus creating a two dimensional contact point.

Have you come across this issue in your organization?  Have you changed your data model to include two dimensional contact points?  If not, has the use of smart phones changed your data model in other ways?

photo by wrayckage via Flickr (Creative Commons license)

---

Caryn has over 20 years experience in providing high-quality data solutions to clients in the areas of Business Intelligence, Data Warehousing and System Integration.  Caryn has expertise in across industries with an emphasis in Pharmaceutical, Manufacturing, and Insurance.  Prior to joining to Baseline, she ran her own consulting company.

A Data Governance Primer (Part 3 of 3)

By Carol Newcomb, Senior Consultant

Diamond in the Rough: Data Quality

The third part of my summertime primer addresses Data Quality Analysis.  Don’t even start a data quality analysis until you have completed the first two steps of your Root Cause Analysis--investigate & prioritize any potential causative factors, and start your metadata assessment.  Otherwise, you may be misled by your findings.

Data quality is defined as complete and accurate data that is ready for business consumption.  Sources of poor data quality may include lack of data entry rules, unclear data element definitions, inconsistent metadata definitions for field type, format or intent, or breakdowns in data transformation processes as data flow between systems or applications.  Poor data quality results in bad business decisions; it contributes to major problems in using data effectively, and costs companies millions of dollars/year in terms of rework and inefficiency.  Data quality, in combination with robust metadata definitions, is part of the foundation of good data governance.

Data Quality Analysis

A Data Quality Management process should be designed to enable an area to start with a simple approach and over time to mature to one that is more proactive and comprehensive.  Initially, investigation may be focused on single data elements or events.  As patterns, data commonalities and other relationships appear, the data quality management process will grow to support complete business processes.   A mature data quality management process will not just resolve individual issues; it will also track relationships between data elements, ensure that business rules are consistent and generate statistical analyses to monitor previously addressed issues to ensure that data quality is stable and that an early warning system is in place as part of the data governance program.  The goal is to design a data quality management lifecycle, as shown in this diagram:

Initial Data Quality Analysis Process

I. Define data scope

• Determine data elements that are associated with or are direct results of the reported issue
• Check that all metadata definitions are present and current
• Enlist the involvement of the Data SME or Data Stewards
• Identify all source systems where the data originates, is  entered or derived

II. Extract and profile the data

• Extract the relevant data from all key source systems.
• Design the profile.  A profile will consist, at a minimum, of total record counts, min/max values, frequency of unique values, and frequency of invalid values (if defined) for each data element profiled. 
• Profile the data to determine key characteristics that are contributing to the issue, such as:
• 1. Wrong values
  2. Missing values
  3. Corrupt transformation processes
  4. Incorrect business rules
  5. Incorrect usage rules

III. Analyze Data Profile Results

• Summarize the key findings from the profile detail
• Determine what key drivers are contributing to the impact
• Determine accountability for the data quality issue
• Involve other Data Stewards in troubleshooting and designing the data quality solution

IV. Design the Corrective Action Plan

Two types of plans should be developed to address known data quality issues: a corrective action plan to fix the immediate source of the problem identified, and an ongoing monitoring plan, where thresholds have been determined and metrics are routinely collected and reported to data stakeholders.  This monitoring process should be scalable based on the number of data elements being tracked.

1. Corrective Action Plan
2. • Does scope of problem warrant change in metadata definitions, business practices or data entry rules?
   • Does scope of problem warrant a data governance standard?
   • Does the corrective action plan include details on how to fix the source of the problem as well as ways to correct historical data in the system?
3. Preventive Action Plan
4. • This plan will be designed to minimize the probability of data quality issues from recurring
   • Determine ‘early warning triggers’ based on designated thresholds.  These thresholds should reflect the business tolerance for inaccurate data (is 95% acceptable?)
   • If data latency is the source of a data quality issue, then latency thresholds should be included in the monitoring plan
   • Determine how frequently results of the monitoring plan will be reported to data stakeholders or governance oversight committees

So, now that summer is officially here, this wraps up my Data Governance Primer series.  Time for some iced tea and my favorite beach towel.  Come August, these little refreshers might be just the thing!

photo by Swamibu via Flickr (Creative Commons License)

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

A Data Governance Primer (Part 2 of 3)

By Carol Newcomb, Senior Consultant

Minding Your Metadata

The second part of my summertime primer addresses ‘Minding your Metadata’.  I can just hear the collective groans and yawns now.  Sorry, but metadata collection is one of those necessary evils that may not be fun in the doing, but having it available as a resource to understand your data and use it appropriately is invaluable.  And you just might find some interesting surprises along the way!

Metadata: What Is It & Why Do I Need It?

As you start your Root Cause Analysis (see last week’s primer), you first need to examine existing data definitions (or lack thereof).  Metadata is the foundation of good data management and forms the basis for Data Governance.   Pardon me for stating the obvious, but metadata is fundamental to investigating and resolving data issues and it is the first place to start when investigating data quality issues.

Metadata is “data about data”.  Plain and simple.  It includes descriptive information about electronic data used in common daily business practice.  Metadata includes items usually found in a data dictionary: field name, field length, retention rules, and security access, as well as additional descriptive information that may include data origin (source or system), creation/entry date, method of creation (key-entry or the result of a calculation), purpose of the data (its intended use), how frequently it gets updated or refreshed, and current location in a database (table, view, schema).  If a data element is the result of calculation logic or groupings (such as age categories), those business rules used to generate the resulting data values should be collected as part of the metadata.

A good example of metadata that you may use every day would be ‘document properties’ in a Word document.  This feature captures data on the original document creation date, most recent access and update times, document creator, count of characters, words and pages.  If the document should be private, this will be indicated in its properties.  You may also tag the document by indicating key words in order to make it easier to find by you or others.

A few of the benefits of Metadata Management include:

• Clarify rules for data entry
• Reduce ambiguity around appropriate use of data elements
• Eliminate problems associated with not having data definitions, business rules or transformation logic available
• Validate legitimate values at the data element level
• Provide evidence to regulators that security and confidentiality are protected
• Centralize the storage and accessibility of metadata for end-users
• Reduce the amount of effort required to research data results.

A Metadata Management Repository is a central location or system to collect and store metadata that may exist in disparate parts of the organization (data dictionaries, systems, spreadsheets, or people’s brains). The metadata repository will store detailed definitions centrally on a network where other users can find it.

There are three general sources of metadata that should be included in this repository:

Business Metadata – Business metadata attributes facilitate identification, understanding, and appropriate use of existing data elements.  These include clear business names and descriptions, relevant business rules, descriptions of the data sources, security and privacy rules, etc. 

Technical Metadata – Describes the technical attributes of data such as physical location (host server, database server, schema, etc.), data types, any transformations applied and domain of valid values, relationships to other data elements, precision, and lineage.  Technical metadata is used by business users and by IT staff to design efficient databases, queries, and applications, and to reduce duplication of data. 

Operational Metadata – Describes the attributes of routine operations on data and related statistics.  These include job schedules and descriptions, data movement and transformation processes, data read, update and performance statistics, volume statistics, backup and archival information.  Operational metadata is used by operations staff, and DBA’s to tune the system and ensure its continued efficient operations.  It is also used by business users to track such events as “last use” of a field, and “last load” of a data element.

Exciting stuff, huh?  Well, the whole point of metadata is to have the information about data available to a multitude of users when they need it, to keep it current, and to avoid confusion around usage.  So if you appreciate having a clean bathroom, and knowing where you keep your antiperspirant, you will also appreciate having good metadata!  The time for spring cleaning is well overdue.

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.

MDM and the Psychology of Data Management Maturity

By Bob Wall, Senior Consultant

I recently read an interesting article on the web entitled “Ascending the Data Infrastructure Hierarchy: The Five Stages of Data Infrastructure Maturity." ¹

In it the author discusses Maslow’s self-actualization psychology² and the Hierarchy of Needs (further information here) and creating an enduring Data Management Infrastructure. I believe it has implications to achieving data management maturity, which is a fundamental ingredient for MDM success.

As Baseline partner Evan Levy has explained in his MDM presentations, MDM is comprised of both “(Master Data) Management—capabilities to support master subject areas such as Customer, Product or Account—and Master (Data Management), in other words sustaining data management via standards, guidelines, and procedures. This includes the following Data Management functions:

• Data stewardship
• Data architecture, analysis and design
• Database administration
• Data security management
• Data quality improvement
• Reference/master data management
• Metadata management  


In trying to better manage these data management functions, reach an optimal level of data management maturity, and support a successful MDM implementation, Maslow’s hierarchy of needs and self-actualization psychology provides some helpful insight. In his epic psychology writing, Maslow organized human need into three broad levels: first, the physiological - air, food and water; then the psychological - safety, love, self-esteem; and finally, self-actualization. His insight was that the higher needs were as much a part of our nature as the lower, indeed were instinctive and biological. Maslow's greatness was in re-imagining what a human being could be.

The following schematic shows Maslow’s Hierarchy of Needs and an analogous Data Management Hierarchy.

 

Maslow’s Hierarchy of Needs and an analogous representation of a Data Management Hierarchy





We can see the 5 areas of the Data Management Hierarchy, mimicking Maslow’s Hierarchy of Needs, as follows:

Level 1: Tribal – Basic Needs (survival) 

At this level, an organization functions from day to day, but without any formal processes or systematic management. Instead, data management is left to the decisions of knowledgeable and skilled individuals.

Level 2: Enforced – Safety Needs (comfort)

As a company advances to the next level of the Data Management Hierarchy, the Enforced stage, it begins to adopt some semblance of governance, which includes documented standard operating procedures and putting in place change controls to better manage workflow within its data environment.

Level 3: Standardized – Psychological Needs

As organizations advance from the Enforced stage to the next level, they reach the Standardized stage, in which, as the name suggests, processes are established to handle various aspects of data management and less importance is placed on individual decision making.

Level 4: Actualized – Self-Actualization

In the Actualized stage, organizations are able to start getting more creative with their data. Data can be extracted from across the enterprise - as well as from outside sources - and combined to create useful information and made actionable to create new knowledge and value added business propositions.

Level 5: Peak Performance Peak Experiences

At last, the organization reaches the height of data management development, where it can devote all its resources to high-level strategic initiatives, rather than administrative issues.

MDM success is directly proportional to how well an organization integrates data management principles and reaches a peak performance/experiences maturity level.  Hence we can see the possibilities of the gradual application of Maslow’s Psychology of the Hierarchy of Needs, leading to the ascendance of data management maturity and re-imagining what a successful MDM project could be.

¹ John Bostic, Ascending the Data Infrastructure Hierarchy: The Five Stages of Data Infrastructure Management Maturity (Auerbach Publishing, 2007).
² Abraham Maslow, A Theory of Human Motivation (Psychosomatic Medicine, 1943).

---

Bob Wall is a senior consultant with Baseline Consulting. He is an information technology specialist with 30 years experience in all areas of data warehouse administration, data architecture, data resource management, training, and applications systems development, as well as in corporate management.

The Metadata ROI Series: Episode 4: Program Management

By Stephen Putman, Senior Consultant

This posting is the fourth in a series discussing various aspects of the financial justification of a metadata management system implementation that are not obvious in many existing analyses. Today, we will discuss the time and effort saved by Information Technology program management staff in system implementation projects.

In Part 3 of this series, the return on investment in a metadata system was shown in relation to end-user reporting groups in your organization. In the fourth and final installment in this series, we will look at how a metadata system can strengthen the program management of the entire system development process.

A great deal of a project manager’s day is spent assembling various lists of functionality to be released into the production environment, and checking the progress of these pieces to ensure that everything is completed in time for the scheduled release. Much of this work is done manually by checking with the various producers, either in person or by status report. It is, frankly, a complicated juggling act that introduces risk into the release process that does not need to be present.

With a robust metadata system, participants in the development and testing of data solutions have a central repository to place updates on the status of various pieces of the product. Unfortunately, status reporting is something that is often neglected, and therefore such a system could easily fall into disrepair. There is a solution that serves all parties, however - the program management group enforces a policy that no piece of software or reporting moves into another state in the standard development-testing-production progression unless the metadata for that piece is complete. This way, the program management staff has an up-to-date view on the status of the entire feature list, and the development staff has incentive to enter data into the system, which benefits all of them during the process and the end-users when the features are released into production. This ultimately saves the organization money in increased efficiency of the entire staff.

A robust and comprehensive metadata system has many benefits to several areas of your organization, and along with effective measurement can provide a large return on your investment in either a purchased or “home-grown” system. It is the basis for effective software development and support in your organization, and should be seriously considered as a best practice for a successful IT function.

photo by perhapstoopink via Flickr

---

Stephen Putman has over 20 years experience supporting client/server and internet-based operations from small offices to major corporations.  He has extensive experience in a variety of front-end development tools, as well as relational database design and administration, and is extremely effective in project management and leadership roles.

The Metadata ROI Series: Episode 3: Report Users

By Stephen Putman, Senior Consultant

This posting is the third in a series discussing various aspects of the financial justification of a metadata management system implementation that are not obvious in many existing analyses. Today, we will discuss the time and effort saved by reporting users in business organizations.

In Part 2 of this series, we took a look at how the software development process is aided by a robust metadata system. In this installment, we will look at the support that a metadata system can give to the end-user reporting groups throughout your organization.

One of the main things that concern end-users is the definition of business terms and how these definitions are represented in the various computer systems that they use daily. This is a huge risk exposure to the enterprise because confusion on definition of terms could lead to conflicting data reported by different groups in the organization, in extreme cases causing financial restatements at potentially great cost to the organization.

A comprehensive metadata system gives reporting users a consistent base of business definitions, data representations, and linkages between the data to mitigate the risk of misstatement of organizational data. It makes report producers more efficient, and reduces service calls to the IT organization for assistance on the business meaning of data. The returns gained by a metadata system can be measured in the reduction of IT service needs and the mitigation of post-auditing of reporting results that often occurs when definitions are unclear.

Another benefit of a robust metadata system involves the integration of operational metadata to determine the “freshness” of the data in reporting systems. The activity of loading of data into reporting structures can be recorded to the data element level, which is utilized by reporting users in dating their output, ensuring the most current data in the report. This instills further confidence in the results produced by reporting users, and although it is nearly impossible to measure this in monetary terms, it increases the general goodwill of the organization.

In the final installment of this series, we will examine the usage of a metadata system by the software development program management organization.

photo by Engin Erdogan via Flickr

---

Stephen Putman has over 20 years experience supporting client/server and internet-based operations from small offices to major corporations.  He has extensive experience in a variety of front-end development tools, as well as relational database design and administration, and is extremely effective in project management and leadership roles.

The Metadata ROI Series: Episode 2: Software Development

By Stephen Putman, Senior Consultant

This posting is the second in a series discussing various aspects of the financial justification of a metadata management system implementation that are not obvious in many existing analyses. Today, we will discuss the time and effort saved by software development staff in system implementation projects.

In Part 1 of this series, we examined the value of a metadata management system in regard to the gathering and documenting of business requirements. In this installment, we will take a look at the next phase of a system implementation - software design and development. A metadata management system can help you save money and time in these areas as well.

For new system development, the system designers and developers can use the data stored in the business requirements process as a basis for their designs. Properly documented requirements leave little question as to the intent of the system, and reduce (or even eliminate) the need for follow-up meetings between the engineering staff and the business analysts. Measuring this saved time by the assigned resources is your first source of return for the investment.

Two other major areas of development that benefit greatly from metadata are Extract-Transform-Load (ETL) development and report development. Both functions need to know where data is located on various systems, which is something a metadata system does well. For ETL development, the location and mapping of source data elements is also available. Developers contribute specific information related to their activities to the metadata system to demonstrate fidelity in design to requirements and design considerations introduced to the system.

The major return on investment of a metadata system in the software development area is primarily related to the time saved by staff in information discovery and application. It is important to measure both pre-metadata-system and post-metadata-system metrics on developer and analyst time to accurately determine the money saved by implementing the metadata system.

In Part 3, we will tackle the potential cost saving to end-user reporting groups in your organization.

Photo, "physical metadata," by Nadya Peek (via Flickr)

---

Stephen Putman has over 20 years experience supporting client/server and internet-based operations from small offices to major corporations.  He has extensive experience in a variety of front-end development tools, as well as relational database design and administration, and is extremely effective in project management and leadership roles.

The Metadata ROI Series: Episode 1: Business Analysis

By Stephen Putman, Senior Consultant

This posting is the first in a series discussing various aspects of the financial justification of a metadata management system implementation that are not obvious in many existing analyses. Today, we will discuss the time and effort saved by Information Technology business analysts in system implementation projects.

In a database implementation project, one of the first activities performed is the gathering of business requirements. These requirements are gathered from business users mainly by interviews and document inspection. The business analysts (and data modelers) take this information and produce documentation describing these requirements, and often perform data mapping between systems when data movement is required.

A basic function of a metadata management system is to describe data elements within computer systems and record attributes of these elements, such as the resident table(s), data types, business rules, and so on. A metadata management system makes it easy for the business analyst to gather relevant information about the existing data elements involved in the data movement in the system, and aids in the design of the data store for the new system by facilitating the data mapping exercise. These two simple uses of the repository save a tremendous amount of time and effort by the analysts in these activities.

It is a simple matter to quantify the cost savings of maintaining a metadata management system for business analysis activities by determining the time spent by staff on collecting and transcribing source data element definitions, subtracting the (very small) time spent extracting the same information from the metadata management system, and multiplying the result by the salaries of the business analysts involved in the project. This result is placed in the “return” bucket of the Return on Investment calculation. This savings is also something that continues to pay dividends over time when more projects utilize the metadata repository in this way.

This example shows that metadata management systems positively affect several areas or your IT organization that are not immediately apparent. I will be detailing more of these functions over the next few months.

---

Stephen Putman has over 20 years experience supporting client/server and internet-based operations from small offices to major corporations.  He has extensive experience in a variety of front-end development tools, as well as relational database design and administration, and is extremely effective in project management and leadership roles.

Disruptive Technologies: Governance in Action

By Carol Newcomb, Senior Consultant

photo by roland

I recently consulted on a Customer Data Governance program implementation project.  I must say, most companies get to the point where they design to death what governance should look like, how it should operate, who should be on which committees and who should have specific decision rights.  But how often have you ever seen data governance work well?  What does a governance policy even LOOK like?
Yeah, yeah, governance is the “4 pillars of People, Processes, Decisions Rights and Controls”, or it’s the “5 boxes of Alignment, Oversight, Measurability, Visibility, and Transparency”, but come on—what does it FEEL like?  How does it really WORK??  What makes it happen??  How can people understand it when they are told it is so important?  Who does what?

Face it.  Governance is disruptive.  We speak of disruptive technologies, like the cell phone or FaceBook, the Hybrid automobile, or the laptop computer.  These were all disruptive in their day.  They changed the way we think about social networks, distances between people and getting around.  Governance is disruptive because it seizes control from the hands (silos) of data analysts, IT staff and report writers, and puts it squarely into the laps of business people who (unknowingly) need the data to make accurate and timely business decisions.  Oh, and did we mention—share the data with each other?  Just think of the power of sharing trusted data and how much more efficient and effective the business would become with strong dependencies instead of weak, redundant independencies.

“Don’t put those decisions in MY LAP.”  We hear this all the time.  Why not?  It’s your bread and butter.  Stand up and be accountable for the fact that you don’t trust the data, but you still report it up the chain of command.  Show that you’re willing to expose who made what data decision, when, and how broad the impacts on actually are.  Be proud of the data that you use to run your portion of the business.    This is uncomfortable stuff.  Trust?  You want me to TRUST the data you lob over the fence? 
Well, let me think about that and I’ll get back to you.

Governance is disruptive because it breaks down the weaknesses we have that permeate most corporations, no matter what industry you’re in.  It forces people to talk to each other.  It forces people whom we call Data Stewards, to beat the bushes, get people to admit what they’ve been doing and how they’ve been doing it.  It wrests the decisions out of the IT cubicle and puts it squarely on the Director’s desk.  Data quality?  Let me show you our data quality.  Ouch.  Metadata?  How do you spell that?

So, get used to the fact that all change is disruptive.  But, the changes that are REALLY disruptive are those that break down the social and cultural barriers (aka personal barriers) that corporations have allowed to take control of their data, weakened its use as a decision-making tool, and contributed to massive cash outlays in data correction, cleaning and rework just to manipulate data so that it can serve its ultimate purpose: to measure the effectiveness of decisions made and to guide more profitable decisions in the future.  It’s that simple.

---

Carol Newcomb is a Senior Consultant with Baseline Consulting. She specializes in developing BI and data governance programs to drive competitive advantage and fact-based decision making. Carol has consulted for a variety of health care organizations, including Rush Health Associates, Kaiser Permanente, OSF Healthcare, the Blue Cross Blue Shield Association and more. While working at the Joint Commission and Northwestern Memorial Hospital, she designed and conducted scientific research projects and contributed to statistical analyses.