What might be interesting is to explore similarities of those customers within each of the classes that can be used in two different ways. The first is for segmentation purposes: to identify characteristics of specific variables that can be used proactively for new customers to predict which class they will fall into. An example might be that many of the “good” customers live in an area with a median annual household income between $75,000 and $95,000 and own their own homes. That would suggest that a new customer whose annual household income is $84,000 and lives in her own home is likely to be a “good” customer.

The benefit of this segmentation is that it can guide other decisions in the marketing and customer acquisition process. One case in point: if “good” customers live in an area with a median annual household income between $75,000 and $95,000 and own their own homes, perhaps the best place for media spend is radio ads in areas where people with those salaries own their own homes.

The second is for promotion: to determine whether there are any customers in one class that have the potential to be promoted into a higher customer classification. As an example, if one of the customers in the “fair” class has an annual household income of $82,500 and lives in his own home, there is a case to be made for trying to influence that customer’s behavior to transition that customer into a “good” one. That might mean making the customer an offer in a way that changes the score, such as urging the customer to spend more, or buy more items, or to remain a customer for a longer time. And, as we will discuss next time, one way to influence behavior is through price.

1. The business requirements were gathered, and they basically stated that the business users wanted the same reports they had now. Mistake #1: it is usually setting up a project to promise the same level of reports. Instead, offer an enhancement using words like “drill thru,” “drill across,” etc. This sets expectations that the new reports are BETTER, AND THEY SHOULD BE!
2. ETL platform chosen
3. Data model for new environment created with referential integrity
4. Project plan created
5. Resources acquired
6. Kick-off planned

During the requirements gathering and the data modeling, no one really addressed the quality of the data. No profiling, and no quick SQL queries to check out the data. This cost time during implementation. Here’s why:

1. As soon as some data didn’t load, they dropped the referential integrity and loaded garbage
2. The garbage had to be filtered out for the reports
3. The data models had to reflect the lack of database enforced referential integrity

Planning for quality is definitely in my top 10!

“Will the migrated data be able to support our business functions post-migration?”

Businesses often place their entire faith in a contractor or supplier. They think that because they have a contract and the supplier has a “proven method” they are protected from the failure caused by data that simply isn’t ready for target system operation.

You only have to read the press and media to learn that a contract is no insurance policy.

Why does this “data readiness problem” occur?

Most businesses don’t perform large data migrations often, so project leaders and sponsors typically lack the expertise and experience to recognise the danger signs. As data migration is such a relatively immature profession, it’s all too easy to omit critical steps in the project plan that can cause poor-quality data to wreak havoc in target systems.

What should businesses be doing to ensure data readiness during data migration?

Business leaders need to take action from the outset because data migration is a business initiative. Yes, you’re migrating data, but the core activity is business transformation and for this reason it demands the necessary levels of business involvement.

Project leaders need to ensure key sponsors and experts from the business community are heavily involved in the decision-making processes for determining data readiness. They need to allocate their most valuable resources to ensure that post-migration the target systems will function like clockwork.

Even if your suppliers or contractors don’t demand it, your organisation has to push for greater involvement, particularly when it comes to ensuring the readiness of data.

Why is this “business integration” tactic so uncommon?

One of the most common practices in data migration is “dumping the problem on the supplier.” This is a flawed approach driven by an assumption that data belongs to IT and can therefore be wholly outsourced to IT teams, either internally or externally.

The key fact that leaders ignore is that data is not some IT by-product that can be shunted over the fence to the contractor of choice. Data drives the business functions that fuel services and business performance. Removing business involvement from the data migration life cycle is akin to saying “we don’t care about the core functions of our business.”

This is commercial suicide and a completely outdated approach. If you want to create successful migrations, the business has to get some skin in the game.

By taking an active part in the development, testing and assurance processes of data migration, the business can guide the correct decisions and actions to ensure healthy, operational data after the data migration and beyond.

What tactics are useful for ensuring data readiness during and after data migration?

• Create contracts that stipulate business involvement during each phase of the migration

• Get the business to determine which data it will need, the rules that will bind it and the quality levels required for go-live; don’t let it use the “we need everything” response

• Perform extensive data quality assessments before, during and after the data migration

• Create use-case tests that reflect the full range of functions required of the target environment

• Perform target system testing based on full-volume data loads, not small sample sets

• Convince senior management to change its perception of data migration being an IT-centric incentive; show managers the business relevance

• At the earliest possible stage, visually demonstrate to the business how its data will look when migrated across to the target system

• Set expectations up front as to what data will be migrated across and get business leaders to sign off on these expectations

• Provide assistance to downstream data users so that they can assess their own data readiness and sign off on any changes to their data feeds

• Use modern software and methods that allow metadata, business rules and data quality management processes to create great value going forward

• Develop migration architectures and software that deliver not just one-time use, but become part of the business and IT landscape, adding value for years to come

How does your data readiness plan compare to the suggested activities above? Please add your comments below if there is anything I’ve missed or anything that needs further explanation.

On his blog, Andrew Dempsey lists three high-level skills:

• Math – They know some blend of statistics, data mining and machine learning
• Code – They can do the above through programming, widget based software or a combination
• Communicate – They can effectively communicate their findings and recommendations

So, how do you find these people? And how do you know when one is right in front of you?

Are these people actually good?

As someone who knows a thing or two about HR, let me chime in. For a long time now, traditional interviews have been terrible predictors of ultimate success on the job. I’ve seen first-hand how people who looked so good on paper did very poorly in their new positions.

The hiring problem is particularly acute with respect to Big Data and data science because the terms are so poorly understood, especially by HR folks who (as a general rule) aren’t terribly skilled at analyzing data. I’d argue that many hiring managers are the least equipped to understand the true skills and abilities of the very people before them.

Of course, there are exceptions to that broad statement. And, lest I overstate things, HR rarely makes these types of hiring decisions sans consulting line employees with whom the prospective new hire will be working.

Simon Says

Rare is the recruiter who understands that hiring the right Big Data people is not about checking boxes on a checklist. Yes, for data scientists, technical skills really do matter. However, finding adroit data scientists requires more than knowing how to code in R. As I write in Too Big to Ignore, it’s more about a state of mind, a natural curiosity to solve problems and ask questions.

Feedback

How do you find your data scientists?

Just as important as performing regular data audits is fostering an environment where everyone in the organization can report data quality issues without fear of blame or reprisal. However, improving an organization’s ability to report data quality issues can sometimes have unintended consequences, similar to improving a city’s ability to report crimes.

“If the police report an increased number of burglaries in a neighborhood,” Nate Silver asked in his book The Signal and the Noise: Why Most Predictions Fail but Some Don’t, “is that because they are being more vigilant and are catching crimes that they had missed before, or have made it easier to report them? Or is it because the neighborhood is becoming more dangerous?”

As an example of the complex relationship between crime reporting and crime rates, Silver explained that “New York does not allow you to file a police report online, while San Francisco does, as I found out when my rental car was broken into there in a reporting trip for this book. San Francisco is doing a better job of helping citizens and visitors to report and prevent crimes. But perversely, this makes its reported crime rate higher.”

The same thing happens in an organization that makes it easier to report data quality issues. As more issues are reported, the organization’s data quality will be perceived as being worse than it was assumed to be when reporting and auditing were not performed on a regular basis.

But it would be a crime to allow this issue with reporting data quality issues prevent you from being more vigilant about auditing your data and reporting any data quality issues that you find.

It might be simple to suggest a specific hierarchy of customer types in relation to some mythical scale of “customer goodness,” and we see this somewhat implicitly applied across the board in the literature surrounding customer centricity and relationship management. Some examples include how to handle your “best customers,” or ways of getting rid of your “worst customers.” The challenge is that without a unit of measure and a scale for goodness, how do organizations classify their customers according to that comparative ranking?

I’d like to suggest two ideas that might help us address customer classification in a way that is easier to manage. The first involves establishing discrete measures and a scale for customer goodness. The second involves having a variety of customer classifications that are not tied to the concept of goodness.

This week, let’s look at the first. Here are some tasks for establishing discrete measures and a scale for customer classification:

• Selecting some key dimensions of value and a unit of measure (such as annual sales in dollars, number of items purchased or duration of the relationship in months),
• Selecting a weighting factor for each measured dimension,
• Deciding on the number of customer goodness levels,
• Setting thresholds for each level, and
• Coming up with discrete measures of goodness.

The weighting factors might be initially set in a somewhat arbitrary way. For some of the other decisions, we can presume that the distribution of the customer base is a normal distribution. That means we can start out using the “banding” of six standard deviations to set the customer goodness levels and the thresholds for each level, since more than 99% of the scores should lie within three standard deviations of the mean:

After evaluating the groupings, you might want to tweak the measures, weights and thresholds – perhaps you did not pull in some expected demographic, or you know a really good customer who didn’t get classified as a good customer. However, this provides at least one starting point for classification. Next time we will look at understanding the characteristics of individuals within each of those classifications.

1. Initial creation of the data warehouse. In the past, we have always entertained the feasibility of the conversion of history data even if the history data resides in the source system, a spreadsheet or another makeshift data warehouse. In some cases, the source system has gotten large and the historical data is not needed or used. So we consider bringing the historical data into the data warehouse that the business requirements dictate. You need to be careful here, as to not make the data warehouse your archival system. We only want the historical data that requires business usage!

2. When you are changing or revamping the data warehouse. In this instance, the business may have changed or you are incorporating enhanced data. The enhanced data may be something that is purchased (probably by marketing or sales), and we want to apply this data to historical data warehouse records.

So you have to ask yourself (and the business users) IS IT WORTH IT? If you google or bing “feasibility studies” it will give you good ideas on what you may want to include in your own feasibility study. Resources that are required to convert history are: people, hardware and software. Always consider giving the option of “START HISTORY FROM IMPLEMENTATION DATE”… this is the cheapest and easiest way to deal with history.

Irrelevant data.

Irrelevant data has no value or place within your business, yet for many reasons it is still being maintained (badly).

It accounts for huge amounts of physical real estate within your data landscape and, if left ignored, it can skew your data quality assessments, confuse the heck out of knowledge workers and cause unnecessary bloat within your systems.

Where does irrelevant data come from?

There are numerous reasons for the creation of irrelevant data, but here are some of the most common:

• COTS: Custom-off-the-shelf systems often cater to the broad industry. As a result, there are often many screens and data structures that simply don’t relate to all organisations.

• M&A inheritance: Legacy systems are integrated or migrated from newly acquired or merged companies.

• Shifting business models and processes: Over time, your underlying business processes have changed, but these changes haven’t been reflected in the data.

• Lack of archival strategy: Old data and data structures are not routinely moved from systems.

Do any of the above scenarios apply to your organisation? If they do, then welcome to the “Data Irrelevance Dimension.”

What are the benefits of removing irrelevant data?

By removing irrelevant data you get immediate gains such as increased query performance and reduced storage requirements. However, the main benefits come from “decluttering” some of the baggage that is slowing down your core user processes.

For example, imagine that you have inherited a COTS system that has field entries for international addresses, yet you only deal with domestic markets. Having to tab past redundant fields can slow up call centre staff or lead to information going in the wrong fields.

Perhaps you’ve inherited an asset management system as a result of a merger. The acquired organisation had a slightly different business model that stored site information for health and safety reasons that isn’t applicable to your organisation. The data remains in the system, and when the original organisation’s data is migrated into the acquired system you now have irrelevant data that persists.

Getting rid of irrelevant data simplifies user processes and operational performance. These are more than enough reason to explore its removal, but where should you start?

How do you get rid of irrelevant data?

I typically approach this with the full support of the business. You need to get them bought into the process.

First, profile the data and look for trends and obvious events in the lifetime history of the data. You’ll often see fields that have not been updated for several years. By having business experts with you there is more chance of spotting occurrences of irrelevance.

Defining what data you need is another critical activity. Getting the business community to agree on common terms and definitions can help in determining what data should be deleted.

Performing functional modelling exercises is another great way of performing a gap analysis of what data and functions you have compared to what you really need. This also helps validate your current business model and business processes.

Somewhat more challenging is removing screen design elements that map to redundant data structures. This can be problematic on COTS solutions where updates to screen and application design are often not supported. However, most modern systems allow you to customise a lot of the application infrastructure. (Hint: This ability to customise an app should also form part of your search for any new COTS solutions).

You will also need to perform an information chain analysis exercise to see which systems depend on the irrelevant data or data structures. Applications and ETL scripts can fail quite easily when the underlying schemas and information sources are changed in an uncontrolled manner. Create an amnesty for downstream users of redundant data structures but give them a deadline for when they need to come forward.

How have you got rid of irrelevant data in the past? How did it impact your organisation?

Welcome your experiences in the comments below.

Dice recently reported that the gap has largely disappeared. Yes, salary differences remain between the sexes, but they can be explained by “legitimate” factors like experience.

Looking at the (Hypothetical) Data

Before I became a proper techie, I used to work in corporate HR. I learned a long time ago that everyone’s dispassionate about other people’s compensation. Once they talk about their own, though, emotions run high. 

So, are men paid more than women? And are there mitigating factors?

Note that, in this post, I use hypothetical data to make a point.

Looking at the average salaries for programmers, there is in fact a legitimate difference between men and women of about eight percent:

But macro-level statistics may mask perfectly valid reasons for this difference. Perhaps the generic title of programmer obscures differences in skills. In 2013, knowing how to code in COBOL isn’t necessarily as valuable as knowing PHP, right? Or what about experience? All else equal, programmers with more experience might make more than those new to the field, irrespective of gender.

Throwing in years of experience, you see that the gender differences are much smaller, even non-existent in some cases:.

Looking at the data above, you’ll notice a clear outlier here. The guy with seven years of experience makes the most money out of everyone. Perhaps he has a unique programming skill? A good recruiter? Maybe he knows something about the VP’s extracurricular activities and gets a little extra to keep his mouth quiet.

Regardless of why, this outlier can throw off the numbers for everyone if the sample size is small. So, is it?

In short, yes. If we were talking about 11 hundred or thousand, then one outlier would have a negligible impact. But with 11, each data point has a disproportionate impact.

Simon Says

I’m not saying that some level of gender discrimination doesn’t exist today. (This post used simple, hypothetical data). However, before making any controversial statements about any issue, make sure that you’ve looked at some data first.

Feedback

What say you?

However, a major problem with defect prevention is assuming that defects are easily detected and can always be known a priori (i.e., not dependent on experience using the data in any business context) when, in fact, many defects can only be detected a posteriori (i.e., dependent on experience using the data in a specific business context), such as data defects causing a failure in a business process.

According to ISO 8000 standards, when it comes to data quality (more specifically, data accuracy) there are only assertions. You need to use data to test its assertions before you can determine its quality. Furthermore, quality is never achieved as a permanent state. Instead, it’s always in flux because of the universality of change. Even data that is defect-free today could be considered defective tomorrow. Therefore, assertions of data quality must be continually reasserted.

And, as Dylan Jones recently blogged, poor quality data isn’t a pitfall, but an opportunity to learn. He recommended transforming the way you approach bad data by finding the hidden story, tracking problems back to their source and asking questions of those who created the data. This story about the people, processes and technologies involved in creating poor quality data can often create considerable opportunities for increased profits, morale and operational efficiency.

Of course, those learning opportunities will often lead to implementing new – or strengthening existing – defect prevention procedures. Just don’t equate defect prevention with defect elimination — that is the preventable defective reasoning that I refer to as the fallacy of defect prevention.