Managing the State of your Data more Effectively

Jun 29, 2012 by Dylan Jones in Data Management, Data Profiling, Data Quality

One of the biggest problems facing any company that manages large amounts of tangible assets such as equipment, parts, stock and physical inventory is ensuring that every item is accounted for. I’ve written in the past about how many assets become “stranded” when data quality defects are introduced.

A big problem here is that inventory is always going to be subjected to human error. During site surveys of asset inventory accuracy (i.e., are the items physically where they’re meant to be according to the data?) I’ve found as many as 40% of high value items incorrectly recorded.

Consider some of the costs associated with this problem:

A £5,000 air conditioning unit (ACU) is repaired and made available for service but the service operator fails to update the system, the next time the telecoms planner designs a new floorplan an entirely new ACU is ordered because none appear available.

Planners design a floor layout based on the premise that certain electrical equipment is already installed but the installation team have “borrowed” the equipment for an emergency on another floor and failed to update the system, the install team quickly provision another piece of equipment but fail to return the loaned equipment back to operational availability.

If you speak to people telecoms they’ll tell you that stranded assets are a massive problem, and it’s because of this reality gap between the data and what’s actually happened on the ground.

State management can help here and it involves a slightly different approach to profiling than you may have been used to.

Most people will be aware of standard data profiling activities. You examine an attribute for a set of frequencies that dictate the “norm” for that attribute. The problem is that when it comes to equipment we need to track the data quality not at a column level but at an entity level. We need to understand the history of a physical piece of equipment and look for anomalies in its state and usage.

For example, let’s imagine that an ACU goes into repairs for a worn out fan motor. If you have date/time attributes located with your inventory data you can use conventional profiling to determine the average length of time to repair this fault. You can then search for equipment that went into repairs and compare its duration in a repair state to the historical average for the fault in question. If an ACU has been in repair for 6 months for a fault that takes on average 5 days to come out of repairs it can give you some indication that the equipment is either available or non-serviceable.

Interestingly, conventional data profiling will fail to flag issues here because there are no data errors per se. The values may be valid from a formatting and domain range perspective but from a business viewpoint there is clearly something suspicious.

One trick I often employ is to infer the state of an inventory item by looking at its relationship with other items. For example, in a previous investigation we found a lot of power equipment would go in for repair, be set with an appropriate repair status and then at some point be reconnected to a power circuit. The repair status would still be set but clearly there were some operational pointers here, you can’t have a piece of equipment feeding power to the floor if it’s sat in the repair bay!

The only way to manage the state of your physical assets more effectively is to understand how they operate in the real world. Spend some time with planners, fitters and other technicians who rely on this equipment. How do these items work in the real world? Which systems do the teams update? Which flags and data elements do they use to change the state of the equipment? How does the equipment interact with other equipment?

Data profiling is NOT just about examining attributes for spurious defects, it’s about discovering how physical objects behave both in the real world and within the data that represents it. You’ll find yourself building entirely new ways to model your data, pulling in 3rd party datasets and other disparate feeds from around the organisation, all in a bid to understand these physical assets more deeply. The richer this model becomes, the easier it will be to trap defects and anomalies at the individual asset level.

It’s a fascinating field to work in, you become a “crime scene detective” for data and I’ll shortly be posting more tips on how to improve your detective skills!

Tags: Data Management, Data Profiling, Data Quality