Skip to content

Contents  ·  Sensing

Data Quality Before Analysis

Every facility's data has the same defects. Finding them before drawing conclusions, because bad data produces confident wrong answers.

Procedure

The most common failure in this field is not a missing sensor. It is confident analysis of data that does not mean what it appears to.

The recurring defects

Clock drift between systems, so cross-system timing is wrong by minutes.

Batch scanning, clustering timestamps that should be spread.

Tasks left open, inflating elapsed times enormously.

Location master data out of date, so movements are computed against locations that no longer exist.

Duplicate events from fixed readers.

Manual corrections that overwrite the original record, removing the evidence of what happened.

Missing events where a step was skipped and reconstructed later.

Time zone and daylight saving handling, which produces an hour of nonsense twice a year.

Finding them

Plot the distribution of every duration. Impossible values, negative durations and enormous outliers are visible immediately.

Look for clustering at the second level, which indicates batch scanning.

Look for durations exactly equal to a shift length, which indicates tasks left open.

Check that consecutive locations are physically reachable in the elapsed time.

Compare event counts across systems for the same period; they should agree and usually do not.

Check clock offsets directly, between every system that contributes timestamps.

Fixing what can be fixed

Synchronise clocks, which is a configuration change and removes a whole class of error.

Auto-close stale tasks with a marker, so they are excluded rather than distorting.

Deduplicate reader events with a dwell window.

Reconcile location master data, which is a physical walk as well as a data exercise.

Fix the cause of batch scanning, which is usually a process or device problem rather than a discipline one.

Handling what cannot

Exclude, do not correct. Removing implausible records is defensible; adjusting them is not.

Document every exclusion rule and the volume it removes.

Report the exclusion rate as a data quality measure, trended. A rising rate is a process signal.

Never silently drop data, because the analysis will be questioned and the answer must be available.

The confidence problem

Bad data does not look bad. It produces plausible charts and specific conclusions.

Which is why the quality check must come first, every time, before any finding is reported.

Sanity-check every result against physical reality. A finding that implies someone travelled at an impossible speed, or that a stage processed more than it received, is a data problem rather than a discovery.

Ask someone who works on the floor whether the finding matches their experience. When it does not, the data is wrong more often than they are.

The ongoing measure

Report data quality alongside every analysis: completeness, exclusion rate, clock agreement, anomaly rate.

Trend it, because it degrades: new devices, new processes, new sites all introduce defects.

Assign it an owner. Data quality with no owner decays within two quarters, and then the analysis is confidently wrong, which is worse than having none.

Documenting the exclusion rules

Every analysis excludes records, and undocumented exclusions make results unreproducible.

Write each rule: what it removes and why.

Report the volume removed, as a proportion.

Keep the rules stable between baseline and follow-up, or the comparison is invalid silently.

Version them, so a change in a figure can be attributed to a rule change rather than to reality.

Never silently drop data. The analysis will be questioned and the answer must be available immediately.

The sanity check against reality

The last step before reporting any finding, and it catches what statistical checks miss.

Does the implied speed exceed walking pace?

Did a stage process more than it received?

Is a duration longer than the shift?

Does the total reconcile with the order data?

Would someone who works there recognise this?

When the data and the floor disagree, the data is wrong more often than the floor is, and that assumption saves a great deal of embarrassment.