Data engineering
The data work that makes AI possible
Before expanding an AI pilot, check the definitions, source records, and evaluation process behind its answers. A readiness review can make the gaps visible.
An incorrect AI answer can have several causes: the model may be unsuitable, retrieval may find the wrong material, or the underlying records may already be inconsistent. Increasing model capability will not resolve every one of these problems. A readiness review helps isolate what needs to change before the business expands a pilot.
Start with one business question
Choose a question that someone needs to act on. For example: which customer orders are overdue? Agree what overdue means, which timestamp matters, and how cancellations or partial deliveries are treated. Then trace the answer through the systems and transformations that produce it. This is more useful than describing an entire data estate as ready or unready.
Make quality checks visible
Google’s engineering guidance treats infrastructure testing, data freshness, silent failures, and ownership as explicit concerns in machine-learning systems. For a business review, translate those concerns into observable checks. The following checklist is a suggested working aid; the appropriate thresholds depend on the process and the consequences of a wrong answer.
Scroll horizontally to view all columns →
| Check | Evidence to request | If the check fails |
|---|---|---|
| Definitions | An agreed definition and owner for each decision-critical field | Resolve the ambiguity before comparing outputs. |
| Freshness | Last successful update and expected update interval | Flag stale results and fix the feed. |
| Completeness | Missing and duplicate records in a representative sample | Repair the source or document the limitation. |
| Access | Tests for what each user role can and cannot retrieve | Correct the access path before wider release. |
| Evaluation | Reviewed questions, expected answers, and failure categories | Create a baseline before claiming improvement. |
Evaluate the path to the answer
Test each boundary separately. Did the source contain the correct record? Was it loaded successfully? Could the intended user access it? Did retrieval select relevant material? Did the final response accurately reflect that material? Keeping these checks separate makes it easier to distinguish a missing record from a generation error.
- Include incomplete records, conflicting definitions, old documents, and questions that cannot be answered.
- Record which source supports the expected answer and who reviewed it.
- Keep some evaluation examples separate from the examples used to tune the system.
- Repeat the review when source structure, access rules, or system behavior changes.
Agree what the system is allowed to conclude
A system should have a defined response when evidence is missing, stale, or outside the user’s permissions. Depending on the process, that may mean declining to answer, showing the limitation, or routing the question to a responsible person. The objective is a dependable decision path, including the cases where the system cannot provide an answer.
Conclude the review with a prioritized record of gaps, owners, and acceptance checks. Some gaps require engineering; others require a business decision about definitions or authority. That record gives the next phase a specific scope and makes later improvement easier to assess.
References & further reading
External guidance informing this perspective. The examples and decision frameworks above are Kōt Dynamics’ editorial guidance.
- Google · Rules of Machine Learning
Rules 5, 8, 10, and 11 cover infrastructure testing, freshness, silent failures, and data ownership.
- NIST · AI Risk Management Framework Playbook
A broader reference for documenting an AI system's context and evaluating its risks.
Apply this to your business.
Discuss the requirements, trade-offs, and next steps with our team.
Start a conversation