$ teds read --post hidden-work-behind-just-add-ai
The Hidden Work Behind Just Add AI
A reality check on the operational work behind AI features: data, evaluation, reliability, UX, safety, compliance, monitoring, and trust.
The Hidden Work Behind Just Add AI
TL;DR
- “Just add AI” hides the system required to make an AI feature useful.
- The model call is usually the easy part.
- The hidden work includes data, evaluation, reliability, UX, safety, compliance, monitoring, and feedback.
- Teams that ignore the hidden work ship demos disguised as products.
Abstract
“Just add AI” sounds simple because the first model call is simple.
Send a prompt. Get an answer. Put it in the product.
That can work for a demo. It rarely works for a product.
AI features depend on an operational system around the model: data access, privacy, evaluation, latency, cost, monitoring, error recovery, user controls, safety review, and feedback loops.
This post names the hidden work so teams can plan for it before it becomes a production problem.
Table of Contents
- The Phrase That Hides Work
- Hidden Layer 1: Data
- Hidden Layer 2: Evaluation
- Hidden Layer 3: Reliability
- Hidden Layer 4: UX
- Hidden Layer 5: Safety and Compliance
- The Real Checklist
- Summary
- Next Steps
The Phrase That Hides Work
“Just add AI” usually means:
Add a model call to an existing workflow.
But the model call is only one piece.
The product still needs to answer:
- what data the model can see
- what the model should do
- how quality is measured
- what happens when it is wrong
- how users verify outputs
- what the latency budget is
- how much each interaction costs
- what data can be logged
- who owns failures
The phrase hides ownership. It makes the work sound like a feature when it is often a system.
Hidden Layer 1: Data
AI systems inherit data problems.
If the source content is stale, the answer may be stale. If permissions are unclear, the system may expose the wrong information. If labels are messy, fine-tuning may teach the model inconsistent behavior. If documents contradict each other, retrieval can surface confusion.
Data work includes:
- access
- quality
- privacy
- freshness
- ownership
- cleaning
- formatting
- permissions
- retention
Teams often discover this late because demos use clean examples. Production uses the real corpus.
Hidden Layer 2: Evaluation
Without evaluation, teams cannot tell whether the AI feature works.
They can see outputs. They can collect reactions. They can feel progress. But they cannot reliably answer:
Did this change make the system better?
Evaluation work includes:
- test cases
- rubrics
- baseline outputs
- edge cases
- known failure cases
- human review
- automated checks
- regression tracking
- production feedback
Evaluation is especially important for technical products, research tools, and AI infrastructure. As discussed in why DevRel is different for research and ML tools, these audiences look for evidence, reproducibility, and workflow fit.
Hidden Layer 3: Reliability
An AI feature has to run under constraints.
Reliability work includes:
- latency
- cost
- rate limits
- retries
- fallbacks
- caching
- monitoring
- model versioning
- prompt versioning
- incident response
Model performance also has infrastructure implications. Sometimes the right work is serving optimization, quantization, or deployment architecture. For examples, see EAGLE-3 speculative decoding in vLLM and AutoRound quantization.
Reliability is not only uptime. It is whether the workflow remains usable at real volume, real cost, and real latency.
Hidden Layer 4: UX
AI UX has to make uncertainty usable.
The user needs to know:
- what the system used
- what it generated
- what can be edited
- what can be trusted
- what needs review
- what happens next
UX work includes:
- citations
- corrections
- review states
- confidence signals
- error recovery
- action approval
- visible assumptions
- clear boundaries
A fluent answer without inspectability can reduce trust. A less magical interface that shows sources, assumptions, and controls may be more useful.
Hidden Layer 5: Safety And Compliance
AI features can create new risk.
Safety and compliance work includes:
- sensitive data handling
- access controls
- prompt injection risks
- harmful outputs
- audit logs
- regulatory review
- human escalation
- policy boundaries
- red teaming
This does not mean every AI feature requires a giant governance program. It means risk should be named early. The amount of process should match the stakes.
The Real Checklist
Before adding AI to a product, ask:
- What data does the system need?
- Who is allowed to access that data?
- How will quality be measured?
- What happens when the model is wrong?
- Can users verify the output?
- What is the latency budget?
- What is the cost per workflow?
- What should be logged?
- What should not be logged?
- Who owns monitoring?
- What feedback returns to the team?
- What claims are we willing to make publicly?
Need someone to find the hidden work before it finds you? We help teams pressure-test AI features before they become production surprises.
Summary
“Just add AI” hides the system.
The real work includes data, evaluation, reliability, UX, safety, compliance, monitoring, and feedback. The model call may be the easiest part.
Teams that plan for the hidden work build products. Teams that ignore it ship demos with support burdens.
Next Steps
Take one proposed AI feature and write down the hidden work under five headings:
- data
- evaluation
- reliability
- UX
- safety and compliance
If any heading is blank, the project is not fully scoped.
Need help turning an AI idea into a working system? We build, break, and explain AI. Then we hand it back working.