Why Your AI System Should Not Handle Everything

When you add AI to your business, the natural instinct is to send every difficult question to the most capable language model available. It sounds simple: collect the relevant documents, place them in the prompt, and let the model decide.

That approach may work during a demonstration. In daily operations, however, it can create inconsistent decisions, difficult-to-trace outcomes, and unnecessary processing for cases that could have been settled by a simple rule. If you are using AI to screen documents, classify customer requests, review supplier information, or flag operational exceptions, “let the model decide” is not a strong control system.

The better approach is to decide what the AI model should never handle. Clear cases should be resolved by rules. Only unclear cases should move to document retrieval and then, if necessary, to the language model.

TL;DR

Use a three-stage process: deterministic rules first, relevant document retrieval second, and an LLM only for genuinely ambiguous cases.

This can reduce unnecessary model usage while making decisions easier to explain, review, and improve. The source article reports that sending only the genuinely ambiguous 10% to 15% of cases to the LLM reduced inference usage by roughly six times in one system.

What This Means

RAG, or retrieval-augmented generation, is a way of giving an AI model access to your own business information before it produces an answer. For example, the system might search your standard operating procedures, past decisions, product records, or internal policy documents, then provide the relevant information to the model.

The weakness is that retrieval does not automatically make a decision correct. If the system finds the wrong document, misses an important condition, or presents conflicting information, the model may still produce an answer that sounds confident. This is especially risky when you need to explain the decision later to a manager, customer, auditor, or business partner.

A cascade architecture places simpler and more reliable checks before the AI model:

Stage	What happens	Suitable cases
Stage 1	Rules and structured checks resolve clear cases	Exact matches, required fields, approved categories
Stage 2	Relevant records and documents are retrieved	Exceptions, conflicting details, similar past cases
Stage 3	The LLM handles the remaining ambiguity	Unclear cases requiring contextual judgement

Stage one should handle anything that has a clear answer. If an invoice number matches an approved purchase order, a required field is present, or a customer request contains a known category, there is little reason to ask a language model to interpret it.

Stage two is about finding the right evidence. A supplier record may need to be compared with previous onboarding decisions. A customer complaint may need to be matched against the correct service policy. A document with conflicting information may need supporting records before anyone makes a decision.

Stage three is the escalation path. The model should see only the cases that remain unclear after the first two checks. This makes the model’s role narrower, easier to monitor, and more appropriate for situations that require context rather than simple matching.

The important AI design question is not “How do I make the model handle everything?” It is “Which decisions should never have belonged to the model?”

How This Applies to Malaysian SMEs

Imagine you run a wholesale business in Shah Alam and receive supplier documents by email and WhatsApp. Your team needs to check whether each document contains the correct company name, registration details, bank information, and supporting forms. The first stage can handle many of these checks using structured fields and approved records. A document with an exact match can move forward without involving an LLM. Only a mismatch or unusual combination needs further review.

For a service company in Penang, the same structure could apply to customer complaints. A request containing a recognised service category, valid order reference, and standard resolution may be routed automatically. If the complaint mentions an unusual failure, conflicting warranty details, or a previous unresolved case, retrieval can bring up similar cases and the relevant policy. The language model then helps summarise the evidence or recommend escalation rather than making every routine decision from scratch.

For an online seller or distributor, you might use AI to classify product returns. Clear cases, such as an unopened item within your documented return conditions, can be checked through rules. Cases involving damage, missing parts, late delivery, or unclear proof require supporting information from order records, courier updates, and previous resolutions. The model should be reserved for cases where those records do not provide a clear answer.

Human resources is another practical area. A leave request that contains the employee’s details, dates, and the required approval may follow a standard workflow. A request involving overlapping leave, incomplete documentation, or an unusual policy situation should be escalated. The AI can help organise the information, but you should keep the final authority with your documented process and an appropriate person.

This approach is also useful for Malaysian SMEs that handle bilingual or mixed-language communication. A rule can identify known form fields, reference numbers, and status codes even when messages contain Bahasa Malaysia and English. Retrieval can then find the matching internal procedure. The model only needs to interpret the remaining context, rather than repeatedly processing information that your system already understands.

Why Auditability Matters Even for a Small Business

You may not operate in a heavily regulated industry, but you still need to answer basic questions when something goes wrong: Which information did the system use? Why was the request approved or flagged? Who reviewed the exception? Was the decision based on a rule, a document, or an AI recommendation?

A rule-based decision is easier to explain. You can record that a reference number matched, a required document was present, or a condition was not met. For a retrieved decision, you can store the documents or records used. For an LLM decision, you can record the prompt version, retrieved evidence, output, confidence indication, and human outcome.

This record gives you a practical trail without requiring your staff to remember what happened weeks or months later. It also helps you identify whether the real problem is a missing rule, weak document retrieval, or poor model judgement.

Practical Takeaways

• List your recurring decisions. Start with document checks, customer routing, supplier onboarding, returns, or internal requests.
• Separate clear cases from unclear cases. Write down which conditions can be checked exactly without interpretation.
• Build rules before prompts. If a decision can be represented by a field comparison or a simple condition, do that first.
• Retrieve evidence selectively. Do not send your entire document library to the model. Retrieve only records relevant to the specific exception.
• Make uncertainty visible. Ask the system to flag unclear cases instead of forcing every case into an automatic approval or rejection.
• Keep a human review queue. Decide who handles cases that contain conflicting information or fall below your confidence standard.
• Record the decision path. Store whether the result came from a rule, retrieved evidence, an AI suggestion, or a human decision.
• Learn from corrections. When a staff member overturns an AI recommendation, preserve the example and the correct outcome for future retrieval.
• Test the difficult cases deliberately. Do not measure the system only on routine requests that are already easy to classify.

Designing Safer AI Prompts

When a case reaches the model, avoid a vague instruction such as “decide whether this is acceptable.” Give the model the business context and explain what should happen when information is incomplete.

For example, your instruction might say: “Use only the supplied records. If the evidence conflicts or an important detail is missing, mark the case for human review. Explain which evidence supports the recommendation and identify any uncertainty.”

This is different from asking the model to sound confident. You are telling it that uncertainty has an operational consequence. A doubtful case should be escalated rather than quietly cleared.

You should also request a structured response containing the recommendation, supporting evidence, missing information, and escalation status. A structured response is easier for your workflow system and staff to process than a long paragraph.

How to Measure Whether It Works

Do not look only at the final answer. Track the rule stage, retrieval stage, model stage, and human review outcome separately. The source article recommends measuring retrieval quality independently from final classification accuracy because finding relevant documents does not guarantee that the final decision will be correct.

Review the cases that reach the model more carefully than routine cases. The source article advises deliberately oversampling stage-three cases in the evaluation set, because those are the cases where judgement is most difficult.

For your business, a simple monthly review can be enough at the start. Examine which rules handled cases correctly, which exceptions were routed unnecessarily, which documents were retrieved, and how often staff changed the recommendation. Use these findings to improve the rules and knowledge base before changing the model.

The Bigger Picture

AI systems for SMEs will become more useful when they are designed as controlled workflows rather than as universal answer machines. The model is valuable, but it does not need to sit in front of every business decision.

Putting clear rules first gives you consistency. Adding retrieval gives your team better evidence for exceptions. Escalating only the genuinely ambiguous cases gives the model a focused job and gives your staff a clearer review queue.

Start with one workflow where the decision criteria are already understood. Document the rules, identify the exceptions, record the evidence, and make human review part of the design. Once that process is reliable, you can apply the same pattern to another area of your business.

The strongest automation is not the one that makes the most decisions independently. It is the one that makes routine decisions consistently, explains unusual decisions clearly, and knows when to ask for help.

Ready to Streamline Your Operations?

Your business should run itself. AutoRunBiz deploys AI agents to automate your daily operations — WhatsApp orders, invoicing, customer follow-ups, and accounting. Book a free 15-min ops audit to see where automation fits your business →