When Your AI Tool Knows Too Much About Your Business
You may be using AI to draft customer replies, summarise documents, analyse sales records, or help your team write code. It is convenient, but every prompt also creates a record of how your business operates. If that information is handled carelessly, your processes, customer details, and internal know-how may travel further than you expect.
A recent report from Anthropic highlights a related issue in the AI industry: “distillation” campaigns, where organisations send large volumes of carefully designed requests to a powerful AI model and use the answers to train another model. Anthropic reported nearly 200 million exchanges linked to five campaigns. The story is about major AI companies, but the lesson applies directly to your business: AI usage needs governance, not just enthusiasm.
TL;DR: Distillation involves collecting model responses at scale to reproduce useful capabilities in another AI system. For your SME, protect sensitive prompts, control staff access, choose vendors carefully, and keep a clear record of what data enters AI tools.
What This Means
Think of a powerful AI model as an experienced specialist. It may be able to reason through a difficult customer complaint, write software, analyse a spreadsheet, or plan a workflow. Distillation is similar to repeatedly asking that specialist for answers, studying the responses, and using them as examples to train a smaller system.
The reported campaigns targeted capabilities such as agentic tasks, tool use, coding, data analysis, and logical reasoning. Anthropic said one campaign used a prompt framed as a translation request to try to expose internal working information. The important point is not the exact wording. It is that attackers may test many unusual approaches to bypass safeguards and collect useful outputs.
Anthropic attributed the largest reported effort to Alibaba. The company said it observed 151 million exchanges between May and July 2026, involving 3,500 accounts and reaching nearly three million exchanges per day. Another campaign attributed to Moonshot AI reportedly routed nearly 300,000 requests through 5,000 accounts over 10 days.
For a business owner, this demonstrates two risks. First, AI providers must defend their models from automated extraction. Second, your own team may unintentionally expose valuable information when using public AI services. Your business does not need millions of prompts to create a problem. One uploaded customer list, supplier agreement, source-code file, or internal operating procedure can be enough.
Key insight: Treat every AI prompt as a business record. If you would not send the information to an unknown third party, do not paste it into an unapproved AI tool.
How This Applies to Malaysian SMEs
Customer service teams: A Malaysian retailer, clinic, tuition centre, or service company may use AI to draft WhatsApp replies and email responses. That can save staff time, but customer conversations often contain names, telephone numbers, order details, medical information, addresses, or complaints. Create a simple rule: remove identifying details before using AI, or use a business-approved system with clear data-handling controls. For example, replace “Siti from Shah Alam, order number 8821” with “Customer A, delayed order.”
Sales and marketing: You may ask AI to improve a proposal, analyse a campaign, or prepare a response to a tender. Those documents can reveal your customer segments, sales process, product roadmap, and negotiation position. Keep sensitive commercial information out of general-purpose tools unless your organisation has approved the provider and the specific use. Ask staff to work with summaries and placeholders instead of uploading complete proposals or contracts.
Operations and finance: AI is increasingly used to classify invoices, identify unusual transactions, generate management reports, and answer questions about spreadsheets. The convenience is real, but financial records may include supplier details, bank information, employee data, and transaction histories. Build workflows that pass only the minimum necessary fields to the model. A report can often use totals, categories, and dates without exposing full account numbers or personal identifiers.
Software and automation projects: If you ask an AI coding assistant to fix an error, do not automatically paste your entire application, API keys, customer database structure, or private business rules. Share a small reproducible example, remove secrets, and review the generated code before it reaches production. This is especially important when an external freelancer or part-time developer is involved, because you may not know which tools they use behind the scenes.
A simple risk view for your team
| AI activity | Possible business information exposed | Safer practice |
|---|---|---|
| Drafting customer replies | Names, phone numbers, complaints | Use fictional labels or approved systems |
| Summarising contracts | Terms, supplier obligations, client details | Remove identifying details and restrict access |
| Analysing spreadsheets | Sales, payroll, banking, inventory data | Upload selected fields, not the full file |
| Generating code | API keys, system design, private logic | Use sample data and scan for secrets |
The Personal Data Protection Act 2010 is relevant when your business handles personal data in commercial transactions. You should review your organisation’s obligations and obtain professional advice for high-risk or regulated activities. The practical starting point is straightforward: know what personal data you collect, where it goes, who can access it, and how long you retain it.
Practical Takeaways
- Create an approved-tool list. Tell employees which AI services they may use for work and which are not approved.
- Classify information before prompting. Mark data as public, internal, confidential, or restricted.
- Minimise what you share. Use summaries, redacted text, sample records, and fictional names whenever possible.
- Remove secrets from code. Never include passwords, access tokens, private keys, or production credentials in a prompt.
- Control accounts. Use individual logins where possible, enable multi-factor authentication, and remove access when staff leave.
- Review vendor terms. Check whether prompts are used for training, how data is retained, and where it may be processed.
- Keep a basic AI register. Record the tool, purpose, users, information types, and review date.
- Train staff with real examples. Show them how a normal-looking prompt can accidentally reveal customer or company information.
- Require human review. AI output should not automatically send customer messages, approve payments, or change records without suitable checks.
The Bigger Picture
The reported distillation campaigns show that AI capability is becoming valuable enough to attract systematic extraction attempts. The same pattern will affect business software: providers will add stronger monitoring, usage limits, identity checks, and restrictions on automated querying. Those measures may sometimes feel inconvenient, but they help protect the reliability and confidentiality of the services you depend on.
For Malaysian SMEs, the long-term advantage will not come from giving every employee unlimited access to every AI tool. It will come from building repeatable workflows that combine automation with sensible boundaries. You want your team to use AI for speed while keeping customer trust, operational knowledge, and confidential records under control.
Start small. Choose one workflow, such as customer-service drafting or invoice classification. Map the information involved, remove unnecessary personal data, define who reviews the output, and document the approved process. Once it works safely, expand to the next workflow.
AI can support a lean team, but it should not become an unmonitored door into your business. The organisations that benefit most will be the ones that make safe usage simple enough for busy staff to follow every day.
Ready to Streamline Your Operations?
Your business should run itself. AutoRunBiz deploys AI agents to automate your daily operations — WhatsApp orders, invoicing, customer follow-ups, and accounting. Book a free 15-min ops audit to see where automation fits your business →