Why Your Software Vendor’s ‘Audit’ Might Be a Black Box

Imagine discovering that your shop’s CCTV system, the one you bought to deter theft, has a “smart” feature that flags suspicious staff behaviour — but the vendor refuses to tell you how the flagging works. Would you trust it? This week, American surveillance startup Flock found itself in exactly that hot seat, and the story holds a sharp lesson for Malaysian SMEs that rely on automated tools to run their businesses.

Flock sells automated licence plate reader cameras to police departments. On Thursday, it announced a suite of policies and a tool called “Audit Assistance” that it says will curb abuse of its systems by law enforcement officers. The tool, according to Flock, detects “abnormal activity” and flags it for an administrator to review, even automatically locking out suspicious users until someone intervenes. More than one-third of Flock’s customers have already enabled it, and Flock is requiring all customers to switch it on by the end of the year, the company says in its blog post (via TechCrunch).

But here’s the catch: Flock has not explained how Audit Assistance works. It says it is not AI or machine learning, but a “data tool that flags atypical search patterns” — for example, if a user searches for the same licence plate under multiple different case codes. Yet when TechCrunch asked what data the tool was trained on, what patterns it surfaces, or whether any statistics prove its effectiveness, the company stayed vague. The ACLU’s senior policy counsel, Chad Marlow, told TechCrunch that there is “no evidence that the tool works consistently,” and pointed out that without knowing how many officers misuse the system, you can’t tell if the tool catches 95% of violators or just 5% (TechCrunch).

Flock claims Audit Assistance has already uncovered real abuse — earlier this week, three former Georgia sheriff’s deputies were arrested for allegedly using Flock cameras to stalk people they had personal relationships with, and the Bibb County Sheriff’s Office specifically credited Flock’s tool for the discovery. But the Electronic Frontier Foundation’s Cooper Quintin told TechCrunch it may not matter whether the tool works: if there are no consequences for abusers, and the tool reports to the same police agency doing the abusing, “it’s just a fig leaf for Flock” (TechCrunch).

Why This Matters for Malaysian SMEs

As a business owner, you likely don’t run a police surveillance network. But you almost certainly run automated systems that make judgment calls about your people and your operations. Your customer relationship management (CRM) software flags leads, your accounting system spots duplicate payments, your cloud security tool blocks suspicious login attempts. Each of these vendors will tout their “smart” features and promise to keep you safe. But how many of them can clearly explain how their algorithms decide what is “suspicious” or “abnormal”?

In Malaysia, this isn’t just an abstract worry. If your company processes personal data — customer contact details, employee records, or even vehicle plate numbers captured by a security camera at your loading bay — you’re accountable under the Personal Data Protection Act (PDPA). If a software tool makes a mistake and accuses a customer or an employee of wrongdoing, you are the one who has to defend that decision. Can you? If your vendor’s auditing feature is a black box, you have a practical problem: you cannot answer the question “why was this flagged?” and you certainly cannot prove your process was fair.

There’s a second, less obvious lesson here about where you place your trust. Flock is effectively asking customers and the public to believe that its Audit Assistance tool makes the system accountable. But without independent verification, that’s just marketing. The same applies to your business tools. Vendors will tell you their automation will improve accuracy, reduce fraud, or optimise your inventory — but if they can’t give you transparent logic, clear exceptions, and a way to override or appeal their decisions, you’re not running your business; you’re vibing on someone else’s code. Malaysian SME owners are often time-poor and resource-constrained, which makes trusting the vendor’s promises tempting. But as Flock shows, a confident press release is not a substitute for accountability.

“If there are no consequences for abuse and whoever the tool reports to is the same police agency that is doing the abuse, it’s just a fig leaf for Flock. The best way to ensure accountability is to pass laws constraining the use of this technology and requiring a warrant for its use.” — Cooper Quintin, EFF, via TechCrunch

That quote speaks to a deeper truth about governance. When you delegate a decision to automation, you still own the outcome. The tool might flag a “pattern” in your sales team’s behaviour, or a “risk score” on a customer’s invoice. But you can only manage that risk if you understand the tool’s inputs, thresholds, and failure modes.

The Bigger Picture

This story points to a broader shift. As software becomes more aware of human behaviour, the line between “technology vendor” and “policy enforcer” blurs — and the requirement for transparency grows. Privacy experts in the Flock story consistently argue that independent auditing is the only way to know if such tools genuinely work. In Malaysia, we’re seeing a similar conversation about data governance, with regulations and frameworks that increasingly expect businesses to demonstrate explainability in their automated decision-making.

So what should you do as an SME owner? Before you adopt any software that claims to monitor, flag, or automatically act on data, ask the vendor the same questions TechCrunch asked Flock: What triggers a flag? What baseline are you comparing against? What are the false positive and false negative rates? And can an independent third party verify your claims?

If the answer is “it’s proprietary” or “don’t worry about it,” walk away. Your business may be small, but your reputation and your customers’ trust are not. The Flock controversy is an uncomfortable reminder that auditing the auditor is your job, not an optional extra.

• Demand explainability: Ask vendors precisely what their “abnormal” or “atypical” rules are, and how you can customise them.
• Insist on independent validation: Look for case studies, third-party audits, or a willingness to show their false-positive rates.
• Keep a human in the loop: Make sure any automated flag requires human review before action, just like Flock’s administrator lockout.
• Document your own decisions: For PDPA compliance, keep an audit trail of why and how your automation made a decision.
• Set clear consequences: If a flag leads to a finding, be prepared to act on it consistently — otherwise the tool is just theatre.

Automation promises to save you time and effort, and it often does. But as the Flock saga shows, a vendor that can’t or won’t explain its safeguards is handing you the risk, not the benefit. For Malaysian SMEs, the smartest move isn’t to avoid automation — it’s to be the boss who asks the uncomfortable questions before you commit.

Ready to Streamline Your Operations?

Technology moves fast. Your operations should keep up. AutoRunBiz builds AI systems that run your daily workflows — from WhatsApp order capture to accounting. Book a free 15-min ops audit →