Krystle and Sean’s Blog

When AI Agents Build Your Backend: A Malaysian SME Guide

When AI Agents Build Your Backend: A Malaysian SME Guide — featured image

—

by

Krystle and Sean

Why You Should Care About AI Agents That Write Code

Imagine asking an AI assistant to fix your company’s customer database, set up a secure login system, or patch a hole that’s exposing your customers’ personal data. Now imagine it doing that in minutes—without a human developer. That’s not science fiction anymore. AI coding agents like Claude Code, Codex, and OpenCode are already handling real backend tasks, and a major open-source company just released a way to test how well they actually perform.

If you run a Malaysian SME with 1-50 employees, you might think this is developer talk. But consider this: every order form, every inventory list, every customer record you manage online depends on a database with security rules. When AI can build and maintain that for you, it changes how much automation you can afford. The catch? You need to know which AI agent to trust. That’s exactly what Supabase’s new open-source benchmark is designed to answer.

What Happened

Supabase, a popular open-source backend platform, has released Supabase Evals—a benchmark and framework that tests how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks: building a database schema, debugging a failed Edge Function, or fixing a broken Row Level Security (RLS) policy. The framework scores each agent’s results, powers a public leaderboard at supabase.com/evals, and runs as an internal regression suite monitored daily.

The benchmark is not a toy. Every scenario runs against a real Supabase stack in containers—no mockups, no shortcuts. Agents get one retry before grading, and scoring combines deterministic checks with an LLM-as-a-judge. The scenarios are grounded in actual support tickets, bug reports, and GitHub issues, meaning they reflect the messy problems real developers (and now real businesses) face.

The initial findings are revealing. In the Build stage, Opus 5 and Kimi K3 both scored 100% unaided—no special instructions needed. Other models improved dramatically when given “skills” or guidance docs: Sonnet 5 rose from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89%.

Why This Matters for Malaysian SMEs

You may not write code, but you likely depend on systems that do. Whether it’s your booking website, your e-commerce store, or your customer relationship database, backend security and reliability are non-negotiable. Here’s where this benchmark hits close to home: one of the key weaknesses it found is agents hand-writing security policies instead of using safer, declarative approaches. As the source article notes, an agent writing a wrong RLS policy is effectively a security incident in regulated backends like fintech or healthcare. For a Malaysian SME, imagine a loyalty app where an AI accidentally exposes all your customers’ phone numbers and home addresses. That’s a reputation killer, and potentially a legal problem.

But there’s a positive side. Supabase Evals shows that AI agents can be tested before you let them touch your business data. Since the framework is open-source under Apache-2.0, it runs locally via pnpm. This means a Malaysian SME (or a trusted local automation partner like AutoRunBiz) can run these tests on your specific workflow, not just a generic benchmark.

Another critical insight for you: the benchmark found that docs usage varies sharply—Codex and GPT-5.6 read roughly 8 documentation pages per scenario, while Claude Code checks docs in fewer than 40% of scenarios even with skills loaded. In plain language: some AI agents are overconfident and skip instructions. For a business owner, this means you should not assume an AI agent will follow your SOPs unless you verify. It’s exactly why benchmark data matters—it gives you a scorecard for which digital “employee” to trust.

The Bigger Picture

We’re entering a phase where AI doesn’t just answer questions; it does work. Databases, backend logic, even security policies are being automated. For Malaysian SMEs with limited technical staff, this could be a game-changer. Instead of hiring a full-time backend developer to maintain a simple system, you could rely on an AI agent—if you know it’s reliable.

“Agents pass most scenarios with no skill loaded… Skills closed the rest of the gap.” — Supabase Evals findings, via MarkTechPost

The bigger picture is about quality control for AI. Just as you wouldn’t hire a new employee without checking their references, you shouldn’t deploy an AI agent without checking its benchmark scores. Supabase Evals is one of the first serious attempts to create that reference check for coding agents. The fact that it’s open source means the results are auditable, and the community can improve the benchmark over time.

For you, the action step is simple: pay attention to AI benchmarks before you adopt automation tools. If a vendor tells you “AI can handle everything,” ask what benchmark they used. If they can’t point to results on real tasks—like building or fixing a database—then you’re trusting a guess, not a test.

Key Takeaways From Supabase Evals

  • Supabase Evals is open source under Apache-2.0 and runs against real containerized stacks, not mocks.
  • Top AI agents (Opus 5, Kimi K3) scored 100% on build tasks with no special skills loaded.
  • Skills/guidance helped mid-tier models dramatically—e.g., Sonnet 5 went from 78% to 100%.
  • Common weaknesses: hand-written migrations instead of declarative schemas, manual auth verification, and inconsistent documentation use.
  • Every scenario grounds in real support tickets and GitHub issues—so the tests reflect actual business problems.

If you’re in a sector like retail, logistics, or professional services, you don’t need to become a database expert. But you do need to be an informed buyer of automation. Supabase Evals is a window into how AI agents truly perform. The next time you’re evaluating an AI tool for your business, ask: has this been tested on real tasks, against real security requirements, in a real environment? That question—once only asked by software engineers—is now a business owner’s question too.

Malaysian SMEs have a huge opportunity to leapfrog with AI, but only if they adopt it with eyes open. Start by familiarising yourself with the benchmark results and the public leaderboard. You don’t need to understand every line of code—just enough to know which AI agent has your back.

Ready to Streamline Your Operations?

Technology moves fast. Your operations should keep up. AutoRunBiz builds AI systems that run your daily workflows — from WhatsApp order capture to accounting. Book a free 15-min ops audit →