Build a Scam-Message Classifier in One Sitting, Inspired by DeepMind's Singapore Work

In one sitting you will know how to build a scam-message classifier that returns structured verdicts you can test, using a prompt and a labelled file.

Why now: Google DeepMind and the Singapore Government are collaborating to build a fine-tuned AI model for scam detection. The headline is all I will vouch for about their model. But it shows where this category is heading: tuned models for a narrow fraud job. Before you tune anything, you need a baseline. This is how you get one.

1. Collect real messages you have seen: SMS, emails, DMs. Save them as JSONL with two fields, text and label.
2. Use three labels only: scam, legit, unsure. Include legit messages that sound urgent, like delivery notices and bank alerts.
3. Fix the output schema before you write the prompt: verdict, tactic, reason. Nothing else.
4. Paste the prompt below into your model of choice, at its lowest temperature.
5. Run every labelled message through it and diff the verdicts against your labels.
6. Read the misses, add the worst ones to the prompt as examples, and rerun.

The prompt:

You are a scam triage classifier. Read the message and respond with JSON only.

Schema:
{"verdict": "scam" | "legit" | "unsure",
 "tactic": "impersonation" | "payment_request" | "credential_request" | "prize_or_refund" | "none",
 "reason": "one sentence"}

Rules:
- Urgency alone is NOT evidence of a scam. Look for what the message asks the reader to do.
- Mark scam only if the message asks for money, credentials, or a click to an unverified destination.
- If you cannot tell, return unsure. Never guess.
- Output nothing outside the JSON object.

Message:
"""
{{MESSAGE_TEXT}}
"""

The gotcha: urgency bias. Your legit-but-urgent rows will come back as scam, with a reason that says something like “creates a sense of urgency.” That is how you recognise it. The model is keying on tone instead of the ask. The rule about requests in the prompt helps. Adding those exact rows as examples helps more. Also treat any reply that is not valid JSON as a miss, not a retry. Schema enforcement on small models is its own discipline, and the Forge write-up on guardrails is a good place to learn it.

My read: prompt first, tune second, and tuning has to earn its place. If a tight prompt with a handful of examples still misses on your own traffic, those misses are your case for fine-tuning. If you skip the baseline, you cannot tell whether tuning bought you anything. A general model told to “spot scams” is a demo. A measured miss list is a product.

Want the next one of these in your inbox? Subscribe here. One AI signal a day. 90 seconds. No fluff.