The demo answers everything. Six weeks after launch the same three complaints arrive every week, somebody quietly turns the assistant off, and the project becomes a thing that was tried. Between those two moments there is usually nothing: no test set, no pass mark, no written definition of what working meant. The supplier showed it working, everybody nodded, and the launch date became the acceptance criterion.
I no longer sign off an Arabic customer assistant without a written acceptance test. This is the one I use: what goes in the set, what to type, what counts as a pass, and who scores it. Half a day to build, two hours to run.
The demo is not the test
A demo is the supplier’s questions, in the supplier’s Arabic, against the supplier’s data. An acceptance test is your customers’ questions, in their Arabic, against your data. Only one predicts what happens after launch. This sits below the reasons most UAE AI pilots stop before production: a pilot can have the right owner, process and budget and still fail on the first hundred real messages.
Build the set from what you have. Export the last two hundred inbound messages across every channel: WhatsApp, Instagram and Facebook, the website form, the notes phone staff type after a call. Take them verbatim. The instinct to tidy them is the instinct that produces a demo.
Strip names and account references first: those messages are personal data the moment a supplier holds them, and the questions the UAE data protection law asks before you connect a model apply to a test set as to production. Then deduplicate: two hundred messages usually collapse into fifty or sixty distinct questions.
Choose forty, weighted by how often each occurs rather than by variety: if a third of your inbound is some version of where is my order, a third of the set is that question in its many spellings.
Test one: the Arabic your customers actually write
Three or four different Arabics arrive on the same channel, often from one customer in one week. Modern Standard Arabic, from people writing formally. Gulf spoken Arabic written down, which is most of it. Arabizi, in Latin letters with digits standing in for sounds Latin has no letter for, so 3 carries ع and 7 carries ح. And code switching inside a sentence, where the question is Arabic and the product name or the whole second half is English.
Treat any claim that this is solved with suspicion. In the 2025 NADI shared task on multidialectal Arabic the best system reached 79.8 per cent accuracy at merely identifying which dialect it was hearing, and the AraDiCE benchmarks found identification, generation and translation all still difficult even for Arabic-specific models. The model supports Arabic is not an answer. It is where the question starts.
Of the forty, put twelve in dialect, six in Arabizi and six code switched. The pass mark has two halves and the second gets skipped: the answer has to be correct, and in the register the customer used. Three paragraphs of Modern Standard Arabic in reply to a five-word message is a failure even when every fact is right.
Abu Dhabi’s Technology Innovation Institute publishes Arabic-first Falcon models, evaluated on dialect comprehension rather than translated English benchmarks. If you are being sold an English-first model behind a translation layer, somebody made that decision, and it is fair to ask why.
Test two: the names, numbers and dates it repeats back
This is the test that catches failures costing money rather than goodwill. An Arabic name has no single correct Latin spelling: Mohammed, Muhammad, Mohamad and Mohd are one customer and, in a careless lookup, four records or none. Put six messages in the set where a name given in Arabic has to be found under the Latin spelling in your system.
Then numerals, which almost nobody tests. Two digit systems are in daily use here, and the W3C’s Arabic and Persian layout requirements note they are not interchangeable to software: European and Arabic-Indic digits carry different bidirectional categories. That is why an order reference can appear in the wrong order beside Arabic text on a phone while being correct in the database.
So: order references, dirham amounts, dates and phone numbers, in both digit sets, buried inside ordinary sentences. The pass mark is that the value repeated back is the value typed, character for character, and reads in the right order on a phone screen. Check it on a phone, not in the admin panel. A transposed digit in a quoted price is a refund and an argument.
Test three: what it does when it does not know
The least tested behaviour and the most valuable. Ten of the forty should be questions your business genuinely cannot answer: a product you do not sell, a discount that does not exist, a delivery promise nobody authorised.
A pass is that it says so and offers the next step. A failure is a fluent, confident, wrong sentence, which is the default output of an unconstrained model. Microsoft’s guidelines for human-AI interaction put this second on a list of eighteen: make clear how well the system can do what it can do. An assistant that never signals doubt has moved the job of catching its errors onto the customer.
Two build requirements follow, both belonging in the quotation rather than a later change request: answers grounded in your own documents rather than the model’s memory, and price, stock and delivery dates read live rather than remembered. That is the narrow version of what to settle before you connect a model to a live system at all.
Test four: the handover
Your supplier will offer deflection rate. It is the wrong number: it pays the assistant for keeping customers away from people. Measure handover accuracy instead: of the conversations that should have reached a human, how many did, and how fast. Three triggers, written down before launch rather than tuned afterwards. The customer asks the same thing twice. The value is above a threshold you set, or the message is a complaint. The customer asks for a person, however they phrase it. A pass is that the colleague receives the whole conversation, in the language it happened in, and does not ask the customer to start again.
One channel constraint catches teams late. Meta’s documentation is explicit that on WhatsApp, template messages are the only type that can be sent outside a customer service window, and templates are reviewed before use. Your agent replying next morning cannot simply type a free-text answer. If nobody designed for that, the handover is a dead end after hours.
Test five: the second and third message
Demonstrations are one turn. Failures are on turn three. Run eight of the forty as scripted conversations: a correction, no, not that one, the larger size; a reference backwards, the same as last time; a change of subject and a return to it later; and the same question asked twice, phrased differently.
That last one is the most informative. An assistant that gives two different answers to one question in one conversation is worse than one that declines to answer.
Scoring it, so the result is not an opinion
Two scorers, both native Arabic readers, one from the part of the business that receives the complaints rather than from technology. Four marks per message. Correct: the answer matches your records. Register: it is the Arabic the customer used. Safe: it invented nothing and promised nothing unauthorised. Escalated: it handed over when a trigger fired.
Agree the pass marks first, because a threshold chosen after seeing results is not a threshold. Mine are ninety per cent correct, eighty-five per cent register, one hundred per cent safe, and every escalation trigger every time. The hundred is the one that matters: a single invented price stops the release.
Record every failure verbatim with the message that produced it. That file is your supplier’s backlog this month, and the regression test from the second release onwards.
What this changes in the contract
Four clauses, none exotic. The forty messages are an annex to the agreement, not a conversation in a meeting. A payment milestone attaches to the acceptance run rather than the go-live date. The supplier re-runs the set before every release and sends you the sheet. The failure file belongs to you, so it travels if the supplier does not.
None of that is unusual in ordinary software procurement, and it is still unusual in AI procurement. With acceptance settled, what remains is what the assistant is allowed to touch and what it costs to run for a year. Building that set against your own inbox rather than a template is part of the AI work I do in the UAE.
Building an assistant that clears these tests in Arabic is my company’s work: custom AI assistants at Tothiq.
Frequently asked questions
Is forty messages really enough to accept a system on?
For a first launch, yes, provided they are weighted by real frequency rather than chosen for variety. Forty real questions find more defects than four hundred invented ones, and a set that scores in an afternoon gets scored before every release. Then grow it honestly: add every failure a customer finds after launch, so it lengthens only where reality proved you were not testing something.
Most of our customers write to us in English. Do we still need the Arabic test?
Look at the channel rather than the website. Businesses that describe themselves as English-speaking usually find their WhatsApp and Instagram messages are substantially Arabic or Arabizi. Count before deciding. If the Arabic share is under a tenth, run a set of ten rather than none, and make certain the handover is language-aware.
Our supplier says the model handles all Arabic dialects. Is that not enough?
That is a claim about a model, and you are buying a system. The model is one component: your retrieval, product data, prompts and escalation rules all sit between it and the customer. The useful response is not to argue but to test it on twelve of your own dialect messages and share the sheet.
Can the supplier score their own acceptance test?
They can run it. They should not be the only ones marking it, for the same reason a builder does not sign their own snagging list. What works: the supplier runs the set and sends raw transcripts, and your two scorers mark them unseen. Disagreements are useful, because most turn out to be a definition problem in the pass mark.
How often should the set be re-run after launch?
Before every release, and once a month whether or not anything was released. The second half surprises people: the models behind these systems are updated by their providers without asking you, so an assistant that passed in March can drift by June with no code change at either end.