Building SafeChat for a Real-Time Marketplace
At QCon AI, Bruna Pereira described how DoorDash built SafeChat to protect consumers, Dashers, and merchants who communicate through chat, calls, images, and in-person interactions. Because these relationships may last only 40 minutes to 6 minutes, safety decisions must happen quickly.
DoorDash processes more than 4 million chat messages, 400,000 delivery-related calls, and 200,000 images each day. Chat moderation must classify messages in a fraction of a second without disrupting delivery.
A purely LLM-based design was too slow and expensive: calls averaged 2 to 10 seconds, and invoking an LLM 4 million times daily was not economically practical.
A Two-Layer Moderation Architecture
The team first spent months studying its data rather than immediately making automated decisions. It instrumented chat and used an asynchronous commercial moderation API to identify categories and estimate the share of unsafe messages. The analysis showed that only a small single-digit percentage of messages were unsafe.
DoorDash then trained an internal classifier to identify messages that were obviously safe. The model was designed to respond in less than 100 milliseconds at the 90th percentile, had no per-call vendor charge, and served as a screening layer rather than a final judge. Messages it could not confidently classify were sent to an LLM, representing less than 10% of traffic.
Instead of asking the LLM for a safe-or-unsafe Boolean, SafeChat requests scores across dimensions such as threat, profanity, and sexual content. Scores allow DoorDash to adjust thresholds, add categories, and apply actions based on severity.
Low-severity profanity can be censored while allowing delivery. Medium-severity insults can be blocked. Threats can trigger message blocking and an option for the affected party to cancel without charge. Very high-severity content can result in cancellation, warnings, message blocking, and removal of the affected party from the interaction.
Images use a commercial vision API as the fast layer for detecting violence, nudity, and other unsafe material. Voice cannot be blocked after transcription because the recipient has already heard it, but calls can be ended and orders can be canceled.
Measured Impact
After SafeChat was implemented, DoorDash recorded an approximately 50% reduction in incidents driven by verbal abuse. Pereira emphasized that this measured a reduction in human harm, not merely an improvement in model accuracy.
From SafeChat to a General Moderation Platform
Other teams requested similar capabilities for profile pictures, signup names, food reviews, fraud detection, and phone or chat interactions. Rather than recreate the system for every use case, DoorDash extracted the underlying pattern: a cheap filter, an expensive model for difficult cases, and graduated actions.
The resulting content-agnostic moderation platform does not interpret business meaning. Teams define that meaning and the desired actions, while the platform handles decision logging, model-provider integration, conditional steps, and orchestration through a no-code UI.
The platform supports three model types:
- Internal models: trained, fine-tuned, and served on DoorDash infrastructure through standardized APIs. They are intended to be fast and avoid per-call costs.
- External models: vendor services used where existing solutions, such as image safety, are already effective.
- External prompts: configurable prompts sent through an LLM gateway to models from different providers.
The gateway supports provider selection, input and output schemas, retries, and fallbacks. A fallback may use another vendor, an internal model, or an LLM when the preferred service is unavailable.
Moderation Agents and Backtesting
Teams compose model steps into moderation agents. Conditions, configured in the UI, route results to another model or an action. The original SafeChat workflow sends a message to the internal model; when its unsafe score exceeds 0.5, the message proceeds to an LLM prompt that returns category scores.
Agents can run synchronously when they must gate a decision, such as whether a chat message is delivered, or asynchronously when latency is less restrictive. Asynchronous executions are acknowledged immediately, run in the background, and publish results to a Kafka topic for subscribed clients.
Backtesting lets teams evaluate complete agents or individual steps against historical data. Humans label results as correct or incorrect and, where appropriate, as true positives, true negatives, false positives, or false negatives. The platform uses those labels to calculate metrics before production deployment.
For gray-area cases, Pereira said DoorDash generally trusts a manually reviewed set of about 1,000 examples. Larger sets can reduce review quality when people begin classifying results casually.
Operational Lessons
Pereira recommended placing a cheap model in front of an LLM on high-volume paths, learning from production data before choosing an architecture, and requesting severity scores rather than binary labels. Thresholds should be adjusted using observed data.
The internal classifier has been retrained nine times for SafeChat. Feedback from safety agents exposed missed patterns such as abbreviations, while human complaints led DoorDash to add a disrespect category. Model training remains the responsibility of the ML platform team; the moderation platform consumes already-trained models.
Her final lesson was to recognize when a successful system should be replaced by a reusable platform. Building code may be inexpensive, but maintaining many separate implementations is not.