Can my chatbot leak customer data?
Yes, if it can reach data the current user is not allowed to see. The usual causes are a shared vector store searched without filtering by customer, conversation memory shared across sessions, tools that query the database with an admin key, and personal information placed in the system prompt. The model cannot keep a secret it has been given, so the fix is never giving it the wrong user's data.
Common causes
| Cause | Fix |
|---|---|
| Retrieval across all customers' documents | Filter by tenant and user before retrieval |
| Tool queries with a service key | Run tools with the user's own permissions |
| Shared conversation memory or cache | Key memory and caches by user |
| Customer data in the system prompt | Pass only the current user's data per request |
| Logs and traces containing prompts | Restrict access and retention; redact |
The model provider
Data sent to a provider is processed under its terms. Use settings that exclude your data from training and check retention. Under PIPEDA you stay accountable for it. See sending data to OpenAI and PIPEDA.
How to test
Create two customer accounts with distinctive data. As one, ask the assistant about the other's data by name, by paraphrase and through injected instructions. Anything returned is a finding. This is LLM02 in the OWASP LLM Top 10.
The forgotten copy
Conversation logs and AI observability tools keep full prompts, which include whatever data was retrieved for each answer. A chatbot that never leaks data to users can still expose it through a logging dashboard shared with a contractor or a third-party monitoring service with broad access. Treat these tools as systems holding personal information: restrict access, set retention, list them as service providers, and redact where you can.
Also check the model provider's data retention settings and whether your plan keeps prompts for abuse monitoring. Know the answer before a customer asks, because it is on most AI questionnaires.
Getting it checked
TrazTech offers AI and LLM security assessments, listed from $4,000 CAD. Get at least one other quote on the same scope; the questions to ask a testing firm help compare them.
Related questions
Get a scope for your app
Tell us what you built, what it stores and who is about to use it.
Get matchedCommon questions
Does the model provider leak my data to other companies?
Major providers' business terms generally say API data is not used for training by default, but check the current terms of your provider and plan.
Is a leak through a chatbot a privacy breach?
Yes, if personal information reaches someone not entitled to it. PIPEDA and Law 25 breach duties apply.