Most voice-AI demos are built for a world with no consequences. A collections call is the opposite: a regulated conversation between a lender and a borrower, where the lender is legally answerable for every sentence, the contact window is fixed by the regulator, and the data flowing through the pipeline is some of the most tightly controlled in the country.
That set of constraints isn’t an obstacle to the architecture. It is the architecture. Get the constraints right and most of the hard design decisions make themselves — including the one that surprises people most, which is that this system cannot run on a cloud speech API.
What DPD actually means
DPD — Days Past Due — is the number of days a payment is overdue. Lenders report it monthly to the credit bureaus, and it’s the spine of every collections operation in the country.
Two codes cause endless confusion. 000 means zero delay in that month — the payment landed on time. XXX means the lender didn’t report payment history for that period at all, which is an absence of data rather than a clean record. So a report full of 000 is a good report; a DPD value of 60 means the borrower is sixty days late.
The buckets are what matter operationally:
| Bucket | Classification | What the conversation actually is |
|---|---|---|
0 (000) |
Standard | Nothing. Don’t call. |
| 1–30 | Standard, early delinquency | A reminder. Often just a failed auto-debit or a forgotten date. |
| 31–60 | SMA-1 | A reason-finding call. Something changed — find out what. |
| 61–90 | SMA-2 | A restructuring conversation. Options, not pressure. |
| 90+ | NPA | Legal and settlement territory. Largely not an automation problem. |
A high DPD can sit on a credit report for up to seven years, which is precisely why the day-5 call and the day-75 call are different products, not different prompts. The day-5 borrower usually isn’t in distress — a NACH mandate bounced, or the salary date moved. The day-75 borrower is in a genuinely different situation, and a system that talks to both the same way is not just ineffective, it’s a compliance risk.
Any voice system worth building takes the bucket as its primary input and changes its goal accordingly: confirm and collect in bucket 1, diagnose in bucket 2, present options in bucket 3, and hand to a human beyond that.
The compliance envelope you are building inside
Before any architecture, the hard limits. These are not internal policy — they’re supervisory requirements, and the lender carries the liability whether the caller is a salaried employee, a third-party agency, or software.
- Contact hours are 08:00 to 19:00, seven days a week, under RBI’s Fair Practices Code. Not “usually”. Not “unless the borrower answers”.
- The lender is responsible for agent conduct, full stop. Outsourcing the calling does not outsource the accountability.
- Specific practices are banned outright: accessing the borrower’s contact list, photos or location; sending shaming messages to people in the borrower’s phonebook; and — directly relevant to anyone building this — collection bots that call from rotating numbers to overwhelm the borrower.
- The Responsible Business Conduct (Second Amendment) Directions took effect on 1 July 2026, tightening agent conduct and escalation requirements further.
That third point deserves emphasis, because it defines what this post is not about. RBI has explicitly prohibited the bot-swarm pattern: many automated calls, rotating CLIs, designed to wear a borrower down. If that’s the system someone wants, the technology isn’t the problem and no architecture will fix it. Everything below assumes the opposite goal — fewer, better, fully logged conversations that a supervisor could audit line by line.
Above these sits RBI’s FREE-AI framework — the Framework for Responsible and Ethical Enablement of Artificial Intelligence, published 13 August 2025. It sets out six pillars, seven guiding principles and twenty-six recommendations, and takes a proportionate line: light-touch for low-risk uses, real supervisory involvement for high-risk ones. An autonomous system that calls retail borrowers about money they owe is not the low-risk end of that spectrum. Two of its principles — accountability, and understandable-by-design — translate directly into engineering requirements later in this post.
Why on-premise isn’t a preference here
This is the part that decides the whole stack.
RBI’s circular of 6 April 2018 requires that payment system data — customer data, payment-sensitive data, payment credentials, transaction data — be stored only in India. There is no exemption for small volumes, small companies, or particular transaction types.
Now consider what is actually present in sixty seconds of a collections call: the borrower’s identity, their loan account, the outstanding amount, the due date, often their reason for non-payment, and sometimes payment credentials as they arrange a transfer. The audio itself is personal data with biometric character — a voiceprint is not anonymised by removing the name.
Streaming that audio to a US-hosted ASR endpoint is not a technical detail. It is the data leaving. The same applies to a cloud LLM that sees the transcript, and to a cloud TTS service that receives the amount and account details in the text it’s asked to speak.
The DPDP Act 2023 doesn’t rescue you either. It permits cross-border transfer by default and reserves the government’s power to restrict specific destinations — a more permissive model than RBI’s. But a lender complies with both frameworks, and the stricter one governs. DPDP’s permissiveness on transfers does not override the RBI mandate on payment data.
So the architecture is forced: ASR, LLM and TTS all run on infrastructure you control, inside the country. FREE-AI’s encouragement of indigenous model development points the same way. This is the clearest example I know of where self-hosting isn’t the cost-optimisation argument made in the local-first router pattern — it’s a licensing condition.
The pipeline, and the latency budget that governs it
Speech-to-speech is three models in a streaming chain: ASR → LLM → TTS. What makes it hard is not any single component but the budget they share.
In natural conversation, the gap between turns sits somewhere around 200–300 ms. Push past roughly 800 ms and the call stops feeling like a conversation — people talk over each other, repeat themselves, and hang up. On a collections call, where the borrower is already uncomfortable, that awkwardness reads as evasiveness.
A workable budget looks roughly like this:
| Stage | Target | What blows it |
|---|---|---|
| Endpointing (VAD) | 100–200 ms | Waiting for silence that never comes; over-eager cut-off mid-sentence |
| ASR final hypothesis | 100–200 ms after speech ends | Batch transcription instead of streaming partials |
| LLM first token | 150–300 ms | A long prompt on every turn — the TTFT tax |
| TTS first audio chunk | 100–200 ms | Synthesising the whole utterance before playing any of it |
The rule that follows: stream everything, wait for nothing. Feed ASR partials to the LLM before the borrower finishes, start TTS on the first clause rather than the full response, and keep the system prompt short and cached — the same prompt and cache discipline that fixes slow LLM apps applies here with a much tighter deadline.
Barge-in is non-negotiable. When the borrower interrupts, playback stops immediately and the system listens. A bot that talks over a person explaining why they can’t pay is not just poor UX; in this domain it is the behaviour the regulator is watching for.
Indic ASR is the hard part
This is where most off-the-shelf pipelines quietly fall apart, and where the engineering effort actually goes.
Collections calls in India run in Hindi, Tamil, Telugu, Marathi, Bengali and a dozen more — and rarely in one language at a time. Real speech is code-mixed: “sir woh payment maine kal hi kar diya, NACH bounce ho gaya tha”. A model trained on clean English audio produces confident nonsense on that sentence.
Four problems worth budgeting for:
- Numbers and amounts must be exact. “Paanch hazaar” and “fifty thousand” and “5,000” have to land on the same value. In a general assistant a misheard number is an annoyance; here, a system that confirms the wrong amount or the wrong date has created a record of a commitment the borrower never made. That’s a compliance incident, not a bug.
- Telephony audio is 8 kHz. Most ASR models are trained on 16 kHz. Narrowband, compressed, often on a moving handset with background noise — the accuracy you measured on clean audio is not the accuracy you will get on a call.
- Domain vocabulary is unusual. EMI, NACH, mandate, bounce, foreclosure, settlement, moratorium, one-time settlement. These need to be in the lexicon or biased in decoding, or they will be transcribed as something phonetically close and semantically useless.
- TTS prosody carries weight. A voice that sounds cheerful while discussing a missed payment is worse than a flat one. Getting Indic TTS to sound calm and neutral rather than either robotic or inappropriately bright is real work, not a config flag.
The honest summary is that language coverage — not model size — is usually what determines whether this system works in production.
Sizing it on your own hardware
The good news after all those constraints: the models involved are not large.
Most turns in a collections call are classification and slot-filling — identify intent, extract a promised date and amount, decide the next state. That is exactly the work small models handle well and where a 7–14B model at 4-bit quantisation is comfortable. The generation is short and highly constrained; you are not writing essays.
The practical sizing questions are about concurrency, not capability. Each simultaneous call holds ASR state, LLM KV cache and a TTS stream, so the number of concurrent conversations per GPU — not tokens per second on a single stream — is the capacity metric that matters. The VRAM and hardware maths works the same way here, with the caveat that peak calling hours are compressed into that 08:00–19:00 window, so you size for the peak and idle the rest of the day. The self-hosting stack is well-trodden ground.
What the system must never do
These belong in code, not in a policy document that a configuration change can override:
- A hard stop at 19:00 and no start before 08:00, enforced at the dialer, in the borrower’s local time — not a campaign setting someone can extend.
- One number, consistently. No rotating CLIs. This is the specific pattern RBI prohibited.
- Disclose that the caller is an AI, early and plainly. A synthetic voice that lets a borrower believe they negotiated with a person has created a commitment on a false premise.
- Immediate handoff on distress, dispute, or any mention of legal action — hardship, bereavement, illness, “I’m going to complain”, “my lawyer”. The model’s job at that point is to recognise the trigger and get out of the way, not to handle it.
- Frequency caps per borrower, counted across every channel, not per campaign.
- Everything retained in India — audio, transcript and decision log, with the retention period set deliberately rather than defaulting to forever.
The audit trail is the product
For a regulated deployment, the logging is not an operational nicety. It is the thing that makes the system defensible when a complaint arrives eight months later.
At minimum, every call needs to be reconstructable end to end: the audio, the transcript with timestamps, the bucket and account state at dial time, the model versions pinned for that specific call, and the reason the system said what it said at each turn. When a borrower alleges harassment or a mis-stated amount, “the model generated it” is not an answer a supervisor will accept.
This is where FREE-AI’s principles stop being abstract. Accountability means being able to attribute a decision to a version, an input and a policy. Understandable by design means the collections head — not just the ML team — can read why the system offered a particular restructuring option. If you cannot produce that trace, you do not have a deployable system; you have a demo.
The honest boundary
Automation earns its place in the early buckets. A day-5 reminder, a confirmation of a promise-to-pay, a re-presentation of a bounced mandate — high volume, low complexity, low emotional stakes, and genuinely better for the borrower when it’s quick and unembarrassing.
It loses its place fast after that. Hardship conversations, disputed amounts, vulnerable borrowers, anything past 90 DPD heading toward settlement or legal action — these need a human, and a system designed to keep talking in those situations is designed wrong. The measure of a good collections voice system is not how many conversations it completes. It’s how reliably it recognises the ones it should never have been in.
Building something in a regulated domain where the audit trail matters as much as the output? That’s the kind of system we build.