Voice AI releases, failover and recovery
Releases move new calls across without dropping one in progress. A second region answers when Sydney cannot, and canary calls test the line around the clock.
Releases without dropped calls
Two voice services take turns. Releases into your account run inside the change windows you set.
Synthetic stage
The new version starts on the idle voice service and takes simulated calls over the real phone path.
Test numbers
Calls to your test numbers go to it next, while customers stay on the current version.
Gates
It moves on only with 0 dropped calls, 0 duplicate replies, and greeting and turn times within 100 ms of the last version.
Cutover
New calls go to the new version, and calls in progress finish where they started.
Rollback
One switch returns new calls to the previous version within 5 seconds.
Failover to Melbourne
Answering
If Sydney does not answer a call event within 5 seconds, Telnyx sends it to a handler in Melbourne, which answers and forwards the caller to your fallback number.
One owner per call
Only the handler that answered a call sends it later commands, so a caller never hears two systems at once.
Restore
The database restores in Melbourne from the daily copy, with a replica of the encryption key.
Backups and recovery
Recovery
- Point-in-time recovery
- 7 days, in Sydney
- Daily copy
- A separate account in Melbourne
- Backup key
- Separate, held in Melbourne
- Copies kept
- 7
- Recovery point if Sydney is lost
- 24 hours
- Recovery time
- Set in your contract
A database failure
Single-tenant deployments run a database reader, which takes over in under 60 seconds. Calls in progress continue, and new calls load the last good copy of each agent from S3.
Monitoring and canary calls
Canary calls
A test call every 10 minutes in business hours and every 30 minutes overnight, checked for a greeting within 3 seconds of answer. Two failures in a row page the on-call engineer.
Daily failover test
Once a day a call goes to a line whose primary handler is switched off, to prove Melbourne answers and forwards within 6 seconds.
Alarms
Alarms live in CloudWatch in the call plane account and page from Melbourne, so a Sydney outage cannot silence them.
Access and change control
No servers to patch
Every service runs on AWS Fargate. A retirement notice from AWS starts a release of the same version within the 7-day notice.
Break-glass access
Engineers reach a deployment through a time-limited role, and every session is logged in the account's CloudTrail.
Change windows
Releases into your account run inside the windows you set, and you can disable our role between them.
Deploys by API
Deploys go through the ECS API, never a stack update, so no deploy waits on or stops a task that is holding a call.
Security questionnaires
We answer security questionnaires in writing and take your team through the architecture on a call. Send yours to hello@verticalai.com.au.