Skip to content

fix(google-genai): Handle transient 500s from Gemini TTS #1069

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/gemini-tts-server-error
Open

fix(google-genai): Handle transient 500s from Gemini TTS #1069
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/gemini-tts-server-error

Conversation

@sentry

@sentry sentry Bot commented Jul 23, 2026

Copy link
Copy Markdown

Issue

Closes #1070

Summary

  • Before: The generate_content call only attempted a single retry on ServerError, causing unhandled exceptions.
  • Now: A retry mechanism with exponential backoff has been implemented, allowing up to 3 retries for ServerError.
  • Added a loop for retries with increasing delays (2 seconds, then 4 seconds).
  • Emitted warning logs before each retry for visibility into transient failures.

Checklist

Before submitting a pull request, please ensure that you mark these task.

  • Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
  • If you've fixed a bug or added code that is tested and has test cases.

Notes

Please add here if any other information is required for the reviewer.

Original PR description

Issue

Closes #PLEASE_TYPE_ISSUE_NUMBER

Summary

This PR addresses the ServerError intermittently reported by the gemini-2.5-flash-preview-tts model, particularly when generating Hindi audio.

Motivation & Problem:
The Google GenAI SDK was raising ServerError (HTTP 500) due to transient issues with the gemini-2.5-flash-preview-tts model, which is currently in preview. The existing retry configuration for the generate_content call was set to only one attempt, meaning any single 500 error would immediately propagate as an unhandled exception, crashing the Celery task.

Root Cause:

  1. Intermittent instability in the gemini-2.5-flash-preview-tts preview model, leading to transient HTTP 500 responses for certain requests (e.g., Hindi audio generation).
  2. Insufficient retry logic: The tenacity configuration for the generate_content call was effectively stop_after_attempt=1, preventing automatic retries for these transient server errors.

Solution:
Implemented a retry mechanism with exponential backoff specifically for genai_errors.ServerError within the _execute_tts method of the GoogleAIProvider (backend/app/services/llm/providers/google_aistudio.py).

  • The self.client.models.generate_content call is now wrapped in a loop that attempts up to 3 retries.
  • Retries occur with increasing delays (2 seconds, then 4 seconds) for ServerError exceptions.
  • A warning log is emitted before each retry, providing visibility into transient failures.
  • If all retries are exhausted, the error is caught and returned gracefully as (None, error_message), preventing unhandled exceptions and allowing the Celery task to fail cleanly.

Checklist

Before submitting a pull request, please ensure that you mark these task.

  • Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
  • If you've fixed a bug or added code that is tested and has test cases.

Notes

Please add here if any other information is required for the reviewer.

Fixes KAAPI-PRODUCTION-J

@github-actions github-actions Bot changed the title fix(google-genai): Handle transient 500s from Gemini TTS fix(gemini-tts): Improve error handling retries Jul 23, 2026
@github-actions

Copy link
Copy Markdown

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

mainfda60c28 · generated by oasdiff

@Prajna1999 Prajna1999 changed the title fix(gemini-tts): Improve error handling retries fix(google-genai): Handle transient 500s from Gemini TTS Jul 23, 2026
@Prajna1999 Prajna1999 self-assigned this Jul 23, 2026
@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 56.25000% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...kend/app/services/llm/providers/google_aistudio.py 56.25% 7 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Prajna1999 Prajna1999 added enhancement New feature or request ready-for-review labels Jul 23, 2026
@Prajna1999 Prajna1999 linked an issue Jul 23, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request ready-for-review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemini TTS: Implement retry mechanism

1 participant