[Proposal] Support Python `asyncio` by migrating to `httpx` from `requests`

[adarshdigievo]

Motivation

• We currently use the requests library for sending HTTP requests. While requests is stable and widely used, it is synchronous-only and does not support Python’s asyncio concurrency model.
• This limits users who build async Python applications. Even if their application code uses async / await, the HTTP layer still blocks unless the underlying client library also supports async. A related request came up in our deprecated library: [Async] Provide async requests for GoogleSearch google-search-results-python#72
• Supporting async is increasingly expected in modern Python SDKs, especially for APIs, since the async pattern works well with I/O blocked apps.
• Most of the popular Python packages are already using httpx: eg, OpenAI and Anthropic Python SDKs, and Python MCP SDK

Plan:

• Migrate the HTTP layer from requests to httpx.
• Ref: Official migration guide from httpx: https://www.python-httpx.org/compatibility/.
• The migration should require minimal changes because HTTPX is mostly compatible with the requests API style.
• The changes will be made with full backward compatibility; there will not be any breakages for current users.
• I will also add benchmarks to assess if there are any changes to performance as a part of this migration.
• We will support the existing pattern:

import os
import serpapi

def send():
  client = serpapi.Client(api_key=os.getenv("SERPAPI_KEY"))
  results = client.search({
    "engine": "google",
    "q": "coffee"
  })

  print(results)

and will also add support for async calls:

import os
import serpapi

async def send():
  client = serpapi.Client(api_key=os.getenv("SERPAPI_KEY"))
  results = await client.search({
    "engine": "google",
    "q": "coffee"
  })

  print(results)

Users can choose the pattern that best fits their application.

[jvmvik]

I agree with the motivation. I have already created a branch with 0.2.0 based on aiohttp which is based on asyncio.

But I stopped in my tracks for 2 reasons related to customer concerns.

• python 2.x is no longer supported.
• dependencies: new dependencies which breaks enterprise stacks.
• synchronization with https://serpapi.com/integrations/python

Historically we have done a bad job synchronizing the web documentation update, and the plugin release.
The plugin doc is pull automatically in the website, but the "Code to integrate" need manual update.

solutions:

1. drop python2 compatibility. (need to check market share left of python 2)
2. add the new library under python3 entry, and keep python2

I'd like 2 because it does break existing enterprise stack, and allow them to migrate slowly. but maybe this argument doesn't stand anymore in the age of AI.

what do you think ?

[adarshdigievo]

Hi @jvmvik,

Thanks for sharing the details. Please see my comments below:

• Breaking changes/updates needed for integrations page: My idea is to keep the existing sync API as-is. We will not break any existing code. Existing users will not require any code changes.

Even after the httpx migration, the below sync usage will work:

import os
import serpapi

def send():
  client = serpapi.Client(api_key=os.getenv("SERPAPI_KEY"))
  results = client.search({
    "engine": "google",
    "q": "coffee"
  })

  print(results)

---

For Async, we can introduce a new client (serpapi.AsyncClient)- this was not mentioned correctly in my previous comment:

import os
import serpapi

async def send():
  client = serpapi.AsyncClient(api_key=os.getenv("SERPAPI_KEY"))
  results = await client.search({
    "engine": "google",
    "q": "coffee"
  })

  print(results)

This is the advantage I see with httpx: it supports both sync and async network requests and has a design that is almost compatible with our current requests library. By migrating to httpx, we can maintain backward compatibility with our user-facing methods and support async as well.

Because of the backward compatibility, we can keep our integrations page as-is and can mention async support in our library docs.

• Regarding the performance benefits, I agree that aiohttp will be more performant. But as you rightly noted, a full migration will be costly - we need to add a new package so that users of our current sync client will not see any breaking changes. Or we need to have sync calls from our package using requests under the hood (keeping existing behavior) and implement new async methods that use aiohttp, which will increase the maintenance burden.

Before starting work on this, I will run a benchmark using our real API calls to measure actual performance.

If we are moving forward with httpx, we can implement support for both sync and async calls with a single dependency with minimal changes to existing code. This is the unique benefit of httpx.

# sync
r = httpx.get(''https://www.example.com/')

# async
async with httpx.AsyncClient() as client:
     r = await client.get('https://www.example.com/')

Also, the community is very active around httpx - Pydantic is maintaining their fork of httpx, and I expect to see performance improvements going forward.

• Python 2 compatibility: I don't think this should be a concern for us. Python 2 reached end of life in 2020, and most of our dependencies do not support it. The current version of requests supports only Python 3.9+. Our pyproject.toml currently lists the required Python version as >=3.6, so Python2 support is safe to ignore. I also verified this using a Python 2.7 Docker image: pip install serpapi fails on Python 2.

• Dependency breakages: As a part of this change, we will have our dependency changed from requests to httpx. Due to backward-compatible design, if a customer integration gets our updated client, their existing code will work without any major updates. Also, since we will be bumping the version number for the release, pinned package install will have the previous version using requests installed.
  I guess internal dependency migrations like this are common (saw that OpenAI Python SDK also made a similar migration to httpx from requests, ref)

[jvmvik]

Convince me httpx is the right choice. Benchmarks between httpx and aiohttp would likely be too close to show
meaningful differences. Most developers avoid async programming because it adds substantial complexity for minimal
returns in real-world applications—the only real benefit is high performance, and Python isn't a high-performance
language anyway.

[jvmvik]

about the package manager, what are you planning to use ? for marketing reason we should document both uv and pip3

[adarshdigievo]

@jvmvik Please see https://serpapi.slack.com/archives/C09AS452KMZ/p1781258667885529 - added a few benchmark results and analysis on our internal discussion thread.