OpenAI has expanded Habitat, its online storage platform, from a Python client library into a distributed service handling more than 70 million requests per second. The company says the platform now supports products used by over 1 billion people each week across almost 40 geographic regions and serves more than 500 petabytes of data.
Habitat began in mid-2024 as a small Python library connected to ChatGPT’s main server and Azure Cosmos DB. It gave product teams a common way to store and retrieve data while handling schema lookup, routing, authorization, encryption, serialization, request shaping and connection pooling, without requiring engineers to manage the underlying storage systems directly.
By the middle of 2025, OpenAI determined that the client-side design could no longer support increasingly complex products and protocol changes. Updates required coordination across dozens of services, and one rollout intended to reduce the impact of a regional outage was undermined when an unrelated service rollback restored a previously buggy client. OpenAI consequently moved Habitat’s storage logic into a standalone service, creating a central point for deployments, observability, access controls, audit logging and restrictions on access to storage resources such as Azure Cosmos DB.
OpenAI initially retained Python despite its network-latency, CPU and memory overhead because the company prioritized platform stability, core APIs and faster product development. The design required close management of tail latency: Habitat combines I/O-heavy request proxying with CPU-heavy work including routing, compression, encryption, checksumming, health checks, shadowing and hedging. OpenAI found that asyncio scheduling delays could reach hundreds of milliseconds and, in some edge cases, several seconds, so it limited concurrent requests per process and scaled out the number of Python workers.
Live CPU profiling also identified periodic parsing of Statsig feature-flag configurations as a cause of high tail latency. Each pod could run up to eight Python processes, and all workers polled and parsed a large production configuration every minute without jitter. OpenAI responded by deploying a smaller targeted configuration, extending the refresh interval and adding timing jitter to background tasks.
Connection pooling produced another failure mode. Some processes received five to 10 times the average number of concurrent requests, while aiohttp’s default last-in, first-out connection reuse repeatedly directed traffic toward slower, overloaded servers. OpenAI says changing the pool to first-in, first-out reuse broke that feedback loop and reduced steady-state request variance. Habitat now largely relies on Istio and Envoy for connection pooling and server-load-aware balancing; Envoy also converts Python HTTP/1 connections to multiplexed HTTP/2 connections and provides a central layer for rate limits and circuit breakers.