OpenAI unveiled more details about its new Jalapeño inference system at the Hot Chips conference this evening (the 25th), including benchmark results for the first time. In SemiAnalysis's InferenceX tests, Jalapeño outperformed the leading inference processors currently on the market in both single-user token throughput and throughput per kilowatt.
Richard Ho, OpenAI's head of hardware, said during a media conference call: “The most important point about the test results is that Jalapeño delivers a very significant performance improvement over the most advanced products available today. It can process more AI workloads with the same amount of power while responding faster. Jalapeño is well suited to efficiently serving large numbers of users while maintaining very low latency.”
Jalapeño was co-developed by OpenAI and Broadcom, with OpenAI's own models also involved in its development. OpenAI hopes to develop Jalapeño into a technology platform that can span multiple generations, with AI products, models, chips, and memory co-designed from the outset.
The chip is primarily responsible for AI inference, meaning it runs already-trained AI models to complete tasks or power AI agents.
OpenAI used the InferenceX platform to test Jalapeño's inference performance and compare it with NVIDIA GB200 and GB300 superchip systems. OpenAI said that across the GPT-OSS 120B, DeepSeek R1, and Kimi K2.5 1T models, Jalapeño can complete 1.5 to 1.9 times as much AI work per watt as the comparison systems, while end-to-end latency is approximately 28% to 59% that of the comparison systems.
Full-stack co-development also enables OpenAI to directly optimize the stages most likely to slow down inference. Jalapeño focuses on reducing latency during prefill and communication, which OpenAI considers major bottlenecks for inference performance.
In the blog post announcing the test results, OpenAI noted that Jalapeño was designed from the outset to minimize data movement and communication latency. Model state such as the KV cache used when generating responses can be explicitly stored and retained locally, while the system allocates suitable compute, memory, and network resources according to the different stages of inference.
IT Home learned that OpenAI plans to deploy Jalapeño on a “small scale” before the end of this year and gradually expand deployment in 2027, but it has not disclosed how many chips it plans to deploy next year.
Reference
OpenAI official post: Jalapeño’s first results show industry-leading speed and efficiency in AI inference