In my earlier submit, Immediate Caching — what it’s, the way it works, and the way it can prevent some huge cash and time when operating AI-powered apps with excessive visitors. In at the moment’s submit, I stroll you thru implementing Immediate Caching particularly utilizing OpenAI’s API, and we focus on some frequent pitfalls.
A short reminder on Immediate Caching
Earlier than getting our palms soiled, let’s briefly revisit what precisely the idea of Immediate Caching is. Immediate Caching is a performance offered in frontier mannequin API companies just like the OpenAI API or Claude’s API, that enables caching and reusing components of the LLM’s enter which are repeated steadily. Such repeated components could also be system prompts or directions which are handed to the mannequin each time when operating an AI app, together with some other variable content material, just like the consumer’s question or data retrieved from a information base. To have the ability to hit cache with immediate caching, the repeated components of the immediate have to be originally of it, specifically, a immediate prefix. As well as, to ensure that immediate caching to be activated, this prefix should exceed a sure threshold (e.g., for OpenAI the prefix needs to be greater than 1,024 tokens, whereas Claude has completely different minimal cache lengths for various fashions). So far as these two circumstances are glad — repeated tokens as a prefix exceeding the dimensions threshold outlined by the API service and mannequin — caching will be activated to realize economies of scale when operating AI apps.
In contrast to caching in different parts in a RAG or different AI app, immediate caching operates on the token degree, within the inner procedures of the LLM. Particularly, LLM inference takes place in two steps:
- Pre-fill, that’s, the LLM takes into consideration the consumer immediate to generate the primary token, and
- Decoding, that’s, the LLM recursively generates the tokens of the output one after the other
Briefly, immediate caching shops the computations that happen within the pre-fill stage, so the mannequin doesn’t must recompute it once more when the identical prefix reappears. Any computations happening within the decoding iterations part, even when repeated, aren’t going to be cached.
For the remainder of the submit, I will probably be focusing solely on the usage of immediate caching within the OpenAI API.
What in regards to the OpenAI API?
In OpenAI’s API, immediate caching was initially launched on the 1st of October 2024. Initially, it provided a 50% low cost on the cached tokens, however these days, this low cost goes as much as 90%. On prime of this, by hitting their immediate cache, extra financial savings on latency will be achived as much as 80%.
When immediate caching is activated, the API service makes an attempt to hit the cache for a submitted request by routing the submitted immediate to an acceptable machine, the place the respective cache is anticipated to exist. That is known as the Cache Routing, and to do that, the API service usually makes use of a hash of the primary 256 tokens of the immediate.
Past this, their API additionally permits for explicitly defining a the prompt_cache_key parameter within the API request to the mannequin. That may be a single key defining which cache we’re referring to, aiming to additional enhance the probabilities of our immediate being routed to the proper machine and hitting cache.
As well as, OpenAI API supplies two distinct kinds of caching with regard to period, outlined by the prompt_cache_retention parameter. These are:
- In-memory immediate cache retention: That is basically the default sort of caching, out there for all fashions for which immediate caching is accessible. With in-memory cache, cached knowledge stay lively for a interval of 5-10 minutes beteen requests.
- Prolonged immediate cache retention: This out there for particular fashions. Prolonged cache permits for conserving knowledge in cache for loger and as much as a most of 24 hours.
Now, with regard to how a lot all these price, OpenAI expenses the identical per enter (non cached) token, both we have now immediate caching activated or not. If we handle to hit cache succesfully, we’re billed for the cached tokens at a significantly discounted value, with a reduction as much as 90%. Furthermore, the value per enter token stays the identical each for the in reminiscence and prolonged cache retention.
Immediate Caching in Observe
So, let’s see how immediate caching truly works with a easy Python instance utilizing OpenAI’s API service. Extra particularly, we’re going to do a practical state of affairs the place a lengthy system immediate (prefix) is reused throughout a number of requests. In case you are right here, I suppose you have already got your OpenAI API key in place and have put in the required libraries. So, the very first thing to do can be to import the OpenAI library, in addition to time for capturing latency, and initialize an occasion of the OpenAI shopper:
from openai import OpenAI
import time
shopper = OpenAI(api_key="your_api_key_here")then we will outline our prefix (the tokens which are going to be repeated and we’re aiming to cache):
long_prefix = """
You're a extremely educated assistant specialised in machine studying.
Reply questions with detailed, structured explanations, together with examples when related.
""" * 200 Discover how we artificially enhance the size (multiply with 200) to ensure the 1,024 token caching threshold is met. Then we additionally arrange a timer in order to measure our latency financial savings, and we’re lastly able to make our name:
begin = time.time()
response1 = shopper.responses.create(
mannequin="gpt-4.1-mini",
enter=long_prefix + "What's overfitting in machine studying?"
)
finish = time.time()
print("First response time:", spherical(finish - begin, 2), "seconds")
print(response1.output[0].content material[0].textual content)So, what will we anticipate to occur from right here? For fashions from gpt-4o and newer, immediate caching is activated by default, and since our 4,616 enter tokens are effectively above the 1,024 prefix token threshold, we’re good to go. Thus, what this request does is that it initially checks if the enter is a cache hit (it isn’t, since that is the primary time we do a request with this prefix), and since it isn’t, it processes the whole enter after which caches it. Subsequent time we ship an enter that matches the preliminary tokens of the cached enter to some extent, we’re going to get a cache hit. Let’s examine this in follow by making a second request with the identical prefix:
begin = time.time()
response2 = shopper.responses.create(
mannequin="gpt-4.1-mini",
enter=long_prefix + "What's regularization?"
)
finish = time.time()
print("Second response time:", spherical(finish - begin, 2), "seconds")
print(response2.output[0].content material[0].textual content)
Certainly! The second request runs considerably quicker (23.31 vs 15.37 seconds). It’s because the mannequin has already made the calculations for the cached prefix and solely must course of from scratch the brand new half, “What’s regularization?”. Because of this, through the use of immediate caching, we get considerably decrease latency and diminished price, since cached tokens are discounted.
One other factor talked about within the OpenAI documentation we’ve already talked about is the prompt_cache_key parameter. Particularly, in response to the documentation, we will explicitly outline a immediate cache key when making a request, and on this manner outline the requests that want to make use of the identical cache. Nonetheless, I attempted to incorporate it in my instance by appropriately adjusting the request parameters, however didn’t have a lot luck:
response1 = shopper.responses.create(
prompt_cache_key = 'prompt_cache_test1',
mannequin="gpt-5.1",
enter=long_prefix + "What's overfitting in machine studying?"
)
🤔
Evidently whereas prompt_cache_key exists within the API capabilities, it isn’t but uncovered within the Python SDK. In different phrases, we can not explicitly management cache reuse but, however it’s reasonably computerized and best-effort.
So, what can go flawed?
Activating immediate caching and truly hitting the cache appears to be type of simple from what we’ve stated to this point. So, what could go flawed, leading to us lacking the cache? Sadly, plenty of issues. As simple as it’s, immediate caching requires plenty of completely different assumptions to be in place. Lacking even a kind of conditions goes to end in a cache miss. However let’s take a greater look!
One apparent miss is having a prefix that’s lower than the edge for activating immediate caching, specifically, lower than 1,024 tokens. Nonetheless, that is very simply solvable — we will at all times simply artificially enhance the prefix token depend by merely multiplying by an acceptable worth, as proven within the instance above.
One other factor can be silently breaking the prefix. Particularly, even after we use persistent directions and system prompts of acceptable dimension throughout all requests, we have to be exceptionally cautious to not break the prefixes by including any variable content material originally of the mannequin’s enter, earlier than the prefix. That may be a assured method to break the cache, irrespective of how lengthy and repeated the next prefix is. Typical suspects for falling into this pitfall are dynamic knowledge, as an example, appending the consumer ID or timestamps originally of the immediate. Thus, a finest follow to comply with throughout all AI app improvement is that any dynamic content material ought to at all times be appended on the finish of the immediate — by no means originally.
In the end, it’s value highlighting that immediate caching is simply in regards to the pre-fill part — decoding isn’t cached. Because of this even when we impose on the mannequin to generate responses following a particular template, that beggins with sure mounted tokens, these tokens aren’t going to be cached, and we’re going to be billed for his or her processing as typical.
Conversely, for particular use circumstances, it doesn’t actually make sense to make use of immediate caching. Such circumstances can be extremely dynamic prompts, like chatbots with little repetition, one-off requests, or real-time personalised techniques.
. . .
On my thoughts
Immediate caching can considerably enhance the efficiency of AI purposes each by way of price and time. Particularly when trying to scale AI apps immediate caching comes extremelly helpful, for sustaining price and latency in acceptable ranges.
For OpenAI’s API immediate caching is activated by default and prices for enter, non-cached tokens are the identical both we activate immediate caching or not. Thus, one can solely win by activating immediate caching and aiming to hit it in each request, even when they don’t succeed.
Claude additionally supplies in depth performance on immediate caching by their API, which we’re going to be exploring intimately in a future submit.
Thanks for studying! 🙂
. . .
Liked this submit? Let’s be pals! Be a part of me on:
📰Substack 💌 Medium 💼LinkedIn ☕Purchase me a espresso!
All photographs by the creator, besides talked about in any other case.
