Prompt caching for faster model inference

Prompt caching is an optional feature that you can use with supported models on Amazon Bedrock to reduce inference response latency and input token costs. Amazon Bedrock supports two types of prompt caching: Implicit Prompt Caching and Explicit Prompt Caching. Support for each type varies by model and API.

Prompt caching can help when you have workloads with long and repeated contexts that are frequently reused for multiple queries. For example, if you have a chatbot where users can upload documents and ask questions about them, it can be time consuming for the model to process the document every time the user provides input. With prompt caching, you can cache the document so that future queries containing the document don't need to reprocess it.

Types of prompt caching

The two types differ in how reusable prompt content is selected:

Implicit Prompt Caching

Implicit Prompt Caching automatically attempts to reuse eligible prompt prefixes without requiring cache controls in your request. Keep static content at the beginning of your prompt and dynamic content at the end to increase the likelihood of an exact prefix match.

Implicit Prompt Caching is best effort. Repeating an identical prompt doesn't guarantee a cache hit, and cache-hit rates can vary.

Explicit Prompt Caching

Explicit Prompt Caching lets you identify reusable prompt prefixes using model-specific cache controls or cache checkpoints. Cache checkpoints are markers that define the contiguous subsection of your prompt that you want to cache. Prompt prefixes should remain static between requests. Changes to a prompt prefix in subsequent requests result in cache misses.

Cache checkpoints have a minimum and maximum number of tokens, depending on the model. You can only create a cache checkpoint if your total prompt prefix meets the minimum number of tokens. Minimums currently range from 512 to 4,096 tokens per cache checkpoint. For the minimum that applies to your model, see Supported models, Regions, and explicit caching limits or the model's model card. The minimum applies cumulatively to the entire prompt prefix before each checkpoint, including, where applicable, content in the tools, system, and messages fields. There is no minimum number of tokens required between cache checkpoints. For a model with a 1,024-token minimum, you can define additional checkpoints fewer than 1,024 tokens apart as long as the total prompt prefix before each checkpoint contains at least 1,024 tokens. If you add a cache checkpoint before the total prompt prefix meets the minimum number of tokens, your inference still succeeds, but your prefix isn't cached.

The cache has a Time To Live (TTL), which resets with each successful cache hit. During this period, the context in the cache is preserved. If no cache hits occur within the TTL window, your cache expires. Many models support a 5-minute TTL. Check the model card for your model to see the exact TTL conditions.

Explicit Prompt Caching provides control over which prompt content is eligible for caching. It doesn't guarantee that an eligible request results in a cache hit.

Billing for cached tokens

For both Implicit Prompt Caching and Explicit Prompt Caching, tokens successfully read from cache are reported as cached tokens and billed at the model's cache-read rate. Tokens that aren't read from cache are billed at the standard input token rate. Depending on the model, tokens written to cache can be billed at a rate that is higher than the standard input token rate. For more information, see the Amazon Bedrock pricing page.

You can use prompt caching when you run inference in Amazon Bedrock with supported models. Availability of each prompt caching type varies by model and API. Prompt caching is available through the following Amazon Bedrock features:

For models that support Explicit Prompt Caching, the APIs provide granular control over the prompt cache. You can set individual cache checkpoints within your prompts and add checkpoints up to the maximum allowed for the model. For more information, see Supported models, Regions, and explicit caching limits.

Supported models, Regions, and explicit caching limits

Prompt caching support varies by model and API. Model cards identify whether a model supports Implicit Prompt Caching, Explicit Prompt Caching, or both. Prompt caching is available in all AWS Regions where the supported models are available. To check model availability by Region, see Regional availability by models.

The following table lists models that support Explicit Prompt Caching, along with their token minimums, maximum number of cache checkpoints, and fields that allow cache checkpoints.

To see which prompt caching types a model supports, refer to Models at a glance, and then choose the model that you're interested in.

* Amazon Nova models support a maximum of 20K tokens for prompt caching. Prompt caching is primarily for text prompts.