提示缓存入门指南：原理、优势与最佳实践

提示缓存是优化LLM API调用成本与延迟的关键技术，通过复用重复的提示前缀，避免重复计算。本文深入解析其工作原理、缓存命中与未命中的机制，并给出Anthropic与OpenAI的实践示例，涵盖工具缓存、多轮对话及图像缓存场景。同时提供最佳实践、成本节省估算、常见错误排查及生产环境优化建议，帮助开发者最大化缓存效率，降低API费用，提升应用性能。

技术背景

随着LLM API的广泛应用，成本和延迟成为开发者面临的主要挑战。每次请求都需要处理完整的提示，即使大量内容重复，也需全额计费，导致资源浪费。提示缓存应运而生，它借鉴Web缓存思想，针对LLM输入处理进行优化。通过缓存静态前缀，可显著降低计算开销，减少响应时间，提高吞吐量。适用于智能体、编码助手、聊天机器人等场景，尤其适合提示较长且包含大量静态内容的场景。主要限制包括：缓存命中依赖前缀完全匹配，动态内容需置于末尾；缓存有生命周期，不常用提示会被逐出；不同提供商的缓存策略和计费方式存在差异。

核心原理

提示缓存的核心原理是：当API请求到达时，系统检查提示的前缀是否与缓存中的条目匹配。若匹配（缓存命中），则复用缓存的计算结果，仅处理新增部分，从而降低延迟和成本；若不匹配（缓存未命中），则完整处理提示，并可能缓存新前缀。缓存以组织为范围，同一组织共享缓存，且符合零数据保留条件。OpenAI对超过1024个令牌的提示自动激活缓存，无需修改请求；缓存命中时，usage.prompt_tokens_details中的cached_tokens字段会显示命中令牌数。缓存折扣基于实际处理的令牌数，包括图像令牌。

架构分析

提示缓存的架构设计遵循'静态前缀、动态后缀'原则。系统提示、少样本示例、工具定义等静态内容应置于提示开头，用户输入等动态内容置于末尾。在Anthropic API中，通过cache_control参数显式标记可缓存部分；OpenAI则自动缓存前缀。数据流为：请求→前缀匹配→命中则复用缓存结果，未命中则完整处理并更新缓存。组件包括缓存存储、匹配引擎和计费模块。设计决策上，缓存粒度通常为前缀，而非整个提示，以最大化复用；缓存逐出策略基于使用频率，不常用内容会被移除。

完整实战教程

提示缓存入门指南

提示缓存是一种强大的技术，可以显著降低大型语言模型（LLM）API调用的成本和延迟。通过缓存重复使用的提示前缀，你可以避免为相同内容重复付费，并加快响应速度。本指南将介绍提示缓存的工作原理、优势以及如何有效利用它。

什么是提示缓存？

提示缓存允许你在多个API请求之间重用相同的提示前缀。当你发送一个请求时，如果提示的前缀与之前的请求匹配，提供商可以跳过重新处理该部分，从而节省计算资源。这类似于Web缓存，但针对的是LLM的输入处理。

为什么使用提示缓存？

提示缓存如何工作？

当你的请求到达API时，提供商检查提示的前缀是否与缓存中的条目匹配。如果匹配，则使用缓存的结果，仅处理新添加的部分。如果不匹配，则处理整个提示，并可能缓存新的前缀。

缓存命中与未命中

如何利用提示缓存？

要最大化缓存命中率，请遵循以下策略：

  1. 保持稳定的前缀：将系统提示、少样本示例和工具定义放在提示的开头，并确保它们在不同请求中保持一致。
  2. 将动态内容放在末尾：将用户输入或变量部分放在提示的末尾，这样前缀可以保持不变。
  3. 使用缓存感知的SDK：一些提供商提供SDK，自动管理缓存，例如Anthropic的Python SDK。

示例：使用Anthropic API

以下是一个使用Anthropic API的示例，展示如何利用提示缓存：

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "你是一个有用的助手。",
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "你好，世界！"}
    ]
)

在这个例子中，系统提示被标记为可缓存，因此后续请求如果使用相同的系统提示，将命中缓存。

缓存策略

成本节省估算

缓存成本节省取决于你的提示结构。例如，如果提示的80%是静态的，那么缓存命中时，你只需支付20%的输入处理费用。许多提供商对缓存读取提供折扣，例如Anthropic提供90%的折扣。

最佳实践

结论

提示缓存是优化LLM API使用的重要工具。通过合理设计提示结构，你可以显著降低成本并提升性能。开始使用提示缓存，并观察你的API账单下降吧！

OpenAI 为超过 1024 个令牌的提示提供折扣提示缓存，对于超过 10,000 个令牌的较长提示，延迟最多可降低 80%。通过在 LLM API 请求之间缓存重复信息，您可以大幅降低延迟和成本。提示缓存在组织级别进行范围界定，这意味着只有同一组织的成员才能访问共享缓存。此外，缓存符合零数据保留的条件，因为在此过程中不会存储任何数据。

提示缓存会自动激活，适用于超过 1024 个令牌的提示——您无需在 completions 请求中更改任何内容。当发出 API 请求时，系统首先检查提示的开头部分（前缀）是否已被缓存。如果找到匹配项（缓存命中），则使用缓存的提示，从而降低延迟和成本。如果没有匹配项，系统会从头开始处理完整提示，并缓存前缀以供将来使用。

考虑到这些优势，提示缓存特别有利的一些关键用例包括：

在本食谱中，我们将介绍几个缓存工具和图像的示例。回想一下，通常您应该将静态内容（如说明和示例）放在提示的开头，而将可变内容（如用户特定信息）放在末尾。这也适用于图像和工具，它们在请求之间必须完全相同，包括顺序。所有请求，包括少于 1024 个令牌的请求，都会在 chat completions 对象的 usage.prompt_tokens_details 中显示 cached_tokens 字段，指示有多少提示令牌是缓存命中。对于少于 1024 个令牌的请求，cached_tokens 将为零。缓存折扣基于实际处理的令牌数量，包括用于图像的令牌，这些令牌也计入您的速率限制。

示例1：工具缓存与多轮对话

在此示例中，我们为客服助手定义了工具和交互，使其能够处理检查交货日期、取消订单和更新支付方式等任务。助手处理两条独立消息，首先响应初始查询，然后延迟响应后续查询。

缓存工具时，工具定义及其顺序必须保持一致，才能被包含在提示前缀中。在多轮对话中缓存消息历史时，需将新元素追加到消息数组的末尾。在响应对象及下方输出中，对于第二次补全 run2，您可以看到 cached_tokens 值大于零，表明缓存成功。

from openai import OpenAI
import os
import json
import time


api_key = os.getenv("OPENAI_API_KEY")
client = OpenAI(organization='org-l89177bnhkme4a44292n5r3j', api_key=api_key)

import time
import json

# Define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_delivery_date",
            "description": "Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID.",
                    },
                },
                "required": ["order_id"],
                "additionalProperties": False,
            },
        }
    },
    {
        "type": "function",
        "function": {
            "name": "cancel_order",
            "description": "Cancel an order that has not yet been shipped. Use this when a customer requests order cancellation.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID."
                    },
                    "reason": {
                        "type": "string",
                        "description": "The reason for cancelling the order."
                    }
                },
                "required": ["order_id", "reason"],
                "additionalProperties": False
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "return_item",
            "description": "Process a return for an order. This should be called when a customer wants to return an item and the order has already been delivered.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID."
                    },
                    "item_id": {
                        "type": "string",
                        "description": "The specific item ID the customer wants to return."
                    },
                    "reason": {
                        "type": "string",
                        "description": "The reason for returning the item."
                    }
                },
                "required": ["order_id", "item_id", "reason"],
                "additionalProperties": False
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "update_shipping_address",
            "description": "Update the shipping address for an order that hasn't been shipped yet. Use this if the customer wants to change their delivery address.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID."
                    },
                    "new_address": {
                        "type": "object",
                        "properties": {
                            "street": {
                                "type": "string",
                                "description": "The new street address."
                            },
                            "city": {
                                "type": "string",
                                "description": "The new city."
                            },
                            "state": {
                                "type": "string",
                                "description": "The new state."
                            },
                            "zip": {
                                "type": "string",
                                "description": "The new zip code."
                            },
                            "country": {
                                "type": "string",
                                "description": "The new country."
                            }
                        },
                        "required": ["street", "city", "state", "zip", "country"],
                        "additionalProperties": False
                    }
                },
                "required": ["order_id", "new_address"],
                "additionalProperties": False
            }
        }
    },
    # New tool: Update payment method
    {
        "type": "function",
        "function": {
            "name": "update_payment_method",
            "description": "Update the payment method for an order that hasn't been completed yet. Use this if the customer wants to change their payment details.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID."
                    },
                    "payment_method": {
                        "type": "object",
                        "properties": {
                            "card_number": {
                                "type": "string",
                                "description": "The new credit card number."
                            },
                            "expiry_date": {
                                "type": "string",
                                "description": "The new credit card expiry date in MM/YY format."
                            },
                            "cvv": {
                                "type": "string",
                                "description": "The new credit card CVV code."
                            }
                        },
                        "required": ["card_number", "expiry_date", "cvv"],
                        "additionalProperties": False
                    }
                },
                "required": ["order_id", "payment_method"],
                "additionalProperties": False
            }
        }
    }
]

# Enhanced system message with guardrails
messages = [
    {
        "role": "system", 
        "content": (
            "You are a professional, empathetic, and efficient customer support assistant. Your mission is to provide fast, clear, "
            "and comprehensive assistance to customers while maintaining a warm and approachable tone. "
            "Always express empathy, especially when the user seems frustrated or concerned, and ensure that your language is polite and professional. "
            "Use simple and clear communication to avoid any misunderstanding, and confirm actions with the user before proceeding. "
            "In more complex or time-sensitive cases, assure the user that you're taking swift action and provide regular updates. "
            "Adapt to the user’s tone: remain calm, friendly, and understanding, even in stressful or difficult situations."
            "\n\n"
            "Additionally, there are several important guardrails that you must adhere to while assisting users:"
            "\n\n"
            "1. **Confidentiality and Data Privacy**: Do not share any sensitive information about the company or other users. When handling personal details such as order IDs, addresses, or payment methods, ensure that the information is treated with the highest confidentiality. If a user requests access to their data, only provide the necessary information relevant to their request, ensuring no other user's information is accidentally revealed."
            "\n\n"
            "2. **Secure Payment Handling**: When updating payment details or processing refunds, always ensure that payment data such as credit card numbers, CVVs, and expiration dates are transmitted and stored securely. Never display or log full credit card numbers. Confirm with the user before processing any payment changes or refunds."
            "\n\n"
            "3. **Respect Boundaries**: If a user expresses frustration or dissatisfaction, remain calm and empathetic but avoid overstepping professional boundaries. Do not make personal judgments, and refrain from using language that might escalate the situation. Stick to factual information and clear solutions to resolve the user's concerns."
            "\n\n"
            "4. **Legal Compliance**: Ensure that all actions you take comply with legal and regulatory standards. For example, if the user requests a refund, cancellation, or return, follow the company’s refund policies strictly. If the order cannot be canceled due to being shipped or another restriction, explain the policy clearly but sympathetically."
            "\n\n"
            "5. **Consistency**: Always provide consistent information that aligns with company policies. If unsure about a company policy, communicate clearly with the user, letting them know that you are verifying the information, and avoid providing false promises. If escalating an issue to another team, inform the user and provide a realistic timeline for when they can expect a resolution."
            "\n\n"
            "6. **User Empowerment**: Whenever possible, empower the user to make informed decisions. Provide them with relevant options and explain each clearly, ensuring that they understand the consequences of each choice (e.g., canceling an order may result in loss of loyalty points, etc.). Ensure that your assistance supports their autonomy."
            "\n\n"
            "7. **No Speculative Information**: Do not speculate about outcomes or provide information that you are not certain of. Always stick to verified facts when discussing order statuses, policies, or potential resolutions. If something is unclear, tell the user you will investigate further before making any commitments."
            "\n\n"
            "8. **Respectful and Inclusive Language**: Ensure that your language remains inclusive and respectful, regardless of the user’s tone. Avoid making assumptions based on limited information and be mindful of diverse user needs and backgrounds."
        )
    },
    {
        "role": "user", 
        "content": (
            "Hi, I placed an order three days ago and haven’t received any updates on when it’s going to be delivered. "
            "Could you help me check the delivery date? My order number is #9876543210. I’m a little worried because I need this item urgently."
        )
    }
]

# Enhanced user_query2
user_query2 = {
    "role": "user", 
    "content": (
        "Since my order hasn't actually shipped yet, I would like to cancel it. "
        "The order number is #9876543210, and I need to cancel because I’ve decided to purchase it locally to get it faster. "
        "Can you help me with that? Thank you!"
    )
}

# Function to run completion with the provided message history and tools
def completion_run(messages, tools):
    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        tools=tools,
        messages=messages,
        tool_choice="required"
    )
    usage_data = json.dumps(completion.to_dict(), indent=4)
    return usage_data

# Main function to handle the two runs
def main(messages, tools, user_query2):
    # Run 1: Initial query
    print("Run 1:")
    run1 = completion_run(messages, tools)
    print(run1)

    # Delay for 7 seconds
    time.sleep(7)

    # Append user_query2 to the message history
    messages.append(user_query2)

    # Run 2: With appended query
    print("\nRun 2:")
    run2 = completion_run(messages, tools)
    print(run2)


# Run the main function
main(messages, tools, user_query2)

示例 2：图像

在第二个示例中，我们在消息数组中包含了多个杂货商品图片的 URL，以及一个用户查询，并运行了三次，每次之间有延迟。图片——无论是链接还是以 base64 编码在用户消息中——都符合缓存条件。请确保 detail 参数保持一致，因为它会影响图片的 token 化方式。请注意，GPT-4o-mini 会添加额外的 token 来覆盖图像处理成本，即使它使用低成本的文本 token 模型。缓存折扣基于实际处理的 token 数量，包括用于图像的 token，这些 token 也会计入你的速率限制。

此示例的输出显示，第二次运行时命中了缓存，但第三次运行未命中，因为第一个 URL 不同（eggs_url 而不是 veggie_url），即使查询相同。

sauce_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/9/97/12-04-20-saucen-by-RalfR-15.jpg/800px-12-04-20-saucen-by-RalfR-15.jpg"
veggie_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/3/31/Veggies.jpg/800px-Veggies.jpg"
eggs_url= "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a2/Egg_shelf.jpg/450px-Egg_shelf.jpg"
milk_url= "https://upload.wikimedia.org/wikipedia/commons/thumb/c/cd/Lactaid_brand.jpg/800px-Lactaid_brand.jpg"

def multiimage_completion(url1, url2, user_query):
    completion = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {
        "role": "user",
        "content": [
            {
            "type": "image_url",
            "image_url": {
                "url": url1,
                "detail": "high"
            },
            },
            {
            "type": "image_url",
            "image_url": {
                "url": url2,
                "detail": "high"
            },
            },
            {"type": "text", "text": user_query}
        ],
        }
    ],
    max_tokens=300,
    )
    print(json.dumps(completion.to_dict(), indent=4))
    

def main(sauce_url, veggie_url):
    multiimage_completion(sauce_url, veggie_url, "Please list the types of sauces are shown in these images")
    #delay for 20 seconds
    time.sleep(20)
    multiimage_completion(sauce_url, veggie_url, "Please list the types of vegetables are shown in these images")
    time.sleep(20)
    multiimage_completion(milk_url, sauce_url, "Please list the types of sauces are shown in these images")

if __name__ == "__main__":
    main(sauce_url, veggie_url)

总体建议

为了充分利用提示缓存，请考虑遵循以下最佳实践：

通过实施这些实践，您可以充分利用提示缓存，确保您的应用程序既响应迅速又成本高效。管理良好的缓存策略将显著减少处理时间、降低成本，并有助于维持流畅的用户体验。

常见问题

常见错误：1. 动态内容置于前缀导致缓存未命中，应调整提示结构，将静态内容前置。2. 频繁更改系统提示或工具定义，导致缓存失效，应保持稳定。3. 忽略图像detail参数一致性，影响缓存命中，需确保参数固定。4. 未监控缓存命中率，无法优化策略，应定期检查cached_tokens指标。排查方法：使用API返回的usage字段分析缓存命中情况；对比不同提示结构的命中率；检查缓存逐出日志。生产优化：设计稳定的提示模板，将变量部分后置；利用SDK自动管理缓存；定期评估缓存命中率并调整策略。

企业应用扩展

生产环境优化方案：1. 设计提示模板，将系统指令、工具定义、示例等静态内容固定，动态数据通过变量插入末尾。2. 使用缓存感知的SDK，如Anthropic Python SDK，自动处理缓存标记。3. 监控缓存命中率、延迟和成本，利用API指标调整策略。4. 对于多轮对话，将历史消息作为静态前缀，新消息追加至末尾。5. 图像缓存需保持URL和detail参数一致。6. 考虑缓存预热，对高频请求提前发送相同前缀以建立缓存。7. 结合其他优化技术，如模型蒸馏、请求合并，进一步降低成本。

总结

提示缓存是优化LLM API调用成本与延迟的关键技术，通过复用重复的提示前缀，避免重复计算。本文深入解析其工作原理、缓存命中与未命中的机制，并给出Anthropic与OpenAI的实践示例，涵盖工具缓存、多轮对话及图像缓存场景。同时提供最佳实践、成本节省估算、常见错误排查及生产环境优化建议，帮助开发者最大化缓存效率，降低API费用，提升应用性能。

内容来源与验证

继续系统学习

站内还有更多完整实战教程

加入会员，解锁全部教程、完整代码与持续更新内容。

加入会员
查看全部API 与工程实践教程