跳到主要内容

提示词缓存

提示词缓存(Prompt Caching)主要用于优化大语言模型 API,通过缓存重复的 prompt 前缀来降低延迟和输入 token 成本。

工作原理​

  1. 缓存命中条件:根据前缀哈希路由到特定机器。提供 prompt_cache_key 可以优化路由并提高命中率。只有完全匹配的前缀才能命中缓存。

  2. 路由机制:系统根据 prompt 前缀的哈希值将请求路由到特定服务器。

  3. 缓存查找:如果匹配,则使用缓存结果;否则处理完整请求,并将前缀缓存以供后续使用。

隐私与计费​

  1. 数据隐私:缓存不在组织之间共享,仅同一组织的成员可访问。

  2. 不影响输出:缓存仅作用于输入 prompt,不影响生成内容。

  3. 计入 TPM 限制:缓存请求计入速率限制。

使用示例​

import requests

url = "https://api.jalapeno-cloud.ai/v1/chat/completions"

payload = {
"model": "${MODEL_NAME}",
"messages": [
{
"role": "system",
"content": "You are a senior clinical physician with 20 years of internal medicine experience, skilled in symptom analysis and differential diagnosis. Diagnosis process: 1) Chief complaint extraction 2) Symptom correlation analysis 3) Initial differential diagnosis (list 3-5 possibilities) 4) Recommended examination items 5) Medical advice. Important note: Only for reference, not a final diagnosis. Patients must visit a regular hospital."
},
{
"role": "user",
"content": "I've had recurring headaches for the past week, accompanied by nausea and photophobia. What could be the problem?"
}
],
"max_tokens": 512,
"prompt_cache_key": "medical_symptom_analysis_v1"
}

headers = {
"Authorization": "Bearer ${API_KEY}",
"Content-Type": "application/json"
}

response = requests.request("POST", url, json=payload, headers=headers)
print(response.text)

监控指标​

通过 API 响应中的 usage.prompt_tokens_details.cached_tokens 字段或监控仪表盘,监控缓存命中率和性能指标。

{
"usage": {
"completion_tokens": 512,
"prompt_tokens": 105,
"prompt_tokens_details": {
"cached_tokens": 80
},
"total_tokens": 617
}
}