提示词缓存
提示词缓存(Prompt Caching)主要用于优化大语言模型 API,通过缓存重复的 prompt 前缀来降低延迟和输入 token 成本。
工作原理
-
缓存命中条件:根据前缀哈希路由到特定机器。提供
prompt_cache_key可以优化路由并提高命中率。只有完全匹配的前缀才能命中缓存。 -
路由机制:系统根据 prompt 前缀的哈希值将请求路由到特定服务器。
-
缓存查找:如果匹配,则使用缓存结果;否则处理完整请求,并将前缀缓存以供后续使用。
隐私与计费
-
数据隐私:缓存不在组织之间共享,仅同一组织的成员可访问。
-
不影响输出:缓存仅作用于输入 prompt,不影响生成内容。
-
计入 TPM 限制:缓存请求计入速率限制。
使用示例
import requests
url = "https://api.jalapeno-cloud.ai/v1/chat/completions"
payload = {
"model": "${MODEL_NAME}",
"messages": [
{
"role": "system",
"content": "You are a senior clinical physician with 20 years of internal medicine experience, skilled in symptom analysis and differential diagnosis. Diagnosis process: 1) Chief complaint extraction 2) Symptom correlation analysis 3) Initial differential diagnosis (list 3-5 possibilities) 4) Recommended examination items 5) Medical advice. Important note: Only for reference, not a final diagnosis. Patients must visit a regular hospital."
},
{
"role": "user",
"content": "I've had recurring headaches for the past week, accompanied by nausea and photophobia. What could be the problem?"
}
],
"max_tokens": 512,
"prompt_cache_key": "medical_symptom_analysis_v1"
}
headers = {
"Authorization": "Bearer ${API_KEY}",
"Content-Type": "application/json"
}
response = requests.request("POST", url, json=payload, headers=headers)
print(response.text)
监控指标
通过 API 响应中的 usage.prompt_tokens_details.cached_tokens 字段或监控仪表盘,监控缓存命中率和性能指标。
{
"usage": {
"completion_tokens": 512,
"prompt_tokens": 105,
"prompt_tokens_details": {
"cached_tokens": 80
},
"total_tokens": 617
}
}