Prompt Caching

Prompt Caching is primarily used to optimize large language model APIs by caching repeated prompt prefixes to reduce latency and input token costs.

How it Works

1. Cache Hit Condition: Routes to a specific machine based on prefix hash. Providing prompt_cache_key can optimize routing and improve hit rate. Only prefixes with exact matches can hit the cache.

2. Routing Mechanism: The system routes requests to specific servers based on the hash value of the prompt prefix.

3. Cache Lookup: If matched, cached results are used; otherwise, the full request is processed and the prefix is cached for future use.

Privacy & Billing

1. Data Privacy: Cache is not shared across organizations, only accessible by members of the same organization.

2. No Output Impact: Cache only applies to input prompts, does not affect generated content.

3. Counted in TPM Limit: Cached requests are counted toward rate limits.

Usage Example

import requests

url = "https://api.jalapeno-cloud.ai/v1/chat/completions"

payload = {
    "model": "${MODEL_NAME}",
    "messages": [
        {
            "role": "system",
            "content": "You are a senior clinical physician with 20 years of internal medicine experience, skilled in symptom analysis and differential diagnosis. Diagnosis process: 1) Chief complaint extraction 2) Symptom correlation analysis 3) Initial differential diagnosis (list 3-5 possibilities) 4) Recommended examination items 5) Medical advice. Important note: Only for reference, not a final diagnosis. Patients must visit a regular hospital."
        },
        {
            "role": "user",
            "content": "I've had recurring headaches for the past week, accompanied by nausea and photophobia. What could be the problem?"
        }
    ],
    "max_tokens": 512,
    "prompt_cache_key": "medical_symptom_analysis_v1"
}

headers = {
    "Authorization": "Bearer ${API_KEY}",
    "Content-Type": "application/json"
}

response = requests.request("POST", url, json=payload, headers=headers)
print(response.text)

Monitoring Metrics

Monitor cache hit rate and performance metrics through the usage.prompt_tokens_details.cached_tokens field in API responses or through the monitoring dashboard.

{
   "usage": {
       "completion_tokens": 512,
       "prompt_tokens": 105,
       "prompt_tokens_details": {
           "cached_tokens": 80
       },
       "total_tokens": 617
   }
}