Cut LLM latency with SageMaker prefix‑aware routing and model caching

Enable prefix‑aware routing in SageMaker and turn on model caching to keep KV caches warm and avoid cold‑starts.
1. Update the endpoint to use prefix‑aware routing:
aws sagemaker update-endpoint-weights-and-capacities \
--endpoint-name my-llm-endpoint \
--routing-config '{"RoutingStrategy":"PrefixAware"}'
2. Create the model with caching enabled (HyperPod):
aws sagemaker create-model \
--model-name my-llm-model \
--primary-container Image=...,ModelDataUrl=...,Environment={SAGEMAKER_MODEL_CACHE=TRUE}
3. Send two requests that share the same prompt prefix to verify warm KV cache:
curl -X POST https://runtime.sagemaker.amazonaws.com/endpoints/my-llm-endpoint/invocations \
-H "Content-Type: application/json" \
-d '{"inputs":"Explain the benefits of prefix‑aware routing..."}'
4. Monitor latency metrics in CloudWatch (Latency, P50, P99).
Gotcha: Prefixes shorter than ~8 tokens may not trigger routing, so craft prompts with a stable leading phrase.