A collection of tools to measure inference latency for foundations models in Amazon Bedrock and OpenAI. Reports time to first token and total time.
- bedrock-latency-benchmark.ipynb - Measure LLM single request latency across scenarios like:
- Different number of in/out tokens
- Compare latency across models from Amazon Bedrock and OpenAI.
- Latency for the same model across AWS Regions.
- stress.test.bedrock.py - A utility to stress test Claude 3 models on Bedrock with high number of concurrent requests (e.g., launch 1000 requests/min).
- clock-based-rolling-window-rpm-tpm-tester - Test whether Bedrock TPM limits are based on absolute minutes (clock-based) or relative minutes (rolling window). Determines if quota resets at xx:00 or 60 seconds after first usage.
git clone https://github.com/gilinachum/bedrock-latency- Open relevant notebook.