Monitoring, Logs, and Troubleshooting

Gateway Overview shows traffic, response behavior, and upstream health. Request logs help investigate individual calls. Personal and organization bills show charges. These sources answer different questions.

Review runtime metrics

Open Admin Console → AI Gateway → Gateway Overview, choose a time range and model, then inspect the charts.

Metric	Purpose and boundary
Time to first token (TTFT)	Time until initial output, distinct from total response time
Concurrency	Active request or connection load, indicating how busy the service is
Prompt cache rate	Reuse of input context cached by the upstream
Traffic and latency trends	Identify when responses slow down relative to traffic
Upstream health	Healthy, sub-healthy, and unavailable nodes
Top five upstream channels	Traffic share and success rates for checking distribution
Token Production SLO Metrics	Monitor TTFT, TPOT, Queue Wait, and token throughput against enterprise SLO targets

Token Production Service Level Objectives (Token Production SLO)

To accurately measure and guarantee LLM service quality, AI Gateway introduces a formal Token Production SLO Metric framework, tracking key P50 and P90 percentiles:

• Time To First Token (TTFT, ttft_ms): Measures latency from request ingress to the first emitted token, assessing initial user responsiveness.
• Time Per Output Token (TPOT, tpot_ms): Measures average token generation latency during decoding, assessing continuous text fluency.
• End-to-End Latency (E2E Latency, e2e_latency_ms): Full duration from request start to the last token received.
• Queue Wait & Depth (queue_wait_ms / queue_depth): Monitors request wait time and backlog volume in gateway admission and priority queue pools.
• Token Throughput (tokens_per_second): Tracks total output tokens produced per second across model services.

Full Alignment with OpenAI Error Specifications

AI Gateway standardizes all authorization, rate-limiting, quota, parameter validation, and upstream error responses to strictly adhere to OpenAI API standards:

• 401 Unauthorized: Missing, expired, invalid, or disabled API Keys;
• 403 Forbidden: Target model is outside the key's allowed model scope, or rejected by IP whitelist access policies;
• 404 Not Found: Requested model service is not deployed or inactive;
• 429 Too Many Requests: Triggered TPM / RPM limits, or exhausted spending allowances/quotas;
• 502 Bad Gateway / 503 Service Unavailable: All upstream providers are unhealthy or circuit-broken.

Standard error response format:

{
  "error": {
    "message": "The model 'qwen2.5-72b' is not permitted for the provided API key",
    "type": "permission_denied",
    "param": null,
    "code": "model_not_allowed"
  }
}

Client SDKs (OpenAI Python/Node.js SDK, LangChain, etc.) seamlessly handle errors without specialized adapters.

See AI Gateway Administration for the complete screen guide.

Request and instance logs

Gateway records can contain prompts, tool calls, and outputs for auditing, quality analysis, and authorized downstream data analysis. Access methods, fields, and retention depend on deployment configuration; see Content Safety and Data Handling.

Self-hosted inference container logs are available on instance pages; see Using Endpoints. Use container logs for startup and runtime issues, request records to analyze model replies, and consumption bills to reconcile charges.

Troubleshoot by symptom

Symptom	First checks	Next step
Authentication fails	Gateway key type, expiry, revocation, request headers	Identity and Access
Model missing or access denied	Model name, enabled status, caller identity, endpoint visibility	Connect and Call Models
Allowance or rate limit reached	Key allowance, account balance, and rate-limit configuration separately	Limits, Metering, and Costs
Upstream failure	Endpoint address, credentials, health, and backups	Routing and Reliability
Slow first token	Traffic, concurrency, cache rate, and instance state in the same period	Optimization
Interrupted stream	Connection, client timeout, upstream errors, content checks	Preserve partial output and avoid unbounded retries
Unexpected charge	Key owner, time range, rate, units, retry history	Reconcile consumption against billing

If a response contains a platform-specific code, consult Error Codes. For upstream HTTP or streaming errors, follow the response details and the corresponding service’s API instructions.

Information for support

Provide the time and time zone, model, interface, request mode, error status, and minimal reproduction steps. Preserve a request identifier when one is returned. Remove credentials and handle request bodies according to enterprise policy before sharing logs.