llmio

LLMIO is a Go-based gateway that consolidates OpenAI, Anthropic, and Gemini models into a single load-balanced service. It provides a unified REST API compatible with OpenAI Chat Completions, Anthropic Messages, and Gemini Native formats, supporting both streaming and non-streaming requests. The gateway implements weight-based scheduling (random or priority strategies) to distribute traffic across providers based on capabilities like tool calling, structured output, and multimodal support. Built-in features include rate limiting, failure handling, session tracking via TraceID, and comprehensive observability with per-request latency breakdown, token usage logging, and cost calculation. A React-based admin UI allows management of providers, models, associations, and request logs with multi-dimensional search filters.

1. **Install LLMIO:** Download the latest binary from [LLMIO GitHub Releases](https://github.com/llmio/llmio/releases) or install via `go install github.com/llmio/llmio@latest`. Run `llmio --init` to generate a default config file. 2. **Configure Providers:** Edit `config.yaml` to add your API keys for OpenAI, Anthropic, and Google. Set weight-based scheduling (e.g., `weights: { "gpt-4o": 5, "claude-3-5-sonnet": 3, "gemini-1.5-pro": 2 }`). 3. **Enable Logging:** Specify a log file path (e.g., `log_file: /var/log/llmio/llmio.log`) and log level (`log_level: info`). 4. **Start the Gateway:** Run `llmio --config config.yaml` and verify the API is accessible at `http://localhost:8080/v1/chat/completions`. 5. **Test Failover:** Use `curl` to send a test request: `curl -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'`. Simulate a provider outage by blocking their API endpoint and observe automatic rerouting. **Tips:** - Use the management interface (`http://localhost:8080/admin`) to monitor real-time metrics and adjust weights dynamically. - For production, deploy LLMIO behind a reverse proxy (e.g., Nginx) with HTTPS and rate limiting. - Enable caching for system prompts to reduce costs (e.g., `cache_enabled: true`).