ai-observer

🥈Silver

AI Observer provides unified local observability for AI coding assistants. It tracks token usage and other metrics for tools like Claude Code, Gemini CLI, and OpenAI Codex CLI. Self-hosted and OpenTelemetry-compatible, it integrates with existing observability stacks.

196130Updated 2w ago

Intermediate30min to implementautomation

Saves ~150 min per use

Quick InstallView Source

git clone https://github.com/tobilg/ai-observer.git

Works with:

Claude

Overview

About This Skill

How to Use

1. **Install and Configure**: Deploy AI Observer in your self-hosted environment with OpenTelemetry exporters. Configure it to monitor your specific AI coding tools (Claude Code, Gemini CLI, etc.) by adding the appropriate instrumentations. 2. **Set Up Data Collection**: Ensure your AI tools are configured to send telemetry to AI Observer. For example, in Claude Code, set `ANTHROPIC_API_KEY` and enable `CLAUDE_CODE_TELEMETRY=true`. 3. **Define Metrics and Alerts**: Create dashboards in your observability stack (Grafana, Kibana, etc.) to visualize token usage, response times, and error rates. Set up alerts for budget thresholds and performance anomalies. 4. **Analyze Reports**: Use the generated reports to identify high-consumption projects and inefficiencies. Focus on projects exceeding 80% of their token budget or showing high error rates. 5. **Optimize and Iterate**: Implement the suggested optimizations (e.g., context pruning, off-peak scheduling) and monitor their impact. Adjust configurations based on new patterns observed in subsequent reports.

Setup & Installation

Quick Install

No install command available. Check the GitHub repository for manual installation instructions.

Alternative Install (Git Clone)

git clone https://github.com/tobilg/ai-observer

Requirements

Claude Code or compatible AI agent
Works with: Claude

Quick Start Guide

Install the Skill

Copy the install command above and run it in your terminal.

Open Your AI Agent

Launch Claude Code, Cursor, or your preferred AI coding agent.

Try It Out

Use the prompt template or examples below to test the skill.

Customize

Adapt the skill to your specific use case and workflow.

Usage Examples

Prompt Template

Generate a report for [AI_TOOL_NAME] usage over the last [TIME_PERIOD] days. Include total token consumption, top 5 projects by token usage, average response time per project, and any anomalies in error rates or latency. Compare this against the [BUDGET_LIMIT] token budget for the team. Suggest 3 optimizations to reduce costs while maintaining productivity.

Example Output

### AI Tool Usage Report for Team Alpha (Period: May 1-15, 2024)

**Summary Metrics**
- Total tokens consumed: 1,247,890 (Budget: 1,500,000 - 17% remaining)
- Average response time: 2.3s (Target: <3s)
- Error rate: 0.8% (Target: <1%)

**Top 5 Projects by Token Usage**
1. **Project Phoenix** (456,780 tokens) - AI-assisted refactoring of legacy monolith
2. **API Gateway Rewrite** (312,450 tokens) - OpenAPI spec generation and validation
3. **Data Pipeline Optimization** (201,340 tokens) - SQL query generation and review
4. **Frontend Migration** (145,670 tokens) - React component generation
5. **Security Audit** (131,650 tokens) - Vulnerability detection and remediation

**Anomaly Detection**
- **Project Phoenix** showed a 40% spike in token usage on May 10 (67,890 tokens vs. daily avg of 32,450) - identified as bulk refactoring session for new feature branch.
- **API Gateway Rewrite** had 3 consecutive errors on May 12 (error rate 2.1%) - resolved by updating OpenAPI spec validation rules.

**Cost Optimization Recommendations**
1. Implement token budget alerts at 80% consumption (currently triggers at 95%) to allow proactive adjustments.
2. For Project Phoenix, enable context pruning for large refactoring sessions to reduce redundant token consumption.
3. Schedule API Gateway Rewrite sessions during off-peak hours to avoid concurrent tool usage spikes.

**Next Steps**
- Review Project Phoenix's refactoring approach with senior engineers to identify opportunities for more efficient AI-assisted patterns.
- Update OpenAPI validation rules based on the error patterns observed in the API Gateway Rewrite project.
- Set up automated weekly reports to track progress against these optimizations.