benchmark-optimization-loop

--- name: benchmark-optimization-loop description: Use when the user asks to make something faster, try many variants, run recursive optimization, benchmark latency/throughput/cost, or choose the best implementation by repeated measured tests. license: MIT metadata: origin: ECC tools: Read, Write, Edit, Bash, Grep, Glob --- # Benchmark Optimization Loop Use this skill to convert "make it 20x faster" or "try 50 recursive optimizations" into a bounded measured loop that can actually improve a system. ## Required Baseline Do not optimize until these exist: - the operation being optimized; - the correctness gate that must stay green; - the metric: wall time, p95 latency, rows/sec, cost/run, memory, error rate; - the current baseline; - the search budget: max variants, max time, max spend, max data impact. If the user asks for an unrealistic target, keep the ambition but make the loop bounded and measurable. ## Loop 1. Measure the baseline. 2. Identify bottlenecks from evidence. 3. Generate variants that test one hypothesis each. 4. Run variants with the same input shape. 5. Reject variants that fail correctness, safety, or reproducibility. 6. Promote the fastest safe variant. 7. Codify the winning path in a script, command, test, config, or doc. 8. Rerun the baseline and winner to confirm the delta. ## Variant Table Track variants like this: ```text Variant | Hypothesis | Command | Time | Correct? | Notes baseline | current path | npm run job | 120s | yes | stable batch-500 | fewer round trips | npm run job -- --batch 500 | 42s | yes | winner parallel-8 | more workers | npm run job -- --workers 8 | 31s | no | rate limited ``` ## Recursive Search For recursive or hyperparameter work: - persist every run to a ledger; - compare against the prior accepted winner, not only the previous run; - keep a holdout or replay check; - stop when improvement is within noise, correctness fails, cost exceeds the budget, or the search starts changing more variables than it can explain. Use phrases like "best measured safe variant" instead of "global optimum" unless the search space was actually exhaustive. ## Promotion Gate A variant cannot become the new default until: - correctness tests pass; - the performance delta is repeated or explained; - rollback is obvious; - the change is encoded in source control or a durable runbook; - the final summary includes exact commands and measurements.

["Define the baseline: Start with your current implementation and document its performance metrics (e.g., latency, throughput, cost). Use tools like `locust`, `Apache Bench`, or custom scripts to measure.","Generate variants: Brainstorm 5-7 alternative approaches (e.g., algorithm changes, hardware optimizations, or architectural shifts). Include at least one \"wildcard\" idea (e.g., GPU offload or precomputation).","Benchmark systematically: Use the same benchmark tool and workload for all variants. Record metrics (mean, median, P95, P99) and note any outliers or anomalies.","Compare and select: Use statistical methods (e.g., paired t-tests) to compare variants. Focus on the top 3 performers for further optimization.","Recursively optimize: Apply the same loop to the best variant (e.g., hyperparameter tuning, parallelization, or caching). Iterate until gains plateau or meet your target.","Tip: Automate the loop with scripts (e.g., Python + Jupyter) to reduce manual effort. Use tools like `Optuna` for hyperparameter optimization or `Ray Tune` for distributed benchmarking."]