Research methods to reduce benchmark matrix size through extrapolation

[anfredette]

Benchmarking all combinations of (model, GPU, traffic profile) may result in an exponentially large matrix. We should investigate whether we can extrapolate missing combinations or build predictive models to reduce the number of explicit benchmarks needed.

Acceptance Criteria

• Research feasibility of extrapolating between traffic profiles (e.g., interpolate TTFT for intermediate prompt lengths)
• Investigate component-level benchmarking to build GPU+LLM performance models
• Prototype an extrapolation or modeling approach
• Validate accuracy against ground-truth benchmarks
• Document approach and limitations

Notes

• This is an exploratory/research task
• Goal: Reduce benchmarking cost while maintaining recommendation accuracy
• May inform Phase 2 architecture

[sriumcp]

How about the following architecture for scenarios where real data may not be available and estimation (e.g., using a simulator or estimator) may be called for?

---

🔁 Data Selection & Simulation Flow

  Given a scenario (model, accelerator, guide-llm settings, etc.)    
└────────────────────────────────────────────────────────
                     │
                     ▼
        ┌────────────────────────────
        │ 1️⃣ Check real-data table  
        │   → If entry exists, use it
        └────────────────────────────
                     │
          (no real data available)
                     ▼
        ┌────────────────────────────
        │ 2️⃣ Check simulation table 
        │   → If entry exists, use it
        └────────────────────────────
                     │
         (no sim data available)
                     ▼
        ┌────────────────────────────
        │ 3️⃣ Run simulator                                                                                
        │   → May take a few extra seconds or minutes    
        │   → Write results back to simulation table       
        └────────────────────────────
                     │
                     ▼
        ┌────────────────────────────
        │  ✅ Use retrieved data for rendering / analysis 
        └────────────────────────────