LLM System Optimization

You don't need a 2 trillion parameter model to tell you the capital of France is Paris. Be smart and route between a panel of models according to query difficulty and model specialty! New paper proposes a framework to train a router that routes queries to the appropriate LLM to optimize the trade-off b/w cost vs. performance. Overview: Model inference cost varies significantly: Per one million output tokens: Llama-3-70b ($1) vs. GPT-4-0613 ($60), Haiku ($1.25) vs. Opus ($75) The RouteLLM paper propose a router training framework based on human preference data and augmentation techniques, demonstrating over 2x cost saving on widely used benchmarks. They define the problem as having to choose between two classes of models: (1) strong models - produce high quality responses but at a high cost (GPT-4o, Claude3.5) (2) weak models - relatively lower quality and lower cost (Mixtral8x7B, Llama3-8b) A good router requires a deep understanding of the question’s complexity as well as the strengths and weaknesses of the available LLMs. Explore different routing approaches: - Similarity-weighted (SW) ranking - Matrix factorization - BERT query classifier - Causal LLM query classifier Neat Ideas to Build From: - Users can collect a small amount of in-domain data to improve performance for their specific use cases via dataset augmentation. - Can expand this problem from routing between a strong and weak LLM to a multiclass model routing approach where we have specialist models(language vision model, function calling model etc.) - Larger framework controlled by a router - imagine a system of 15-20 tuned small models and the router as the n+1'th model responsible for picking the LLM that will handle a particular query at inference time. - MoA architectures: Routing to different architectures of a Mixture of Agents would be a cool idea as well. Depending on the query you decide how many proposers there should be, how many layers in the mixture, what the aggregate models should be etc. - Route based caching: If you get redundant queries that are slightly different then route the query+previous answer to a small model to light rewriting instead of regenerating the answer

A candidate interviewing for Staff Engineer at Meta was asked to design an LLM router that sends 80% simple queries to a cheap 8B model and 20% harder ones to a 70B model without users noticing. Another candidate interviewing for L5 at Google was asked a very similar question. One person says: “I’ll classify the query as simple or hard. Simple queries go to the 8B model. Hard queries go to the 70B model.” Another person starts differently. They ask: - What does “simple” actually mean? - How confident is the router in that decision? - Which query categories should never go to the small model? - What happens when the 8B answer is low quality? - How do we measure whether users noticed the downgrade? One loses the offer. One gets hired. Btw, if you’re preparing for Senior to Principal-level system design interviews, I’ve put together 90+ fundamentals like this into a guide. You can check it out here: puneetpatwari.in This is only a high-level approach, but here is how I would break it down: [1] Define “simple” before routing anything A query is simple only when the intent is clear, the answer is factual, the risk is low, and the response does not need deep reasoning, tool use, personalization, or multi-step context. Examples: - password reset - basic FAQ - return policy - feature explanation - simple summarization Harder queries include: - billing disputes - legal or compliance questions - ambiguous user intent - multi-document reasoning - anything involving private account state The first move is not routing. The first move is classification with risk awareness. [2] Use confidence-based routing, not hard rules I would not build a binary router that blindly says “8B” or “70B.” I would create a routing layer that scores: - query complexity - intent clarity - domain risk - context required - user tier or business impact - historical success rate for similar queries If confidence is high, send it to the 8B model. If confidence is low, send it directly to 70B. If confidence is medium, use the 8B model first, but verify the answer before returning it. [3] Add a quality gate before the user sees the answer This is the part most candidates miss. The router should not only decide which model answers. It should also decide whether the answer is safe to show. For medium-risk queries, I would add: - lightweight judge model - factuality check against retrieved context - answer completeness check - fallback to 70B if confidence drops So the user does not feel the cheaper model. They only feel a fast, correct response. [4] Measure the router like a product system The real system will not be perfect on day one. So I would track: - cost per query - fallback rate - 8B success rate by category - user satisfaction - escalation to human support - answer correction rate - latency difference between routes Then I would roll it out gradually. Start with low-risk categories, prove quality, then expand coverage.

TCS asked this question for AI Engineer Role: Your AI system uses multiple models for routing. A cheaper model misclassifies requests and sends them to the wrong workflow. How do you design intelligent model routing with guardrails? Here's how to answer it: 1. Route on a confidence signal, not just cost Have the cheap router return a calibrated confidence; low-confidence requests escalate to a stronger classifier instead of committing to a workflow. 2. Add a verification gate after routing, before execution Cross-check the chosen workflow against the request intent so a misclassification is caught before it triggers the wrong pipeline. 3. Make misroutes recoverable, not terminal Design workflows to detect 'this doesn't belong here' and hand back to the router, instead of failing deep inside the wrong path. 4. Cascade: cheap model first, escalate the ambiguous ones Most traffic is easy and the cheap model handles it fine; only borderline cases pay for the expensive model. 5. Log every routing decision with its confidence Each misroute becomes a labeled example to tune thresholds and build a routing regression set. 6. Set a safe fallback route for low-confidence cases When the router isn't sure, default to a general handler or a human — not a confident wrong guess. The lesson: Model routing isn't a one-shot classification — it's a decision with a confidence budget and a verification gate. The cheap model choosing fast is fine; the cheap model choosing wrong with no recovery path is the actual bug. ======================================== We've covered questions like these in our AI Engineering Interview Master Bundle, a comprehensive set of 22 courses designed for real interview prep. Explore the full guide here → https://lnkd.in/dNhcjdf4

I’m jealous of AI Because with a model you can measure confidence Imagine you could do that as a human? Measure how close or far off you are? here's how to measure for technical and non-technical teams For business teams: Run a ‘known answers’ test. Give the model questions or tasks where you already know the answer. Think of it like a QA test for logic. If it can't pass here, it's not ready to run wild in your stack. Ask for confidence directly. Prompt it: “How sure are you about that answer on a scale of 1-10?” Then: “Why might this be wrong?” You'll surface uncertainty the model won't reveal unless asked. Check consistency. Phrase the same request five different ways. Is it giving stable answers? If not, revisit the product strategy for the llm Force reasoning. Use prompts like “Show step-by-step how you got this result.” This lets you audit the logic, not just the output. Great for strategy, legal, and product decisions. For technical teams: Use the softmax output to get predicted probabilities. Example: Model says “fraud” with 92% probability. Use entropy to spot uncertainty. High entropy = low confidence. (Shannon entropy: −∑p log p) Language models Extract token-level log-likelihoods from the model if you have API or model access. These give you the probability of each word generated. Use sequence likelihood to rank alternate responses. Common in RAG and search-ranking setups. For uncertainty estimates, try: Monte Carlo Dropout: Run the same input multiple times with dropout on. Compare outputs. High variance = low confidence. Ensemble models: Aggregate predictions from several models to smooth confidence. Calibration testing: Use a reliability diagram to check if predicted probabilities match actual outcomes. Use Expected Calibration Error (ECE) as a metric. Good models should show that 80% confident = ~80% correct. How to improve confidence (and make it trustworthy) Label smoothing during training Prevents overconfident predictions and improves generalization. Temperature tuning (post-hoc) Adjusts the softmax sharpness to better align confidence and accuracy. Temperature < 1 → sharper, more confident Temperature > 1 → more cautious, less spiky predictions Fine-tuning on domain-specific data Shrinks uncertainty and reduces hedging in model output. Especially effective for LLMs that need to be assertive in narrow domains (legal, medicine, strategy). Use focal loss for noisy or imbalanced datasets. It down-weights easy examples and forces the model to pay attention to harder cases, which tightens confidence on the edge cases. Reinforcement learning from human feedback (RLHF) Aligns the model's reward with correct and confident reasoning. Bottom line: A confident model isn't just better - it's safer, cheaper, and easier to debug. If you’re building workflows or products that rely on AI, but you’re not measuring model confidence, you’re guessing. #AI #ML #LLM #MachineLearning #AIConfidence #RLHF #ModelCalibration

I didn't ship a chatbot. I shipped a state machine. The prompt didn't fail. The memory routing did. Three weeks into building the vendor onboarding agent, I hit a hard wall. The flow looked clean in my notebook: ingest to extract to push to HubSpot. In production? It hallucinated compliance flags. Skipped signature blocks. Retried without context. I was treating the LLM like control flow. It's not. It's a stochastic function. You need explicit state. So I rebuilt it as a directed graph in LangGraph: 🔹 Ingest Node: PDF to chunking to semantic embedding to Pinecone for episodic memory storage 🔹 Extract Node: ReAct planner with Pydantic schema validation. If confidence drops below 0.82, route to fallback. 🔹 Validate Node: Guardrails AI plus regex plus business rule engine. Catches hallucinated tags before they hit the CRM. 🔹 Router Node: State-based conditional edges. New vendor triggers full flow. Existing vendor triggers delta update. Ambiguous input routes to HITL queue. The breakthrough wasn't better prompts. It was state management. I stopped asking the model to remember. I gave it a shared State object passed between nodes. I stopped hoping for accuracy. I set confidence thresholds, built semantic similarity checks, and routed failures to humans instead of guessing. I stopped tracking tokens. I tracked latency per node, cache hit rates, and fallback frequency. What changed in production: - Task completion: 62% to 89% - Hallucination rate: 0.3%, caught pre-CRM - HITL intervention: 38% to 12% - Cost per qualified record: $4.20 Agents aren't conversational UIs. They're state machines with stochastic components. If you're building GTM agents, stop optimizing for flow. Start optimizing for: ✅ Explicit state transitions, not chain-of-thought ✅ Confidence-based routing, not open-ended retries ✅ Observability traces via LangSmith or Phoenix, not open rates ✅ Human-in-the-loop as a fallback node, not an afterthought What's the hardest state transition you've had to engineer in your agent workflows? Drop your stack below. 👇 #AIGTM #LangGraph #MultiAgent #PMM #SystemDesign #AIEngineering #RAG #HITL #FieldNotes

💡 Why are we sending everything to expensive LLMs? The smarter architecture pattern I’m seeing work in the real world: ➡️ Route 70–90% of tasks to SLMs (Small Language Models) ➡️ Escalate only complex, ambiguous work to LLMs This isn’t just optimization—it’s necessary for scale. 🧠 Hybrid SLM + LLM Architecture (What Works) At the center is an AI Gateway that acts as the decision engine: - Classifies request complexity - Routes to the right model tier - Applies guardrails (PII, compliance) - Tracks cost + performance Execution model: - 🟢 SLMs → fast, cheap, high-volume tasks - 🔵 LLMs → deep reasoning, edge cases - 🔁 Fallback → escalate when confidence is low ⚡ Proven Patterns ✔️ SLM-first strategy (default routing) ✔️ Confidence-based escalation ✔️ Task decomposition (SLM → LLM chain) ✔️ RAG before generation ✔️ Aggressive caching ⚠️ Pitfalls I keep seeing ❌ Sending everything to LLMs → 💸 cost explosion ❌ Over-orchestrating “agentic” workflows → unnecessary complexity ❌ Ignoring latency → poor UX ❌ No cost observability → no control ❌ Same prompts for SLMs and LLMs → bad results 🧭 Simple mental model SLM = Worker LLM = Expert Let the workers handle the bulk. Call the expert only when it truly matters. 📊 What good looks like - 60–90% cost reduction - 2–5x faster response times - Better scalability without overengineering Most teams start with LLM-heavy designs. The winning approach is the opposite: 👉 Start small. Escalate to large. #AI #EnterpriseArchitecture #GenAI #AIArchitecture #CostOptimization #DigitalTransformation

We’ve moved past the stage of simply asking, “Which LLM is best for this prompt?” The real optimization challenge now is selecting the right model paired with the right tool for the job. A new paper from Tsinghua University, "ATLAS", tackles this by treating model selection and tool usage not as separate steps, but as a joint optimization problem. The framework orchestrates heterogeneous agents (like linking a coding-specialized model with a Python interpreter, or a math model with a calculator) based on the complexity of the query. Here is how ATLAS approaches the problem using a "Dual-Path Framework": 🔹 Path 1: Efficiency (Cluster-Based Routing) For familiar tasks, it groups similar queries based on historical data. It quickly routes new prompts to the Model-Tool pair that has statistically performed best for that specific "cluster" of problems. This is fast and cost-effective. 🔹 Path 2: Generalization (RL-Based Routing) For complex or unseen tasks ("Out-of-Distribution"), it switches to a Reinforcement Learning agent. This agent doesn't rely on cached stats; it dynamically explores different reasoning paths to find the best solution, effectively learning how to solve the problem rather than just memorizing who solved it before. The Results: By orchestrating a pool of open-source 7B and 8B models (like Llama-3 and Qwen2.5), ATLAS outperformed significantly larger proprietary models like GPT-4o on complex reasoning benchmarks. Specifically, the RL-based routing showed a +13.1% improvement on unseen tasks compared to standard routing methods. It’s a strong indicator that smaller, specialized agents working in concert can rival massive generalist models. Limitations: The framework currently focuses on text and visual reasoning, leaving out audio/video modalities for now. It also assumes reliable API access to all candidate models, meaning network latency could impact real-time performance in production. #MachineLearning #LLMs #AIResearch #Orchestration #ReinforcementLearning