Problem
Teams using multiple model providers eventually need hard decisions about cost, latency, rollout risk, and fallback behavior. Those decisions belong in a reliable control surface, not scattered through application code.
Ratna
Project Stash Vault
Engineering archive shaped for long-horizon work, not short-term portfolio churn.
Vault 02 / Systems
Back to Systems vault
Case file / 2025
Featured case fileCost / SLO Inference Router
A control plane for routing inference traffic across models while respecting latency budgets, spend constraints, and rollout safety.
Recommended starting point for the systems archive.
GoAWSOpenTelemetryPolicy DesignService Control
#control plane#ai systems#routing#observability
20-second project scan
What to understand first
A fast scan for recruiters and engineers: ownership, technical depth, proof status, and current outcome in one place.
Category
Systems
Status
Operational
Proof ready
01
Stack surface
5 tools
My role
Routing policy design, provider abstraction, observability strategy, and control-plane architecture.
GoAWSOpenTelemetryPolicy DesignService Control
Technical highlights
01
Separated policy evaluation from provider adapters so routing rules can evolve independently of vendor integrations.
02
Made latency and cost first-class inputs in the decision path instead of treating them as dashboard-only metrics.
03
Modeled route selection as a policy problem with explicit inputs and outputs instead of ad hoc branching logic.
04
Used telemetry to keep request outcomes visible across providers, not just inside the primary service.
Impact
Provides a concrete example of production-minded thinking around intelligent infrastructure.
Architecture overview
Problem, solution, and system shape
The project framed as a system: the problem, the solution boundary, and the architecture choices that make the implementation credible.
Solution
Cost / SLO Inference Router frames model serving as an operational systems problem. The project focuses on policy evaluation, provider abstraction, telemetry, and fallback behavior so model access can be treated like infrastructure instead of a thin wrapper over API calls.
Architecture
01
Separated policy evaluation from provider adapters so routing rules can evolve independently of vendor integrations.
02
Made latency and cost first-class inputs in the decision path instead of treating them as dashboard-only metrics.
03
Included fallback and rollback behavior in the architecture from the beginning rather than layering them in after initial success.
Technical highlights
Where the engineering depth shows up
The implementation details a technical reviewer should notice before reading the full case file.
Highlight 01
Separated policy evaluation from provider adapters so routing rules can evolve independently of vendor integrations.
Highlight 02
Made latency and cost first-class inputs in the decision path instead of treating them as dashboard-only metrics.
Highlight 03
Modeled route selection as a policy problem with explicit inputs and outputs instead of ad hoc branching logic.
Highlight 04
Used telemetry to keep request outcomes visible across providers, not just inside the primary service.
Proof surface
Artifacts, references, and public access points
Ready links and planned proof artifacts are shown together so reviewers can distinguish published evidence from reserved case-study slots.
Repository
AvailableGitHub repository
Public source code, implementation structure, and current engineering baseline.
View RepoDiagram
PlannedRouting policy diagram
Reserved for the control-plane view showing policy evaluation, provider selection, and fallback behavior.
Reserved for future publication once the supporting material is ready.
Walkthrough
PlannedTraffic replay walkthrough
Reserved for a recorded pass through route evaluation using realistic latency and cost constraints.
Reserved for future publication once the supporting material is ready.
System view
Architecture and implementation
The architectural boundaries and implementation choices that make the system coherent, maintainable, and operationally meaningful.
Architecture
01
Separated policy evaluation from provider adapters so routing rules can evolve independently of vendor integrations.
02
Made latency and cost first-class inputs in the decision path instead of treating them as dashboard-only metrics.
03
Included fallback and rollback behavior in the architecture from the beginning rather than layering them in after initial success.
Implementation Notes
01
Modeled route selection as a policy problem with explicit inputs and outputs instead of ad hoc branching logic.
02
Used telemetry to keep request outcomes visible across providers, not just inside the primary service.
03
Designed the project as reusable platform infrastructure rather than a single-application integration.
Metrics and outcomes
What is proven so far
Honest status, proof readiness, and results. Qualitative markers are used where exact production metrics are not available yet.
Status
Operational
Current maturity of the project record.
Proof
Repo available
Public source code, implementation structure, and current engineering baseline.
Architecture
3 notes
Documented architecture decisions and boundaries.
Outcomes
01
Provides a concrete example of production-minded thinking around intelligent infrastructure.
02
Strengthens the systems vault with work that sits between backend reliability and applied AI operations.
Future Work
01
Add offline policy simulation against recorded traffic to evaluate routing changes before deployment.
Related entries
More from the systems vault
Vault 02 / 2025
Limit Order Book Engine
Featured file
Operational
A compact matching engine centered on deterministic sequencing, cache-aware data structures, and replayable correctness.
Built
Core engine implementation, replay tooling, benchmark harness design, and code structure for a maintainable performance-sensitive system.
Low-latency software is easy to romanticize and hard to build well. This project matters because it demonstrates discipline around determinism, hot-path simplicity, and observability in a domain where vague correctness is not acceptable.
Strongest proof
Trace planned
C++20BenchmarkingReplay ToolingMarket Data Simulation
#matching engine#low latency#determinism#systems
View ProjectView Repo
Internal case file is live; public repo is not linked yet.
Vault 02 / 2025
Code Search Index v2
Operational
A retrieval stack for large codebases with incremental indexing, responsive query serving, and developer-oriented search ergonomics.
Built
System architecture, indexing and retrieval design, storage strategy, and product framing for a developer-facing search surface.
Search quality in engineering tools is not just about embeddings or ranking models. It is also about update cost, cache design, index freshness, and whether results arrive fast enough to stay inside a real workflow.
Strongest proof
Diagram planned
PythonGoPostgreSQLFaissRedis
#search#retrieval#developer infrastructure#indexing
View ProjectView Repo
Internal case file is live; public repo is not linked yet.