https://ait.bsc.es/handbook/planning-your-deployment/ Recent content in Planning your deployment on AI Tools Hugo en https://ait.bsc.es/handbook/planning-your-deployment/choosing-the-right-model/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/planning-your-deployment/choosing-the-right-model/ <p>The largest model that fits is rarely the right choice. Model selection sets the floor for your serving cost and the ceiling for your output quality at the same time, and the two pull in opposite directions.</p> <p>This page frames the decision as an evaluation problem tied to your own task, not a leaderboard comparison.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Which model properties actually affect serving cost.</li> <li>How to evaluate candidates against your own task rather than a public benchmark.</li> <li>How licensing and provenance constrain deployment options.</li> <li>When a smaller specialised model beats a larger general one.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/planning-your-deployment/choosing-the-right-gpu-number/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/planning-your-deployment/choosing-the-right-gpu-number/ <p>Accelerator count is driven by two independent requirements: fitting the model and its KV cache in memory at all, and serving the expected concurrent load within your latency targets. Either can dominate.</p> <p>This page separates the two questions so they can be sized independently and then reconciled.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>How to separate the memory requirement from the throughput requirement.</li> <li>When to add accelerators to one replica versus adding replicas.</li> <li>How concurrency targets translate into hardware count.</li> <li>What headroom to leave for traffic variation.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/planning-your-deployment/calculating-gpu-memory/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/planning-your-deployment/calculating-gpu-memory/ <p>Running out of accelerator memory in production is one of the more common and more avoidable inference failures. A memory budget computed in advance turns it into a capacity decision rather than an incident.</p> <p>This page sets out the components of that budget and how each one scales.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>The components that make up accelerator memory consumption.</li> <li>How to estimate weight memory from parameter count and precision.</li> <li>How KV cache size scales with batch size and context length.</li> <li>How much headroom to reserve for fragmentation and runtime overhead.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/planning-your-deployment/choosing-the-right-inference-framework/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/planning-your-deployment/choosing-the-right-inference-framework/ <p>Serving frameworks differ less in the features they list than in what they optimise for and how they behave at the edges of their operating range. The right comparison is against your own traffic pattern.</p> <p>This page gives the criteria to compare on, and a method for evaluating candidates without committing to one first.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>The criteria that meaningfully separate serving frameworks.</li> <li>Which optimizations to check for, and why their presence alone is not enough.</li> <li>How to benchmark candidates against your own traffic pattern.</li> <li>What operational characteristics to weigh alongside raw performance.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p>