Skip to content
J·BJon BishopFractional growth marketer

Artificial intelligence · llms.txt file breakdown

Modular llms.txt

A directly retrieved Markdown index for Modular's documentation, with 57 links across 2 detected headings.

The actual file

Source Markdown

The saved llms.txt text, displayed as source. The file belongs to its publisher; the live endpoint may change.

Open saved text ↗

Complete saved Markdown shown below.

# LLM Inference Handbook

> A practical handbook for engineers building, optimizing, scaling, and operating LLM inference systems in production.

Markdown versions of all pages are available by appending `.md` to any URL.

- [LLM Inference Handbook](https://handbook.modular.com/index.md): A practical handbook for engineers building, optimizing, scaling, and operating LLM inference systems in production.


## Table of Contents

- [Bring Your Own Cloud (BYOC)](https://handbook.modular.com/getting-started/bring-your-own-cloud.md): Bring Your Own Cloud (BYOC) is a deployment model where vendors run software in your cloud, combining managed orchestration with complete data cont...
- [Calculating GPU memory for serving LLMs](https://handbook.modular.com/getting-started/calculating-gpu-memory-for-llms.md): Learn how to calculate GPU memory for serving LLMs.
- [Choosing the right GPU](https://handbook.modular.com/getting-started/choosing-the-right-gpu.md): Select the right NVIDIA or AMD GPUs (e.g., L4, A100, H100, B200, MI250X, MI300X, MI350X) for LLM inference.
- [Choosing the right inference framework](https://handbook.modular.com/getting-started/choosing-the-right-inference-framework.md): Learn what LLM inference frameworks do, why raw model execution is not enough for production, and how to choose the right inference frameworks for ...
- [Choosing the right model](https://handbook.modular.com/getting-started/choosing-the-right-model.md): Select the right models for your use case.
- [Planning your deployment](https://handbook.modular.com/getting-started.md): Before you can run an LLM in production, you first need to make a few key
- [On-prem LLM deployments](https://handbook.modular.com/getting-started/on-prem-llms.md): On-prem LLMs are large language models deployed within an organization’s own infrastructure, such as private data centers or air-gapped environment...
- [Serverless vs. self-hosted LLM inference](https://handbook.modular.com/getting-started/serverless-vs-self-hosted-llm-inference.md): Understand the differences between serverless LLM APIs and self-hosted LLM deployments.
- [Where is LLM inference run?](https://handbook.modular.com/llm-inference-basics/cpu-vs-gpu-vs-tpu.md): Learn the differences between CPUs, GPUs, and TPUs and where you can deploy them.
- [How does an LLM work?](https://handbook.modular.com/llm-inference-basics/how-does-llm-inference-work.md): Learn how an LLM works, from tokenization and Transformer architecture (attention, attention masks) to the prefill and decode stages of inference.
- [Foundations](https://handbook.modular.com/llm-inference-basics.md): LLM inference is where models meet the real world. It powers everything from
- [Key metrics for LLM inference](https://handbook.modular.com/llm-inference-basics/llm-inference-metrics.md): Measure key metrics like latency and throughput to optimize LLM inference performance.
- [Training vs. inference](https://handbook.modular.com/llm-inference-basics/training-inference-differences.md): LLM training builds the model while LLM inference applies it to generate real-time outputs from new inputs.
- [What is LLM inference?](https://handbook.modular.com/llm-inference-basics/what-is-llm-inference.md): LLM inference is the process of using a trained language model to generate responses or predictions based on prompts.
- [Data, tensor, pipeline, expert and hybrid parallelisms](https://handbook.modular.com/inference-optimization/data-tensor-pipeline-expert-hybrid-parallelism.md): Understand the differences between data, tensor, pipeline, expert and hybrid parallelisms.
- [Inference optimization](https://handbook.modular.com/inference-optimization.md): Running an LLM is just the starting point. Making it fast, efficient, and
- [Inference routing](https://handbook.modular.com/inference-optimization/inference-routing.md): Route LLM requests using cache locality, queue depth, KV cache pressure, and worker state for lower latency and better utilization.
- [KV cache offloading](https://handbook.modular.com/inference-optimization/kv-cache-offloading.md): Learn how KV cache offloading improves LLM inference by reducing GPU memory usage, lowering latency, and cutting compute costs.
- [LLM performance benchmarks](https://handbook.modular.com/inference-optimization/llm-performance-benchmarks.md): LLM performance benchmarks are standardized tests that measure how LLMs perform under specific conditions. They focus on practical metrics such as ...
- [Offline batch inference](https://handbook.modular.com/inference-optimization/offline-batch-inference.md): Run predictions at scale with offline batch inference for efficient, non-real-time processing.
- [PagedAttention](https://handbook.modular.com/inference-optimization/pagedattention.md): Improve LLM memory usage with block-based KV cache storage via PagedAttention.
- [Prefill-decode disaggregation](https://handbook.modular.com/inference-optimization/prefill-decode-disaggregation.md): Disaggregate prefill and decode for better parallel execution, resource allocation, and scaling.
- [Prefix caching](https://handbook.modular.com/inference-optimization/prefix-caching.md): Prefix caching speeds up LLM inference by reusing shared prompt KV cache across requests.
- [Speculative decoding](https://handbook.modular.com/inference-optimization/speculative-decoding.md): Speculative decoding accelerates LLM inference with draft model predictions verified by the target model.
- [Static, dynamic and continuous batching](https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching.md): Optimize LLM inference with static, dynamic, and continuous batching for better GPU utilization.
- [FlashAttention](https://handbook.modular.com/kernel-optimization/flashattention.md): FlashAttention is a fast, memory-efficient attention algorithm for Transformers that accelerates LLM training and inference and helps achieve longe...
- [GPU memory hierarchy](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/gpu-memory.md): Understand GPU registers, shared memory, caches, bank conflicts, HBM, and the trade-offs across the memory hierarchy.
- [GPU architecture](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals.md): Build a practical model of GPU hardware and the execution model used to write and optimize GPU kernels.
- [Streaming multiprocessors](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/streaming-multiprocessors.md): Understand streaming multiprocessor internals, warp scheduling, block residency, and GPU occupancy.
- [Tensor Cores](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/tensor-cores.md): Learn how Tensor Cores execute tiled matrix operations and what precision, shape, and layout constraints kernels must satisfy.
- [GPU threads, warps, blocks, and grids](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/threads-warps-blocks.md): Learn how GPU kernels organize work into threads, warps, thread blocks, and grids.
- [Kernel optimization](https://handbook.modular.com/kernel-optimization.md): Kernel optimization is about making GPU kernels run faster and more efficiently
- [Kernel optimization for LLM inference](https://handbook.modular.com/kernel-optimization/kernel-optimization-for-llm-inference.md): Kernel optimization for LLM inference improves GPU utilization and performance by writing or generating optimized kernels tailored to the compute p...
- [Choosing the right kernel optimization tool](https://handbook.modular.com/kernel-optimization/kernel-optimization-tools.md): Compare the main tools for kernel optimization in LLM inference, from cuBLAS and cuDNN to TVM, XLA, Triton, custom CUDA kernels, Mojo and MAX.
- [Model preparation](https://handbook.modular.com/model-preparation.md): Model preparation is the process of preparing the model for inference.
- [LLM distillation](https://handbook.modular.com/model-preparation/llm-distillation.md): Learn how LLM distillation works, how it compares to quantization, and how to use it to build smaller, faster, and more efficient models for infere...
- [LLM fine-tuning](https://handbook.modular.com/model-preparation/llm-fine-tuning.md): Understand LLM fine-tuning and different fine-tuning frameworks.
- [LLM quantization](https://handbook.modular.com/model-preparation/llm-quantization.md): Understand LLM quantization and different quantization formats and methods.
- [Agent harnesses](https://handbook.modular.com/model-interaction/agent-harnesses.md): An agent harness is the runtime around a model that turns an LLM into an agent. Learn why LLMs need one, which harnesses are popular, and how to ru...
- [Anthropic-compatible API](https://handbook.modular.com/model-interaction/anthropic-compatible-api.md): An Anthropic-compatible API mirrors Anthropic's Messages API so Claude-based clients, SDKs, and agent tools can use another model or provider with ...
- [Function calling](https://handbook.modular.com/model-interaction/function-calling.md): Learn what function calling is and its use case.
- [Model interaction](https://handbook.modular.com/model-interaction.md): Model interaction is the process of interacting with the model to get the
- [LLM inference parameters](https://handbook.modular.com/model-interaction/inference-parameters.md): LLM inference parameters are request-time settings that control randomness, output length, repetition, stopping behavior, reproducibility, and stru...
- [Model Context Protocol](https://handbook.modular.com/model-interaction/model-context-protocol.md): Learn what Model Context Protocol (MCP) is and its use case.
- [OpenAI-compatible API](https://handbook.modular.com/model-interaction/openai-compatible-api.md): An OpenAI-compatible API implements the same request and response formats as OpenAI's official API, allowing developers to switch between different...
- [Prompt engineering](https://handbook.modular.com/model-interaction/prompt-engineering.md): Understand prompt engineering for LLM inference. Learn system & user prompts, zero-shot & few-shot prompting, KV cache impact, token costs, and pro...
- [Structured outputs](https://handbook.modular.com/model-interaction/structured-outputs.md): Structured outputs are model responses in defined formats like JSON or XML, making AI-generated data predictable, machine-readable, and easy to int...
- [Build and maintenance cost](https://handbook.modular.com/infrastructure-and-operations/build-and-maintenance-cost.md): Building LLM infrastructure in-house is costly, complex, and slows AI product development and innovation.
- [LLM observability](https://handbook.modular.com/infrastructure-and-operations/comprehensive-observability.md): LLM observability explained. Learn which metrics, logs, and traces to monitor for LLM inference, how to diagnose issues, and which tools to use.
- [What is distributed inference?](https://handbook.modular.com/infrastructure-and-operations/distributed-inference.md): Distributed inference is the practice of running model inference across multiple GPUs, workers, nodes, or regions to achieve scalable, reliable, an...
- [Fast scaling](https://handbook.modular.com/infrastructure-and-operations/fast-scaling.md): Fast scaling enables AI systems to handle dynamic LLM inference workloads while minimizing latency and cost.
- [Infrastructure and operations](https://handbook.modular.com/infrastructure-and-operations.md): LLMs don't run in isolation. They need robust infrastructure behind them, from
- [InferenceOps and management](https://handbook.modular.com/infrastructure-and-operations/inferenceops-and-management.md): Scale LLM inference confidently with InferenceOps workflows and infrastructure best practices.
- [Multi-cloud and cross-region inference](https://handbook.modular.com/infrastructure-and-operations/multi-cloud-and-cross-region-inference.md): Multi-cloud and cross-region inference is the practice of running LLM workloads across multiple cloud providers or regions to improve latency, avai...
- [Multi-model inference pipelines](https://handbook.modular.com/infrastructure-and-operations/multi-model-inference-pipelines.md): Multi-model inference pipelines chain multiple models into one application path, improving specialization and control, but at the cost of extra lat...
- [What is LLM inference infrastructure?](https://handbook.modular.com/infrastructure-and-operations/what-is-llm-inference-infrastructure.md): Explore the LLM inference infrastructure stack, from GPUs and serving runtimes to scaling, routing, observability, and the choice to build or buy.

Measured from the saved file

Linked document

UTF-8 text size
12.4 KB12688 bytes
Token count
2769o200k_base encoding
Markdown links
5757 distinct URLs
Fenced code blocks
067 lines in the file

Size measures the captured UTF-8 text, not compressed transfer size. Tokens are counted with a named tokenizer; other models can count differently. Neither measure establishes context compatibility or answer quality.

Project H1
LLM Inference Handbook

Opening blockquote

A practical handbook for engineers building, optimizing, scaling, and operating LLM inference systems in production.

Context architecture

How the content is divided

1 H2 sections organize 57 parsed Markdown links. 57 links point to .md or .markdown paths; 0 link occurrences use another hostname. Linked destinations have not been recursively validated.

H2 sectionLinksUTF-8 bytesTokens
Table of ContentsLine 1056122582682

Link counts exclude parsed fenced and inline code and resolve relative destinations against the final endpoint. Section sizes cover the text below each H2 until the next H2.

Embedded material

Code, installation and API examples

Fenced blocks
0 closed
Declared languages
No fenced blocks detected
Package-install commands
0 detected in fenced or inline code
Authentication examples
0 fenced lines match API-key, Authorization, Bearer, OAuth or access-token patterns
HTTP request examples
0 fenced lines match request-method patterns
Authentication links
0 links have authentication-related labels

Embedded example text and links to further documentation are counted separately. A code block does not establish that a sample runs or that an authentication flow succeeds.

Publisher prioritization

The Optional section

No H2 section named “Optional” was detected in this file.

The community proposal uses Optional by convention for secondary links that can be skipped for a shorter context. This label does not establish that everything outside it is critical, or that the material inside it is deprecated.

Machine-readable routes

MCP references and configuration text

MCP-related text was detected in the captured file. 1 links have MCP-related labels or paths. 0 fenced lines match mcpServers, mcp_servers or an mcp add command.

These are text and link observations. No MCP server connection, authentication or tool call was tested.

Index and bundle

llms-full.txt companion

Not found at checked URLs

A verified text response establishes a companion at the checked URL. It does not prove the bundle contains all product documentation, that generation is automated, or that it fits an assistant’s context window.

Change evidence

What can be said about freshness

This capture establishes a baseline. Update frequency cannot be determined from one observation.

File changes can reveal documentation edits or navigation changes. They do not by themselves measure product release velocity, accuracy, agent adoption or AI visibility.

Capture identity and measurement method
Requested endpoint
https://handbook.modular.com/llms.txt
Final endpoint
https://handbook.modular.com/llms.txt
HTTP status
200
Content type
text/plain
Text retrieval
Firecrawl literal text capture
Tokenizer
tiktoken 0.14.0 · o200k_base
SHA-256 of saved UTF-8 text
622a23450724963f8ae1c76a417ef54b201e448a84121b464e5129461c861e40

Counts use a deterministic parser, not a spec-compliance grade. Unusual Markdown and indented code can fall outside its recognized patterns. See the llms.txt proposal and tokenizer implementation.

Related tools and resources

Contact Jon