A long time ago, on a server far, far away 👋

Overview An inference engine is software that runs a model to produce output based on input from incoming request. On high level, it provides runtime environment to execute the model. There are multiple implementations and they differ in hardware support, model capabilities, performance priorities, deployment etc. Responsibilities No matter the implementation, all inference engines have several responsibilities: Responsibility Description Load model weights Load the trained parameters into the memory used for inference. Manage request state Maintain intermediate state, including key and value tensors for active requests. Schedule requests Select which requests to process and how to group concurrent work. Compute outputs Execute the model operations that produce tokens. Serve requests Expose an API through which applications submit requests and receive responses. Here weights are the trained model parameters, while request state is created and maintained during inference. The distinction is important from operational perspective: weights impose a relatively fixed memory cost, whereas request state varies with active workloads. ...

LLM compression is the set of techniques used to make large language models cheaper, faster, smaller, and easier to deploy while preserving as much quality as possible. It can compress the model itself, the runtime state used during inference, or the input context sent to the model. What is LLM compression? LLM compression means reducing one or more of the resources required to use a model: Resource Why it matters Disk size Model distribution, storage cost, startup time RAM/VRAM Whether the model fits and how many requests can run concurrently Memory bandwidth Often the dominant bottleneck for token generation Compute Matrix multiply cost, attention cost, and server throughput Compression can target different parts of the system: ...

GitHub Copilot CLI can export OpenTelemetry data that helps inspect model calls, tool invocations, MCP activity, token usage and latency. This article relates to OpenTelemetry for Codex CLI and focuses on the Copilot-specific setup and telemetry behavior. Use the Compose example from the Codex article to bring up local OTel stack. The important Copilot-specific differences are: Copilot CLI exports OTLP over HTTP to http://127.0.0.1:4318. Codex CLI exports OTLP over gRPC to http://127.0.0.1:4317. Traces Copilot CLI telemetry is most useful when inspected as traces. From local captures, the most useful span categories are: ...

Codex CLI can export traces, metrics, OTel log records with event names and attributes. With that telemetry, you can track API requests, tool invocations, token usage, MCP calls and run latency. This article walks through a local OpenTelemetry stack and Codex CLI configuration needed to send telemetry to it. The setup should help you explore the traces, logs and metrics, build dashboards and answer questions such as: Which models were used? How many tool calls were made? Was MCP used? How long did the run take? Traces While Codex CLI can export traces, its trace spans are not fully documented, and there is no official list of span names. ...

During 2025 the AI race has shifted from creating best models to creating best agents. By end of year, most developers argue that Claud Code has taken lead. Standard Claude Code usage typically requires an expensive Anthropic subscription. But beyond the cost savings, using Ollama with Claude Code offers significant advantages regarding data privacy, confidentiality, and service autonomy. By redirecting the Claude Code interface to a local instance, you effectively bypass several of Anthropic’s cloud-based data collection and usage policies. These include model training opt-out, technical information collection, usage tracking, regional constraints, intellectual property ownership and more. ...