Inference engines
Overview An inference engine is software that runs a model to produce output based on input from incoming request. On high level, it provides runtime environment to execute the model. There are multiple implementations and they differ in hardware support, model capabilities, performance priorities, deployment etc. Responsibilities No matter the implementation, all inference engines have several responsibilities: Responsibility Description Load model weights Load the trained parameters into the memory used for inference. Manage request state Maintain intermediate state, including key and value tensors for active requests. Schedule requests Select which requests to process and how to group concurrent work. Compute outputs Execute the model operations that produce tokens. Serve requests Expose an API through which applications submit requests and receive responses. Here weights are the trained model parameters, while request state is created and maintained during inference. The distinction is important from operational perspective: weights impose a relatively fixed memory cost, whereas request state varies with active workloads. ...