IBM Developer

Article

Fast inference and furious scaling with vLLM and KServe

Discover how to deploy and scale high-performance LLM inference workloads using vLLM, KServe, Kubernetes, and distributed serving architectures to improve throughput, latency, and production AI scalability

By Rafael Vasquez

On this page

The title "Fast inference and furious scaling" captures the pace of generative AI technology. With new optimization techniques and tools emerging daily, generative AI is a rapidly evolving landscape that often makes it difficult for beginners to get started.

In this article, beginners who are new to large language model (LLM) inferencing and serving can learn about why it's a complicated thing to do and gain a clearer idea of how to get started using two open source tools: vLLM and KServe. Instead of focusing on technical details, this article covers the 'why' and 'how' of LLM inferencing and serving to give background context for those who want to get started, while including resources for further reading linked along the way.

What does it mean to "serve" an LLM?

Model serving means making a pre-trained model usable. When you try a cloud-based service like ChatGPT, models have been made available for you to send prompts (that is, inference requests) and receive a response (that is, output). Those models are being served for you to consume. The model has been made available through an API.

This sounds simple, but it is not.

Serving LLMs isn't just about wrapping them in an API. Just like any memory-intensive software program, large models bring performance trade-offs, infrastructure constraints, and cost considerations.

Here are just a few reasons that model serving is not simple:

  • Massive LLMs require massive resources
  • LLMs are complex
  • Scaling is hard

Massive LLMs require massive resources

Hosting the 70-billion parameter version of Meta's Llama-3 can require ~150GB of vRAM. To put that in perspective, OpenAI's GPT-4 is estimated to have over 1.5 trillion parameters. Hosting these kinds of models requires high-end GPUs (for example, A100s or H100s), well beyond what a standard consumer GPU or cloud instance provides. Running them efficiently and reliably requires specialized resources and infrastructure.

Those top-tier GPUs aren’t just expensive to buy, they’re expensive to operate. And in 2025, they're scarce too.

LLMs are complex

Aside from their size, LLMs face technical hurdles under the hood.

  • Tokenization: Each prompt they receive has to be processed or tokenized. (Learn more about tokenization in this article, "Token optimization: The backbone of effective prompt engineering.")
  • Autoregressive generation: LLMs generate one token at a time. After predicting a token, that token becomes part of the input to predict the next one. In many cases, compute requirements scale quadratically with the length of a sequence: for instance, if the number of input tokens doubles, the model needs 4 times as much processing power to handle it.
  • Context window: An LLM's working memory. Every time the model predicts the next token in a sequence, it computes the relationships between that token and every token before it. Longer prompts and conversations result in slower inference.

Scaling is hard

The more users and requests you have, the harder it gets to maintain fast, consistent responses. Serving LLMs is one thing, but scaling LLMs brings its own challenges. Here are a few:

  • When demand spikes, you need to make more resources available quickly. But dynamically scaling up GPU-bound workloads without dropping requests or disrupting service is complex.
  • You can’t just send incoming requests to any available server. If someone is "chatting" with a model, there could be session context that gets lost. That makes load balancing more complicated than in stateless systems.
  • Spinning up a new model isn’t instant. Large models can take minutes to load into GPU memory, and that might affect user experience.

Enter fast inference: vLLM

With an appreciation for the hurdles LLM inferencing brings, it's easier to see why so many new tools and techniques are emerging to address those challenges.

Currently, one of the most impactful is vLLM, an open-source LLM inference engine that improves memory efficiency and throughput, the rate at which a model can process input requests or generate output over time. (Learn more about LLM inference performance metrics like throughput in this blog).

What makes vLLM fast?

There are several innovations under the hood that help vLLM achieve high performance:

  • PagedAttention: vLLM uses this technique which rethinks how memory is allocated for the attention mechanism, drastically reducing memory fragmentation and overhead. Instead of storing every token's key/value cache in contiguous blocks, PagedAttention lets the model store these in virtual memory “pages,” allowing more requests to run concurrently without running out of GPU memory.

  • Continuous Batching: vLLM can dynamically batch incoming requests together, even if they arrive at different times. That means vLLM can serve many users at once without waiting for batch windows to fill up, which dramatically improves throughput without increasing latency.

  • Optimized Compute Unified Device Architecture (CUDA) Kernels: vLLM includes optimized CUDA kernels to maximize performance on specific GPUs and other hardware. This improves efficiency and reduces costs.

So, with vLLM:

  • You can serve more concurrent requests using fewer GPUs.
  • You get significantly better throughput compared to other engines.
  • You reduce memory waste, letting you deploy bigger models on the same hardware.

Enter furious scaling: KServe

KServe is a powerful open-source platform built on Kubernetes for deploying, scaling, and managing models in production.

While vLLM optimizes how fast your model does its thing, KServe ensures it's accessible and running, under unpredictable workloads, across multiple model versions, and across the cloud. KServe abstracts the complexity of model serving infrastructure while benefiting from all of Kubernetes’ resilience and scaling features.

How does KServe help?

Some of its key capabilities include:

  • Autoscaling: Automatically scales model instances up or down based on demand, even to zero when idle, which helps save on costs.
  • Multi-model management: Deploy, update, and rollback different model versions with minimal downtime.
  • Advanced routing: Automatically route traffic to new model versions for canary testing or A/B deployments.
  • Observability: Integrates with tools like Grafana for monitoring performance, latency, and error rates.

So, with KServe:

  • You can run and manage multiple models at once, within the same system.
  • You can scale those models up or down automatically, even to zero if no one’s using them.
  • You get the benefits of Kubernetes-native features like scalability and fault-tolerance, so your services restart automatically if they crash and scale reliably across clusters.

Putting it together

So far, we’ve looked at some of the key challenges in LLM serving — from GPU limitations to scaling pains. And while there are several ways to scale LLM inference (for example, llm-d or vLLM Production Stack), this article focuses on how vLLM and KServe fit in the landscape as a practical, open-source solution that’s Kubernetes-native, flexible, and approachable for beginners.

  • vLLM makes inference fast and efficient, using techniques like PagedAttention and continuous batching to squeeze the most out of GPUs.

  • KServe handles deployment, autoscaling, routing, and observability — making models production-ready.

Hands-on with KServe and vLLM (on a CPU)

You don't have to start with GPUs or complex stacks to play around with LLM serving. This walkthrough shows how you can get started and learning by running a small model on a local Kubernetes cluster using KServe and vLLM on your own laptop.

This example will use the following:

Creating your local cluster with KServe

Start by following the first page of the KServe Quickstart. It'll help you easily install tools like kind, kubectl, and helm, and then run the installation script which will automatically set up KServe's control plane, webhooks, ingress, and out-of-the-box serving runtimes to deploy models. A ServingRuntime defines how models of a certain format are served: which container image to use, what arguments or environment variables are needed, and what model formats are supported.

KServe provides several built-in runtimes like:

  • tritonserver: TensorFlow, ONNX, PyTorch, TensorRT
  • sklearnserver: SKLearn
  • huggingfaceserver: HuggingFace transformers

This demo will focus on the HuggingFace runtime, which for text generation uses vLLM by default.

Deploying a model

With a cluster set up, you can deploy a model using a KServe InferenceService. This custom resource definition is the main interface that KServe uses for managing models and represents the model's logical endpoint for serving inferences. Below, we'll deploy the facebook/opt-125m model. We pass runtime arguments pointing to the model hosted on Hugging Face. We also specify the key-value (KV) cache space through VLLM_CPU_KVCACHE_SPACE, from vLLM's CPU runtime variables. Setting it larger allows vLLM to run more requests in parallel, which can affect CPU performance. Depending on your constraints, you may need to adjust the resource requests and limits.

kubectl apply -f - <<EOF
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: opt-125
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=opt-125
        - --model_id=facebook/opt-125m
      image: kserve/huggingfaceserver:v0.15.0
      env:
        - name: VLLM_CPU_KVCACHE_SPACE
          value: "1"
      resources:
        requests:
          cpu: "4"
          memory: "4Gi"
        limits:
          cpu: "4"
          memory: "4Gi"
EOF

If you watch the logs of your ServingRuntime Pod, you'll see the vLLM server spinning up behind the scenes:

kubectl get pods -l serving.kserve.io/inferenceservice=opt-125
kubectl logs <POD_NAME> -c kserve-container --follow

It should look something like:

INFO 06-18 22:53:05 [__init__.py:256] Automatically detected platform cpu.
2025-06-18 22:53:12.550 1 kserve INFO [model_server.py:register_model():395] Registering model: opt-125
INFO 06-18 22:53:37 [config.py:583] This model supports multiple tasks: {'generate', 'reward', 'score', 'embed', 'classify'}. Defaulting to 'generate'.
WARNING 06-18 22:53:37 [_logger.py:72] device type=cpu is not supported by the V1 Engine. Falling back to V0.
WARNING 06-18 22:53:37 [_logger.py:72] uni is not supported on CPU, fallback to mp distributed executor backend.
INFO 06-18 22:53:37 [llm_engine.py:241] Initializing a V0 LLM engine (v0.8.1) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=facebook/opt-125m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False,
...

From there, check out the status of your InferenceService.

kubectl get isvc

Output:

NAME   URL                              READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION     AGE
opt-125    http://opt-125.default.example.com   True            100                     opt-125-predictor-00001 3m30s

You can use kubectl describe on the InferenceService to get descriptive status information too. If you checked it before it was ready, you will see useful information for debugging such as a Waiting for runtime Pod to become available message.

Making an inference request

There are a few ways to make an inference request, but a simple approach is to port forwarding the service:

INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')
kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80

And in a separate terminal, send a request.

export INGRESS_HOST=localhost
export INGRESS_PORT=8080
export SERVICE_HOSTNAME=$(kubectl get inferenceservice opt-125 -o jsonpath='{.status.url}' | cut -d "/" -f 3)

curl -v "http://${INGRESS_HOST}:${INGRESS_PORT}/openai/v1/completions" \
-H "content-type: application/json" -H "Host: ${SERVICE_HOSTNAME}" \
-d '{
  "model": "opt-125",
  "prompt": "One thing about Denver is ",
  "stream": false,
  "max_tokens": 30,
  "stop": "."
}'

After a few seconds, you should receive a response.

{
  "id": "cmpl-a36be1a1-91ef-4b44-b9c6-f7aa5d8b84bc",
  "object": "text_completion",
  "created": 1750288691,
  "model": "opt-125",
  "choices": [
    {
      "index": 0,
      "text": " you are free to have different opinions",
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": ".",
      "prompt_logprobs": null
    }
  ],
  "usage": {
    "prompt_tokens": 7,
    "total_tokens": 15,
    "completion_tokens": 8,
    "prompt_tokens_details": null
  }
}

Your response will likely be different. Remember that with smaller models, response quality can vary and often depends on how clearly and specifically you write your prompt.

Wrapping up

This hands-on example is specifically designed to be accessible, allowing you to experiment with LLM serving concepts on your local machine without the need for expensive GPUs or complex cloud setups.

By standing up a simple KServe cluster with a small model running on CPU, you’ve peeled back the curtain on how modern LLM inference systems actually work. You’ve seen how:

  • vLLM serves text generation requests behind an OpenAI-compatible API
  • KServe abstracts away Kubernetes complexity with declarative model deployment
  • Concepts like InferenceService and ServingRuntime connect infrastructure to model logic

This local setup gives you the space to:

  • Dive deeper into LLM serving without needing expensive GPUs or cloud clusters
  • Observe the behavior of real serving runtimes like vLLM in action
  • Learn how scalable LLM inference works under the hood

Conclusion

LLM inference and serving can be difficult to understand — especially for beginners. As this article aimed to show, you can start small and still build a working mental model of how the pieces fit together.

By deploying a lightweight model using vLLM and KServe on a CPU, and by understanding why those tools exist in the first place, you've seen what it takes to get from a raw model to a live, queryable endpoint. Along the way, you explored core concepts like inference optimization and scaling.

You now have a sandbox to learn from. Break things, try other models, tweak performance settings, and gradually build toward understanding what scalable and production-ready really mean.

Further reading and next steps

Explore vLLM:

Learn more about KServe: