Skip to main content

Inference Frameworks Introduction

Text-to-Text Model Inference Framework​

The ChanShen community supports the following four text-to-text inference frameworks: VLLM, SGLang, TGI and llama.cpp. Each framework has its unique features and use cases, suitable for different inference tasks. When selecting a framework, you can decide on the most appropriate tool based on specific scenarios and requirements.

1. VLLM​

vLLM is a fast and easy-to-use LLM inference and service library. Originally developed by UC Berkeley's Sky Computing Lab, vLLM has evolved into a community-driven project that encompasses contributions from both academia and industry.

Features:​

  • High Performance: Delivers leading service throughput, supports paged attention management, fast CUDA/HIP graph execution, quantization support (GPTQ, AWQ, INT4, INT8, FP8), optimized CUDA kernels, and integrates FlashAttention and FlashInfer.
  • Flexibility: Supports popular models from Hugging Face and a variety of decoding algorithms (e.g., parallel sampling, beam search), supports tensor parallelism and pipeline parallelism for distributed inference.
  • Multi-Platform Support: Compatible with NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPUs, and AWS Neuron platforms.
  • Model Support: Seamlessly supports most open-source models from Hugging Face, including Transformer-based LLMs (e.g., Llama), mixture-of-experts LLMs (e.g., Mixtral), and multimodal LLMs (e.g., LLaVA).

Use Cases:​

Suitable for scenarios requiring high throughput, high-performance inference, and multi-platform support; especially strong in large-scale inference tasks where vLLM can provide robust computational support.

2. SGLang​

SGLang is a fast service framework that supports large language models and visual language models. It enables quicker and more controllable interactions with models through a jointly designed backend runtime and frontend language.

Features:​

  • Efficient Backend Runtime: Includes RadixAttention for prefix caching, jump-forward constraint decoding, zero-cost CPU scheduling, batched throughput, tensor parallelism, FlashInfer kernel, etc.
  • Flexible Frontend Language: Provides an intuitive interface for programming LLM applications, including chain generation calls, complex prompts, multimodal inputs, and parallelization.
  • Wide Model Support: Supports generation models like Llama, Gemma, Mistral, QWen, DeepSeek, and LLaVA, as well as embedding models (e.g., e5-mistral) and reward models (e.g., Skywork).
  • Active Community: SGLang is open-source and backed by an active community, widely used across industry.

Use Cases:​

Ideal for scenarios requiring a flexible and efficient inference framework, especially in multimodal applications, customized generation, and situations needing fine control over the generation process.

3. TGI (Text Generation Inference)​

TGI is a toolkit for deploying and serving large language models, designed to provide high-performance text generation services for the most popular open-source LLMs.

Features:​

  • Easy Startup: Supports quick service initialization for popular LLMs like Llama, Falcon, StarCoder, and more.
  • Production Ready: Supports distributed tracing (Open Telemetry) and Prometheus metrics, ensuring stability in production environments.
  • Efficient Inference: Supports tensor parallelism, token streaming (SSE), and batched throughput, optimizing inference performance.
  • Quantization and Customization: Supports various quantization methods (e.g., GPT-Q, AWQ, FP8) and offers features for custom prompt generation and model fine-tuning support.

Use Cases:​

Suitable for production environments requiring efficient and low-latency inference, particularly in large-scale text generation tasks and customized text generation tasks, where TGI provides flexible configuration and efficient execution capabilities.

4. llama.cpp​

llama.cpp is an open-source C++ implementation for running large language models locally, designed to efficiently perform text generation and natural language processing tasks in resource-constrained environments.

Features​

  • High Performance Inference : With optimized C++ code and multi-threading support, the ability to efficiently run LLaMA models on the CPU is achieved.
  • Cross-platform support : Compatible with Windows, Linux, macOS and other operating systems, easy to deploy and use on different platforms.
  • Easy to use : Provides a concise command line interface and rich API interface, easy for developers to quickly integrate and call models.
  • Lightweight : It takes up less resources and is suitable for running in resource-limited environments such as personal computers and embedded devices.

Use Cases:​

Suitable for scenarios where large-scale language models need to be run in a local environment for text generation, natural language processing research, development, and experimentation. Especially in applications with high data privacy and security requirements, llama.cpp provides an efficient and controllable solution. It is also suitable for educational and learning purposes, allowing developers to gain insight and explore how large language models work.