SGLang

SGLang
SGLang
Developer	LMSYS
Release	January 17, 2024; 2 years ago
Written in	Python, Rust, CUDA, C++
Type	Large language model inference engine
License	Apache License 2.0
Website	sglang.io
Repository	github.com/sgl-project/sglang

SGLang (short for Structured Generation Language) is an open-source framework for programming and serving large language models and multimodal models. It was introduced by researchers affiliated with LMSYS^[1] and other institutions as a system combining a Python-embedded language for structured generation with a runtime for high-throughput inference.^[2]^[3]^[4]

The project is designed for low latency and high-throughput inference workloads, and its documentation describes support for features such as structured outputs, speculative decoding, continuous batching, quantization, and compatibility with OpenAI-style APIs.^[5]

History

SGLang was publicly introduced in January 2024 by researchers affiliated with Stanford, UC Berkeley, Texas A&M, and Shanghai Jiao Tong University.^[2] Its academic description later appeared in the proceedings of NeurIPS 2024.^[3] In January 2026, TechCrunch reported that contributors associated with the project had formed the startup RadixArk to commercialize services around SGLang while continuing its open-source development.^[6]^[7]

Architecture

According to the NeurIPS paper, SGLang consists of two main components: a front-end language embedded in Python and a back-end runtime for executing language model programs efficiently.^[3] The front end provides primitives for generation, selection, and parallel control flow, while the runtime uses a set of optimizations intended to reduce repeated computation and improve throughput.^[3]

Among the techniques described by the project are RadixAttention for reusing key–value cache state across multiple generation calls, compressed finite-state machines for faster constrained decoding, and speculative execution for API-based models.^[3] The current documentation also describes support for serving both language models and multimodal models across a range of hardware back ends.^[5]

References

^ "LMSYS". GitHub. GitHub, Inc. Retrieved April 22, 2026.
^ ^a ^b "Fast and Expressive LLM Inference with RadixAttention and SGLang". LMSYS Org. January 17, 2024. Retrieved April 19, 2026.
^ ^a ^b ^c ^d ^e Zheng, Lianmin; Yin, Liangsheng; Xie, Zhiqiang; Sun, Chuyue; Huang, Jeff; Yu, Cody Hao; Cao, Shiyi; Kozyrakis, Christos; Stoica, Ion; Gonzalez, Joseph E.; Barrett, Clark; Sheng, Ying (2024). SGLang: Efficient Execution of Structured Language Model Programs (PDF). Advances in Neural Information Processing Systems 37. Retrieved April 19, 2026.
^ "SGLang". UC Berkeley Sky Computing Lab. April 25, 2024. Retrieved April 22, 2026.
^ ^a ^b "SGLang Documentation". SGLang. Retrieved April 19, 2026.
^ Hu, Krystal (January 21, 2026). "Sources: Project SGLang spins out as RadixArk with $400M valuation as inference market explodes". TechCrunch. Retrieved April 19, 2026.
^ R, Vignesh (January 23, 2026). "From Berkeley lab to $400M startup: SGLang becomes RadixArk". TFN. Retrieved April 22, 2026.

External links

[1] "LMSYS". GitHub. GitHub, Inc. Retrieved April 22, 2026.

[lmsys-launch-2] "Fast and Expressive LLM Inference with RadixAttention and SGLang". LMSYS Org. January 17, 2024. Retrieved April 19, 2026.

[neurips-3] Zheng, Lianmin; Yin, Liangsheng; Xie, Zhiqiang; Sun, Chuyue; Huang, Jeff; Yu, Cody Hao; Cao, Shiyi; Kozyrakis, Christos; Stoica, Ion; Gonzalez, Joseph E.; Barrett, Clark; Sheng, Ying (2024). SGLang: Efficient Execution of Structured Language Model Programs (PDF). Advances in Neural Information Processing Systems 37. Retrieved April 19, 2026.

[4] "SGLang". UC Berkeley Sky Computing Lab. April 25, 2024. Retrieved April 22, 2026.

[docs-5] "SGLang Documentation". SGLang. Retrieved April 19, 2026.

[techcrunch-6] Hu, Krystal (January 21, 2026). "Sources: Project SGLang spins out as RadixArk with $400M valuation as inference market explodes". TechCrunch. Retrieved April 19, 2026.

[7] R, Vignesh (January 23, 2026). "From Berkeley lab to $400M startup: SGLang becomes RadixArk". TFN. Retrieved April 22, 2026.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

v t e Large language models (LLMs)
List of LLMs AI Companies Benchmarks List of chatbots Foundation model Generative AI
Concepts	Language model NLP NLG Computational linguistics Foundation model Small language model Reasoning model Generative pre-trained transformer Transformer Attention KV cache Context window Tokenization Word embedding Parameter Hyperparameter Autoregression Mixture of experts (MoE) Inference Model compression Knowledge distillation Speculative decoding PagedAttention Neural scaling law Multimodality Open weights
Training, prompting, and alignment	Self-supervised learning Supervised learning Fine-tuning Instruction tuning RLHF Constitutional AI AI alignment AI safety Mechanistic interpretability Prompt engineering In-context learning Chain-of-thought prompting RAG Prompt injection Adversarial machine learning Hallucination Stochastic parrot Glitch token Generative engine optimization
Models	Apertus BERT BLOOM Chinchilla Claude DBRX Gemini Gemma GloVe GLM GPT GPT-J PanGu Granite Jais LaMDA Llama MiMo Minerva Mistral and Mixtral Nemotron PaLM Phi Qwen Seq2seq T5 Vicuna Word2vec XLNet
Chatbots and assistants	Amazon Q ChatGPT Character.ai Claude DeepSeek Doubao Ernie Bot GLM Google Gemini Grok Kimi Microsoft Copilot Meta AI Perplexity AI Sparrow You.com
Agents, coding, and applications	AI agent Intelligent agent AutoGPT CrewAI LangChain Manus Model Context Protocol Agent2Agent OpenAI Codex Vibe coding Code generation Question answering Machine translation Text summarization Chatbot Virtual assistant LLMs in higher education
Software	PyTorch TensorFlow Hugging Face Inference engine llama.cpp Ollama SGLang TensorRT-LLM vLLM ONNX OpenVINO Vector database ChromaDB Deep learning software Open-source AI software
Hardware and infrastructure	AI data center AI accelerator GPU CUDA TPU High-bandwidth memory
Benchmarks, evaluation, and detection	Language model benchmark MMLU Humanity's Last Exam LMArena LLM-as-a-Judge Perplexity metric GPTZero Artificial intelligence content detection Undetectable.ai
Datasets and data	Data set Text corpus Common Crawl The Pile Web scraping Synthetic data Training, validation, and test data sets
Organizations	AI21 Labs Alibaba Anthropic Baidu Cohere DeepSeek EleutherAI Google DeepMind Hugging Face Meta AI Microsoft AI Mistral AI MiniMax Moonshot AI Nvidia OpenAI Sarvam AI StepFun Technology Innovation Institute Thinking Machines Lab xAI
People	Sam Altman Dario Amodei Yoshua Bengio Aidan Gomez Demis Hassabis Geoffrey Hinton Andrej Karpathy Yann LeCun Percy Liang Liang Wenfeng Christopher D. Manning Arthur Mensch Mira Murati Alec Radford Noam Shazeer Ilya Sutskever Ashish Vaswani Andrew Ng
Social, economic, and governance	AI boom AI bubble AI slop AI anthropomorphism AI arms race Chatbot psychosis Competition Copyright Deaths linked to chatbots Dependency (GAID) Environmental impact Regulation Ethics Existential risk In education In healthcare Workplace impact
Category:Large language models

SGLang

History

Architecture

See also

References

External links

Content Disclaimer