#

inference-optimization

Here are 522 public repositories matching this topic...

syv-ai / HyperQwen

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

cuda quantization inference-optimization kv-cache vllm local-llm llm-inference qwen speculative-decoding qwen3 consumer-gpu rtx-3090

Updated Nov 10, 2026
Python

google / XNNPACK

High-efficiency floating-point neural network inference operators for mobile, server, and Web

cpu neural-network inference multithreading simd matrix-multiplication neural-networks convolutional-neural-networks convolutional-neural-network inference-optimization mobile-inference

Updated Nov 11, 2026
C

mryab / efficient-dl-systems

Efficient Deep Learning Systems course materials

machine-learning deep-learning cuda pytorch performance-optimization distributed-training ml-infrastructure mlops inference-optimization efficient-deep-learning ml-systems

Updated May 28, 2026
Jupyter Notebook

alibaba / BladeDISC

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

machine-learning deep-learning neural-network compiler tensorflow pytorch inference-optimization mlir

Updated Dec 30, 2024
C++

jiazhihao / TASO

The Tensor Algebra SuperOptimizer for Deep Learning

deep-neural-networks deep-learning inference-optimization

Updated Jan 26, 2023
C++

brontoguana / krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

transformer inference-engine inference-optimization mixture-of-experts cpu-inference large-language-models gpu-inference llm-inference high-performance-inference hybrid-inference gguf-model-support llama-cpp-alternative

Updated Nov 8, 2026
C++

modular / llm-inference-handbook

Everything you need to know about LLM inference

inference-optimization llm llm-inference inference-handbook inference-infrastructure

Updated Nov 10, 2026
TypeScript

llm-inference-engineering

amitshekhariitbhu / llm-inference-engineering

Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

inference-optimization large-language-models llm llms llm-inference llm-engineering inference-engineering

Updated Nov 9, 2026
Markdown

mit-han-lab / inter-operator-scheduler

[MLSys 2021] IOS: Inter-Operator Scheduler for CNN Acceleration

acceleration cnn parallelism inference-optimization

Updated Apr 27, 2022
C++

imedslab / pytorch_bn_fusion

Batch normalization fusion for PyTorch. This is an archived repository, which is not maintained.

deep-neural-networks deep-learning pytorch batch-normalization inference-optimization

Updated Apr 6, 2020
Python

nari-labs / nari-qwen3-tts

Ultrafast Qwen3-TTS: sub-50 ms time-to-first-audio at 10 requests per second.

text-to-speech streaming ai inference streaming-audio inference-optimization audio-generation speech-generation qwen3-tts

Updated Aug 19, 2026
Python

Scottcjn / ram-coffers

LLM infrastructure cost reduction via NUMA-aware weight banking: 147 t/s (8.8x stock llama.cpp) on refurbished enterprise POWER8. Self-hosted inference, no cloud APIs. Part of the Proof of Physical AI stack.

Updated Nov 7, 2026
Python

truespar / paddock

Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

macos rust ai metal cuda inference nvidia mlx inference-engine model-serving inference-optimization blackwell aimodels

Updated Nov 5, 2026
Rust

ZFTurbo / Keras-inference-time-optimizer

Optimize layers structure of Keras model to reduce computation time

keras inference-optimization

Updated Jul 18, 2020
Python

EfficientContext / ContextPilot

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

ai-agents context-api inference-optimization prompt-engineering context-engineering openclaw hermes-agent

Updated Oct 29, 2026
Python

RightNow-AI / auto

the agi compiler: records llm agent behavior, proves what repeats, and compiles it into verified, sandboxed wasm binaries that run for microdollars. nothing figured out twice, paper: https://arxiv.org/abs/2607.04542

agent compiler webassembly program-synthesis distillation conformal-prediction inference-optimization llm ai-infrastructure

Updated Jul 7, 2026
Rust

pleasedodisturb / awesome-llm-token-optimization

A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.

Updated Nov 8, 2026
Shell

htobty / ReadyLLM

Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.

react self-hosted quantization model-deployment inference-optimization fastapi llm llm-serving llama-cpp comfyui vllm local-llm llm-inference gguf speculative-decoding sglang

Updated Nov 6, 2026
Python

vbdi / divprune

[CVPR 2025] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models

pruning inference-optimization multi-modality llm token-pruning vision-language-model llava multimodal-large-language-models

Updated Apr 16, 2026
Python

RabbitLLM

ManuelSLemos / RabbitLLM

Run 70B+ LLMs on a single 4GB GPU — no quantization required.

inference inference-optimization llm qwen airllm

Updated Feb 28, 2026
Python

Add this topic to your repo

To associate your repository with the inference-optimization topic, visit your repo's landing page and select "manage topics."