#

consumer-gpu

Here are 35 public repositories matching this topic...

Sandermage / sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

Updated Jul 27, 2026
Python

Alberto-Codes / turboquant-vllm

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

compression transformer triton quantization inference-optimization kv-cache llm vllm consumer-gpu turboquant

Updated Apr 10, 2026
Python

giannisanni / neutronstar

Giant MoE models on a single consumer GPU by streaming experts from SSD. CUDA fork of antirez/ds4: runs GLM-5.2 (743B), Tencent Hy3 (295B), and DeepSeek 4 Flash, with io_uring expert streaming, LFU host cache, cross-layer expert prefetch, and the first MTP speculative decoding for GLM-5.2.

cuda glm mixture-of-experts llm-inference gguf speculative-decoding deepseek consumer-gpu ssd-streaming

Updated Jul 12, 2026
C

MELLM

Rahul-14507 / MELLM

Lightweight Modular AI Routing Engine for Local LLMs — Run specialised experts efficiently on consumer GPUs using smart Mixture-of-Experts routing.

python mixture-of-experts llama-cpp local-llm gguf consumer-gpu ai-router

Updated Apr 6, 2026
Python

sipsalabs / ultracompress

Verified AI infrastructure for regulated deployment. UltraCompress (our wedge): near-lossless 5-bit compression with SHA-256-reproducible reconstruction - prove the model in production is the one you validated. 23 architectures (0.6B-405B), Hermes-3-405B @ 1.0066x. OpenAI-compatible API. pip install ultracompress

python compression cuda inference pytorch transformer lossless quantization mlops deep-tech openai-api llm patent-pending ai-infrastructure 405b consumer-gpu 5-bit sipsa-labs experimental-tech

Updated Jul 27, 2026
Python

kimari-local-ai

smouj / kimari-local-ai

🦁 Local AI for Consumer GPUs — Run powerful LLMs on GTX 1060/1080. No cloud. No subscriptions. Built on llama.cpp + CUDA.

python cli nextjs cuda local-first llama-cpp llm-inference gguf open-webui openai-compatible-api consumer-gpu openclaw gtx-1060

Updated Jul 27, 2026
Python

anna-claudette / angruvadal

RAM-Backed MCP Memory Architecture for Consumer LLM Inference — 900K token context on 16GB VRAM

amd mcp rocm llm llama-cpp local-llm context-window rdna4 consumer-gpu rotorquant

Updated Mar 27, 2026
Python

bojobh609 / TurboQuant

Optimize FAISS-compatible vector quantization for fast, accurate vector search with TurboQuant

python machine-learning deep-learning metal pytorch nearest-neighbor quant attention embedding mlx iclr rag inference-optimization kv-cache kv-cache-compression consumer-gpu turboquant

Updated Jul 27, 2026
Python

arbitrary-number / arbitrary-number

Arbitrary Numbers

python machine-learning deep-learning tensorflow gpu cuda pytorch nvidia model-serving numerical-computing model-optimization edge-inference ai-inference ai-performance consumer-gpu

Updated Aug 12, 2025
Python

obisin / dgls

Dynamic GPU Layer Swapping: Train large models on consumer GPUs with intelligent memory management

training pytorch gpu-memory memory-optimization consumer-gpu layer-swapping

Updated Sep 12, 2025
Python

Novamind-CS / Novamind-CS

Surgical reasoning on consumer silicon. Hybrid SSM + causal memory architecture with entropy-gated System 1/2 dispatch, O(1) inference memory, and continual learning — designed for 16 GB VRAM.

machine-learning research deep-learning pytorch lora language-model mamba causal-inference reasoning low-memory fine-tuning state-space-model continual-learning neuro-symbolic llm consumer-gpu

Updated Mar 22, 2026
Python

esonhjz / Llama-server-Extreme-VRAM-Saver

Run MoE models on consumer GPUs by offloading Attention to GPU and shunting experts to CPU — 75% less VRAM, 9× faster than brute force.

cuda moe llama-cpp gpu-inference qwen low-vram consumer-gpu llm-benchmark vram-optimization

Updated Jul 27, 2026
PowerShell

JigSawPT / moe-autopilot

Adaptive hot/cold MoE expert cache for llama.cpp: keep the hottest experts in VRAM to speed up decode when offloading to RAM on a consumer GPU. Measured +26% (Qwen3-Coder-Next 80B) and +31.6% (gpt-oss-120B) on one RTX 5090.

cuda moe quantization mixture-of-experts llama-cpp gpu-inference local-llm llm-inference consumer-gpu expert-offloading

Updated Jul 23, 2026
Python

ikaganacar1 / ismail

ismail is a from-scratch Turkish language model implementation designed for low-end hardware, built and trained on a single RTX 5070 (12GB).

moe mla pytorch-implementation turkish-nlp llm llms llm-training deepseek-v3 turkish-llm consumer-gpu low-resource-llm

Updated Nov 19, 2025
Python

Alex-888888 / memory-layers-consumer-hardware

Independent reconstruction of Memory Layers at Scale (Berges et al., 2024) on a single 24GB consumer GPU — Qwen2.5-7B, frozen backbone, with honest lessons learned.

machine-learning pytorch reproducibility rocm llm qwen consumer-gpu memory-layers

Updated Jul 23, 2026
Python

AIdevsmartdata / ramp-quant

RAMP: RL-guided Adaptive Mixed-Precision quantization for GGUF models. Data-free sensitivity analysis, evolutionary search, per-tensor type optimization. Produces hardware-optimized GGUF for consumer GPUs.

moe quantization sensitivity-analysis ramp mixed-precision llm llama-cpp qwen gguf qwen3 consumer-gpu imatrix ik-llama

Updated Apr 16, 2026
Python

jyimu / AeloRu

AeloRu (Adaptive Elastic Learning with Orthogonal Robust Units) enables real-time, continuous learning on resource-constrained devices. Drop-in memory module for LLMs.

lora hebbian-learning dora continuous-learning peft hebbian llm consumer-gpu relora

Updated Jul 21, 2026
HTML

yalun753 / moe-l2

Run large MoE models on consumer GPUs — save up to 91% VRAM with expert caching

moe llama-cpp llm-inference deepseek consumer-gpu vram-optimization expert-caching

Updated Jul 27, 2026
Python

Zorac

chris-colinsky / Zorac

Self-hosted LLM chat client with streaming UI for vLLM servers. Run Mistral-24B locally on RTX 4090/3090. Privacy-focused ChatGPT alternative for homelab/gaming PCs. Python/Rich terminal UI.

python cli ai self-hosted chat-client homelab mistral nvidia-gpu awq llm vllm chatgpt-alternative local-llm llm-inference offline-ai consumer-gpu

Updated Feb 27, 2026
Python

magicmayonaise / flux-local-inference

Reproducible local inference for FLUX.1 [schnell] on 8 GB Turing GPUs (RTX 2070). Sequential CPU offload, fp16 compute, mock-tested pipeline init - portfolio piece for a neuroscience to ML transition.

flux neuroscience pytorch text-to-image diffusion-models huggingface-diffusers local-inference flux-schnell consumer-gpu

Updated May 18, 2026
Python

Improve this page

Add a description, image, and links to the consumer-gpu topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the consumer-gpu topic, visit your repo's landing page and select "manage topics."