Tagged
Meta's open-weight model line: releases, fine-tunes and local inference. Curated and summarized from dozens of sources by AIBriefs.
A r/LocalLLaMA post claims a llama.cpp fork combining QFN with a specific quantization and hardware config yields unusually high prompt-processing and token-generation speeds. The post gives no reproducible benchmarks or configuration details.
Pull request #31356 by max-krasnyansky targets prompt-processing speedups for the Gemma 4 E2B and E4B variants in ggml-org/llama.cpp, described as massive gains on pp.
FlashProve generates proofs for Llama 4 Maverick (400B-parameter MoE) at ~110 tokens/second on a single H100, with ~19 MB proof size per 1,024-token request in 4-bit weights. That is roughly 70-95x the GPU time of running the model itself on vLLM.
Pull request #29510 by am17an adds flash_attn_ext_rows to fix unified KV cache handling when running multiple sequences on CUDA. A LocalLLaMA poster calls it a "really nice speedup" for running multiple local agents.
Berth hands GPU access to applications through leases instead of letting every app compete for VRAM, per its GitHub repo. It is pitched for live-swapping workloads like llama.cpp, ComfyUI and Strata.
Reddit user benchmarked LFM3-1.7B (Q8_0 GGUF, llama.cpp, ROCm) at 4.1k t/s prefill and ~78 t/s decode on a Strix Halo iGPU, or ~7.1k/~151 t/s tensor-split across a 7900 XTX + XT. They report it stays far more coherent at 32k-64k prompts than expected for its size.
Dense 27B model with a 262,144-token window (16,384 default) that returns option probabilities in one forward pass rather than generating text. Ships bf16 weights, a Q8_0 GGUF, an affine Q4 checkpoint and forks of llama.cpp and Ollama; Nace reports 67.52 on Decision Index 0.4, 6.27 points over Drex v2.0.
Community GGUFs of the DFlash2 drafter for Qwen3.8-Flash-Next now run in llama.cpp v0.8.0, the first release with DFlash2 support. The original authors report roughly 20% faster than MTP at concurrency 1; one tester measured +17% on average.
Run on a MacBook Pro M5 Max with 64 GB, Qwen 3.8 27B Q4 in llama.cpp with thinking off, across 5 small coding tasks each containing a trap. Token use and cache reuse varied more between agents than pass rates did.
Community release pairs jeffpeng3's v2 heretic weights with a new quantization and imatrix recipe from u/jjusko20. The poster says v2 preserves both reasoning ability and output length, calling it a big step up from the v1 weights.
Pull request #31188 reports up to +17.2% prompt-processing throughput on a DGX Spark (GB10) running Qwen3-30B-A3B Q4_K_M, and +9.9% on Qwen3.6-35B-A3B Q4_K_M. A related PR, #29948, fuses FFN gate/up matmuls with interleaved rows for +7.1% on Qwen3.8-27B Q4_K_M.
Strata, Basalt, tandem.cpp, capy-flash and QEngine all target the 125B MoE model, with Basalt hitting up to 275 tok/s generation and 5,500 tok/s prompt processing on a single RTX 5080. tandem.cpp claims 3.3× llama.cpp at 97 tok/s on a 4090, and capy-flash v0.7.1 runs the model in 64 GB of RAM.
A LocalLLaMA user switched from DSV4 Flash 0731 to Qwen3.8-Flash-Next and reports it works significantly better than expected. The main complaint: it does unnecessarily deep work, thinking and fetching information more often than not, making it not very token efficient.
The 1.58-bit ternary MoE quantization runs with 368K context on a 12 GB GPU, hitting 172 tok/s decode and 7000+ tok/s prefill. One GGUF file works in both stock llama.cpp and the PrismML fork.
Pull request #31028 by am17an in ggml-org/llama.cpp merges Qwen tensors during GGUF conversion, prompting LocalLLaMA users to ask whether existing GGUFs need reconverting.
Pull request #31086 by am17an adds DeepSeek-V4.1-Flash to llama.cpp; the model is listed at 763B parameters on Hugging Face. A DeepSeek_V41_Tech_Report.pdf ships in the same repo.
Saluki is a 7.9 GB quantization of Qwen3.8-27B that its makers say keeps tool calling intact at ~2 bits per weight and beats the full 54 GB model on their own 120-task benchmark. The tutorial compares it against ISTA-DASLab's GSQ-RCO quantization in stock llama.cpp, scoring both on seeded BFCL tasks.
Pull request #30965 in ggml-org/llama.cpp rebuilds the QSA selection without the kq_mask, reportedly saving gigabytes of VRAM for Qwen Flash Next. The change is described as part of what Strata does to squeeze more from the experts' cache.
Strata's docs add a Qwen 3.6 35B guide; one user reports it ran slower than llama.cpp on decode speed on a gfx906 (MI50 16GB) build.
A LocalLLaMA poster describes running GGUF Q4 weights paged into WebGL2 compute buffers via GLSL inside HTML, and says tokenizer/vocab handling was "endless debug hell." Their advice: just clone llama.cpp instead.
Free, open-source app from the llama.cpp team lets Mac users find models that fit their hardware, download with one click, and chat offline. Conversations stay on the computer, and apps supporting the OpenAI API can connect.
Pull request #29864 by budiony reworks Intel XMX matrix engines in llama.cpp, which the poster says gave a huge speculative-decoding speedup on a B70 running SYCL. Benchmarks use Muse Glimmer Q6_K (30B) with DFlash2; lower token-verification cost allowed a higher n-max.
A llama.cpp pull request (PR #30971) proposes an approach to beat strata decode performance using an R9700 GPU. Shared on r/LocalLLaMA by user keyboardhack.
Pull request #30640 by am17an adds a fused QSA mask operation to ggml-org/llama.cpp, targeting speedups for sparse attention models.
AbliteratedOutTheBox is a llama.cpp fork that strips refusal behavior from any loaded model at inference time, with no fine-tuning and no extra files. The author says it adds no additional VRAM overhead.
Cohere's Nick Frosst announced llama.cpp support via PR #30062, following the earlier Command A+ launch and North Mini Code beta. He cited local-runner feedback as a driver.
HardenedLinux's GPLv3 SonicBoom runtime has no Python at build or run time — one shared library, a C ABI, and Guile Scheme as its first-class language. The community is now crowdsourcing GPU benchmarks for v0.0.3.
Starting with v0.7.0, llama server's default port moves from 8080 to 9931 (ggml). Users who never set a port explicitly will hit a breaking change on upgrade.
A Reddit user loaded the 560B-parameter MoE model, with 25B active parameters per token, across four DGX Sparks after it wouldn't fit on his two-machine setup. A contributor in Germany provided access to additional hardware.
Adaptive KV streaming plus RDNA 4/ROCm patches to llama.cpp let Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) hold its full 262,144-token window on one 16GB RX 9070 XT at 80+ t/s decode. The IQ4_XS weights take roughly 14.3GB.
A fork of PS5LM runs GGUF models natively on jailbroken PS5s, hitting 20 tok/s on the console's GPU with Qwen 3.8 27B versus roughly one token per minute on its CPU. The native fork accesses the GPU and additional RAM, with reports of over 100 tok/s.
A £299.99 Lenovo IdeaCentre Mini (Snapdragon X1-26-100, 32GB, CPU-only) writes at 17-18 tok/s on Qwen3-Next 80B A3B with MTP, despite the model not fitting in RAM. Qwen3.6 35B A3B coder reads ~118 tok/s and writes 24.5, rising to ~30 with MTP.
Microsoft AI built a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model, per a Microsoft command-line blog post. A llama.cpp pull request references a 137B parameter model with 6.8B active parameters.
Hugging Face's "Local AI 202" livestream walks through speculative decoding techniques including EAGLE, DFlash and MTP using llama.cpp, transformers and the speculators library. Speakers include Merve Noyan (Hugging Face) and Eldar Kurtić (Red Hat).
AnythingLLM Desktop's built-in engine now runs llama.cpp managed by Lemonade, AMD's open-source local AI server, replacing the bundled Ollama it shipped since launch. The team says Ollama upgrades over the past year were as likely to break AnythingLLM as fix it, and that llama.cpp gets new architectures, quantization formats, and GPU kernels first.
Video breaks down how PewDiePie's assistant Odysseus works, including the agent harness built from scratch and the local model Ajax trained to power it. Covers running models locally with llama.cpp and Hugging Face, plus fine-tuning a small model.
AWS published a how-to for serving quantized llama.cpp models on Amazon SageMaker AI, targeting applications that score a fixed set of choices rather than generate free-form text. The managed path supports CPU or GPU inference for routing, agent-action selection, and policy assignment.
llama.cpp PR #27694 adds probabilistic multi-token prediction, now merged. Reported decode speedup of +14% on prose generation, with optimal draft-n-max and draft-p-min settings tracking greedy sampling.
A user reports a CUDA megakernel for Qwen3.8-27B running 1.4-1.9x faster than llama.cpp with MTP on an RTX 3090, hitting 140 tok/s on code generation. The kernel was written with Claude Opus 5.5.
A hobbyist built a conversational plush toy with all inference on a local LAN: Whisper for speech-to-text, Hermes 3 Llama 3.1 8B at Q8_0 for the LLM, and Kokoro for text-to-speech. Per-person memory via LangGraph and dlib face identity let it recognize and recall past conversations.
LlamAmpere's latest release is 3-4% faster with a few hundred MB smaller runtime, targeting 3090-class 12GB Ampere GPUs. It supports Qwen3.8 27B at 200K context with MTP, and YaRN users get a new option.
The Algorithmic Bridge essay argues Meta fell behind on LLMs after missing DeepMind in 2014 and OpenAI in 2015, then rebuilt its research org to return to the top. It describes Meta and Anthropic becoming "briefly and improbably, very good friends" in early 2026, contrasting Zuckerberg with Dario Amodei.
Qwen 3.8-27B's UD-IQ4_XS build from Unsloth is a 13.3GB download running on a 16GB RTX 4070 Ti Super at 16K context, served via llama.cpp at ~33.7 tokens per second. It built a working Snake game in one shot in a Pygame test.
Pull request #29887 by am17an adds a GPU cache for MoE experts stored in host memory, targeting speedups for MoE models that don't fully fit in VRAM.
At its Windows and Surface event, Microsoft demoed Copilot using local files and models to take actions across the OS, with features rolling out over the "next couple months." A new Windows search bar with quick actions arrives this fall on Windows 11.
Pull request #29928 to ggml-org/llama.cpp adds GLM5Next MTP support, letting users run GLM 5 Flash MTP locally. The change was submitted by contributor pwilkin.
Unsloth is updating its Qwen3.8-Flash-Next-GGUF repo on Hugging Face to work with llama.cpp, aiming to resolve the issue of two different GGUF versions of Qwen Flash Next circulating.
Release adds Clef support (text + vision), high-quality support for Qwen3.8-Flash-Next, and a new llama_batch_ext API. Metal performance improved substantially; the project site was also refreshed.