Tagged
Meta's open-weight model line: releases, fine-tunes and local inference. Curated and summarized from dozens of sources by AIBriefs.
OpenVINO 2026.4 adds CPU support for Gemma-3n and CPU/GPU support for Kokoro-82M and Qwen3-VL-4B with eagle3. The release emphasizes more Gen AI coverage and framework integrations to minimize code changes.
A LocalLLaMA thread asks whether uncensored coding models perform better because they skip safety checks on every prompt. The poster asks if the claim is "hogwash" or true, and whether anyone actually uses them for coding.
Pull request #28901 in ggml-org/llama.cpp, authored by am17an, adds hc ops for qwen4exp. A LocalLLaMA commenter says it's "time to re-benchmark Qwen Flash Next again."
Over a month after Meta promised to release Muse Spark weights, the model has moved from Spark 1.2 to 1.3 with nothing published. r/LocalLLaMA users question whether 1.2 weights will arrive only once 1.4 drops.
A Reddit user released a llama.cpp fork tuned for Ampere GPUs, claiming 90+ tokens per second at temperature 1 for agentic and coding workloads up to 100K context. Some optimizations also carry over to Blackwell and Lovelace cards.
Pull request #27000 by AlexGabbia adds CPU support for the Maple 20B-A1B ternary mixture-of-experts architecture to ggml-org/llama.cpp. A preview of the model is hosted at huggingface.co/deepgrove/maple-preview.
A r/LocalLLaMA poster says a 9.4B-parameter dense model is ready to train, using a 1/2/3 Engram table, Moonshot's AttnRes modeling, and 3:1 RoPE/NoPE layering. It uses the Llama 3 tokenizer and LM head.
A r/LocalLLaMA thread asks users to share llama.cpp settings and system setups for Qwen3.8 Flash Next, with the poster noting the model is quite big and that testing many option combinations takes a lot of time.
A r/LocalLLaMA post requests hot expert reload on GPU for llama.cpp, claiming decode-speed gains on MoE models with few active parameters. Named examples include Qwen3.8-Flash-Next, Deepseek V4/V4.1 Flash and GLM 5.3 Flash, with larger gains claimed on 2x 3090 setups.
A LocalLLaMA user reports fitting roughly 170k tokens of Qwen3.8-27B-UD-Q4_K_XL on a 32 GB GPU with MTP and mmproj enabled, avoiding KV cache quantization entirely. They say even q8_0 KV cache quantization is noticeably worse, and Qwen3.8 burns through the context at xhigh effort.
Pull request #28102 by pwilkin adds CUDA/HIP Flash Attention tuning for gfx1201, targeting RDNA4 cards including the R9700, RX 9060 XT and 8060S. The PR reports prompt-processing gains and improved numbers at large context sizes.
A llama.cpp maintainer recounts a comment confusing llama.cpp with Meta's Llama models, calling it a 'FB business using the pipeline to make profits.' The maintainer clarifies the distinction, noting audio.cpp follows the same philosophy for audio models.
Hugging Face's tutorial shows how to run local models in Pi using llama.cpp, covering GGUF model selection and quantization via its hardware compatibility feature. Fully private, offline, and no per-token cost.
A r/LocalLLaMA post claims official llama.cpp struggles to reach 50% of Strix Halo (gfx1151) hardware theoretical throughput, and that ~90% of the community uses it anyway.
A blog post argues Ollama, the popular local LLM tool, obscures its reliance on llama.cpp, the engine by Georgi Gerganov, and has drifted from its local-first mission while taking VC money. It recommends alternatives.
On 2x 20GB RTX 3080s, 128GB DDR4 and a Xeon 6148, llama.cpp ran Unsloth's Q4_K_XL quant of Qwen 3.8 Flash Next at ~270tps prefill and ~13tps decode, dropping from near 20tps.
A Reddit guide details 9 steps for using llama.cpp with a local GGUF model (e.g., Qwen3.8-27B-UD-Q4_K_M) and FreeCAD to generate solid objects suitable for 3D printing or milling. The setup runs on Linux and includes a pi coding agent.
XHToken released Spark-X2.5-4B and Spark-X2.5-1.7B, compact models with their own architecture. GGUF quantizations are available, and llama.cpp support is in PR #27868.
A pull request to llama-cpp-turboquant introduces block KV cache streaming via a shared CUDA phase arena, bounding VRAM usage at long context. The author ported and extended Raymond's work to multiple models beyond Qwen, benchmarking to confirm value.
A Reddit user built a villager simulation game proof-of-concept with Qwen3.8-27B-UD-Q3_K_XL.gguf, fully offloaded on a 16GB RTX 5070 Ti under Windows, with vision running on CPU. It uses beellama.cpp with kvarn performance enhancements and MTP n-max = 2.
A reproducible ROCm stack for Qwen3.8-27B on Ryzen AI Max / Max+ (gfx1151) with calibrated IQ4_XS weights, an IQ4_XS DFlash2 drafter, and retained-PM4 dispatch. Benchmarked on a Radeon 8060S with a 31,497-token prompt and 256-token output, the custom ROCm build claims fastest prefill and decode.
Reddit user benchmarked NInfer, llama.cpp, and vLLM serving Qwen3.8-27B in NVFP4 on an RTX 5090, comparing output quality and speed for a production content-intelligence pipeline with long-context retrieval and structured extraction.
NVIDIA announced up to 1.9x faster local inference via new llama.cpp and vLLM optimizations, available now through LM Studio and Ollama. RTX Spark Windows PCs from Lenovo and Acer arrive in October, alongside NVIDIA PAIR, a router that spreads inference across local-network PCs.
NVIDIA agreed to acquire Hugging Face for $12,930,300,000, per Jensen Huang's announcement. Hugging Face will stay an open platform supporting multi-cloud and multi-accelerator deployment, with NVIDIA compute not required to build on or deploy through it.
A user ran Qwen3.6-35B-A3B at IQ2_XXS quantization on an i3 laptop with 8GB RAM and no VRAM, generating an RPG in 24 minutes via llama-server with an 8192 context window.
Workflow combines Plaguekind's v8 pipeline with a custom prompt-enhancer node wired to llama.cpp for prompt enhancement and reference-image alignment.
A Reddit user asks for the fastest abliterated/safety-removed versions of the 3.6 and 3.8 27B models, citing community claims that 3.6 works better without thinking. The thread gathers recommendations for low-resource use.
Google's Android Studio uses llama.cpp to run Gemma 4 locally, likely via Vulkan and QAT versions. The integration enables on-device AI assistance within the IDE.
llama.cpp PR #28133 adds vision support for DeepSeek-V4-Flash-Vision-Exp, with Unsloth publishing matching GGUF weights on Hugging Face.
A developer ran Qwen 3.8 27B at Q4KM quantization and got a working Super Mario clone in a single take. Setup: Windows PC with a 4070 Ti (12GB VRAM, 32GB RAM) plus an M5 MacBook Air over llama.cpp RPC.
With build b10726, llama.cpp's default --lazy-mode change keeps Qwen 3.8 Flash Next's 51B-parameter PLE n-gram embedding table mmap'd on disk, reading rows on demand even with --load-mode none. Passing --lazy-mode off restores loading it into RAM.
Pull request #27402 by bartowski1182 speeds up large batch size prompt processing of IQ models on CPU using AVX2 instructions.
LLMFan46 released multiple uncensored models in GGUF format, including LongCat-Flash-Lite-Sparse with MTPs and LSAs, Qwen3.8-27B, Qwen3.5-122B-A10B, Qwen3-Coder-Next, and Laguna-S2.1 with vision. LongCat-Flash-Lite-Sparse required creating Heretic support from scratch and adding support to llama.cpp.
A 78.9 GB UD-Q2_K_XL Unsloth quant of Qwen3.8-Flash-Next ran a 358,400-token context slot on a 128 GB M5 Max MacBook Pro via llama.cpp b10686, using YaRN to extend from the native 262,144 tokens. The 3.5-hour, 100-turn test charted speed against context depth.
A Reddit post in r/LocalLLaMA lists open PRs and discussions for CPU/RAM/disk/hybrid inference in llama.cpp, aiming to improve CPU-only and hybrid performance. The author notes they are "just 50 PRs away from more faster inference" and hopes for progress by end of year.
A Reddit user shares a setup running Qwen3.8-27B at 50 tok/s with 100k context on an RTX 4070 Ti SUPER (16GB VRAM) using beellama.cpp and an IQ4_XS GGUF quantization.
An audit of 443 GGUF quants across 25 repos found 64 files whose filenames claim a lower bit-width than they actually are, due to llama-quantize silently swapping in a ~4.5 bpw type when tensor rows aren't divisible by 256. On Nemotron-3.5-Lightning, all four IQ2 rungs are the same 4.58 bpw file.
Qwen3.8 27B (Q4_K_M, 17GB) generates at ~14 tokens/s on a Mac Studio M3 Ultra via Ollama, while its predecessor qwen3.6:27b does ~28.6 tokens/s. The 1-bit quant (6.7GB) runs at 27 tokens/s in llama.cpp but cannot commit to an answer.
A developer reverse-engineered the Axera AX8850 NPU's engine format, storing int8 weights as two nibble planes, to run GGUFs without model conversion. It achieves 1.5× speedup over the vendor's runtime, running Qwen3-0.6B at 13.5–14.5 t/s on the M5Stack LLM-8850 card.
A user reports 220 tokens per second (averaging 170s) running a 27B model with Ninfer on an RTX 5090, roughly doubling throughput over llama.cpp.
llama.cpp merged PR #27342 adding DFlash2 support, including local convolution and candidate selector. The merge enables running DFlash2 models locally via llama.cpp.
Q4_K_M quantization of Qwen3.8 27B matches the full BF16 model on Terminal-Bench 2.1 while fitting in 17 GB, but 1-bit performs near random chance on GPQA Diamond. Benchmarks cost about $3,000 on Modal GPUs.
Blog post walks through squeezing performance out of a multi-GPU home server running LLMs with llama.cpp, covering transformer basics and multi-GPU parallelism. Focuses on practical settings and techniques, not new kernels.
llama.cpp version 0.2.0 is out, with docs now hosted at llama.app/docs. Recent PRs add DFlash2 support, an --n-cpu-ffn option for low-VRAM dense models, and adaptive speculation for faster inference.
ConvRot, a quantization method that reportedly achieves Q8-level accuracy at Q6 size, is now available in the llama-cpp-turboquant repository. The method was previously discussed in a Reddit thread.
A Reddit user in r/LocalLLaMA reports successfully running Qwen 3.8 27B after fixing their llama.cpp Docker config, connecting it to HomeAssistant. They thanked the community for troubleshooting help.
llama.cpp PR #26534 enables MTP for GLM-4.5-Air, a 106B MoE with 12B active parameters, offering speedups on memory-rich, compute-limited hardware like Strix Halo or DGX Spark.
A user reports a 30-50% speed boost after switching from llama.cpp on Windows to vLLM on Linux. The post on r/LocalLLaMA has 31 upvotes and 26 comments.
A llama.cpp fork optimized for AMD GFX906 (Mi50, Mi60, Radeon VII) promises up to double prompt processing speeds for deep infill. It adds pipeline parallelism, a cost-based split mode, and custom GCN HIP kernels for q8_0 KV cache quantization.
A Reddit benchmark of DFlash 2 (PR build) in llama.cpp on Qwen 3.8 27B reports 2.26x speedup on 100 real coding prompts, 4.68x with an n-gram drafter, and up to 8x on specific cases. Tested over 3 days on one RTX PRO 6000.
A Reddit user reports running Qwen3.8-27B at 262K context on an AMD Strix Halo (128 GB unified) plus RTX 3090 Ti, achieving 9.5 to 153 tok/s after 159 logged experiments. The setup also beats a dual-3090 vLLM box on HumanEval.