Tagged
Meta's open-weight model line: releases, fine-tunes and local inference. Curated and summarized from dozens of sources by AIBriefs.
llama.cpp PR #26563 adds an expert-use heatmap that caches frequently used MoE experts in VRAM while cold experts stay on CPU; the author reports 33 → 56 tok/s on an 8GB VRAM system.
A new pull request moves sampling to the GPU, yielding an 8% increase in tokens per second for Qwen3.6:35b on an NVIDIA 5090. Testing on an NVIDIA P40 showed a 4% inference speed boost.
A r/LocalLLaMA post asks users about moving from LM Studio to llama.cpp, drawing 30 upvotes and 57 comments. The poster wants to know what others had to learn to recreate their setup and which front-end harnesses or GUIs they switched to.
A r/LocalLLaMA post proposes a community-driven website where users share hardware specs paired with llama.cpp flags that work, aiming to crowd-source working configurations for local LLM inference.
Pull request #25589 by yomaytk adds multi-token prediction (MTP) support for Qwen3-Next in llama.cpp, letting the model run 'at full speed.'
llama.app is an official Mac frontend for llama.cpp that pairs with the llama serve command to run local LLMs. The r/LocalLLaMA PSA notes even longtime llama.cpp users often miss that it exists.
Recent llama.cpp builds load MTP/NextN tensors by default for any draft-mtp architecture, even when speculative decoding is disabled. Previously these tensors were skipped unless users enabled it explicitly; GGUFs bundling MTP weights (GLM-5.2, hy_v3, qwen35moe, step35) are affected.
A developer built a GBNF grammar compiler for llama.cpp, enabling 8B-parameter models to reliably call ~50 tools. The system, called Eris, uses an Obsidian vault as memory and runs locally in Rust.
Recent llama.cpp commits broke preserve_thinking behavior for chat templates baked into older DSV4 GGUF files, degrading the model in coding-agent use. Fix: pass --chat-template-file with an updated template.
A pull request to llama.cpp adds text-only Kimi K3 support; the author says it still needs someone to run the conversion and model to verify it works.
The llama.cpp repository has merged support for the Minimax-M3 vision model, enabling local inference for the architecture.
Pull request #24908 merges MiniMax M3 support, including MSA, into llama.cpp, enabling local inference of the model.
llama.cpp now supports MCP over stdio servers, complementing existing HTTP support. The integration was led by contributor ngxson.
A Reddit user asks whether a 128GB MacBook Pro can run local models comparable to today's frontier coding assistants, citing concerns about cloud pricing. The discussion explores hardware options for local AI development.
A llama.cpp fork called MindControl injects guidance during sampling to improve reasoning consistency of smaller local models. The author created it after frustration with unreliable reasoning behavior in Qwen3.6-27B at low temperatures.
The b10087 release adds support for Laguna XS.2 and M.1 models.
Benchmarks four popular open-source LLM fine-tuning frameworks: Unsloth rewrites kernels for speed, Axolotl composes parallelism strategies, TRL defines the RLHF pipeline, and LLaMA-Factory offers a modular interface. The comparison covers speed, VRAM usage, and multi-GPU scalability.
A Reddit post in r/Singularity claims Google's Gemini is now behind Meta's models. The post provides no evidence or specifics. It has 36 upvotes and 16 comments.
A llama.cpp trick using VRAM as disk cache reaches 340 pp/s and 9.6 tg/s running a 204GB Kimi-K2.7-Code GGUF on a single DGX Spark. The strategy keeps MoE experts on the CUDA compute path by caching them in VRAM over disk.
pi 0.81.0 now features native integration with llama.cpp via llama-server router, replacing the pi-llama extension. A video demo is available.
A Reddit user describes using an eGPU on a laptop to run Qwen3.6 35B A3B with llama.cpp. The post highlights llama.cpp's built-in benchmark feature for tuning inference on modest hardware.
Users report running DeepSeek V4 Flash using the Q8_K_XL quantization on an NVIDIA RTX 5090. The implementation leverages recent llama.cpp updates to support a 1 million token context window.
On an RTX 6000 PRO with Qwen 3.6 27B, llama.cpp's n-gram stack hit ~6x speedup, DFlash ~3.7x, and MTP ~2.7x on real coding. DFlash recently merged via PR #22105; an earlier run posted 4.44x at 36K context.
Libargus.cc is a new library that runs local LLMs directly from the JVM using the OpenJDK Panama Foreign Function & Memory API in Java 22, eliminating the need for a REST sidecar. It interfaces directly with llama.cpp for low-latency inference. The project is open-source on GitHub.
llama.cpp release b9978 fixes a checkpoint bug that hit agentic workloads hardest: every agent turn created a new checkpoint, bypassing min-step spacing and collapsing the context coverage window.
TurboQuant v0.3.0 ships a training-free KV-cache compression method for llama.cpp. The update fixes a CUDA precision flag issue that caused silent errors on older GPUs like the Tesla P100.
A Reddit user built an interactive Jacobian-Lens visualizer and live steerer for GGUF models on llama.cpp. The tool is inspired by Anthropic's interpretability research and provides real-time steering.
A Reddit user reports that Qwen3.6 35B-A3B (Q8_0, no KV quant) generated a complete flight simulator HTML with mountains, clouds, and procedural terrain from a single prompt in opencode. The user noted significantly better performance at Q8_0 on CPU compared to Q4_K_M on GPU.