MiniMax-Music3 backend

Native C++/GGML port of MiniMaxAI/MiniMax-Music3, built 2026-08-13 (release day) as HOT-Step's second generation backend behind an N-backend abstraction. Status: rudimentary text2music only (caption + lyrics + duration + seed); no covers/repaint/stems/adapters/training. Output = raw 44.1 kHz stereo WAV (app norm is 48 k — post-chain steps that hardcode 48 k are skipped for MM3).

Deep docs (local, gitignored): docs/plans/multi-backend-architecture.md (architecture plan, day-0 findings, op inventory) and docs/plans/mm3-gguf-layout.md (GGUF contract + loader addendum). Caption format: the mm3-captioning skill.

Model + pipeline (25 fps frames; every module parity-proven vs the diffusers reference)

caption+lyrics → Qwen2 BPE → Global LM 8.59B (Qwen3 arch, semantic codes @ ids 151675–168058,
  EOS 151670, AR CFG 1.5 as persistent 2-row batch) → per frame: RVQ depth decoder 0.6B
  (7 acoustic codebooks) → frame_hiddens [F,8,4096] → per 200-frame window (hop 100):
  condition encoder 25M (×3.4453125 nearest resample) → flow DiT 2.4B (30 Euler steps,
  CFG 1.7, zeros-cond uncond as separate pass) → vocoder 54M (DAC-style, ×512 → 44.1 kHz)
  → overlap-crop stitch

LM sampling knobs (2026-08-25)

The AR stage's semantic draw takes the full knob set (engine fields on MM3GenRequest, wire names lm_*, UI via the minimax backend param registry keys mm3Lm*): lm_temperature, lm_top_k (0 = the checkpoint's 50), lm_top_p (nucleus over the top-k survivors), and lm_rep_penalty with the ACE LM's three modes ported (dry default / frequency / presence).

• Time constants are 25fps-rescaled: window 320 (~12.8 s), DRY min-match 15 frames (0.6 s). Never copy ACE's 5 Hz numbers (64 / 3-6) literally.
• DRY punishes only codes that would extend a verbatim recent cycle — the memorising-adapter loop failure — and leaves musical restatement alone. Useful range 1.05-1.15.
• Parity is proven: at default knobs the sampler takes the exact pre-knob code path; fixed-seed renders hash bit-identical across the change (da869838…, old and new builds, identical launch). Knobs at defaults are omitted from the wire so the engine recipe stays authoritative.
• The depth decoder's sampler is untouched on purpose: loops do not live in the per-frame acoustic codes, and perturbing its input distribution re-opens the timbre question the acoustic loss closed (training skill).

Where the knobs render: the group field (2026-08-27)

Declared knobs (capabilities().extensions) carry an optional group: 'generation' | 'lm', and each generic top-bar dropdown renders its own group — BackendGenerationDropdown and BackendLmDropdown, both on the shared schema renderer in BackendExtensionControls.tsx. An untagged knob is a Generation knob, which is where every one of them lived before groups existed, so an older manifest still renders exactly as it did.

MM3's group: 'lm' set is the six mm3Lm* sampling knobs plus the four that decide what happens to the planner's output: mm3ArSeed, mm3ReuseAr, mm3SaveArCodes, mm3PlankPath.

Two things to know before touching the LM cluster:

• features.lm does not mean "has an LM." It means "has ACE's CoT metadata LM" — a stage that is genuinely optional. MM3 reports lm: false and still has an LM; it is just an autoregressive planner that always runs. The bar shows the LM tab on features.lm || any knob tagged group:'lm'.
• No global on/off in MM3 mode. GlobalParamBar hangs the section's headerToggle (skipLm) only when features.lm is true. There is no MM3 render without the planner, so a switch there would be a lie.

File map

Piece	Where
Engine modules	engine/src/minimax/ — mm3-model.h (loader/residency), mm3-tokenizer.h, mm3-lm-graph.h, mm3-ar-loop.h, mm3-sample.h, mm3-depth-graph.h, mm3-cond-graph.h, mm3-dit-graph.h, mm3-vocoder-graph.h, mm3-pipeline.h (e2e + chunking), mm3-request.h (prompt assembly/hygiene), mm3-job.h (job queue + VRAM arbitration), mm3-server.h (endpoints)
Hooks	one include in engine/tools/hot-step-server.cpp (+ mm3_register_routes/mm3_register_job_routes call sites); checked by engine/verify-hooks.ps1 (hooks 4/4b/4c)
Server backend	server/src/services/backends/ — types.ts (EngineBackend + capability manifest), registry.ts, ace/, minimax/{client,index,generate}.ts; routes server/src/routes/backends.ts; generation branch at top of runGeneration in routes/generate.ts
UI	stores/backendStore.ts, hooks/useCapabilities.ts, global-bar/BackendToggle.tsx (hidden until ≥2 backends), shared/BackendCapabilityGate.tsx (studio guards), gating in GlobalParamBar.tsx
Models	5-way split since 2026-08-14 (ported from ServeurpersoCom/minimaxmusic.cpp): models/mm3/mm3-{lm,depth,cond,dit,voc}-<quant>.gguf (archs qwen3 / mm3-{depth,cond,dit,voc}), legacy mm3-synth-* bundles still load (fill any role; split file wins per quant token). Per-role quant mixing (LM Q8_0 + DiT Q4_K_M is the headline combo); a DiT/adapter swap reloads only cond+dit+voc — LM stays warm. cond/voc are never quantised (f16 only). Hosted scragnog/MiniMax-Music3-GGUF; registry role mm3, packs rebuilt on split components in server/src/data/model-registry.json
Converter	engine/tools/convert-mm3.py (safetensors→GGUF bundle; folds weight-norm; refuses pruned/int8_convrot) then engine/tools/split-mm3.py (byte-exact bundle→5-way split; idempotent; cond/voc only from native bundles)
Fixtures / parity	D:\Ace-Step-Latest\mm3-weights\fixtures\ (manifest.json + raw f32 dumps + reference WAVs), seed-spread study in ..\seed-spread-2026-08-13\; venvs: .venv-convert (numpy/gguf), .venv-ref (patched diffusers @ dafe3733 — patch_venv.py --restore; capture_fixtures.py --replay rebuilds dumps without rerunning the model)

Engine endpoints (:8085 via app, standalone tests on :8086)

GET /mm3/props (files/config/loaded/limits — blocks while an MM3 generation runs; always call with ~2.5 s timeout and keep last-known-good), POST /mm3/warm / POST /mm3/unload (idempotent; unload frees weights+KV), POST /mm3/synth (production, rides the same FIFO GPU worker as ACE /synth; standard /job?id= progress/cancel/result; request contract documented in mm3-request.h/mm3-job.h), GET /mm3/job?id= (MM3-vocabulary progress, never blocks; &ar=1 returns the Plank code blob — see below), GET /mm3/stream?id= (live audio of a running job — chunked WAVs, one reader, never takes the MM3 mutex; see "Streaming player" below), POST /mm3/tokenize-check (cold-capable; 5000-token limit), plus deprecated bring-up endpoints (/mm3/voc-decode, /mm3/dit-forward, /mm3/flow-sample, /mm3/depth-frame, /mm3/cond-encode, /mm3/lm-plan, /mm3/synth-e2e) kept for parity work — they run GPU work on httplib threads; never build production paths on them.

Standalone launch gotcha: ace-server exits 0xC0000135 with zero output unless engine/trtllm-libs + engine/deps/tensorrt_libs are prepended to PATH (aceEngineProcess.ts does this; engine/server.cmd does not).

Caption echo (added 2026-08-21). POST /mm3/synth prints the caption to stderr at job creation, so it reaches the terminal, ace_engine.log and the in-app Terminal — the MM3 analogue of ACE's [LM-Phase2] CoT[0] dump, which MM3 had no equivalent of:

[MM3-Job] <id> created - 63 prompt tokens, ...
[MM3-Job] <id> caption (149 bytes in, 143 cleaned), lyrics 46 bytes:
<the cleaned caption>

It prints the cleaned caption (post mm3_clean_caption), not the raw body, because the two differ exactly where a markdown-emitting tool pasted **bold** headings or - bullets in — the drift you would otherwise only hear. MM3_LOG_PROMPT=1 swaps it for the whole assembled template (<|im_start|><|caption_start|>…<|lyrics_start|>[start]…<|audio_start|>). The Node-side [Generate] … caption=N chars line is the send-side half; a mismatch between the two counts localises a drop to the wire rather than the UI.

Natural-ending candidates (SHIPPED 2026-09-09, 00e3e7af)

POST /mm3/synth accepts require_eos: true and eos_rounds: N (1..16) with takes: K. The planner runs K takes in one batched pass (seed+t); any take that reaches max_frames without EOS is DROPPED before the flow stage; if none ended the plan repeats at seed + K (round r plans seed + r*K + t) up to eos_rounds times, then fails with "no candidate ended naturally". The job JSON's takes is the number RENDERED; takes_planned, takes_dropped, eos_rounds_used, require_eos and a per-take round are added, and take_detail is emitted whenever candidates were in play. Ignored on an interleaved stream (logged). Server: mm3RequireEnding (default on, Generation dropdown) sends takes 3, eos_rounds 4 and reads the surviving count/seeds from the completion detail. Duration on MM3 is ALWAYS auto (a requested length was a hard cap that cut endings off); the Create panel hides the control in MM3 mode. Batched take 0 is a different song from the same seed by design (check-mm3-ensemble.mjs). Knock-ons: Save Plan To Disk is dead while the toggle is on; each ended candidate costs its own flow pass.

The trap list (each cost real debugging — do not relearn)

1. ComfyUI's wrapper NEGATES the DiT output; the diffusers reference (and our port) does not. mm3.dit.output_negated in the GGUF records Comfy's behavior. Do not "fix" the sign.
2. tokenizer.ggml.pre = qwen2 is misleading — the reference uses the slow Qwen2Tokenizer (single-digit regex = classic GPT-2 pre-tokenization, which bpe.h implements). Matching the KV's llama.cpp meaning ({1,3} digit grouping) breaks token parity.
3. Scheduler sigmas must replicate float32 linspace(1, 1/30, 30) rounding — deriving i/steps is wrong in the 7th digit and it matters.
4. AR iteration 0 is fed back but never emitted (emitted frame j = iteration j+1). A one-frame indexing slip degrades conditioning parity 49×.
5. The semantic code embeds via the LM's token_embd, not depth.audio_embd.
6. Caption hygiene: splitlines() for caption, split("\n") for lyrics — mixing them leaks a trailing \n into the template. Empty lyrics → we substitute [instrumental] (the reference rejects empty; this substitution is a HOT-Step decision).
7. Condition resample is plain nearest, not nearest-exact (differs on 199/689 positions).
8. Never use std::normal_distribution for reproducible noise (stdlib-dependent bytes) — mm3_fill_noise uses splitmix64 + Box-Muller.
9. GGUFs live in the models/mm3/ subdir deliberately: the ACE registry scan globs only the models root (unknown-arch warnings + 17 GB header reparse per boot if placed there).
10. Single-seed spectral/genre judgments are meaningless — the reference's own 11-seed spread spans 272× in flatness and wanders off-genre with minimal captions. Structured 3-section captions (mm3-captioning skill) are the adherence lever. Compare distributions, not takes.
11. VRAM: f16 stack ≈ 22.5 GB + KV (288 kB/position) + ~3 GB compute headroom. Engine-side arbitration evicts idle ACE modules before MM3 warm; Node-side releaseVram() handles the reverse on backend switch and before ACE gens. ~600 MB stays in the CUDA pool after unload (returns on process exit — not a leak).
12. The LM GGUF is not interchangeable with stock Qwen3-8B GGUFs (extended 200 k vocab, untied head) and llama.cpp alone cannot run music generation. It IS interchangeable with a depth-pruned distilled composer — see "Alternative composer LMs" below.
13. read_wav_buf returns INTERLEAVED [T,2]; the DAV encoder wants PLANAR [L:T][R:T]. Use audio_io_read_wav_buf (audio-io.h), which de-interleaves — never the raw reader. mm3-preprocess sliced the raw reader's output as {p, p+T} and made "left" the FIRST HALF of the song with L/R alternating. Since L≈R, that duplicates every sample: an exact 2× time stretch, one octave down. Every cached target was the song in slow motion and five LoRA runs learned to generate slow motion (2026-08-15, fixed 82b2852).
14. VERIFY PREPROCESSING BY DECODING A TARGET AND LISTENING — metadata cannot catch this class of bug. #13 survived a full day because T is PER-CHANNEL frames, so latent_frames / duration stayed at exactly 86.1328 Hz and every arithmetic check on the manifest passed. The DAV parity gate passed too (it is fed encode_ref.py's planar dump). The manifest was written by the same buggy code being checked, so it corroborated itself. One listen to a decoded target found it. POST /mm3/voc-decode?frames=N with raw f32 [128,N] returns a WAV — there is no excuse not to. Objective version of the same gate: encode a 440 Hz sine and measure what comes back (was 220.0 Hz, i.e. ratio 0.5000; correct is 440.0 Hz / 1.0000). A pure tone cannot be argued with, and it brackets which stage is at fault.
15. Rob's ear beat every metric, twice. He called "slow motion, too deep" on the first adapter and again on the third; both times it was explained away as regression-to-the-mean (which produces a genuinely similar description) and five runs of hyperparameter tuning followed on corrupt data. When the user reports a physical symptom — speed, pitch, duration — treat it as literal and test it literally before reaching for a statistical explanation.
16. Gate every trained adapter on ||delta||/||W|| BEFORE any ear test. Healthy LoRA merges move weights 1–5% Frobenius; at lr 5e-4 × 8k steps ours hit median 17% (max 34%) and at scale 1.0 that is a damaged model, not a strong style — jumbled inside a single 689-latent window, invariant to rank/crop/CFG (AdamW makes total movement ≈ lr×steps regardless of rank, which is why every knob "did nothing"). Measure against the ComfyUI f16 checkpoint (mm3-weights/comfy/diffusion_models/), whose keys match the export directly; target median ≤5%. SimpleTuner's reference recipe is lr 5e-5.
17. Training crops must not straddle conditioning-rollout seams. mm3-condition builds the cache from independent 60 s segments; a crop across a seam pairs continuous audio with conditioning that jumps to an unrelated rollout mid-window — teaching "conditioning lies, smooth over it" (mean-collapse pressure). The seams were parsed and never consulted for a week (~13% of crops at 689, 27% at 1378); fixed 5117281 with reject-and-retry.
18. Filter groups at EXPORT, never constrain them at TRAINING. Measured (runs 09/10, matched ~2.6% delta, same groups): full-set training + MLPV surgery = coherent with clear lyrics; --target mlpv trained-from-scratch = intrusions and jumble, with LESS style at matched delta. Gradient denied its natural pathway (q,k routing) emulates it destructively through the remaining groups, so "safe-group" deltas from a constrained run carry structure-entangled content the base attention cannot support. The winning recipe: train ALL sites at modest delta, then zero q,k rows + proj heads in the export (groupfilter.py pattern — q,k rows are B[0:4096] of the fused qkv; ablation-proven: q,k = structure poison, proj_in/out = seed-dependent fuzz, MLP+V+out = timbre).
19. An MM3-only install must not kill the server at boot. The startup gates in hot-step-server.cpp (registry_scan empty → exit 1; partial ACE synth without LM → exit 1) predate MM3 and knew nothing about it: a user with only models/mm3/*.gguf got a dead engine → empty model dropdowns for BOTH backends + the MM3 "weights missing" CTA, while the Model Manager (Node disk scan, checks subdirs) said everything was installed (GitHub issue #118). Both gates now fall through when mm3_weights_present() (mm3-model.h — filename-only probe of <models> + <models>/mm3) is true; ACE handlers already degrade per-request with an empty registry. Any future boot-time hard-exit must ask "can MM3 still serve?" first.

Alternative composer LMs (depth-pruned + guidance-distilled) — 2026-08-22

The composer LM is independently swappable: the 5-way split means a variant replaces mm3-lm-*.gguf only, and depth/cond/dit/voc stay as they are. Proven with Mothersuperior/minimax-music3-composer-5.7b-distilled (36 → 21 blocks, 5.69B, repair-distilled against the teacher's CFG-guided distributions; two LR arms). Everything but the block count is bit-for-bit stock — the depth decoder consumes a 4096-wide hidden and the 200 k audio vocabulary is what makes it a music model, so those cannot move.

python engine/tools/convert-mm3.py --src <arm-dir> --out models/mm3 \
  --components lm --quant q8_0 --lm-layers 21 --ar-cfg-scale 1.0 \
  --suffix=-d21-lr6e5 --tokenizer <official>/tokenizer/tokenizer.json

• --suffix (needs =, else argparse eats the leading -) makes the whole trailing token the variant name, so tagged files appear as extra entries in the LM dropdown next to the stock quants. That is the A/B mechanism — no file juggling.
• --lm-layers is guarded by the leftover-tensor diff, not by trust: a wrong count leaves whole model.layers.N.* groups unconsumed and the run dies.
• source layout: unknown on a bare Qwen3ForCausalLM dir is expected and harmless.

CFG 1.0 means single-row, and the engine acts on it. mm3_cfg_rows() (mm3-model.h) returns 1 when mm3.ar.cfg_scale == 1.0, because u + (c-u)*1.0 is identically c — the unconditional row would be computed, read back and cancelled. The LM graph, its KV cache and the depth decoder all build single-row; the AR loop mirrors row 0 into row 1 so every consumer downstream stays unconditional. Keyed on the arithmetic, never a model name.

Measured (RTX 5090, matched caption/seed/duration, only the LM swapped):

	teacher 36L / 2 rows	distilled 21L / 1 row
LM decode	8.0 ms/step	3.8
depth decode	9.4 ms/frame	7.9
AR stage	5317 ms	3534
end to end (12 s clip)	9.5 s	7.7 s
LM KV cache	288 kB/pos	84 kB/pos
Q8_0 file	9.13 GB	6.05 GB

The depth decoder is the clean control for the row change alone — identical weights in both runs, so its 1.19× is bought purely by dropping the row. Design note A ("2 rows are ~free because decode is bandwidth-bound") is therefore only mostly right: at these tiny per-row matmuls the second row costs ~20 %, not ~0 %. The rest of the LM's 2.1× is the 36→21 prune.

Casualty: LRC alignment. MM3_ALIGN_HEADS (mm3-align.h) pins layers 12/19/24, found empirically on the teacher. On 21 layers, 24 does not exist — the replay loop clamps (mm3-lm-graph.h, i < m.lm.blk.size()) so nothing crashes, but the heads are teacher-specific and the timestamps are not to be trusted. Re-discovery (MM3_ALIGN_DUMP=1) would be needed per variant. Audio is unaffected.

Not yet judged by ear. Renders staged in _experiments/_LISTENING/2026-08-22_mm3-distilled-lm/. Remember trap #10 before drawing any conclusion from them: this is a planner swap, so 02/03 are different takes, not degraded copies of 01, and a single seed proves nothing.

Low step counts go THIN, not dull — and why (root-caused + fixed 2026-08-21)

Dropping steps below the checkpoint's 30 degrades in a specific, non-obvious way. Measured on a matched 10-vs-30-step pair (same seed/caption/structure):

	10-step vs 30-step
L/R correlation	−0.07 vs +0.77 (anti-phase mids, 160 Hz–2.6 kHz)
side/mid ratio	+0.5 dB vs −9.5 dB
Mid spectrum	−8 dB @ 60 Hz, tapering to 0 dB above 2 kHz

So it is thin and phasey ("tinny", "cheap radio"), not dull — HF is already at the correct absolute level. Do not reach for a treble fix; the tilt is the illusion.

Root cause, three facts that only bite together:

1. The vocoder decodes latent channels 0–63 as LEFT and 64–127 as RIGHT in two INDEPENDENT passes (mm3-vocoder-graph.h:596). Zero cross-channel coupling.
2. Initial noise is i.i.d. across all 128 channels (mm3-pipeline.h:462), so the two halves start completely uncorrelated.
3. The schedule is uniform, shift=1 (mm3-dit-graph.h:744), faithful to upstream — and the GGUF declares mm3.flow.steps = 30. There is no low-step compensation in the reference at all.

⇒ Every bit of stereo coherence must be manufactured by the DiT along the trajectory. Coarse Euler steps leave that work unfinished, and the same starved high-noise phase costs the low end. It is NOT residual noise: the excess side energy tracks the music envelope at +0.93 and drops in quiet passages — that measurement is what rules the noise hypothesis out, so run it before assuming otherwise.

The fix (shipped, server-side, no rebuild): shift = 29 / (steps − 1), derived by setting the shifted grid's first step 1/(shift·(steps−1)+1) equal to the 30-step native 1/30. Returns exactly 1.0 at 30 steps, so the curve is continuous and can never perturb a default render. Lives in mm3LowStepShift() (server/src/services/backends/minimax/generate.ts), applied by the mm3AutoLowStep extension (default ON). It forces scheduler + shift only — forwarding the shared inferMethod/guidanceMode pickers would silently swap MM3 onto ACE's APG default.

EAR-VALIDATED at 10 steps (shift 3.2). 8–29 is interpolation on a curve anchored at both ends. Below 8 is extrapolation — the first-step match is bought with an ever-larger final leap to clean (0.54 @ 6 steps, 0.86 @ 2), which must break down somewhere. Symptom to tune against: muddy/smeared = shift too high for the budget; thin/wide again = too low. Slider min is now 2 steps.

DSP fallback (built, measured, NOT shipped): a linear-phase M/S correction — +8 dB mid low-shelf, −3 dB side with a −8 dB bell at 1.4 kHz — recovers the 30-step balance to 0.63 dB (mid) / 0.83 dB (side) RMS error. It cannot restore HF coherence (only ~0.2 correlated with the 30-step above 2.5 kHz), so fixing the trajectory beats correcting after the vocoder. Analysis scripts + A/B renders: D:/Ace-Step-Latest/_experiments/_LISTENING/2026-08-21_lowstep-dsp/.

Sampler plugins: shared with ACE (built 2026-08-16 — NOT YET COMPILED OR HEARD)

The same Lua solver / scheduler / guidance plugins that drive the ACE DiT now drive MM3's flow DiT. No plugin was modified and no plugin API was widened — the plugin layer never had an ACE dependency (every lua_call_* entry point takes raw float * + counts + a param map); what was ACE-specific was the sampler, not the plugins. Bridge: engine/src/minimax/mm3-plugins.h.

Two conventions differ, and both mappings are exact:

1. Time runs the other way. ACE t descends 1→0 with xt -= vt*dt; MM3 sigma ascends 0→1 with x += dsigma*v. Substituting sigma = 1-t and v_ace = -v_mm3 makes them the same expression — including the terminal step, where MM3's last increment (1 - sigma[steps-1])*v is character-for- character ACE's engine-owned x0 = xt - t_curr*vt. MM3's steps+1 sigma array IS ACE's "N timesteps + engine-owned final step".
2. The latents are transposed. ACE is time-major [T][Oc], MM3 is channel-major [C=128][L]. This is NOT cosmetic: apg_forward normalises per channel over time and indexes [t*Oc+c] to do it, so a channel-major buffer would be grouped along neither axis. The bridge transposes into the ACE view before any plugin sees a buffer. 4 transposes/step of 88k floats against two 2.4B forwards — free.