- Python 97.7%
- Shell 2.3%
| tests | ||
| .gitignore | ||
| fla_causal_conv1d_shim.py | ||
| LICENSE | ||
| moe_routing_cache_patch.py | ||
| preflight.py | ||
| pyproject.toml | ||
| README.md | ||
| run.sh | ||
| train.py | ||
| uv.lock | ||
deterministic-stack
A reproducible LoRA fine-tuning smoke-test recipe for Qwen3.6-MoE on AMD MI100 / ROCm.
The verified claim is deliberately narrow: with the same base, data, seed, and environment, the smoke run produced byte-identical adapter weights twice in a row. The failure logs prove gradient-checkpoint forward/recompute tensor count/metadata mismatches; the MoE routing explanation is the mechanism that fits the failure shape and the fixes, not a standalone router-index ablation.
Status: working as of ROCm 7.2.2 + torch 2.10 (rocm7.0 wheel) + transformers 5.4.0 + gptqmodel 6.0.3 + peft 0.19.0 + trl 0.12.0. Tested on 2× MI100 (64 GB combined VRAM). YMMV on other ROCm versions, other AMD parts, or full-length production runs.
Why this exists
Most fine-tuning recipes published in 2025 don't reproduce. Same script, same data, same seed, different weights. On AMD MI100s the situation is worse: the upstream Hugging Face stack assumes a CUDA-shaped world, and the divergences are subtle enough to look like training noise.
This repo is the working list of what we had to pin, patch, or replace to get
the same adapter bytes twice on the smoke workload. It is meant to be read as
a recipe, not a library — the load-bearing pieces are three short files plus a
pyproject.toml that pins the Hugging Face/GPTQ stack and the ROCm build
flavor of torch.
The three load-bearing patches
1. fla_causal_conv1d_shim.py
Qwen3.6-MoE's GatedDeltaNet layer wires causal_conv1d_fn to the
CUDA-only causal-conv1d PyPI package. When that package isn't installed
(it can't be, on ROCm), the attribute is set to None and the layer falls
back to a plain F.silu(self.conv1d(...)) path. Replacing that fallback was
one layer in the final working stack; the archived artifacts don't isolate it
alone.
We replace the fallback with fla's
Triton-based causal_conv1d, which works on the ROCm stack we tested. The
shim handles the [B, D, T] ↔ [B, T, D] layout disagreement between the two
functions.
2. moe_routing_cache_patch.py
The failure pattern is consistent with near-tied top-k routing decisions changing across recompute. With 256 experts and 8-active routing, many softmax scores are near-tied; a flip changes which experts' weights get touched, which changes the tensor shapes/counts PyTorch sees during recompute, which trips checkpoint's checks. The logs prove the checkpoint mismatch; they do not contain a clean "forward top-k != recompute top-k" printout.
We can't bypass the gate's compute (the count check enforces equal
op-graph shape across forward and recompute). What we can do: run the full
linear → softmax → topk chain on both passes — same ops, same tensor count —
but on recompute, replace topk's indices with the cached ones from forward
after topk has run. The gather op runs on both passes, so the autograd
graph is structurally identical. The only difference is the integer index
tensor's values, and integer tensors carry no gradient.
forward: linear → softmax → topk → [cache indices] → gather → divide → cast
recompute: linear → softmax → topk → [override w/ cache] → gather → divide → cast
Result: tensor count matches, per-expert shapes match, gate weights still train normally.
3. attn_implementation = "eager"
For this stack, SDPA/flash were the wrong surface to debug through. Eager
(pure-PyTorch matmul + softmax) keeps the full-attention layers on ordinary
ops where use_deterministic_algorithms(True) can do its job or fail loudly.
It is slower per step. We force it on the loaded model in train.py.
Companion knobs (boring but mandatory)
torch.use_deterministic_algorithms(True, warn_only=False)— fails loudly if any op slips through with a non-deterministic kernel.CUBLAS_WORKSPACE_CONFIG=:4096:8— set inrun.sh. Keeps the BLAS side inside the determinism contract used by the smoke test. (cuBLAS reads this env var on ROCm too.)gradient_checkpointing_kwargs={"use_reentrant": False}— the older reentrant path doesn't enforce the same tensor-count invariants and silently corrupts gradients in some cases. Non-reentrant is required.- GPTQ-prequantized base, loaded via
gptqmodel. bitsandbytes 0.49.2 segfaults during weight load on ROCm 7.0 with this MoE; gptqmodel sidesteps bnb entirely. PEFT'sprepare_model_for_kbit_trainingworks on a GPTQ-loaded base.
Layout
deterministic-stack/
├── README.md # this file
├── LICENSE # MIT
├── pyproject.toml # uv project: pinned ROCm-flavored deps
├── preflight.py # env validator — RUN THIS FIRST
├── run.sh # canonical entrypoint (sets CUBLAS_WORKSPACE_CONFIG)
├── train.py # the trainer
├── fla_causal_conv1d_shim.py # patch #1
├── moe_routing_cache_patch.py # patch #2
└── tests/hash_adapter.py # byte-for-byte adapter comparison
There is no data/ or models/ directory. The training corpus, the base
weights, and the produced adapter are deliberately out of scope — what's
reproducible here is the procedure, not someone else's training run.
Reproduce
# 1. Set up the environment
git clone <this repo>
cd deterministic-stack
uv venv
source .venv/bin/activate
# 2. Install ROCm-built PyTorch FIRST so uv doesn't pull the CUDA wheel
uv pip install --index-url https://download.pytorch.org/whl/rocm7.0 torch
# 3. The rest of the deps
uv sync --no-install-project
# 4. Validate the environment — MUST pass before train.py
python preflight.py
# torch import PASS torch 2.10.x (with HIP)
# HIP build PASS HIP 7.0.x
# GPU visibility PASS available=True, count=2
# expected GPU count PASS 2 >= 2
# AMD GPU detected PASS all AMD: ['AMD Instinct MI100', ...]
# bf16 compute PASS bf16 matmul OK
# import transformers PASS transformers 5.4.x
# ...
# RESULT: all critical checks passed.
# 5. Smoke test (1 epoch, 5 examples — proves the loop runs end-to-end)
./run.sh --base-model ./models/qwen3.6-moe-gptq-int4 \
--train-data ./data/train.jsonl \
--smoke
# 6. Full run
./run.sh --base-model ./models/qwen3.6-moe-gptq-int4 \
--train-data ./data/train.jsonl
You provide the base model (a GPTQ-prequantized Qwen3.6-MoE — there's
nothing magic about how it's quantized, just that it is) and the training
data (JSONL with {"messages": [...]} per line, OpenAI chat-format).
Determinism, verified
Run twice with the same seed, the same data, the same base. Hash the saved adapter artifacts:
OUT=/tmp/deterministic-stack-smoke
./run.sh --base-model ./models/qwen3.6-moe-gptq-int4 \
--train-data ./data/train.jsonl \
--output-root "$OUT" \
--run-name run-a \
--smoke \
--seed 42
./run.sh --base-model ./models/qwen3.6-moe-gptq-int4 \
--train-data ./data/train.jsonl \
--output-root "$OUT" \
--run-name run-b \
--smoke \
--seed 42
uv run python tests/hash_adapter.py "$OUT/run-a" "$OUT/run-b"
Verified on 2026-06-11 with /home/daniele/models/Qwen3.6-35B-A3B-GPTQ-Int4-self,
data/curate/train-v3.jsonl from lares-finetune, --smoke, seed 42,
2× MI100-visible ROCm runtime:
ADAPTER_BYTES_IDENTICAL=yes
adapter_model.safetensors
a: 288c2ee7ca0e511ccbfaf350bc15ec3c0c10572a98d9f01132564cb0049a4bbc
b: 288c2ee7ca0e511ccbfaf350bc15ec3c0c10572a98d9f01132564cb0049a4bbc
checkpoint-2/adapter_model.safetensors
a: 288c2ee7ca0e511ccbfaf350bc15ec3c0c10572a98d9f01132564cb0049a4bbc
b: 288c2ee7ca0e511ccbfaf350bc15ec3c0c10572a98d9f01132564cb0049a4bbc
If any tensor differs, start with the training log: check for
Patched causal_conv1d_fn on N GatedDeltaNet layers..., the eager-attention
line, and the routing-cache install line before looking for more exotic
causes.
What this is not
- Not a Qwen-specific library. The two patches reach into class names
(
Qwen3_5MoeGatedDeltaNet,Qwen3_5MoeTopKRouter) — both will need renaming for other MoE architectures with the same problems. Note: theQwen3_5Moe...class prefix is a Hugging Face naming holdover and does not match the model line. The model is Qwen3.6-MoE; the classes are whattransformerscalls them. - Not the full Lares fine-tune. The corpus, the rubrics, the held-out eval set, the actual produced adapter — those live elsewhere and aren't shipped with the recipe. This repo is the infrastructure, not the artifact it produces.
- Not a replacement for FSDP/Accelerate at scale. We deploy on 2× MI100
with
device_map="auto". For larger setups you'll want FSDP, which has its own determinism story — none of these patches are incompatible, but the verification above only covers the single-process / 2-card case. - Not proof of full-run production determinism. The checked artifact was a smoke run. That is valuable because it exercises load, GPTQ hooks, LoRA, checkpointing, optimizer step, and adapter save; it is not the same claim as reproducing a full corpus/schedule end to end.
What we still don't know
- Whether the routing-cache patch is necessary on batched_mm /
grouped_mm experts implementations (we use the default
linear_loop_experts_forwardbecause gptqmodel unrolls per-expert modules; the fused-tensor paths don't apply to our layout). - Whether ROCm 7.3+ improves determinism around the fused/numeric surfaces enough to make patch #2 unnecessary. Re-test when the next ROCm drops.
- Whether the same recipe works on MI200 / MI300. Should — they have better fp determinism than MI100 — but unverified.
License
MIT. See LICENSE.
Citing
If this saved you a week:
@misc{salatti_deterministic_stack_2026,
title = {Deterministic LoRA SFT for Qwen3.6-MoE on AMD MI100},
author = {Salatti, Daniele},
year = {2026},
howpublished = {\url{https://v37.ai/research/deterministic-stack-mi100}},
note = {v37.ai},
}