A user stated that by simply duplicating specific intermediate layer blocks without making any changes to the weights of Qwen2-72B, they achieved performance improvements across all benchmarks on the HuggingFace Open LLM Leaderboard.
Using this method, they secured 1st place on the leaderboard. It is reported that all models currently ranked in the top four are derivatives of this approach.
Regarding the discovery of layer duplication, the user noted that duplicating a single layer has no effect, and performance declines if the number of layers is either too few or too many. They claim that only blocks with a circuit size of approximately seven layers are functional.
This suggests that pre-training carves out discrete functional circuits within the layer stack, and these only operate when they are preserved in their entirety.
Development was carried out using two RTX 4090s in a home basement. The user is currently running current models such as GLM-4.7, Qwen3.5, and MiniMax M2.5 on a dual GH200 configuration.
The code and the new model are scheduled to be released soon.
Source: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs（HN 495pt・126コメント） (HN Search (backfill), 2026-03-10)