SynMOE routes syntax labels into expert gating: LoRA and IA3 gain on code tasks while training about 1.90% of parameters
Synopsis
Parameter-efficient fine-tuning usually trails full fine-tuning on code tasks, partly because it relies on low-rank or sparsity assumptions without explicitly modeling code structure. SynMOE first extracts a syntax label for each leaf node of the abstract syntax tree, aligns those labels to the code language model's input tokens, and lets the labels take part in the routing decision of a mixture-of-experts gate. Across code search, code clone detection and line-level code completion with five code models, SynMOE improves both LoRA and IA3, with a maximum gain of 8.78%, and matches or exceeds full fine-tuning while training about 1.90% and 0.41% of parameters.
Interpretation
Parameter-efficient fine-tuning for code language models gains an explicit syntax channel: every input token carries a syntax label taken from the abstract syntax tree, and the label participates in mixture-of-experts routing instead of leaving token roles to implicit attention. Earlier parameter-efficient methods rest on assumptions that parameter updates are low-rank or sparse, without explicit modeling of code-related information, and generally fall below full fine-tuning on code intelligence tasks. On code search, code clone detection and line-level code completion with five code models, SynMOE combined with LoRA and IA3 improves consistently over the corresponding baselines, with a maximum gain of 8.78%, and 15 of the metric comparisons pass significance testing, the exception being SynMOE with IA3 on Deepseek-Coder.
The procedure has two steps: parse source code into an abstract syntax tree and label each leaf node, then let those labels shape a weighted combination of lightweight experts. Labels do not line up naturally with the language model's tokenization, since special tokens, subword splits and comments all create mismatches, so a left-to-right two-pointer alignment propagates each leaf label onto its input tokens. Aligned labels are encoded into high-dimensional vectors and mixed with the gating network's scores through a hyperparameter lambda to set expert weights; experiments fix the expert count at 4 and reduce the rank of LoRA experts to control the added trainable parameters relative to the single module.
Syntax information is not added directly to token embeddings but routed through the gating network to influence expert selection, so the token's original contextual representation is not overwritten. Summing syntax-label embeddings onto code tokens is the simpler integration route, but the labels can overwhelm the code token's own representation. Ablations show that removing syntax extraction or collapsing the mixture of experts to a single expert both degrade performance, with the single-expert collapse costing more, indicating that the routing mechanism carries a key role.
Perspective
Validation covers code search, code clone detection and line-level code completion, with CodeBERT, UniXcoder, CodeGPT, Deepseek-Coder and Qwen2.5-Coder as base models up to about 1.5 billion parameters; the efficiency comparison measures per-epoch training time and memory footprint for CodeGPT on line-level code completion.
Tasks such as code translation and program repair are not evaluated, and behavior on larger models remains unexamined; the effect of the hyperparameter lambda is non-monotonic, its best value differs across base models and parameter-efficient methods, and performance drops sharply as it approaches 1.0.
