SynMOE routes syntax labels into expert gating: LoRA and IA3 gain on code tasks while training about 1.90% of parameters
Parameter-efficient fine-tuning usually trails full fine-tuning on code tasks, partly because it relies on low-rank or sparsity assumptions without explicitly modeling code structure. SynMOE first extracts a syntax label for each leaf node of the abstract syntax tree, aligns those labels to the code language model's input tokens, and lets the labels take part in the routing decision of a mixture-of-experts gate. Across code search, code clone detection and line-level code completion with five code models, SynMOE improves both LoRA and IA3, with a maximum gain of 8.78%, and matches or exceeds full fine-tuning while training about 1.90% and 0.41% of parameters.