Skip to main content
Search

# Parameter-Efficient Fine-Tuning

12 related articles

Subscribe to updates
  1. arXiv

    300 damaged buildings are enough: PUMA adapts a geospatial foundation model to map post-disaster damage

    Post-disaster building damage mapping is limited by labels: parameter-efficient fine-tuning adapts geospatial foundation models but still assumes large reliable annotation sets. PUMA recasts the task as positive-unlabeled learning, fine-tuning with a few confirmed damaged buildings and many unlabeled ones, while non-contrastive representation refinement and latent-state label disambiguation reinforce each other. On the xBD tsunami, hurricane, tornado and wildfire events, PUMA reaches the highest overall accuracy and F1 among the compared methods on tsunami, hurricane and wildfire, with F1 gains of 5.98 and 4.47 percentage points over the strongest competitor on tsunami and hurricane; on tornado it reaches 95.72% overall accuracy and 92.88% F1, competitive with the best NcPU.

  2. Low-rank adaptation learns one matrix: SingLoRA beats LoRA and LoRA+ on LLaMA-7B at 60% of their parameter budget

    Low-rank adaptation (LoRA) writes the weight update as a product of two small matrices, but their norms often differ sharply during training, producing imbalanced gradient flow and unstable optimization. SingLoRA instead writes the update as a single low-rank matrix times its own transpose, removing inter-matrix scale conflicts by construction and roughly halving the trainable parameters. Fine-tuning LLaMA-7B on MNLI, SingLoRA reaches 91.3% accuracy against 89.1% for LoRA and 90.2% for LoRA+, while using only 60% of their parameter budget.

  3. arXiv

    Weather downscaling no longer just tunes the backbone: PROLA splits a fixed rank budget between principal and orthogonal directions, cutting Z500 error to 50.58

    Mapping coarse-resolution history directly to future high-resolution weather trajectories requires forecasting, spatial downscaling and temporal densification at once, yet existing low-rank fine-tuning does not separate reusing pretrained principal directions from correcting outside them under a fixed rank budget. PROLA splits that budget into a principal branch and an orthogonal branch and rescales rank-1 optimizer updates with gradient signals; adapting Aurora on WeatherBench 2 with 3.92M trainable parameters, it reaches a 3-hour Z500 RMSE of 50.58, below AdaLoRA's 53.40.

  4. arXiv

    SEEM stumbles on one-pixel vessels: a 0.47M-parameter read-out adapter lifts 3 mm large-vessel Dice to 91.93%

    The interactive segmentation model SEEM performs poorly on OCTA retinal vessels out of the box, reaching only 75.83% Dice on large vessels with a frozen encoder, and every candidate mask it can produce is read from the same quarter-resolution feature maps, too coarse for vessels one to three pixels wide. The Read-out Basis Adapter (RBA) adds 32 half-resolution basis maps at the read-out, which each candidate accesses through a zero-initialised projection, for 0.47M trainable parameters. On OCTA-500 3 mm large vessels, averaged over three seeds, it reaches 91.93% Dice and 85.16% IoU, 1.72 and 2.90 points above Conv-LoRA, with gains concentrated on thin vessels and edges and no significant difference on the FAZ.

  5. arXiv

    SVC selects singular-vector channels to raise continual-learning performance while preserving pretrained ability across four LLM families

    The work defines each singular value's paired left and right singular vectors as a singular-vector channel, estimates each channel's adaptation benefit from current-task gradients and its forgetting cost from activation second moments on an unlabeled general-domain corpus, then selects trainable channels via knee-based cost screening, Pareto-front filtering, and Otsu thresholding; across four LLM families, eight TRACE tasks, and three task orders, SVC preserves pretrained general capabilities while achieving strong downstream continual-learning performance.

  6. arXiv

    AutoAdapt trains independent LoRA adapters via automatic latent domain discovery, matching an all-domain single adapter across 14 benchmarks while enabling retraining-free extension

    AutoAdapt is a modular instruction-tuning framework that first discovers latent domains with three methods (K-means, BERTopic, MNLI), then trains one LoRA adapter per domain independently and in parallel, and routes at inference with a parameter-free centroid mechanism; across 14 domain-specific benchmarks and GPT-4o pairwise judgements it reaches parity with a single LoRA adapter trained on all domains, while adding or updating a domain requires training only one adapter instead of full-model retraining.

  7. arXiv

    GPart maps a d-dimensional trainable vector directly into the full weight space via a sparse isometric partition matrix, matching or beating existing PEFT methods at ultra-low parameter budgets

    The work proposes GPart (Global Partition fine-tuning), a highly parameter-efficient fine-tuning method that maps a d-dimensional trainable vector directly into the full weight space through a sparse, isometric partition matrix, retains a fixed global parameter-sharing prior while removing the additional low-rank reconstruction used by LoRA-based methods, and thereby yields a single main hyperparameter (d), exact end-to-end isometry, and a minimal checkpoint representation consisting of the trainable vector and a random seed, matching or improving over existing PEFT methods at ultra-low parameter budgets across natural language understanding, computer vision, and mathematical reasoning benchmarks.

  8. arXiv

    SEA-PEFT raises mean Dice by 2.4–2.8 points in 1/5/10-shot 3D medical segmentation while training under 1% of parameters

    The authors propose SEA-PEFT, which treats adapter configuration as an online allocation problem solved during fine-tuning through a search–audit–allocate loop that trains active adapters, estimates each adapter's Dice utility by momentarily toggling it off, and reselects the active set under a parameter budget with a greedy knapsack allocator, stabilized by EMA+IQR smoothing and a finite-state machine; on TotalSegmentator and FLARE'22 it improves mean Dice by 2.4–2.8 points over the strongest fixed-topology PEFT baselines across 1/5/10-shot settings while training under 1% of parameters.

  9. arXiv

    VA-Adapter lets an ultrasound foundation model guide the echo probe, cutting trained parameters about 33-fold while lowering guidance error below strong baselines

    The work proposes a Vision-Action Adapter (VA-Adapter) inserted into the deep layers of a frozen ultrasound foundation model image encoder (EchoCLIP, USFM, BiomedCLIP) to online-inject understanding of individual 3D cardiac structure by encoding historical vision-action sequences; on a dataset of 178 adults, 356 expert scans and 1.31M image-action pairs, it reaches lower translation and rotation mean absolute error than strong probe-guidance baselines with roughly 2.61M-3.97M trainable parameters.

  10. Tsinghua survey on parameter-efficient fine-tuning: LoRA trains only 4.7M parameters on GPT-3, saving over 99.97% and slightly improving on full fine-tuning

    This survey by Tsinghua University's Knowledge Engineering Group systematically reviews parameter-efficient fine-tuning (PEFT) for foundation models (FMs), organizing methods into five categories—selective, additive, prompt, reparameterization, and hybrid—and reviewing their applications across five model structures (LLMs, vision foundation models, vision-language models, visual content generation models, and multimodal foundation models), while deriving three trend observations from Semantic Scholar citation counts (PEFT growing broadly; LLMs and VFMs dominating; MFMs relatively underexplored) and identifying reliability, interpretability, and unified benchmarks as future directions.