Skip to main content
Search

# Low-Rank Adaptation

26 related articles

Subscribe to updates
  1. arXiv

    AUROC alone picks the wrong warning probe: Prewarn scores alarms against the onset, and two models warn before most loops

    Language-model failures often begin at a specific token, yet the usual response-level AUROC asks only whether a finished response failed, not when an alarm fired. Prewarn fixes each monitor's threshold on validation data at an error rate chosen for the application, then scores classification, detection and early warning separately against an annotated onset. On repetitive degeneration, a length-only rule reaches near-perfect AUROC but can never warn; frozen probes warn before the onset on most degenerate responses of Apertus and Llama, while on Mistral they mostly fire after it.

  2. Multimodal trackers stop correcting two streams in parallel: CCLT lets the second correction read the freshly updated state, training 0.212% of parameters

    RGB-X tracking pairs visible frames with thermal, depth or event data, but paired annotations are far scarcer than RGB pretraining data, so only a small set of parameters is usually tuned. Earlier bidirectional cross-modal corrections computed both directions side by side from uncorrected input states, so the second correction could not use the first one's result. CCLT inserts low-rank exchange modules at blocks 6 and 10 of a frozen backbone, letting one direction correct first and the other read the freshly corrected state; against a parameter-matched parallel version, mean LasHeR SR rises by 1.980 points (95% interval [0.360, 2.690]), with 196.6K trainable parameters, 0.212% of the 92.7M-parameter tracker.

  3. Low-rank adaptation learns one matrix: SingLoRA beats LoRA and LoRA+ on LLaMA-7B at 60% of their parameter budget

    Low-rank adaptation (LoRA) writes the weight update as a product of two small matrices, but their norms often differ sharply during training, producing imbalanced gradient flow and unstable optimization. SingLoRA instead writes the update as a single low-rank matrix times its own transpose, removing inter-matrix scale conflicts by construction and roughly halving the trainable parameters. Fine-tuning LLaMA-7B on MNLI, SingLoRA reaches 91.3% accuracy against 89.1% for LoRA and 90.2% for LoRA+, while using only 60% of their parameter budget.

  4. arXiv

    Frozen pathology foundation models stumble on single-cell gene expression from H&E, while lightweight adaptation lifts 5K-panel correlation from 0.065 to 0.174

    Frozen pathology foundation-model embeddings transfer poorly to single-cell spatial gene expression prediction from H&E: on the 5K gene panel, mean per-gene Pearson correlation falls from 0.375 at pseudo-spot resolution to 0.065 at single-cell resolution. A set of single-cell-oriented lightweight adaptations, namely cell-localized token pooling, 40x field-of-view inputs, Poisson count supervision and LoRA fine-tuning, raises the same panel to 0.174 and beats SpatialEx on the 5K panel with AUROC 0.800 versus 0.642. Gains concentrate in spatially variable and disease-relevant genes, while absolute correlation across the full panel stays low.

  5. bioRxiv

    TriStageHLA splits epitope discovery into three stages, lifting immunogenicity AUROC from 60.1% to 81.4%

    A team including City University of Hong Kong released TriStageHLA, which splits HLA class I epitope discovery into peptide-only screening, allele-specific binding prediction and immunogenicity prediction in sequence. On held-out immunogenicity data the immunogenicity module reached AUROC 81.4% and AUPRC 73.8%, above ImmuneApp at 70.7%/63.6% and BigMHC-IM at 60.1%/51.5%; the binding module reached AUROC 96.2% and AUPRC 95.2% on combined BA/EL evaluation. In a simulated screen over 300,000 human-proteome background peptides, the screening stage kept 82.4% of true candidate peptides while discarding 87% of background peptides.

  6. Mixed-precision quantization stops treating each layer as one bit width: A2BQ assigns bits per block by attention sensitivity, reaching 7.884 perplexity on TinyLlama

    LLM quantization usually gives an entire layer one bit width, ignoring that weight blocks within a layer differ in sensitivity to quantization noise. A2BQ scores each 128-row weight block by post-softmax attention variance from an FP16 model on a WikiText-2 calibration set, allocates 4/6/8 bits under a 5.7-bit average budget, and sets qLoRA+ LoRA ranks from layer sensitivity and assigned precision. On TinyLlama-1.1B, the attention-scope configuration with three protected layers at each boundary reaches 7.884 test perplexity at 73.11 tokens/s, while runtime GPU memory stays near FP16 because the reference backend dequantizes weights into floating-point buffers before computing.

  7. arXiv

    Diffusion language models without pretraining from scratch: LoRA-only conversion turns four autoregressive models diffusive in hours on one GPU

    BYOD is a LoRA-only recipe for converting autoregressive language models into diffusion language models: every base parameter stays frozen, each layer's causal mask becomes bidirectional, and a masked-diffusion objective trains at most 102.4M tokens in 3.1–4.0 hours on one Google Colab GPU, turning Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B into models that generate from fully masked sequences, with adapters at 4.1–5.8% of final parameters. The strongest conversion, BYOD-Gemma, reaches 50.6% average accuracy on nine tasks against 51.9% for LLaDA 8B Instruct in the same harness; open-generation quality at low NFE budgets remains the main limitation.

  8. Buildings stop merging forecasts blindly: a local cross-fit gate avoids the 3% negative transfer that unconditional federation caused on Electricity

    Distributed building energy management depends on forecasts, yet federated forecasting is usually scored on error without testing the decisions those forecasts support. UtilityGossip keeps a private and a collaborative low-rank adapter at each node and accepts a peer adapter only when randomized four-fold cross-fitting on local held-out data shows a meaningful error reduction. On Electricity, unconditional federation raised mean squared error more than 3% above local training while the gated method avoided that degradation; on Solar-Energy at the 96-step horizon it lowered mean squared error by 1.59% relative to local training using half the communication of federated averaging.

  9. A paraphrase shouldn't flip a refusal: attribution consistency cuts a small tool agent's flip rate from 11.3% to 9.3%

    Tool-using language model agents can refuse one phrasing of a harmful request and comply with another, and a missed refusal becomes an action. Llama-3.2-1B-Instruct was fine-tuned with LoRA on data not drawn from AgentHarm, comparing latent attribution regularization (LAR) against an augmentation-only control over three seeds. At 10% over-refusal on harmless tool tasks, LAR lowered the share of tasks with mixed refusal decisions from 25% to 19% and paraphrase-induced flips from 11.3% to 9.3%, while harmful-versus-harmless detection did not change significantly.

  10. Patient-safety reasoning without the knowledge graph loses 0.047 F1, while retrieval's gain straddles zero

    Patient-safety reasoning requires linking clinical narratives, medications, laboratory findings and adverse events, and a language model alone can produce plausible but unsupported conclusions. The same MIMIC-III/IV cases ran under four conditions with one LLaMA 2 7B model and one prompt: model only, plus retrieval-augmented generation, plus a patient-safety knowledge graph and ontology, and both. The full framework reached F1 0.89 for concept extraction, 0.046 above baseline; removing the knowledge graph cost the most, with F1 0.047, AUROC 0.084 and AUPRC 0.110, while removing retrieval gave an F1 difference of 0.029 whose interval crossed zero.