Skip to main content
Search

# Preference Optimization

8 related articles

Subscribe to updates
  1. arXiv

    Reasoning models learn where to stop: LoCo cuts math completions by 41.7%–45.4% by training only the continue-or-conclude choice

    Large reasoning models often keep deliberating after enough information is available, raising decoding latency and memory cost. LoCo moves the compression decision to individual reasoning states: it finds candidate overthinking states in the model's own traces, freezes that prefix, regenerates shorter continuations whose answers verify as correct, and learns a continue-or-conclude preference with DPO. Across five math benchmarks, completion length drops 41.7%–45.4% while accuracy holds or improves; trained on math only, the same model cuts length by 23.4%–36.3% on three code benchmarks.

  2. arXiv

    Preference optimization gains a linguistic constraint: folding information uniformity into the reward brings model wording closer to human habits

    A preference optimization reward built on the psycholinguistic uniform information density hypothesis matches SimPO on AlpacaEval 2, Arena-Hard and MT-Bench, and moves English complement-clause that usage closer to natural language.

  3. arXiv

    Kuaishou and Nanjing University propose BPS: adding reverse preference pairs for boundary failures so the same response is rejected where wrong and chosen where right

    Kuaishou Technology and Nanjing University introduce Bidirectional Preference Synthesis (BPS): for each validated boundary failure, BPS keeps the conventional forward pair under the original prompt and adds a reverse pair under a synthesized achieved prompt, so the same response is rejected where it is wrong and chosen where it is right; on Qwen3-4B-Instruct-2507, BPS preserves original-side pairwise ranking while raising achieved-side ranking accuracy on held-out crossed anchors from 6.8% to 62.3%, without changing the DPO objective, training a reward model, or requiring online sampling.

  4. arXiv

    Suan builds the preference-optimization objective directly at the gradient level, achieving stronger safety alignment while preserving response utility across baselines and benchmarks

    The work introduces Suan, a new preference optimization algorithm whose objective is formulated directly at the gradient level, bypassing the standard variational derivation and yielding more interpretable and robust training dynamics; extensive evaluations across a diverse suite of competitive baselines and benchmarks show Suan achieves superior safety alignment while fully preserving response utility.

  5. arXiv

    ENPO uses adversarial policy exploration to remove iterative NLHF's exponential dependence on the inverse KL-regularization parameter, and consistently beats the evaluated RLHF and NLHF baselines on Llama-3-8B-Instruct

    This work studies online iterative Nash learning from human feedback (NLHF) and identifies exploration as a key obstacle: standard iterative NLHF can incur an exponential dependence on the inverse KL-regularization parameter, showing that implicit exploration through policy updates can be insufficient; it proposes ENPO, which combines SFT-type regularization with adversarial policy exploration to eliminate this exponential dependence without minimax oracles or explicit preference-model estimation, further introduces BENPO, which uses additional oracles to achieve an O(log T) regret bound, and develops DENPO, a practical variant of ENPO for fine-tuning LLMs; with Llama-3-8B-Instruct, DENPO shows consistent improvements over the evaluated RLHF and NLHF baselines across multiple benchmarks.

  6. arXiv

    First survey of post-training and alignment for video generation unifies four method families and reports dimension-wise cross-benchmark evidence

    This TMLR survey presents the first comprehensive review of post-training and alignment for video generation models, distinguishing implicit from explicit alignment by how alignment signals are enforced and organizing methods into supervised fine-tuning, self-training and distillation, preference- and reward-based methods, and inference-time methods, while reviewing datasets, benchmarks, and evaluation practices and discussing open challenges such as scalable reward design, long-horizon temporal consistency, stability-expressiveness trade-offs, and safety-aware generation.

  7. arXiv

    PsPLUG uses a "personalization residual" plug-in to curb style-instruction suppression of user preferences, outperforming baselines on LaMP

    The work identifies that explicit style instructions can erode the user-specific traits that personalized LLMs aim to preserve (a failure mode it calls personalization collapse), proposes modeling personalization as a distributional residual between the user's true linguistic distribution and the base model's neutral distribution under the same input, and builds PsPLUG: a lightweight plug-in that prepends a 3-token prefix (system instruction vector, user vector, input vector) to a frozen Qwen3-8B backbone, learns the user-specific residual with a style-conditioned Bradley–Terry preference objective, and tunes personalization strength at inference via a scaling coefficient; on the LaMP benchmark, PsPLUG generally outperforms non-personalized, RAG, PAG, PPlug, and OPPU baselines without styl

  8. arXiv

    ClinCoT pushes preference optimization from answer-level correction down to lesion-region reasoning: it beats MMedPO and other baselines on most metrics across SLAKE, VQA-RAD and IU-Xray, with more consistent gains after SFT initialization

    The work proposes ClinCoT, a clinical-aware visual chain-of-thought framework: it uses hypotheses-driven region proposals (disease-conditioned activation maps from a clinical-aware tool such as MedKLIP, thresholded into candidate regions), has the target Med-VLM generate multiple region-conditioned reasoning chains by jointly processing the original image and each candidate region, scores them with two Med-LLM evaluators that combine current-step score with next-step influence and penalize evaluator disagreement, and trains with a score-margin-aware DPO plus iterative regeneration of preference data; across SLAKE, VQA-RAD and IU-Xray it outperforms DPO, Self-Rewarding, STLLaVA-Med, POVID, SIMA, FiSAO and MMedPO on most metrics, is strongest on report generation, falls below MMedPO on VQA-R

You’re all caught up