Skip to main content
Search

# Emergent Functional Specialization

1 related articles

Subscribe to updates
  1. arXiv

    Affective Adaptation in Multimodal Foundation Models: FFN Tuning Beats Attention Modules and Gate Projections Specialize Under Joint Optimization

    Analyzing 13 affective model instances spanning nine model designs, multiple scales, tasks, and training paradigms at the module level, with controlled functional analyses on representative models, the work finds a consistent yet non-exclusive functional organization in affective adaptation: under matched trainable-parameter budgets, tuning only the FFN consistently outperforms tuning only attention modules and nearly matches tuning all major Transformer projections, while gate, up, and down projections differentiate functionally after joint optimization, with gate_proj particularly prominent, motivating GET, which retains 96.2-98.0% of full-projection performance using only 19.3-24.5% as many trainable parameters.

You’re all caught up