TY  - EJOU
AU  - Sun, Jiakang 
AU  - Chen, Ke 
AU  - He, Xinyang 
AU  - Liu, Xu 
AU  - Li, Ke 
AU  - Peng, Cheng 

TI  - UniTrans: Unified Parameter-Efficient Transfer Learning and Multimodal Alignment for Large Multimodal Foundation Model
T2  - Computers, Materials \& Continua

PY  - 2025
VL  - 83
IS  - 1
SN  - 1546-2226

AB  - With the advancements in parameter-efficient transfer learning techniques, it has become feasible to leverage large pre-trained language models for downstream tasks under low-cost and low-resource conditions. However, applying this technique to multimodal knowledge transfer introduces a significant challenge: ensuring alignment across modalities while minimizing the number of additional parameters required for downstream task adaptation. This paper introduces UniTrans, a framework aimed at facilitating efficient knowledge transfer across multiple modalities. UniTrans leverages Vector-based Cross-modal Random Matrix Adaptation to enable fine-tuning with minimal parameter overhead. To further enhance modality alignment, we introduce two key components: the Multimodal Consistency Alignment Module and the Query-Augmentation Side Network, specifically optimized for scenarios with extremely limited trainable parameters. Extensive evaluations on various cross-modal downstream tasks demonstrate that our approach surpasses state-of-the-art methods while using just 5% of their trainable parameters. Additionally, it achieves superior performance compared to fully fine-tuned models on certain benchmarks.
KW  - Parameter-efficient transfer learning; multimodal alignment; image captioning; image-text retrieval; visual question answering

DO  - 10.32604/cmc.2025.059745