Recently, at AAAI 2025, a leading global artificial intelligence conference, a joint team from Ant Group, Zhejiang University, the University of Liverpool, and East China Normal University introduced a novel cross-domain fine-tuning framework—ScaleOT. This framework enhances privacy protection by 50% without compromising model performance. Compared to knowledge distillation techniques, it significantly reduces computational costs by 90%, providing a highly efficient and lightweight solution for cross-domain fine-tuning of large-scale models (with billions of parameters). The paper's innovation secured it an oral presentation at AAAI (only 4.6% of the nearly 13,000 submissions received oral presentations).
Cross-domain fine-tuning is a prevalent approach in the industry for protecting model intellectual property and data privacy. It involves lossy compression to convert large models into simulators. Data holders train adapters and return them to the large model for fine-tuning. Both data and models remain within their respective domains, ensuring privacy. However, limitations exist: firstly, the "uniformly extracting building blocks" approach can lead to the loss of crucial model layers, significantly degrading performance; secondly, using distillation techniques to compensate for performance loss is computationally expensive; and existing methods lack flexibility in privacy protection.
The Ant Group technical team explains that ScaleOT introduces three innovative approaches to balance model performance and privacy security. Firstly, it assesses the importance of intelligent layers in large models, using reinforcement learning to automatically identify key layers and dynamically retain "core layers" to minimize performance loss. Secondly, it "masks" the retained original layers to prevent attackers from restoring the original model, enhancing privacy protection with minimal performance impact. Thirdly, it allows flexible assembly based on different scenarios, enabling adjustable privacy strength.
Addressing data and model privacy security is crucial for the industrial application of large models, particularly in the financial sector. This innovative algorithm from Ant Group has been integrated into its Morse large model privacy protection product and is among the first in China to pass the trusted execution environment product special test of the China Academy of Information and Communications Technology (CAICT).
