Model Level: Privacy-Preserving Fine-Tuning

Model Level: Private and Decentralized Fine-Tuning of Large Biomedical Models

Modern biomedical research increasingly relies on large pre-trained models, particularly transformer-based architectures such as BioBERT and ClinicalBERT. These models are trained on biomedical literature, clinical notes, and genomic datasets (Lee et al., 2020; Alsentzer et al., 2019).

Fine-tuning these models for downstream tasks—such as clinical decision support, genomic prediction, or drug-target interaction—requires institution-specific datasets that are often sensitive or governed by strict privacy policies (Gu et al., 2021; Rasmy et al., 2021; Wang et al., 2023). Pooling data across hospitals or research groups remains infeasible due to regulatory and ethical concerns, limiting the practical impact of these models (Kim et al., 2021; Berger et al., 2019).

Examples

• Clinical Decision Support Systems Hospitals adapt models like ClinicalBERT using local EHRs. However, cross-hospital data integration is blocked by privacy laws, limiting generalization and robustness (Alsentzer et al., 2019; Kim et al., 2021).

• Genomic and Precision Medicine Personalized genomic models must be fine-tuned using high-dimensional, sensitive data. Re-identification risks and ethical constraints hinder collaboration and scale (Cho et al., 2022).

• Drug Discovery and Development Joint efforts between pharmaceutical companies and academic labs are rare due to concerns about IP leakage and competitive advantage (Gu et al., 2021).

Limitations of Existing Approaches

Several decentralized training approaches have been explored to mitigate privacy concerns, but all present critical limitations:

• Federated Learning (FL) FL avoids raw data sharing but is vulnerable to gradient inversion attacks, where adversaries reconstruct sensitive inputs from shared updates—especially dangerous with large transformer gradients (Zhu et al., 2019; Berger et al., 2019).

• Trusted Execution Environments (TEEs) Platforms like Intel SGX offer isolated execution, but are limited by small memory footprints, poor scalability, and susceptibility to side-channel attacks (Costan & Devadas, 2016; Lee et al., 2020 [Occlum]).

• Homomorphic Encryption (HE) HE allows privacy-preserving computation but introduces severe computational overhead, especially in nonlinear, high-dimensional models like transformers (Gilad-Bachrach et al., 2016; Brutzkus et al., 2019).

These barriers underscore the urgent need for a scalable, cryptographically robust method for fine-tuning biomedical foundation models across institutional boundaries—without compromising privacy, utility, or efficiency.

---

References

• Alsentzer, E., et al. (2019). Publicly available ClinicalBERT embeddings. Clinical NLP Workshop.

• Berger, B., et al. (2019). Federated learning in biomedical research.

• Brutzkus, A., et al. (2019). Low latency privacy-preserving inference. ICML.

• Cho, H., et al. (2022). Privacy-preserving genomic analysis. Annual Review of Biomedical Data Science.

• Costan, V., & Devadas, S. (2016). Intel SGX explained. IACR ePrint Archive.

• Gilad-Bachrach, R., et al. (2016). CryptoNets: Applying neural networks to encrypted data. ICML.

• Gu, Y., et al. (2021). Domain-specific language model pretraining for biomedical NLP. ACM Transactions on Computing for Healthcare.

• Kim, M., et al. (2021). Privacy-preserving federated learning in medicine. JAMIA.

• Lee, J., et al. (2020). BioBERT: A pre-trained biomedical language model. Bioinformatics.

• Lee, Y., et al. (2020). Occlum: Secure and efficient multitasking in SGX. ASPLOS.

• Rasmy, L., et al. (2021). Med-BERT: Contextualized embeddings for structured EHRs. NPJ Digital Medicine.

• Wang, S., et al. (2023). Large language models encode clinical knowledge. Nature.

• Zhu, L., et al. (2019). Deep leakage from gradients. NeurIPS.