> For the complete documentation index, see [llms.txt](https://rexis.gitbook.io/rexis/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://rexis.gitbook.io/rexis/model-level-privacy-preserving-fine-tuning.md).

# Model Level: Privacy-Preserving Fine-Tuning

### Model Level: Private and Decentralized Fine-Tuning of Large Biomedical Models

Modern biomedical research increasingly relies on large pre-trained models, particularly transformer-based architectures such as BioBERT and ClinicalBERT. These models are trained on biomedical literature, clinical notes, and genomic datasets (Lee et al., 2020; Alsentzer et al., 2019).

Fine-tuning these models for downstream tasks—such as clinical decision support, genomic prediction, or drug-target interaction—requires institution-specific datasets that are often sensitive or governed by strict privacy policies (Gu et al., 2021; Rasmy et al., 2021; Wang et al., 2023). Pooling data across hospitals or research groups remains infeasible due to regulatory and ethical concerns, limiting the practical impact of these models (Kim et al., 2021; Berger et al., 2019).

#### Examples

* **Clinical Decision Support Systems**\
  Hospitals adapt models like ClinicalBERT using local EHRs. However, cross-hospital data integration is blocked by privacy laws, limiting generalization and robustness (Alsentzer et al., 2019; Kim et al., 2021).
* **Genomic and Precision Medicine**\
  Personalized genomic models must be fine-tuned using high-dimensional, sensitive data. Re-identification risks and ethical constraints hinder collaboration and scale (Cho et al., 2022).
* **Drug Discovery and Development**\
  Joint efforts between pharmaceutical companies and academic labs are rare due to concerns about IP leakage and competitive advantage (Gu et al., 2021).

#### Limitations of Existing Approaches

Several decentralized training approaches have been explored to mitigate privacy concerns, but all present critical limitations:

* **Federated Learning (FL)**\
  FL avoids raw data sharing but is vulnerable to *gradient inversion attacks*, where adversaries reconstruct sensitive inputs from shared updates—especially dangerous with large transformer gradients (Zhu et al., 2019; Berger et al., 2019).
* **Trusted Execution Environments (TEEs)**\
  Platforms like Intel SGX offer isolated execution, but are limited by small memory footprints, poor scalability, and susceptibility to side-channel attacks (Costan & Devadas, 2016; Lee et al., 2020 \[Occlum]).
* **Homomorphic Encryption (HE)**\
  HE allows privacy-preserving computation but introduces severe computational overhead, especially in nonlinear, high-dimensional models like transformers (Gilad-Bachrach et al., 2016; Brutzkus et al., 2019).

These barriers underscore the urgent need for a scalable, cryptographically robust method for fine-tuning biomedical foundation models across institutional boundaries—*without compromising privacy, utility, or efficiency*.

***

#### References

* Alsentzer, E., et al. (2019). Publicly available ClinicalBERT embeddings. *Clinical NLP Workshop*.
* Berger, B., et al. (2019). Federated learning in biomedical research.
* Brutzkus, A., et al. (2019). Low latency privacy-preserving inference. *ICML*.
* Cho, H., et al. (2022). Privacy-preserving genomic analysis. *Annual Review of Biomedical Data Science*.
* Costan, V., & Devadas, S. (2016). Intel SGX explained. *IACR ePrint Archive*.
* Gilad-Bachrach, R., et al. (2016). CryptoNets: Applying neural networks to encrypted data. *ICML*.
* Gu, Y., et al. (2021). Domain-specific language model pretraining for biomedical NLP. *ACM Transactions on Computing for Healthcare*.
* Kim, M., et al. (2021). Privacy-preserving federated learning in medicine. *JAMIA*.
* Lee, J., et al. (2020). BioBERT: A pre-trained biomedical language model. *Bioinformatics*.
* Lee, Y., et al. (2020). Occlum: Secure and efficient multitasking in SGX. *ASPLOS*.
* Rasmy, L., et al. (2021). Med-BERT: Contextualized embeddings for structured EHRs. *NPJ Digital Medicine*.
* Wang, S., et al. (2023). Large language models encode clinical knowledge. *Nature*.
* Zhu, L., et al. (2019). Deep leakage from gradients. *NeurIPS*.