Author(s)

MeiLin Cheng

Affiliation(s)

Department of Computer Science, City University of Hong Kong, Hong Kong Region, China.

Corresponding Author

MeiLin Cheng

ABSTRACT

Clinical text classification is a critical task in medical informatics, enabling applications such as automated diagnosis coding, patient outcome prediction, and adverse event detection. However, the scarcity of labeled medical data and the domain-specific language used in clinical records pose significant challenges. This paper proposes a transfer learning framework that leverages pretrained language models—specifically BioBERT and ClinicalBERT—for clinical text classification tasks. The framework incorporates a domain-adaptive fine-tuning strategy and task-specific adaptation layer to bridge the gap between general language understanding and specialized medical text. Experimental results on benchmark clinical datasets demonstrate substantial improvements in classification accuracy, F1-score, and robustness compared to traditional supervised learning approaches.

KEYWORDS

Clinical text classification; Transfer learning; Pretrained language models; BioBERT; ClinicalBERT; Natural language processing; Electronic health records

CITE THIS PAPER

MeiLin Cheng. A transfer learning framework for clinical text classification using pretrained language models. AI and Data Science Journal. 2025, 2(2): 1-6. DOI: https://doi.org/10.61784/adsj3018.

REFERENCES

[1] Doppalapudi S, Wang T, Qiu R. Transforming unstructured digital clinical notes for improved health literacy. Digital Transformation and Society, 2022, 1(1): 9-28.

[2] Arowoogun J O, Babawarun O, Chidi R, et al. A comprehensive review of data analytics in healthcare management: Leveraging big data for decision-making. World Journal of Advanced Research and Reviews, 2024, 21(2): 1810-1821.

[3] Wu B, Qiu S, Liu W. Addressing Sensor Data Heterogeneity and Sample Imbalance: A Transformer-Based Approach for Battery Degradation Prediction in Electric Vehicles. Sensors, 2025, 25(11): 3564.

[4] Sheikhalishahi S, Miotto R, Dudley J T, et al. Natural language processing of clinical notes on chronic diseases: systematic review. JMIR medical informatics, 2029, 7(2): e12239.

[5] Li P, Ren S, Zhang Q, et al. Think4SCND: Reinforcement Learning with Thinking Model for Dynamic Supply Chain Network Design. IEEE Access, 2024.

[6] Guo L, Hu X, Liu W, et al. Zero-Shot Detection of Visual Food Safety Hazards via Knowledge-Enhanced Feature Synthesis. Applied Sciences, 2025, 15(11): 6338.

[7] AlShuweihi M, Salloum S A, Shaalan K. Biomedical corpora and natural language processing on clinical text in languages other than English: a systematic review. Recent advances in intelligent systems and smart applications, 2022: 491-509.

[8] Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR medical informatics, 2020, 8(3): e17984.

[9] Mars M. From word embeddings to pre-trained language models: A state-of-the-art walkthrough. Applied Sciences, 2022, 12(17): 8805.

[10] Nazi Z A, Peng W. Large language models in healthcare and medical domain: A review. MDPI, 2024, 11(3): 57.

[11] Naseem U, Dunn A G, Khushi M, et al. Benchmarking for biomedical natural language processing tasks with a domain specific ALBERT. BMC bioinformatics, 2022, 23(1): 144.

[12] Laparra E, Mascio A, Velupillai S, et al. A review of recent work in transfer learning and domain adaptation for natural language processing of electronic health records. Yearbook of medical informatics, 2021, 30(01): 239-244.

[13] Yang Y, Wang M, Wang J, et al. Multi-Agent Deep Reinforcement Learning for Integrated Demand Forecasting and Inventory Optimization in Sensor-Enabled Retail Supply Chains. Sensors (Basel, Switzerland), 2025, 25(8): 2428.

[14] Wang J, Zhang H, Wu B, et al. Symmetry-Guided Electric Vehicles Energy Consumption Optimization Based on Driver Behavior and Environmental Factors: A Reinforcement Learning Approach. Symmetry, 2025.

[15] Abdollahi M. Improving Medical Document Classification via Feature Engineering (Doctoral dissertation, Open Access Te Herenga Waka-Victoria University of Wellington). 2024.

[16] Aydo?an M. Adaptive Contextual Embeddings for Detecting Social Determinants of Health in Patient Narratives. Applied Science, Engineering, and Technology Review: Innovations, Applications, and Directions, 2024, 14(10): 27-41.

[17] Banerjee I, Ling Y, Chen M C, et al. Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification. Artificial intelligence in medicine, 2019, 97: 79-88.

[18] Willemink M J, Koszek W A, Hardell C, et al. Preparing medical imaging data for machine learning. Radiology, 2020, 295(1): 4-15.

[19] Zhang Q, Chen S, Liu W. Balanced Knowledge Transfer in MTTL-ClinicalBERT: A Symmetrical Multi-Task Learning Framework for Clinical Text Classification. Symmetry, 2025, 17(6): 823.

[20] Min B, Ross H, Sulem E, et al. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 2023, 56(2): 1-40.

[21] Aharoni R, Goldberg Y. Unsupervised domain clusters in pretrained language models. arXiv preprint arXiv: 2004.02105, 2020.

[22] Wankhade M, Rao A C S, Kulkarni C. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 2020, 55(7): 5731-5780.

[23] Buonocore T M, Crema C, Redolfi A, et al. Localizing in-domain adaptation of transformer-based biomedical language models. Journal of Biomedical Informatics, 2023, 144: 104431.

[24] Ahmed T, Aziz M M A, Mohammed N. De-identification of electronic health record using neural network. Scientific reports, 2020, 10(1): 18600.

[25] Wang Y. Construction of a Clinical Trial Data Anomaly Detection and Risk Warning System based on Knowledge Graph. In Forum on Research and Innovation Management, 2023, 3(6).

[26] Laparra E, Mascio A, Velupillai S, Miller T. A review of recent work in transfer learning and domain adaptation for natural language processing of electronic health records. Yearbook of medical informatics, 2021, 30(01): 239-244.

[27] Laparra E, Mascio A, Velupillai S, et al. A review of recent work in transfer learning and domain adaptation for natural language processing of electronic health records. Yearbook of medical informatics, 2021, 30(01): 239-244.

[28] Gillioz A, Casas J, Mugellini E, et al. Overview of the Transformer-based Models for NLP Tasks. In 2020 15th Conference on computer science and information systems (FedCSIS).IEEE, 2020: 179-183.

[29] Xing S, Wang Y, Liu W. Multi-Dimensional Anomaly Detection and Fault Localization in Microservice Architectures: A Dual-Channel Deep Learning Approach with Causal Inference for Intelligent Sensing. Sensors, 2025.

[30] Wang Y. RAGNet: Transformer-GNN-Enhanced Cox–Logistic Hybrid Model for Rheumatoid Arthritis Risk Prediction. 2025.

[31] Hosna A, Merry E, Gyalmo J, et al. Transfer learning: a friendly introduction. Journal of Big Data, 2022, 9(1): 102.

[32] Abdullah T A, Zahid M S M, Ali W. A review of interpretable ML in healthcare: taxonomy, applications, challenges, and future directions. Symmetry, 2021, 13(12): 2439.

[33] Tan Y, Wu B, Cao J, Jiang B. LLaMA-UTP: Knowledge-Guided Expert Mixture for Analyzing Uncertain Tax Positions. IEEE Access, 2025.

[34] Jin J, Xing S, Ji E, Liu W. XGate: Explainable Reinforcement Learning for Transparent and Trustworthy API Traffic Management in IoT Sensor Networks. Sensors (Basel, Switzerland), 2025, 25(7): 2183.