Research on indicators for evaluating the quality of real-world data

Published on Jan. 28, 2026Total Views: 102 times Total Downloads: 38 times Download Mobile

Author: JIE Wan ^1, ^2, ³ YAO Minghong ^1, ^2, ³ ZHANG Jun ⁴ WANG Mingqi ^1, ^2, ³ JIA Yulong ^1, ^2, ³ LIU Yanmei ^1, ^2, ³ WANG Yuning ^1, ^2, ³ Larry Z. LIU ^5, ⁶ ZOU Kang ^1, ^2, ³  SUN Xin ^1, ^2, ³

Affiliation: 1. Hainan Lecheng Institute of Real-World Study, Qionghai 571400, Hainan Province, China 2. Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, Chengdu 610041, China 3. Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China 4. Department of Medical Affairs, MSD R&D (China) Co., Ltd., Beijing 100012, China 5. V&I Outcomes Research, Merck & Co., Inc., Rahway, NJ 07065, USA 6. Weill Cornell Medical College, New York, NY 10021, USA

Keywords: Real world study Real world data Quality evaluation Data management Data governance

DOI: 10.12173/j.issn.1005-0698.202411008

Reference: JIE Wan, YAO Minghong, ZHANG Jun, WANG Mingqi, JIA Yulong, LIU Yanme, WANG Yuning, Larry Z. LIU, ZOU Kang, SUN Xin. Research on indicators for evaluating the quality of real-world data[J]. Yaowu Liuxingbingxue Zazhi, 2026, 35(1): 62-74. DOI: 10.12173/j.issn.1005-0698.202411008.[Article in Chinese]  Copied

Abstract
Full-text
References

Abstract

Objective To develop an index system for the quantitative evaluation of real-world data quality.

Methods Computerized searches were conducted across literature databases, including PubMed, Europe PMC, WanFang Data, and CNKI, as well as the official websites of 13 international academic organizations and regulatory agencies. Based on predefined inclusion and exclusion criteria, literature, guidelines, and standards related to real-world data (RWD) quality assessment, were ultimately selected. Through thematic induction and content summarization, core elements from the selected documents and guidelines were extracted to construct an initial set of indicators. These indicators were subsequently revised and refined using expert consultation methods.

Results A total of 35 publications were included. Incorporating the research team’s practical experience, this paper explored data management processes for prospective data and data governance processes for retrospective data. Corresponding data quality evaluation indicators were established accordingly. In both frameworks, data quality was categorized into three primary dimensions: raw data quality, process quality, and outcome quality. These dimensions were further organized according to the chronological sequence of clinical research. For the data management component, 15 secondary indicators and 43 tertiary indicators were developed. For the data governance component, 13 secondary indicators and 29 tertiary indicators were formulated.

Conclusions The evaluation indicators developed through literature analysis and expert consultation method demonstrate a considerable degree of scientific rigor and feasibility. They can serve as valuable references for regulatory authorities, sponsors, and researchers involved in the quality assessment of real-world data.

Full-text

Please download the PDF version to read the full text: download

References

1.U.S. FDA. Real-world evidence[EB/OL]. (2023-05-10) [2023-09-20]. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence.

2.国家药监局药审中心. 关于发布《用于产生真实世界证据的真实世界数据指导原则（试行）》的通告（2021年第27号）[EB/OL]. (2021-04-13) [2023-10-19]. https://www.cde.org.cn/main/news/viewInfoCommon/2a1c437ed54e7b838a7e86f4ac21c539.

3.孙鑫, 谭婧, 王雯, 等. 建立真实世界数据与研究技术规范, 促进中国真实世界证据的生产与使用[J]. 中国循证医学杂志, 2019, 19(7): 755-762. [Sun X, Tan J, Wang W, et al. Developing technical guidance for real-world data and studies to achieve better production and use of real-world evidence in China[J]. Chinese Journal of Evidence-Based Medicine, 2019, 19(7): 755-762.] https://kns.cnki.net/kcms2/article/abstract?v=IU01-AKyIMB5ageovNFcG7GldflPMMCR12QD1DBq8vV20k6-jORe66EAdVrzMg0oeuH81NrWjTNP2pN379E8NlFm61drLnc41Z1aFC3ZZ59U7wNIUmH33BBdZ4ehh3Zd9Le9FyEpEzuGvJatzRGRztVBbSb86-EbjamZtaCzRalb2xj6jgy-mw==&uniplatform=NZKPT&language=CHS.

4.国家药品监督管理局. 关于发布真实世界证据支持药物研发与审评的指导原则（试行）的通告（2020年第1号）[EB/OL]. (2020-01-03) [2023-12-05]. https://www.nmpa.gov.cn/xxgk/ggtg/ypggtg/ypqtggtg/20200107151901190.html.

5.Knatterud GL, Rockhold FW, George SL, et al. Guidelines for quality assurance in multicenter trials: a position paper[J]. Control Clin Trials, 1998, 19(5): 477-493. DOI: 10.1016/s0197-2456(98)00033-6.

6.European Medicines Agency. High-quality data to empower data-driven medicines regulation in the European Union-Medicines Agency[EB/OL]. (2022-10-10) [2023-09-20]. https://www.ema.europa.eu/en/news/high-quality-data-empower-data-driven-medicines-regulation-european-union.

7.Shang N, Wend C, Hripcsak G. A conceptual framework for evaluating data suitability for observational studies[J]. J Am Med Inform Assoc, 2018, 25(3): 248-258. DOI: 10.1093/jamia/ocx095.

8.姚晨. 利用好真实世界数据生产高质量真实世界证据支持药械监管[J]. 中国食品药品监管, 2020(2): 22-27. [Yao C. Use real world data to produce high-quality real world evidence to support drug device regulation[J]. China Food & Drug Administration Magazine, 2020, (2): 22-27.] DOI: 10.3969/j.issn.1673-5390.2020.02.

9.贾玉龙, 姚明宏, 刘艳梅, 等. 特许医疗政策下真实世界数据治理的模式探索[J]. 中国循证医学杂志, 2021, 21(12): 1373-1380. [Jia YL, Yao MH, Liu YM, et al. Exploring the patterns of real-world data governance in the context of special healthcare policy[J]. Chinese Journal of Evidence-Based Medicine, 2021, 21(12): 1373-1380.] DOI: 10.7507/1672-2531.202108147.

10.谭婧, 程亮亮, 王雯, 等. 患者登记研究的策划与患者登记数据库构建: 基于观察性设计的真实世界研究[J]. 中国循证医学杂志, 2018, 17(12): 1365-1372. [Tan J, Cheng LL, Wang W, et al. Real world study using observational designs: plan for patient registry study and development of registry database[J]. Chinese Journal of Evidence-Based Medicine, 2018, 17(12): 1365-1372.] https://kns.cnki.net/kcms2/article/abstract?v=IU01-AKyIMDw_SXDvj-SY7maiuhSjVKP8jsvFdTmtkmfhFG7NMbzWVkuKFfDMkiA5Nadh0Hw6EQp9q6shy-v2kLAy1VNW_IEwtjbqjB2ijf5CsedKnOWTgcq0oJGSHMCVoj4KKezhUPrWiLbbF2pmhPsDMapTJPVoe7xRzpqF0uTGpjonG20EA==&uniplatform=NZKPT&language=CHS.

11.国家药品监督管理局. 关于发布临床试验的电子数据采集技术指导原则的通告（2016年第114号）[EB/OL]. (2016-07-27) [2023-09-22]. https://www.nmpa.gov.cn/xxgk/ggtg/ypggtg/ypqtggtg/20160729184001958.html.

12.国家药监局药审中心. 关于发布《药物临床试验数据递交指导原则（试行）》的通告（2020年第16号）[EB/OL]. (2020-07-20) [2023-09-22]. https://www.nmpa.gov.cn/xxgk/ggtg/ypggtg/ypqtggtg/20200720171201514.html.

13.刘艳梅, 任燕, 贾玉龙, 等. 真实世界数据体系构建的模式探索[J]. 中华流行病学杂志, 2022, 43(3): 418-423. [Liu YM, Ren Y, Jia YL, et al. Exploration of construction mode of real-world data system[J]. Chinese Journal of Epidemiology, 2022, 43(3): 418-423.] DOI: 10.3760/cma.j.cn112338-20210824-00675.

14.赵国桢, 闫世艳, 郭玉红, 等. 基于既有医疗数据构建研究型数据库的方法学探讨及实例解读（二）: 数据治理的方法[J]. 中国中医药信息杂志, 2023, 30(9): 17-21. [Zhao GZ, Yan SY, Guo YH, et al. Methodological discussion and case interpretation of construction research database based on existing medical data (Ⅱ): methods of data governance[J]. Chinese Journal of Information on TCM, 2023, 30(9): 17-21.] DOI: 10.19879/j.cnki.1005-5304.202210065.

15.李姮. 医疗大数据质量评价指标体系研究[D]. 武汉: 华中科技大学, 2020. DOI: 10.27157/d.cnki.ghzku.2020.003499.

16.Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research[J]. J Am Med Inform Assoc, 2013, 20(1): 144-151. DOI: 10.1136/amiajnl-2011-000681.

17.Naik S, Voong S, Bamford M, et al. Assessment of the nursing quality indicators for reporting and evaluation (NQuIRE) database using a data quality index[J]. J Am Med Inform Assoc, 2020, 27(5): 776-782. DOI: 10.1093/jamia/ocaa031.

18.Bian J, Lyu T, Loiacono, et al. Assessing the practice of data quality evaluation in a national clinical data research network through a systematic scoping review in the era of real-world data[J]. J Am Med Inform Assoc, 2020, 27(12): 1999-2010. DOI: 10.1093/jamia/ocaa245.

19.熊兴江. 医疗大数据质量评价指标体系构建研究[D]. 武汉: 华中科技大学, 2019. DOI: 10.27157/d.cnki.ghzku.2019.002068.

20.Liaw S, Guo JGN, Ansari S, et al. Quality assessment of real-world data repositories across the data life cycle: a literature review[J]. J Am Med Inform Assoc, 2021, 28(7): 1591-1599. DOI: 10.1093/jamia/ocaa340.

21.Wang SV, Schneeweiss S, Berger ML, et al. Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1.0[J]. Value Health, 2017, 20(8): 1009-1022. DOI: 10.1016/j.jval.2017.08.3018.

22.Cook JA, Collins GS. The rise of big clinical databases[J]. Br J Surg, 2015, 102(2): e93-e101. DOI: 10.1002/bjs.9723.

23.Duke-Margolis Center for Health Policy. Characterizing RWD quality and relevancy for regulatory purposes[EB/OL]. (2018-10-01) [2023-11-10]. https://healthpolicy.duke.edu/publications/characterizing-rwd-quality-and-relevancy-regulatory-purposes-0.

24.Reimer AP, Milinovich A, Madigan EA. Data quality assessment framework to assess electronic medical record data for use in research[J]. Int J Med Inform, 2016, 90: 40-47. DOI: 10.1016/j.ijmedinf.2016.03.006.

25.Wand Y, Wang RY. Anchoring data quality dimensions in ontological foundations[J]. Commun ACM, 1996, 39(11): 86-95. DOI: 10.1145/240455.240479.

26.World Health Organization. Data quality assurance (DQA)[EB/OL]. (2023-02-14) [2023-12-05]. https://www.who.int/data/data-collection-tools/health-service-data/data-quality-assurance-dqa.

27.Kahn MG, Raebel MA, Glanz JM, et al. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research[J]. Med Care, 2012, 50 Suppl(0): S21-S29. DOI: 10.1097/MLR.0b013e318257dd67.

28.Black N, Payne M. Directory of clinical databases: improving and promoting their use[J]. Qual Saf Health Care, 2003, 12(5): 348-352. DOI: 10.1136/qhc.12.5.348.

29.Verma AA, Pasricha SV, Jung HY, et al. Assessing the quality of clinical and administrative data extracted from hospitals: the general medicine inpatient initiative (GEMINI) experience[J]. J Am Med Inform Assoc, 2021, 28(3): 578-587. DOI: 10.1093/jamia/ocaa225.

30.Buchheit RB, Garrett JH, Mcneil S, et al. Automated procedure to assess civil infrastructure data quality: method and validation[J]. J Infrastruct Syst, 2005, 11(3): 180-189. https://doi.org/10.1061/(ASCE)1076-0342(2005)11:3(180).

31.Evans P. Scaling and assessment of data quality[J]. Acta Crystallogr D Biol Crystallogr, 2006, 62(Pt 1): 72-82. DOI: 10.1107/S0907444905036693.

32.Blacketer C, Defalco FJ, Ryan PB, et al. Increasing trust in real-world evidence through evaluation of observational data quality[J]. J Am Med Inform Assoc, 2021, 28(10): 2251-2257. DOI: 10.1093/jamia/ocab132.

33.Rajan NS, Gouripeddi R, Mo P, et al. Towards a content agnostic computable knowledge repository for data quality assessment[J]. Comput Methods Programs Biomed, 2019, 177: 193-201. DOI: 10.1016/j.cmpb.2019.05.017.

34.王雯, 刘艳梅, 谭婧, 等. 回顾性数据库研究的概念、策划与研究数据库构建[J]. 中国循证医学杂志, 2018, 18(2): 230-237. [Wang W, Liu YM, Tan J, et al. Concept and planning of retrospective database study and development of research database[J]. Chinese Journal of Evidence-Based Medicine, 2018, 18(2): 230-237.] https://kns.cnki.net/kcms2/article/abstract?v=IU01-AKyIMBtkQzmSIstN2XKSUGIZj2PiwWaVjqZR0_We0Jf1ZQbLDXcglndcxLoz1i0LoD6U7iKgsQnzfgHRTMs7kqZqWg_Jq6v8njV0hYjQ0-AfQDVklvmst_larST5lWf6UMfQ91FwOFyHW3xOvvW5dLmC7UMU1arfRktNEBO39HryLDxFg==&uniplatform=NZKPT&language=CHS.

35.Carl VW, Shane E, Peter CA. Administrative database code accuracy did not vary notably with changes in disease prevalence[J]. J Clin Epidemiol, 2016, 79: 86-89. DOI: 10.1016/j.jclinepi.2016.05.009.

Multi-dimensional quantitative evaluation of intravenous iron formulations based on A Quick Guideline for Drug Evaluation and Selection in Chinese Medical Institutions (the 2nd edition)