Natural Language Processing Improves the Speed, Reliability of Determining Cognitive Status From Electronic Health Records
Key findings
- When electronic health records are used for research into dementia, phenotyping of the disease is a major challenge because dementia is underrecognized, underdiagnosed, and underreported in claims data
- Massachusetts General Hospital researchers have developed a semiautomated natural language processing–powered annotation tool (NAT) that reviews both structured data and clinical notes; experts review its summary to make their final adjudications
- When two teams of adjudicators evaluated 100 patients, the agreement between NAT-supported adjudications of cognitive status and adjudications based on manual chart reviews was moderate (κ, 0.65–0.68)
- The agreement between the two teams was very good (κ=0.89) and was higher than with manual chart review (κ=0.80)
- When a separate 527 patients were evaluated, NAT-supported adjudications were substantially faster than manual reviews (mean time difference, 1.4 minutes, P<0.001), and clinicians spent less time using NAT on the second half of patients than the first
Electronic health records (EHRs) are used for various purposes beyond clinical care, including conducting epidemiological research, examining healthcare utilization, and formulating healthcare policies. An important first step is to phenotype the disease or condition—distinguish between patients who are or aren't affected.
Subscribe to the latest updates from Neuroscience Advances in Motion
Phenotyping cognitive status in EHRs is a major challenge because dementia is underrecognized, underdiagnosed, and underreported in claims data. Moreover, several studies have documented that information on cognitive impairment or dementia is often found only in free text, since clinicians often don't make a formal diagnosis, refer to a specialist, or prescribe medication. Thus, accurate phenotyping of cognitive status requires using structured data (e.g., diagnosis codes, medications and laboratory test results) and unstructured clinical notes.
In the first approach that combines those forms of information, researchers at Massachusetts General Hospital used a natural language processing (NLP) tool to assist annotators in making their final judgments. In the Journal of Medical Internet Research, Ayush Noori, a research student at the Mass General Institute for Neurodegenerative Disease; Colin Magdamo, BS, research intern in the Department of Neurology, Shibani S. Mukerji, MD, PhD, associate director of the Neuro-infectious Diseases Unit, and Sudeshna Das, PhD, director of the Biomedical Informatics Core, and colleagues, reports use of the tool improved the interrater reliability and speed of phenotyping cognitive status.
Background
The researchers have developed a semiautomated NLP-powered annotation tool (NAT) to facilitate the adjudication of cognitive status from EHRs. The tool processes structural data from the EHR and then ranks clinical notes based on a deep learning NLP algorithm that classifies whether each note indicates the patient is cognitively normal (CN), indicates cognitive impairment (CI), or provides no relevant information.
Upon logging in to NAT, the annotator sees a dashboard with information such as the identification numbers, ages, sexes of their assigned patients, and each patient's label (CN, CI, or undetermined). Selecting a patient navigates the user to a summary of demographic, clinical, imaging, and laboratory data.
In addition, notes with highlighted keywords are presented. Each sequence of notes is classified as CN, CI, or neither to expedite the review of the entire chart history during the relevant period for adjudication of cognitive status.
Interrater Reliability
The research team evaluated interrater agreement using EHR data on 100 patients seen at Mass General Brigham between January 1, 2017, and December 31, 2018. The patients were selected from 1,002 Medicare beneficiaries who had their cognitive status expert-annotated as part of a previous study published in the Journal of the American Geriatrics Society. Selection was random, but five categories of cognitive status were represented equally.
The 100 patients were randomly assigned to two teams of expert annotators for adjudication supported by NAT:
- The agreement between the NAT-supported adjudications and the adjudications in the previous study, which were based on manual Epic chart reviews, was moderate for both team 1 (κ=0.68) and team 2 (κ=0.65)
- The agreement of NAT adjudication between team 1 and team 2 was very good (κ=0.89) and was higher than the interrater agreement between manual reviews in the previous study (κ=0.80)
Speed of Adjudication
The data on the 100 patients didn't include information about the time required for manual review, so it couldn't be used to compare the time required for phenotyping with NAT versus manual chart review.
Instead, the researchers used data on 527 patients with SARS-CoV-2 infection confirmed between March 1 and December 31, 2020. Their cognitive status was determined by manual review in Epic as part of a case–control study of how COVID-19 affects people with and without HIV.
Adjudications supported by NAT were substantially faster than manual reviews (mean time difference, 1.4 minutes; P<0.001). Furthermore, clinicians spent less time using NAT on the second half of patients than the first, a learning effect not observed with manual reviews.
Using NAT to adjudicate cognitive status is expected to increase the feasibility of using EHRs to build large-scale cohorts for the study of cognitive decline.
view original journal article Subscription may be required
Learn more about the Department of Neurology
Learn more about the Mass General Institute for Neurodegenerative Disease