- This project examined whether a natural language processing (NLP) algorithm could be developed that would identify a group of patients at high risk of severe maternal morbidity (SMM) at the time of admission for delivery
- The algorithm was derived from history and physical free-text clinical notes on 10,250 patients, tested on records for 3,322 patients from the same hospital, and validated on records for 23,397 patients from another hospital in the same system
- In the internal test dataset, the area under the curve (AUC) for the ability of the algorithm to predict overall SMM was 0.67 (95% CI, 0.63–0.72), and the AUC for its ability to predict non-transfusion SMM was 0.72 (95% CI, 0.65–0.80)
- In the external validation dataset the respective AUCs were 0.72 (95% CI, 0.70–0.74) and 0.76 (95% CI, 0.73–0.79)
- These results are early evidence that the NLP algorithm could scan an admission note and identify a patient presenting for labor and delivery as being at high risk of severe morbidity, prospectively and without delaying clinical workflows
The rates of severe maternal morbidity (SMM) and mortality are higher in the U.S. than in other developed countries. More accurate risk stratification is a public health priority.
Subscribe to the latest updates from OB/GYN Advances in Motion
Mark A. Clapp, MD, MPH, a specialist in the Maternal–Fetal Medicine Program in the Department of Obstetrics and Gynecology at Massachusetts General Hospital, Thomas H. McCoy, MD, the director of research at the Center for Quantitative Health in the Department of Psychiatry, and colleagues are developing a machine learning solution. As they explain in the American Journal of Obstetrics & Gynecology, their new natural language processing (NLP) algorithm identified a group of patients at high risk of SMM using only the history and physical notes available at the time of admission for delivery.
An NLP computer algorithm filters through the free text of electronic health records and identified words that predict an outcome. In this case, the research team limited the algorithm to single words that occurred in ≥5% and ≤80% of history and physical notes, for a vocabulary of 1,126 unique words.
The records came from patients admitted for delivery at Mass General and Brigham and Women's Hospital between July 1, 2016, and June 30, 2020. At one of the hospitals, there were records on 13,572 patients, which were split between a training set of records for deriving the algorithm (n=10,250) and a test set for retrospectively validating it internally (n=3,322).
At the other hospital, there were records on 23,397 patients, which were used for external validation.
The primary outcome was SMM, meaning any of the conditions listed in the composite definition of the Centers for Disease Control and Prevention:
- Internal test dataset—area under the curve (AUC) for the ability of the algorithm to predict SMM, 0.67 (95% CI, 0.63–0.72)
- External validation dataset—AUC, 0.72 (95% CI, 0.70–0.74)
In the external validation dataset, the algorithm distinguished a high-risk group, patients in the top two deciles of predicted risk (8.3% developed SMM vs. 1.9% in the lower two deciles; P<0.001).
Many of the conditions included in the CDC composite definition of SMM relate to transfusion, so the researchers also examined non-transfusion SMM:
- Internal test dataset—AUC, 0.72 (95% CI, 0.65–0.80)
- External validation dataset—AUC, 0.76 (95% CI, 0.73–0.79)
In the external validation dataset, the high-risk group was patients in the top decile of predicted risk (7.1% developed non-transfusion SMM vs. 0.9% of those in lower deciles; P<0.001).
Laying the Groundwork
These results are early evidence that the NLP algorithm could scan an admission note and identify a patient presenting for labor and delivery as being at high risk of severe morbidity prospectively and without delaying clinical workflows. Notably, the algorithm doesn't require any demographic information or diagnosis codes.
Once developed into a tool, the algorithm would be an adjunct to clinician judgment. Local protocols would presumably influence the exact thresholds for what constitutes "high risk" and the subsequent actions to be taken, such as alerting the attending provider or initiating a care team huddle.
Learn more about the Department of Obstetrics & Gynecology
Refer a patient to the Department of Obstetrics & Gynecology