Machine learning-driven flattening of model priors: A comparative simulation study across multiple compounds
URL
Abstract
Introduction
Model-informed precision dosing (MIPD) supports clinical decision-making by leveraging mathematical models and individual drug or biomarker measurements [1]. A typical MIPD workflow involves estimating individual maximum a posteriori (MAP) values for parameters which include interindividual variability (IIV). MAP values represent the mode of the posterior distribution and are obtained by maximising the product of the likelihood of the data and the parameter’s prior distribution [2]. The selected prior distribution needs to properly reflect the uncertainty about the parameter and is often defined by a distribution around the covariate-adjusted typical parameter value with a variance given by the model’s IIV estimate. However, this representation of uncertainty is only accurate when the model closely represents the clinical population (CP). In reality, deviations between model and CP are expected [3]. This deviation gave rise to the method of machine learning (ML)-assisted flattening of model priors [4], where the variance of the prior is selectively increased to account for these deviations. While this method has shown considerable improvement in predictive performance for vancomycin [4], its potential benefit across other drugs has not been studied.
Objectives
To assess the impact of ML-driven flattening of priors on the predictive performance for five compounds which are commonly applied in Therapeutic Drug Monitoring (TDM) programs by conducting a simulation study.
Methods
Five compounds were investigated: vancomycin (VAN), meropenem (MEM), methotrexate (MTX), infliximab (IFX), and tacrolimus (TAC). To have a plausible representation of the differences between model and CP to which the model is applied, two published models per drug [5-14] were selected and split into a data-generating model (DM) and an applied model (AM) used for Bayesian forecasting.
A plausible TDM dataset was simulated with the DM (n=5000) using mrgsolve [15]. The PK-Sim® [16] population builder was used to generate demographically realistic virtual patients. All other covariates were drawn from lognormal distributions based on reported point estimates. Two administrations, each followed by a single drug concentration measurement, were simulated for each individual. RUV was included to account for the expected noise in clinical readouts.
MAP estimation using the AM was performed with prior weights λ=1 and λ=1/8 for standard priors (SP) and flattened priors (FP), respectively. While the first concentration was included for MAP estimation, the second was used as a reference to evaluate the predictive performance. For each patient and λ, the residual between prediction and withheld datapoint was calculated.
The labeled dataset was then split into a training/test cohort (75%/25%) to build a ML model which predicts whether FP or SP should be applied for a given individual. All ML steps were conducted using the tidymodels [17] framework in R. The XGBoost (XGB) algorithm [18] was employed, using a 5-fold cross-validation with 5 repeats and a grid search for hyperparameter tuning. Optimal hyperparameters were selected based on precision.
To assess the impact of the method on the predictive performance of the model, the test dataset was bootstrapped (n=1000). For each sample, RMSE and MPE along with their relative improvement compared to SP were calculated across patients.
Results
Results from the simulation study, presented as the median relative improvement (-) or deterioration (+) compared to SP, were:
- VAN
- RMSE: -16.0% (-11 to -22% [4])
- MPE: -40.8% (-42 to -74% [4]) -MEM
- RMSE: -0.12%
- MPE: -2.89%
- MTX
- RMSE: +1.06%
- MPE: -4.15%
- IFX
- RMSE: -11.8%
- MPE: +1.69%
- TAC
- RMSE: -32.6%
- MPE: +18.0%
Overall, the simulated impact ranged from -40.8% to +1.06% (RMSE) and -16.0% to +18.0% (MPE). Precision for the test dataset was 72.1% (VAN), 25.8% (MEM), 32.8% (MTX), 37.9% (IFX), and 71.3% (TAC).
Conclusion
The predicted improvements in RMSE and MPE for VAN are in agreement with the reported values. However, the impact of ML driven-flattening of priors on the predictive performance for other drugs varied substantially: The largest reduction in RMSE/MPE was indeed found for VAN with heterogenous improvements and even deteriorations for the other compounds. Missing impact of modified priors was also previously reported [19]. Moving forward, we plan to extend the investigation to identify variables which potentially explain the observed differences.