Dynamic Optimization of Population Pharmacokinetic Models Using Reinforcement Learning Techniques

Abstract

ABSTRACT : Pharmacometrics - i.e., the science of quantitative pharmacology - has disrupted the way drugs are dosed. It has gone from an empirical methodology, based solely on the experience of the pharmacologist, to a methodology that uses all the available information on the drug, the population, the subject, and the disease, to design a dosage regimen tailored to the needs of each individual patient. Pharmacometrics is generally centered around the development of pharmacokinetic and pharmacodynamic models that represent an individual or population drug concentration behavior and the disease’s response. After the development of these models, pharmacometricians can simulate different sets of scenarios and evaluate the potential patient’s outcome. By doing so, it is possible to calculate the probability of reaching the therapeutic goal for each of the dosage schedules, as well as to find the ones that can reduce the probability of outcomes associated with toxicities. The final objective of the model depends on the decisions made by the pharmacometrician. Currently, the population’s pharmacokinetic and pharmacodynamic model development process requires a large amount of data, ideally on patients that belong to the population of interest. This is usually an expensive and difficult undertaking, since any mistakes made when collecting the blood samples or taking times associated with them can directly lead to erroneous conclusions or even make it impossible to model the data. At this point, the pharmacometrician proceeds to transform the data obtained from the experimental process into the format associated with the pharmacokinetic and pharmacodynamic modeling software of choice. Afterward, they begin the iterative process of fitting the different structural and error models, including or excluding covariates, and selecting the appropriate search space. By doing so, the pharmacometrician expects, based on their experience, to find the best set of initial conditions that would drive the modeling software to find the parameters that adequately fulfill their needs. Having said that, there are different potential problems that could limit the scope and use of this type of pharmacometrics techniques, especially in contexts such as the Colombian one. At the outset, data acquisition is an expensive process and unfortunately, the data itself has a fairly short life span. By this, we mean that after being obtained and used to fit a model, the data normally cannot be reused and, although many authors agree to share their data when requested, this is seldom the reality. The second problem associated with the development of these models is the vast technical knowledge that is required of the pharmacometrician, as well as the highly iterative and time-consuming nature of this process. The time required of the researched leads to an increase in costs, which, in turn, limits the number of possibilities that can be tested and, consequently, introduces a bias associated with the researcher's prior modeling experience. The general aim of this thesis is, therefore, to provide a set of tools designed to mitigate the possible repercussions of the abovementioned problems. Furthermore, we hope these tools contribute to the massification of pharmacometric therapy-individualization techniques, particularly in the Colombian context. The first chapter introduces and provides the proper context of problem addressed in this thesis. The general and specific objectives are also presented, along with the methodology used to address them, including: the data collection process, the development of the pharmacokinetic data transformation and storage platform, and the automatic adjustment system. This chapter ends with a brief description of the framework around which this thesis is built. The second chapter focuses on building the theoretical framework on which this thesis is developed, namely: pharmacological and pharmacometrics definitions, the uses of the latter, and the mathematical framework around which non-parametric algorithms are built. Moreover, we provide context to the reinforcement learning tools that serve as the basis of this thesis. Chapters 3 to 6 compile three articles that resulted from the research on which this thesis is constructed, each one responding to a specific set of objectives set in chapter 2. Chapter 3 presents a systematic review and analysis of the approaches used in population pharmacokinetic modeling of the drugs used in the treatment of mycobacterium tuberculosis infection. We consider the different structural models, parameter choices and their ranges, covariates and statistical approximations used. Finally, we proceeded to compare the results obtained by models which are fitted using parametric versus nonparametric techniques. Chapter 4 accompanies the process of design, development and analysis of a web tool that storages, categorizes, filters and transforms pharmacokinetic data sets to the various formats used by the different modeling software. This development is based on the information found in the systematic search of chapter 3. Chapter 5 presents the methodology for the automatic adjustment of population pharmacokinetic models using reinforcement learning. Along with this methodology, we also include the training process of two agents using a dataset previously published in the literature. This exercise was performed by using two different non-parametric population adjustment algorithms and, subsequently, comparing the results of both agents with the result of the original study. In chapter 6, we summarize of the results obtained for each of the objectives and their main limitations, and prepare the ground for future work derived from this thesis. This work is preliminary and its evaluation will be carried out continuously. Both the pharmacokinetic data storage platform and the automatic adjustment system are going to be continually fed with data uploaded by its users. We hope in the future to continue working with these types of tools and to report their results. In summary, this thesis aims to take advantage of the computational capacities that are currently available, by proposing a protocol that allows pharmacometricians to share their data, regardless of their software of choice. Said data will, then, be used to train a Reinforcement Learning agent that will design the Population Pharmacokinetic model that best fits that data. By providing tools that are open-source and free to use, modify or extend, we expect to help remove current access barriers and to improve the adoption of Population Pharmacokinetics techniques among general practitioners.