Understanding the forecasting process and making decisions based on those predictions is not done easily. Without time-machines or real fortune tellers to give us the answers of what will happen, we’re left with the practice of modeling the future, and subsequently base our decisions on what those models tells us. As these results provide the basis for our decisions, the models must be sufficiently realistic and describe the most important components of the decision reality.
The most central unknown factor for pharmaceutical projects is how good the developmental drug really is, i.e how great is the treatment effect of the drug we’re trying to develop. Nevertheless, we do know that the whole process of drug development is affected by the unknown magnitude of the drug’s effectiveness. Decisions made throughout the development process aim to progress drug candidates with a beneficial treatment effect, while terminating projects with inferior effect. Thus, with the actual treatment effect being unknown, we need to include the corresponding uncertainty into the model to adequately mirror the circumstances for the decision. For this purpose, Captario has developed the Clinical Effect Model.
A Realistic Approach To Modeling Treatment Effects
We may assume that each drug candidate has an inherent treatment effect that would be realized, should the compound be given as a drug to the targeted patient population. We call this inherent property the True Clinical Effect. A distinction can be made to the effect that we can estimate in a clinical trial. We refer to this as the True Study Effect. When assessing a clinical study it’s important to remember that all patient groups holds a certain level of variance in the effects of the drug. Therefore, we need to distinguish between three different kinds of effects, which are True Clinical Effect, True Study Effect, and Observed Study Effect.
The True Clinical Effects are the effects on the end patient after we have concluded the studies and the drug is being used on the market, and we can only truly understand the full true clinical effects by examining the patients after their death. The True Study Effects on the other hand are the measured effects within the boundaries of the study. In order to be able to exactly assess the True Study Effect, trials would have to be conducted on basically the whole population of patients, which obviously is not possible. Instead, we’re left with Observed Study Effect, i.e assessing a small sample of patients in the clinical trial, letting them represent the remaining population of patients.
If you for example only evaluate operationally how many patients you should include in a study, the answer would surely be that it’s better to have fewer (less cost and shorter time). However, we also know that by including many patients in a study, the estimate of the treatment’s efficacy will have greater precision, decreasing the random difference between Observed Study Effect and the True Study Effect. If the drug has great market potential, we want to be as certain as possible and thus have more patients. Especially if we are dealing with a disease with high drug effect variability that is difficult to get a good answer to, such as depression or other psychiatric illnesses. As such, we need to construct a model that quantifies the benefit we can get from having many patients in the study in order to find an optimal number.
In order to generate such a model of sufficient realistic relevance, the model should generally address at least these three key components:
1. Uncertainty in the treatment effect
2. How the observed treatment effect controls the outcome in the decision points
3. How the treatment effect affects sales in the market
Understand the False Positives & True Negatives in Clinical Studies
Looking at the two by two below, the two axes represent the fictitious outcome of observed study effect, and the underlying true clinical effect (observed study effects on the X-axis and true clinical effects on the Y-axis). When assessing the effects for each conducted study, and supporting the subsequent investment decision, we assign a numerical value which then defines whether the study is considered successful, and whether it leads to continuing with the project or not. These decision rules are referred to as Stop/Go (or Go/NoGo) criteria.
For this fictitious example, the chosen value for continuing is any value larger than 0 (X ≥ 0 implies GO), which consequently entails that any study with a value of 0 or lower would get terminated (X < 0 implies STOP). This means that any study with observed effect results in the two squares to the far right would imply that the project proceeds. At the same time, we have the underlying true clinical effects, which means that had we known the true clinical effects for each study, every study in the two top squares would have been classified as drugs with positive effect (where Y ≥ 0).
Hence, if the true clinical effect and the observed effect both are over 0 we have a True Positive, if the true clinical effect and the observed effect both are below 0 we have a True Negative, if the true clinical effect is over 0 and the observed effect is below 0 we have a False Negative, and if the true clinical effect is below 0 and the observed effect is over 0, we have a False Positive.
Analyzing the quadrants of the table above illustrates that there is value to be lost. Lost value in the form of investing in a project that demonstrates a positive outcome in the study, but has limited true effect and eventually does not get FDA approval, a False Positive. Furthermore, lost value in the form of terminating trials that otherwise would have gone all the way to market, i.e a False Negative. Incorporating the treatment effect as a part of drug project modeling enables the assessment and analysis of these important properties of investment decisions made in a drug project.
A Holistic Approach Increases Model Relevancy
Modeling can take on many different forms, but it’s important to remember that no matter how many variables you insert into the model, the relevance and usefulness of your model depends on its potential to adequately capture the most relevant aspects. A core element of a model for clinical projects is the treatment effect on the targeted patients. As the treatment effect is largely uncertain, we suggest incorporating uncertainty into the model with holistic linkages along the whole development process, which in turn provides a more realistic decision basis than what is traditionally used today. By turning the tables and constructing the model to mimic the outcomes that forms the basis for project decisions (e.g. by appreciating the distinction between Observed Study Effect, True Study Effect, and True Clinical Effect), it leaves the model more apt to assess and mitigate the risk of erroneous decisions in clinical projects. By and large, decisions are then made in a more relevant context, and provides a more informed ground for decision-making.
Being able to adequately link different parts of the development process is key when creating a model intended to hold a high level of reality relevance. Linking the decision points of what the development process should look like with the chance of success in the various decisions, and ultimately with the chance of getting good sales in the market, creates a balance between the pros and cons of different courses of action. Further, it allows for clearer distinctions of what constitutes a good or bad clinical study outcome, limiting the risk of making False Positive or False Negative decisions. This increases the chance of making positive investment decisions for those projects that will eventually receive FDA approval, hence providing more value both to the company and to all the patients with medical needs that we are trying to address.
***
Captario has successfully applied the Clinical Effect Model with several Big Pharma companies. Do you want to know more about the Clinical Effects Model and how to leverage it in your business? Contact Us or read the Scientific Paper A Modeling Framework For Improved Design and Decision-Making in Drug Development by Captario Head of Science, Stig-Johan Wiklund.