Previous month:
November 2010
Next month:
February 2011

Are clinical trial results all wrong?

And what to do about it if it's true.

One would not normally consult the New Yorker magazine about the scientific method, but a recent article in that publication makes it necessary that we do just that.

In short, this article concludes that most of the controlled clinical trials that have been performed and used to get medical treatments on the market had (and will continue to have) false results.

Now, read that again and let it sink in!

Since we have learned to be paranoid about the integrity of what we do in this industry, it would be natural to think that this claim must be based on evidence that the studies have been rigged. This is not the case. Rather, the various individuals cited in the article claim that it's simply human nature that leads us to reach the results that we wish to reach and that we actually use the scientific method to help us get there.

Now, it's not my objective to summarize or rehash what's in this article. You are perfectly capable of doing that yourself. Rather, I want to share with you my ideas on the ramifications of the core finding of the article. Let me repeat those here in my own words:

  1. It is nearly impossible to come up with a truly objective scientific study design;
  2. It is human nature to (unknowingly) design the study to prove what we wish to prove;
  3. We use scientific tools and techniques to fool ourselves that we are being objective;
  4. We do this all with good intentions (most of the time); and
  5. Other scientists reproduce the false results for the same reasons (see 1 - 4)

If we take all of this at face value, we would need to ask why we continue to design and run controlled clinical trials. Of course, we know that this is what the regulators (i.e. FDA, EMEA) want. Since the regulators are not necessarily smarter than the sponsor's scientific staff or, for that matter, the experts that they call on to pass judgment on marketing submissions, we need to conclude that everyone pretty much has deep faith in the current and accepted methods we use to carry out these trials. Based on this article, we thus need to also conclude that all of these people are wrong!

At this point, you may ask yourself "What is this guy talking about? He must be an idiot!" If you think this, it means that you have not yet read the article (see link above.)

Personally, I found this article to be quite disturbing. After reading it, I could have done one of two things: 1. Put my head in the sand and pretend that I never read it; or 2. Bring it to your attention to give the claims more visibility and lead to its evaluation by our industry as a whole.

Now for the good news:

If we can accept the hypothesis and the evidence presented that it's inevitable that the results of clinical trials are normally false, we can move on to a few ideas that may help get past this problem. Here are the ones that I have formulated:

  1. Continue to design and execute controlled trials but strictly limit the number of them required to gain marketing authorization;
  2. For the trials that remain, focus primarily on safety and less so on efficacy;
  3. Allow for adaptive study design for Phase II and III studies;
  4. Provide marketing authorization earlier for the claimed indication;
  5. Require rigorous follow-up of the actual patient population receiving the treatment including the analysis and reporting of pooled data;
  6. Require the reporting of outcomes to show both safety and efficacy in the actual patient population;
  7. Over time, allow outcomes from different treatments to determine whether a drug stays on the market and/or ascertain its cost/benefit value to society

Does any of this sound familiar? The answer is yes. It's just that we now have even more reason to do it.