A little while back, I did a brief review of Alan Aragon’s Research Review. After hounding him for quite some time, I finally got the man to send me a guest article so that I could expose people to a little bit more of his work.
I was hoping he’d give us all the secrets to freaky lean muscle mass gains.
Instead, in keeping with his status as a research nerd (probably not as bad as me but damn close), he sent me the following piece, a look at where potential biases can enter into the world of scientific research.
For those of us who follow research (or see those long lists of references at the end of articles these days), it’s an important reminder that science can be subject to biases just as much as anything else.
In any case, if research is your gig, check out his monthly research review and enjoy the following guest article.
Don’t worry, I’ll get back to the dieting series here before too long.
Ghosts in the Machine: Commercial Bias in Scientific Research
By Alan Aragon, July 2008
How do we know what we know?
The acquisition of scientific knowledge is a delicate process. The range of sources is broad; from the gossip of the grapevine, all the way to the peer-reviewed literature. Ironically, the process of drawing conclusions from scientific research is more of an art than a science. While the lay press is dominated by emotionally striking topics, the scientific literature has similar underlying principles that govern publication, with a few added twists. Funding source is a significant factor influencing study outcomes. Publication bias and funding bias are often overlooked yet important considerations for sizing up the evidence. Unfortunately, in many cases these factors are pre-determinants of the fate of drugs, supplements, and other products within the sphere of human health and disease. For this reason, every aspect of research – even aspects that aren’t plainly visible – must be scrutinized through a skeptical lens.
Publication bias has been defined as the submission or acceptance of manuscripts for publication based on the strength or direction of the study findings . Various factors are associated with the failure to reach publication, but whether the roadblock is primarily on the part of the investigators or the journal editors is unclear. Dickersin and colleagues tracked 737 manuscripts accepted by institutional review boards, and only 6 of the 124 studies that failed to be published were reported to have been rejected . In contrast, Lee and colleagues recently tracked 1107 manuscripts submitted to the British Medical Journal, The Lancet, and the Annals of Internal Medicine. 70% of the studies were rejected outright, and another 24% were rejected after peer review . Only 6% actually made it to publication. In spite of the differing outcomes in these investigations, they both found ‘positive outcome bias’, a preference toward the publication of studies that showed statistically significant results.
Implications of the Problem, and a Solution in Progress
The fact that a sizable body of data goes unpublished means that a significant portion of the truth (or close to it) is left in limbo. This bias of omission gets exacerbated when transferred to meta-analyses and systematic reviews, which often do not include unpublished or “grey” research. Non-publication also can have legal implications since it breaches the agreement researchers make with study participants, ethics boards, and sponsors . Realizing this problem, the scientific community has spent the last couple of decades pressuring the powers-that-be to form a centralized registry of clinical trials.
The first far-reaching breakthrough in this regard was in 2004 when the International Council of Medical Journal Editors (ICMJE) released a document issuing mandatory registration of clinical trials before participant enrollment . This encouraged researchers to register their investigations, creating a portal to access unpublished data, and allowing a more comprehensive inclusion of studies in meta-analyses and systematic reviews. The World Health Organization (WHO) plans to assemble an international network of registries that will function through a web-based platform. There currently is no formal tracking system for trial registration, nor is there any system for penalizing the failure to register. Global regulation of publication bias is still in its infancy, but it’s making steps in the right direction.
Funding Bias – The Other Invisible Beast
Funding bias is the strong tendency for a commercially sponsored trial to yield positive effects caused by the sponsor’s product. A perfect model for examining funding bias is the pharmaceutical industry. A recent review by Sismondo discussed actions by researchers that contribute to sponsorship bias . Tactics include ‘publication planning’, which is a calculated progression wherein drug companies and their agents influence the outcomes and publication of articles for their benefit. The details involve ghostwriters, or unnamed authors hired to assist in data manipulation, tabulation, and editing.
Fulfilling this niche, a multitude of Contract Research Organizations (CROs) advertise their expertise in efficiently shuttling manuscripts through every step of the publication process – from study design, to reporting, and finally to acceptance in the target journals. Pharmaceutical companies have transferred the majority of their clinical trial support from academic institutions to CROs, which in part function as personalized data farms. Company-sponsored research is highly susceptible to design bias, including insufficient trial durations to show side effects, treatment imbalance (unrealistic or uneven comparisons), and inappropriate doses.
“Ghost-Management” in Action
Hiring an unacknowledged third party to design and execute the full range of trial procedures can produce high-quality research, but it’s also a profound source of bias. Healy and Cattell examined the impact of Pfizer hiring Current Medical Directions (CMD) on the surrounding outcomes of sertraline (Zoloft) studies, and found a distinct advantage to outsourcing to CMD . A comparison was made between CMD-assisted trials and trials done without hiring a medical writing agency. Sertraline trials managed by CMD were more numerous for the given time period (85 trials versus 41) and had a 7-fold higher impact factor, thus a higher frequency of citation by other literature. The CMD-assisted trials also had higher-profile authors and appeared in more prestigious journals. And now the kicker – only 50% of the non-CMD trials showed positive results, while 100% of the CMD-assisted trials showed positive results.
Another example of ghostwriting and compromised integrity is the case of Merck’s handling of rofecoxib (Vioxx), which the company withdrew from the market in 2004 due to safety (and legal) issues. Ross and colleagues recently examined the nature and extent of ghostwriting for rofecoxib studies from 1996-2004 . Only 50% of the review articles published either a disclosure of Merck sponsorship or a disclosure of any financial compensation to the authors from the company. These articles were often prepared by ghostwriters, yet were attributed to academically affiliated investigators who often did not disclose financial support by Merck.
Investigator Affiliations & Other Slippery Slopes
At this point I’ll shift gears from the drug discussion and delve into sports supplements, an arena similarly rife with commercial influence. One of the most blatant scenarios of investigator bias is when the lead researcher of a given trial is employed by the company sponsoring the trial, and whose product is being tested in the trial in question. For example, Ronald Stanko is the principal investigator in the majority of trials showing pyruvate’s effectiveness for fitness-related purposes [9-12]. He also holds patents on the pyruvate-based compounds used in the studies.
A similar situation is seen with Steve Nissen, who has led the majority of research on beta-hydroxy-beta-methylbutyrate (HMB). Nissen is also a patent holder of HMB’s use as a sports supplement. I found it amusing that Nissen did a meta-analysis on supplements used for lean mass and strength, concluding that only creatine and (surprise) HMB had sufficient data supporting their use . I was tickled when Decombaz and colleagues took issue with this and brought it to light. Let me quote their concern regarding the inclusion of HMB in the same league of evidence as creatine :
“Demonstrations of efficacy produced by only a small number of separate sources are intuitively less persuasive than those from a large number of independent laboratories. Among reasons for this, we believe that investigations led by connected authors or institutions are likely to share similar approaches to a given question and use a smaller variety of methods than truly independent ones.”
A final investigator affiliation worth mentioning is Paul Cribb and the supplement company AST Sports Science. Cribb is AST’s research director, and is also the principal investigator in the AST-sponsored trials involving their whey (VP2) and creatine (Creatine HSC). Thanks to the partnership of Cribb and AST, these products have spanked some serious Pubmed ass [15-19]. I’m not suggesting that we should completely write off all research produced by company-investigator alliances. However, I am urging everyone to remain skeptical and think critically. Scientific research is like any other tangle of information – it needs to be combed for bugs.
1. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990 Mar 9;263(10):1385-9.
2. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992 Jan 15;267(3):374-8.
3. Lee KP, et al. Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006 Jun 19;184(12):621-6.
4. Krzyzanowska MK, et al. Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA. 2003 Jul 23;290(4):495-501.
5. Abaid LN, et al. Reducing publication bias through trial registration. Obstet Gynecol. 2007 Jun;109(6):1434-7.
6. Sismondo S. How pharmaceutical industry funding affects trial outcomes: causal structures and responses. Soc Sci Med. 2008 May;66(9):1909-14. Epub 2008 Mar 4.
7. Healy D, Cattell D. Interface between authorship, industry and science in the domain of therapeutics. Br J Psychiatry. 2003 Jul;183:22-7.
8. Ross JS, et al. Guest authorship and ghostwriting in publications related to rofecoxib: a case study of industry documents from rofecoxib litigation. JAMA. 2008 Apr 16;299(15):1800-12.
9. Stanko RT, et al. Body composition, energy utilization, and nitrogen metabolism with a 4.25-MJ/d low-energy diet supplemented with pyruvate. Am J Clin Nutr. 1992 Oct;56(4):630-5.
10. Stanko RT, et al. Body composition, energy utilization, and nitrogen metabolism with a severely restricted diet supplemented with dihydroxyacetone and pyruvate. Am J Clin Nutr. 1992 Apr;55(4):771-6.
11. Stanko RT, et al. Body composition, nitrogen metabolism, and energy utilization with feeding of mildly restricted (4.2 MJ/d) and severely restricted (2.1 MJ/d) isonitrogenous diets. Am J Clin Nutr. 1992 Oct;56(4):636-40.
12. Stanko RT, Arch JE. Inhibition of regain in body weight and fat with addition of 3-carbon compounds to the diet with hyperenergetic refeeding after weight reduction. Int J Obes Relat Metab Disord. 1996 Oct;20(10):925-30.
13. Nissen SL, Sharp RL. Effect of dietary supplements on lean mass and strength gains with resistance exercise: a meta-analysis. J Appl Physiol. 2003 Feb;94(2):651-9. Epub 2002 Oct 25.
14. Decombaz J, et al. HMB meta-analysis and the clustering of data sources. J Appl Physiol. 2003 Nov;95(5):2180-2; author reply 2182.
15. Cribb PJ, et al. A creatine-protein-carbohydrate supplement enhances responses to resistance training. Med Sci Sports Exerc. 2007 Nov;39(11):1960-8.
16. Cribb PJ, et al. Effects of whey isolate, creatine, and resistance training on muscle hypertrophy. Med Sci Sports Exerc. 2007 Feb;39(2):298-307.
17. Cribb PJ, Hayes A. Effects of supplement timing and resistance exercise on skeletal muscle hypertrophy. Med Sci Sports Exerc. 2006 Nov;38(11):1918-25.
18. Cribb PJ, et al. The effect of whey isolate and resistance training on strength, body composition, and plasma glutamine. Int J Sport Nutr Exerc Metab. 2006 Oct;16(5):494-509.
19. Cribb PJ, Hayes A. Effects of supplement timing and resistance exercise on skeletal muscle hypertrophy. Med Sci Sports Exerc. 2006 Nov;38(11):1918-25.