You’re a Dirty Little Conclusion Jumper

 

In Physical Therapy literature, I have noticed a common occurrence of persons reading abstracts and conclusions with a disregard for all in between. There are likely a multitude of reasons why this happens including: lack of time, patience, and a general poor understanding of methodology or statistical analysis. I hope to bridge this gap with a series of posts on basic statistical options and why authors choose them. I also want to highlight the important concept that when authors skip crucial steps in the process both before and during data collection, we must be skeptical of some or all of their conclusions.

Power

Power is a statistical calculation in order to provide a researcher with an idea of how many subjects will be required to make a conclusion. The researchers typically perform an a priori power analysis to determine this number. This word is Latin for “what is before”. One issue is that many authors do not perform this analysis. Even the best methods of all time can come to a quick halt if they do not have enough subjects to study. Not performing a power analysis can be somewhat of a harikari to a paper that the primary researchers spent months out of their lives working on during data collection and publication.

Why?

If a study is underpowered (not enough subjects) it becomes challenging to find a true statistical difference between groups (if there is a difference). If the study is too small, the conclusions cannot be inferred to the general population. Post hoc power analyses can be performed, but frequently are of little value. There is a great deal of controversy between statisticians and researchers on this topic, but typically post hoc testing can be excellent for planning of future, better designed studies and poor for substantiating a conclusion.

Example

Underpowering can be seen in the hallmark paper on Vaccines and Autism. This is one of the many issues that have come up with this publication, and a reason for its redaction. In this paper, they associate 8 of 12 pediatric patients who have a developmental disorder diagnosed post-measles, mumps, rubella vaccination. What fails to be acknowledged, is that compared to the rate of total vaccinations, this number is meaningless. In 2015, the CDC reported 91.5% of children in the United States received the MMR vaccine. If a sample of 12 with a developmental disorder was taken from the population, it is likely that nearly all would have received the vaccine. In other words, the likelihood of those children with developmental disorders receiving the vaccine and being diagnosed cannot be distinguished from pure luck (1, 2).

Issues Researchers Come Across

Sometimes it is not plausible to complete a study with a properly powered group in physical therapy. Researchers are constantly under constraints of time, money, and population to recruit from. An example could be a longitudinal study on pulmonary rehabilitation in elderly persons with cystic fibrosis. Typically, due to the nature of this chronic, progressive, and frequently fatal disease you will have dropout due to personal issues, lack of adherence, and mortality. Occasionally, authors can account for this and exceed the power requirements of the study, but in a population such as elderly persons with Cystic Fibrosis, it is unlikely to find a large enough cohort willing to participate.

What happens when they did their a priori analysis and cannot meet power requirements after dropout? It is both the decision of the authors and you as the reader whether or not their outcomes can be seen as valid. If our Cystic Fibrosis paper needed 20 patients for the control group (assuming there was a control) and treatment group and they ended up with 10 each, we need to look at their conclusions and decide whether or not they are validated. Far too many times I see someone citing a paper to back up their biases with 5 subjects per group in PT literature that likely needed 40 in order to find a true effect. Side note: In clinical practice guidelines, any recommendations must be based on HIGH-level evidence, which meet power requirements along with a number of other methodological factors.

Recommendation

Next time you read a paper, go to see if an a priori power analysis was done. If upon completion of data collection their dependent variables all met power requirements, you can have improved confidence that the conclusions are justified. If they did not meet power, take a skeptical approach to outlandish claims such as recommending to use a product or technique for a pathology. An example of this would be the authors of our underpowered cystic fibrosis study recommending clinicians to use an acapella device based on their results from 10 subjects.

 

The purpose of this post is to improve clinician confidence with a simplified version about powering statistics in PT/medical literature.


References:

  1. RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Wakefield, AJ et al. The Lancet , Volume 351 , Issue 9103 , 637 – 641
  2. National Center for Health Statistics. Health, United States, 2015: With Special Feature on Racial and Ethnic Health Disparities. Hyattsville, MD. 2016.

 

One Reply to “You’re a Dirty Little Conclusion Jumper”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s