Statistics Matter Thursday, February 16, 2023
Weak or strong indicators?
- STATISTICAL INDICATORS
WEAKER STRONGER
p value < 0.05 Confidence Intervals
surrogate marker clinical outcome
disease-oriented patient-oriented
days ill or days hospitalized mortality
relative risk absolute risk
non-inferiority superiority
sub-group analysis primary outcome
secondary outcome
retrospective prospective
(correlation only)
observational randomized
(non-randomized)
non-placebo placebo-controlled
under-powered powered for
“inconclusive” positive or negative
small sample size large sample size
dichotomizing outcome “continuous” outcome
*even if everything is STRONGER; it says nothing about the potential for bias. In fact, almost all published RCTs have at least 3 inherent biases:
- publication bias: positives studies >> published than negative ones even though negative studies are a much more common result.
- acuity bias: this is a subset of selection bias in which the RCT selects which patients to be in their study. Some of this may not be avoidable due to legality and pragmatism (eg. you don’t want a woman of childbearing years in your study to take drug X, afraid she may drop out if she becomes pregnant or drug X could cause abortifacient or teratogenic complications before she knows she is pregnant). Some acuity bias is very intentional. Drug companies want really sick people to prove effect ,but not with lots of co-morbidities because their mortality is higher and will make the mortality rate of the drug look bad. And they don’t want mentally unhealthy persons, poor persons, or persons that can not or do not comply well with medication taking - though this population can be a substantial portion of people on the drug.
- Hawthorne effect- people perform better when they know they are being “watched” or studied. Health care workers perform protocols better when they know they are in a study. Patients are more compliant with medication taking and lifestyle choices when they are being compensated to take part in a research study.
Here is the dilemma:
Very few studies have the power to be anything more than “inconclusive” when it comes to important clinical outcomes like mortality (“….though underpowered for mortality…”.)
So researchers find ways to “Power-boost”
POWER -BOOSTERS (will cover more later)
- composite outcome
- meta-analysis
- registries
- reviews
While increasing the total N of results will often (not always) increase the power of your study, especially for more rare events like mortality, — it is all dependent upon what you include in your outcomes or your studies. “Garbage in” will equal “garbage out.” Many are now grading the “strength” and “quality” of their studies. This is a necessary and good step with the understanding that there may not be good standards on what distinguishes a “poor quality” study from a “good quality” study.
A final way to create better results is to take weaker types of data and adjust them to try an mimic a randomized controlled trial.
PAPER- ADJUSTERS (will cover later)
- “Intention-to-treat”
- Propensity scores