Imagine you've had major surgery, maybe a hip
replacement. You get asked later, perhaps a year
or two on, how you're doing via a questionnaire.
Now this feedback, yours and thousands of others,
helps gauge how successful these operations really
are. Seems simple enough, right? But... What
if the people who don't send back those forms,
what if they aren't just, you know, randomly
busy? What if they're actually a specific group
and their absence completely changes the picture
of success that the data shows us? That's the
crucial question we're exploring today on the
deep dive. We're looking closely at a particular
research paper on patient -reported outcome measures,
PROMs after total hip replacement, and the significant
bias that can crop up when patients don't complete
their follow -up. Now, these PROMs, they're really
vital tools. Organizations like NICE here in
the UK, or the FDA over in the US, use them to
understand the real world impact of surgery beyond
just, say, survival rates or basic complications.
They're meant to capture the patient's own view
on their quality of life, their recovery. And
joining us to walk us through this paper and
this, well, critical area of health care research
is Professor Mo Imam. Professor Imam is a leading
orthopedic surgeon and researcher, very involved
in outcomes analysis, and importantly, one of
the authors of the actual study we're discussing
today. Welcome. Thank you. It's a pleasure to
be here. So maybe let's set the scene with some
headline numbers from your study. You tracked
patients having total hip replacements, looked
at their response rates to questionnaires over
time. How did that response rate change from
before the operation? Well, our initial capture
rate preoperatively was actually excellent, almost
99 percent. Oh, wow. OK. But yes, as time went
on, we did see a steady decline at six weeks
after the operation. It was down to about 90
.6 percent. Right. By six months, it had dipped
slightly further to 89%. Then one year out, it
was 83%. And by the time we reached the two -year
mark, we were down to 79 % response. 79%. So
quite a drop. Yes. A considerable proportion
of that initial group was, well, no longer providing
feedback at that point. So doing the maths quickly,
that's roughly a fifth of the original patients
who aren't represented in that two -year data.
Now, was that drop off just random chance? Did
it happen sort of? equally across all patient
types? Or did your analysis find specific things
linked to who didn't respond? Our analysis was
very clear on this point. The non -response was
absolutely not random. Right. We found quite
significant associations between certain patient
characteristics and their likelihood of responding
to those follow -up questionnaires. OK. And that's
really the crux of it, isn't it? So tell us,
who were these patients who were less likely
to respond? What did your study identify? Well,
we found several key factors. One was age. Age,
interesting. Yes, the non -responders were significantly
younger than the responders, and this is true
at all the follow -up points, with a p -value
less than .001, so highly significant. For instance,
at the two -year point, the median age of those
who responded was about 69, but for the non -responders,
the median age was 65. And why might that be?
Does the paper suggest reasons? Well, the paper
speculates a bit. It could be linked to younger
patients perhaps having busier working lives,
finding it harder to make time for questionnaires,
or maybe feeling less connected to the health
care system once that immediate post -op phase
is over. OK. Younger patients not responding.
That's quite an interesting finding, maybe a
bit counterintuitive. What else stood out? Another
really significant factor, and this is something
we highlighted as quite a novel finding in this
specific context, was the association with deprivation.
Deprivation. How did you measure that? We used
the income deprivation affecting children index,
the IDACI score. It gives a measure of socioeconomic
deprivation for an area here in the UK. And we
saw that patients from more deprived areas, those
with higher IDCI scores, were significantly less
likely to respond. Again, this was a very robust
finding, p -value less than .001, and it held
true across all the time points we looked at.
And the reasons for that? Likely multifaceted,
I'd say. It could involve things like lower health
literacy, perhaps, or maybe less stable housing
or social situations, or simply having other
more pressing concerns in their daily lives that
mean completing and returning a form just isn't
a priority. So, age and socioeconomic background
are factors. What about their actual health status
before the surgery? Did their starting point
matter? Yes, it mattered profoundly. We found
that patients who were less likely to respond
later on had significantly lower preoperative
scores on both the Oxford hip score and the EQ5D.
Right, the Oxford hip score measures pain and
function specifically for the hip. Exactly. So
those who started with the worst baseline, meaning
more pain, more functional limitation before
their surgery, were actually less likely to engage
with the follow -up process afterwards. To give
you an example, patients who responded to all
the post -op questionnaires had a median baseline
OHS of 20. But those who missed all of them,
they started lower, with a median OHS of just
16. So that suggests that poor health -related
quality of life right at the beginning is some
kind of predictor for non -response later. It
certainly seems to be, yes. And maybe most tellingly,
you looked at whether a patient's previous experience
or their score at an earlier point influenced
whether they responded next time. What did that
show about why people might not respond? Yes.
This finding, I think, really strengthens the
argument for disengagement being linked to the
outcome itself. We found a strong correlation.
If a patient had lower outcome scores or reported
lower satisfaction at an earlier time point,
they were significantly less likely to respond
to the next questionnaire down the line. Oh,
okay. So, for example, patients who didn't respond
at the 24 -month mark, their median Oxford HIP
score back at the 12 -month follow -up was 38.
Compare that to those who did respond at 24 months.
Their median OHS at 12 months was 43. That's
a five -point difference. Is that significant?
It is, yes. It's clinically meaningful and statistically
significant. It strongly implies that if a patient
felt they weren't achieving the outcome they'd
hoped for, or perhaps wasn't satisfied with their
result or the service received, they were then
less inclined to participate in future data collection.
It's almost like they've voted with their feet
by simply not responding. That's a really powerful
insight. It suggests it's not always just about
practical barriers, but maybe a psychological
disengagement based on how well they feel they've
done. Precisely. Were there any factors you perhaps
expected to be linked to non -response but actually
weren't, according to your study? Well, interestingly,
in our particular cohort, we didn't find a significant
link between gender or whether the patient experienced
surgical or medical complications after the operation
and their likelihood of responding. Really? No
link to complications? Not a statistically significant
one in our data set, no. And the paper does note
that the finding on gender actually differs from
some other studies out there. Which just highlights,
I suppose, that the factors influencing non -response
can vary a bit, depending on the specific patient
group and the setting you're looking at. Right.
Context matters. OK. So we've got a clearer picture
now of who tends to be missing from this follow
-up data. Younger patients, those from more deprived
areas, and crucially, those who started with
lower scores or had poorer outcomes or satisfaction
earlier on. So let's bridge that now to the core
problem this paper really tackles, the bias.
If that specific group is less likely to respond,
what does it fundamentally mean for the outcome
data that we do see reported? Well, the fundamental
consequence, the big issue, is that the reported
outcomes from these data sets or registries that
suffer from significant loss to follow -up are
inherently biased. Biased how? Because the group
that is disproportionately not responding are
the very patients with the worst scores, perhaps
less improvement or lower satisfaction. So, the
group that does respond is therefore skewed.
It's weighted towards patients who are, generally
speaking, doing well and are satisfied. So, the
picture of success painted by those who reply
is potentially overly optimistic compared to
what the entire group of patients actually experienced.
Precisely. And our study provides some quite
concrete evidence for this bias. We look at the
actual improvement in the Oxford HIPPS score,
from baseline to follow -up. At 12 months, the
median improvement for patients who responded
was three points higher than for those who didn't
respond at that point. Three points higher. And
by 24 months, that difference had actually widened
slightly to four points. Four points. Now, does
that difference, three or four points, sound
small or is it meaningful? While it might sound
small numerically, it's statistically significant
and it represents a tangible skew in the reported
outcome data. If you only look at the responders,
you are seeing results that are, on average,
notably better than what the non -responders
actually experienced. Is it unique to hip replacements
or PROMs data? Or is it a much wider concern
in health care data and registries generally?
Oh, it's a much broader concern. This phenomenon
where loss to follow up isn't random, where it
represents a subset with different and often
less favorable outcomes. Well, that's a critical
consideration that really extends far beyond
just hip PROMs. It's relevant for, I'd say, virtually
any study or registry data set that relies on
ongoing patient participation or follow up over
time. So any long term tracking? Yes. The paper
touches on its importance for survival analysis
in joint replacement registries, for instance,
and it acknowledges that the profile of non -responders
might differ depending on the procedure. For
example, with hip resurfacing, you might actually
lose younger, more active patients to follow
up precisely because they're doing so well and
have just got back to their busy lives. That's
a different dynamic to what we saw in total hip
replacements. But the key point remains, the
missing data is rarely, if ever, random. You
also made a really fascinating comparison in
the paper between the UK's national PROMS data
and the Swedish HIP Arthroplasty Register. What
did that comparison tell you about the impact
of response rates? Ah, yes. That comparison offers
quite powerful sort of external support for our
findings on bias. The Swedish Register, you see,
consistently achieves a significantly higher
response rate, typically around 90%. 90 %? That's
high. It is. Compare that to the national UK
PROMS data, which was around 80 % at the time
we did our study. Now, what's really striking
is that despite presumably performing similar
surgeries with similar patients, the Swedish
registry reported a lower patient satisfaction
rate, about 84 % satisfied. That's compared to
a much higher rate, over 95 % reporting improvement
in the UK PROMS data with its lower response
rate. So higher response rate, lower reported
satisfaction. How does that support the bias
idea? Well, our interpretation is that because
the Swedish registry captures a larger proportion
of the total patient population, including more
of those patients with less ideal outcomes, their
reported satisfaction rate is actually a more
accurate reflection. It's naturally lower because
it includes more of the less satisfied people.
It really underscores how the response rate itself
directly impacts the picture of success that
gets reported. That makes sense. Given this significant
bias that can creep in, what are some potential
ways forward? What strategies or efforts does
the paper mention to try and reduce this loss
to follow up and get a more complete picture?
The paper really emphasizes that substantial
effort is needed first to understand and then
to overcome the barriers stopping patients from
following up. This could involve practical things
beyond just sending out standard postal questionnaires,
offering different ways to respond, maybe email
or telephone calls, perhaps even app -based systems.
Using technology more? Potentially, yes. And
also considering things like language barriers
or providing extra assistance for patients who
might have lower literacy levels or perhaps less
access to digital technology. It's really about
making the process as easy and accessible as
we can for all patient groups, particularly those
we've now identified. as being less likely to
respond otherwise. Right, targeting the effort.
And just briefly, what were some of the limitations
of your own study that you acknowledged? Important
for context? Yes, of course. Like any research,
there were limitations. We didn't have access
to really comprehensive socio -demographic details
for all patients, things like education level
or specific language needs. Nor did we have detailed
clinical data on comorbidities or mental health
conditions, which could certainly also influence
response. Okay. Also, it's worth noting a low
hip score isn't always solely due to the hip
surgery itself. It can be influenced by other
coexisting health conditions. And while our institutional
data capture rate was relatively high compared
to the national average at the time, these findings
would definitely benefit from being validated
in other populations and different health care
settings. Transparency about limitations is always
good practice. Okay, let's switch gears slightly
lightning round. Quick questions, sharp insights.
Professor, what's the single most crucial takeaway
for anyone looking at patient feedback data?
Assume non -responders are different. And assume
their outcomes are likely less favorable than
those who did respond. Never take the data you
have as the whole picture unless your follow
-up is near perfect. Good point. If you could
implement one change tomorrow to improve PROM's
data quality based purely on this study, what
would it be? I'd shift focus and importantly
resources towards actively understanding why
certain patients aren't responding and then putting
targeted strategies in place to try and re -engage
them or capture their feedback effectively. Makes
sense. And finally, what's the one essential
question professionals should always ask themselves
when they see published outcome data that's based
on patient feedback? Always ask. What was the
follow -up rate? And, truthfully, was any attempt
made to characterize the non -responders? Because
the lower that response rate, the greater the
potential for bias, and the more cautious you
absolutely must be about the conclusions being
drawn from it. Excellent. Very clear points.
So just to recap some of the most crucial takeaways
from our deep dive today for everyone listening.
Firstly, Patient response rates in long -term
studies like PROMs often drop off significantly,
which leaves quite a substantial gap in the data
we collect. Secondly, and this is critical, this
non -response isn't just random. Studies, like
the one we've discussed, show that the patients
less likely to respond are often younger, come
from more deprived areas, and significantly had
lower baseline scores or experienced poorer outcomes
and satisfaction earlier on. Thirdly, this leads
directly to a major issue. Bias. The reported
outcome data, based only on those patients who
do respond, is likely painting an overly positive
picture compared to the true average outcome
for all patients because those less successful
results are disproportionately missing. Fourthly,
this isn't just an orthopedics issue. This problem
of non -random loss to follow -up and the bias
it creates is a vital thing to consider for anyone
working with patient reported outcomes, registry
data, or really any research relying on ongoing
patient feedback over time. And finally, getting
a genuinely accurate understanding of how well
treatments are truly working requires real, dedicated
effort. Effort to understand and reduce the barriers
to follow -up, making sure the voices and experiences
of patients who had less favorable outcomes are
also included so we get a more complete and much
less biased picture. That was a really essential
deep dive into a critical aspect of healthcare
data validity. Understanding this potential for
bias and patient follow -up is just fundamental
for anyone making decisions based on these kinds
of metrics. And if you found this deep dive valuable,
please do take a moment to rate and share the
show. It really helps other professionals discover
insights like these. Yes. I think this research
really does underscore that the methodology of
how we collect data profoundly impacts the reliability
of the conclusions we can draw about clinical
effectiveness. It pushes us all, I hope, to be
more critical consumers of outcome data. Absolutely.
So perhaps a final thought for you to consider
as you look at outcome data in your own professional
world. Always pause and ask yourself who might
not be represented in these dumpers and what
might their story and their outcome truly be?