1
00:00:00,000 --> 00:00:05,860
Welcome to the Clinician Researcher podcast, where academic clinicians learn the skills

2
00:00:05,860 --> 00:00:11,260
to build their own research program, whether or not they have a mentor.

3
00:00:11,260 --> 00:00:17,340
As clinicians, we spend a decade or more as trainees learning to take care of patients.

4
00:00:17,340 --> 00:00:22,380
When we finally start our careers, we want to build research programs, but then we find

5
00:00:22,380 --> 00:00:27,780
that our years of clinical training did not adequately prepare us to lead our research

6
00:00:27,780 --> 00:00:29,200
program.

7
00:00:29,200 --> 00:00:35,480
Through no fault of our own, we struggle to find mentors, and when we can't, we quit.

8
00:00:35,480 --> 00:00:40,580
However, clinicians hold the keys to the greatest research breakthroughs.

9
00:00:40,580 --> 00:00:46,200
For this reason, the Clinician Researcher podcast exists to give academic clinicians

10
00:00:46,200 --> 00:00:51,800
the tools to build their own research program, whether or not they have a mentor.

11
00:00:51,800 --> 00:01:01,100
Now introducing your host, Toyosi Onwuemene.

12
00:01:01,100 --> 00:01:03,060
Welcome to the Clinician Researcher podcast.

13
00:01:03,060 --> 00:01:07,220
I'm your host, Toyosi Onwuemene, and I'm super excited to be talking with you today because

14
00:01:07,220 --> 00:01:08,220
I have a special guest.

15
00:01:08,220 --> 00:01:10,980
Actually, Leila is not new to the show.

16
00:01:10,980 --> 00:01:12,420
She's been here before.

17
00:01:12,420 --> 00:01:18,700
She is a wonderful and amazing medical information specialist, also known as a librarian.

18
00:01:18,700 --> 00:01:23,700
And she in particular is here today because of her expertise in systematic reviews.

19
00:01:23,700 --> 00:01:29,740
In fact, she is chair of the Systematic Review Caucus for the Medical Library Association.

20
00:01:29,740 --> 00:01:32,700
And so she's here to talk to us today about systematic reviews.

21
00:01:32,700 --> 00:01:34,260
Leila, welcome to the show.

22
00:01:34,260 --> 00:01:36,240
Thank you so much for having me again.

23
00:01:36,240 --> 00:01:40,420
This is super fun, and I love talking about systematic reviews.

24
00:01:40,420 --> 00:01:43,600
Well, systematic reviews is definitely what we're talking about today.

25
00:01:43,600 --> 00:01:48,420
And I'm so glad that you're here because sometimes it seems as if it should be intuitive.

26
00:01:48,420 --> 00:01:50,220
You're doing a systematic review.

27
00:01:50,220 --> 00:01:53,780
Just search the literature, pull the papers together, write the paper.

28
00:01:53,780 --> 00:01:57,700
It seems like it should be straightforward, but it's a lot more complicated than that.

29
00:01:57,700 --> 00:02:03,380
So I will start by saying when we first started working together, I had no idea about systematic

30
00:02:03,380 --> 00:02:05,300
reviews and how long it would take.

31
00:02:05,300 --> 00:02:08,820
My first thought when someone told me that it would take about six months to do a good

32
00:02:08,820 --> 00:02:13,580
systematic review was to laugh and say, why would anybody need so much time?

33
00:02:13,580 --> 00:02:18,220
I feel like a little bit wiser now, but I want you to just talk about what does it take

34
00:02:18,220 --> 00:02:24,460
to do a good systematic review, not just in terms of the time, the effort, but also the

35
00:02:24,460 --> 00:02:28,020
resources that you need in personnel and all things.

36
00:02:28,020 --> 00:02:29,020
Okay.

37
00:02:29,020 --> 00:02:30,020
Wow.

38
00:02:30,020 --> 00:02:31,020
This is a big question.

39
00:02:31,020 --> 00:02:35,260
So you may have to refocus me as we cruise through the answer to that.

40
00:02:35,260 --> 00:02:37,540
So maybe we can break it up into parts.

41
00:02:37,540 --> 00:02:44,300
So we kind of joke that systematic reviews are one of the research methods that nobody

42
00:02:44,300 --> 00:02:47,740
seems to go to the manual for.

43
00:02:47,740 --> 00:02:53,340
There's manuals and checklists and things to tell you how to do them, but nobody does

44
00:02:53,340 --> 00:02:54,340
that.

45
00:02:54,340 --> 00:02:57,620
They just decide that they're just going to launch themselves in.

46
00:02:57,620 --> 00:03:02,580
And then as they sort of cruise down the road, we're doing things, they're surprised by what's

47
00:03:02,580 --> 00:03:04,540
involved in it.

48
00:03:04,540 --> 00:03:10,100
So some of the things that we say, and you remember this at our first consultation where

49
00:03:10,100 --> 00:03:17,100
we sit down with a new team is we ask people how many people are on their team.

50
00:03:17,100 --> 00:03:23,660
So we encourage the minimum, minimum, minimum would be two people, but we encourage you

51
00:03:23,660 --> 00:03:28,820
to include a librarian or team to help you with your searches because you would prefer

52
00:03:28,820 --> 00:03:31,180
to have an expert do that than to do it yourself.

53
00:03:31,180 --> 00:03:35,820
It really ends up, we can talk about that a little bit later, ends up being very complicated.

54
00:03:35,820 --> 00:03:42,180
You're going to need, and we usually say six months at the minimum for an experienced team

55
00:03:42,180 --> 00:03:47,860
with a really tight question and enough like hands on deck because right, you could end

56
00:03:47,860 --> 00:03:51,900
up with fed teams of like 18,000 things to screen.

57
00:03:51,900 --> 00:03:56,780
So on an average, we actually say it takes about 12 months and on the outside, kind of

58
00:03:56,780 --> 00:03:59,480
like 18 is not unusual.

59
00:03:59,480 --> 00:04:03,500
So that's because you have to figure out what your question is.

60
00:04:03,500 --> 00:04:05,700
You have to do the searches.

61
00:04:05,700 --> 00:04:09,820
You might end up having to refine your question depending on how big your question is.

62
00:04:09,820 --> 00:04:16,880
It might be, you know, like 18,000 things and that's about 200 things an hour.

63
00:04:16,880 --> 00:04:20,500
So if you divvy that up amongst your right, it could take months and months and months

64
00:04:20,500 --> 00:04:22,980
just to get through your screening.

65
00:04:22,980 --> 00:04:27,340
Then you have to read the papers and then you have to, and you probably know this, but

66
00:04:27,340 --> 00:04:31,860
nobody thinks about it is when you get to where you actually have to pull the data from

67
00:04:31,860 --> 00:04:33,500
the papers you've selected.

68
00:04:33,500 --> 00:04:36,500
That's the most time consuming piece.

69
00:04:36,500 --> 00:04:40,180
And I feel like we should spend more time talking about like, what are you going to

70
00:04:40,180 --> 00:04:42,360
collect and how are you going to collect it?

71
00:04:42,360 --> 00:04:44,220
How do you plan to normalize your data?

72
00:04:44,220 --> 00:04:46,060
And so like all these things.

73
00:04:46,060 --> 00:04:52,180
And in the meantime, you've got software and you have manuals and you have reporting guidelines

74
00:04:52,180 --> 00:04:56,340
of things that you have to meet in order to publish the work that you've done.

75
00:04:56,340 --> 00:04:59,260
There's just so many things in that whole process.

76
00:04:59,260 --> 00:05:00,260
Is that a good overview?

77
00:05:00,260 --> 00:05:01,260
Yes.

78
00:05:01,260 --> 00:05:02,480
No, it's good.

79
00:05:02,480 --> 00:05:06,940
It's good because what I'm hearing you say, and I'm so glad that you're communicating

80
00:05:06,940 --> 00:05:11,620
to the audience is that this is not just a walk in the park.

81
00:05:11,620 --> 00:05:14,980
This is not just a pull a couple of papers together and do a review.

82
00:05:14,980 --> 00:05:16,580
It really is systematic.

83
00:05:16,580 --> 00:05:20,140
There's a process to it and it takes time and it takes energy.

84
00:05:20,140 --> 00:05:22,340
It takes people to do it as well.

85
00:05:22,340 --> 00:05:26,820
So speaking about people, your librarian, why should you involve your librarian in the

86
00:05:26,820 --> 00:05:27,820
systematic review?

87
00:05:27,820 --> 00:05:32,180
Aren't we just putting a few keywords into PubMed and finding a couple of papers?

88
00:05:32,180 --> 00:05:33,180
I love this.

89
00:05:33,180 --> 00:05:36,540
I mean, you can, but what we usually warn people.

90
00:05:36,540 --> 00:05:41,540
So a systematic review is a very defined methodology.

91
00:05:41,540 --> 00:05:47,140
And one of the reasons it is at the highest level of the pyramid of your evidence pyramid

92
00:05:47,140 --> 00:05:52,760
is that what you've said is we have gone out there and found all of the studies about this

93
00:05:52,760 --> 00:05:56,980
particular topic and we've done things to them.

94
00:05:56,980 --> 00:06:02,160
One of the things that also defines a systematic review is evaluating the quality of the article

95
00:06:02,160 --> 00:06:04,420
so we can talk about that a little bit later.

96
00:06:04,420 --> 00:06:10,100
But you go out and you have to find all of that and it's really embarrassing if you go

97
00:06:10,100 --> 00:06:13,220
to publish something and you have not found it.

98
00:06:13,220 --> 00:06:20,420
It's also a little bit dangerous because your audience trusts that you have found everything

99
00:06:20,420 --> 00:06:25,460
and that the recommendation you are giving with your paper is accurate because you have

100
00:06:25,460 --> 00:06:26,900
found everything.

101
00:06:26,900 --> 00:06:32,860
And you have worked with me and with other librarians and we're just used to, just like

102
00:06:32,860 --> 00:06:38,020
clinicians are, we're used to thinking about the problem within our scope of expertise

103
00:06:38,020 --> 00:06:40,500
in a really specific way.

104
00:06:40,500 --> 00:06:43,940
So we're used to thinking about the other ways topics are talked about.

105
00:06:43,940 --> 00:06:49,180
We're really, we have to do at least three databases for a systematic review so we know

106
00:06:49,180 --> 00:06:52,140
the language and syntax for each one.

107
00:06:52,140 --> 00:06:57,020
We know how to pull back enough stuff but not more than you can handle.

108
00:06:57,020 --> 00:07:01,140
There's all these little tips and tricks that come just with the expertise of searching

109
00:07:01,140 --> 00:07:02,780
for information.

110
00:07:02,780 --> 00:07:06,180
And that's your data that you base the rest of your paper on.

111
00:07:06,180 --> 00:07:11,700
So all the methodologies and stuff say we really recommend working with some sort of

112
00:07:11,700 --> 00:07:17,420
informationist in order to get the data that you then work on for the paper.

113
00:07:17,420 --> 00:07:19,780
So we have to collect your data.

114
00:07:19,780 --> 00:07:21,660
Yeah, that's awesome.

115
00:07:21,660 --> 00:07:26,260
So you talk about the need to be systematic.

116
00:07:26,260 --> 00:07:28,820
So this is a research methodology.

117
00:07:28,820 --> 00:07:34,260
Systematically searching the literature is a unique skill set.

118
00:07:34,260 --> 00:07:38,800
It takes expertise to be able to do it well and make sure that you're systematically capturing

119
00:07:38,800 --> 00:07:42,200
all articles on that topic.

120
00:07:42,200 --> 00:07:47,100
And so in the same way as clinicians we spend a long time trying to figure out how to do

121
00:07:47,100 --> 00:07:48,580
our clinical specialty.

122
00:07:48,580 --> 00:07:53,860
Well, as librarians you take time figuring out how to do these systematic searches to

123
00:07:53,860 --> 00:07:55,820
the best of your ability.

124
00:07:55,820 --> 00:08:01,660
Yes, and I could teach someone if they wanted to spend the time practicing just like any

125
00:08:01,660 --> 00:08:08,340
sport or skill if you had the time and the opportunity to practice then you could be

126
00:08:08,340 --> 00:08:10,300
just as good at it.

127
00:08:10,300 --> 00:08:18,420
It really does take months to years to really learn all the tips and tricks and skills to

128
00:08:18,420 --> 00:08:19,420
end up doing that.

129
00:08:19,420 --> 00:08:24,260
And I continue to learn things from other experts who have figured stuff out.

130
00:08:24,260 --> 00:08:27,220
And we're just like you guys in a way.

131
00:08:27,220 --> 00:08:29,020
It's a clinical skill.

132
00:08:29,020 --> 00:08:31,060
Absolutely, absolutely.

133
00:08:31,060 --> 00:08:36,060
So we started with team and as part of your team you want at least two people but probably

134
00:08:36,060 --> 00:08:41,460
more and your librarian needs to be part of that team so that you can do a good systematic

135
00:08:41,460 --> 00:08:42,460
search.

136
00:08:42,460 --> 00:08:43,460
So we've got the team assembled.

137
00:08:43,460 --> 00:08:45,860
Now let's talk about the protocol.

138
00:08:45,860 --> 00:08:48,780
Why do you need to write a protocol before you get started?

139
00:08:48,780 --> 00:08:51,980
Why not just define the question and go?

140
00:08:51,980 --> 00:08:59,860
Yeah, so I liken it to making a decision about where you go before you start the car and

141
00:08:59,860 --> 00:09:01,460
driving, right?

142
00:09:01,460 --> 00:09:07,940
So you type in your GPS and if you're smart you'll have a quick look to make sure it's

143
00:09:07,940 --> 00:09:11,380
taking you someplace that makes sense, right?

144
00:09:11,380 --> 00:09:15,540
Instead of just being like okay and then getting to where the bridge is out.

145
00:09:15,540 --> 00:09:16,860
How did I get here?

146
00:09:16,860 --> 00:09:20,540
Or the traffic's terrible or whatever, right?

147
00:09:20,540 --> 00:09:26,780
So a protocol is your roadmap or your planning document for your project.

148
00:09:26,780 --> 00:09:31,980
So just like you would do a clinical trial, any kind of experiment you would set up how

149
00:09:31,980 --> 00:09:34,700
you plan, execute all of those things.

150
00:09:34,700 --> 00:09:41,260
So then you have to stop every time you hit a place and say now what do we do?

151
00:09:41,260 --> 00:09:43,180
Okay we've done the search, now what?

152
00:09:43,180 --> 00:09:46,620
Okay now we've loaded it in the software, okay now what?

153
00:09:46,620 --> 00:09:49,140
All right now we're onto the next step, right?

154
00:09:49,140 --> 00:09:54,460
So you end up if you don't have a protocol kind of coming with screeching stop and it

155
00:09:54,460 --> 00:10:01,420
really slows down your flow and I just feel like kind of knowing where you're going, what

156
00:10:01,420 --> 00:10:07,860
data you plan to collect, how you plan to evaluate the articles, all of that makes you

157
00:10:07,860 --> 00:10:10,440
actually better at the earlier steps.

158
00:10:10,440 --> 00:10:15,460
So really having a good idea of your final destination and that's not just well I plan

159
00:10:15,460 --> 00:10:21,540
to do X but like the things that you plan to collect and the way you plan to do it really

160
00:10:21,540 --> 00:10:24,220
makes a huge difference early on in the process.

161
00:10:24,220 --> 00:10:28,100
Have you found that to be the case now that you know where you're going?

162
00:10:28,100 --> 00:10:34,780
So your protocol is that roadmap and also it's a little bit to keep you from cheating.

163
00:10:34,780 --> 00:10:38,940
I mean this in the like nice way, just something you can't just as you're cruising along and

164
00:10:38,940 --> 00:10:45,540
you realize this is not going the way I expected it to which is like 50% of science maybe more.

165
00:10:45,540 --> 00:10:50,820
You can't just pivot, it's supposed to keep you honest, you know so this is what I said

166
00:10:50,820 --> 00:10:56,380
I was going to do and because it's not working out the way I wanted it to you can either

167
00:10:56,380 --> 00:11:01,060
quit and just opt not to but you can't cheat and change it, right?

168
00:11:01,060 --> 00:11:02,940
The protocol keeps you honest.

169
00:11:02,940 --> 00:11:08,340
Absolutely, absolutely and I love the what you say about it being the roadmap.

170
00:11:08,340 --> 00:11:11,340
What it allows you to do because you're answering these questions as you're going through the

171
00:11:11,340 --> 00:11:15,900
protocol it allows you to think early about challenges that you might face and address

172
00:11:15,900 --> 00:11:19,540
and decide how you're going to address them before you get to those challenges so it's

173
00:11:19,540 --> 00:11:21,820
definitely very helpful.

174
00:11:21,820 --> 00:11:26,500
Now you can't just you so you put your protocol together but you also have to register it.

175
00:11:26,500 --> 00:11:28,380
What is this thing about registering protocols?

176
00:11:28,380 --> 00:11:30,380
Can you speak to that?

177
00:11:30,380 --> 00:11:36,860
Yes, so there are a protocol is also helpful because it helps you plant your flag in your

178
00:11:36,860 --> 00:11:42,900
topic, it lets people know hey don't even bother we've already started working on this

179
00:11:42,900 --> 00:11:47,980
here's our question here's what we plan to search and so you can go in and look at these

180
00:11:47,980 --> 00:11:54,620
registries and say oh Toyosi is working on this already I'm going to find another aspect

181
00:11:54,620 --> 00:11:59,860
of this topic that I'm particularly interested in so it saves you from wasting any time on

182
00:11:59,860 --> 00:12:02,540
a topic somebody else is already working on.

183
00:12:02,540 --> 00:12:08,060
It also saves you from having like a competing paper out there it's just that's better all

184
00:12:08,060 --> 00:12:14,720
around so the registries are a little bit like registering a car I mean there's nobody

185
00:12:14,720 --> 00:12:22,540
really policing the quality of the protocols there is a organization called Prospero they

186
00:12:22,540 --> 00:12:28,100
are out of the University of York in the UK and they will give you like a registration

187
00:12:28,100 --> 00:12:35,500
number for your protocol and they do do like a quick look through to make sure that everything

188
00:12:35,500 --> 00:12:41,020
looks legit you know did you talk about what kind of data you're collecting did you say

189
00:12:41,020 --> 00:12:46,140
you were going to search certain number of databases it doesn't mean your systematic

190
00:12:46,140 --> 00:12:50,340
review is going to be any good but they are checking to make sure you fit all the high

191
00:12:50,340 --> 00:12:56,180
points so there is Prospero they only accept systematic review registration so there are

192
00:12:56,180 --> 00:13:02,500
other evidence syntheses like scoping reviews rapid reviews there's a ton of other types

193
00:13:02,500 --> 00:13:08,860
of evidence syntheses and those will go into another registry the most common one I use

194
00:13:08,860 --> 00:13:15,460
is the OSF which is the Open Science Foundation I believe I can never remember what the F

195
00:13:15,460 --> 00:13:24,420
stands for but they also accept registration and will give you a number and a DOI and copyright

196
00:13:24,420 --> 00:13:31,340
for it and all that good stuff so that is a way for you to say look here is my a priori

197
00:13:31,340 --> 00:13:38,020
project this is how I planned to conduct it we didn't talk about another thing that protocols

198
00:13:38,020 --> 00:13:45,140
do which is a lot of journals now require protocols if you're nodding your head I actually

199
00:13:45,140 --> 00:13:51,180
had some poor team say that they tried to get their manuscript published and they hadn't

200
00:13:51,180 --> 00:13:58,140
registered a protocol and the journal that they wanted to publish in was like no you

201
00:13:58,140 --> 00:14:02,940
needed that to do one can we do one now and I'm like no that has to be done ahead of time

202
00:14:02,940 --> 00:14:06,220
or you're out of luck you're just going to have to find another journal and that's happened

203
00:14:06,220 --> 00:14:11,060
enough times now that I try to warn people because it seems annoying right you know why

204
00:14:11,060 --> 00:14:16,860
should I just like to get on with it but yeah so a lot of journals do it keeps you honest

205
00:14:16,860 --> 00:14:22,060
keeps you on track lets you plan things out it helps you with plant your flag on your

206
00:14:22,060 --> 00:14:27,740
topic yeah they're terrific I'm a huge proponent of doing them absolutely and my experience

207
00:14:27,740 --> 00:14:32,440
they also give you feedback they'll say well think again about your analysis you haven't

208
00:14:32,440 --> 00:14:37,740
included X Y Z so it actually is is helpful feedback so it's definitely the turnaround

209
00:14:37,740 --> 00:14:41,580
gosh I feel like is a couple of weeks to months depending on how busy they are they will get

210
00:14:41,580 --> 00:14:47,940
feedback that's really nice I have not heard you've gotten the most useful feedback I've

211
00:14:47,940 --> 00:14:53,220
heard a lot of times they're just like this is not a good question and you know and zoom

212
00:14:53,220 --> 00:14:58,020
it out but it sounds like you got some really I like hearing that yeah yeah I would say

213
00:14:58,020 --> 00:15:02,980
that one experience we've had is where I think they rejected the protocol and then when we

214
00:15:02,980 --> 00:15:07,140
asked and said what was the problem then they listed the issues and we were able to go back

215
00:15:07,140 --> 00:15:12,420
and fix those which is great that's fabulous I'm gonna let my teams know that that I had

216
00:15:12,420 --> 00:15:18,940
not heard that so that's great there is actually a like a reporting that there are actually

217
00:15:18,940 --> 00:15:23,780
resources out there to help you do a protocol so this again and your librarian will help

218
00:15:23,780 --> 00:15:29,140
you again you don't have to go into this not knowing what's going on I heard from another

219
00:15:29,140 --> 00:15:34,140
clinician that I work with we were asking him you know like do you do protocols now

220
00:15:34,140 --> 00:15:41,140
what is a barrier to this and he said that if as a brand new person to systematic reviews

221
00:15:41,140 --> 00:15:46,500
I had asked him to do a protocol he would have that would have been just too big of

222
00:15:46,500 --> 00:15:51,220
a hump to get over that that was just too scary but I'd like to tell people in your

223
00:15:51,220 --> 00:15:56,700
audience that it actually isn't that big of a deal and maybe you can reassure them it's

224
00:15:56,700 --> 00:16:01,340
literally just kind of like an outline with things that you've decisions you've made and

225
00:16:01,340 --> 00:16:06,100
thoughts you've had about your project did you want to talk about that a little bit absolutely

226
00:16:06,100 --> 00:16:11,180
I will say that the most helpful resource I found which I always go back to and our

227
00:16:11,180 --> 00:16:16,180
library website is where there's a systematic review protocol template and it's a series

228
00:16:16,180 --> 00:16:22,060
of questions and you just answer the questions and it makes it so easy it helps you think

229
00:16:22,060 --> 00:16:25,500
about every aspect so you're not really creating a protocol from scratch as much as answering

230
00:16:25,500 --> 00:16:29,500
a series of questions and that becomes your protocol template so that's that's been really

231
00:16:29,500 --> 00:16:35,100
helpful yes and it doesn't it's like a couple of hours right it's not days and days or months

232
00:16:35,100 --> 00:16:40,940
or months right absolutely absolutely so now when I work with trainees who are working

233
00:16:40,940 --> 00:16:45,060
on systematic reviews that's the first thing I give them is here's the template go fill

234
00:16:45,060 --> 00:16:49,620
this out and then we have conversations around it which is great yeah it's it's really I

235
00:16:49,620 --> 00:16:53,900
love hearing you talk about it because it's sometimes a struggle to get teams to want

236
00:16:53,900 --> 00:16:58,500
to do it again it's the reading the manual there's the people who read the manual and

237
00:16:58,500 --> 00:17:05,940
the people who do not and and I know about this because you taught me well let's talk

238
00:17:05,940 --> 00:17:09,540
about let's talk about the conversation with the librarian when you say I want to do a

239
00:17:09,540 --> 00:17:14,260
systematic review and I have this protocol what information do you need to get to your

240
00:17:14,260 --> 00:17:20,380
librarian other than the protocol we don't always actually require the protocol from

241
00:17:20,380 --> 00:17:25,020
the get-go it's nice if you do if you've done that work ahead of time and you have lovely

242
00:17:25,020 --> 00:17:29,660
teens that almost always do this so but a lot of people just arrive on my doorstep but

243
00:17:29,660 --> 00:17:33,620
they're like little bag that says you know I want to do a systematic review we almost

244
00:17:33,620 --> 00:17:42,060
always talk about their question so that is probably it you might agree with this probably

245
00:17:42,060 --> 00:17:48,700
the biggest struggle is really refining your question I always warn people you're not writing

246
00:17:48,700 --> 00:17:54,940
a textbook this isn't the Moby Dick of systematic reviews you have to include all the things

247
00:17:54,940 --> 00:18:00,980
in fact it's actually better at least in my opinion is these like sports car systematic

248
00:18:00,980 --> 00:18:06,140
reviews small and fast so that you can then have another one and you can get that published

249
00:18:06,140 --> 00:18:10,700
right you can get it done in six to twelve months get that published you can start working

250
00:18:10,700 --> 00:18:15,780
on another one but the people who want to do these like big warship of systematic reviews

251
00:18:15,780 --> 00:18:19,580
I'm like oh it's gonna take you forever and by the time you finish that somebody else

252
00:18:19,580 --> 00:18:25,340
will have published it's full it's let's try to keep it fast like a sports car so we talk

253
00:18:25,340 --> 00:18:30,740
to them about their question and like find out what's how much information is out there

254
00:18:30,740 --> 00:18:35,740
should we make the question bigger sometimes we have to where do we have to make it smaller

255
00:18:35,740 --> 00:18:44,660
we ask them for what papers they already know exist for their topic that is exceptionally

256
00:18:44,660 --> 00:18:51,180
helpful for a lot of reasons one it lets the librarian we use it a lot we use it to sort

257
00:18:51,180 --> 00:18:57,100
of get to know your topic we use it to gather search terms both those mesh terms that everybody's

258
00:18:57,100 --> 00:19:02,660
always talking about the subject headings and also other ways your topic is talked about

259
00:19:02,660 --> 00:19:08,380
it lets us check our searches so we can make sure that the articles you know you want to

260
00:19:08,380 --> 00:19:13,540
include in your systematic review are and pulled up in the search there's just so many

261
00:19:13,540 --> 00:19:19,540
ways we can use it it's very very helpful so we'll ask for the question we'll ask for

262
00:19:19,540 --> 00:19:26,340
those example article maybe like exemplars or what I usually tell people is articles

263
00:19:26,340 --> 00:19:32,820
you expect to include in your in your final set and more than five would be helpful but

264
00:19:32,820 --> 00:19:36,540
if you've got three you've got three did you know that you can publish a systematic review

265
00:19:36,540 --> 00:19:43,500
with nothing in it I mean more I know isn't that interesting so we just published one

266
00:19:43,500 --> 00:19:52,180
you can this usually happens when a team is asked to do a review on a particular topic

267
00:19:52,180 --> 00:19:56,820
and then they go digging and they don't find anything that meets their criteria and so

268
00:19:56,820 --> 00:20:03,340
it ends up being a empty review what this particular team just recently happened did

269
00:20:03,340 --> 00:20:08,980
is they it was a mental health one so they used it as a call to action look nobody is

270
00:20:08,980 --> 00:20:18,140
doing this work we should be you know developing studies and testing if this if this works

271
00:20:18,140 --> 00:20:25,580
if this intervention works so that was fascinating so don't be scared off and I've also had

272
00:20:25,580 --> 00:20:32,060
a review with I had three articles in it don't be scared off by a small amount of of reviews

273
00:20:32,060 --> 00:20:35,580
now it has to be an interesting enough topic that people be interested there's nothing

274
00:20:35,580 --> 00:20:43,820
like that was if you know so what about salamanders I guess you know but that particular thing

275
00:20:43,820 --> 00:20:50,620
was really really I thought a glaring hole in the research and so they did get that published

276
00:20:50,620 --> 00:20:56,220
and it was very interesting so yes you can have zero and I would not encourage you to

277
00:20:56,220 --> 00:21:01,060
do more than how many of you have you had a lot in your reviews if you had like more

278
00:21:01,060 --> 00:21:07,780
than 50 rarely rarely I think that I feel like the numbers in my head are anywhere from

279
00:21:07,780 --> 00:21:14,100
11 to probably the most you've had is 32 yeah I try to encourage people to like stay under

280
00:21:14,100 --> 00:21:19,620
the 50s if you're like there are hundreds of papers out there about this particular

281
00:21:19,620 --> 00:21:24,420
question then I would usually tell people you might want to break your question we might

282
00:21:24,420 --> 00:21:29,420
want to focus more part of that is you're going to have to mention let's say there's

283
00:21:29,420 --> 00:21:35,540
a 250 papers this has happened to one of my teams we have 276 papers they said in our

284
00:21:35,540 --> 00:21:42,100
final set and I'm like you're gonna have to write about all 276 papers in your manuscript

285
00:21:42,100 --> 00:21:46,100
that's gonna have to go in a table in your manuscript you're gonna have to quality assess

286
00:21:46,100 --> 00:21:50,220
all 276 we're gonna have a data extract it that's gonna have to go in your references

287
00:21:50,220 --> 00:21:55,300
where your journal even allow you to have 276 things in your references and they're

288
00:21:55,300 --> 00:22:02,860
like oh so we talked about like what you could do to narrow their topic and one of the things

289
00:22:02,860 --> 00:22:06,980
they did was they actually decided on their follow-up time for this particular thing and

290
00:22:06,980 --> 00:22:11,820
once they decided on a narrower follow-up time that scooched right down to 35 papers

291
00:22:11,820 --> 00:22:16,820
which is perfect in my opinion that is great so one of the things I hear you talking about

292
00:22:16,820 --> 00:22:21,160
is really a librarian as your partner in this process not not just the person who comes

293
00:22:21,160 --> 00:22:25,860
and searches and then it's done but really a partner and so in thinking about the partnership

294
00:22:25,860 --> 00:22:31,860
so you help with the search your co-authors run off and they do the title and abstract

295
00:22:31,860 --> 00:22:36,860
screening and then they're like wow we found these maybe 72 articles that we think meet

296
00:22:36,860 --> 00:22:41,900
inclusion criteria and now we actually have to go find the articles how are you helpful

297
00:22:41,900 --> 00:22:49,220
at that point so one of the things that we can do is we can use software to help pull

298
00:22:49,220 --> 00:22:55,540
articles for that for your manuscripts so let's say there's 76 of them so we can use

299
00:22:55,540 --> 00:23:02,220
and know or Zotero to use our proxy to pull as many of those as it can as you may have

300
00:23:02,220 --> 00:23:06,680
experienced it's not always super accurate at that so we usually figure it gets about

301
00:23:06,680 --> 00:23:12,380
50 percent of that and then there's like another 50 percent that you have to go pull by hand

302
00:23:12,380 --> 00:23:19,020
or in order via document delivery interlibrary loan if there's a lot of papers what will

303
00:23:19,020 --> 00:23:24,140
we all usually do is we'll sort of divvy it up amongst the team and so everybody goes

304
00:23:24,140 --> 00:23:28,460
and pulls 20 papers or something like that so that it not it doesn't land all on one

305
00:23:28,460 --> 00:23:34,300
poor person so that's one thing that we do also the library provides a free document

306
00:23:34,300 --> 00:23:40,180
delivery so if we can get the paper for you from somewhere else besides Duke that we will

307
00:23:40,180 --> 00:23:45,720
and and you should do that when you're doing a systematic review if it's not at Duke you

308
00:23:45,720 --> 00:23:52,580
are sort of obligated to go get it if you possibly can at another institution leaving

309
00:23:52,580 --> 00:23:58,780
it out just because we don't have it in full text is is not a legitimate thing and I actually

310
00:23:58,780 --> 00:24:05,020
do I'm a reviewer for a lot of journals I'm right I'm the librarian methodology expert

311
00:24:05,020 --> 00:24:12,100
and if I see that in a manuscript it's a very quick reject now speaking of leaving out papers

312
00:24:12,100 --> 00:24:18,900
one thing that I see that that I struggle with is the language the language thing so

313
00:24:18,900 --> 00:24:24,140
if something is not in English how do you decide what to do with that what is the best

314
00:24:24,140 --> 00:24:32,580
practice versus kind of like what's most practical okay so I am a huge fan of including as many

315
00:24:32,580 --> 00:24:40,420
people as you can on your team so and a multi-disciplinary multi-national team is fantastic because then

316
00:24:40,420 --> 00:24:47,980
you have native speakers that can then go and translate those papers we always keep

317
00:24:47,980 --> 00:24:55,200
all languages in the search and this is so that you're not doing you know a western bias

318
00:24:55,200 --> 00:25:00,220
in your search for information great research is done in China great research is done in

319
00:25:00,220 --> 00:25:05,080
Africa great research is done in Germany you can't just because they don't speak English

320
00:25:05,080 --> 00:25:10,140
is a primary language that they can write a scientific paper in doesn't mean it isn't

321
00:25:10,140 --> 00:25:16,180
good science so it's a very rude of us to be like yeah they wrote it in German and it's

322
00:25:16,180 --> 00:25:20,580
probably not worth looking at right let's so we try not to have that sort of English

323
00:25:20,580 --> 00:25:26,500
language bias so we included in the search so that you can at least have a look at it

324
00:25:26,500 --> 00:25:31,660
what most people do is they don't end up unless they have someone on their team that speaks

325
00:25:31,660 --> 00:25:37,340
the language they don't end up using those papers because it's very expensive to have

326
00:25:37,340 --> 00:25:42,900
them translated several thousand dollars a paper I have a friend who does this so I've

327
00:25:42,900 --> 00:25:48,340
got the skinny from her and she's like yes very expensive so we don't usually do that

328
00:25:48,340 --> 00:25:52,900
but you've shown you're like look there's this paper in Polish and there's this paper

329
00:25:52,900 --> 00:25:58,780
in Afrikaans and they really seem because the title and abstract are in English they

330
00:25:58,780 --> 00:26:02,940
really seem to address our question but we didn't have the resources now if you know

331
00:26:02,940 --> 00:26:06,420
you're going to be doing it maybe scoop a little bit of money in your grant off to do

332
00:26:06,420 --> 00:26:11,860
translations but if this is a surprise SR and you didn't have funding for it what you

333
00:26:11,860 --> 00:26:17,340
can say is here are the papers that we really think would be useful and if someone in your

334
00:26:17,340 --> 00:26:22,700
audience wants to spend the money to translate they can or if they speak the language they

335
00:26:22,700 --> 00:26:28,340
can go have a look at it but you're that you're addressing that it's there and not just ignoring

336
00:26:28,340 --> 00:26:32,820
it I usually encourage people to put in like a little table that said these are some papers

337
00:26:32,820 --> 00:26:38,620
that were interesting and we couldn't translate them but look we found them and I think that

338
00:26:38,620 --> 00:26:42,900
just makes the team look great because you're trying really hard to minimize the bias in

339
00:26:42,900 --> 00:26:44,700
your review.

340
00:26:44,700 --> 00:26:48,340
That's awesome thank you thank you for that perspective for giving us the ideal and the

341
00:26:48,340 --> 00:26:54,380
practical now the I think last question I want to ask is about authorship okay your

342
00:26:54,380 --> 00:26:59,380
librarians done all this great work and then you go to write the paper how should you be

343
00:26:59,380 --> 00:27:03,700
thinking about including your librarian as a co-author with you in the process?

344
00:27:03,700 --> 00:27:12,300
I love that you asked this bless you so we're right that if you think about and I guarantee

345
00:27:12,300 --> 00:27:16,540
that if anyone works with a librarian you're going to definitely want to give them authorship

346
00:27:16,540 --> 00:27:21,400
because we do quite a bit of work we're not just you know it doesn't take us five minutes

347
00:27:21,400 --> 00:27:26,900
to run the search and then you never see us again we help you with all your methodology

348
00:27:26,900 --> 00:27:31,940
we help you make decisions about what risk of bias tool you might want to use right all

349
00:27:31,940 --> 00:27:37,100
those things plus we will write up your methodology for you for the bits that we do so we can

350
00:27:37,100 --> 00:27:44,180
say this is how we conducted our search we provide you reproducible searches that are

351
00:27:44,180 --> 00:27:49,300
transparent that can go in your supplementary material I offer a lot of advice for a lot

352
00:27:49,300 --> 00:27:54,500
of other pieces of the review I offer a lot of advice on data extraction here's what I

353
00:27:54,500 --> 00:28:00,940
would do here's best practices so we usually just say you know we're a team member we don't

354
00:28:00,940 --> 00:28:06,780
have to be in on all the nitty-gritty things because I don't know anything about hematology

355
00:28:06,780 --> 00:28:12,300
or whatever your particular topic is you don't want me screening you don't want me data collecting

356
00:28:12,300 --> 00:28:16,460
because I don't know what I'm doing but I will be there to help you with the entire

357
00:28:16,460 --> 00:28:23,220
like process of doing a review so I think that work I think genuinely a lot of times

358
00:28:23,220 --> 00:28:29,140
are even part of conceptualizing your question so I do think that that helps us deserve being

359
00:28:29,140 --> 00:28:35,260
part of the author team and most of my teams have said oh absolutely you we feel like you've

360
00:28:35,260 --> 00:28:40,380
deserved it I sometimes every now and again I'll have one that was just super super easy

361
00:28:40,380 --> 00:28:43,740
and they didn't need me and I'm like you don't have to include me so if I feel like I didn't

362
00:28:43,740 --> 00:28:48,620
do the work come say have at it you don't need me on this but I feel like it also makes

363
00:28:48,620 --> 00:28:53,660
your review look really good look we used a professional librarian so there's been someone

364
00:28:53,660 --> 00:28:58,380
here sort of keeping an eye and then one of the things that I do is I go through the manuscript

365
00:28:58,380 --> 00:29:04,180
at the very end because I'm also a reviewer and I say this is what I look at when I'm

366
00:29:04,180 --> 00:29:09,420
asked to look at a review what things did you hit what things are missing and so I feel

367
00:29:09,420 --> 00:29:15,340
like just that helps us sort of get over the finish line of publication absolutely absolutely

368
00:29:15,340 --> 00:29:22,980
thank you thank you so I'm thinking my final question has to do with tools what are some

369
00:29:22,980 --> 00:29:29,540
favorite tools software tools those kinds of tools gosh we have some really cool ones

370
00:29:29,540 --> 00:29:37,500
so at Duke we subscribe to software called covenants which is a systematic review screening

371
00:29:37,500 --> 00:29:42,660
software but also helps with the data extraction it doesn't do it for you we have not gotten

372
00:29:42,660 --> 00:29:47,540
there yet people always ask me that they're like does it do it for you and I'm like not

373
00:29:47,540 --> 00:29:52,780
yet not yet but what people are working on it they're trying but it does really help

374
00:29:52,780 --> 00:30:00,220
with the sort of data management of your systematic review so helps you your yes and no's tagging

375
00:30:00,220 --> 00:30:06,220
things deciding on your criteria all that stuff and it provides a really pretty prisma

376
00:30:06,220 --> 00:30:12,140
flow diagram for you at the end and all the good stuff plus forms for keeping all your

377
00:30:12,140 --> 00:30:17,380
collecting all your data and keeping it all in one ecosystem so we love that software

378
00:30:17,380 --> 00:30:23,180
there is other software out there we just happen to have this when we find this to be

379
00:30:23,180 --> 00:30:30,780
the most user friendly other tools I really like to use are there is this is super nerdy

380
00:30:30,780 --> 00:30:37,420
but there is a software out there called the polyglot translator which allows us to take

381
00:30:37,420 --> 00:30:43,100
the initial search that we do and it will translate it for you it doesn't do a spectacular

382
00:30:43,100 --> 00:30:49,540
job there's things it doesn't do for you but it does actually save some time with syntax

383
00:30:49,540 --> 00:30:52,620
so I teach some of the students about this and I say here's the things you're going

384
00:30:52,620 --> 00:30:56,620
to have to look out for but if you don't want to work with a library and this will sort

385
00:30:56,620 --> 00:31:03,220
of get you halfway there there is a super cool tool out there called citation chaser

386
00:31:03,220 --> 00:31:08,180
so one of the things that you're expected to do is when you get your final set of papers

387
00:31:08,180 --> 00:31:13,460
so let's say we have 10 papers at the end you're expected to look through the references

388
00:31:13,460 --> 00:31:18,540
of that we used to be called hand searching but what they now call it I've heard a lot

389
00:31:18,540 --> 00:31:23,100
of different things I call it forward and backward citation searching so you look through

390
00:31:23,100 --> 00:31:27,780
the papers just to sort of make sure that there isn't anything living in the references

391
00:31:27,780 --> 00:31:33,140
that might fit your topic but then there's also papers out there since let's say there's

392
00:31:33,140 --> 00:31:38,820
a paper that was done in 2004 and it's been cited by 15 other papers you'll want to make

393
00:31:38,820 --> 00:31:43,260
sure that you've looked at those 15 papers to see if they have if they answer your question

394
00:31:43,260 --> 00:31:47,980
so there's a software out there there's a couple of them the one I like is called citation

395
00:31:47,980 --> 00:31:52,700
chaser and you can actually just put in the doi of the papers and it will go get all the

396
00:31:52,700 --> 00:31:58,180
references and all the cited papers and and then you can just turn around and drop it

397
00:31:58,180 --> 00:32:03,460
into that software that covenant software and it keeps track of everything I just it's

398
00:32:03,460 --> 00:32:08,940
really really nice and it's super easy to use so yes those are some faves there's some

399
00:32:08,940 --> 00:32:13,060
other ones I like to use those are the ones that I think are probably used the most often

400
00:32:13,060 --> 00:32:17,380
that's awesome citation chaser is one I had yeah so that's that's pretty awesome thank

401
00:32:17,380 --> 00:32:22,740
you next time when you hit that spot we will we will touch it we will touch it I'll show

402
00:32:22,740 --> 00:32:27,420
you how now for anyone who's as they're coming to the end of the podcast I'm just wondering

403
00:32:27,420 --> 00:32:33,580
what major thing about about systematic reviews haven't we talked about that you want to highlight

404
00:32:33,580 --> 00:32:38,900
or maybe we've talked about it you just want to emphasize it we haven't really talked about

405
00:32:38,900 --> 00:32:44,980
the process of doing a risk of bias or quality assessment with papers that we mentioned it

406
00:32:44,980 --> 00:32:49,300
super briefly but that's one of the defining characteristics of doing a systematic review

407
00:32:49,300 --> 00:32:56,220
and I find that a lot of teams don't realize that that has to be done you can choose to

408
00:32:56,220 --> 00:33:02,140
do a review and not do the quality assessment risk of bias but you may not call it a systematic

409
00:33:02,140 --> 00:33:07,820
review at that point so you can call it a literature review a narrative review a state

410
00:33:07,820 --> 00:33:12,220
of the art something but you may not call it a systematic review without doing that

411
00:33:12,220 --> 00:33:18,340
risk of bias portion and what that is is just looking at the paper and and making sure you

412
00:33:18,340 --> 00:33:23,580
can only tell what they reported but there it's relatively easy there are a checklist

413
00:33:23,580 --> 00:33:28,780
usually that says you know did they ask a good question did they I'm going to use randomized

414
00:33:28,780 --> 00:33:35,260
control trials as an example did they randomize was the allocation blinded you know how did

415
00:33:35,260 --> 00:33:39,820
they crunch the data was you know all those kinds of things just so you can look at how

416
00:33:39,820 --> 00:33:47,780
well conducted this is really just how well reported a study was and so that way when

417
00:33:47,780 --> 00:33:53,500
you're summarizing everything at the end you can say and you have to include I've seen

418
00:33:53,500 --> 00:33:58,740
this done but people will throw out the poor scores and I'm like no no no no no because

419
00:33:58,740 --> 00:34:03,540
you don't do that either it's a little bit like getting rid of patients that don't do

420
00:34:03,540 --> 00:34:11,020
well in your study that this is no no bueno no you still want to report on them because

421
00:34:11,020 --> 00:34:15,380
someone might say well I thought that Smith article was great you're like no actually

422
00:34:15,380 --> 00:34:20,860
we looked at the Smith article and there's a really some weaknesses some really a high

423
00:34:20,860 --> 00:34:27,580
chance of bias creeping into the paper because of X Y or Z so that portion is actually really

424
00:34:27,580 --> 00:34:34,460
really important and then you can say look the quality of the research done on this topic

425
00:34:34,460 --> 00:34:39,660
is really quite good or it's a middle in or you know nobody's doing really high quality

426
00:34:39,660 --> 00:34:45,540
research on this yet so it's just another portion of a review that makes it a systematic

427
00:34:45,540 --> 00:34:49,780
review that makes it super valuable if you do a scoping review you do not have to do

428
00:34:49,780 --> 00:34:55,660
it but that whether or not is a scoping or a systematic review really depends on your

429
00:34:55,660 --> 00:34:59,780
question and a librarian can help you sort that out.

430
00:34:59,780 --> 00:35:04,020
That is super awesome I've learned new things today Leila thank you.

431
00:35:04,020 --> 00:35:05,940
You're welcome.

432
00:35:05,940 --> 00:35:10,020
Well yeah that's been a it's been a great great discussion about systematic reviews

433
00:35:10,020 --> 00:35:16,380
I want to thank you for coming on and sharing your expertise I wonder why is there a caucus

434
00:35:16,380 --> 00:35:23,100
for systematic reviews what makes it so so so impressive because you told me it's a it's

435
00:35:23,100 --> 00:35:28,500
the second largest group relative to other committees or other groups that you have within

436
00:35:28,500 --> 00:35:30,180
the library association.

437
00:35:30,180 --> 00:35:36,140
Yeah we have almost 700 members I think the reason one systematic reviews are super fun

438
00:35:36,140 --> 00:35:42,300
to do this is also a skill that librarians are really reviews as you probably know just

439
00:35:42,300 --> 00:35:48,500
getting more and more popular and medical professions are showing up at a lot of universities

440
00:35:48,500 --> 00:35:54,800
and also at you know pharmaceutical companies and right there's lots of places that people

441
00:35:54,800 --> 00:36:01,860
are doing literature review type research and so and librarians are really important

442
00:36:01,860 --> 00:36:07,820
to that process and so more and more librarians are being asked to participate and so we want

443
00:36:07,820 --> 00:36:14,300
to be good at this for our clinicians for our researchers and so there's a lot of sharing

444
00:36:14,300 --> 00:36:20,460
of knowledge we are doing a whole one of our librarians here I invited her to come speak

445
00:36:20,460 --> 00:36:25,140
about including gray literature and their systematic reviews and like how to find that

446
00:36:25,140 --> 00:36:34,740
so it's just a way for us to a little bit of shop talk and a little bit of some commiseration

447
00:36:34,740 --> 00:36:39,220
and also this is what's happening on the sort of bleeding edge this is what people are doing

448
00:36:39,220 --> 00:36:46,260
like is AI there yet can we use AI yet to do a systematic reason not yet I've had a

449
00:36:46,260 --> 00:36:51,060
couple teams asked me recently and I like not yet it's getting there but we're a couple

450
00:36:51,060 --> 00:36:56,400
years out so those kinds of things now that's really awesome I think systematic reviews

451
00:36:56,400 --> 00:37:02,220
are very valuable and so having the right people in the room to do them especially your

452
00:37:02,220 --> 00:37:06,700
dedicated amazing librarian is important so I want to thank you for coming on the show

453
00:37:06,700 --> 00:37:11,940
to talk about it today thank you so much this is really fun I will come back and talk as

454
00:37:11,940 --> 00:37:19,100
long as you want me to I will invite you again okay all right everyone you've heard it from

455
00:37:19,100 --> 00:37:24,660
Layla Ledbetter talking to you about how to do systematic reviews in the best way possible

456
00:37:24,660 --> 00:37:28,720
Layla if people have questions and they just want to connect with you how can they reach

457
00:37:28,720 --> 00:37:35,300
you you may go to the Duke University Medical Center Libraries website and you may find

458
00:37:35,300 --> 00:37:40,780
me there in the staff list we also have a guide on systematic reviews it's pretty popular

459
00:37:40,780 --> 00:37:46,060
if you google Duke systematic reviews you can reach us that way and we are happy to

460
00:37:46,060 --> 00:37:51,220
answer any questions that you have about the process and direct you to resources on how

461
00:37:51,220 --> 00:37:56,580
to learn how to do that there's a Coursera course done by Johns Hopkins that's fantastic

462
00:37:56,580 --> 00:38:01,700
so there's lots of places we can direct you if you'd like to learn more sounds good well

463
00:38:01,700 --> 00:38:07,140
thank you so much Layla you're welcome bye all right everyone have a great time we'll

464
00:38:07,140 --> 00:38:17,180
see you again on the next episode

465
00:38:17,180 --> 00:38:22,540
thanks for listening to this episode of the clinician researcher podcast where academic

466
00:38:22,540 --> 00:38:28,260
clinicians learn the skills to build their own research program whether or not they have

467
00:38:28,260 --> 00:38:34,020
a mentor if you found the information in this episode to be helpful don't keep it all to

468
00:38:34,020 --> 00:38:41,380
yourself someone else needs to hear it so take a minute right now and share it as you

469
00:38:41,380 --> 00:38:46,780
share this episode you become part of our mission to help launch a new generation of

470
00:38:46,780 --> 00:38:52,820
clinician researchers who make transformative discoveries that change the way we do health

471
00:38:52,820 --> 00:39:17,620
well