1 00:00:00,000 --> 00:00:13,800 Welcome to LITE Bites, an occasional podcast from Leeds Institute for Teaching Excellence 2 00:00:13,800 --> 00:00:15,420 at the University of Leeds. 3 00:00:15,420 --> 00:00:18,520 Episodes will be hosted by members of the LITE team. 4 00:00:18,520 --> 00:00:24,880 And we'll be showcasing the scholarship of teaching and learning from across the university. 5 00:00:24,880 --> 00:00:33,760 Hello and welcome to another episode of the LITE Bites podcast. 6 00:00:33,760 --> 00:00:38,740 It's Emma Peasland here, research and impact officer in Leeds Institute for Teaching Excellence 7 00:00:38,740 --> 00:00:40,520 at the University of Leeds. 8 00:00:40,520 --> 00:00:46,160 And today I'm chuffed to be joined by Samantha Stark, who is a course design manager in the 9 00:00:46,160 --> 00:00:51,880 digital education service here at the university and has recently completed a Light fellowship 10 00:00:51,880 --> 00:00:56,440 evaluating the impact of a pedagogical framework for a fully online education. 11 00:00:56,440 --> 00:00:57,440 Hi Samantha. 12 00:00:57,440 --> 00:00:58,440 Hi, thanks for having me. 13 00:00:58,440 --> 00:01:02,200 Well, it's great to have you here. 14 00:01:02,200 --> 00:01:06,240 And I just mentioned that you work in the digital education service here. 15 00:01:06,240 --> 00:01:11,960 So I wondered if a good place to start would be for you to tell us a little bit about what 16 00:01:11,960 --> 00:01:14,360 you do in the digital education service. 17 00:01:14,360 --> 00:01:15,360 Yes. 18 00:01:15,360 --> 00:01:17,880 So the digital education service, it's a bit of a mouthful. 19 00:01:17,880 --> 00:01:21,880 So we tend to call it DES for short, just to help you out. 20 00:01:21,880 --> 00:01:25,400 And there are many different departments in DES that do completely different things. 21 00:01:25,400 --> 00:01:27,480 We're a very big department now. 22 00:01:27,480 --> 00:01:34,060 But the team in which I used to work up until the end of my Light fellowship works with 23 00:01:34,060 --> 00:01:40,040 academics from across the university to design master's degrees that students will study 24 00:01:40,040 --> 00:01:41,040 fully online. 25 00:01:41,040 --> 00:01:48,520 And so we have completed master's degrees in a wide variety of disciplines, engineering, 26 00:01:48,520 --> 00:01:53,000 artificial intelligence, sustainable business leadership, digital design communication, 27 00:01:53,000 --> 00:01:57,720 international fashion marketing and design management, data science. 28 00:01:57,720 --> 00:02:00,880 And the portfolio is increasing all the time. 29 00:02:00,880 --> 00:02:04,220 So that's what that team in the in DES does. 30 00:02:04,220 --> 00:02:07,960 And that's the team I was working for when I conducted this research. 31 00:02:07,960 --> 00:02:09,320 And what do you do now? 32 00:02:09,320 --> 00:02:14,640 So now I'm still at DES, but I've just moved to a different team. 33 00:02:14,640 --> 00:02:22,520 The focus of that team is to, we kind of make a wide variety of different courses to support 34 00:02:22,520 --> 00:02:24,300 LEAD students. 35 00:02:24,300 --> 00:02:29,440 And also we create courses for short courses for future learn. 36 00:02:29,440 --> 00:02:34,320 Some examples of courses we've done, I worked on an academic listening course to help students 37 00:02:34,320 --> 00:02:38,320 to prepare for studying at the LEADS University. 38 00:02:38,320 --> 00:02:45,640 And we've also done a course for like supporting healthy relationships for students. 39 00:02:45,640 --> 00:02:49,480 So basically we're focused on enhancing student education at the University of Leeds, but 40 00:02:49,480 --> 00:02:53,680 also helping to attract people to the university as well. 41 00:02:53,680 --> 00:02:54,680 Very interesting. 42 00:02:54,680 --> 00:02:58,720 It's really diverse kind of range of activity going on in DES. 43 00:02:58,720 --> 00:02:59,720 Absolutely. 44 00:02:59,720 --> 00:03:00,720 Excellent. 45 00:03:00,720 --> 00:03:04,360 So let's talk a little bit about your Light fellowship then, since we're on the Light 46 00:03:04,360 --> 00:03:05,360 Bytes podcast. 47 00:03:05,360 --> 00:03:11,080 So do you want to start by telling us a little bit about what it was that you were investigating 48 00:03:11,080 --> 00:03:13,880 and why you wanted to do that? 49 00:03:13,880 --> 00:03:14,880 Sure. 50 00:03:14,880 --> 00:03:23,960 So when designing master's degrees, it can be quite difficult because academics might 51 00:03:23,960 --> 00:03:28,480 be used to teaching their module face to face. 52 00:03:28,480 --> 00:03:30,640 They have to change it to online delivery. 53 00:03:30,640 --> 00:03:36,320 And that means changing a lot of things about how the course is designed and the type of 54 00:03:36,320 --> 00:03:38,560 content that we can include. 55 00:03:38,560 --> 00:03:46,320 So for the master's modules, they involve six weeks of teaching and each week involves 56 00:03:46,320 --> 00:03:48,880 about 20 hours of learning. 57 00:03:48,880 --> 00:03:51,600 And then after the six weeks, learners have a week to prepare. 58 00:03:51,600 --> 00:03:55,720 And then in week eight, they submit their final assessment and then they start their 59 00:03:55,720 --> 00:03:57,120 next module. 60 00:03:57,120 --> 00:04:04,000 So the way we used to design master's modules is that it was almost like a blank paper approach. 61 00:04:04,000 --> 00:04:10,600 So each week of learning, which is a unit of 20 hours, was split into like four lessons. 62 00:04:10,600 --> 00:04:12,680 But that was kind of the only structure that was there. 63 00:04:12,680 --> 00:04:14,920 And each week ended with a webinar. 64 00:04:14,920 --> 00:04:19,240 So there just ended up being 20 hours that we had to fill. 65 00:04:19,240 --> 00:04:23,440 And it meant that we ended up with too much work. 66 00:04:23,440 --> 00:04:26,000 The design and development process was taking too long. 67 00:04:26,000 --> 00:04:30,240 We ended up with too much content, which meant the academics were overloaded. 68 00:04:30,240 --> 00:04:33,360 We and the team were overloaded and the students were overloaded. 69 00:04:33,360 --> 00:04:39,440 It meant there was a predominance of passive learning and not a lot of active learning. 70 00:04:39,440 --> 00:04:43,240 So it wasn't a good learning experience and it wasn't a good design and development experience 71 00:04:43,240 --> 00:04:44,240 either. 72 00:04:44,240 --> 00:04:47,560 So we wanted to improve that. 73 00:04:47,560 --> 00:04:54,920 And when a former colleague, Sarah Briggs, joined our team, she had sort of used the 74 00:04:54,920 --> 00:04:58,520 design framework in her previous role before she came to the university. 75 00:04:58,520 --> 00:05:01,400 So she suggested that as a way forward. 76 00:05:01,400 --> 00:05:06,040 And I got heavily involved in the design and implementation of that framework. 77 00:05:06,040 --> 00:05:09,120 So the new framework was very different from what we've done before. 78 00:05:09,120 --> 00:05:14,560 So instead of having 20 hours of learning split into four lessons, we had this pedagogical 79 00:05:14,560 --> 00:05:21,320 framework, which was basically meant that we could have more active learning interventions. 80 00:05:21,320 --> 00:05:26,880 So it split into three phases, prepare, apply and reflect. 81 00:05:26,880 --> 00:05:30,280 And the prepare section was about two hours. 82 00:05:30,280 --> 00:05:32,960 And that was where a lot of teaching input happened. 83 00:05:32,960 --> 00:05:39,760 So there might have been short videos, articles to read, maybe other sorts of infographics 84 00:05:39,760 --> 00:05:43,160 to interact with. 85 00:05:43,160 --> 00:05:48,780 And that prepared them for a key sort of a substantive activity, which would take learners 86 00:05:48,780 --> 00:05:51,200 between three and five hours to complete. 87 00:05:51,200 --> 00:05:53,080 And we would create like a brief for that. 88 00:05:53,080 --> 00:05:57,600 So the learners had to take what they'd learned in the prepare phase and use it to actually 89 00:05:57,600 --> 00:06:02,080 achieve something that would be relevant to something that they would do in real life 90 00:06:02,080 --> 00:06:05,000 and would have a clear outcome. 91 00:06:05,000 --> 00:06:09,760 So for the design degrees, for example, it would be like, I don't know, create some 92 00:06:09,760 --> 00:06:12,640 kind of a wire frame. 93 00:06:12,640 --> 00:06:17,560 Or they had to do a SWOT analysis of something for one of the international fashion marketing 94 00:06:17,560 --> 00:06:18,560 models. 95 00:06:18,560 --> 00:06:20,740 So it had to be something that had a clear outcome. 96 00:06:20,740 --> 00:06:23,680 And if it could be related to real life, then that would be good because then the learners 97 00:06:23,680 --> 00:06:27,760 have almost a portfolio of things that they've done that they'll be able to show to future 98 00:06:27,760 --> 00:06:30,360 employers potentially. 99 00:06:30,360 --> 00:06:36,760 And so once they've completed that, they share the outcome with their peers for feedback, 100 00:06:36,760 --> 00:06:39,920 for them to discuss and learn from each other. 101 00:06:39,920 --> 00:06:46,040 And then the reflect stage was about four hours, a one hour webinar, a one hour preparation 102 00:06:46,040 --> 00:06:49,920 for that webinar, and then two hours for reflection at the end. 103 00:06:49,920 --> 00:06:55,160 In addition to those three phases, there was also two hours allowed for a substantive discussion 104 00:06:55,160 --> 00:06:56,160 on a different point. 105 00:06:56,160 --> 00:07:00,080 So that meant there were kind of two points of discussion in each week. 106 00:07:00,080 --> 00:07:04,080 There was a separate substantive discussion where the module leader set questions that 107 00:07:04,080 --> 00:07:07,400 we would design as part of the design and development process. 108 00:07:07,400 --> 00:07:11,280 But there was also this kind of student led discussion that happened once they'd completed 109 00:07:11,280 --> 00:07:15,200 their substantive activity where they're sharing their outcome with each other and getting 110 00:07:15,200 --> 00:07:16,200 feedback. 111 00:07:16,200 --> 00:07:23,240 So I'm curious to know how much of this is students in these programs attending synchronous 112 00:07:23,240 --> 00:07:29,720 live sessions and how much of it is working on their own self-directed learning? 113 00:07:29,720 --> 00:07:30,720 Yeah. 114 00:07:30,720 --> 00:07:32,480 So it's kind of a mix. 115 00:07:32,480 --> 00:07:38,280 So the synchronous is the live webinar at the end of each unit, end of each week. 116 00:07:38,280 --> 00:07:41,120 The rest of it is asynchronous. 117 00:07:41,120 --> 00:07:45,520 That's why they're not sharing their outcomes in an online seminar or something. 118 00:07:45,520 --> 00:07:47,280 They're sharing their outcomes in a discussion board. 119 00:07:47,280 --> 00:07:52,000 They can all share and give feedback asynchronously because they're all in different time zones. 120 00:07:52,000 --> 00:07:53,800 All our students are from all over the world. 121 00:07:53,800 --> 00:07:57,040 So organising synchronous is quite difficult. 122 00:07:57,040 --> 00:08:01,400 And another thing to note about our students is that they are older than the on-campus 123 00:08:01,400 --> 00:08:02,880 master's students might be. 124 00:08:02,880 --> 00:08:05,120 They're usually around their mid-30s. 125 00:08:05,120 --> 00:08:06,120 They're working. 126 00:08:06,120 --> 00:08:09,680 So they have to kind of fit their studies around what they do. 127 00:08:09,680 --> 00:08:10,680 OK. 128 00:08:10,680 --> 00:08:13,800 And so you had this new kind of framework that you introduced. 129 00:08:13,800 --> 00:08:17,640 And so how did your fellowship work to kind of evaluate that? 130 00:08:17,640 --> 00:08:18,640 Yes. 131 00:08:18,640 --> 00:08:22,880 So we often introduce things that we think, oh, this is amazing, but we don't necessarily 132 00:08:22,880 --> 00:08:23,880 get the feedback on it. 133 00:08:23,880 --> 00:08:29,680 So I was really keen to think, well, to find out how it was actually landing with our students 134 00:08:29,680 --> 00:08:33,880 and with the academics that we worked with, but also within DES itself. 135 00:08:33,880 --> 00:08:36,520 Were my colleagues finding it easy to work with? 136 00:08:36,520 --> 00:08:41,280 And was there any way that it could be improved to make it better and make it work better 137 00:08:41,280 --> 00:08:42,800 for everybody? 138 00:08:42,800 --> 00:08:47,420 So you mentioned that you were wanting to evaluate how the framework or this new process 139 00:08:47,420 --> 00:08:50,080 landed with students and also with DES staff. 140 00:08:50,080 --> 00:08:54,440 And was there anything kind of collecting data from the academics who were involved 141 00:08:54,440 --> 00:08:56,040 in that as well? 142 00:08:56,040 --> 00:09:00,320 So what we wanted to find out specifically was, you know, was this framework helping 143 00:09:00,320 --> 00:09:02,360 to reduce the amount of content? 144 00:09:02,360 --> 00:09:05,800 Therefore was it speeding up the design and development process? 145 00:09:05,800 --> 00:09:09,360 And you know, was it meaning that the students had a better learning experience? 146 00:09:09,360 --> 00:09:14,360 And I wanted to find out a diverse range of opinions. 147 00:09:14,360 --> 00:09:18,200 So I interviewed academics on a one-to-one basis. 148 00:09:18,200 --> 00:09:22,960 I had semi-structured interviews both with module leaders and with program leaders, because 149 00:09:22,960 --> 00:09:28,320 the program leaders can see, you know, all the modules and the program leaders have seen 150 00:09:28,320 --> 00:09:32,000 the modules that were not designed with the framework and those that were. 151 00:09:32,000 --> 00:09:34,840 So they could provide a useful point of comparison. 152 00:09:34,840 --> 00:09:39,200 Some of the module leaders have also worked on non-framework and framework modules, but 153 00:09:39,200 --> 00:09:41,360 some of them hadn't. 154 00:09:41,360 --> 00:09:47,320 So I was speaking to all of them, as many as I could, during my fellowship. 155 00:09:47,320 --> 00:09:50,120 So those were the one-to-one semi-structured interviews. 156 00:09:50,120 --> 00:09:54,560 Then for DES staff, I ran two focus groups. 157 00:09:54,560 --> 00:09:56,800 Well, I designed the questions for two focus groups. 158 00:09:56,800 --> 00:10:02,040 Haley Bullard from Light actually ran them for me, which was very helpful. 159 00:10:02,040 --> 00:10:07,120 And I also designed an anonymous online questionnaire for the students. 160 00:10:07,120 --> 00:10:13,040 It had to be online because, as I mentioned before, our students are all over the world, 161 00:10:13,040 --> 00:10:14,240 different time zones, et cetera. 162 00:10:14,240 --> 00:10:18,320 It would have been quite difficult to arrange to meet up with them. 163 00:10:18,320 --> 00:10:25,400 And I wanted it to be anonymous so that they felt more able to express their opinions honestly. 164 00:10:25,400 --> 00:10:28,840 So yeah, that was an online questionnaire I did for them. 165 00:10:28,840 --> 00:10:32,000 And what did you want to find out from the students specifically? 166 00:10:32,000 --> 00:10:35,920 Well, again, I couldn't ask the students to say, hey, was this module better than the 167 00:10:35,920 --> 00:10:41,960 other one, because there was no way of knowing if that student had done that module. 168 00:10:41,960 --> 00:10:45,280 And I couldn't track the students and ask the same students the same questions. 169 00:10:45,280 --> 00:10:51,280 So all I could do was collect, ask all the students the same question and ask which module 170 00:10:51,280 --> 00:10:52,280 it was from. 171 00:10:52,280 --> 00:10:55,960 So I could see if their response is related to a non-framework or a framework, and then 172 00:10:55,960 --> 00:10:58,720 just see if there were any trends. 173 00:10:58,720 --> 00:11:04,040 Is the feedback generally more positive or more negative about non-framework or framework? 174 00:11:04,040 --> 00:11:06,160 OK, that's really interesting. 175 00:11:06,160 --> 00:11:09,920 So now might be a good time to ask what you found out. 176 00:11:09,920 --> 00:11:10,920 Yes. 177 00:11:10,920 --> 00:11:14,200 So got some really, really useful feedback. 178 00:11:14,200 --> 00:11:19,640 So in relation to academic staff, there were many positive comments. 179 00:11:19,640 --> 00:11:24,680 They found that the structured approach was really helpful, that the framework was simple 180 00:11:24,680 --> 00:11:29,280 and logical, that it did help to save time, and that the substantive activities that we 181 00:11:29,280 --> 00:11:31,160 designed were extremely valuable. 182 00:11:31,160 --> 00:11:33,960 So that was the positive feedback from them. 183 00:11:33,960 --> 00:11:39,080 But there were some areas of concern. 184 00:11:39,080 --> 00:11:44,960 One bit of feedback actually from a module leader that led a non-framework module said 185 00:11:44,960 --> 00:11:47,120 that too much structure can kill content. 186 00:11:47,120 --> 00:11:53,120 So I think there was a real concern that this structure would kind of be imposed on different 187 00:11:53,120 --> 00:11:56,280 disciplines when it may not be suitable. 188 00:11:56,280 --> 00:11:59,260 And so flexibility is really important. 189 00:11:59,260 --> 00:12:04,200 Another concern was, yes, although there's less content perhaps for module leaders to 190 00:12:04,200 --> 00:12:08,760 write, designing a decent substantive activity that the learners are going to be able to 191 00:12:08,760 --> 00:12:13,880 do independently for three to five hours takes a lot of time to make sure that they've got 192 00:12:13,880 --> 00:12:17,840 what they need and that it's well designed and that the module leader is not going to 193 00:12:17,840 --> 00:12:24,680 end up having to field lots of questions from the students while the module is running. 194 00:12:24,680 --> 00:12:28,760 Another concern was, as I mentioned before, our learners, they have jobs, they might have 195 00:12:28,760 --> 00:12:34,360 children that they're fitting this in around their jobs, et cetera. 196 00:12:34,360 --> 00:12:38,680 And so it could be difficult for them to find the chunk of time that they need to do the 197 00:12:38,680 --> 00:12:42,840 substantive activity because they might be just dipping in and out, doing half an hour 198 00:12:42,840 --> 00:12:44,360 here, half an hour there. 199 00:12:44,360 --> 00:12:47,760 And so for something that's going to take three to five hours, it might be quite difficult 200 00:12:47,760 --> 00:12:51,440 for them to find that time. 201 00:12:51,440 --> 00:12:57,280 Regarding workload, the framework itself doesn't necessarily reduce it because the workload 202 00:12:57,280 --> 00:12:58,720 is a complex thing. 203 00:12:58,720 --> 00:13:00,520 It can depend on the individual. 204 00:13:00,520 --> 00:13:03,060 It can depend on the experience that they've had. 205 00:13:03,060 --> 00:13:04,640 Have they done a module before? 206 00:13:04,640 --> 00:13:06,240 If they have, it's going to be a lot quicker. 207 00:13:06,240 --> 00:13:10,160 They're going to find it less work, whereas somebody else is going to find it more. 208 00:13:10,160 --> 00:13:13,360 So the framework by itself is not going to solve that issue. 209 00:13:13,360 --> 00:13:14,360 Yeah. 210 00:13:14,360 --> 00:13:19,400 And I guess for some people, perhaps that designing that substantive activity might 211 00:13:19,400 --> 00:13:22,600 take a bit more time than for others. 212 00:13:22,600 --> 00:13:24,480 And then you mentioned about discipline as well. 213 00:13:24,480 --> 00:13:28,760 So did you find that using the framework was particularly effective for some disciplines 214 00:13:28,760 --> 00:13:30,240 and not for others? 215 00:13:30,240 --> 00:13:31,240 Yeah. 216 00:13:31,240 --> 00:13:37,280 It's difficult for me to say that because I had to base my research on the modules that 217 00:13:37,280 --> 00:13:40,240 were running at the time of my Light Fellowship. 218 00:13:40,240 --> 00:13:47,580 And so I was able to collect data on a couple of modules from Sustainable Business Leadership, 219 00:13:47,580 --> 00:13:49,920 one of which used the framework. 220 00:13:49,920 --> 00:13:54,360 And that was very much about data analytics. 221 00:13:54,360 --> 00:13:59,440 And there was maybe one or two that was from digital design communication, so much more 222 00:13:59,440 --> 00:14:01,000 design focused. 223 00:14:01,000 --> 00:14:06,520 And then International Fashion Marketing and Design Management, which was a kind of maybe 224 00:14:06,520 --> 00:14:08,640 a bit of design, but maybe a bit more business as well. 225 00:14:08,640 --> 00:14:16,840 So I found it did seem to lend itself in some ways quite well to design, but in other ways, 226 00:14:16,840 --> 00:14:23,960 not because when learners are studying face to face, something I found when I was designing 227 00:14:23,960 --> 00:14:28,400 myself on that module is that the module leader said, normally they're working on their final 228 00:14:28,400 --> 00:14:30,560 assessment in class. 229 00:14:30,560 --> 00:14:32,800 They're getting feedback from each other on the final assessments. 230 00:14:32,800 --> 00:14:36,200 Their coursework is their assessment. 231 00:14:36,200 --> 00:14:41,600 But because of academic integrity rules and the nature of online learning, we can't make 232 00:14:41,600 --> 00:14:44,960 the substantive activities the final assessment. 233 00:14:44,960 --> 00:14:48,200 And so the substantive activities, they're going to mirror what they need to do in the 234 00:14:48,200 --> 00:14:52,600 final assessment, but it's not actually going to be what the final assessment is, which 235 00:14:52,600 --> 00:14:57,280 then made it quite difficult for the module leader because of the way they had to design 236 00:14:57,280 --> 00:14:58,680 it for online learning. 237 00:14:58,680 --> 00:15:02,640 But that would have been the case, whether it was the pedagogical framework or not. 238 00:15:02,640 --> 00:15:06,360 Yeah, that's very interesting. 239 00:15:06,360 --> 00:15:10,880 So you talked a little bit about what you found out from academic staff. 240 00:15:10,880 --> 00:15:13,240 So there was one more thing about the academic staff. 241 00:15:13,240 --> 00:15:18,880 That's all right, so the negative, one more sort of area of concern was the substantive 242 00:15:18,880 --> 00:15:22,160 activities are such an important part of each unit. 243 00:15:22,160 --> 00:15:26,240 So we need to make sure that the learners actually do it because how much value are 244 00:15:26,240 --> 00:15:28,480 they going to get if they're not actually doing it? 245 00:15:28,480 --> 00:15:33,920 So we need to make sure that when we're designing, we're designing in motivations for the learner 246 00:15:33,920 --> 00:15:34,920 to do it. 247 00:15:34,920 --> 00:15:38,800 But I'll come to the suggestions for improvements a bit later. 248 00:15:38,800 --> 00:15:46,800 So in relation to DES staff, the positive responses were that the structure provides 249 00:15:46,800 --> 00:15:51,680 really useful scaffolding and a platform upon which we can discuss and explore pedagogy. 250 00:15:51,680 --> 00:15:57,920 So the framework is informed by different pedagogical theories, universal design for 251 00:15:57,920 --> 00:16:04,120 learning and also Laurel Arts conversational framework and also constructive alignment 252 00:16:04,120 --> 00:16:05,260 from BICS. 253 00:16:05,260 --> 00:16:10,040 Those are some of the key sort of theories that have informed it. 254 00:16:10,040 --> 00:16:17,320 And so this platform, this framework, we know that it is based on that pedagogical theory. 255 00:16:17,320 --> 00:16:22,580 And so by having it, we automatically have a platform to engage with academics on that. 256 00:16:22,580 --> 00:16:25,420 So they found that really useful. 257 00:16:25,420 --> 00:16:28,920 They said it does help to control the amount of content in each unit, which was one of 258 00:16:28,920 --> 00:16:33,160 the main goals of the framework was to do that. 259 00:16:33,160 --> 00:16:37,200 It does help to make learning more active, again, one of the key roles of the framework 260 00:16:37,200 --> 00:16:39,200 and that model design is quicker. 261 00:16:39,200 --> 00:16:43,200 So based on that positive feedback, it looks like it's certainly achieving some of the 262 00:16:43,200 --> 00:16:44,200 goals. 263 00:16:44,200 --> 00:16:50,320 But again, they had areas of concern, also about structure and rigidity, that the framework 264 00:16:50,320 --> 00:16:53,640 could be too restrictive and could limit creativity. 265 00:16:53,640 --> 00:16:59,040 Again, they said what I mentioned before, that it can conflict with discipline pedagogy 266 00:16:59,040 --> 00:17:02,040 and therefore cause a problem. 267 00:17:02,040 --> 00:17:06,280 Another concern is that two hour prepare phase that learners complete prior to do the substantive 268 00:17:06,280 --> 00:17:10,800 activity, there has been a sort of predominance of passive learning there. 269 00:17:10,800 --> 00:17:14,000 So it's almost like we're just throwing a lot of knowledge at them and then they have 270 00:17:14,000 --> 00:17:18,760 to apply it in the substantive activity rather than there being a bit more active learning 271 00:17:18,760 --> 00:17:22,200 in that prepare phase. 272 00:17:22,200 --> 00:17:27,180 Another point that it's not always appropriate to implement the framework in the first week 273 00:17:27,180 --> 00:17:31,240 of the module because learners might not feel particularly confident and they suddenly have 274 00:17:31,240 --> 00:17:35,400 to go off on their own and do something that's quite substantive, might be quite new to them. 275 00:17:35,400 --> 00:17:40,200 So maybe flexibility is needed about perhaps in the first week you're not doing it or you're 276 00:17:40,200 --> 00:17:45,280 doing something a lot smaller than you would in the subsequent weeks. 277 00:17:45,280 --> 00:17:49,800 And finally, similar to what the academics said, that the speed of the design and development 278 00:17:49,800 --> 00:17:56,080 process is impacted by factors outside of whether we use the framework or not, such 279 00:17:56,080 --> 00:17:58,400 as academic availability, etc. 280 00:17:58,400 --> 00:18:03,320 So it's never going to solve all the problems, but it looks like from their point of view, 281 00:18:03,320 --> 00:18:06,760 it's certainly contributing to make things a bit better. 282 00:18:06,760 --> 00:18:07,760 That's cool. 283 00:18:07,760 --> 00:18:08,760 And I'm interested in that. 284 00:18:08,760 --> 00:18:15,480 I think, or I think it's interesting that there's this perhaps need to flex the framework 285 00:18:15,480 --> 00:18:19,240 a little bit early on in a module so that students can get used to that kind of new 286 00:18:19,240 --> 00:18:20,240 way of learning. 287 00:18:20,240 --> 00:18:23,360 Did that come out in the student responses? 288 00:18:23,360 --> 00:18:31,800 So in relation to the student responses, unfortunately, I couldn't identify any solid trends because 289 00:18:31,800 --> 00:18:39,540 there was quite a low response rate, which I suppose is to be expected to a certain degree. 290 00:18:39,540 --> 00:18:44,120 So we collected data, as I said before, about non-framework and framework, but the student 291 00:18:44,120 --> 00:18:50,560 experience seems to be impacted far more by their prior knowledge of the subject. 292 00:18:50,560 --> 00:18:57,600 So if they're coming in with some prior knowledge of analytics or design processes or business, 293 00:18:57,600 --> 00:19:02,240 then that's going to impact how well they're able to keep up with the module and how easy 294 00:19:02,240 --> 00:19:06,440 or difficult they find it and how long it takes them to complete. 295 00:19:06,440 --> 00:19:10,640 So it very much depends on the individual student. 296 00:19:10,640 --> 00:19:17,160 But as an indication of the kind of difficulty I had with identifying any trends, two students 297 00:19:17,160 --> 00:19:22,360 on framework modules said that there was insufficient taught content, again, because there's only 298 00:19:22,360 --> 00:19:23,960 that two hour prepare phase. 299 00:19:23,960 --> 00:19:26,720 But then another three said that there was sufficient. 300 00:19:26,720 --> 00:19:28,920 So there's just complete diversity of opinions. 301 00:19:28,920 --> 00:19:31,740 So you can't really identify a particular trend. 302 00:19:31,740 --> 00:19:36,480 But I think what was most insightful was that two framework students said that there was 303 00:19:36,480 --> 00:19:43,000 too much going on with a substantive activity, with the separate discussion. 304 00:19:43,000 --> 00:19:49,160 I was the standalone discussion and responding to and giving feedback to their peers. 305 00:19:49,160 --> 00:19:50,760 So it's difficult to keep up. 306 00:19:50,760 --> 00:19:55,520 And this was really interesting for me because I think because we were going from designing 307 00:19:55,520 --> 00:20:00,360 20 hours of content to designing just those two in the prepare phase, I think we were 308 00:20:00,360 --> 00:20:06,760 trying to kind of make up for that and still demonstrate value to the students by having 309 00:20:06,760 --> 00:20:10,320 things for them to read in preparation for the webinar, sometimes putting in things for 310 00:20:10,320 --> 00:20:15,360 them to read in preparation for a discussion on top of preparation, substantive activity 311 00:20:15,360 --> 00:20:16,880 and giving feedback. 312 00:20:16,880 --> 00:20:21,600 So I think by trying to make sure that they felt that they were getting value for money, 313 00:20:21,600 --> 00:20:24,600 we overloaded them even with the framework. 314 00:20:24,600 --> 00:20:26,440 So it was really useful to get that feedback. 315 00:20:26,440 --> 00:20:32,280 That is really interesting, isn't it, that going from 20 hours of taught content to, 316 00:20:32,280 --> 00:20:34,440 I mean, what does it add up to? 317 00:20:34,440 --> 00:20:39,660 It was something like two hours for prepare, three to five hours for the substantive activity 318 00:20:39,660 --> 00:20:41,120 and four hours for reflect. 319 00:20:41,120 --> 00:20:42,120 Yes. 320 00:20:42,120 --> 00:20:45,760 So something I've neglected to mention at the beginning, which is quite important, is 321 00:20:45,760 --> 00:20:50,000 that there is still the 20 hours of learning, but it's just split up differently. 322 00:20:50,000 --> 00:20:56,880 So we end up the framework takes up 10 to 12 hours and self-directed learning, as in 323 00:20:56,880 --> 00:21:01,160 their choosing completely what they want to do is the eight to 10 hours. 324 00:21:01,160 --> 00:21:05,680 And in that time, they might be using it to spend a bit more time on the substantive activity. 325 00:21:05,680 --> 00:21:09,520 They might be using it to catch up on things they've missed, prepare for their assessments, 326 00:21:09,520 --> 00:21:10,520 et cetera. 327 00:21:10,520 --> 00:21:14,360 So there was also that factor of, oh, we're no longer designing 20 hours. 328 00:21:14,360 --> 00:21:16,440 We're only designing 10 to 12. 329 00:21:16,440 --> 00:21:18,960 So again, we want to give them lots to do. 330 00:21:18,960 --> 00:21:23,040 But it was taking them, some of them, it was taking them over 20 hours, I think, just to 331 00:21:23,040 --> 00:21:27,560 work through that kind of framework, those framework activities. 332 00:21:27,560 --> 00:21:33,280 Yeah, I guess having lots and lots of touch points, I guess, throughout each kind of week. 333 00:21:33,280 --> 00:21:35,320 Yeah, it ends up being a lot. 334 00:21:35,320 --> 00:21:42,480 And I suppose different people will engage with it with different levels of time or amounts 335 00:21:42,480 --> 00:21:47,760 of time, dependent, perhaps, on that knowledge and prior experience that you mentioned before. 336 00:21:47,760 --> 00:21:49,160 Yeah, and their personal circumstances. 337 00:21:49,160 --> 00:21:51,280 So it all varies. 338 00:21:51,280 --> 00:21:56,960 So I think, yeah, it was really, really, really useful to get that feedback from them. 339 00:21:56,960 --> 00:22:01,640 Another bit of data I collected in relation to students, which I, again, forgot to mention 340 00:22:01,640 --> 00:22:05,400 before, I apologise, is in relation to Minerva data. 341 00:22:05,400 --> 00:22:07,520 So we also got some Minerva data about engagement. 342 00:22:07,520 --> 00:22:10,600 And Minerva is our VLE here at Leeds. 343 00:22:10,600 --> 00:22:12,560 Yes, based on Blackboard, yes. 344 00:22:12,560 --> 00:22:16,120 So that's where we build these master's degree modules. 345 00:22:16,120 --> 00:22:22,600 So that's what learners access in order to get to access their learning materials. 346 00:22:22,600 --> 00:22:28,760 So I got some data about engagement in those standalone discussion forums and also webinar 347 00:22:28,760 --> 00:22:30,280 attendance. 348 00:22:30,280 --> 00:22:36,200 And I noted that participation in those standalone discussions was limited. 349 00:22:36,200 --> 00:22:41,000 And perhaps that indicates the fact that there was just too much, what those students were 350 00:22:41,000 --> 00:22:42,680 saying, there was just too much to do. 351 00:22:42,680 --> 00:22:46,480 And it was the discussion that fell by the wayside. 352 00:22:46,480 --> 00:22:49,720 So that may be a question, well, why are we doing it? 353 00:22:49,720 --> 00:22:53,640 If it's not being engaged with, it takes a lot of time to design discussion questions. 354 00:22:53,640 --> 00:22:58,980 It's time for the module leader to engage in them themselves while the module is running. 355 00:22:58,980 --> 00:23:03,080 And so it was really insightful to see that data that the learners perhaps just don't 356 00:23:03,080 --> 00:23:05,760 have the time for it. 357 00:23:05,760 --> 00:23:10,360 And so the final interesting thing about the discussions, which one of the academics touched 358 00:23:10,360 --> 00:23:16,120 on is that engagement in the post-substantive activity discussion, so the student-led one 359 00:23:16,120 --> 00:23:20,520 where they're comparing what they've done, those discussions were much more active. 360 00:23:20,520 --> 00:23:24,800 So those are the ones that the learners were engaging with rather than the standalone discussion. 361 00:23:24,800 --> 00:23:29,240 OK, so that standalone discussion happened on a board, and then the other discussion 362 00:23:29,240 --> 00:23:32,120 was in a synchronous session where people were participating. 363 00:23:32,120 --> 00:23:37,360 It was asynchronous as well, but it just had a different purpose of them sharing their 364 00:23:37,360 --> 00:23:38,360 work with each other. 365 00:23:38,360 --> 00:23:42,080 And the module leader, I mean, it's up to the module leader how much they engage with 366 00:23:42,080 --> 00:23:43,080 that. 367 00:23:43,080 --> 00:23:46,760 But it wasn't something where they had, you know, that we set discussion questions. 368 00:23:46,760 --> 00:23:50,320 We simply said, share your work with your peers and get feedback. 369 00:23:50,320 --> 00:23:54,640 And apparently in those, the academics were saying, yeah, there's a lot more engagement. 370 00:23:54,640 --> 00:23:59,120 Perhaps it seemed more useful for the learners to engage in that one rather than the standalone 371 00:23:59,120 --> 00:24:02,520 one, which wasn't related to the substantive activity. 372 00:24:02,520 --> 00:24:05,080 OK, so that's all really interesting. 373 00:24:05,080 --> 00:24:10,080 And so I'm curious to know, given you've got this kind of diverse and interesting set of 374 00:24:10,080 --> 00:24:16,480 data, what did you do with that in terms of making recommendations for future practice 375 00:24:16,480 --> 00:24:19,840 or suggested changes or anything like that? 376 00:24:19,840 --> 00:24:27,200 Yeah, so I think my overall conclusions were that the framework was helping to make things 377 00:24:27,200 --> 00:24:30,360 quicker and it was helping to control content. 378 00:24:30,360 --> 00:24:35,960 And it did result in a more active learner centred experience. 379 00:24:35,960 --> 00:24:43,440 But that it can mean it can mean that things are too rigid and too uniform. 380 00:24:43,440 --> 00:24:49,920 So my suggested improvements were that the framework is used as just that, a framework, 381 00:24:49,920 --> 00:24:56,320 and is applied flexibly depending on what is needed for that particular discipline and 382 00:24:56,320 --> 00:24:59,920 to, you know, not to be prepared to cut things out. 383 00:24:59,920 --> 00:25:05,960 For example, the two hour reflection at the end, some disciplines are not reflective disciplines. 384 00:25:05,960 --> 00:25:08,440 It's you're right or you're wrong and that's it. 385 00:25:08,440 --> 00:25:13,720 And so insisting on including that reflection for a discipline that doesn't really do that 386 00:25:13,720 --> 00:25:14,960 wasn't helpful. 387 00:25:14,960 --> 00:25:22,520 So it kind of sort of gave us a justification, I suppose, for being more flexible in that 388 00:25:22,520 --> 00:25:23,520 regard. 389 00:25:23,520 --> 00:25:28,520 There was the suggestion to yet to make the prepare phase a bit more active. 390 00:25:28,520 --> 00:25:31,120 As I said, I've left the team, so I don't know if people are doing that. 391 00:25:31,120 --> 00:25:37,780 But that was one of my kind of recommendations that were flexible as to when the pedagogical 392 00:25:37,780 --> 00:25:42,680 framework starts so that we don't have to have these independent, these huge independent 393 00:25:42,680 --> 00:25:44,680 tasks in week one. 394 00:25:44,680 --> 00:25:50,360 And that task size sort of, when I say task, I mean the substantive activity and that the 395 00:25:50,360 --> 00:25:56,040 size of that substantive activity can vary along the way through the module so that when 396 00:25:56,040 --> 00:26:00,080 we know that learners are going to be doing a formative assessment, for example, they 397 00:26:00,080 --> 00:26:04,040 don't have to spend as much time on substantive activity that week. 398 00:26:04,040 --> 00:26:08,840 And so that was one main recommendation, more flexibility. 399 00:26:08,840 --> 00:26:13,680 The second main recommendation was that we make sure that we're encouraging engagement 400 00:26:13,680 --> 00:26:16,720 in the tasks, in the substantive activities. 401 00:26:16,720 --> 00:26:22,560 For example, that any feedback is written in such a way that it only makes sense if 402 00:26:22,560 --> 00:26:27,440 the learners have done the activity, that the activity is linked to the webinar in some 403 00:26:27,440 --> 00:26:32,720 way, that it's always linked to some sort of summative assessment in some way, even 404 00:26:32,720 --> 00:26:35,960 if we just make clear the skills that you're going to develop, the knowledge that you're 405 00:26:35,960 --> 00:26:39,880 going to develop in this substantive activity is going to help you with your final assessment 406 00:26:39,880 --> 00:26:45,000 because obviously learners are very assessment focused. 407 00:26:45,000 --> 00:26:49,600 To give those clear reasons for completing that activity and breaking it down into component 408 00:26:49,600 --> 00:26:51,600 parts when that suits. 409 00:26:51,600 --> 00:26:55,560 So it may not always be possible to break down an activity into smaller sections, but 410 00:26:55,560 --> 00:26:59,240 where it is possible to help those learners that have only got little pockets of time 411 00:26:59,240 --> 00:27:05,760 here and there, it means that it might make it easier for them to do that. 412 00:27:05,760 --> 00:27:14,120 So since completing this research, I did design one module before I left that team and I put 413 00:27:14,120 --> 00:27:20,000 this into practice and it did make the design process much, much easier because I wasn't 414 00:27:20,000 --> 00:27:23,440 coming up against the academics in the same way to say, this is the framework and this 415 00:27:23,440 --> 00:27:25,040 is what we need to be doing. 416 00:27:25,040 --> 00:27:27,520 It was, oh, well, reflection is not going to work for you. 417 00:27:27,520 --> 00:27:28,520 Okay, fine. 418 00:27:28,520 --> 00:27:31,240 So what would we do there instead that would work for your discipline? 419 00:27:31,240 --> 00:27:34,920 Is there something that the learners could usefully discuss this week that's separate 420 00:27:34,920 --> 00:27:37,160 from their sort of post activity discussion? 421 00:27:37,160 --> 00:27:40,480 No, okay, let's just not worry about that. 422 00:27:40,480 --> 00:27:46,240 So it meant that we could just not, you know, exclude things that were not useful and save 423 00:27:46,240 --> 00:27:50,360 a lot of time on those discussions and debates about those particular sections. 424 00:27:50,360 --> 00:27:55,560 I mean, one more point I should note is that when we were piloting the framework, we were 425 00:27:55,560 --> 00:27:59,840 kind of deliberately applying it rigidly because we wanted to test it out. 426 00:27:59,840 --> 00:28:03,800 It's only once you've properly done something, you know, by the book, as it were, that you 427 00:28:03,800 --> 00:28:07,040 can then work out where you can flex or where you can't. 428 00:28:07,040 --> 00:28:12,880 So that kind of feedback about rigidity and now applying it more flexibly doesn't come 429 00:28:12,880 --> 00:28:17,720 as a surprise, but it's good to have it sort of justified in the research. 430 00:28:17,720 --> 00:28:19,320 Yeah, that's really interesting. 431 00:28:19,320 --> 00:28:28,120 And I guess it'd be really interesting to extend the research that you've done, perhaps 432 00:28:28,120 --> 00:28:34,000 someone else, given that you've left the team, but to see how that is being received in practice, 433 00:28:34,000 --> 00:28:39,080 especially since your anecdotal experience of that was that it was positive for you. 434 00:28:39,080 --> 00:28:44,080 And did you get any anecdotal feedback from the academics? 435 00:28:44,080 --> 00:28:45,640 In relation to that module I designed? 436 00:28:45,640 --> 00:28:47,440 Yeah, being a bit more flexible. 437 00:28:47,440 --> 00:28:50,560 Well, it's difficult because they wouldn't have designed one before. 438 00:28:50,560 --> 00:28:51,560 This was their first experience. 439 00:28:51,560 --> 00:28:54,160 It's not like they could compare it to anything else. 440 00:28:54,160 --> 00:28:57,480 This was their first experience of working with the framework and working with us. 441 00:28:57,480 --> 00:29:03,080 OK, but I guess from your perspective, it felt like it was easier for everyone. 442 00:29:03,080 --> 00:29:08,640 Easier and much quicker, definitely much quicker and a much kind of smoother process. 443 00:29:08,640 --> 00:29:09,640 Brilliant. 444 00:29:09,640 --> 00:29:14,560 And so I'm curious to know if there's anything that you would do differently if you're going 445 00:29:14,560 --> 00:29:16,600 to start your light fellowship over. 446 00:29:16,600 --> 00:29:18,800 Absolutely, yes. 447 00:29:18,800 --> 00:29:24,600 So I think I would have written better research questions. 448 00:29:24,600 --> 00:29:30,600 I think I used words that were kind of vague and I knew it at the time when I wrote them, 449 00:29:30,600 --> 00:29:33,320 but I just couldn't think of a better way to word them. 450 00:29:33,320 --> 00:29:38,760 And I think what I should have done is word the research questions so that they were grounded 451 00:29:38,760 --> 00:29:43,760 in the problems we were trying to solve in a much more succinct and concise manner. 452 00:29:43,760 --> 00:29:48,680 So there's too much content, is the framework helping you to control content? 453 00:29:48,680 --> 00:29:54,640 You know, much, much simpler, shorter questions, because the research questions I had ended 454 00:29:54,640 --> 00:29:57,720 up being quite difficult to work with throughout the project. 455 00:29:57,720 --> 00:30:00,360 Yeah, when you get to that kind of analysis phase. 456 00:30:00,360 --> 00:30:08,120 Yeah, I always remember one chapter of my PhD having similarly, perhaps to you, research 457 00:30:08,120 --> 00:30:12,240 questions that I could have written better and finding that analysis and reporting phase 458 00:30:12,240 --> 00:30:13,240 really laborious. 459 00:30:13,240 --> 00:30:18,080 Absolutely, it does make it a lot more difficult because you're trying to answer research questions 460 00:30:18,080 --> 00:30:22,760 that you were never really going to be able to answer, but you have got things that you 461 00:30:22,760 --> 00:30:27,320 can answer that are useful and they are related to the research questions, but the research 462 00:30:27,320 --> 00:30:32,920 questions are just not very well written. 463 00:30:32,920 --> 00:30:35,800 I think that's the main thing I would have done differently. 464 00:30:35,800 --> 00:30:41,240 I think other things that were challenging were things that were outside of my control 465 00:30:41,240 --> 00:30:44,040 and outside of lights control. 466 00:30:44,040 --> 00:30:47,880 It is challenging to do a light fellowship in one year when you've only got one day a 467 00:30:47,880 --> 00:30:53,200 week because you end up spending the first part of that day working out where you've 468 00:30:53,200 --> 00:30:59,560 got to and then that only leaves you sort of half a day or three quarters of a day. 469 00:30:59,560 --> 00:31:03,440 So that was challenging, but I just had to get really good at project management to get 470 00:31:03,440 --> 00:31:05,240 it all done in that time. 471 00:31:05,240 --> 00:31:07,280 So I think it was a benefit in the end. 472 00:31:07,280 --> 00:31:11,440 And then actually I think as well perhaps from the interactions that we had throughout 473 00:31:11,440 --> 00:31:15,120 your fellowship, you were very good at having one day a week that was your light fellowship 474 00:31:15,120 --> 00:31:16,120 day. 475 00:31:16,120 --> 00:31:19,960 Yes, I think it was really important because initially in the first month I didn't have 476 00:31:19,960 --> 00:31:23,840 it blocked out of my diary and then my line manager was like, you have to block out your 477 00:31:23,840 --> 00:31:24,840 diary. 478 00:31:24,840 --> 00:31:30,160 So then for the whole year, one day was blocked and so no one could book any meetings. 479 00:31:30,160 --> 00:31:34,760 Sometimes obviously urgent things come up and I had to deal with them, but equally sometimes 480 00:31:34,760 --> 00:31:38,160 there was light events earlier on in the week, so then I would be coming onto campus and 481 00:31:38,160 --> 00:31:41,040 attending those, which was also part of my light allocation. 482 00:31:41,040 --> 00:31:44,520 So a bit of flexing there as well. 483 00:31:44,520 --> 00:31:50,360 And then that perhaps moves quite nicely into thinking about your experience of your light 484 00:31:50,360 --> 00:31:51,360 fellowship. 485 00:31:51,360 --> 00:31:57,200 And so I'm wondering if you experienced any kind of benefits for yourself of having taken 486 00:31:57,200 --> 00:32:00,280 part in this fellowship or completed this fellowship? 487 00:32:00,280 --> 00:32:01,280 Very much so. 488 00:32:01,280 --> 00:32:08,840 So as I mentioned before, last October I started a new position and I think having done the 489 00:32:08,840 --> 00:32:15,640 light fellowship, it helped me to demonstrate that I had experience of evaluation of our 490 00:32:15,640 --> 00:32:20,280 courses and evaluation is really important in our current team. 491 00:32:20,280 --> 00:32:27,240 It also helped me to develop my research experience ahead of my PhD, which I started in October, 492 00:32:27,240 --> 00:32:32,840 and it may well have contributed to me getting ESRC funding for my PhD because I was able 493 00:32:32,840 --> 00:32:37,040 to include my research experience in the application. 494 00:32:37,040 --> 00:32:44,040 And aside from that, another personal benefit was that it provided a lot of networking opportunities 495 00:32:44,040 --> 00:32:49,840 with people outside of my department and across the university, which I have found really 496 00:32:49,840 --> 00:32:55,880 useful because having contacts across the university helps you to understand how other 497 00:32:55,880 --> 00:32:57,880 parts of the university work. 498 00:32:57,880 --> 00:33:00,680 And in my current team, that's really important because we're working with different parts 499 00:33:00,680 --> 00:33:02,680 of the university all the time. 500 00:33:02,680 --> 00:33:03,680 Excellent. 501 00:33:03,680 --> 00:33:06,680 Well, congratulations on your funding and your PhD. 502 00:33:06,680 --> 00:33:07,680 Thank you very much. 503 00:33:07,680 --> 00:33:08,680 That's very interesting. 504 00:33:08,680 --> 00:33:11,880 Perhaps we can have another conversation one day about that. 505 00:33:11,880 --> 00:33:12,880 Of course. 506 00:33:12,880 --> 00:33:17,880 But yeah, it's been wonderful to hear a bit more about your project today. 507 00:33:17,880 --> 00:33:23,160 And we spoke occasionally during your fellowship, but we didn't work particularly closely. 508 00:33:23,160 --> 00:33:28,120 So it's really interesting to hear a little bit more detail about what you found out and 509 00:33:28,120 --> 00:33:30,240 how that might be useful and everything like that. 510 00:33:30,240 --> 00:33:31,880 So thank you for coming along. 511 00:33:31,880 --> 00:33:33,520 Yes, thank you very much for having me. 512 00:33:33,520 --> 00:33:34,520 It's been a pleasure. 513 00:33:34,520 --> 00:33:35,520 Yeah. 514 00:33:35,520 --> 00:33:39,280 And you know, all the best with your continued studies and research and thanks for sharing. 515 00:33:39,280 --> 00:33:40,280 Thank you very much. 516 00:33:40,280 --> 00:33:56,700 Well, thank you again.