WEBVTT 00:00:08.490 --> 00:00:11.169 Welcome to LITE Bites, an occasional podcast 00:00:11.169 --> 00:00:13.289 from Leeds Institute for Teaching Excellence 00:00:13.289 --> 00:00:16.269 at the University of Leeds. Episodes will be 00:00:16.269 --> 00:00:18.550 hosted by members of the LITE team. And we'll 00:00:18.550 --> 00:00:20.370 be showcasing the scholarship of teaching and 00:00:20.370 --> 00:00:28.429 learning from across the university. Hello and 00:00:28.429 --> 00:00:31.329 welcome to LITE Bites. I'm Robert Averys. You 00:00:31.329 --> 00:00:33.549 thought we'd forgotten, but no. You're about 00:00:33.549 --> 00:00:36.009 to listen to part two of Manoj Ravi and Mohsen 00:00:36.009 --> 00:00:38.840 Besharat's LITE Bites episode. on their LITE 00:00:38.840 --> 00:00:41.340 Fellowship, Assessment Flexibility to Embed Real 00:00:41.340 --> 00:00:44.460 Professional Practice. Not to put you to any 00:00:44.460 --> 00:00:47.119 pressure, but the first part, which we released 00:00:47.119 --> 00:00:50.100 in October 2024 and saw you present the findings 00:00:50.100 --> 00:00:52.020 from the first phase of your fellowship, was 00:00:52.020 --> 00:00:55.340 our most listened to episode to date. What do 00:00:55.340 --> 00:01:00.479 you put that down to? I think, I don't know, 00:01:00.500 --> 00:01:03.619 probably we just distributed that on different 00:01:03.619 --> 00:01:06.079 platforms and people were interested, so to listen 00:01:06.079 --> 00:01:08.480 to that. Yeah, we were quite lucky. We really 00:01:08.480 --> 00:01:12.140 appreciate that. Clickbait title, probably. You're 00:01:12.140 --> 00:01:15.239 being modest. So for the benefit of those who 00:01:15.239 --> 00:01:17.340 haven't listened to part one, Manoj and Mohsen 00:01:17.340 --> 00:01:19.180 are now going to introduce themselves and the 00:01:19.180 --> 00:01:21.760 purpose of their LITE Fellowship. Very good. 00:01:21.799 --> 00:01:23.819 Thank you so much for having us again, Robert. 00:01:24.719 --> 00:01:28.200 So I'm now Associate Professor at the School 00:01:28.200 --> 00:01:31.540 of Civil Engineering and I'm still Programme 00:01:31.540 --> 00:01:36.400 Leader for three undergraduate programmes. I'm 00:01:36.400 --> 00:01:39.239 involved in other university -wide initiatives 00:01:39.239 --> 00:01:43.819 like sustainable curriculum and yeah with Manage 00:01:43.819 --> 00:01:47.620 we started to do this LITE fellowship and the 00:01:47.620 --> 00:01:50.060 aim for the fellowship was actually to understand 00:01:50.060 --> 00:01:53.079 how we can support our learners in order to get 00:01:53.079 --> 00:01:56.540 that sort of like skills and the knowledge they 00:01:56.540 --> 00:01:59.430 need for professional practices we found that 00:01:59.430 --> 00:02:03.170 sometimes students are a bit deviated from that 00:02:03.170 --> 00:02:08.490 path and they probably focus on Mark and also 00:02:08.490 --> 00:02:10.909 the process of the assessment is not actually 00:02:10.909 --> 00:02:13.990 well reflecting what we are doing in industry. 00:02:14.210 --> 00:02:19.750 So, yeah. Brilliant. Thanks, Mohsen. Thank you, 00:02:19.770 --> 00:02:22.370 Robert, for having us back on The LITE Podcast. 00:02:22.530 --> 00:02:25.310 Great to be back. Yeah, Manoj Ravi over from 00:02:25.310 --> 00:02:26.870 the School of Chemical and Process Engineering, 00:02:27.110 --> 00:02:29.030 where I work as a lecturer, joined through the 00:02:29.030 --> 00:02:32.090 Curriculum Redefined Initiative at the university. 00:02:32.289 --> 00:02:34.250 I'm also the program lead for the undergraduate 00:02:34.250 --> 00:02:37.370 chemical engineering program in our school. And 00:02:37.370 --> 00:02:40.909 yes, Mohsen's kindly introduced the remit and 00:02:40.909 --> 00:02:44.129 the broad scope and aims of our fellowship. We 00:02:44.129 --> 00:02:46.569 were looking at authentic assessments and flexible 00:02:46.569 --> 00:02:49.629 assessments, kind of reinforcing. each other. 00:02:50.169 --> 00:02:52.469 We're here for part two because in part one we 00:02:52.469 --> 00:02:55.530 spoke primarily around the findings from the 00:02:55.530 --> 00:02:58.710 student survey that we'd done largely in the 00:02:58.710 --> 00:03:00.590 engineering and physical sciences faculty with 00:03:00.590 --> 00:03:02.449 our students and it was a pretty large sample 00:03:02.449 --> 00:03:06.349 size and there were findings which we kind of 00:03:06.349 --> 00:03:08.189 expected when we put together the project proposal 00:03:08.189 --> 00:03:10.180 but there were A couple of other interesting 00:03:10.180 --> 00:03:13.699 things as well. So in terms of like a quick recap, 00:03:13.960 --> 00:03:16.280 of course, do go listen to our first podcast 00:03:16.280 --> 00:03:19.659 if not. But sort of quick recap would be we're 00:03:19.659 --> 00:03:21.819 looking at a wide array of questions picked based 00:03:21.819 --> 00:03:24.080 on the Authentic Assessment Framework and other 00:03:24.080 --> 00:03:26.000 literature which has been published in that direction, 00:03:26.099 --> 00:03:30.080 specifically to our students at EPS, Engineering 00:03:30.080 --> 00:03:32.659 and Physical Sciences. A vast, vast majority 00:03:32.659 --> 00:03:35.479 of students said that... They would incentivize 00:03:35.479 --> 00:03:38.020 early submission if given an opportunity to get 00:03:38.020 --> 00:03:40.919 formative feedback before they hand in their 00:03:40.919 --> 00:03:44.400 summative piece. But it was not something necessarily 00:03:44.400 --> 00:03:47.719 they were used to at scale and as often as possible 00:03:47.719 --> 00:03:50.560 in their current curriculum across any of the 00:03:50.560 --> 00:03:53.479 programs offered in the faculty. Yeah, so only 00:03:53.479 --> 00:03:56.139 7 % have got my numbers printed out. 7 % said 00:03:56.139 --> 00:03:57.800 that they experienced any kind of flexibility 00:03:57.800 --> 00:04:01.479 in assessments in that survey. And closely linked 00:04:01.479 --> 00:04:05.889 to flexibility was our... desire to couple creativity 00:04:05.889 --> 00:04:08.789 to this and how do we kind of foster creativity 00:04:08.789 --> 00:04:11.849 in students when they're handing in these assessments. 00:04:11.990 --> 00:04:14.330 And again, only around 14 % said that they can 00:04:14.330 --> 00:04:16.310 express creativity in coursework assessments. 00:04:16.569 --> 00:04:18.550 So those were some of the things that we potentially 00:04:18.550 --> 00:04:21.250 wanted to address in our pilot in our second 00:04:21.250 --> 00:04:23.269 year, which is what we're going to talk about 00:04:23.269 --> 00:04:26.529 on this podcast. So yeah, that's a brief recap 00:04:26.529 --> 00:04:28.350 from the findings from the student survey in 00:04:28.350 --> 00:04:30.670 terms of what scored high, what scored relatively 00:04:30.670 --> 00:04:33.110 low. and the interventions that we potentially 00:04:33.110 --> 00:04:35.350 wanted to make. And yeah, if you stay tuned to 00:04:35.350 --> 00:04:37.149 this episode, you'll find out whether we failed 00:04:37.149 --> 00:04:39.550 or we've had some reasonable success. That's 00:04:39.550 --> 00:04:44.189 the hook. So you've now completed the second 00:04:44.189 --> 00:04:46.110 and final year of your LITE Fellowship, and 00:04:46.110 --> 00:04:48.310 I think we're all eager to hear how it's gone. 00:04:48.550 --> 00:04:52.790 How and where was the pilot implemented? So for 00:04:52.790 --> 00:04:56.509 pilot, we decided to implement the ideas we got 00:04:56.509 --> 00:04:59.509 from the survey in a couple of modules. For myself, 00:04:59.730 --> 00:05:02.110 I'm speaking from... My school then manager actually 00:05:02.110 --> 00:05:06.430 can explain how he piloted the idea. So in School 00:05:06.430 --> 00:05:09.129 of Civil Engineering, I piloted these ideas in 00:05:09.129 --> 00:05:13.870 two MSC modules, which are in our two very successful 00:05:13.870 --> 00:05:16.370 kind of MSC programs. One of them is called Wash, 00:05:16.389 --> 00:05:19.050 Water, and Sanitation, and Health. And the other 00:05:19.050 --> 00:05:21.290 one, EAPM, Environmental Engineering and Project 00:05:21.290 --> 00:05:24.600 Management. And the name of the modules, just 00:05:24.600 --> 00:05:26.720 for listeners, if they want, they can go and 00:05:26.720 --> 00:05:29.139 check the modules on our module catalog, are 00:05:29.139 --> 00:05:32.459 called Water Supply and the other one is Advanced 00:05:32.459 --> 00:05:36.420 Wastewater Management. So from what we discussed 00:05:36.420 --> 00:05:38.500 and what I actually managed now mentioned, we 00:05:38.500 --> 00:05:41.439 tried to understand how we can implement ideas 00:05:41.439 --> 00:05:45.629 of fostering the creativity and... providing 00:05:45.629 --> 00:05:48.970 students with flexibility and more importantly 00:05:48.970 --> 00:05:51.589 to give them this kind of feedback loop that 00:05:51.589 --> 00:05:56.209 they wanted because 86 percent of students they 00:05:56.209 --> 00:05:58.649 said that they incentivize actually they like 00:05:58.649 --> 00:06:03.069 to have feedback early in order to work based 00:06:03.069 --> 00:06:05.490 on the feedback not actually looking at the feedback 00:06:05.490 --> 00:06:09.589 when they have got the mark that looking back 00:06:09.589 --> 00:06:13.329 kind of approach but forward feedback so we improved 00:06:13.329 --> 00:06:16.600 the authentic delivery because another finding 00:06:16.600 --> 00:06:19.339 from the first phase of the project was that 00:06:19.339 --> 00:06:23.399 authentic assessment is not actually one term 00:06:23.399 --> 00:06:26.199 is actually the design of the assessment plus 00:06:26.199 --> 00:06:28.839 the delivery of that so we try to focus on authentic 00:06:28.839 --> 00:06:32.959 delivery and like I said we increase the flexibility 00:06:32.959 --> 00:06:37.639 we focus on feedback quality and also we encourage 00:06:37.639 --> 00:06:42.939 reflection in the in the process How we did that, 00:06:43.000 --> 00:06:47.079 I introduced a group project in the assessment. 00:06:47.220 --> 00:06:50.879 So students actually started to work on a design 00:06:50.879 --> 00:06:54.139 task, which was like an open -end, very authentic 00:06:54.139 --> 00:06:58.319 kind of problem. And through that, they work 00:06:58.319 --> 00:07:01.120 in the groups through tutorial sessions that 00:07:01.120 --> 00:07:02.959 were designed in order to provide them with the 00:07:02.959 --> 00:07:06.040 feedback. And then they were given this opportunity 00:07:06.040 --> 00:07:09.500 to submit their work from the group work at some 00:07:09.500 --> 00:07:13.269 point. get feedback on that from from lectures 00:07:13.269 --> 00:07:16.670 of the module and plus peer review from from 00:07:16.670 --> 00:07:19.910 people in the group and eventually they could 00:07:19.910 --> 00:07:23.629 take this feedback into the final phase which 00:07:23.629 --> 00:07:25.610 is the submission of the of the report in terms 00:07:25.610 --> 00:07:29.709 of the timetable I mean time frame of that it 00:07:29.709 --> 00:07:34.170 was like the submission of the group activity 00:07:34.170 --> 00:07:36.089 was at the end of there for example November 00:07:36.089 --> 00:07:39.120 because this was like a semester one module then 00:07:39.120 --> 00:07:42.139 at the beginning of the December they got feedback 00:07:42.139 --> 00:07:45.180 from the group submission which was in the format 00:07:45.180 --> 00:07:48.459 of a poster and we saw a lot of like creative 00:07:48.459 --> 00:07:53.019 design of the poster then we had an oral presentation 00:07:53.019 --> 00:07:55.740 of the poster which they received the feedback 00:07:55.740 --> 00:08:00.439 from module contributors and then at the end 00:08:00.439 --> 00:08:02.779 of the December they were due to submit their 00:08:02.779 --> 00:08:06.939 final report so And during this journey, we did 00:08:06.939 --> 00:08:09.420 journaling. So it was like six -week journaling 00:08:09.420 --> 00:08:11.680 of the student to understand how they are developing 00:08:11.680 --> 00:08:15.759 over the course of this experience. In terms 00:08:15.759 --> 00:08:19.339 of some indices that we picked from the first 00:08:19.339 --> 00:08:22.939 phase of the research, like stress and perception 00:08:22.939 --> 00:08:27.040 of the workload and creativity, we saw a big 00:08:27.040 --> 00:08:29.199 jump from week one to week six. For example, 00:08:29.360 --> 00:08:31.319 a huge amount of students, they said that they 00:08:31.319 --> 00:08:34.049 were really stressed by the... assessment at 00:08:34.049 --> 00:08:36.509 the beginning at week one but then at week six 00:08:36.509 --> 00:08:41.389 we had about 100 percent actually of students 00:08:41.389 --> 00:08:43.970 they said they found the workload manageable 00:08:43.970 --> 00:08:48.490 and they feel less stressed by the work so I 00:08:48.490 --> 00:08:54.990 think the process overall supported student to 00:08:54.990 --> 00:08:58.490 develop this sort of like developing different 00:08:58.490 --> 00:09:01.009 skills and learning different things designed 00:09:01.009 --> 00:09:05.809 for the project during the six weeks. And an 00:09:05.809 --> 00:09:08.809 interesting part was that towards the end of 00:09:08.809 --> 00:09:12.490 the journey, towards the week six, they were 00:09:12.490 --> 00:09:16.110 provided with more tasks to do. But despite that, 00:09:16.210 --> 00:09:20.029 they found the experience quite kind of manageable 00:09:20.029 --> 00:09:24.179 and less stressed. The quality of the work eventually 00:09:24.179 --> 00:09:27.039 was really good. We had very low failure rate, 00:09:27.080 --> 00:09:29.100 only one student actually in the cohort failed, 00:09:29.179 --> 00:09:34.620 and we had a lot of marks above 70%. Brilliant. 00:09:34.820 --> 00:09:38.409 Thanks for that, Mohsen. Very extensive. work 00:09:38.409 --> 00:09:41.009 over at the School of Civil Engineering. In terms 00:09:41.009 --> 00:09:42.529 of the motivations, yeah, it was very similar 00:09:42.529 --> 00:09:45.309 to me, where I implemented the pilot at the School 00:09:45.309 --> 00:09:47.750 of Chemical and Process Engineering for a first 00:09:47.750 --> 00:09:50.389 -year undergraduate module that's taken by chemical 00:09:50.389 --> 00:09:53.970 engineering students. And this was around a lab 00:09:53.970 --> 00:09:55.909 -based assessment, so a practical work, and then 00:09:55.909 --> 00:09:57.870 a piece of coursework, which is usually linked 00:09:57.870 --> 00:10:02.080 to the lab practical. Just to probably put it 00:10:02.080 --> 00:10:04.519 in context, the way it's usually worked is we've 00:10:04.519 --> 00:10:06.379 designed, this is largely again to first year 00:10:06.379 --> 00:10:08.240 students, so we've designed the experiment to 00:10:08.240 --> 00:10:12.759 a fairly nuanced degree. Students come in in 00:10:12.759 --> 00:10:14.879 small groups, they perform the experiment in 00:10:14.879 --> 00:10:17.639 the lab, they take away the data from it and 00:10:17.639 --> 00:10:20.019 the coursework usually involves data analysis 00:10:20.019 --> 00:10:24.159 and interpretation, usually working to a two 00:10:24.159 --> 00:10:27.000 week. turnaround time from the day to the practical 00:10:27.000 --> 00:10:29.679 to when the submission is due. So in terms of 00:10:29.679 --> 00:10:32.080 the pilot, this assessment was already in place, 00:10:32.179 --> 00:10:34.600 but we went about making three specific changes 00:10:34.600 --> 00:10:37.539 in terms of looking back at what students told 00:10:37.539 --> 00:10:41.039 us from that all faculty survey in terms of flexibility. 00:10:41.240 --> 00:10:44.019 So we kind of wanted to embed this. three ways. 00:10:44.259 --> 00:10:47.259 We started off by giving them, firstly, flexibility 00:10:47.259 --> 00:10:50.080 in the operating conditions. So we did not necessarily 00:10:50.080 --> 00:10:52.399 specify the exact experiments that they're going 00:10:52.399 --> 00:10:55.879 to run in the lab. We gave them a kind of window 00:10:55.879 --> 00:10:59.620 of operation of things like temperature and flow 00:10:59.620 --> 00:11:02.070 rates. Don't want to go into the nitty gritties 00:11:02.070 --> 00:11:04.350 of the experiment. But yeah, there were choices 00:11:04.350 --> 00:11:06.370 in the operating conditions that they could collectively 00:11:06.370 --> 00:11:09.409 decide as a group. So that was the first element 00:11:09.409 --> 00:11:12.570 of flexibility. The second thing which we also 00:11:12.570 --> 00:11:14.490 started embedding based on the fact that, as 00:11:14.490 --> 00:11:16.409 Mohsen very rightly said, that students greatly 00:11:16.409 --> 00:11:19.470 value formative feedback. So we wanted to move 00:11:19.470 --> 00:11:21.950 away from the approach of they do the experiment 00:11:21.950 --> 00:11:24.629 in the lab, go away, work in silos for two weeks, 00:11:24.690 --> 00:11:26.990 trying to get the data interpreted and then hand 00:11:26.990 --> 00:11:30.169 it in for a mark. But we kind of embedded. this 00:11:30.169 --> 00:11:33.649 opportunity to get formative feedback from the 00:11:33.649 --> 00:11:36.570 instructor. So for me on that module, one week 00:11:36.570 --> 00:11:39.769 into their assessment period. So after a week 00:11:39.769 --> 00:11:41.169 that they'd done the lab, they could come and 00:11:41.169 --> 00:11:43.870 show any kind of work. This need not be, it could 00:11:43.870 --> 00:11:45.750 have been a data table. It could have been a 00:11:45.750 --> 00:11:47.830 paragraph on the introduction. It could have 00:11:47.830 --> 00:11:49.669 been an interpretation of the results. It was 00:11:49.669 --> 00:11:52.090 up to the students in terms of what they wanted 00:11:52.090 --> 00:11:55.169 to get feedback on and how extensive they wanted 00:11:55.169 --> 00:11:57.450 it to be. But it was verbal feedback. So there 00:11:57.450 --> 00:11:59.649 was no pressure on them in terms of. You have 00:11:59.649 --> 00:12:01.730 to give me a PowerPoint slide or you have to 00:12:01.730 --> 00:12:05.090 write up a specific A4 paper. It was very much 00:12:05.090 --> 00:12:08.320 up to them. And the third and final element of 00:12:08.320 --> 00:12:11.399 flexibility, which we tried to link this more 00:12:11.399 --> 00:12:13.840 to the creativity really, was in terms of the 00:12:13.840 --> 00:12:16.620 format of their submission. So we said you could 00:12:16.620 --> 00:12:20.019 either hand this in as a report or it could also 00:12:20.019 --> 00:12:23.980 be a video recording. But the rubric and the 00:12:23.980 --> 00:12:25.620 assessment brief was designed in a way where 00:12:25.620 --> 00:12:27.700 we explicitly showed how the learning outcomes 00:12:27.700 --> 00:12:29.539 were the same in terms of what they were meeting 00:12:29.539 --> 00:12:32.070 it. irrespective of whether they decided to hand 00:12:32.070 --> 00:12:34.549 it in as a report or if they decided to do it 00:12:34.549 --> 00:12:37.110 as a video recording. So those were the three 00:12:37.110 --> 00:12:41.490 elements of flexibility. In terms of how we analyzed 00:12:41.490 --> 00:12:43.370 the data from the pilot, very similar to what 00:12:43.370 --> 00:12:45.429 Mohsen did, but because we were working to a 00:12:45.429 --> 00:12:47.889 two -week turnaround period, it was effectively 00:12:47.889 --> 00:12:51.299 a... sort of end of first week journaling, and 00:12:51.299 --> 00:12:53.240 then very similar set of questions for the end 00:12:53.240 --> 00:12:56.379 of assessment survey. So that's what worked there. 00:12:56.440 --> 00:12:59.279 In terms of the main positives, and certainly 00:12:59.279 --> 00:13:03.000 formative feedback, again, was extremely valued. 00:13:03.139 --> 00:13:06.799 We saw 100%. return rate in terms of did you 00:13:06.799 --> 00:13:10.200 find the feedback useful going into your eventual 00:13:10.200 --> 00:13:13.120 submission is a 100 % positive response on that. 00:13:13.320 --> 00:13:15.639 And then what we did was make a comparison across 00:13:15.639 --> 00:13:18.200 the other questions in terms of we had the averages 00:13:18.200 --> 00:13:20.480 from the faculty survey and wanted to test if 00:13:20.480 --> 00:13:23.019 our interventions in the pilot was successful. 00:13:23.200 --> 00:13:27.919 So 82 % like the flexibility offered on this 00:13:27.919 --> 00:13:31.350 particular lab practicals that compares to a 00:13:31.350 --> 00:13:33.570 low 7 % in the faculty pool who said they'd experienced 00:13:33.570 --> 00:13:36.950 any kind of flexibility. So that was a certain 00:13:36.950 --> 00:13:40.110 win on that count. The other thing which... we 00:13:40.110 --> 00:13:43.110 also found worked was just in terms of challenging 00:13:43.110 --> 00:13:45.350 assumptions when it comes to analyzing and interpreting 00:13:45.350 --> 00:13:47.789 data, because that is pretty key when, again, 00:13:47.830 --> 00:13:49.669 it might not be just for engineering students, 00:13:49.950 --> 00:13:52.710 but you tend to work in very idealized environments 00:13:52.710 --> 00:13:54.809 in academia. But when you graduate, you should 00:13:54.809 --> 00:13:56.950 be in a position to challenge assumptions and 00:13:56.950 --> 00:14:00.929 theories. And there, again, it was close to 50 00:14:00.929 --> 00:14:04.090 % on our pilot study, which was down around about 00:14:04.090 --> 00:14:09.169 15, 16 % in the faculty poll. But the third thing 00:14:09.169 --> 00:14:11.190 that that I will highlight, and we'll speak a 00:14:11.190 --> 00:14:12.690 little bit about this as we go along when we 00:14:12.690 --> 00:14:15.929 go to staff perspectives, is about Having these 00:14:15.929 --> 00:14:18.529 assessment guidelines in place, which should 00:14:18.529 --> 00:14:20.830 be clear enough to set out the expectations in 00:14:20.830 --> 00:14:22.870 terms of what we want from the students, but 00:14:22.870 --> 00:14:24.830 they should not potentially be hindering the 00:14:24.830 --> 00:14:27.289 expression of creativity. So that is what we 00:14:27.289 --> 00:14:28.889 asked students if they felt that the guidelines 00:14:28.889 --> 00:14:32.789 restricted expression of creativity. 33 % said 00:14:32.789 --> 00:14:35.769 yes in the faculty poll, and it was down to 0 00:14:35.769 --> 00:14:38.850 % on our pilot. So no one felt that the guidelines 00:14:38.850 --> 00:14:41.809 was restricting them from expressing creativity. 00:14:42.600 --> 00:14:44.580 Having said that, all of that, I don't want to 00:14:44.580 --> 00:14:46.779 say that whatever we did, whatever we touched 00:14:46.779 --> 00:14:50.639 turned to gold. There were things where potentially 00:14:50.639 --> 00:14:53.960 did not work as envisioned or as planned. One 00:14:53.960 --> 00:14:57.279 of the things that we are keen on as pedagogic 00:14:57.279 --> 00:14:59.559 researchers, but also as educators, is to make 00:14:59.559 --> 00:15:01.820 the assessment experience enjoyable for students 00:15:01.820 --> 00:15:04.759 during the process. So one of these key questions 00:15:04.759 --> 00:15:07.259 that we ask them is... Do you experience sort 00:15:07.259 --> 00:15:09.480 of a sense of passion and purpose when working 00:15:09.480 --> 00:15:11.679 on the assessment? So that's over the two week 00:15:11.679 --> 00:15:14.720 window. And then there's the overriding sense 00:15:14.720 --> 00:15:17.080 of accomplishment and achievement once they hand 00:15:17.080 --> 00:15:19.539 the submission in after a couple of weeks. And 00:15:19.539 --> 00:15:21.519 usually we see there's a lot of students say, 00:15:21.559 --> 00:15:23.740 yes, we feel as if we've accomplished something 00:15:23.740 --> 00:15:26.360 once they've handed it in. But the experience 00:15:26.360 --> 00:15:28.480 during the course of that two week window is 00:15:28.480 --> 00:15:32.620 probably not as enjoyable. Now, we still continue 00:15:32.620 --> 00:15:34.940 to observe, at least in the pilot that I led 00:15:34.940 --> 00:15:36.440 in the School of Chemical Process Engineering, 00:15:36.700 --> 00:15:38.179 there was still that divide. There were a lot 00:15:38.179 --> 00:15:40.980 more students who felt that, you know, there's 00:15:40.980 --> 00:15:44.720 a sense of achievement after handing in the report, 00:15:44.820 --> 00:15:47.799 the video recording, as opposed to how much actually 00:15:47.799 --> 00:15:49.740 felt that sense of passion, purpose during the 00:15:49.740 --> 00:15:51.500 journey. So I think that's potentially going 00:15:51.500 --> 00:15:53.340 forward is something that we have to continue 00:15:53.340 --> 00:15:56.379 to look at and see how we could improve that 00:15:56.379 --> 00:15:58.559 experience for our students. Thank you both. 00:15:58.600 --> 00:16:02.539 That was a really excellent, comprehensive kind 00:16:02.539 --> 00:16:04.980 of overview of how you drew upon the findings 00:16:04.980 --> 00:16:07.240 from year one and embedded it into your authentic 00:16:07.240 --> 00:16:10.779 assessment delivery in your own contexts. I wonder 00:16:10.779 --> 00:16:13.320 what you'd say are the main sort of implications 00:16:13.320 --> 00:16:16.299 for people who might be listening to this and 00:16:16.299 --> 00:16:18.759 might be interested, hopefully, in embedding 00:16:18.759 --> 00:16:22.679 this as an approach into their own work. Very 00:16:22.679 --> 00:16:26.870 good question, Robert. So some... Probably reflections 00:16:26.870 --> 00:16:29.490 from what we did we can actually share here. 00:16:29.950 --> 00:16:33.970 I think the first point is focusing on assessment 00:16:33.970 --> 00:16:36.970 delivery and the feasibility of that in their 00:16:36.970 --> 00:16:40.070 assessment. So I think it would be really good 00:16:40.070 --> 00:16:44.350 for teaching staff to understand how they can 00:16:44.350 --> 00:16:48.649 incorporate this concept of delivery and provide 00:16:48.649 --> 00:16:51.789 students with the opportunity to have feedback 00:16:51.789 --> 00:16:55.629 through the journey, not just at the end. in 00:16:55.629 --> 00:16:57.649 order to enhance the quality of the assessment 00:16:57.649 --> 00:16:59.769 delivery. I think that was something that we 00:16:59.769 --> 00:17:02.970 really enjoyed ourselves and also students positively 00:17:02.970 --> 00:17:09.569 reacted to that because the journey, and as Manner 00:17:09.569 --> 00:17:13.849 said, if they find the learning process pleasant 00:17:13.849 --> 00:17:17.109 and if they learn something through the journey, 00:17:17.170 --> 00:17:20.089 then the shift will be on the learning rather 00:17:20.089 --> 00:17:24.210 than on the mark. will be much more engaged and 00:17:24.210 --> 00:17:26.410 also the quality of the work as we experience 00:17:26.410 --> 00:17:30.230 in our pilot will be increased. Another thing 00:17:30.230 --> 00:17:33.670 which I can mention here is focus on creativity, 00:17:33.809 --> 00:17:36.589 particularly for disciplines like engineering. 00:17:36.710 --> 00:17:38.829 Creativity is very important because we are not 00:17:38.829 --> 00:17:43.529 only dealing with problems that the final answer 00:17:43.529 --> 00:17:47.970 is clear, is defined. We are working with open 00:17:47.970 --> 00:17:51.089 -ended problems in engineering that particularly 00:17:51.089 --> 00:17:54.480 in this environment in this world everything 00:17:54.480 --> 00:17:56.740 is getting more complex and multidisciplinary 00:17:56.740 --> 00:18:00.660 so the creativity is something that definitely 00:18:00.660 --> 00:18:03.000 we need we need to improve that we need to help 00:18:03.000 --> 00:18:07.319 our learners to have that creativity and so again 00:18:07.319 --> 00:18:09.200 Manoj mentioned different kind of submission 00:18:09.200 --> 00:18:12.460 formats I had the same thing in my pilot so I 00:18:12.460 --> 00:18:15.720 actually helped students to walk through different 00:18:15.720 --> 00:18:19.059 formats like poster the report also oral presentation 00:18:19.059 --> 00:18:25.250 so I think these sort of opportunities give students 00:18:25.250 --> 00:18:30.430 again room for creativity. There's a lot of literature 00:18:30.430 --> 00:18:33.089 in this area. A couple of places people might 00:18:33.089 --> 00:18:36.289 want to do some reading is around super complexity, 00:18:36.589 --> 00:18:39.809 Barnett's concept, you're familiar. Also wicked 00:18:39.809 --> 00:18:44.089 problems, I'm not sure who introduced that. Go 00:18:44.089 --> 00:18:47.390 away and look it up. But that's a super interesting 00:18:47.390 --> 00:18:51.369 concept as well. There are a lot of pedagogies. 00:18:52.960 --> 00:18:56.019 Cradle. It's a Deakin University. Definitely. 00:18:56.200 --> 00:19:00.539 There are a lot of pedagogies on that. Yeah, 00:19:00.660 --> 00:19:05.359 so I mentioned the mark focus thing and probably 00:19:05.359 --> 00:19:08.779 we'll speak about that a little bit more throughout 00:19:08.779 --> 00:19:12.519 this podcast. But focus on mark is something 00:19:12.519 --> 00:19:16.039 that sometimes completely hinders students from 00:19:16.039 --> 00:19:19.259 developing that sort of creativity or the learning 00:19:19.259 --> 00:19:22.339 because they are just... kind of targeting the 00:19:22.339 --> 00:19:24.680 bare minimum and trying to get the mark and pass 00:19:24.680 --> 00:19:27.920 some element of the whole module or element of 00:19:27.920 --> 00:19:33.160 the module. And another thing which we face, 00:19:33.319 --> 00:19:35.640 and I think it's very important, we need probably 00:19:35.640 --> 00:19:38.240 to elaborate on that a little bit more, is the 00:19:38.240 --> 00:19:40.720 institutional support through this journey. Because 00:19:40.720 --> 00:19:44.279 it's not possible to do these things because 00:19:44.279 --> 00:19:47.440 of the workload implications and time limitations 00:19:47.440 --> 00:19:50.369 to do that. any sort of like an institutional 00:19:50.369 --> 00:19:55.470 support any sort of like tools or I don't know 00:19:55.470 --> 00:19:58.710 mechanism that can support educators in order 00:19:58.710 --> 00:20:01.369 to go through this journey and provide more and 00:20:01.369 --> 00:20:04.829 more kind of assessment delivery and flexible 00:20:04.829 --> 00:20:08.289 assessment ideas would be really appreciated 00:20:08.289 --> 00:20:11.609 I think fantastic we could spend a lot of time 00:20:11.609 --> 00:20:13.809 talking about some of these wider implications 00:20:13.809 --> 00:20:16.210 and I think you'd probably encourage people to 00:20:16.210 --> 00:20:19.890 reach out to you to engage with your work What 00:20:19.890 --> 00:20:22.789 sorts of research outputs have you produced? 00:20:23.170 --> 00:20:25.549 And is there anything else that's in the works 00:20:25.549 --> 00:20:28.569 that people can look out for? Yeah, thanks, Robert. 00:20:28.630 --> 00:20:31.210 It's very much in the pipeline. We did take some 00:20:31.210 --> 00:20:34.410 of these results to the CEFI European Engineering 00:20:34.410 --> 00:20:37.309 Education Conference recently, where we ran a 00:20:37.309 --> 00:20:39.549 workshop kind of using our assessment briefs 00:20:39.549 --> 00:20:42.029 in a way, but challenging the delegates to think 00:20:42.029 --> 00:20:45.009 of how would you embed authenticity if you had, 00:20:45.130 --> 00:20:47.640 you know. full reins over how you're going to 00:20:47.640 --> 00:20:51.039 design the assessment. So yeah, that conference 00:20:51.039 --> 00:20:54.059 workshop paper should be online pretty soon. 00:20:54.220 --> 00:20:57.680 But we've also working on a manuscript with specific 00:20:57.680 --> 00:21:00.480 findings from these pilot studies, which hopefully 00:21:00.480 --> 00:21:03.160 should see the light of public domain soon. I'm 00:21:03.160 --> 00:21:05.000 sure it will. And we'll also include a link to 00:21:05.000 --> 00:21:08.420 the show notes, a link in the show notes to Manoj 00:21:08.420 --> 00:21:10.839 and Mohsen's LITE Fellowship profile, which 00:21:10.839 --> 00:21:12.720 will have the snapshot with a summary of the 00:21:12.720 --> 00:21:14.960 findings that we're talking about in this episode. 00:21:16.170 --> 00:21:18.930 Should we move on to your findings from the staff 00:21:18.930 --> 00:21:21.529 perspectives, which helped inform the pilot study? 00:21:21.970 --> 00:21:26.789 Yes. So just to do a short recap, we did the 00:21:26.789 --> 00:21:31.829 staff interview in order to also bring staff 00:21:31.829 --> 00:21:34.990 perspective into the assessment, the authentic 00:21:34.990 --> 00:21:38.369 assessment and flexible assessment ideas. In 00:21:38.369 --> 00:21:42.009 terms of how we did that, we interviewed, we 00:21:42.009 --> 00:21:45.000 had like a semi... structure interview interviewed 00:21:45.000 --> 00:21:48.640 10 academic staff mostly in faculty of engineering 00:21:48.640 --> 00:21:50.859 in different schools and different disciplines 00:21:50.859 --> 00:21:56.039 and also we had stuff from different kind of 00:21:56.039 --> 00:22:00.220 career level we had high level like management 00:22:00.220 --> 00:22:02.700 level stuff like a director of student education 00:22:02.700 --> 00:22:06.140 and also lecturers professors so that was really 00:22:06.140 --> 00:22:09.500 good kind of diverse mix we had from different 00:22:09.500 --> 00:22:12.579 perspective of staff at different levels How 00:22:12.579 --> 00:22:16.779 we brought what we found from that was trying 00:22:16.779 --> 00:22:20.940 to understand, are they reinforcing some of our 00:22:20.940 --> 00:22:24.599 findings or not? One thing that was really highlighted 00:22:24.599 --> 00:22:28.420 during the conversation was the focus on authentic 00:22:28.420 --> 00:22:31.799 delivery again. And most of the stuff they mentioned 00:22:31.799 --> 00:22:34.359 that they find sometimes the assessment process 00:22:34.359 --> 00:22:38.200 is not actually reflecting the professional practices 00:22:38.200 --> 00:22:42.119 completely. Something else which was really interesting 00:22:42.119 --> 00:22:45.660 was because we are in the era of AI, artificial 00:22:45.660 --> 00:22:48.619 intelligence, now they are everywhere. It's a 00:22:48.619 --> 00:22:50.140 great era, I haven't heard anything about it. 00:22:52.059 --> 00:22:57.180 So some of the staff, they mentioned that students 00:22:57.180 --> 00:23:01.440 nowadays are very reliant actually on AI. There 00:23:01.440 --> 00:23:05.279 are a lot of like work academy integrity issues 00:23:05.279 --> 00:23:09.279 because of using AI as like a... black box just 00:23:09.279 --> 00:23:13.299 producing some sort of mostly text and they found 00:23:13.299 --> 00:23:15.500 that probably the assessment delivery would be 00:23:15.500 --> 00:23:19.500 really good tool in order to reduce that over 00:23:19.500 --> 00:23:22.539 reliance on AI because we are giving them this 00:23:22.539 --> 00:23:26.759 opportunity to work on the assessment bit by 00:23:26.759 --> 00:23:30.160 bit through the feedback that we provide and 00:23:30.160 --> 00:23:33.059 I think something that I found myself in my modules 00:23:33.059 --> 00:23:37.420 was that students will rely on AI when they are 00:23:37.420 --> 00:23:39.759 not actually completely sure what they want to 00:23:39.759 --> 00:23:43.079 do, how to do the work. Because in that case, 00:23:43.160 --> 00:23:45.420 they see that they have limited time to deliver 00:23:45.420 --> 00:23:48.599 the work and then they start to rush to do something 00:23:48.599 --> 00:23:50.880 in order to get something from AI and submit 00:23:50.880 --> 00:23:53.819 it. So if we give them the feedback, if we help 00:23:53.819 --> 00:23:56.059 them to learn how to do the work step by step 00:23:56.059 --> 00:24:01.519 and how to deliver the work, get feedback, improve 00:24:01.519 --> 00:24:03.619 the work based on that, I think that would be... 00:24:04.170 --> 00:24:08.369 something that enhances or reduces actually the 00:24:08.369 --> 00:24:13.829 over -reliance on AI. Highlighting the implementation 00:24:13.829 --> 00:24:18.849 barrier occurred a lot actually during the interviews. 00:24:19.349 --> 00:24:23.230 A lot of stuff mentioned that it is really hard 00:24:23.230 --> 00:24:26.230 because of the workload implications to bring 00:24:26.230 --> 00:24:28.809 flexible assessment into their modules because 00:24:28.809 --> 00:24:31.450 they need to sometimes design different rubrics 00:24:31.450 --> 00:24:34.440 for that. They need to change the So it is not 00:24:34.440 --> 00:24:36.819 actually an easy work. It requires a lot of time 00:24:36.819 --> 00:24:40.619 and effort. And despite being really kind of 00:24:40.619 --> 00:24:43.059 interested to do that, but time and workload 00:24:43.059 --> 00:24:45.059 implication is something that they mentioned. 00:24:45.140 --> 00:24:47.960 So that's probably linked to what I mentioned 00:24:47.960 --> 00:24:50.599 before in terms of like institutional support. 00:24:50.940 --> 00:24:55.640 This is something that we can probably we need 00:24:55.640 --> 00:24:59.000 to focus on. Yeah, I completely agree. And just 00:24:59.000 --> 00:25:01.049 a couple of points too. pretty much reinforce 00:25:01.049 --> 00:25:03.809 what Mohsen has just said. So when we interviewed 00:25:03.809 --> 00:25:05.930 staff, and these were semi -structured interviews, 00:25:06.349 --> 00:25:08.549 the overarching themes for me personally on reflection 00:25:08.549 --> 00:25:11.069 kind of centered around two main things. One 00:25:11.069 --> 00:25:14.670 was to go from, we design assessments with specific 00:25:14.670 --> 00:25:17.210 intended learning outcomes is what we're assessing 00:25:17.210 --> 00:25:20.089 for, which is very much the correct approach. 00:25:20.390 --> 00:25:23.789 But also to think of in, as Mohsen has just alluded 00:25:23.789 --> 00:25:28.210 to, in the AI -based era, to kind of make that 00:25:28.210 --> 00:25:30.849 shift potentially towards more a process -based 00:25:30.849 --> 00:25:33.450 assessment and not just looking at the final 00:25:33.450 --> 00:25:36.789 deliverable and the final outcome was very much 00:25:36.789 --> 00:25:39.430 an emerging theme when we had these conversations 00:25:39.430 --> 00:25:42.369 with staff. And it was about how do we make sure 00:25:42.369 --> 00:25:44.329 that we, A, create the right environment for 00:25:44.329 --> 00:25:48.750 students to work on tasks authentically, express 00:25:48.750 --> 00:25:50.789 creativity, and that would mean setting up the 00:25:50.789 --> 00:25:52.970 right environment. And then you kind of monitor 00:25:52.970 --> 00:25:56.000 the process as they... work towards the deliverable. 00:25:56.119 --> 00:26:00.099 So that is the first point to emphasize. And 00:26:00.099 --> 00:26:02.460 the second one that staff also picked up a lot 00:26:02.460 --> 00:26:04.559 on in the interviews that we did was sort of 00:26:04.559 --> 00:26:06.299 this approach potentially to go more towards 00:26:06.299 --> 00:26:09.539 holistic marking as opposed to reductionist marking 00:26:09.539 --> 00:26:11.650 of saying, okay, you've got... that element of 00:26:11.650 --> 00:26:14.589 the assessment done so you get a mark for that 00:26:14.589 --> 00:26:18.950 kind of approach, which, again, a lot of us continue 00:26:18.950 --> 00:26:22.430 to use that approach. And that is very much clear, 00:26:22.549 --> 00:26:24.049 kind of that kind of approach when you explain 00:26:24.049 --> 00:26:25.869 to students and when you set up the rubrics. 00:26:26.009 --> 00:26:28.509 But that's probably where that element of training, 00:26:28.690 --> 00:26:31.609 both from a staff training, but also a student 00:26:31.609 --> 00:26:34.569 training perspective of how does holistic marking 00:26:34.569 --> 00:26:37.029 work? And because it might not be necessarily 00:26:37.029 --> 00:26:40.569 as clear as the conventional rubric. and marking 00:26:40.569 --> 00:26:43.349 grading criteria, which is what staff and students 00:26:43.349 --> 00:26:45.990 are more used to. So those are the kind of changes 00:26:45.990 --> 00:26:48.509 where potentially we have to kind of envision 00:26:48.509 --> 00:26:52.470 going forward. Great. And something I'd really 00:26:52.470 --> 00:26:55.309 like listeners to learn more about, because it 00:26:55.309 --> 00:26:58.390 wasn't just you interpreting the findings, was 00:26:58.390 --> 00:27:00.890 people you worked with. Could you tell us a bit 00:27:00.890 --> 00:27:03.279 more about that? Absolutely. That was one of 00:27:03.279 --> 00:27:05.400 the most exciting things, the fellowship, wasn't 00:27:05.400 --> 00:27:09.220 it? So, yeah, spoken about this a little bit 00:27:09.220 --> 00:27:11.839 at some of the LITE events, but for people who 00:27:11.839 --> 00:27:14.420 weren't there, the idea was we were sitting on 00:27:14.420 --> 00:27:16.940 this data, the staff interview data that we've 00:27:16.940 --> 00:27:20.880 just spoken about. And we as two academics pretty 00:27:20.880 --> 00:27:22.619 much said, OK, so we're going to go through this, 00:27:22.660 --> 00:27:26.420 apply effects of thematic analysis. But we wondered 00:27:26.420 --> 00:27:28.799 if the richness of the data would not be fully 00:27:28.799 --> 00:27:31.039 captured if we were just two members of academic 00:27:31.039 --> 00:27:34.799 staff looking at staff interview data. So one 00:27:34.799 --> 00:27:37.640 of the things that we wanted. to do was involve 00:27:37.640 --> 00:27:40.400 student research assistants as core researchers 00:27:40.400 --> 00:27:44.579 on the staff interviews. So we could tap into 00:27:44.579 --> 00:27:47.519 that valuable student positionality when doing 00:27:47.519 --> 00:27:49.440 the reflexive thematic analysis. So it's not 00:27:49.440 --> 00:27:52.119 just a staff perspective of looking at things. 00:27:52.500 --> 00:27:55.740 And we ended up getting two students who were 00:27:55.740 --> 00:27:58.039 certainly far more competent than the two of 00:27:58.039 --> 00:28:01.480 us and ended up doing a wonderful job of lending 00:28:01.480 --> 00:28:04.640 that student positionality to looking at the 00:28:04.640 --> 00:28:07.579 data. was interesting because we ended up analyzing 00:28:07.579 --> 00:28:09.980 our data independently, of course, because we 00:28:09.980 --> 00:28:13.220 didn't want to have any kind of bias or influencing 00:28:13.220 --> 00:28:16.440 them and vice versa. Then when we got together 00:28:16.440 --> 00:28:18.400 through a series of meetings, eventually sharing 00:28:18.400 --> 00:28:20.539 our notes and seeing how we'd interpreted things, 00:28:20.740 --> 00:28:23.400 it was really refreshing to see how those two 00:28:23.400 --> 00:28:25.319 different perspectives on technically the same 00:28:25.319 --> 00:28:28.430 data set, but I think that is... Something really, 00:28:28.470 --> 00:28:30.630 truly unique to qualitative research and as people 00:28:30.630 --> 00:28:32.369 who come from a very quantitative background 00:28:32.369 --> 00:28:35.769 where it doesn't matter who's looking at a quantitative 00:28:35.769 --> 00:28:37.789 data set, you can interpret it only one way, 00:28:37.869 --> 00:28:40.309 but that's where qualitative data is very different. 00:28:40.329 --> 00:28:42.549 It's been a very rich learning experience for 00:28:42.549 --> 00:28:44.829 us as well. So yeah, a great experience and I 00:28:44.829 --> 00:28:47.349 recommend anyone who's interested in pedagogic 00:28:47.349 --> 00:28:49.789 research where you can embed that student perspective, 00:28:50.190 --> 00:28:51.930 go for it. It's a richly rewarding experience. 00:28:52.410 --> 00:28:54.309 I think whether you're a staff member considering 00:28:54.309 --> 00:28:56.849 doing pedagogical research, possibly for a LITE 00:28:56.849 --> 00:28:59.519 Fellowship, Look into possibilities for working 00:28:59.519 --> 00:29:02.700 co -creatively with students. We have a team 00:29:02.700 --> 00:29:05.059 of student research assistants. Equally, if you're 00:29:05.059 --> 00:29:06.539 a student that happens to be listening to this 00:29:06.539 --> 00:29:08.380 podcast, look out for opportunities to get involved 00:29:08.380 --> 00:29:10.960 in this kind of research. I'm sure there are 00:29:10.960 --> 00:29:13.480 some students who are really keen on our podcast 00:29:13.480 --> 00:29:18.740 series. Maybe then you could share with any particular 00:29:18.740 --> 00:29:21.440 elements of the findings that I think it was 00:29:21.440 --> 00:29:25.359 Winnie and Asa were struck by. played a particular 00:29:25.359 --> 00:29:30.920 role in you developing your findings? So something 00:29:30.920 --> 00:29:35.559 that came out from that work was that authenticity 00:29:35.559 --> 00:29:39.799 actually is about like underlying the skills. 00:29:39.920 --> 00:29:43.119 So the development of the skills, which it's 00:29:43.119 --> 00:29:45.000 really good because at University of Leeds, we 00:29:45.000 --> 00:29:48.259 have very nice kind of approach towards skill 00:29:48.259 --> 00:29:51.640 development. And we have this idea of skill matrix. 00:29:52.799 --> 00:29:56.220 So I think if listeners probably can search about 00:29:56.220 --> 00:29:57.980 skill metrics at the University of Leeds, that's 00:29:57.980 --> 00:30:01.359 a really nice idea to learn about that. So I 00:30:01.359 --> 00:30:06.619 think they believe that the focus should be on 00:30:06.619 --> 00:30:09.420 underlying skills like critical thinking, problem 00:30:09.420 --> 00:30:13.319 solving and reflection and this sort of stuff, 00:30:13.440 --> 00:30:18.039 because these skills. will prepare learners for 00:30:18.039 --> 00:30:20.099 the professional environment. When they are in 00:30:20.099 --> 00:30:22.420 that environment, that sort of skill that they 00:30:22.420 --> 00:30:25.519 learn through the journey of education will help 00:30:25.519 --> 00:30:30.920 them to walk towards different problems. Also, 00:30:31.019 --> 00:30:37.319 the flexibility is suggested as backbone of authentic 00:30:37.319 --> 00:30:44.129 assessment because not only it supports students 00:30:44.129 --> 00:30:47.450 through the creativity and working towards different 00:30:47.450 --> 00:30:49.410 kind of submission form and this sort of stuff. 00:30:49.490 --> 00:30:52.509 But also it's very inclusive approach. It is 00:30:52.509 --> 00:30:56.569 very nice. It can actually cater for the requirement 00:30:56.569 --> 00:30:59.349 of students from different backgrounds and different 00:30:59.349 --> 00:31:02.869 needs. The crucial role of feedback, I think 00:31:02.869 --> 00:31:07.029 we discussed this a lot and that is something 00:31:07.029 --> 00:31:11.779 that again and again, even through the... analysis 00:31:11.779 --> 00:31:15.059 carried out by Winnie and Asa was highlighted 00:31:15.059 --> 00:31:18.759 and also they from their own feedback they found 00:31:18.759 --> 00:31:21.680 that feedback from their own experience they 00:31:21.680 --> 00:31:24.380 found that feedback would help them to improve 00:31:24.380 --> 00:31:28.000 the work a lot. Something else which I can share 00:31:28.000 --> 00:31:32.960 about that is some implications of the assessment 00:31:32.960 --> 00:31:36.599 deadlines and assessment kind of arrangements 00:31:36.599 --> 00:31:39.539 that probably needs to be looked at. from the 00:31:39.539 --> 00:31:41.980 program level, not only module level, because 00:31:41.980 --> 00:31:44.500 sometime in the program level, we have series 00:31:44.500 --> 00:31:48.539 of modules and by adding more assessments in 00:31:48.539 --> 00:31:51.900 order to make the process more flexible, we might 00:31:51.900 --> 00:31:55.039 actually have a couple of like deadlines happening 00:31:55.039 --> 00:31:57.160 at the same time or clashes between deadlines 00:31:57.160 --> 00:31:59.900 or sometime the assessment fatigue might be part 00:31:59.900 --> 00:32:04.220 of that. So I think the program level view of 00:32:04.220 --> 00:32:06.680 the assessment delivery design is something that 00:32:06.680 --> 00:32:09.990 we need to focus. when we are speaking about 00:32:09.990 --> 00:32:15.109 that. So, obviously, training, again, as Manoj 00:32:15.109 --> 00:32:18.049 mentioned and also as highlighted before, even 00:32:18.049 --> 00:32:22.490 for stuff like for CPD sessions and this sort 00:32:22.490 --> 00:32:25.450 of stuff, for stuff to learn how to implement 00:32:25.450 --> 00:32:28.890 these ideas and effectively actually do the work 00:32:28.890 --> 00:32:31.690 without putting a lot of time and effort on that 00:32:31.690 --> 00:32:38.670 would be really good. And eventually the co -creation 00:32:38.670 --> 00:32:41.990 part of that. I think if we can sit down with 00:32:41.990 --> 00:32:44.650 students and listen to them in terms of their 00:32:44.650 --> 00:32:46.970 perception about the assessment. Because something 00:32:46.970 --> 00:32:49.990 that, again, was mentioned by Manoj and we found 00:32:49.990 --> 00:32:52.890 during the first part of the research was that 00:32:52.890 --> 00:32:55.970 the guidance needs to be clear for students. 00:32:56.049 --> 00:32:58.410 But at the same time, that shouldn't be very 00:32:58.410 --> 00:33:01.210 perspective to limit them and to put them in 00:33:01.210 --> 00:33:04.220 a specific channel. So something like that, I 00:33:04.220 --> 00:33:07.539 think it would be really nice. And our student 00:33:07.539 --> 00:33:09.460 researchers, they mentioned that it would be 00:33:09.460 --> 00:33:12.579 really useful to sit down with students to understand 00:33:12.579 --> 00:33:16.460 how to actually design a specific guidance that 00:33:16.460 --> 00:33:19.700 provides that sort of freedom, but also provides 00:33:19.700 --> 00:33:22.680 a clear understanding about the expectation of 00:33:22.680 --> 00:33:26.000 the work. Fantastic. Now, the next question I've 00:33:26.000 --> 00:33:28.619 got, what sorts of changes would you like to 00:33:28.619 --> 00:33:32.000 see going forwards? I feel as though listeners 00:33:32.000 --> 00:33:33.500 might already have a good sense of what some 00:33:33.500 --> 00:33:35.180 of those might be, which I think suggests that 00:33:35.180 --> 00:33:37.240 you embedded that sort of impact thinking into 00:33:37.240 --> 00:33:39.400 your research really effectively. What would 00:33:39.400 --> 00:33:42.440 you say to that? I think I'd probably go back 00:33:42.440 --> 00:33:45.759 to one of the top points that we've tried to 00:33:45.759 --> 00:33:47.759 make through this podcast, but also in part one, 00:33:47.839 --> 00:33:50.980 was the delivery aspect of authenticity. So it's 00:33:50.980 --> 00:33:52.960 really creating that environment and thinking 00:33:52.960 --> 00:33:56.259 about, not as an educator, how is the student 00:33:56.259 --> 00:33:59.000 going to prove that? you know, they've met a 00:33:59.000 --> 00:34:01.220 specific learning outcome, which is certainly 00:34:01.220 --> 00:34:04.200 something that should be front and centre. But 00:34:04.200 --> 00:34:06.480 then how will we be enabling students over that 00:34:06.480 --> 00:34:09.500 journey of evidencing meeting the learning outcome? 00:34:09.659 --> 00:34:12.280 And I think that requires a lot of conscious 00:34:12.280 --> 00:34:15.000 thought and design, and it requires quite a lot 00:34:15.000 --> 00:34:17.699 of effort as well, making sure that that scaffolding 00:34:17.699 --> 00:34:21.239 is in place. And as Mohsen said, that delicate 00:34:21.239 --> 00:34:24.000 balance of how do you foster creativity to the 00:34:24.000 --> 00:34:26.010 sense that... Yeah, the guidelines shouldn't 00:34:26.010 --> 00:34:28.070 be stifling them from potentially expressing 00:34:28.070 --> 00:34:30.730 creativity. But at the same time, there should 00:34:30.730 --> 00:34:34.090 be a reasonably clear set of expectations of 00:34:34.090 --> 00:34:37.210 what the assessment is asking them. um to do 00:34:37.210 --> 00:34:39.010 which was what our student research assistants 00:34:39.010 --> 00:34:41.309 picked up as well when looking through that data 00:34:41.309 --> 00:34:43.309 where you know it's easy for us to say you know 00:34:43.309 --> 00:34:45.590 we want to pioneer flexible assessments and let's 00:34:45.590 --> 00:34:48.349 give them a lot of choice but choice can be overwhelming 00:34:48.349 --> 00:34:50.989 at times and uh and that's not where we want 00:34:50.989 --> 00:34:53.550 to take our students uh necessarily right so 00:34:53.550 --> 00:34:56.230 i think it is those things that we need to it's 00:34:56.230 --> 00:34:58.409 a delicate trade -off on a number of those elements 00:34:58.409 --> 00:35:00.730 um which requires as i said i think conscious 00:35:00.730 --> 00:35:04.210 thought and design and something we touched on 00:35:04.210 --> 00:35:06.469 earlier as well as Possible areas for further 00:35:06.469 --> 00:35:08.989 research? Yeah, personally for me, reflecting 00:35:08.989 --> 00:35:11.030 on the pilot and some of the things that I said 00:35:11.030 --> 00:35:13.929 there, Robert, in terms of, yeah, things that 00:35:13.929 --> 00:35:16.110 we potentially wanted to see further up, not 00:35:16.110 --> 00:35:18.510 quite there, was this element of creativity. 00:35:19.010 --> 00:35:20.829 I think one thing which has certainly worked 00:35:20.829 --> 00:35:23.150 is the formative feedback. And in terms of what 00:35:23.150 --> 00:35:26.429 next on that count, I would go. How do we scale 00:35:26.429 --> 00:35:28.710 this up, particularly if you're dealing with 00:35:28.710 --> 00:35:31.309 significantly large student cohorts, well in 00:35:31.309 --> 00:35:34.429 excess of 100 students? How do you embed that 00:35:34.429 --> 00:35:37.170 formative feedback loop multiple number of times 00:35:37.170 --> 00:35:39.989 if you're the solo instructor? So that that is 00:35:39.989 --> 00:35:43.909 very much the I think the obvious research question 00:35:43.909 --> 00:35:46.730 on that basis. But the other one that I'd be 00:35:46.730 --> 00:35:49.050 interested in to kind of tap into a lot more 00:35:49.050 --> 00:35:53.639 is around creativity, because I think we. Sort 00:35:53.639 --> 00:35:55.840 of undid the first element, which is we did not 00:35:55.840 --> 00:35:58.760 want our guidelines to stifle creativity, as 00:35:58.760 --> 00:36:00.719 I said, and our students certainly agreed with 00:36:00.719 --> 00:36:02.880 that over the course of the pilot. They did say 00:36:02.880 --> 00:36:05.500 that the guidelines did not restrict them from 00:36:05.500 --> 00:36:08.239 expressing creativity. But just saying the guidelines 00:36:08.239 --> 00:36:10.920 did not restrict them does not necessarily translate 00:36:10.920 --> 00:36:13.159 into, yes, I could express creativity through 00:36:13.159 --> 00:36:15.460 this assessment, because those two questions 00:36:15.460 --> 00:36:18.239 did not necessarily receive the same sort of 00:36:18.239 --> 00:36:20.719 commensurate. proportionate percentage responses. 00:36:21.139 --> 00:36:23.280 So I think that's the sort of the next step in 00:36:23.280 --> 00:36:26.079 the ladder, if you will, of, okay, yes, the guidelines 00:36:26.079 --> 00:36:28.079 are not restricting them, but what more can we 00:36:28.079 --> 00:36:31.300 do to kind of, you know, encourage them to actively 00:36:31.300 --> 00:36:34.300 be more creative? So those are sort of personally 00:36:34.300 --> 00:36:37.820 for me, what's next in life. Very good points 00:36:37.820 --> 00:36:41.119 mentioned by Manoj. But on top of that, I'm personally 00:36:41.119 --> 00:36:45.590 interested in... some initiatives like automatic 00:36:45.590 --> 00:36:48.969 kind of feedback formative feedback systems if 00:36:48.969 --> 00:36:52.269 we can implement that in high quality level in 00:36:52.269 --> 00:36:56.670 order to provide constructive and feedback the 00:36:56.670 --> 00:36:59.309 student can work based on that in order to improve 00:36:59.309 --> 00:37:03.710 their work so in this case instead of like reliance 00:37:03.710 --> 00:37:07.889 on AI they might be much more interested to submit 00:37:07.889 --> 00:37:10.190 the piece of work get feedback and at the same 00:37:10.190 --> 00:37:13.949 time the implications of additional workload 00:37:13.949 --> 00:37:18.449 for big cohorts particularly can be somehow resolved 00:37:18.449 --> 00:37:21.130 there are a lot of like nice works that we saw 00:37:21.130 --> 00:37:24.369 actually when we were in the conference in the 00:37:24.369 --> 00:37:26.969 CEFI conference in Finland people are doing an 00:37:26.969 --> 00:37:30.050 automatic kind of feedback methods which can 00:37:30.050 --> 00:37:34.449 be looked at I think as a solution for these 00:37:34.449 --> 00:37:36.840 so I believe there are a lot of things that can 00:37:36.840 --> 00:37:39.860 be done in order to enhance the assessment experience 00:37:39.860 --> 00:37:42.980 for students and make this very rich assessment 00:37:42.980 --> 00:37:46.880 journey. And probably we mentioned a few of them. 00:37:46.940 --> 00:37:51.219 People might end up with much richer ideas in 00:37:51.219 --> 00:37:54.920 that sense. So last but not least, perhaps my 00:37:54.920 --> 00:37:58.079 favourite question. Do you have any further advice 00:37:58.079 --> 00:38:00.280 in addition to the advice you gave this time 00:38:00.280 --> 00:38:02.159 last year? I don't know if you can remember what 00:38:02.159 --> 00:38:05.289 advice you gave. do you have any additional advice 00:38:05.289 --> 00:38:07.989 for others conducting or thinking of conducting 00:38:07.989 --> 00:38:10.710 pedagogical research scholarship of teaching 00:38:10.710 --> 00:38:14.329 and learning that's a difficult question to finish 00:38:14.329 --> 00:38:17.090 the podcast isn't it yeah I'd probably go just 00:38:17.090 --> 00:38:19.809 being one of the things in general with research 00:38:19.809 --> 00:38:23.420 is yeah is that You have to trust your ability. 00:38:23.900 --> 00:38:27.579 It is uncharted territory and you'd have to be 00:38:27.579 --> 00:38:29.780 exposed to potentially new research methodologies. 00:38:29.820 --> 00:38:32.280 It's very much been the case with us for this 00:38:32.280 --> 00:38:34.599 fellowship project. But it is just embracing 00:38:34.599 --> 00:38:38.119 that uncertainty. And, well, I think we have 00:38:38.119 --> 00:38:39.460 to be thankful that here at the University of 00:38:39.460 --> 00:38:41.940 Leeds we have a good support infrastructure for 00:38:41.940 --> 00:38:45.219 pedagogic research. I think it's leveraging all 00:38:45.219 --> 00:38:47.920 of those different support opportunities and, 00:38:47.940 --> 00:38:50.530 yeah, continuing to... to trust in your abilities 00:38:50.530 --> 00:38:53.429 and being driven by that passion of all of us 00:38:53.429 --> 00:38:55.969 shared this vision of wanting to improve the 00:38:55.969 --> 00:38:58.590 student learning experience. And I think if we 00:38:58.590 --> 00:39:01.030 continue to draw inspiration from that, we should 00:39:01.030 --> 00:39:03.570 be very successful pedagogic researchers as well. 00:39:04.610 --> 00:39:06.929 I remember during the first episode, actually, 00:39:06.969 --> 00:39:09.130 I said I would not give any advice. I give suggestions, 00:39:09.269 --> 00:39:11.469 but this time I will give one single advice. 00:39:12.570 --> 00:39:16.269 I think my single advice would be for people 00:39:16.269 --> 00:39:19.880 thinking about pedagogical research. please if 00:39:19.880 --> 00:39:22.019 you are interested do that this is really nice 00:39:22.019 --> 00:39:24.519 journey particularly for my engineering fellows 00:39:24.519 --> 00:39:27.599 because in engineering field people usually look 00:39:27.599 --> 00:39:30.079 at that as something very strange something very 00:39:30.079 --> 00:39:32.739 new they are not interested but through this 00:39:32.739 --> 00:39:36.219 journey we found that really nice and very providing 00:39:36.219 --> 00:39:38.739 us with very deep kind of insight in different 00:39:38.739 --> 00:39:42.269 things and also it was really nice to have network 00:39:42.269 --> 00:39:44.610 with other kind of engineering fields in other 00:39:44.610 --> 00:39:47.070 settings around the globe that they are working 00:39:47.070 --> 00:39:48.969 on the same things they are actually facing the 00:39:48.969 --> 00:39:51.829 same challenges so that's my single advice get 00:39:51.829 --> 00:39:54.309 into this pedagogical research kind of field 00:39:54.309 --> 00:39:56.369 particularly engineering field because we need 00:39:56.369 --> 00:39:58.329 that we need actually to improve the assessment 00:39:58.329 --> 00:40:01.650 particularly in this kind of over complex world 00:40:01.650 --> 00:40:04.829 to provide students with the skills that they 00:40:04.829 --> 00:40:08.900 need for professional practices I wonder if there's 00:40:08.900 --> 00:40:10.820 anything particular about sort of your experience 00:40:10.820 --> 00:40:13.000 working with students in the research process 00:40:13.000 --> 00:40:14.960 that hopefully this is something that listeners 00:40:14.960 --> 00:40:17.579 will be considering doing. What was that process 00:40:17.579 --> 00:40:19.960 like? I know you spoke about that at a LITE 00:40:19.960 --> 00:40:24.460 event recently. Yeah. I think you did. Yeah, 00:40:24.519 --> 00:40:26.179 I certainly did. Yeah, I was sort of scratching 00:40:26.179 --> 00:40:29.559 my head as to what I said there. No, I think 00:40:29.559 --> 00:40:31.739 one of the things with working with the student 00:40:31.739 --> 00:40:34.699 research assistants, I think is it's a really 00:40:34.699 --> 00:40:37.369 rewarding experience because. A, you do not know 00:40:37.369 --> 00:40:40.130 what the outcome is, what the output is necessarily 00:40:40.130 --> 00:40:42.969 going to be, but it is ensuring that you provide 00:40:42.969 --> 00:40:47.369 reasonably good structure to the research process, 00:40:47.510 --> 00:40:49.570 which in a way, if I had to draw a parallel, 00:40:49.650 --> 00:40:52.650 it works reasonably well to co -creation of assessments, 00:40:52.929 --> 00:40:56.269 for instance, which is not a research exercise 00:40:56.269 --> 00:40:59.530 on its own. But you're very much there saying 00:40:59.530 --> 00:41:02.250 there's a fair degree of scaffolding in terms 00:41:02.250 --> 00:41:04.500 of where you potentially... where, you know, 00:41:04.519 --> 00:41:08.460 just direction of travel. But the eventual destination 00:41:08.460 --> 00:41:11.500 is a little bit flexible there in terms of the 00:41:11.500 --> 00:41:14.559 student co -creation element. So, yeah, I would 00:41:14.559 --> 00:41:16.219 say it was something that we'd not envisioned 00:41:16.219 --> 00:41:18.340 at the start of our LITE Fellowship that we 00:41:18.340 --> 00:41:20.460 would. We certainly wanted to do the staff interviews, 00:41:20.579 --> 00:41:24.340 but in terms of the analysis, we hadn't thought 00:41:24.340 --> 00:41:26.380 that far in terms of we would be lacking the 00:41:26.380 --> 00:41:28.159 student positionality. It was something that 00:41:28.159 --> 00:41:31.039 as we sort of gained exposure to reflexive thematic 00:41:31.039 --> 00:41:35.130 analysis that... that kind of entered our, yeah, 00:41:35.210 --> 00:41:37.210 our sort of thought space and we took it from 00:41:37.210 --> 00:41:39.309 there. So I would certainly, I think the advice 00:41:39.309 --> 00:41:41.230 there, well, again, don't think I'm necessarily 00:41:41.230 --> 00:41:44.369 qualified enough to be giving advice for people 00:41:44.369 --> 00:41:46.409 to be doing pedagogic research. But yeah, it 00:41:46.409 --> 00:41:48.570 would just be being open to ideas and exploring 00:41:48.570 --> 00:41:50.869 those possibilities. And I think students do 00:41:50.869 --> 00:41:54.190 provide a great perspective that we easily miss. 00:41:54.389 --> 00:41:58.309 Yeah. Thumbs up from us then. And you're not 00:41:58.309 --> 00:42:01.889 giving any more advice, you've said that. The 00:42:01.889 --> 00:42:04.329 theme of sort of like there's uncertainty in 00:42:04.329 --> 00:42:06.909 pedagogical research. You're not necessarily 00:42:06.909 --> 00:42:10.190 using methods or theories that you are familiar 00:42:10.190 --> 00:42:12.349 with. You might be working with students in a 00:42:12.349 --> 00:42:14.829 new way. But that is also part of the learning 00:42:14.829 --> 00:42:16.369 and part of the research. And I think that's 00:42:16.369 --> 00:42:17.789 a really interesting, it's something that comes 00:42:17.789 --> 00:42:23.550 up a lot in these episodes. Well, all that remains 00:42:23.550 --> 00:42:26.190 to say is thanks again for joining us. Thank 00:42:26.190 --> 00:42:28.250 you very much for hosting us, Robert. Happy year 00:42:28.250 --> 00:42:32.070 two. The fellowship went well. And don't worry, 00:42:32.130 --> 00:42:33.610 you won't miss us too much. I'm sure we'll be 00:42:33.610 --> 00:42:36.690 inviting you back to future events to talk about 00:42:36.690 --> 00:42:40.269 your fellowship and your experiences. Great, 00:42:40.329 --> 00:42:45.250 I'll write that down. Thanks for listening, everybody. 00:42:45.670 --> 00:42:48.650 Do check out the show notes for further links 00:42:48.650 --> 00:42:51.710 to the research. And we'll see you next time. 00:42:52.190 --> 00:42:53.670 Thank you so much for having us. Thank you very 00:42:53.670 --> 00:42:53.909 much.