Welcome to Light Bites, an occasional podcast from Leeds Institute for Teaching Excellence
at the University of Leeds. Episodes will be hosted by members of the Light team. And
we'll be showcasing the scholarship of teaching and learning from across the University.
Hello you're listening to Light Bites. I'm Robert Avery's Research and Impact Officer
at the Leeds Institute for Teaching Excellence. Two for the price of one today, because we're
joined by not one but two Light Fellows who are halfway through their two-year Light Fellowship,
which is called Assessment Flexibility to Embed Real Professional Practice. Good afternoon,
Mr. Ravie, Mohsen Besharad. How are you both? Very good, thank you. Very good, thanks.
Very good, thanks. Excited to get started? Yeah, absolutely. Excellent. So why don't
we begin by you telling us a little bit about what you do at the University of Leeds. Good
afternoon, Robert. Thank you for having us in this podcast session. So my name is Mohsen
Besharad. I'm a Curriculum Redefined Lecturer at the School of Civil Engineering, joined
the University of Leeds in 2022. I'm also Program Leader for Undergraduate, three undergraduate
programs, Civil and Structural Engineering, Civil and Environmental Engineering and Civil
Engineering. And yeah, thank you so much for having us. Thank you, Robert. Delighted to
be on the podcast here with you. Look forward to discussing what we've really gathered in
terms of data and findings over the course of the last year. So I'm Manoj Ravie over
from the School of Chemical and Process Engineering. So I work as a lecturer there and like Mohsen,
joined the University in 2022 as a Curriculum Redefined Lecturer. Fantastic. Let's get on
to talking about your fellowship then. What motivated you to undertake this piece of research?
Great. So there are a lot of motivations, but for me, actually, I have background in
industry and also in academia. And when I was working in industry, I observed a lot
of like graduates that they come to industry, they are struggling to use their skills developed
during education in terms of their communication, developing the reports, doing the work itself,
because in academia mostly we teach students to focus on a perfect piece of the work to
get the final mark. And yeah, so that motivated me to look at the gap between these kind of
approach we have in academia and the requirement of industry to see how we can improve students
to be more work ready. Yeah, just building on what Mohsen said there, I think my interests
were very similar as well. So in terms of when students are going out there, there is
this general narrative which is painted outside of higher education that students are unnecessarily
ready to take on the challenges of the professional world. But having said that, when we think
of it from within academia and within higher education circles, the whole concept of authentic
assessments has been around for a really long time. So it was that dichotomy which I wanted
to address saying we as educators are saying that we're making assessments increasingly
authentic across all disciplines. But despite our claims of doing that, it's not being met
by the same opinion on the outside. So what is the reason or potentially multiple reasons,
but what are these and how could they be addressed going forward? That was the real thing that
I want to hammer in on over the course of this fellowship. Fantastic. Could you explain
to listeners a little bit about how authentic assessment delivery differs from authentic
assessment design? So the authentic assessment design is mostly focusing on the design of
the assessment brief or work itself that mimics the work in people doing the professional
world. But that's the brief which we deliver to students in order to start to work on that.
The delivery side of that starts right after that when students start to work on their
assessment. So the whole process from reading the brief to develop the work itself, to communicate
with other peers, to communicate with instructors, to go through different resources to understand
how to solve that problem and also to the final delivery point is the delivery of an
assessment that we have in mind. Would you say then that the relevance of your fellowship
and its findings can transcend your discipline and impact authentic assessment delivery in
other disciplines? Oh yes, very much so. Absolutely. Because if you look at a lot of the recently
published pedagogic research and scholarship articles, potentially critiquing some of the
use of the term authentic assessment because it has become a buzzword but it's potentially
not being used in the real depth and richness that the term encompasses. And that's where
we were looking at it in terms of this design and delivery component as you referred to
Robert. So where we're not just looking at the question or the task which has been given
to the student which might or might not resemble to some extent the challenge of a professional
but it's about how does the student actually go about addressing that challenge. So if
they have a couple of weeks to come up with a report, poster, whatever the assessment
is, how much of that working practice is similar to how a professional would work on a similar
challenge over two weeks time. And that when I say professional pretty much sweeps across
all disciplines. It's not just engineers or scientists but any discipline which is why
I think the project has relevance for everyone. Fantastic. You're one year into your fellowship
and you have preliminary findings. Looking forward to hearing about what they are and
how that's going to inform the second year of your fellowship. Before we go into findings,
I'm sure that listeners are really keen to hear about how you've got to the point that
you're at now. How then did you go about research design and conducting research up until this
point? Well Robert, this is a two-year fellowship
which the first year is the planning phase and the second year would be the pilot phase.
So we are at the end of the first year. For the planning phase, we went after all the
ethical approval and this sort of stuff, we started to do a survey in the Faculty of Engineering
and Physical Science and the survey was designed based on a couple of criterias. For example,
we have something which is called employer indicator framework in university developed
based on a project with industrial and external partners which gave us a good insight at what
are the main criterion that employers are looking when they are searching for a graduate
to do employ. So we took something like peer review, the commercial awareness, reflective
practices, engaging with multiple conflicting requirements like main takeaways from that
project. Also, we look at the broader assessment practice that can replicate the professional
practices in order to bring something also from that like time and project management,
controlling the stress, anxiety, the passion and accomplishment that comes after doing
the project, the creativity and also we looked at the potential solution for that which is
assessment flexibility. We embedded these points into the survey and also we, so most
of these things were embedded through a Likert scale questionnaire and then we added one
open end text based question also in order to have more in depth insight into the perception
of students regarding the assessment and also these criteria. So that phase as I mentioned
has been done, we have responses from students, we have analyzed them which we will speak
about them more in depth today and the second phase, the pilot phase starts from the next
academy year 2024-25 when we, for now we have verified three modules in undergraduate and
also postgraduate levels in order to apply this concept in the assessment and we are
assessing two additional modules to potentially get that to five modules to pilot this and
yeah we are here to discuss findings from the student survey mostly from the planning
phase.
Great, so could you give us a sense then of the kinds of questions students have asked?
Yeah, indeed so we ask questions like I said in terms of the criterion about how much the
submission deadline stress you, we mentioned questions like to what extent do you feel
you can express creativity in your coursework assessment, how much flexibility do you currently
have in your assessment, also we try to understand how, what is the student's perception in
terms of the incentivizing the flexibility for like something like early submission as
something that can be a flexibility, do they prefer that through bonus marks or do they
prefer through the feedback, also we had some because you know we mentioned the authenticity
can be through the design and also delivery so we had some questions in order to verify
the design side of the authenticity in terms of to what extent students are using different
resources for their coursework, how they feel in terms of learning on the go and these kind
of questions and also another part of that like I mentioned was an open end question
which was reflecting on a recent coursework submission comment on what changes could be
made to make the assessment less stressful for you and that would have helped you come
up with a more creative and high quality submission, yeah those were some sample questions.
Brilliant that's a really nice segue because I was just about to ask what did you find
out in this phase of data collection?
Thank you Robert, yeah lots of interesting findings that's come through and I'll probably
go back again to that to the two branches that we were looking at initially which was
authenticity in design and authenticity in delivery.
Students survey across all engineering schools and the sciences within the Faculty of Engineering
and Physical Sciences pretty much unanimously say that students are indeed using a diverse
range of resources to prepare their assessments for instance for coursework so it's not that
they're relying on just lecture notes to prepare things be it posters, reports, whatever form
of assessment so we're really doing a good job in terms of embedding authenticity in
design on that front.
There's still some scope for improvement in terms of more systems thinking competencies
and by systems thinking I mean bringing together various skills it's not just considering the
technical aspect of a question or a task but it's potentially colliding those or the technical
factor alongside economic considerations, business considerations, social considerations
and so on.
So on that front there's still some scope for improvement but by and large authenticity
in design as it has been referred to in literature as well we're doing a very good job.
However as we'd expected authenticity in delivery has a lot to do or a lot to improve in that
sense because for instance when we ask students how often do you engage in peer review and
peer feedback it was a very low percentage of students who said that they often or you
know pretty much always engage in peer review practice and so consequently what this gets
to is the fact that yes as educators we are designing assessment questions that are more
or less resembling challenges in the professional world because students within the same survey
also agree to the fact that they're challenged to learn on the go by these assessments so
it's not that assessments are inauthentic or unauthentic whichever way you call it by
design but it's more that the environment that we're setting up for them to go address
this particular challenge is quite rigid you know it does not give them the freedom to
be creative and it's potentially there where we need to concentrate more of our efforts
to say well how do we kind of remove that rigid structure embrace flexibility and allow
students to be more creative engage in peer review and peer feedback because that's what
people will do graduates would do once they take up any job again irrespective of which
discipline they're going to go into because they're not going to work in silos teamwork
is important so it's how do we you know create that environment which is where kind of all
the findings came together.
Just wanted to build on this in terms of my experience working in industry it's very normal
kind of approach and working routine industry for people to interact with each other on
a daily basis to give each other feedback so that shows in industry people have more
freedom and that freedom gives more passion to kind of show their creativity show their
self in working in different kind of problems you know because the problems in industry
is not like specific problems that they have all similar answers you face with different
kind of challenges and you need to be creative you need to be like a good problem solver
in order to understand how to answer those questions how to kind of find a solution for
those problems and this is what we want to to improve in terms of the education for students
to interact with each other as as managed mention that to work in isolation but to get
out to get feedback from from instructor from peers and use all this interactivity to be
more authentic and to to find solution for like very challenging problems that makes
them completely ready when they get into the into the professional career.
Great and I'll say this because I've because I've read your journal submission which I'd
point listeners towards reading if you'd like to do that please get in touch with Roussein
or Manage and they'll point you towards where to find it and when it's going to be published
the three I suppose teleological aims of assessment that you talk about quite a lot creativity
innovation and passion is it the case then that these aims which link explicitly to the
realities of the workplace should be considered throughout the assessment design and delivery
process not only in your discipline but has relevance to others too.
Absolutely Robert I think you've hit the nail on the head there and it links in very well
with what Mohsen was just talking about so you know broad brush if I were to stereotype
based on the responses that we've got from students again this is very broad by a broad
brush kind of picture making based on the responses that we've got is that as an engineering
or a physical science student or even more broadly we're asking them in terms of course
work here's a question which more or less resembles a professional challenge so they're
going about saying okay we've got a professional challenge to address but I've got two weeks
to produce a report where I cannot share it with my peers because we'd run into academic
integrity issues and this is an individual piece of assessment and I have two weeks as
a solid deadline I need to get as much work as possible done over the course of say the
first week week and a half so I give myself enough time towards the end of the deadline
to perfect that document you know dot the i's cross the t's look at the formatting issues
and then hand it in that is pretty much the majority of student responses if I had to
stereotype it into what kind of assessment experience they're having in terms of that
entire assessment cycle that is the experience that they're having on the whole which is
chalk and cheese when you compare to what Mohsen was just talking about with what happens
in the professional world so if you were working out there and your boss calls you over and
says you have two weeks to address this you're not going to be doing it in exactly the same
way that the student has done it you'd probably take a day or two to curate some data look
at it individually but then you'd go about sharing the draft with your peers potentially
with your seniors try and get some feedback and almost value multiple creative outputs
as opposed to just perfecting one piece of document without even knowing how much value
it has so it's kind of addressing that through the three words that you brought in there
which is about yes being innovative feeling more passionate about it and embedding that
creativity so it's valuing creativity ahead of perfection which unfortunately our rigid
structures rigid assessment structures tend to promote I think it relates to what I was
reading in your manuscript around in the survey many more students referred to a sense of
accomplishment when submitting assessments or assignments than did students who had a
sense of passion or a sense of purpose interestingly you spoke about sense of accomplishment more
as sense of relief could you tell us a bit about why you did that and some insight into
that interpretation sure so I mean when we were designing the survey it was again because
we were thinking of authenticity in delivery so we were thinking about what is the lived
experience of the student when they're preparing the assessment deliverable alongside what
is that overriding emotion let's say once they hand the deliverable in and kind of building
those two questions in which is well how do you feel during the assessments deliverable
period and then how do you feel after submitting the assessment piece kind of lends itself
a little bit from Daniel Kahneman the Nobel Prize winners concept of well experienced
happiness but there's also the self remembering happiness of what do you remember about a
period of time in terms of when you were happy and what we found from the survey results
as you just alluded there Robert is that a high percentage I think close to 70% up in
my head say well they felt a great sense of accomplishment when they handed in the deliverable
and then when we asked them did you feel a sense of passion or purpose when working on
the assessment deliverable that number jumps way down from about 70% around about 40% which
is why we thought well so there is a high chance that a number of these students who
are not feeling that sense of passion or purpose when working on the assessment deliverable
but have this overriding positive emotion when they submit the piece is largely because
they could be engaging with assessment for assessment sake they're here to obtain a
degree and passing an assessment is an important component of it and they're happy that they've
managed to get their submission in on time so it's again trying to see how to be addressed
that divide if 70% are feeling that sense of hopefully authentic accomplishment again
that the same 70% also feel that sense of passion and purpose when working on it.
Yeah and you know this links very well to something that we touched a little bit for
in terms of the guidelines that actually we provide to students so the guidelines we provide
it seems to be mostly restrictive because we ask students to focus on something like
an authentic task.
Having said that I think the guidelines should be clear enough to pave the way for students
in order to provide a work that they can express themselves but at the same time should it
be very descriptive to completely limit them on a specific framework and in this way that
improves the passion for students they will be able to show some creativity and go through
some solutions that might be even beyond the requirement of the task more creative more
challenging when we give them freedom.
Okay and that reminds me of your project title and the importance of flexibility right?
Exactly Robert so making that transition from rigid structures which is what I've been talking
about to embracing more flexibility and there are multiple means that we as educators can
embed flexibility so one as I was just commenting on would be through flexible deadlines and
how we could potentially incentivise student submissions according to these flexible deadlines
working on to the same time scales I was commenting on earlier if it's a two week submission can
we build in an opportunity where a student is able to hand in a draft one week in and
we as instructors can give them feedback that they can effectively feed forward into their
summative submissions so incentivising in terms of flexible deadlines is one option
but that's not the only option the other one which we've also kind of wondered a little
bit about is about assessment format so yes the challenge is set in terms of the assessment
task but it need not always be done through a report it need not always be done through
a presentation it could be done for instance through a pitch deck which might be more commonplace
in industry or start ups so again offering that flexibility to students saying this is
the learning outcome that we want you to meet but how you're going to demonstrate it is
up to you the format is up to you as for the student to choose and likewise with what skills
they want to emphasise in terms of linking in well with the surfacing skills project
here at Leeds as well saying this is the technical skill that we do want you to address because
that is a hardcore learning outcome but what are the other skills that go along with it
offer the flexibility to students to choose that.
Did anything in particular shine through in the open text question of the survey?
Yeah a couple of things that stood out Robert one was the importance that students associate
with timely feedback I think it's something that we speak a lot about as educators but
tend to easily underestimate how much importance students really give to feedback and particularly
formative feedbacks and not this decoupling between formative assessment and summative
assessment but if students can get feedback on a draft version that they're going to hand
in for a summative piece eventually they do tend to really appreciate that.
That was a reoccurring theme on the open text question.
Yeah on that context I was listening to one episode of Light Bites by Gianni Liu that the
title was assessment literacy and student success in the context of language education
so it was a different discipline rather than engineering and interestingly the finding
was the students are asking for more contact and also feedback related to the assessment
literacy and so just wanted to mention is that that fellowship somehow has the same
funding in terms of the feedback similar to what I found from our survey.
And the second point I wanted to touch on yes we need to be more flexible in terms of
can be marking schemes rubrics the assessment briefs but it's also that delicate trade
off because when we were looking at student responses across all year groups starting
from first year of undergraduate study through to postgraduate level we see this delicate
balance between the fact that an assessment brief must have reasonably enough information
in there so that it's clear what the expectations of the assessment are to the students however
it should not be overly prescriptive to the extent that it stifles creativity so it's
about getting that delicate balance right and when we ran reflexive thematic analysis
over the responses that we received for that open text question that theme was very obvious
but that's where again students say that please engage in partnerships with us in terms of
whether it be assessment design or even more high level curriculum design it's kind of
taking students on board looking at what those mutual expectations are and how do we give
them as I said the best possible environment for them to thrive the question the authenticity
in the question is there but how do we create the environment for them to thrive be more
creative and come up with a high quality submission.
So I think it's great to hear you referring to your use of reflexive thematic analysis
there that certainly came through in reading your manuscript especially when you're talking
about what I referred to previously about how you interpret some of the things that
people said whether in the light cut survey or in the open text question for example around
accomplishment and actually relief which I suppose brings us on now to what I'm sure
listeners at home or work interested in knowing how is this going to inform your work going
forwards and your pilot phase in year two of your fellowship.
As I mentioned before the pilot phase start from the October this year so academic year
twenty twenty four twenty five we have a couple of modules in both undergraduate and postgraduate
which means that taught MSc programs we have selected for these pilot phase we are trying
to introduce some ideas from these flexible assessment and also the perception of the
assessment we got from student to provide more feedback more peer review opportunities
and also flexible assessment hand in dates with the incentive of providing feedback.
These are like initial thoughts that we have in terms of getting from the survey to the
modules and to apply in the pilot phase.
Sure to just back up on what Mohsen said so we've identified these modules where we're
going to pilot flexible assessments and we'd be collecting some more data in terms of journey
experiences to go back and compare with the larger student survey that we've of course
just run so that would be the primary source of data and trying to yeah hopefully spread
the word and get more educators across the university take on such practice and in that
sense we've been having conversations with assessment strategy leads both at within the
faculty of EPS but also more broadly at the university level to see how our findings can
feed into upcoming policy guidance.
Excellent and just a note to listeners Manoj and Mohsen will be coming back in a year's
time to share these findings from implementing the pilot and personally we're looking forward
to hearing about how that turns out for you both.
You mentioned Mohsen your plans for disseminating in addition to this podcast and achieving
impact.
Could you tell us a bit more about your plans going forwards?
Yeah for dissemination as you previously mentioned we already wrote a manuscript submitted which
is under review currently and we hope to get a result for that soon to share with listeners.
Also we have plan in order to publish kind of not publish but participate in couple of
conferences in Europe and also abroad in probably in America and Asia to be engaged in different
conversation not only in scale of the Europe but in other scales to understand how these
concept of flexible assessment have been explored and have conversation with other colleagues
in different educational settings to get their feedback to see how we can improve this and
all these to a level as Manoj mentioned to have more educators more lecturers and instructors
on board with their idea.
Great and are there any particular strategic frameworks at the university that you'd be
looking to inform?
Absolutely Robert the one we've identified is the access and student success strategy
links in very well with some of the points that we've just mentioned here on the podcast
as well to particularly the attainment and the progression pillars because we've collected
data as I stated across all year groups we really can look into that granular level detail
of what we need to be doing across different year groups to help them progress and attain.
So those are the things that we are looking at at the moment.
Okay and just taking a step back then how's the overall experience been for you both so
far in your light fellowship?
Brilliant Robert if I had to put it in one word I think just the experience of you know
systematically investigating a pedagogic research question was very interesting certainly learnt
a lot because yeah I come from an engineering background so I do not quite have that hat
as a pedagogic researcher or as a scholarship person but still kind of going through the
data going through the entire experience has been a rich learning curve.
I've had networking opportunities a lot of them through light so I have to thank light
for that through talking to engineers also some staff interviews which we did not mention
over the course this podcast but it's data that we'll be looking at going forward.
So yeah just the fact that light has given me the time and space to think about the project
and potentially influence the practice of others it's just absolutely brilliant.
Yeah same here it was an excellent experience for me too and a big learning curve in terms
of learning how to work with qualitative data because we are engineers literally so working
with digits and quantitative data is something normal for us but qualitative data is a big
learning curve for us.
How to go to different thematic analysis methods the reflexive thematic analysis how to analyze
the data that is not in digits it was very interesting learning curve.
So the challenge of time was a bit evident for myself because I have other commitments
to do in conjunction of this fellowship but overall it was brilliant experience and the
support I had from the light also other as managed mentioned opportunities for networking
and speak with people from other disciplines was amazing.
I'm really glad that for both of you stepping into the world of social science research
has been so far an enriching experience it's not unusual for it to be a difficult process
in terms of adapting to in many ways very different way of seeing data and interpreting
data and generating findings from that.
It's great to hear that learning how to apply reflexive thematic analysis in particular has
been enriching to you I know that my colleague Lucy has been in touch with you supporting
you with that and is currently developing a resource on reflexivity for light fellows
to use and anyone doing pedagogic research.
Great what advice then would you have for other people that are starting out in their
scholarship of teaching and learning or pedagogic research projects?
Probably some suggestion rather than advice you know like I said it makes sense for people
to prepare for that experience in terms of the learning and reading some resources before
going into the work itself and also be aware of some challenges because you know when we
are dealing with this social science approach we mostly go through the route of survey and
questionnaire and this sort of stuff and in higher education I think we have the survey
fatigue somehow for students we do a lot of survey and evaluation so I would say for someone
that tries to enter actually this realm be aware of that survey fatigue not to push students
with a lot of like different surveys because that reduces the number of respondents diminishes
the value of the survey.
Yeah it can definitely be a challenge another shameless plug from another colleague of mine
is he's been developing a resource hosted on our website which has a range of qualitative
data collection methods so for people that want to consider something else and have a
way of learning about an issue that's definitely worth looking into and considering.
Thank you I pretty much second what Mohsen has just said there in terms of more suggestions
than advice really and since he's touched on more specific learnings in terms of the
methodology and the findings for our project I'd probably go more high level and say again
more suggestions than advice would be to have bias towards action especially if you are
not from a very social sciences discipline so in our case for instance as engineers stepping
into a pedagogic research project you can have that imposter syndrome which comes up
more than once throughout the entire project so it's about having that bias towards action
and continuing to look at data feeling confident and you know one tiny step forward and will
eventually given us the results that we have in a year's time and we look forward to seeing
what the next year holds.
Thanks so much that is incredibly valuable feedback and suggestions for people that are
considering undertaking a light fellowship or pedagogic research and thanks to both of
you for being such great light fellows so far and I know I'll speak for everybody at
light in saying that we're really excited to see what's in store in a second year of
your fellowship and again I'll reiterate to everyone listening that Manoj and Mohsen
are very happy to be contacted if you have any questions or would like access to the
journal article or would like to apply the findings from the fellowship into their own
work so thanks again to you both.
Thank you very much Robert absolute pleasure doing this yes if you want to get in touch
with us by all means too because as we said we all here are passionate about student education
so if whatever work we've done can feed into your practice we'd be more than happy to see
that happen thank you very much.
Yes same for me I would be very glad to connect with people if they have any question would
be really interested to respond and to join forces in the mission that we have in order
to educate a generation that could be useful for our world and also to have great contributions
yeah glad to be here thank you so much.
Brilliant thank you and thanks to everybody listening to this episode of Light Bites just
a reminder that we'll be revisiting the final findings of this fellowship this time next
year thanks for listening and see you next time.