Welcome to LITE Bites, an occasional podcast from Leeds Institute for Teaching Excellence
at the University of Leeds.
Episodes will be hosted by members of the LITE team.
And we'll be showcasing the scholarship of teaching and learning from across the university.
Hello and welcome to another episode of the LITE Bites podcast.
It's Emma Peasland here, research and impact officer in Leeds Institute for Teaching Excellence
at the University of Leeds.
And today I'm chuffed to be joined by Samantha Stark, who is a course design manager in the
digital education service here at the university and has recently completed a Light fellowship
evaluating the impact of a pedagogical framework for a fully online education.
Hi Samantha.
Hi, thanks for having me.
Well, it's great to have you here.
And I just mentioned that you work in the digital education service here.
So I wondered if a good place to start would be for you to tell us a little bit about what
you do in the digital education service.
Yes.
So the digital education service, it's a bit of a mouthful.
So we tend to call it DES for short, just to help you out.
And there are many different departments in DES that do completely different things.
We're a very big department now.
But the team in which I used to work up until the end of my Light fellowship works with
academics from across the university to design master's degrees that students will study
fully online.
And so we have completed master's degrees in a wide variety of disciplines, engineering,
artificial intelligence, sustainable business leadership, digital design communication,
international fashion marketing and design management, data science.
And the portfolio is increasing all the time.
So that's what that team in the in DES does.
And that's the team I was working for when I conducted this research.
And what do you do now?
So now I'm still at DES, but I've just moved to a different team.
The focus of that team is to, we kind of make a wide variety of different courses to support
LEAD students.
And also we create courses for short courses for future learn.
Some examples of courses we've done, I worked on an academic listening course to help students
to prepare for studying at the LEADS University.
And we've also done a course for like supporting healthy relationships for students.
So basically we're focused on enhancing student education at the University of Leeds, but
also helping to attract people to the university as well.
Very interesting.
It's really diverse kind of range of activity going on in DES.
Absolutely.
Excellent.
So let's talk a little bit about your Light fellowship then, since we're on the Light
Bytes podcast.
So do you want to start by telling us a little bit about what it was that you were investigating
and why you wanted to do that?
Sure.
So when designing master's degrees, it can be quite difficult because academics might
be used to teaching their module face to face.
They have to change it to online delivery.
And that means changing a lot of things about how the course is designed and the type of
content that we can include.
So for the master's modules, they involve six weeks of teaching and each week involves
about 20 hours of learning.
And then after the six weeks, learners have a week to prepare.
And then in week eight, they submit their final assessment and then they start their
next module.
So the way we used to design master's modules is that it was almost like a blank paper approach.
So each week of learning, which is a unit of 20 hours, was split into like four lessons.
But that was kind of the only structure that was there.
And each week ended with a webinar.
So there just ended up being 20 hours that we had to fill.
And it meant that we ended up with too much work.
The design and development process was taking too long.
We ended up with too much content, which meant the academics were overloaded.
We and the team were overloaded and the students were overloaded.
It meant there was a predominance of passive learning and not a lot of active learning.
So it wasn't a good learning experience and it wasn't a good design and development experience
either.
So we wanted to improve that.
And when a former colleague, Sarah Briggs, joined our team, she had sort of used the
design framework in her previous role before she came to the university.
So she suggested that as a way forward.
And I got heavily involved in the design and implementation of that framework.
So the new framework was very different from what we've done before.
So instead of having 20 hours of learning split into four lessons, we had this pedagogical
framework, which was basically meant that we could have more active learning interventions.
So it split into three phases, prepare, apply and reflect.
And the prepare section was about two hours.
And that was where a lot of teaching input happened.
So there might have been short videos, articles to read, maybe other sorts of infographics
to interact with.
And that prepared them for a key sort of a substantive activity, which would take learners
between three and five hours to complete.
And we would create like a brief for that.
So the learners had to take what they'd learned in the prepare phase and use it to actually
achieve something that would be relevant to something that they would do in real life
and would have a clear outcome.
So for the design degrees, for example, it would be like, I don't know, create some
kind of a wire frame.
Or they had to do a SWOT analysis of something for one of the international fashion marketing
models.
So it had to be something that had a clear outcome.
And if it could be related to real life, then that would be good because then the learners
have almost a portfolio of things that they've done that they'll be able to show to future
employers potentially.
And so once they've completed that, they share the outcome with their peers for feedback,
for them to discuss and learn from each other.
And then the reflect stage was about four hours, a one hour webinar, a one hour preparation
for that webinar, and then two hours for reflection at the end.
In addition to those three phases, there was also two hours allowed for a substantive discussion
on a different point.
So that meant there were kind of two points of discussion in each week.
There was a separate substantive discussion where the module leader set questions that
we would design as part of the design and development process.
But there was also this kind of student led discussion that happened once they'd completed
their substantive activity where they're sharing their outcome with each other and getting
feedback.
So I'm curious to know how much of this is students in these programs attending synchronous
live sessions and how much of it is working on their own self-directed learning?
Yeah.
So it's kind of a mix.
So the synchronous is the live webinar at the end of each unit, end of each week.
The rest of it is asynchronous.
That's why they're not sharing their outcomes in an online seminar or something.
They're sharing their outcomes in a discussion board.
They can all share and give feedback asynchronously because they're all in different time zones.
All our students are from all over the world.
So organising synchronous is quite difficult.
And another thing to note about our students is that they are older than the on-campus
master's students might be.
They're usually around their mid-30s.
They're working.
So they have to kind of fit their studies around what they do.
OK.
And so you had this new kind of framework that you introduced.
And so how did your fellowship work to kind of evaluate that?
Yes.
So we often introduce things that we think, oh, this is amazing, but we don't necessarily
get the feedback on it.
So I was really keen to think, well, to find out how it was actually landing with our students
and with the academics that we worked with, but also within DES itself.
Were my colleagues finding it easy to work with?
And was there any way that it could be improved to make it better and make it work better
for everybody?
So you mentioned that you were wanting to evaluate how the framework or this new process
landed with students and also with DES staff.
And was there anything kind of collecting data from the academics who were involved
in that as well?
So what we wanted to find out specifically was, you know, was this framework helping
to reduce the amount of content?
Therefore was it speeding up the design and development process?
And you know, was it meaning that the students had a better learning experience?
And I wanted to find out a diverse range of opinions.
So I interviewed academics on a one-to-one basis.
I had semi-structured interviews both with module leaders and with program leaders, because
the program leaders can see, you know, all the modules and the program leaders have seen
the modules that were not designed with the framework and those that were.
So they could provide a useful point of comparison.
Some of the module leaders have also worked on non-framework and framework modules, but
some of them hadn't.
So I was speaking to all of them, as many as I could, during my fellowship.
So those were the one-to-one semi-structured interviews.
Then for DES staff, I ran two focus groups.
Well, I designed the questions for two focus groups.
Haley Bullard from Light actually ran them for me, which was very helpful.
And I also designed an anonymous online questionnaire for the students.
It had to be online because, as I mentioned before, our students are all over the world,
different time zones, et cetera.
It would have been quite difficult to arrange to meet up with them.
And I wanted it to be anonymous so that they felt more able to express their opinions honestly.
So yeah, that was an online questionnaire I did for them.
And what did you want to find out from the students specifically?
Well, again, I couldn't ask the students to say, hey, was this module better than the
other one, because there was no way of knowing if that student had done that module.
And I couldn't track the students and ask the same students the same questions.
So all I could do was collect, ask all the students the same question and ask which module
it was from.
So I could see if their response is related to a non-framework or a framework, and then
just see if there were any trends.
Is the feedback generally more positive or more negative about non-framework or framework?
OK, that's really interesting.
So now might be a good time to ask what you found out.
Yes.
So got some really, really useful feedback.
So in relation to academic staff, there were many positive comments.
They found that the structured approach was really helpful, that the framework was simple
and logical, that it did help to save time, and that the substantive activities that we
designed were extremely valuable.
So that was the positive feedback from them.
But there were some areas of concern.
One bit of feedback actually from a module leader that led a non-framework module said
that too much structure can kill content.
So I think there was a real concern that this structure would kind of be imposed on different
disciplines when it may not be suitable.
And so flexibility is really important.
Another concern was, yes, although there's less content perhaps for module leaders to
write, designing a decent substantive activity that the learners are going to be able to
do independently for three to five hours takes a lot of time to make sure that they've got
what they need and that it's well designed and that the module leader is not going to
end up having to field lots of questions from the students while the module is running.
Another concern was, as I mentioned before, our learners, they have jobs, they might have
children that they're fitting this in around their jobs, et cetera.
And so it could be difficult for them to find the chunk of time that they need to do the
substantive activity because they might be just dipping in and out, doing half an hour
here, half an hour there.
And so for something that's going to take three to five hours, it might be quite difficult
for them to find that time.
Regarding workload, the framework itself doesn't necessarily reduce it because the workload
is a complex thing.
It can depend on the individual.
It can depend on the experience that they've had.
Have they done a module before?
If they have, it's going to be a lot quicker.
They're going to find it less work, whereas somebody else is going to find it more.
So the framework by itself is not going to solve that issue.
Yeah.
And I guess for some people, perhaps that designing that substantive activity might
take a bit more time than for others.
And then you mentioned about discipline as well.
So did you find that using the framework was particularly effective for some disciplines
and not for others?
Yeah.
It's difficult for me to say that because I had to base my research on the modules that
were running at the time of my Light Fellowship.
And so I was able to collect data on a couple of modules from Sustainable Business Leadership,
one of which used the framework.
And that was very much about data analytics.
And there was maybe one or two that was from digital design communication, so much more
design focused.
And then International Fashion Marketing and Design Management, which was a kind of maybe
a bit of design, but maybe a bit more business as well.
So I found it did seem to lend itself in some ways quite well to design, but in other ways,
not because when learners are studying face to face, something I found when I was designing
myself on that module is that the module leader said, normally they're working on their final
assessment in class.
They're getting feedback from each other on the final assessments.
Their coursework is their assessment.
But because of academic integrity rules and the nature of online learning, we can't make
the substantive activities the final assessment.
And so the substantive activities, they're going to mirror what they need to do in the
final assessment, but it's not actually going to be what the final assessment is, which
then made it quite difficult for the module leader because of the way they had to design
it for online learning.
But that would have been the case, whether it was the pedagogical framework or not.
Yeah, that's very interesting.
So you talked a little bit about what you found out from academic staff.
So there was one more thing about the academic staff.
That's all right, so the negative, one more sort of area of concern was the substantive
activities are such an important part of each unit.
So we need to make sure that the learners actually do it because how much value are
they going to get if they're not actually doing it?
So we need to make sure that when we're designing, we're designing in motivations for the learner
to do it.
But I'll come to the suggestions for improvements a bit later.
So in relation to DES staff, the positive responses were that the structure provides
really useful scaffolding and a platform upon which we can discuss and explore pedagogy.
So the framework is informed by different pedagogical theories, universal design for
learning and also Laurel Arts conversational framework and also constructive alignment
from BICS.
Those are some of the key sort of theories that have informed it.
And so this platform, this framework, we know that it is based on that pedagogical theory.
And so by having it, we automatically have a platform to engage with academics on that.
So they found that really useful.
They said it does help to control the amount of content in each unit, which was one of
the main goals of the framework was to do that.
It does help to make learning more active, again, one of the key roles of the framework
and that model design is quicker.
So based on that positive feedback, it looks like it's certainly achieving some of the
goals.
But again, they had areas of concern, also about structure and rigidity, that the framework
could be too restrictive and could limit creativity.
Again, they said what I mentioned before, that it can conflict with discipline pedagogy
and therefore cause a problem.
Another concern is that two hour prepare phase that learners complete prior to do the substantive
activity, there has been a sort of predominance of passive learning there.
So it's almost like we're just throwing a lot of knowledge at them and then they have
to apply it in the substantive activity rather than there being a bit more active learning
in that prepare phase.
Another point that it's not always appropriate to implement the framework in the first week
of the module because learners might not feel particularly confident and they suddenly have
to go off on their own and do something that's quite substantive, might be quite new to them.
So maybe flexibility is needed about perhaps in the first week you're not doing it or you're
doing something a lot smaller than you would in the subsequent weeks.
And finally, similar to what the academics said, that the speed of the design and development
process is impacted by factors outside of whether we use the framework or not, such
as academic availability, etc.
So it's never going to solve all the problems, but it looks like from their point of view,
it's certainly contributing to make things a bit better.
That's cool.
And I'm interested in that.
I think, or I think it's interesting that there's this perhaps need to flex the framework
a little bit early on in a module so that students can get used to that kind of new
way of learning.
Did that come out in the student responses?
So in relation to the student responses, unfortunately, I couldn't identify any solid trends because
there was quite a low response rate, which I suppose is to be expected to a certain degree.
So we collected data, as I said before, about non-framework and framework, but the student
experience seems to be impacted far more by their prior knowledge of the subject.
So if they're coming in with some prior knowledge of analytics or design processes or business,
then that's going to impact how well they're able to keep up with the module and how easy
or difficult they find it and how long it takes them to complete.
So it very much depends on the individual student.
But as an indication of the kind of difficulty I had with identifying any trends, two students
on framework modules said that there was insufficient taught content, again, because there's only
that two hour prepare phase.
But then another three said that there was sufficient.
So there's just complete diversity of opinions.
So you can't really identify a particular trend.
But I think what was most insightful was that two framework students said that there was
too much going on with a substantive activity, with the separate discussion.
I was the standalone discussion and responding to and giving feedback to their peers.
So it's difficult to keep up.
And this was really interesting for me because I think because we were going from designing
20 hours of content to designing just those two in the prepare phase, I think we were
trying to kind of make up for that and still demonstrate value to the students by having
things for them to read in preparation for the webinar, sometimes putting in things for
them to read in preparation for a discussion on top of preparation, substantive activity
and giving feedback.
So I think by trying to make sure that they felt that they were getting value for money,
we overloaded them even with the framework.
So it was really useful to get that feedback.
That is really interesting, isn't it, that going from 20 hours of taught content to,
I mean, what does it add up to?
It was something like two hours for prepare, three to five hours for the substantive activity
and four hours for reflect.
Yes.
So something I've neglected to mention at the beginning, which is quite important, is
that there is still the 20 hours of learning, but it's just split up differently.
So we end up the framework takes up 10 to 12 hours and self-directed learning, as in
their choosing completely what they want to do is the eight to 10 hours.
And in that time, they might be using it to spend a bit more time on the substantive activity.
They might be using it to catch up on things they've missed, prepare for their assessments,
et cetera.
So there was also that factor of, oh, we're no longer designing 20 hours.
We're only designing 10 to 12.
So again, we want to give them lots to do.
But it was taking them, some of them, it was taking them over 20 hours, I think, just to
work through that kind of framework, those framework activities.
Yeah, I guess having lots and lots of touch points, I guess, throughout each kind of week.
Yeah, it ends up being a lot.
And I suppose different people will engage with it with different levels of time or amounts
of time, dependent, perhaps, on that knowledge and prior experience that you mentioned before.
Yeah, and their personal circumstances.
So it all varies.
So I think, yeah, it was really, really, really useful to get that feedback from them.
Another bit of data I collected in relation to students, which I, again, forgot to mention
before, I apologise, is in relation to Minerva data.
So we also got some Minerva data about engagement.
And Minerva is our VLE here at Leeds.
Yes, based on Blackboard, yes.
So that's where we build these master's degree modules.
So that's what learners access in order to get to access their learning materials.
So I got some data about engagement in those standalone discussion forums and also webinar
attendance.
And I noted that participation in those standalone discussions was limited.
And perhaps that indicates the fact that there was just too much, what those students were
saying, there was just too much to do.
And it was the discussion that fell by the wayside.
So that may be a question, well, why are we doing it?
If it's not being engaged with, it takes a lot of time to design discussion questions.
It's time for the module leader to engage in them themselves while the module is running.
And so it was really insightful to see that data that the learners perhaps just don't
have the time for it.
And so the final interesting thing about the discussions, which one of the academics touched
on is that engagement in the post-substantive activity discussion, so the student-led one
where they're comparing what they've done, those discussions were much more active.
So those are the ones that the learners were engaging with rather than the standalone discussion.
OK, so that standalone discussion happened on a board, and then the other discussion
was in a synchronous session where people were participating.
It was asynchronous as well, but it just had a different purpose of them sharing their
work with each other.
And the module leader, I mean, it's up to the module leader how much they engage with
that.
But it wasn't something where they had, you know, that we set discussion questions.
We simply said, share your work with your peers and get feedback.
And apparently in those, the academics were saying, yeah, there's a lot more engagement.
Perhaps it seemed more useful for the learners to engage in that one rather than the standalone
one, which wasn't related to the substantive activity.
OK, so that's all really interesting.
And so I'm curious to know, given you've got this kind of diverse and interesting set of
data, what did you do with that in terms of making recommendations for future practice
or suggested changes or anything like that?
Yeah, so I think my overall conclusions were that the framework was helping to make things
quicker and it was helping to control content.
And it did result in a more active learner centred experience.
But that it can mean it can mean that things are too rigid and too uniform.
So my suggested improvements were that the framework is used as just that, a framework,
and is applied flexibly depending on what is needed for that particular discipline and
to, you know, not to be prepared to cut things out.
For example, the two hour reflection at the end, some disciplines are not reflective disciplines.
It's you're right or you're wrong and that's it.
And so insisting on including that reflection for a discipline that doesn't really do that
wasn't helpful.
So it kind of sort of gave us a justification, I suppose, for being more flexible in that
regard.
There was the suggestion to yet to make the prepare phase a bit more active.
As I said, I've left the team, so I don't know if people are doing that.
But that was one of my kind of recommendations that were flexible as to when the pedagogical
framework starts so that we don't have to have these independent, these huge independent
tasks in week one.
And that task size sort of, when I say task, I mean the substantive activity and that the
size of that substantive activity can vary along the way through the module so that when
we know that learners are going to be doing a formative assessment, for example, they
don't have to spend as much time on substantive activity that week.
And so that was one main recommendation, more flexibility.
The second main recommendation was that we make sure that we're encouraging engagement
in the tasks, in the substantive activities.
For example, that any feedback is written in such a way that it only makes sense if
the learners have done the activity, that the activity is linked to the webinar in some
way, that it's always linked to some sort of summative assessment in some way, even
if we just make clear the skills that you're going to develop, the knowledge that you're
going to develop in this substantive activity is going to help you with your final assessment
because obviously learners are very assessment focused.
To give those clear reasons for completing that activity and breaking it down into component
parts when that suits.
So it may not always be possible to break down an activity into smaller sections, but
where it is possible to help those learners that have only got little pockets of time
here and there, it means that it might make it easier for them to do that.
So since completing this research, I did design one module before I left that team and I put
this into practice and it did make the design process much, much easier because I wasn't
coming up against the academics in the same way to say, this is the framework and this
is what we need to be doing.
It was, oh, well, reflection is not going to work for you.
Okay, fine.
So what would we do there instead that would work for your discipline?
Is there something that the learners could usefully discuss this week that's separate
from their sort of post activity discussion?
No, okay, let's just not worry about that.
So it meant that we could just not, you know, exclude things that were not useful and save
a lot of time on those discussions and debates about those particular sections.
I mean, one more point I should note is that when we were piloting the framework, we were
kind of deliberately applying it rigidly because we wanted to test it out.
It's only once you've properly done something, you know, by the book, as it were, that you
can then work out where you can flex or where you can't.
So that kind of feedback about rigidity and now applying it more flexibly doesn't come
as a surprise, but it's good to have it sort of justified in the research.
Yeah, that's really interesting.
And I guess it'd be really interesting to extend the research that you've done, perhaps
someone else, given that you've left the team, but to see how that is being received in practice,
especially since your anecdotal experience of that was that it was positive for you.
And did you get any anecdotal feedback from the academics?
In relation to that module I designed?
Yeah, being a bit more flexible.
Well, it's difficult because they wouldn't have designed one before.
This was their first experience.
It's not like they could compare it to anything else.
This was their first experience of working with the framework and working with us.
OK, but I guess from your perspective, it felt like it was easier for everyone.
Easier and much quicker, definitely much quicker and a much kind of smoother process.
Brilliant.
And so I'm curious to know if there's anything that you would do differently if you're going
to start your light fellowship over.
Absolutely, yes.
So I think I would have written better research questions.
I think I used words that were kind of vague and I knew it at the time when I wrote them,
but I just couldn't think of a better way to word them.
And I think what I should have done is word the research questions so that they were grounded
in the problems we were trying to solve in a much more succinct and concise manner.
So there's too much content, is the framework helping you to control content?
You know, much, much simpler, shorter questions, because the research questions I had ended
up being quite difficult to work with throughout the project.
Yeah, when you get to that kind of analysis phase.
Yeah, I always remember one chapter of my PhD having similarly, perhaps to you, research
questions that I could have written better and finding that analysis and reporting phase
really laborious.
Absolutely, it does make it a lot more difficult because you're trying to answer research questions
that you were never really going to be able to answer, but you have got things that you
can answer that are useful and they are related to the research questions, but the research
questions are just not very well written.
I think that's the main thing I would have done differently.
I think other things that were challenging were things that were outside of my control
and outside of lights control.
It is challenging to do a light fellowship in one year when you've only got one day a
week because you end up spending the first part of that day working out where you've
got to and then that only leaves you sort of half a day or three quarters of a day.
So that was challenging, but I just had to get really good at project management to get
it all done in that time.
So I think it was a benefit in the end.
And then actually I think as well perhaps from the interactions that we had throughout
your fellowship, you were very good at having one day a week that was your light fellowship
day.
Yes, I think it was really important because initially in the first month I didn't have
it blocked out of my diary and then my line manager was like, you have to block out your
diary.
So then for the whole year, one day was blocked and so no one could book any meetings.
Sometimes obviously urgent things come up and I had to deal with them, but equally sometimes
there was light events earlier on in the week, so then I would be coming onto campus and
attending those, which was also part of my light allocation.
So a bit of flexing there as well.
And then that perhaps moves quite nicely into thinking about your experience of your light
fellowship.
And so I'm wondering if you experienced any kind of benefits for yourself of having taken
part in this fellowship or completed this fellowship?
Very much so.
So as I mentioned before, last October I started a new position and I think having done the
light fellowship, it helped me to demonstrate that I had experience of evaluation of our
courses and evaluation is really important in our current team.
It also helped me to develop my research experience ahead of my PhD, which I started in October,
and it may well have contributed to me getting ESRC funding for my PhD because I was able
to include my research experience in the application.
And aside from that, another personal benefit was that it provided a lot of networking opportunities
with people outside of my department and across the university, which I have found really
useful because having contacts across the university helps you to understand how other
parts of the university work.
And in my current team, that's really important because we're working with different parts
of the university all the time.
Excellent.
Well, congratulations on your funding and your PhD.
Thank you very much.
That's very interesting.
Perhaps we can have another conversation one day about that.
Of course.
But yeah, it's been wonderful to hear a bit more about your project today.
And we spoke occasionally during your fellowship, but we didn't work particularly closely.
So it's really interesting to hear a little bit more detail about what you found out and
how that might be useful and everything like that.
So thank you for coming along.
Yes, thank you very much for having me.
It's been a pleasure.
Yeah.
And you know, all the best with your continued studies and research and thanks for sharing.
Thank you very much.
Well, thank you again.