The Use and Usefulness of Peer Feedback

Dr Charlotte Newey and Dr Steph Rennick

c.newey@reading.ac.uk

Humanities (Philosophy)

Overview

We undertook a pilot study into the use and usefulness of peer feedback, involving undergraduates and postgraduates from Philosophy at Cardiff University – where peer feedback was not widely used – and for comparison, undergraduates from Law (where peer feedback was well-established).

The study identified three main concerns students have with peer feedback: the expertise of their peers, their motivation and investment, and their ability to interpret and apply grading criteria. Here we outline some simple recommendations to help to mitigate these concerns while allowing educators and students to repeat the many benefits of incorporating peer feedback.

Objectives

Our study’s overarching aims were three-fold (https://www.cardiff.ac.uk/learning-hub/view/improving-peer-review-a-pilot-study):

  1. To trial different opportunities for, and kinds of, peer feedback.
  2. To gather qualitative and quantitative data on the perceived usefulness of peer feedback, before, during, and after interventions.
  3. To improve students’ ability to identify and utilise different kinds of feedback (including, but not limited to, peer feedback).

Ultimately, we wished to improve the perception of peer feedback among students by helping them to understand its usefulness, identify the conditions under which it is most valuable, and gain insight into the barriers that can hinder its success. We hypothesised that a better understanding of student perceptions regarding feedback would help us to improve our teaching and feedback practices.

Context

The research was undertaken by Dr Charlotte Newey and Dr Steph Rennick when we were Philosophy lecturers at Cardiff University. We noted an apparent mismatch between workload constraints and the quality and volume of feedback the university aimed to provide. We trialled interventions in two undergraduate and one postgraduate Philosophy module and held focus groups with Philosophy students and students from Law (where peer feedback was already an established practice). Given the time pressures on academics throughout the UK, this research remains highly relevant and is applicable across disciplines.

The research also bears on student experience. Students may give lower scores on module evaluations and NSS in relationship to the timeliness or usefulness of feedback if they do not recognise the different forms that feedback can take and have in mind only written comments made by academics on summative work. There is therefore additional benefit to improving students’ understanding and recognition of peer feedback.

Implementation

At the start of the study, we held focus groups to canvas opinions among undergraduates regarding peer feedback.  We then trialled six interventions across three modules over the course of a semester (two undergraduate and one postgraduate). These included critically commenting on their peers’ individual and group work over different tasks, providing feedback both verbally and in writing. Throughout this period, we measured the difference in perceived usefulness between the different interventions, including instances identified explicitly as constituting peer feedback versus those described merely in terms of the activity (e.g., ‘a group exercise’). We used in-class surveys and Mentimeter polls. Finally, we held a third focus group at the end of the semester to capture whether attitudes had changed as a result of the interventions. In the first half of semester, we described activities in terms of their specific learning outcomes without identifying them explicitly as peer feedback (e.g., this is a group activity in which you’ll practise reading and interpreting an ancient text); in the second half, we indicated how and why the interventions were forms of peer feedback. We wanted to discover whether the phrase ‘peer feedback’ put students off, and whether they were correctly identifying the various opportunities when they were receiving feedback, rather than assuming feedback was limited to comments from a staff member on written work.

We found that reception of peer feedback varied depending on a number of factors. Most strikingly, students seemed to appreciate peer feedback most when it didn’t apply to a particular assessment, but rather in the context of checking their understanding and/or skills development. While they were reluctant to have their peers ‘mark’ their work, they reported significant benefits from defending their ideas, critiquing the structure of others’ arguments, and comparing their understanding. Perhaps because ‘assessment’ and ‘feedback’ are so often discussed together, students didn’t always recognise this non-assessment-specific feedback as ‘feedback’.

Students highlighted three main concerns with peer feedback:

  1. What do their peers know? (The expertise of those giving feedback)
  2. Do their peers care? (The attitude, investment, and motivation of those giving feedback)
  3. Do their peers understand the grading criteria, and would they apply it accurately and reliably?

The focus groups revealed differences in attitudes between disciplines that affected reception of peer feedback: Philosophy students tended to view their peers more as collaborators while Law students viewed them as competitors.

Impact

Encouragingly, each of the central concerns raised by students can be overcome. Based on the study and our subsequent practice, we make the following recommendations for improving the effectiveness and reception of peer feedback:

  • Ensure that exercises involving peer feedback are overseen by staff. This helps to avoid the problem of ‘student expertise’.
  • Provide opportunities for students to practise giving feedback, increasing their confidence in themselves and each other.
  • Incentivise giving helpful feedback. For example, by making peer feedback a component of summative assessment. This helps to overcome the problem of student investment.
  • Foster an environment where students see each other as collaborators, rather than competitors, which might be done differently in different departments.
  • Develop students’ literacy in interpreting grading criteria by having them apply them, rather than merely distributing copies of the criteria.
  • Do not limit the discussions of feedback to discussions of assessments: highlight the diversity of opportunities for, and benefits of, feedback. An example of this from within Humanities, which is likely to have wider application, occurs in group discussions. A seminar leader might delay their own response to a students’ question or opinion, and instead invite others to provide answers or suggestions.

Reflection

This study provided helpful insights into the attitudes of students towards peer feedback and how its benefits could be maximised and best communicated. Part of the project’s success came from the sustained nature of the interventions, the ongoing evaluations, and comparisons between modules and with another department. However, this was a small pilot study so there were limitations on what could be achieved. The postgraduate class we trialled interventions in was small, and so the bulk of our analysis focussed on our more robust undergraduate data – exploring postgraduates’ experience of peer feedback would be a fruitful avenue for future research. We had chosen Law as our comparison discipline as peer feedback was more established there, but unfortunately the attendees of the Law focus group were less familiar with this practice than we had hoped. In future research, additional cross-discipline comparisons would be useful.

Follow Up

We have both continued to incorporate peer feedback into our practice and encourage our colleagues to do the same.

Links

https://www.cardiff.ac.uk/__data/assets/pdf_file/0006/922929/Rennick-Newey-Peer-Feedback-Final-Report.pdf

Using student feedback to make university-directed learning on placement more engaging

Anjali Mehta Chandar: a.m.chandar@reading.ac.uk

Charlie Waller Institute, School of Psychology and Clinical Language Sciences

 

Overview

Our vocational postgraduate courses in Cognitive Behavioural Therapy include University Directed Learning (UDL) days that are completed within the placement setting (e.g. their NHS trust). A qualitative student feedback survey allowed us to collaboratively adapt this format, with favourable outcomes in how interesting, enjoyable and useful the students found the day.

Objectives

Our objectives were as follows:

-To ascertain how interesting, enjoyable and useful the UDL days were, as perceived by the students, based on pedagogical findings that students engage best and are most satisfied, if these characteristics are met (e.g. Ramsden, 2003).

-To make improvements to the UDL days based on qualitative student feedback.

-To ascertain whether our improvements had made the UDL days more interesting, enjoyable and useful, as perceived by the next cohort of students.

Context

The Educational Mental Health Practitioner (EMHP) and Children’s Wellbeing Practitioner (CWP) programmes are one-year vocational postgraduate courses. The students are employed by an NHS trust, local authority or charity, and study at UoR to become qualified mental health practitioners.

UDL days make up a small proportion of the teaching days. They are self-guided teaching days, usually containing elements of e-learning, designed to complement and consolidate face to face teaching (live or remote). A combination of learning methods, including e-learning, is shown to be effective in increasing clinical skills (e.g. Sheikhaboumasoudi et al., 2018).

UDL days had been poorly received by our two 2019/2020 cohorts, according to feedback in the student rep meetings and Mentimeter feedback after each UDL e.g.  comments included: ‘there was too much [content] for one day’, ‘I felt pressured to fill [the form] out rather than focussing on the readings themselves’ and ‘[the reflective form] was too long and too detailed’. Whilst this gave us some ideas on changes to make, I was aware of the low completion rates of the Mentimeter feedback. Therefore, to hear from more voices, we decided to create a specific feedback survey about the UDLs to help us make amendments in a collaborative way.

Implementation

We started by creating a survey for the current students to ascertain their views on how interesting, enjoyable and useful the UDL days were. We also had qualitative questions regarding what they liked and disliked and ideas for specific improvements.

I then led a meeting with my course team to explore the key findings. We agreed to make several changes based on the specific feedback, such as:

– variety of activities (not purely e-learning, but roleplays, videos, self-practice self-reflection tasks, group seminars run by lecturers, etc, to provide a more engaging day)
– fewer activities (we aimed for one main activity for every 1-1.5 hours to manage workload demands)
– an option to complete the programme’s reflective form (designed to be more simple, by asking them to provide their own notes on each task) or provide their notes in a format of their choice (e.g. mindmaps, vlogs, etc) to increase accessibility.
– share these reflections on a discussion board for other students and the lecturer to comment on.

We were unable to implement these changes to the current cohort as they had finished all their UDL days in the timetable, so made the changes for the following cohorts in 2020/2021.

We then sought their feedback via a new survey to ascertain their views on how interesting, enjoyable and useful the UDLs are, with additional questions relating to specific feedback on the new elements.

Impact

The survey results for the newer cohorts were much more positive than the original cohort, after changes were made to the UDL format.

There was a significant increase in how interesting, enjoyable and useful the students found the days.

The trainees also largely agreed that the UDLs had an appropriate workload, e.g. one task per 1-1.5 hours.

They also largely agreed that UDLs included interactive and varied tasks. This finding is in contrast to some of the aforementioned literature of the importance of e-learning, and it must be remembered that too much e-learning can be less engaging for trainees.

The students also praised the simple reflective form as a helpful tool, and many appreciated the option to submit notes in their own preferred way.

Although we neglected to explore the role of the lecturer feedback in the new UDL survey, research shows that this makes for a more engaging e-learning session (Dixson, 2010), and may explain why the UDLs were now more favourable.

Moreover, the process of collecting data from the students via a feedback form seemed effective, in that we used feedback to adapt the specific teaching method, thus improving student satisfaction. Pedagogical research shows the importance of using qualitative questions (instead of, or as well as, quantitative methods) to elicit student feedback (Steyn et al., 2019).

Reflection

Overall, this redesign was successful, which may be down to the fact we used the student voice to make meaningful changes. This is in line with Floden’s (2017) research that student feedback can help to improve courses.

Furthermore, the changes we have made are in line with effective practice amongst other courses and universities, e.g. appropriate workload (Ginn et al., 2007), student choice of discussion format (Lin & Overbaugh, 2007), accessibility of resources (Mahmood et al., 2012) and lecturer interaction (Dixson, 2010).

There is a possible limitation in this case study, in that our more recent cohorts are generally happier on the course, and therefore may be more positive about the UDL. In future projects, it would be useful if we can notice themes within module evaluation/student rep meetings earlier, to then elicit specific survey feedback earlier in the course and make amendments sooner, allowing feedback from the same cohort.

In future variations of the survey, I would also wish to explicitly ask how trainees find sharing reflections on the Blackboard discussion groups, as this is one change we had not elicited feedback on.

Follow Ups

We have continued to utilise these changes in the UDL format with future cohorts,  e.g. reduced workload, variety of activities, simplified forms, choice of discussion format and lecturer interaction. We no longer receive concerns about these days in the student rep meetings since the original cohort. The Mentimeter feedback at the end of each UDL is generally positive, with one person recently commenting: ‘this was a very engaging day’.

References

References:

Dixson, M. D. (2010). Creating effective student engagement in online courses: What do students find engaging?. Journal of the Scholarship of Teaching and Learning, 1-13.

Flodén, J. (2017). The impact of student feedback on teaching in higher education. Assessment & Evaluation in Higher Education42(7), 1054-1068.

Ginns, P., Prosser, M., & Barrie, S. (2007). Students’ perceptions of teaching quality in higher education: The perspective of currently enrolled students. Studies in higher education32(5), 603-615.

Lin, S. Y., & Overbaugh, R. C. (2007). The effect of student choice of online discussion format on tiered achievement and student satisfaction. Journal of Research on technology in Education39(4), 399-415.

Mahmood, A., Mahmood, S. T., & Malik, A. B. (2012). A comparative study of student satisfaction level in distance learning and live classroom at higher education level. Turkish Online Journal of Distance Education13(1), 128-136.

Ramsden, P. (2003). Learning to teach in higher education. Routledge.

Sheikhaboumasoudi, R., Bagheri, M., Hosseini, S. A., Ashouri, E., & Elahi, N. (2018). Improving nursing students’ learning outcomes in fundamentals of nursing course through combination of traditional and e-learning methods. Iranian journal of nursing and midwifery research, 23(3), 217.

Steyn, C., Davies, C., & Sambo, A. (2019). Eliciting student feedback for course development: the application of a qualitative course evaluation tool among business research students. Assessment & Evaluation in Higher Education, 44(1), 11-24.

Links

CWI website: https://sites.reading.ac.uk/charlie-waller-institute/

DIGITAL EDUCATION & ACCESSIBLE LEARNING

Prof Suzanne Graham: s.j.graham@reading.ac.uk;

Prof Richard Mitchell: r.j.mitchell@reading.ac.uk

Dr Yota Dimitriadi:  y.dimitriadi@reading.ac.uk

Schools: MPCS and Institute of Education

Overview

This article reports on the joint Institute of Education / Department of Computer Science Leverhulme funded project concerned with improving online learning for three groups of students. Various recommendations are made, some relating to the Yuja lecture capture system. More details are on our DEAL site https://blogs.reading.ac.uk/deal/.

Objectives

  • To identify optimal conditions for presenting learning information for students in HE, specifically across three student groups: – deaf/hearing impaired (DHH); with dyslexia (DYS), those with English as a second language (L2)
  • To establish a platform for developing an agile system responsive to different user needs
  • To use this evidence base to inform the development of guidance to providers of online learning

Context

Although the prime focus of the project was on Deaf and Hard of Hearing (DHH) learners, research suggests that students with literacy and language difficulties also benefit from captioning and visual materials. We used material from two of the University’s well established MOOCS on FutureLearn Begin Robotics and Understanding Anxiety, Depression and CBT which had material in various formats.

Implementation

We created two versions of online learning materials across these content areas. One version of the materials was ‘unenhanced’ – termed MOOC. The other version was ‘enhanced’ to offer greater support to learners (for example, through British Sign Language, BSL), termed DEAL.

Participants (109, randomly allocated to conditions) viewed the materials in an online interview after completing tests of literacy and prior knowledge. After viewing they completed quizzes to assess learning, and questionnaires and an interview to gauge their views on and attention to various features of the materials. Additional participants (26) viewed the Robotics materials while their eye-movements were tracked.

The University’s lecture capture system Yuja stores information from the lecturn computer and an optional camera. Students can view any result on the YuJa server, controlling captions and the two streams from the computer and camera. The camera can be used to record a signer live, but a video of a signer done elsewhere can be merged with that from the lecturn.

Impact

The project recommends

  • Adding Advance Organisers (signposts given to students before they undertake an activity to help them structure the information they are about to learn and to direct their attention to key points).
  • Pre-viewing explanations of key subject specific terminology.
  • Breaking some of the information down into smaller segments with summaries.
  • Adding British Sign Language to video clips.
  • Drawing participants’ attention to how to modify and use captions.

The overriding message for online learning is that personalisation of modifications is key, and that can be achieved by systems such as

  • Ally, where students can access material in different formats
  • YuJa, where students can personalise captions and signing.

Reflection

  • For both content areas, post-viewing quiz scores for MOOC and DEAL were very similar.
  • For Robotics, DHH DEAL participants had higher average post-viewing scores than DHH MOOC participants, giving some indication that the DEAL modifications helped.
  • Across all groups, DEAL modifications were found helpful by many participants. There was however a lot of individual variation regarding what was helpful/unhelpful. Participants wanted to be able to personalise their viewing: for example, by moving the BSL to a certain area of the screen, to lessen cognitive overload.

Specifically for the DHH students, we recommend

  • Provide a BSL version that can be turned on or off. Not all DHH students find BSL helpful or use it as their first and preferred language so the option to select BSL is likely to be helpful.
  • The option to move the interpreter to other parts of the screen was also favoured, which can be achieved for instance by Yuja, see link below.

Follow Up

Currently, relevant videos in the Begin Robotics MOOC are being enhanced to incorporate relevant findings from the project. These will be available in future runs, which will also be taken by the first year Computer Science students.

Links

The project web site https://blogs.reading.ac.uk/deal/

Viewing a video and a signer on Yuja  https://reading.yuja.com/V/Video?v=186538&a=492227066

The CBT MOOC  https://www.futurelearn.com/courses/anxiety-depression-and-cbt

The Begin Robotics MOOC https://www.futurelearn.com/courses/begin-robotics

The university as a lending service – STEM learning with Lego

Dan James: Daniel.james@reading.ac.uk

Institute of Reading

Overview

The case study will describe a TLEP (Teaching and Learning Enhancement Project) to develop STEM teaching and learning.  Over 400 Reading Partnership student teachers (RPTs) have received some training, which was well received in university sessions.  This case study will also describe how the programmable Lego kits were used successfully with 60 children who visited the university during their science week, as well as discussing future plans.

Objectives

The aims for the TLEP were:

  • To provide the students the skills, resources and confidence within university to deliver cross-curricular STEM activities whilst on their school placement, either in lessons and / or as an after-school club.
  • To promote and model cross-curricular working within a module and collaborative working within the community.
  • To maximise the use of expensive resources and to demonstrate ‘proof-of-concept’ that lending of resources to partnership schools is viable.

Context

The majority of the incoming primary teaching ITT (Initial Teacher Training) students do not take STEM A-levels and lack confidence teaching STEM in schools. Over the course of the teacher training programme, knowledge in the individual subjects improves.  However, due to the siloed nature of university module teaching, students do not always see the cross-curricular opportunities to teach the subjects, nor have the confidence to deliver STEM initiatives, especially design and engineering challenges to children. With an ever-increasing shortage of a skilled STEM workforce, promoting STEM skills in children from a primary age is important for the future of Britain’s economy (ASPIRES project, 2013). One major barrier to achieving this goal is schools having the finances to purchase equipment.  Even if schools have access to the necessary resources, they can often be used once and then lost in the back of the store cupboard.  Further barriers include teachers having the necessary skills and confidence to deliver these sessions.

Implementation

By modelling working in a cross-curricular fashion and exposing students to carrying out the activity experientially, the aim was to upskill the RPTs (Reading Partnership student teachers) in their skills and confidence to deliver these types of STEM projects in their training placements and in their future schools.

Funding from the TLEP was used to purchase 10 Lego Spike Prime kits which provided sufficient resources for a class set both for use in university science teaching sessions, and with schools, where class numbers are approximately 30 children.  By the university purchasing the resources and making them available for lending through the Learning Hub at London Road, the barrier in terms of resource cost to schools using these kits (~£300 per box new) was therefore eliminated.

The ambition was that after the workshop session as part of their university course, the Reading Partnership Teachers (student teachers) would then be inspired to borrow the kits and use with their own classes.  However, the use of practical resources was severely hampered by covid-19, so the anticipated impact was reduced, and the timescale delayed.

But as part of a ‘Science Week’ in May 2022, a partnership school was invited to the science teaching labs on the London Road Campus.  The children from a years 5 and 6, took part in a session to build 2 different motorizable grabber arms, and evaluated which was the best grabber to pick up different plastics in a plastic recycling sorting facility.

Impact

Aim 1:  The university session provided the students the skills, resources and confidence, as evidenced by module reviews and overhearing many of the university students saying, “that was really fun!”.  It also gave them the chance to work together and problem-solve groups, developing relationships amongst the cohort.

Aim 2:  By inviting a school into the university’s science teaching lab, we met the aim of working collaboratively with the local community, meeting the aims of the Community, Engagement  and Sustainability strands of the University Strategy. The children very much enjoyed this with them rating the session 9 or 10, out of 10.

Aim 3:  This aim was to maximise the use of expensive resources and demonstrate  that this is an option when working in partnership with schools.  The fact that this work has already been used by 400 university children and 50 children, the means that the cost per participant is now down to approximately £5, based on the purchase cost of £2500.  As further students and children, use these resources this cost per participant will further reduce.

Reflection

This was a good first good start to supporting delivery of these cross-curricular STEM sessions with a physical computing component. Part of this aim was also to model cross-curricular working within a module with the computing training happening in computing sessions.  Due to covid-19 disrupting the delivery of both the science and computing sessions, this was not as well implemented as initially hoped.

Further confidence and awareness in how these kits can be used would be beneficial to support increased implementation in schools, both for the trainee teachers and for experienced teachers mentoring them.

Follow Up

I am aiming to train experienced teachers in using the kits, to ensure even greater impact and support for our RPTs on school placements.

Using more low-tech hybrid/hyflex teaching methods in English Literature modules – benefits and limitations.

Professor Cindy Becker: l.m.becker@reading.ac.uk

Literature and Languages

 

Overview

During the pandemic, hybrid learning was used in two modules I taught – one at Foundation Level and one at Part One. I found it worked well for discussion-based sessions. I recognize that there are potential gains and losses to continuing this practice post-pandemic and I hope this case study might contribute to our institutional conversation.

Objectives

The objectives were to:

  1. Include as many students as possible in live teaching sessions.
  2. Maintain the energy of a group learning setting for all students.
  3. Reassure students that they were still part of their learning cohort, even off campus.
  4. Ensure good attendance and engagement on key modules at Foundation Level and Part 1, which might be seen as ‘at risk’ stages for student retention and attainment.

Context

Seminars in Arts and Humanities offer a specific type of learning experience, based upon developing ideas through discussion in a group setting. Students did not respond well to the idea of one-to-one (or two/three) sessions as a replacement if they were off campus. They struggled to attend or to engage in sessions which could become ‘information-giving’ tutorials rather than ‘knowledge-sharing’ seminars.

Implementation

Hybrid learning, in my case through a Blackboard Collaborate session running simultaneously with a campus seminar, was used for some modules in my department to include students who were unable to come onto campus due to the pandemic. I used the desktop computer in the teaching room, and I had the camera facing me (the online students preferred this to looking at their fellow students). The sessions were low tech, so I did no more than share my screen for visual material used in the seminar. I put the handouts in my Blackboard module and shared them in the Blackboard Collaborate chat. At the end of each seminar the online students stayed in the session so that I could check in with them once the on-campus students had left the room

For the Part 1 module, the module convenor alerted me to those students to whom I would need to send the Blackboard session link. No students other than these were offered the option of attending online

Impact

The first three objectives were achieved with the Part 1 module. An unplanned outcome was that we opened the hybrid sessions to a student whose mental health precluded on-campus attendance for two weeks, so the impact was wider than expected.

I then introduced hybrid learning in a Foundation module on which there had been poor attendance, for seminars in which I was offering important information about assessment.

I sent the online link to all students on the Foundation module. An unexpected outcome was that this did not significantly reduce the number of students who attended the on-campus seminar, but it did draw in the students on the module who had been regular nonattenders.

Reflection

I think hybrid teaching and learning worked well because our subject lends itself to relatively low-tech, conversation-based seminar learning. It worked for me as a seminar leader surprisingly well; this might in part have been because students were happy to give me leeway as I drifted off camera or took a few moments to catch up with their chat contributions.

On reflection I wish I had explored the option of students in the room interacting on their laptops with the online-only students. Responses would be passed to me from students online via students in the room on WhatsApp during general discussions, and I would have liked to facilitate that more formally. I wonder if we are underestimating our students’ abilities to multi-task in this way when we offer them campus-only sessions and I am keen to see whether the blended learning landscape of our future might also allow for more hybrid learning opportunities.

Follow Up

I led a professional conversation on this topic in my department to share our experiences and, perhaps, to consider what we might lose if we abandon hybrid teaching next year, weighing this against the potential risks that might be associated with its continuance. I would be pleased to hear from any colleagues who would like to become involved in a wider conversation.

Links

Note: this entry is submitted in conjunction with Gemma Peacock’s T&L Exchange entry in which she shares her experiences of hybrid/hyflex learning in language learning and academic skills development courses in ISLI (add link to that blog here).

Hybrid Teaching and Learning Network (on Teams) for sharing good practice: https://tinyurl.com/ye6awyuz

General survey on hybrid teaching and learning practice at the University of Reading: https://forms.office.com/r/bF9k3amY3d

Using more high-tech hybrid/hyflex teaching methods in language and academic skills learning contexts – benefits and limitations.

Gemma Peacock: g.peacock@reading.ac.uk

ISLI (International Study and Language Institute)

Overview

This case study reports on a successful pilot of hybrid/hyflex teaching in the International Study and Language Institute (ISLI) where one class contained fully online remote students and blended on-campus students together. There are both benefits and limitations to using this approach for language and skills learning contexts.

Objectives

The objectives of the pilot were:

  1. To investigate and refine the technical aspects of running hybrid lessons.
  2. To develop guidance for teachers.
  3. To ascertain whether hybrid/hyflex teaching and learning methods can be used in contexts where more complex interaction patterns are required.
  4. To gather feedback from teachers and students on their experiences.

Context

The hybrid/hyflex pilot was run initially because small cohort numbers for ISLI’s autumn 2021 Pre-sessional English course precluded the running of two separate classes: one online and one on-campus. While evidence existed in other institutions of the successful adoption of hybrid in lecture-style classes, it was not known if hybrid would work in ISLI’s context as it specialises in English language teaching and academic skills development for international students. These fields require complex interaction patterns between students themselves and with their teachers.

Implementation

During the pilot we refined the technology and processes necessary to deliver hybrid successfully as follows:

  1. A Teams meeting runs during the lesson, displayed on a smart board. Remote students attend this meeting and on-campus students can also do the same using their own devices to receive documents or links easily in the chat during the lesson.
  2. A device called a Meeting Owl Pro takes a constant 360 degree panoramic shot of the whole classroom and also shares video and audio of the speaker as they speak and move around the room.
  3. Two monitors on the teacher desk means they can interact with the Teams meeting functionality (such as displaying slides or documents) and they can use the other screen for other purposes (such as teacher notes).
  4. Teachers are thus able to speak to all students and remote students can speak to and see on-campus students via the Owl and vice versa for the implementation of a wide variety of interactive tasks.
  5. Focus groups were held with teachers and students on the pilot to gather data on their experiences.

NB: Hybrid/hyflex is possible without a Meeting Owl so long as a reasonable quality microphone and camera exists in the classroom.

Impact

ISLI’s hybrid/hyflex pilot achieved its outcomes. We investigated and refined the technical aspects of running hybrid lessons through trial and error. This resulted in the production of:

  • a Meeting Owl Pro set-up guide for teachers.
  • a guidance document for teaching and learning via hybrid/hyflex methods.
  • a Teaching and Learning Sub-committee report on the pilot.
  • future recommendations for hybrid/hyflex delivery in ISLI.

The feedback gathered from teachers and students on their experiences was generally positive. When combined with the feedback from ISLI’s TEL team, it was agreed that while it is possible to use hybrid/hyflex in language learning or skills development contexts it may not be desirable. Some recommendations include:

  • Comprehensive teacher training in hybrid technology and pedagogy.
  • Lesson design and staging must enable both remote and on-campus students to participate equally and to receive equal attention from the teacher.
  • Where there is a small cohort (<5) these should be integrated into an online/F2F class to form a hybrid cohort to improve the student experience.

Reflection

The pilot study was successful as it allowed for on-the-job teacher-training through action research, and a more granular understanding of how hybrid/hyflex can work in terms of both technology and pedagogy. I believe ISLI’s expertise of hybrid/hyflex teaching and learning methods could be called on more widely across the university as the blended learning landscape of the future takes shape.

Hybrid/hyflex teaching and learning has been hailed as a more inclusive and accessible mode of study since it gives students agency to choose whether to study from home or on campus according to their immediate needs. This has proved beneficial in other teaching contexts (add link to Cindy Becker’s T&L blog post). Current visa regulations, however, do not permit international students on Pre-sessional English courses to switch between online and face-to-face delivery within a course, as students receive an offer for one mode of delivery only. This means that some of the potential benefits of hybrid/hyflex delivery are not available to them at this time.

 

Follow Up

Since the pilot, professional conversations about hybrid/hyflex have taken place with schools across the university to include the Department of English Literature and Henley Business School. In May 2022, presentations on hybrid were delivered to HE colleagues at the JISC Change Agent’s conference, and with English Language Teaching professionals at the IATEFL conference in Belfast. Data on hybrid methodology usage is currently being gathered from a survey Gemma Peacock and Cindy Becker have circulated with the aim of writing a journal article in the near future.

 

Links

Note: this entry is submitted alongside Cindy Becker’s T&L Exchange entry in which she shares her experiences of low tech hybrid/hyflex learning in English Literature seminars (add link to that blog here).

Hybrid Teaching and Learning Network (on Teams) for sharing good practice: https://tinyurl.com/ye6awyuz

General survey on hybrid teaching and learning practice at the University of Reading: https://forms.office.com/r/bF9k3amY3d

Co-Creating a Programme Focussed Assessment Strategy in BSc Food Science

Julia Rodriguez Garcia: Department of Food and Nutritional Science

j.rodriguezgarcia@reading.ac.uk

Link back to case studies on the T and L Exchange website

Overview

This student-staff partnership involved students from all year groups, academics and the programme administrator of the BSc Food Science. In this PlanT project the Transforming the Experience of Students through Assessment (TESTA) methodology was used to develop programme focus assessment strategy that could lead to a reduction in volume and improved distribution of assessment, and overall enhance student and staff experience

Objectives

The main aim of this project was to take an evidence-based approach to enabling a cultural shift required from ‘my module’ to ‘our programme’ to engage staff in an integral restructuration of the assessment design, volume, and distribution in the programme.

The main objectives of the project were:

  • To assess if there is assessment is evenly distributed in terms of volume and weighting and type across the programme
  • To explore and propose changes in the design of assessment tasks to move to a programme level assessment strategy that could improve student and staff experience

Context

Restructuration of programmes have been usually performed at modular level, resulting in limited coordination of the learning (including assessment) strategy at programme level. In the BSc Food Science this led to student complains about high volume of assessment tasks due in a short period of time. External examiners also commented on the high volume of coursework components. And staff was overload with marking and feedback tourn around deadlines.

Having small assessment tasks set within a module, results in only a small number of concepts assessed as part of each task which leads to losing the holistic perspective of the subject area creating the effect of fragmenting knowledge and promoting surface learning, such as memorisation of content. Thus, a change from assessment of learning to assessment for/as learning will facilitate a change in behaviours improving students’ motivation and rewarding staff efforts

Implementation

Figure 1. Structure of the Transforming the Experience of Students through Assessment’ (TESTA method)

We developed Programme Level Assessment Maps that allowed us to calculate the number of assignments per credits, the weighting distribution per week and per term. These maps helped to create a visual interpretation of the students’ experience to increase staff awareness of the real situation at programme level (not just in their modules).

Assessment Map of Part 1 of an BSc in Food Science. Assessment activities are labelled in colours: purple for formative activities, pink for summative activities from compulsory modules, yellow for summative activities from optional modules, red for exams. The type of assessment activities and the weighting are also presented in the map.

In collaboration with students, we modify the Assessment Experience Questionnaire to give students more space to reflect in their experience when performing certain type of assessments. This questionnaire was run two consecutive years.

One of the students leading the PlanT project carried out a focus groups with students from all year groups to reflect and discuss on the way they learn, the skills the develop and the challenges they faced completing assessment tasks.

We combined all the data, discussed it in light of published literature and draw some final suggestions to modify our assessment design practices. The results and future strategies were presented in a staff meeting lead by a student.

Impact

This project has facilitated and achieved a change in staff mindset from module level to programme level. This has been reflected in a 9% reduction on assessment activities from 2018-2019 to 2019-2020.

Figure 3: Principles and practices that will inform and define the implementation of a programme focused assessments trategy

This shift in focus will allow us to perform more holistic changes following a common set of principles that underpin the development of authentic assessment to promote higher order thinking skills, integration of knowledge (horizontally and vertically), and students’ motivation and independence.

The conclusions from this project and the proposed strategies to develop a programme level assessment approach have been implemented in the department through the Portfolio Review

Reflection

The development of a cohesive staff community of Programme Directors, Module Convenors, and support staff from the teaching hub with strong communication links is crucial for the design and delivery of a high-quality programme.

Moreover, student voice and partnership are crucial for the co-development of teaching and assessment approaches, working collaborations are transformational both for the community wellbeing and for achieving highly successful outcomes. Development of a sense of community within the department is something distinctive in Food Science.

Follow up

Brown, Sally, and Kay Sambell. 2020a. “Changing assessment for good: a major opportunity for educational developers.” Assessment, Learning and Teaching in Higher Education; Sally Brown. https://sally-brown.net/kay-sambell-and-sally-brown-covid-19-assessment-collection/.

Murphy, Vanessa, James Fox, Sinéad Freeman, and Nicola Hughes. 2017. ““Keeping It Real”: A Review of the Benefits, Challenges and Steps Towards Implementing Authentic Assessment.”. The All Ireland Journal of Teaching and Learning in Higher Education (AISHE-J) 9 (3):2801.

O’Neil, Geraldine Mary. 2019. “Why don’t we want to reduce assessment?”  All Ireland Journal of Higher Education 11 (2):1-7.

TESTA. Transforming the Experience of Students Through Assessment. https://www.testa.ac.uk/

Villarroel, V.; Bloxham, S.; Bruna, D.; Bruna, C.; Herrera-Seda, C. Authentic assessment: creating a blueprint for course design. Assessment & Evaluation in Higher Education 2018, 43, 840-854, doi:10.1080/02602938.2017.1412396.

The One Where a Timetable Merger Gives Rise to a Curriculum Implementation Review

Emma-Jayne Conway, James Kachellek and Tamara Wiehe

t.wiehe@reading.ac.uk

Link back to case studies on the T and L Exchange website

Overview

Staff and students in CWI collaborated on a project initially designed to merge two timetables of sister programmes to aid cross programme working (objective 1) but gave rise to the perfect opportunity to review the way our PWP curriculum is implemented following the pandemic (objective 2). In this blog, we reflect on achieving both objectives within our original timeframe!

Objectives

1.     To create a single timetable to aid cross-programme working for teaching and administrative staff.

2.     To review curriculum implementation including structure and modality on a modular and programme level with all key stakeholders.

Context

In response to a departmental restructure, we required more efficient ways of working across programmes starting with a uniform timetable. Early on, the project evolved to also review the structure and modality of the curriculum. Our two sister PWP training programmes (one undergraduate and one postgraduate) are virtually identical with a few exceptions but historically had been managed separately.

Over the course of 2021, we planned, designed, and implemented a timetable merger for our September cohorts. This impacted on 3 modules (4 for undergraduates) that form the PWP training year for the MSci Applied Psychology (Clinical) students and the Postgraduate/graduate Certificate in Evidence-Based Psychological Treatments (IAPT Pathway).

Taking both Higher Education and Mental Health Care processes into consideration was no easy feat, including those specific to University of Reading (e.g., term dates), our national PWP curriculum specifying the content and learning outcomes for our 26 teaching days and 19 study days, and British Psychological Society (BPS) accreditation requirements. Modality was a particularly important topic throughout this project, taking key learnings from remote delivery during the pandemic as well as awaiting guidance from our professional accrediting body.

Overall, it served as an excellent opportunity to work collaboratively with staff and students to review the implementation of PWP training at the University of Reading.

Implementation

  1. Early 2021: The PWP team met on several occasions to discuss the possibility of merging the two timetables, including transitioning to a “blended” format of online and face-to-face teaching post-Covid. We set out a timeline for planning, designing, and implementing the project.
  2. Advice was sought from the Director of Training in CWI and colleagues in Academic Development, CQSD based on their experience of timetable mergers and a green light was given based on our draft plans!
  3. Several options were considered before the final format was arrived at: Face-to-face teaching is weighted towards the first module/term with progressive increase to the online taught content as the course progresses. (Rationale supplied elsewhere in this blog).
  4. The educator team were able to draw on feedback from online teaching to gauge the attitude of the student body to online learning, as well as expectations and concerns related to a return to the classroom (see Impact, below). The student voice was important in terms of utilising partnership to create meaningful change to curriculum implementation. However, engaging professional practice students via the course reps was a challenge due to time constraints, therefore, we were able to engage graduate instead. This is something we would consider earlier on in future projects.
  5. The educator team unanimously agreed that the externally taught content of the VEC module could be effectively taught with mixed cohorts from the Core-PWP and MSci cohorts using an online approach.
  6. Information on the changes was disseminated to Program Administrators to enable efficient implementation. External Educators were made aware of the retention of online lecture sessions, and the mixed-cohort approach, by the VEC module convenor.
  7. Timetables were updated by the Program Director, in collaboration with Module Convenors; consideration has been given to the potential Workflow impact of closely aligning multiple cohorts (see below). Timetables have been looked at by the team ‘side-by-side,’ to ensure that Workflow balance is maintained for educators across all cohorts. We can continue to monitor the impact on workload while adjustments are made to teaching (such as with the Working Document mentioned in the Follow-Up section, below).
  8. IAPT Services were made aware of the changes to the timetables

Impact

As of October 2021, the merged timetables are proving effective, with no major barriers, having been detected. Predicted barriers included those to effective teaching of (previously face-to-face) content, student/staff dissatisfaction with a blended approach, and significant administrative/technical difficulties.

Face-to-Face teaching resumed in September 2021 and has been a successful return to the classroom. Educators report being able to switch between live sessions and face-to-face teaching days without significant barriers.

The educator team plan to continue to gather feedback on the student experience of the blended and merged approach. We will be able to assess feedback when the first cohorts fully complete in June 2022.

Feedback will be sought from module convenors, educators, and program administrators using “menti” feedback forms, bi-weekly team meetings and informal qualitative discussion, to gauge the impact of the changes on workflow. Student feedback will also be monitored through end-of-module feedback collated by the School.

Reflection

  • The challenge of engaging professional practice students and utilising graduates to overcome this. We will consider setting up graduate/patient advisory group for future projects.
  • Using feedback from a MSci graduate led to timetable changes to ensure readability and clarity for students. This included points such as colour coding F2F v online teaching days, explaining acronyms, etc.
  • Involving all members of the team (especially Module Convenors) felt like a much more meaningful and collaborative process than Programme Director decisions alone. It gave Module Convenors autonomy over their modules as well as aligning learning outcomes across the 3 modules of the programme which is particularly important for clinical training. Other courses may wish to replicate this approach to build team cohesion and allow all colleagues to make meaningful contributions to programme changes and delivery.

Follow up

  • Working document has been created for the educator team to comment on the teaching they have just delivered i.e., was there enough time to deliver all content? This has allowed changes to be made within a matter of weeks as the same day is delivered across the programmes. As a result, we can fine-tune the timetable and delivery of the programme quicky and efficiently to improve the student experience.
  • We will review module by module and at the end of each cohort to continue making any necessary adjustments. Module and programme evaluations, student-staff rep meetings and any feedback from individual teaching days will also help to inform this.

 

Driving programme development in the IOE: student focus groups and paper writing

Jo Anna Reed Johnson – Institute of Education

j.a.reedjohnson@reading.ac.uk

Link back to case studies on the T and L Exchange website

Overview

This article outlines the thinking to drive programme development through student focus groups across three IOE programmes.  The outcome to write a paper and present at a conference helped me to frame this project with a team of academics focusing on changes made during Covid-19 (2020-2021).  This article will share reflections on setting up and running of the focus groups, the delivery of the conference presentation and the final paper writing.  Finally, it will discuss what we have learnt from this and what we will continue to do.

Objectives

  • Share 4 academic perspectives on the redesigning of three modules (SKE, STEAM, PGCE Sec Science) that all have practical elements (laboratory or school), due to Covid-19, by sharing what we did and exploring the student perspectives
  • Show how we designed opportunities for discussion and collaboration when conducting practical work or school related work online
  • Consider the use of student focus groups for programme co-development
  • Reflect on the collaborative nature of paper writing and co-programme reflections

Context

At the IOE there are a range of teacher education programmes, with a practical focus.  The four colleagues engaged in this article were involved with Skills in Schools (ED2TS1 – March to July 2020), SKE (IESKEP and PFTZSKEMATHSA– March to Aug 2020) and PGCE Secondary Science (GFSTSCIENCE – September 2020 to June 2021).  These programmes all require students to work in schools and engage in a science laboratory (if science focused).  As COVID hit in March 2020 we had to think quickly and imaginatively, transforming our provision to be online where required.  Having worked across all three programmes I felt it was pedagogically appropriate to engage our students in the ways we had throughout their learning during the pandemic, where they worked in online communities of practice to reflect.  Thus, we decided to set up a series of focus groups with students reflecting on the impact of the changes and to provide insights for future programme innovations.  This culminated in a conference presentation and paper.

Implementation

The focus was to drive programme development through reflections and shared experiences of academics and students.  I set up a project timeline and MS Team to manage and drive the deliverables, with the end goal to engage students as co-programme developers and to culminate in a conference presentation and paper.  It required framing the project, seeking ethical approval and funding, setting up focus groups to collect data, then reflections and writing up.

Framing the project allowed me to maintain the focus for the redesigning of three modules that all had practical elements (laboratory or school), due to Covid-19.  And then exploring how that had impacted on students through focus groups. It was the conference and paper deadlines that drove this activity and timeline.  At first colleagues wondered why we were writing a paper for a submission related to the School of Architecture (Manchester and Manchester Metropolitan University), but in fact it was because it was about ‘place’.  The remit was a paper related to ‘online education: teaching in a time of change’.

Seeking ethical approval and funding all required knowing where to go and what to do.  Ethical approval required submission of an ethical approval form (including consent form, interview schedule, focus group details) to the IOE ethics committee.  Then applying for funding through the University Travel Grants Scheme – Tasha Easton – e.saxon@reading.ac.uk

Data Collection was initially carried out using MS Forms, for the initial feedback request.  Consent was also required, so where this could not be achieved in person, there was a need to have consent approval attached to the top of the MS Form.  Once participants had consented and those who were willing had indicated taking part in the focus groups, I could set up a series of focus groups across the three programmes, to take place on MS Teams.  We decided to split the four sets of interviews into subject specific groups so that the conversations and reflections could be driven by the students.  One student was nominated as the chair, and they had a template of questions to guide their discussions.

Paper Writing was a challenge as we needed to fit this around our Teaching Focused roles.  I created a writing template after attending an IOE Research and Scholarship Writing Workshop with Professor Alan Floyd.  I scheduled meetings to review, discuss and allocate sections of writing.

The whole process began in December 2020 and continued through to 30 May 2021, with the conference in 21-23 April 2021 (July 2021- Paper Publication).

 

Impact

There were several elements of impact:

  • Working collaboratively with colleagues to reflect on programme development
  • Engaging students as co-programme developers
  • Attending a conference (where funding allowed)
  • Conference paper presentation
  • Conference paper publication

Reflection

In terms of the setting up of focus groups and driving data collection, we learnt that we needed to be organised, and the timeline/plan really helped to keep that focus.  There were times where we were too busy, but we had to create time as we had deliverables to meet.  If we had not have had those deliverables of a conference presentation and paper, we may have let this slip and do it ‘next year’.

Writing the paper was a challenge in that we had not done this together before, and some colleagues had not written an academic paper in a very long time, or even an educational one.  So, creating that writing template and allocating tasks worked.

Gaining conference funding can always be a challenge.  But reaching out and asking was the first thing to do. Then finding out what could be offered at the University/School Level.  Next time, we would all like to attend the conference.  Being an online conference made it more difficult to engage, and I think next time we would plan to all get funding an attend a face-to-face conference so that we too can benefit from being part of the Community of Practice.

What we will continue to do….

  • Develop students as programme co-developers through focus groups, engaging them in the paper writing.
  • Use focus groups to help us (academics) reflect on our own practice and discuss developments across programmes.
  • Drive programme development through the sharing of practices, building communities of practice with timelines and deliverables.

What else will we do…

  • Engage students in the paper writing and conference process.
  • Seek funding to attend a F2F conference with colleagues to allows us time and space to continue to reflect on practice.

Links

Research and Travel Grants Committee: https://www.reading.ac.uk/closed/research/committees/res-researchtravelgrantsubcommittee.aspx

AMPS Conference 21-23 April 2021 – https://architecturemps.com/online-ed-conference/

Merging the Academic Tutor System into Compulsory Core Skills Modules

Lizzy Lander – School of Chemistry Food and Pharmacy

e.r.lander@reading.ac.uk

Link back to case studies on the T and L Exchange website

Overview

This blog will outline the successful integration of core (compulsory) skills modules with the academic tutor system via a curriculum of tutorials designed in this project to be delivered by tutors. This project involved the successful design and scheduling of tutorials for tutors to deliver that supported content (e.g. writing and referencing) in core skills modules to allow better support for student academic skill development and also more closely link tutors into modular taught material.

Objectives

  • Link new academic tutor system with existing Key Skills modules through newly designed academic tutorials discussing core skills to be delivered by tutors.
  • Design this curriculum of tutorials to improve engagement and development of skills at relevant points in the academic year.
  • Design tutorial resources for tutors to ensure consistent support for tutees.
  • Schedule tutorials so tutors and tutees have a place/time to meet in their timetable to facilitate engagement.

Context

SBS had been delivering core skill “Key Skills” modules in parts 1 & 2 for a number of years (since 2015) that focused on academic skills (e.g. writing, referencing,). The introduction of the academic tutor system (2018) with greater focus on academic skills development was closely aligned with the learning outcomes of these modules therefore it was proposed to link the two together.

Implementation

Firstly, an audit of the core skills taught in the Key Skills modules took place to identify which would be most impactful for student development to be reinforced by being integrated into tutorials with academic tutors. Then assignment timetabling was examined to create a schedule of tutorials for the identified skills which allowed practice and formative feedback before assignments, as well as post-assignment feedback to allow students to identify areas of development.

Next, tutorials were formally timetabled, so students viewed these sessions as part of their “normal” academic schedule rather than optional meetings with their tutors.

Afterwards, resources for tutors were created so they could facilitate these tutorials. This consisted of a one to two page pro-forma to inform tutors about the running of the session. Other resources created included materials for activities such as essays to critique as a group. This would also help improve consistency between tutors delivering these sessions as all tutors would have the same session to deliver.

Finally, this project was presented to staff along with details on how resources and information would be disseminated (initially email). Throughout the year tutorials were run by academic tutors directed by the pro-formas and resources with support if needed. Tutors also marked their tutees’ assignments in Key Skills and gave them feedback in their tutorials. Outside of the scheduled tutorials tutors gave one-to-one support for tutees as needed.

Impact

The implementation and influence of these structured and timetabled tutorials was highly effective in supporting tutees in improving academic skills and improving the consistency of tutors engaging with their tutees. Positive impact was clear from the students (surveying parts 1 + 2 in 2019); 53% felt supported/very supported by their tutor and overall, 65% were satisfied/very satisfied with their sessions with 62% finding the summative feedback from tutors helpful and 72% found it useful/very useful to have tutorials in their timetable. Tutors also fed back they have a much clearer idea of what do at tutorials and how best to support student development, whilst valuing the resources provided in this project.

Overall staff and student experience was positively impacted with staff being led and guided to successfully support student development more effectively and consistently.

Reflection

This activity was successful in the way in which it blended together academic tutors,  compulsory modules, as well as assessment and feedback. This generated a platform from which students could learn and practise academic skills for success at university in both a compulsory module and with their tutor through formative and summative feedback. This also helped formalise the role of the tutor for both staff and students giving both groups direction, which ultimately benefitted the students’ academic development. Given that each tutorial had a pro-forma of discussions and activities this helped all tutorials stay consistent so all students got broadly the same development opportunities. Finally, the timetabling of meetings made the tutorials like a normal part of the academic calendar encouraging engagement.

Implementation when supplying information and resources via email to academics was shown to not be the most efficient distribution method and ultimately, some students did not attend tutorials despite reminders of the purpose of these sessions, meaning not all students benefitted from this project.

Follow up

The core outcome of the project in that Key Skills modules would be linked by academic tutorials run by tutors and assessments marked by tutors has continued to be implemented in SBS. However, some alterations have been made for more efficient accessing of materials, by placing resources on a OneDrive that could be accessed at any time. This then evolved into an MS Team to store these resources and also allow tutors to ask questions.

References