E-submission, marking and feedback – Pilar Gray-Carlos

By laurenmccann Comment Closed

OBJECTIVES

  • To facilitate the administrative process in submission of summative assessment
  • To inform module convenors and language teaching fellows of the tools supported by the University LMS Blackboard Learn
  • To provide the opportunity to apply the above tools, gather experience and inform decision on best approaches and best practice
  • To explore usability and applicability of existing marking criteria in the form of Tii (Turnitin) rubrics
  • To explore and facilitate a transition to use a basic set of QuickMarks across the Department whilst enabling room to create language specific amendments
  • To facilitate timely and transparent accessibility of results for students via the Grade Centre

CONTEXT

As part of summative assessment, students of intermediate to advanced language courses in IWLP Chinese, French, Italian, Japanese, German and Spanish submit a project (between 600 and 1000 words or characters according to language and stage) researched and written in the Target Language.

IWLP deals with a large volume of students each year so it was important to explore ways of facilitating a point of submission that would enable staff to easily follow up submission deadlines and late submissions eliminating paper based trails and multiple parties involved in the process, making it timely and easily accessible for staff to keep track of submission.

As the majority of language teaching staff works on part-time basis, it was felt that it would be of advantage to have a point of access to student´s work from different locations. This also meant adopting electronic marking and feedback as a way to facilitate marking and moderation remotely.

Three years ago it was unclear whether Tii would support the modern languages provided by the IWLP programme. Once it was established that it did support modern languages, it was felt that the use of similarity reports would both assist teachers in detecting plagiarism and be good for student learning as it would force students to revise not just content but language as well and re-write when necessary.

One of the advantages of using electronic submission, marking and feedback is that both the marking criteria and the feedback can be provided in the same space, therefore avoiding reprinting and waiting for students to collect feedback. Language projects are assessed on the following areas: content, structure, vocabulary, grammatical accuracy, range of expression, syntax and variety of grammatical structure. The aim was to upload the project marking criteria in the form of rubrics hence facilitating all the tools for marking and feeding back in one place for tutors, providing an area readily available for moderation, and granting ease of access to results and feedback for students.

There were two e-submission options to be explored: e-submission with inline grading or e-submission via Tii assignment submission, the latter supplying the facility to use rubrics and quick marks via the Turnitin Suite.

IMPLEMENTATION

Initial meetings took place three years ago with members of the TEL team which highlighted the advantages of using the electronic submission of written work. The meetings involved coordinators and module convenors of the languages that initially provided intermediate to advanced stages: English for Erasmus, French, German and Spanish. It was then agreed to pilot the use of electronic submission and to initially explore the use of “inline marking” tools for marking and providing feedback.

Further training was arranged, delivered by both the TEL team and Pilar Gray Carlos and on-going support was provided on an ad-hoc basis.

The first round of assessments took place and the feedback collected from tutors was varied. Some colleagues developed feedback systems utilising tools such as colour underlying and text boxes. As not only the content but the language is assessed, and identifying, correcting and explaining language mistakes can be a detailed process it was felt that, not only it took time to get familiar with the new system but that the result of the corrections and feedback was not easily accessible to students, making it necessary to print student´s work and go over corrections and feedback again with students in the classroom.

A period of required e-submission, but voluntary use of electronic marking and feedback followed until there was confirmation that Tii supported other modern languages. At this time modern languages such as Mandarin Chinese and Japanese had added intermediate courses to their provision. It was then decided to take the opportunity to start using Tii also as a formative tool, and in doing so, familiarising students with its use and enabling them to self-evaluate and readdress their own work. The use of similarity report was enabled for formative submission during the course and in view of the final submission of summative coursework.

Opportunities for training by the TEL team and in-house training were organised and provided by Pilar Gray Carlos and more experienced colleagues within ISLI. In this way module convenors and tutors were shown how rubrics and QuickMarks are used for marking and feed back in language teaching (see Rob Playfair case study and Jonathan Smith´s interview).
At the same time, and in parallel with work on e-submission, marking and feedback, a Grade Centre was created for the 31 modules provided in the 10 different languages. Weighted columns were created per assessment per module, teachers could directly input results and students would have direct access to marks as they were released.
Having all that data available also meant that, although limited, some reports could be printed with regards to module performance per assessment and even for languages where classes are taught in parallel groups, group performance data reports could be produced.

Since then the EMA Core Systems Team has delivered a more streamlined process which produces similar data sets on RISIS (for more detailed information see the EMA Programme short videos link below).

IMPACT

The use of e-submission has enabled a variety of approaches to formative assessment to flourish, some languages have made the most of using e-submission to collect student´s work and to feedback on line.

As per summative assessment, the adoption of QuickMarks is facilitating marking, and once the teachers get accustomed to using them it becomes an efficient way to point out generic language errors.

The use of the Grade Centre was a success, as it cut down on administration, freeing time on the side of administrators and teachers and it provides helpful information as to the performance of certain cohorts and groups. The only drawback was the missing step between Grade Centre and RISIS. At that point in time the only way to update records in RISIS was by downloading all marks in the form of a spreadsheet and manually inputting them in RISIS. The EMA Programme Core Systems Workstream are working to improve the integration between Blackboard and RISIS.

REFLECTIONS

The feedback obtained from the teachers indicates that there is a healthy satisfaction surrounding e-submission, it is also positive with regards to marking content but it is divided about how to approach correction and feedback on language items as they can be as particular as the individual but also as the language itself. In this sense written adjustments and examples need to be inserted in the text, an option that seems to be faster in paper rather than electronically but in the very specific context of inserting grammatical symbols in a text in language teaching there might be some additional thinking. The EMA Team are looking at requirements surrounding scientific, mathematical and grammatical type notations within the University and possible ways
forward.

The general consensus is that at present out of the two options Tii is a better option for language projects than inline marking. In order to enable that transition we need to look into the set of rubrics we are using and adopt sets of QuickMarks applicable to all languages, with perhaps addition of specific sets for non-Latin language scripts.

FOLLOW UP

There will be a small working group set up to revise QuickMarks across all languages. This working group will also look into the rubrics and how can we best customise them for our assessment purposes and in line with CEFR (Common European Framework of Reference for languages)

LINKS

EMA Project Reading Resources

http://www.reading.ac.uk/internal/ema/ema-resources.aspx

Common European Framework of Reference for Languages (CEFR)

https://www.coe.int/en/web/common-european-framework-reference- languages/?

Using wikis for assessed group work in new history modules

By laurenmccann Comment Closed

Shirin Irvine – TEL Adviser, CQSD

Image of Shirin Irvine

Overview

For the academic year 2015/16, the Department of History offered a brand-new Part 1 programme as part of the History Project. This resulted in the development of three new core modules.

Dr Mara Oliva transformed common practice by using technology to carry out full electronic assessment for her module. This project included multiple aspects of digital pedagogy, using Blackboard to perform engaging assessment.  This was achieved through innovative and effective use of Blackboard Groups in combination with Blackboard Wikis and Turnitin Assignments, in addition to the Grade Centre for administering students’ marks.

What is a wiki?

A wiki is a collaborative tool that allows students to work as a group on one project and write shared content in the form of a website. They can create a series of web pages that can include images, web links and videos, collectively responding to a theme.

Dr Mara Oliva – Lecturer in Modern American History (20th century)

Image of Mara Olive

Mara explains how she used the wiki tool within Blackboard as a new tool for summative assessment.

The Culture Wiki

Journeys through History 2 aims to introduce students to major historical ideas, concepts, beliefs and knowledge systems, and to show how these are exemplified in material culture, with reference to artefacts, buildings, paintings and other works of art, literature and media.

We wanted the assessment tools we chose to reflect the cultural and visual elements of the module. Therefore we decided to use a group wiki of 2,000 words (50% of the module mark), which we called the Culture Wiki, and an individual 2,000-word essay on one of the historical concepts.

The Culture Wiki allowed students to create and contribute to several web pages of course-related material. They were expected to display their research, analytical and communication skills by building a website meant for public consumption. In small groups, students created their wikis based on a theme discussed during lectures. Lecturers provided themes in the module handbook and on Blackboard.

Our aims for using this form of assessment were to teach students the importance of teamwork and how to write in a concise and accessible way in order to develop an understanding of public history, which offers many employability opportunities to history graduates.

Impact – great results! 

Overall, the exercise was very successful! According to the feedback, both students and staff enjoyed working on the Culture Wiki. Students said it gave them a chance to look at history from a different angle and realise how many flexible and transferable skills they can gain through studying history.

We then decided to take this a step further and extend full electronic assessment to the individual assay, using Turnitin Assignments. This was received very enthusiastically by the students, who appreciated the immediacy and flexible, 24/7 access technology can offer.

The project, however, would have never taken off without the invaluable support of the TEL team, in particular Shirin Irvine, Lauren McCann and Maria Papaefthimiou. With their help we arranged training and guidance for the department staff on creating and assessing wikis, using Turnitin for e-assessment, and using the Grade Centre.

To support students, we provided a separate handbook with “how to build a wiki” guidelines, which was uploaded on Blackboard. I then dedicated part of the first lecture to introducing the exercise and answering the questions. Overall, students did not need much support and were very quick at learning – their questions were mainly content related.

We are very pleased with the outcome of the project, so we have decided to continue for the foreseeable future!

Using quickmarks and rubrics in online assessment – Catherine Foley

By laurenmccann Comment Closed

Catherine Foley is a lecturer in Primary Maths Education in the Institute of Education. She is Director of the Primary School Direct programme which trains people to be teachers whilst they are working in schools.

Image of Catherine Foley

OBJECTIVES

Catherine describes her experience of using the Feedback Studio to move from Word-based marking an assignment to full use of Grademark.

CONTEXT

Catherine Foley is a lecturer in Primary Maths Education in the Institute of Education. She is Director of the Primary School Direct programme which trains people to be teachers whilst they are working in schools. Her experience of electronic marking relates primarily to a 20 credit postgraduate module which is part of this programme, developing the reflective practice and critical thinking of the trainees. The module is assessed through one piece of written work which is assessed formatively and summatively and is taken by approximately 80 students each year.

IMPLEMENTATION

Up until the current academic year, although students would submit their work through Turnitin (for both formative and summative attempts), they would receive feedback in the form of underlined grading sheets and text-based comments which would be completed for each student and uploaded to be released to them via Grade Centre. As with other IoE programmes, all submission, grading and feedback for this assessment is now carried out electronically.

This year, we decided to use the full electronic feedback option for both assessments since the first formative experience would give students (and staff) the chance to get used to the system. We
developed our own rubric for the assessment. For the formative assessment, we decided not to use quickmarks but just to focus on becoming familiar with using the rubric. For the summative
assessment, both a rubric and quickmarks were used: the quickmark set is the same as that used for other initial teacher training programmes.

In my own marking, I found it helpful, when getting started, to open out the full rubric in a grid from the sidebar in the feedback studio. After a while, I was clear what the different statements meant and so could use the sliders more confidently.

IMPACT

  • Speed of marking. Although marking has not been any quicker so far overall, it is likely that this will speed up as the administrative problems are ironed out and we get to know the
    system. Not having to save individual files saves a lot of time which can be spent on quality feedback.
  • Ease of moderation. Because all the assessment and feedback is in the same place, it is much more straightforward and a module convenor is easily able to quality-assure the marking
    that is taking place.
  • Curriculum review opportunity. Developing our own rubric for the assessment encouraged us to review what we had been doing. It made use stop and examine our taken-for-granted practice.
  • Student ownership of feedback. We had a workshop on developing academic writing and it was interesting to see all the students with their laptops open, looking at very specific
    pieces of contextualised feedback received online for their first assignment.
  • Using rubric reports for bespoke study advice sessions. We used the function in Turnitin to generate a report on how well students had achieved as a cohort in relation to the different
    rubric themes. We sent the report to one of the study advisers who was then able to use this to pinpoint areas to focus upon in helping students work towards their next assignment.

REFLECTIONS

Many of the challenges we experienced were due to the fact that the assessment is marked by five different members of staff:

  • When we were using Word-based documents for feedback, we could shape and guide the feedback which tutors were giving more easily (for example with a writing frame). In the feedback studio, the text comment box presents markers with a blank space so it has been harder to ensure a common approach across markers. We therefore agreed a common structure for feedback in this box.
  • The marking team had differing levels of experience with electronic marking. Because the quickmark set had to be uploaded by each marker to their Blackboard account and not all markers were present on campus at the same time, this was a logistical challenge.
  • With the options for quickmarks, rubric statements and open text comments, it would be easy for markers to over-assess each piece of work. Our agreement was that, since students were getting extra feedback in terms of the first two kinds of feedback, the final text comments should be brief and simply recognise specific areas of success then pinpoint areas for
    development.
  • Limitations in functionality of the feedback studio. Some markers liked to be able to use Word to check the number of times a student has used a particular phrase or look at the
    consistency between citations and references: you can’t currently move around the document so easily (unless you download it). Some warning or confirmation messages from
    the system (for example when moving onto the next piece of work) would make it still more user-friendly. With several people involved in marking an assignment, it is easy for markers
    to accidentally change each other’s grades – it would be helpful if grades and comments could be ‘locked’ in some way. Are different levels of access possible, so that external examiners can see the feedback studio but without being able to change feedback?
  • There are still issues (mostly to do with administrative protocols) to iron out. The IoE is currently reviewing its moderation processes and determining the extent to which
    students know they have been included. Programme directors are working with their admin teams to determine exactly how
    academics will be informed when an ECF assignment has been submitted.

Developing innovative teaching: The importance of reflective practice

By peteandrews Comment Closed

Dr Allán Laville, School of Psychology and Clinical Language Sciences

a.laville@reading.ac.uk

Overview

In the training of Psychological Wellbeing Practitioners (PWPs), it is crucial to support students in the development of their reflective thinking and writing skills. Therefore, I implemented the Self-Practice/Self-Reflection (SP/SR; Bennett-Levy, 2001) approach into our PWP training programmes. Impact was measured by asking students to complete questionnaires on their experience of SP/SR and the results informed my PGCAP research project.

Objectives

  • To improve the level of support and guidance for reflective thinking and writing within the programme.
  • To support students to review their current clinical practice and to create action points in order to develop their practice based on the use of SP/SR.

Context

As part of the BPS PWP national curriculum, students are required to complete summative pieces of reflective writing based on their clinical practice. This is contained within the Evidence-based low-intensity treatment of common mental health problems module (PY3TRT1 and PYMTRT) in both PWP training programmes at the University.

Implementation

The first step was to review the current literature on SP/SR to see how we could implement this approach in the training of low-intensity Cognitive Behavioural Therapy (LICBT) for PWPs. Previous use of this approach was for Cognitive Behavioural Psychotherapists who completed training over two years, whereas PWPs train over 9 months, and so there is significantly less time for implementation.

Based on the length of the training programme, we agreed on including 6 SP/SR activities. We then explored the three different components to this. The first component, which was already in the programme, is for students to receive teaching on a LICBT intervention, such as Behavioural Activation i.e. supporting patients with depression to increase their amount of routine, necessary, and pleasurable activities. The second component, which was new to the programme, was for students to practice completing the LICBT intervention on themselves to identify what went well and what could have gone better.  The third component, which was also new to the programme, was for students to then blog about their experience via the discussion board feature on Blackboard.

Impact

The student feedback, elicited in the questionnaires, were very positive with comments such as ‘practicing the intervention and then blogging about it really made me see the difficulties that patients might face’ and ‘completing SP/SR really made me review my current practice and see what I can do to improve.  Our experience of including the SP/SR training was presented at the National British Association of Behavioural and Cognitive Psychotherapies Conference at the University of Birmingham. Overall, I think that the activity did meet the objectives however, an unexpected outcome was the need to reduce the number of activities due to student feedback.

Reflections

In relation to the success of the activity, the consultation with other Clinical Educators and Programme Directors (both at Reading and other Universities) enabled us to implement the activity with very little difficulty. Furthermore, within our teaching, we provide an ‘Introduction to SP/SR’ session so students are informed of the evidence-base for this approach, what the requirements are for SP/SR on our programmes, as well as when they will need to complete the activity, and how to post their experiences on Blackboard. This has been commented on as very useful within student feedback.

In relation to better implementation, the earlier versions of the activity included 6 SP/SR tasks however, students commented that whilst SP/SR is very useful, they found completing 6 tasks too much when considering the rest of their workload. Therefore, we reduced the number to 4 SP/SR tasks, which has been working well.

Follow up

In the previous two years, we have developed our practice by providing students with the opportunity to receive written staff feedback on one of their SP/SR blogs. The student feedback in regard to this has been very positive and we have seen an improvement in the reflective writing skills of our students in their summative reflective assignments.

Links

https://www.researchgate.net/publication/231747637_The_value_of_self-practice_of_cognitive_therapy_techniques_and_self-reflection_in_the_training_of_cognitive_therapists

Communicating Ancient Sport

By peteandrews Comment Closed

Barbara Goff     School of Humanities     b.e.goff@reading.ac.uk

Overview

In my Part 2 module ‘Ancient Sport’ I offer students a choice between a traditional essay and an ‘outreach project’, which requires them to communicate an aspect of ancient sport to a non-academic audience, perhaps for schools or for the general public.

Objectives

  • To develop students’ communication skills in an attractive way
  • To diversify assessment in a relevant way (I first taught the module in an Olympics year)
  • To foster students’ sense of their own employability by developing a range of skills.
  • To engage students more fully in an assessment that draws on creativity and imagination.
  • I also hoped that students would have fun with the assessment, which they definitely have done.

Context

The module ‘Ancient Sport’ investigates Ancient Greek and Roman sporting activities with a focus on relating these to concepts of gender, desire, citizen identity, political power, and empire.  The histories of art, architecture and engineering are also important.  Amy Smith, the Curator of the Ure Museum, suggested the outreach project when I started planning the new module.  I consulted with other colleagues in Study Advice, and the then Teaching and Learning Dean, in order to design the assessment effectively.   I monitored the success of the outreach project via evaluations and discussion with students as well as via assessing the work itself, and recursively amended rubric and feedback sheet in order to communicate what students needed to do, and to guide their practice by clarifying criteria.

Implementation

Each outreach project has to be accompanied by a commentary on a relevant ancient text, a bibliography of secondary literature, and a reflective essay.   I start talking to the students about the assessment choices at the beginning of term.  Towards the end of term, students discuss their chosen project with me and get some feedback on how it is developing.   The module includes a workshop on outreach communications, run by Kim Shahabudin, a colleague from Study Advice, and we share with the students the specific rubric and feedback form which I have developed to address the various elements of the assessment.  We also situate the assessment in the context of employability, pointing towards the importance of being able to reflect on one’s own work, as well as stressing research and communication skills.

Impact

The outreach project assessment has been very successful, with many evaluations picking it out as a strength of the module.  In informal conversations, it has become clear that students understand the link with employability, e.g. with their ambitions towards teaching, journalism or museum work. Over the years students have produced work such as videos both educational and entertaining, board games, museum trails, short stories, comics and magazines.  I have been impressed by the effort, imagination, humour and creativity that students have put into their work, and also by their ability to reflect on their achievements, any limitations of their projects, and the decisions that they had to make along the way.  I have been particularly gratified when students who have struggled with the traditional essay, for a variety of reasons, have found an assessment activity in which they can really shine.  We have used several projects on Open Days and in workshops for local schools.

Reflections

What has mainly contributed to the success of this activity is simply the effort and commitment of the students, and I am very glad to have elicited such good work.  This activity has also been very well supported by colleagues in Study Advice and in the Ure Museum, for which I am grateful.  The activity has required me to rethink things like assessment criteria and rubrics, which I have found useful overall in my teaching.

Follow up

I find it very productive to approach assessment as a way of fostering employability and a variety of skills.  As Departmental Director of Teaching and Learning I am keen for the Department to continue to extend such opportunities for students to engage with a variety of assessment.  I have given extra publicity to our Independent Project module, which offers an alternative to the dissertation.  Although I shall rest ‘Ancient Sport’ for a while, I shall develop a creative writing assessment in a Part 3 module.  We are going to investigate the transformations of the figure of Helen of Troy, across different literary genres and periods, and students will have the opportunity to produce their own version of Helen, in poetry, short story, script, or other text.  Reflection as well as research will be a significant part of this assessment.

 

From boning a duck to reflecting on utopia in English Literature: Using blogs in teaching, learning and assessment

By peteandrews Comment Closed

Lauren McCann, Centre for Quality Support and Development   l.j.mccann@reading.ac.uk

Chloe Houston, School of Literature and Languages   c.houston@reading.ac.uk

In the 2009 hit film ‘Julie et Julia’, real life American office worker Julie Powell (Amy Adams) spends a year cooking her way through culinary legend Julia Child’s ‘Mastering the Art of French Cooking’ and ‘blogs’ about it. Powell picks up a following and generates a dialogue with her readers who comment on her posts and offer her advice, from how to prepare a lobster to how to bone a duck. Blogs are, of course, more than just about French cooking. There are blogs about all sorts of things. They have never been bigger and have become an increasingly useful tool in education too. In this article, we’ll explore this tool and find out how it’s been used for summative assessment on the BA in English Literature.

What is a blog?

A blog – short for web log – is a personal online journal that can include various media and is intended for sharing with others, like an open web-based diary. Most blogs have some kind of commenting system so that people can share their thoughts on entries. Blogs encourage students to clearly express their ideas and engage in social learning.

In Blackboard Learn, instructors can create and manage blogs from within a course. Enrolled users can then view and create entries and comments in them. They can be used for various purposes and as a tool for both formative and summative assessment, providing an alternative to more traditional methods.

Case study: Using blogs in English Literature at the University of Reading

Dr Chloe Houston has used the blog tool this year in a new third year module, ‘Utopia: The Ideal Society in English and American Literature’. Chloe was interested in diversifying the assessment methods experienced by her students and in moving away from the conventional essay. Aware that after graduation, students could be expected to write in a variety of media for a range of audiences, she was keen to give them the opportunity to write in a different format and share their ideas with their peers.

In getting ready to use the tool, Chloe did a good deal of preparation which was key to her ultimate success, contacting TEL CQSD for advice and researching academic blogs. She set up a Blackboard blog to be used as 50% of the module’s assessment in which students were expected to post entries during the term. An inexperienced blogger, she made use of a post-graduate student with relevant experience to help prepare the students and provided support materials. Mid-term evaluation suggested students were enjoying working in this way and end of module evaluations confirmed this, with the additional benefit that Chloe found the assessments more varied and interesting to mark! When asked if she had any advice for other staff thinking about trying out blogging, she exclaimed, “Immerse yourself in the blogging culture and just do it!”

Student Josie Palmer was one of a number who reported positively on her experiences of using the blog tool: “With students having grown up around technology… I feel a blog is a positive step forward in the way work is assessed. It’s easy to access and manage, it’s interactive, as you can read other student’s work and comment on what they have written… This differs greatly from essays… [The blog] gives students the opportunity to upload work and receive feedback more frequently… We are given more of an opportunity to explore ideas in different ways, with a simple format, as opposed to putting all the work collected over a term into one final essay. I think that as a format of assessment the blog works brilliantly!”

In this short video, Chloe discusses her use of the blog in her module:

What next?

If this article has inspired you to find out more about using the blog tool in your own teaching, please see Blackboard’s Support for Staff tab and/or contact the TEL CQSD team for advice. You can also subscribe to the TEL team’s very own blog at http://blogs.reading.ac.uk/tel/

Bibliography

Salmon, G, 2013. E-tivities: The Key to Active Online Learning . 2nd ed. Abingdon: Routledge.

Downes, S, 2004. Educational Blogging . EDUCAUSE Review , [Online]. 39 (Number 5), 14-26. Available at: http://www.downes.ca/post/40939 [Accessed 01 March 2016].

Hammond , M , (2006). Blogging within Formal and Informal Learning Contexts: Where are the Opportunities and Constraints?’  In Networked Learning. University of Warwick , 2006.

Julie et Julia , 2009. [DVD] Nora Ephron, USA: Sony Pictures.

Sharing the ‘secrets’: Involving students in the use (and design?) of marking schemes

By peteandrews Comment Closed

Rita Balestrini, School of Literature and Languages, r.balestrini@reading.ac.uk

Overview

Between 2016 and 2018, I led a project aiming to enhance the process of assessing foreign language skills in the Department of Modern Languages and European Studies (MLES). The project was supported by the Teaching and Learning Development Fund. Its scope involved two levels of intervention: a pilot within one Part I language module (Beginners Italian Language) and other activities involving colleagues in all language sections and students from each year of study. The project enabled the start of a bank of exemplars for the assessment of a Part I language module; promoted discussion on marking and marking schemes within the department; and made possible a teacher-learner collaborative appraisal of rubrics.

Objectives

  • To enhance Beginners Italian Language students’ understanding of rubrics and their assessment literacy
  • To increase their engagement with the assessment process and their uptake of feedback
  • To engage MLES students as agents of change in the assessment culture of the department
  • To stimulate innovation in the design of rubrics within the MLES Language Team and contribute to develop a shared discourse on assessment criteria and standards informed by the scholarship of assessment

Context

In recent years, there has been an increasing demand to articulate explicitly the standards of assessment and to make them transparent in marking schemes in the form of rubrics, especially in Foreign Languages. It is widely held that the use of rubrics increases the reliability of assessment and fosters autonomy and self-regulation in students. However, it is not uncommon that students do not engage with the feedback that rubrics are supposed to provide. In 2016, the language team of the Department of Modern Languages and European Studies started to work at the standardisation and enhancement of marking schemes used to assess language skills. The aim of this multi-layered project was to make a positive contribution to this process and to pilot a series of activities for the enhancement of foreign language assessment.

Implementation

  • Review of research literature and scholarly articles on the use of standard-based assessment, assessment rubrics, and students-derived marking criteria.
  • Presentation on some of the issues emerged from the review at a School T&L Away Day on assessment attended by the MLES language team (April 2017) and at a meeting of the Language Teaching Community of Practice (November 2017).
  • Organisation of a ‘professional conversation’ on language assessment, evaluation and marking schemes as a peer review activity in the School of Literature and Languages (SLL). The meeting was attended by colleagues from MLES and CQSD (February 2018).
  • 2016-17 – Two groups of students on the Beginners Italian Language module were asked for permission to use exemplars of their written and oral work for pedagogic practice and research. Ten students gave their informed consent.
  • Collection of written and oral work, double-marked by a colleague teaching one of the groups.
  • 2017-2018 – Organization of two two-hour workshops on assessment for a new cohort of students. Aim: To clarify the link between marking criteria, learning outcomes and definitions of standards of achievement of the module. An anonymised selection of the exemplars collected the previous year was used a) ‘to show’ the quality of the standards described in the marking schemes and b) for marking exercises.
  • 2017 – Organisation of three focus groups with students – one for each year of study – to gain insights into their perspectives on the assessment process and understanding of marking criteria. The discussions were recorded and fully transcribed.
  • The transcriptions were analysed by using a discourse analysis framework.
  • Some issues emerged from the analysis: atomistic approach of rubrics; vagueness of the standards; subjectivity of the evaluation; problematic measuring of different aspects of achievement; rating scales anchoring (for a more comprehensive account of the focus groups see the Engage in T&L Blog post Involving students in the appraisal of rubrics for performance-based in Foreign Languages).
  • Developed, in collaboration with three students from the focus groups, a questionnaire on the use of rubrics. The questionnaire was intended to gather future students’ views on marking schemes and their use.

Impact

This multi-layered project contributed to enhance the process of assessing foreign language skills in MLES in different ways.

  • The collection of exemplars for the Beginners Italian Language module proved to be a useful resource that can also be used with future cohorts. The workshops were not attended by all students, but those who did attend engaged in the activities proposed and asked several interesting questions about the standards of achievement described in the marking schemes (e.g. grade definitions; use of terms and phrases).
  • The systematic analysis of the focus groups provided valuable insights into students’ disengagement with marking schemes. It also brought to light some issues that would need to be addressed before designing new rubrics.
  • The literature review provided research and critical perspectives on marking schemes as a tool of evaluation and a tool for learning. It suggested new ways of thinking about marking and rubrics and provided a scholarly basis for potential wider projects. The discussion it stimulated, however different the opinions, was an important starting point for the development of a shared discourse on assessment.

Reflections

The fuzziness of marking students’ complex performance cannot be overcome by simply linking numerical marks to qualitative standard descriptors. As mentioned in a HEA document, even the most detailed rubrics cannot catch all the aspects of ‘quality’ (HEA, 2012) and standards can be better communicated by discussing exemplars. There is also an issue with fixing the boundaries between grades on a linear scale (Sadler, 2013) and the fact that, as Race warns, the dialogue between learners and assessors (Race, HEA) can easily be broken down by the evaluative terms typically used to pin down different standards of achievement. Despite all these pitfalls, in the current HE context, rubrics, if constructed thoughtfully and involving all stakeholders, can benefit learning and teaching.

By offering opportunities to discuss criteria and standards with students, rubrics can help to build a common understanding of how marks are assigned and so foster students’ literacy, especially if their use is supported by relevant exemplars.

The belief that rubrics need to be standardised across modules, levels and years of study makes designing rubrics particularly difficult for ‘foreign languages’. Cultural changes require time and the involvement of all stakeholders, especially where the changes concern key issues that are difficult to address without a shared view on language, language learning and assessment. A thorough discussion of rubrics can provide chances to share ideas on marking, assessment and language development not only between students and staff but also within a team of assessors.

I have tried to engage students in the appraisal of rubrics and to avoid a market research approach to focus groups. It is clear that, if we are committed to make any assessment experience a learning experience and to avoid the potential uneasiness that rubrics can cause students, we need to explore new ways of defining the standards of achievement in foreign languages. Establishing pedagogical partnerships with students seems a good way to start.

Follow up

I will encourage a differentiation of rubrics based on level of language proficiency and a collection of exemplars for other language modules. The natural follow up to this project would be to continue enhancing the rubrics used for evaluation and feedback in languages in the light of the analysis of the focus group discussions and the review of the literature on assessment, ideally with the collaboration of students. Possible connections between the marking schemes used to assess language modules and cultural modules will be explored.

References

HEA, 2012. A Marked Improvement. Transforming assessment in HE. York: Higher Education Academy.

Race, P. Using feedback to help students to learn [online] Available at https://www.heacademy.ac.uk/knowledge-hub/using-feedback-help-students-learn   [accessed on 15/8/2018]

Sadler, D. R. 2013. The futility of attempting to codify academic achievement standards. Higher Education 67 (3): 273-288.

 

Links to related posts

‘How did I do?’ Finding new ways to describe the standards of foreign language performance. A follow-up project on the redesign of two marking schemes (DLC)

Working in partnership with our lecturers to redesign language marking schemes 

Involving students in the appraisal of rubrics for performance-based assessment in Foreign Languages By Dott. Rita Balestrini

The EMA Symposium: Sharing Knowledge, Good Practice, and Cake

By edwardwhite Comment Closed

Dr Madeleine Davies and Dr Emma Mayhew

On Tuesday 22nd May 2018, over 150 colleagues from across the university gathered in the Meadow Suite to hear a series of presentations and to engage in conversations about the work that the EMA Programme has been doing to prepare for roll-out of online assessment and feedback. Colleagues from the majority of Schools and units across the university were represented and staff from academic Schools, Support Centres, and CQSD shared experience and good practice. An active Twitter feed provided live commentary on the presentations and activities (120 tweets and re-tweets) and reflected an atmosphere that was informative and wholly positive.

The event was organised by Dr Emma Mayhew (EMA Academic Director) and Dr Madeleine Davies (EMA Academic Partner) and was introduced by Professor Gavin Brooks, the Programme’s sponsor. The Vice-Chancellor attended for the first hour of the event which was supported throughout by colleagues from TEL. As well as seven panels involving presentations given by 21 colleagues, the day of activities included a Q&A session, a roundtable, a Menti quiz, a talk about Learning Analytics, and a demonstration of online marking for beginners: the result was a permanent movement between learning and dialogue. Piles of EMA cup-cakes and biscuits retained energy levels throughout the day.

The Symposium was designed to offer events for colleagues new to online assessment as well as to those with more experience. Colleagues attending the Symposium provided information about their level of technological confidence: of 40 surveyed participants, 5% rated themselves as having low levels of confidence with technology, 45% rated themselves as having average levels of confidence, and 48% rated themselves as having advanced confidence. The Symposium offered talks and events for all levels.

Presentations from the Early Adopter Schools discussed the process of change and offered advice about successfully implementing online assessment in departments; a panel was also convened where ‘nervous adopters’ who had already transitioned to online marking spoke about the training and tactics that had worked for them. For more experienced users the Symposium offered presentations on the use of rubrics and QuickMarks, explaining the several ‘hidden’ benefits of online marking and demonstrating the potential of previously unidentified buttons.

Throughout the day, the emphasis remained on student experience and marking quality: several presentations commented on ways in which online assessment could enhance feedback and release a range of marking options that had not been available previously. At the same time, presentations emphasised that ‘good marking is still good marking’, as Rob Hosfield stated: the change to a new delivery model does not alter the fact that it is the usability of feedback that matters most in relation to teaching and learning.

Participants also heard presentations from the Support Centres and their experience of change: Luisa Ciampi’s presentation explained how online assessment benefits our colleagues in the Support Centres and Marguerite Gascoine spoke of the process of change. Advice about managing the potential impact of increased screen-time was given in a session led by Dr Eileen Hyder, and a presentation by Dr Calvin Smith demonstrated how new screens in RISIS support the move towards the Academic Tutoring System (Calvin was the winner of the ‘Best Title’ Menti Prize for ‘Spotting Crisis in RISIS’).

The feedback that was collected at the end of the afternoon was overwhelmingly positive:

39/40 colleagues rated the event as good or excellent and 38/40 said that the event would impact positively on their teaching and learning provision. Colleagues commented positively on the benefit of hearing the experience of a wide range of colleagues, the broad range of topics covered, and the advice on good practice. The feedback also noted how useful it had been to bring academic and professional staff together to discuss experiences, and several colleagues mentioned how they had enjoyed putting faces to the names of Support Centre colleagues. Also praised was the emphasis on how new data can be used, sessions on the practical use of marking tools, and hearing the experiences of Early Adopter schools. There was a great deal of positive feedback on a talk given by Dr Alex Knox on Learning Analytics.

As the organisers of the Symposium, we were delighted by colleagues’ positivity, collegiality, and eagerness to share online assessment experience. We hope that our colleagues will be able to join us again in September when the Programme is hosting a conference in association with the the Advance HE’s (formerly HEA) Assessment and Feedback Community of Practice. In the meantime, screencasts of key presentations will be posted on the EMA website together with a collection of helpful links and information.

Thank you to all our colleagues who attended the Symposium to share experience and to listen to that of others. We were overwhelmed by the generosity of staff who contributed their time and expertise to the event, and by the goodwill of our colleagues who attended.

Engaging students in the design of assessment criteria

By peteandrews Comment Closed

Dr Maria Kambouri-Danos, Institute of Education                                                                                                                m.kambouridanos@reading.ac.uk                                                                                                                                                                                            Year of activity 2017/18

Overview

I recently led a group of colleagues while working in partnership with students to develop a new module in BA in Children’s Development and Learning (BACDL) delivered at the Institute of Education (IoE). This approach to working in partnership with students is a core part of the project’s aim and the work described here has been part of a Partnerships in Learning & Teaching (PLanT) project.

Objective

The team’s aim was to develop and finalise a new module for BACDL in close partnership with the students. The new module will replace two existing modules (starting from 2018-19), aiming to reduce overall assessment (programme level), a need identified during a Curriculum Review Exercise. The objective was to adopt an inclusive approach to student engagement when finalising the new module, aiming to:

  • Go beyond feedback and engage students by listening to the ‘student voice’
  • Co-develop effective and student-friendly assessable outcomes
  • Identify opportunities for ‘assessment for learning’
  • Think about constructive alignment within the module
  • Encourage the development of student-staff partnerships

Context

To accomplish the above, I brought together five academics and six students (BACDL as well as Foundation Degree (FDCDL) students). Most of the students on this programme are mature students (i.e. with dependants) who are working full time while attending University (1 day/week). To encourage students from this ‘hard to reach group’ to engage with the activity, we secured funding through the Partnerships in Learning & Teaching scheme, which enabled the engagement of a more diverse group (Trowler, 2010).

Implementation

The team participated in four partnership workshops, during which staff and students engaged in activities and discussions that helped to develop and finalise the new module. During the first workshop, we discussed the aims of the collaborative work and went through the module’s summary, aims and assessable outcomes. We looked at the two pre-existing modules and explored merging them into a new module, maintaining key content and elements of quality. During the second workshop, we explored chapter two from the book ‘Developing Effective Assessment in Higher Education: a practical guide’ (Bloxham & Boyd, 2007), which guided the discussions around developing the assessment design for the new module.

During the third workshop, we discussed aspects of summative and formative tasks and finalised the assessment design (Knight, 2012). We then shared the new module description with the whole BACDL cohort and requested feedback, which enabled us to get other students’ views, ensuring a diverse contribution of views and ideas (Kuh, 2007). During the last workshop, with support from the Centre of Quality Support and Development (CQSD) team, we implemented a game format workshop and created a visual ‘storyboard’, outlining the type and sequence of learning activities required to meet the module’s learning outcomes (ABC-workshop http://blogs.ucl.ac.uk/abc-ld/home/abc-workshop-resources/). This helped to identify and evaluate how the new module links with the rest of the modules, while it also helped to think about practical aspects of delivering the module and ways to better support the students (e.g. through a virtual learning environment).

Photos from staff-student partnership workshops

Impact

The close collaboration within the team ensured that the student voice was heard and taken into account while developing the new module. The partnership workshops provided the time to think collaboratively about constructive alignment and ensure that the new module’s assessment enables students to learn. It also ensured that the module’s assessable outcomes are clearly defined using student-friendly language.

A pre- and post-workshop survey was used to evaluate the impact of this work. The survey measured the degree to which students appreciate the importance of providing feedback, participate in activities related to curriculum review/design, feel part of a staff-student community and feel included in developing their programme. The survey results indicate an increase in relation to all of the above, demonstrating the positive impact of activities like this on student experience. All students agreed that it has been beneficial to take part in this collaborative work, mentioning that being engaged in the process, either directly (attending the workshops) or indirectly (providing feedback) helped them to develop a sense of belonging and feel part of the community of staff and students working together (Trowle, 2010; Kuh, 2005;2007).

Reflections

This project supported the successful development of the new module, from which future students will benefit (Kuh, 2005). The work that the team produced has also informed the work of other groups within the IoE. At the institutional level, this work has supported the development of the CQSD ‘Student Engagement’ projects. All the above were achieved because of close collaboration, and could not have been done by a group of individuals working on their own (Wheatley, 2010). Because of that, our team was awarded the University Collaborative Awards for Outstanding Contributions to Teaching and Learning.

References

Bloxham, S. & Boyd, P. (2007). Developing effective assessment in higher education: a practical guide. Maidenhead: Open University Press.

Knight, P. (Ed.). (2012). Assessment for learning in higher education. Routledge.

Kuh, G.D. (2005). Putting Student Engagement Results to Use: Lessons from the Field, Assessment Update. 17(1), 12–1.

Kuh, G.D. (2007). How to Help Students Achieve, Chronicle of Higher Education. 53(41), 12–13.

Trowler, V. (2010). Student engagement literature review. The Higher Education Academy.

Wheatley, M. (2010). Finding our Way: Leadership for an Uncertain Time. San Francisco: Berrett-Koehler