DMLES – T&L Exchange

‘How did I do?’ Finding new ways to describe the standards of foreign language performance. A follow-up project on the redesign of two marking schemes (DLC)

20 November 20199 March 2020 By edwardwhite Comment Closed

Rita Balestrini and Elisabeth Koenigshofer, School of Literature and Languages, r.balestrini@reading; e.koenigshofer@reading.ac.uk

Overview

Working in collaboration with two Final Year students, we designed two ‘flexible’, ‘minimalist’ rubric templates usable and adaptable across different languages and levels, to provide a basis for the creation of level specific, and potentially task specific, marking schemes where sub-dimensions can be added to the main dimensions. The two marking templates are being piloted this year in the DLC. The project will feature in this year’s TEF submission.

Objectives

Design, in partnership with two students, rubric templates for the evaluation and feedback of writing tasks and oral presentations in foreign languages which:

were adaptable across languages and levels of proficiency
provided a more inclusive and engaging form of feedback
responded to the analysis of student focus group discussions carried out for a previous TLDF-funded project

Context

As a follow-up to a teacher-learner collaborative appraisal of rubrics used in MLES, now DLC, we designed two marking templates in partnership with two Final Year students, who had participated in the focus groups from a previous project and were employed through Campus Jobs. ‘Acknowledgement of effort’, ‘encouragement’, ‘use of non-evaluative language’, ‘need for and, at the same time, distrust of, objective marking’ were recurrent themes that had emerged from the analysis of the focus group discussions and clearly appeared to cause anxiety for students.

Implementation

We organised a preliminary session to discuss these findings with the two student partners. We suggested some articles about ‘complexity theory’ as applied to second language learning, (Kramsch, 2012; Larsen-Freeman, 2012; 2015a; 2015b; 2017) with the aim of making our theoretical perspective explicit and transparent to them. A second meeting was devoted to planning collaboratively the structure of two marking schemes for writing and presentations. The two students worked independently to produce examples of standard descriptors which avoided the use of evaluative language and emphasised achievement rather than shortcomings. At a third meeting they presented and discussed their proposals with us. At the last meetings, we continued working to finalise the templates and the two visual learning charts they had suggested. Finally, the two students wrote a blog post to recount their experience of this collaborative work.

The two students appreciated our theoretical approach, felt that it was in tune with their own point of view and that it could support the enhancement of the assessment and marking process. They also found resources on their own, which they shared with us – including rubrics from other universities. They made valuable suggestions, gave us feedback on our ideas and helped us to find alternative terms when we were struggling to avoid the use of non-evaluative language for our descriptors. They also suggested making use of some visual elements in the marking and feedback schemes in order to increase immediateness and effectiveness.

Impact

The two marking templates are being piloted this year in the DLC. They were presented to colleagues over four sessions during which the ideas behind their design were explained and discussed. Further internal meetings are planned. These conversations, already begun with the previous TLDF-funded project on assessment and feedback, are contributing to the development of a shared discourse on assessment, which is informed by research and scholarship. The two templates have been designed in partnership with students to ensure accessibility and engagement with the assessment and feedback process. This is regarded as an outstanding practice in the ‘Assessment and feedback benchmarking tool’ produced by the National Union of Students and is likely to feature positively in this year’s TEF submission.

Reflections

Rubrics have become mainstream, especially within certain university subjects like Foreign Languages. They have been introduced to ensure accountability and transparency in marking practices, but they have also created new problems of their own by promoting a false sense of objectivity in marking and grading. The openness and unpredictability of complex performance in foreign languages and of the dynamic language learning process itself are not adequately reflected in the detailed descriptors of the marking and feedback schemes commonly used for the objective numerical evaluation of performance-based assessment in foreign languages. As emerged from the analysis of focus group discussions conducted in the department in 2017, the lack of understanding and engagement with the feedback provided by this type of rubrics can generate frustration in students. Working in partnership with them, rather than simply listening to their voices or seeing them as evaluators of their own experience, helped us to design minimalist and flexible marking templates, which make use of sensible and sensitive language, introduce visual elements to increase immediateness and effectiveness, leave a considerable amount of space for assessors to comment on different aspects of an individual performance and provide ‘feeding forward’ feedback. This type of ‘partnership’ can be challenging because it requires remaining open to unexpected outcomes. Whether it can bring about real change depends on how its outcomes are going to interact with the educational ecosystems in which it is embedded.

Follow up

The next stage of the project will involve colleagues in the DLC who will be using the two templates to contribute to the creation of a ‘bank’ of descriptors by sharing the ones they will develop to tailor the templates for specific stages of language development, language objectives, language tasks, or dimensions of student performance. We also intend to encourage colleagues teaching culture modules to consider using the basic structure of the templates to start designing marking schemes for the assessment of student performance in their modules.

Links

An account written by the two students partners involved in the project can be found here:

Working in partnership with our lecturers to redesign language marking schemes

The first stages of this ongoing project to enhance the process of assessing writing and speaking skills in the Department of Languages and Cultures (DLC, previously MLES) are described in the following blog entries:

National Union of Students 2017. The ‘Assessment and feedback benchmarking tool’ is available at:

http://tsep.org.uk/wp-content/uploads/2017/07/Assessment-and-feedback-benchmarking-tool.pdf

References

Bloxham, S. 2013. Building ‘standard’ frameworks. The role of guidance and feedback in supporting the achievement of learners. In S. Merry et al. (eds.) 2013. Reconceptualising feedback in Higher Education. Abingdon: Routledge.

Bloxham, S. and Boyd, P. 2007. Developing effective assessment in Higher Education. A practical guide. Maidenhead: McGraw-Hill International.

Bloxham, S., Boyd, P. and Orr, S. 2011. Mark my words: the role of assessment criteria in UK higher education grading practices. Studies in Higher Education 36 (6): 655-670.

Bloxham, S., den-Outer, B., Hudson J. and Price M. 2016. Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria. Assessment in Higher Education 41 (3): 466-481.

Brooks, V. 2012. Marking as judgement. Research Papers in Education. 27 (1): 63-80.

Gottlieb, D. and Moroye, C. M. 2016. The perceptive imperative: Connoisseurship and the temptation of rubrics. Journal of Curriculum and Pedagogy 13 (2): 104-120.

HEA 2012. A Marked Improvement. Transforming assessment in HE. York: The Higher Education Academy.

Healey, M., Flint, A. and Harrington K. 2014. Engagement through partnership: students as partners in learning and teaching in higher education. York: The Higher Education Academy.

Kramsch, C. 2012. Why is everyone so excited about complexity theory in applied linguistics? Mélanges 33: 9-24.

Larsen-Freeman, D. 2012. The emancipation of the language learner. Studies in Second Language Learning and Teaching. 2(3): 297-309.

Larsen-Freeman, D. 2015a. Saying what we mean: Making a case for ‘language acquisition’ to become ‘language development’. Language Teaching 48 (4): 491-505.

Larsen-Freeman, L. 2015b. Complexity Theory. In VanPatten, B. and Williams, J. (eds.) 2015. Theories in Second Language Acquisition. An Introduction. New York: Routledge: 227-244.

Larsen-Freeman, D. 2017. Just learning. Language Teaching 50 (3): 425-437.

Merry, S., Price, M., Carless, D. and Taras, M. (eds.) 2013. Reconceptualising feedback in Higher Education. Abingdon: Routledge.

O’Donovan, B., Price, M. and Rust, C. 2004. Know what I mean? Enhancing student understanding of assessment standards and criteria. Teaching in Higher Education 9 (3): 325-335.

Price, M. 2005. Assessment standards: the role of communities of practice and the scholarship of assessment. Assessment & Evaluation in Higher Education 30 (3): 215-230.

Sadler, D. R. 2009. Indeterminacy in the use of preset criteria for assessment and grading. Assessment and evaluation in Higher Education 34 (2): 159-179.

Sadler, D. R. 2013. The futility of attempting to codify academic achievement standards. Higher Education 67 (3): 273-288.

Torrance, H. 2007. Assessment as learning? How the use of explicit learning objectives, assessment criteria and feedback in post-secondary education and training can come to dominate learning. Assessment in Education 14 (3): 281-294.

VanPatten & J. Williams (Eds.) 2015. Theories in Second Language Acquisition, 2nd edition. Routledge: 227-244.

Yorke, M. 2011. Summative assessment dealing. Dealing with the ‘Measurement Fallacy’. Studies in Higher Education 36 (3): 251-273.

Working in partnership with our lecturers to redesign language marking schemes

20 November 201917 January 2020 By edwardwhite Comment Closed

Sian Wells (BA French and Italian) and Sophia Acton (BA Italian and Spanish)

During the Spring and Summer term of 2019, we participated in a project within the Department of Modern Languages and European Studies (MLES, now Department of Languages and Cultures) with the aim of redesigning the marking schemes used for oral and written pieces of work. This was a follow-up to a 2016 project by Rita Balestrini, which aimed to improve the process of assessing foreign language skills within the department. As we had both taken part in the focus groups in our second year, we were asked to take part in the follow-up project and work alongside Rita Balestrini and her colleague Elisabeth Koenigshofer to improve the rubrics currently used in the department.

This project was influenced by Complexity Theory in the context of Second Language Development. Before the project started, we as students were not aware of research into Second Language Learning and research into the effectiveness of marking schemes. At the beginning of the project we were provided with reading material to gain a better understanding of the science and theory behind the project. We did find the reading quite difficult initially but it became clearer how second language development is a complex process which involves a number of interacting factors. Diane Larson-Freeman, a leading researcher into Complexity Theory defines the process as ‘a complex nonlinear system’ (1997), by which language learners do not simply master one item and then move on to another. Instead, they acquire the second language at different rates: some grammatical structures will be consolidated early on whilst others will take longer to master, and “mistakes” will still occur at all levels (Larson-Freeman, 1997). Thus, feedback becomes an integral part of the learning process. Effective feedback will eventually help learners to reach a high level of proficiency. There are many aspects involved in the process of Second Language Learning, for example the source language and the target language with their respective components (eg. phonology, morphology, lexicon, syntax, semantics and pragmatics); the amount and type of input, interaction, and of feedback (Larson-Freeman, 1997). This complexity raises the question of how to effectively evaluate Second Language Learning. Some research has found that rubrics are not very useful for measuring and evaluating objectively complex processes like Second Language Development. They do not acknowledge setbacks which are part of the learning process and cannot account for the effort put in by the student. This is a problem which also emerged from the analysis of focus group discussions.

Working with our lecturers was a very positive experience because we were able to contribute ideas to the project. We came up with examples of new descriptors and we hope our collaborative effort has produced new rubrics which will support better the process of language learning in the DLC. Being students, it meant that we could provide a unique insight because we have used the marking schemes throughout our four years of studying in the department. We could use our own experience to improve the experience of future students. We also realised how difficult it is to effectively evaluate the learning of a language from the perspective of a lecturer, as there are so many components that go into one single piece of work and also because language tasks can vary enormously, from writing a recipe to writing an essay. The difficulties of creating a marking scheme, which adequately evaluate the level of a student in all pieces of work, became apparent to us. One of the hardest parts of the project was coming up with new descriptors for each grade boundary. Based on the analysis of focus group discussions and on research on the use of rubrics, descriptors should avoid evaluative language and be encouraging for students, both, for those getting lower grades and needing to improve, and those achieving higher grades and wanting to continue to improve. Firstly, Sophia and I were given time to come up with our own new descriptors which we then shared with Rita and Elisabeth. Our lecturers used our ideas to come up with a proposal of what a new marking scheme would look like and which criteria and descriptors could be used. This was a really good way of working because it allowed equal input from both, us students, and lecturers, and meant that we could fully contribute to the project and have our ideas incorporated into the final marking and feedback schemes.

Overall, it was a really positive experience as we valued the expertise of our lecturers regarding the theory surrounding the project and could share the difficulties in evaluating pieces of work. Our lecturers in turn valued our experience as second-language learners who had used the rubrics as students and could provide ideas of how to improve them. We felt that our ideas and input were really appreciated and we hope to have helped create more effective marking schemes which will improve how students are assessed in the department.

Links to related posts

The report about the the follow-up project can be found here:

‘How did I do?’ Finding new ways to describe the standards of foreign language performance. A follow-up project on the redesign of two marking schemes (DLC)

Involving students in the appraisal of rubrics for performance-based assessment in Foreign Languages

Sharing the ‘secrets’: Involving students in the use (and design?) of marking schemes

11 January 201922 November 2019 By peteandrews Comment Closed

Rita Balestrini, School of Literature and Languages, r.balestrini@reading.ac.uk

Overview

Between 2016 and 2018, I led a project aiming to enhance the process of assessing foreign language skills in the Department of Modern Languages and European Studies (MLES). The project was supported by the Teaching and Learning Development Fund. Its scope involved two levels of intervention: a pilot within one Part I language module (Beginners Italian Language) and other activities involving colleagues in all language sections and students from each year of study. The project enabled the start of a bank of exemplars for the assessment of a Part I language module; promoted discussion on marking and marking schemes within the department; and made possible a teacher-learner collaborative appraisal of rubrics.

Objectives

To enhance Beginners Italian Language students’ understanding of rubrics and their assessment literacy
To increase their engagement with the assessment process and their uptake of feedback
To engage MLES students as agents of change in the assessment culture of the department
To stimulate innovation in the design of rubrics within the MLES Language Team and contribute to develop a shared discourse on assessment criteria and standards informed by the scholarship of assessment

Context

In recent years, there has been an increasing demand to articulate explicitly the standards of assessment and to make them transparent in marking schemes in the form of rubrics, especially in Foreign Languages. It is widely held that the use of rubrics increases the reliability of assessment and fosters autonomy and self-regulation in students. However, it is not uncommon that students do not engage with the feedback that rubrics are supposed to provide. In 2016, the language team of the Department of Modern Languages and European Studies started to work at the standardisation and enhancement of marking schemes used to assess language skills. The aim of this multi-layered project was to make a positive contribution to this process and to pilot a series of activities for the enhancement of foreign language assessment.

Implementation

Review of research literature and scholarly articles on the use of standard-based assessment, assessment rubrics, and students-derived marking criteria.
Presentation on some of the issues emerged from the review at a School T&L Away Day on assessment attended by the MLES language team (April 2017) and at a meeting of the Language Teaching Community of Practice (November 2017).
Organisation of a ‘professional conversation’ on language assessment, evaluation and marking schemes as a peer review activity in the School of Literature and Languages (SLL). The meeting was attended by colleagues from MLES and CQSD (February 2018).
2016-17 – Two groups of students on the Beginners Italian Language module were asked for permission to use exemplars of their written and oral work for pedagogic practice and research. Ten students gave their informed consent.
Collection of written and oral work, double-marked by a colleague teaching one of the groups.
2017-2018 – Organization of two two-hour workshops on assessment for a new cohort of students. Aim: To clarify the link between marking criteria, learning outcomes and definitions of standards of achievement of the module. An anonymised selection of the exemplars collected the previous year was used a) ‘to show’ the quality of the standards described in the marking schemes and b) for marking exercises.
2017 – Organisation of three focus groups with students – one for each year of study – to gain insights into their perspectives on the assessment process and understanding of marking criteria. The discussions were recorded and fully transcribed.
The transcriptions were analysed by using a discourse analysis framework.
Some issues emerged from the analysis: atomistic approach of rubrics; vagueness of the standards; subjectivity of the evaluation; problematic measuring of different aspects of achievement; rating scales anchoring (for a more comprehensive account of the focus groups see the Engage in T&L Blog post Involving students in the appraisal of rubrics for performance-based in Foreign Languages).
Developed, in collaboration with three students from the focus groups, a questionnaire on the use of rubrics. The questionnaire was intended to gather future students’ views on marking schemes and their use.

Impact

This multi-layered project contributed to enhance the process of assessing foreign language skills in MLES in different ways.

The collection of exemplars for the Beginners Italian Language module proved to be a useful resource that can also be used with future cohorts. The workshops were not attended by all students, but those who did attend engaged in the activities proposed and asked several interesting questions about the standards of achievement described in the marking schemes (e.g. grade definitions; use of terms and phrases).
The systematic analysis of the focus groups provided valuable insights into students’ disengagement with marking schemes. It also brought to light some issues that would need to be addressed before designing new rubrics.
The literature review provided research and critical perspectives on marking schemes as a tool of evaluation and a tool for learning. It suggested new ways of thinking about marking and rubrics and provided a scholarly basis for potential wider projects. The discussion it stimulated, however different the opinions, was an important starting point for the development of a shared discourse on assessment.

Reflections

The fuzziness of marking students’ complex performance cannot be overcome by simply linking numerical marks to qualitative standard descriptors. As mentioned in a HEA document, even the most detailed rubrics cannot catch all the aspects of ‘quality’ (HEA, 2012) and standards can be better communicated by discussing exemplars. There is also an issue with fixing the boundaries between grades on a linear scale (Sadler, 2013) and the fact that, as Race warns, the dialogue between learners and assessors (Race, HEA) can easily be broken down by the evaluative terms typically used to pin down different standards of achievement. Despite all these pitfalls, in the current HE context, rubrics, if constructed thoughtfully and involving all stakeholders, can benefit learning and teaching.

By offering opportunities to discuss criteria and standards with students, rubrics can help to build a common understanding of how marks are assigned and so foster students’ literacy, especially if their use is supported by relevant exemplars.

The belief that rubrics need to be standardised across modules, levels and years of study makes designing rubrics particularly difficult for ‘foreign languages’. Cultural changes require time and the involvement of all stakeholders, especially where the changes concern key issues that are difficult to address without a shared view on language, language learning and assessment. A thorough discussion of rubrics can provide chances to share ideas on marking, assessment and language development not only between students and staff but also within a team of assessors.

I have tried to engage students in the appraisal of rubrics and to avoid a market research approach to focus groups. It is clear that, if we are committed to make any assessment experience a learning experience and to avoid the potential uneasiness that rubrics can cause students, we need to explore new ways of defining the standards of achievement in foreign languages. Establishing pedagogical partnerships with students seems a good way to start.

Follow up

I will encourage a differentiation of rubrics based on level of language proficiency and a collection of exemplars for other language modules. The natural follow up to this project would be to continue enhancing the rubrics used for evaluation and feedback in languages in the light of the analysis of the focus group discussions and the review of the literature on assessment, ideally with the collaboration of students. Possible connections between the marking schemes used to assess language modules and cultural modules will be explored.

References

HEA, 2012. A Marked Improvement. Transforming assessment in HE. York: Higher Education Academy.

Race, P. Using feedback to help students to learn [online] Available at https://www.heacademy.ac.uk/knowledge-hub/using-feedback-help-students-learn [accessed on 15/8/2018]

Sadler, D. R. 2013. The futility of attempting to codify academic achievement standards. Higher Education 67 (3): 273-288.

Links to related posts

‘How did I do?’ Finding new ways to describe the standards of foreign language performance. A follow-up project on the redesign of two marking schemes (DLC)

Working in partnership with our lecturers to redesign language marking schemes

Involving students in the appraisal of rubrics for performance-based assessment in Foreign Languages By Dott. Rita Balestrini

Supporting Inclusivity and Diversity in Language Teaching and Learning at the University of Reading Authored by Laura Brown, Regine Klimpfinger, Daniela Standen and Enza Siciliano Verruccio

12 July 20174 October 2019 By edwardwhite Comment Closed

Language learning and disability: how to avoid the ‘avoidance’?

When the university disability office was approached in 2003 by a new member of staff for guidance on the assessment of a dyslexic student enrolled on a language module, the reply was that students with dyslexia are better advised to avoid foreign language courses. Fast-forward to 2017, and issues of ‘course substitution’, or ‘avoidance’,[i] when it comes to the study of foreign languages and learning difficulties, are still emerging today, as anecdotally reported by prospective secondary school applicants to this university.

When the principles of inclusivity and diversity, fresh from the new University of Reading Curriculum Framework, were chosen as the focus of this year’s university Teaching and Learning conference (January 2017), the discussion and thinking it provoked pointed clearly towards the need – within our institution and within our discipline in this institution – for a thorough reflection on how our current language teaching practices, our language curricula, and the general university procedures can best support students with disabilities who do not wish to avoid learning a foreign language.

Reflecting on disabilities and language teaching and learning practices: a workhop

This is when the idea of the Disability and Language Teaching & Learning Workshop was born. On 18 May, 22 language teaching practitioners from the Institution-Wide Language Programme (IWLP), the Department of Modern Languages and European Studies (DMLES), the Department of Classics and the Institute of Education gathered to explore and discuss experiences and practices of, as well as aspirations to, inclusivity and diversity in language teaching and learning here at Reading. They were guided by Laura Brown from the university Disability Office, with the support of Regine Klimpfinger (DMLES Disability Officer), Daniela Standen (International Study and Language Institute Disability Officer), and Enza Siciliano Verruccio (DMLES Language Coordinator).

The workshop consisted of a blend of theory and practice, with a strong focus on group discussion and activity, given the collaborative approach we wanted to engender. We set the scene with Enza recounting the experiences described above. To further examine the kinds of assumptions we may make about certain disabilities, the group then engaged in a ‘Fact or Fiction’ exercise to indicate whether statements were true or false, unearthing potential stereotypes and preconceptions, such as ‘Students with Asperger’s Syndrome can’t do group work’.

In smaller groups, participants then prioritised skills and attributes needed to learn languages, such as phonological processing skills, memory, curiosity and motivation, using a pyramid shape to indicate the most important at the top ranging to least important at the bottom (Picture 1). Skills and attributes were discussed in terms of how disabilities can affect those skills and attributes, for example the advantage of extroversion in acquiring spoken fluency and how this can be impeded by severe social anxiety. This led to a broader presentation on the experiences that disabled students may have in relation to the four key aspects of language learning – speaking, writing, reading and listening – looking both at barriers and strengths that disabled students may experience in relation to various elements of a languages course, such as oral examinations, classroom conversation exercises, timed translation examination papers, etc.

Groupwork: prioritised language learners’ attributes and skills

The group were then subjected to an impossible memory test and a note-taking exercise using their non-writing hand. These gave them a feel for what it can be like for disabled students to try to fit in with traditional assessment and teaching methods which are unsuited to their learning style.

The group reflected, via Mentimeter, on their experiences of students on their modules who, despite adequate intelligence and effort, struggled with aspects of language learning due to disability (Picture 2). This led to consideration of techniques that can be applied to enhance accessibility and inclusivity in language teaching, across the three core areas of curriculum design, delivery and assessment (Picture 3). The challenges and limitations in applying these techniques were acknowledged as well as the benefits.

2. Workshop attendees report own experiences.

Laura Brown from the university Disability Office leads the discussion on embedding inclusivity and diversity in the language curriculum

Case study examples of disabled students successfully studying languages were presented, highlighting particular aspects that helped them to achieve – this led to one of the key messages from the day in the plenary discussion, that small changes can make a huge difference. We also emphasised how people are not on their own in supporting disabled students and that the day’s collaborative approach provided a platform for further building support networks.

Moving forward

The workshop left the participants with solid advice on how to support students as individuals, but more importantly with ideas and possibilities to explore to make the curriculum more inclusive. From the feedback received there is a clear need and willingness to push these conversations forward. Many expressed the need for more specific information and a forum to share practical ideas and good practice about language teaching and disability, and felt it was paramount to do so collaboratively across departments in order to implement and embed changes. So, keep a look out for the Special Interest Group on disability coming to ISLI and DMLES soon!!

[i] DiFino, S. M. & Lombardino, L. (2004), Language Learning Disabilities: The Ultimate Foreign Language Challenge. Foreign Language Annals, 37, 3, pp. 390-400