Below are some of the frequently asked questions (FAQs) regarding the Electronic Management of Assessment programme (EMA).
If you can't find the information you're looking for, take a look at the About the Programme page. Alternatively, if you have a question about Blackboard or RISIS, you can contact the Technology Enhanced Learning (TEL) team.
The Electronic Management of Assessment Programme (EMA) ran from January 2017 until January 2020 to deliver the University's long-term vision for online assessment. It was a complex, transformative programme which has the potential to place Reading in a sector leading position in areas of assessment provision.
The EMA Programme was a collection of projects brought together to significantly improve the assessment experience of staff and students and reduce the administrative burden of assessment across the University. The Programme supported the continued move towards online submission, marking and feedback of assessed work, underlying integration between our core systems (Turnitin, Blackboard and RISIS), the creation of a student dashboard for assessment information and further development in our approach toward learning analytics.
The EMA Programme supported the University’s vision to provide a more efficient and significantly improved end-to-end assessment process to meet student expectations, improve the assessment experience for staff and students and enable the University to accommodate growth in student numbers. It also supported the overall improvement of university performance to continue to build towards the University’s ambition to enhance its standing as a leader in research and higher education.
The programme focused on 3 core objectives:
- Enabling a consistently good student assessment experience. This means that all students will have a similarly good experience when submitting coursework, receiving feedback, viewing marks and accessing overall results.
- Providing an improved and supported assessment experience for staff. This means that colleagues receive full support in their use of online submission, feedback and grading tools, that they have access to better information about student performance and that they spend less time administering assessment.
- Reducing the administrative burden of assessment for the University. This means that staff will benefit from more efficient, less manual business processes. There will be improved integration between Blackboard and RISIS and more reliable systems from which to deliver a service. This will give the University improved management information and a scalable system able to cope with increases in student numbers.
The Programme ran for three years from January 2017 until January 2020 although there is ongoing support work following the end of the Programme. All projects within the Programme ran simultaneously except for the development of the Student Progress Dashboard which began in Autumn 2018.
You can see the timeline for delivery in the About EMA section of the website.
The Programme was led by Professor Gavin Brooks, then Pro Vice Chancellor (Teaching and Learning) with Professor Emma Mayhew as Academic Director and Shelley Connor leading on programme management. To ensure that the activity of the programme represented the needs of the University, a collaborative approach was adopted. Integral to the delivery team were colleagues from existing University of Reading teams, such as Student Services, Exams, ADE and TEL (Technology Enhanced Learning).
Academic Partners were seconded to the Programme on a part-time basis, alongside their existing teaching, research and administrative roles. In most cases they worked within the team as representatives of the entire academic community and not just their respective schools. In addition, Graduate Partners joined the team to provide vital student insight and champion the interests of the student body within all Programme activity.
The EMA Programme supported the move to electronic assessment for the majority of summative assessment (where practical), using the University’s existing core systems of RISIS (SITS), Blackboard and Turnitin.
Sub modular marks were made available and are now managed within RISIS for the majority of Undergraduate, PGT and Foundation programmes that are run from the UK and UoR Malaysia campuses.
Integration between RISIS and Blackboard was achieved. This included the automation of submission point creation in Blackboard and the automatic transfer of marks from Blackboard and Turnitin into RISIS.
A Student Progress Dashboard was created for students and relevant staff, providing graphical representation of assessment data held within RISIS only.
Formative assessment is not in scope, however the tools within Blackboard and Turnitin may be used for formative assessment if desired.
- Massive Open Online Courses (MOOCs), Open Online Courses (OOCs) and Post graduate research assessment are not included.
- Programmes not using the Blackboard and Turnitin for electronic submission and marking, such as Degree Apprenticeships, AHEP, Henley “Greenlands” Post Experience courses. However, we are working closely to align where appropriate.
- Programmes not managed from the UK, such as those managed at BIT, NUIST and UoR South Africa. However, when NUIST students transfer to Reading in Part 3, their sub modular marks will be managed in RISIS.
- It is not possible to automate the creation of submission points in Turnitin at this time.
Assessment and feedback involves a complex set of processes. Moving to electronic management of assessment brings a wide range of benefits to support and streamline these processes as well as promote effective student learning. It also contributes to wider, strategic aims to meet rising student expectations and improve the student experience.
The benefits anticipated as a result of managing assessment electronically can be seen in the Benefits page. It is recognised that everyone’s experience will be different and therefore the benefits described will be felt to varying degrees.
The Programme provided enhanced process and improved functionality using the existing core systems of RISIS (SITS), Blackboard and Turnitin. Integration between RISIS and Blackboard/Turnitin was developed to automate key processes.
A key outcome of the Programme was the availability of increased data and information. For example sub modular marks became available in RISIS. This provided an opportunity to create enhanced reports, information screens and a Student Progress Dashboard to present richer and more graphical information on assessment and marks.
Where new functionality was introduced, training and guidance was provided.
A core Programme Board, chaired by the Programme Sponsor, reported into UEB and the Strategy and Finance Committee. Each of the 5 Workstreams had a Project Board, chaired by a member of the Programme Board. A Policy Advisory Group was also created to provide support and guidance on any policy related queries. Matters were referred to existing University committees and governance groups as necessary.
All projects within the Programme were undertaken in accordance with PRINCE2 standards. PRINCE2 offers a framework of tools and techniques to assist in the management of projects and as such each project will use gateways, product based planning and have management by exception at their core. Inevitably, some flexibility was required for individual projects and the methodology has been tailored to suit the University, however, this standard way of working ensures the best possible chance of delivery. The Programme methodology in use is the MSP (Managing Successful Programmes) methodology which similarly offers a framework from which the Programme as a whole can be managed.
With any large programme of work there will be risks to success. As such individual risk registers were maintained for each project and an overarching one for the Programme. We use these to identify risks, rank them and then consider mitigating actions to reduce or avoid the risks. Risks were reviewed regularly and reported on by each Project Manager in regular highlight reports and by the Programme Manager in the monthly Programme Highlight Report to the Programme Board.
An effective institution-wide engagement and communications strategy was fundamental to the delivery of key programme goals. The strategy was driven by a number of key principles including:
- Broad and effective engagement with students, academics, professional services and other stakeholders from the start and continuing until programme implementation and reflection. Ongoing engagement methods are designed to identify existing good practice, future requirements, expectations and concerns from key users but also, crucially, to ensure that the stakeholder voice clearly impacts on project design.
- Being aware of, and remain sensitive to, the needs of different stakeholders and the wider environment in which they operate.
- Adopting an honest approach to elements that do not meet expectations at first attempt and seeking out input from stakeholders to re-work solutions.
The Dashboard provides students with an interactive dashboard of graphs and charts presenting their summative sub modular marks, module marks and part marks over time, as well as tools to enable target setting and progress reflection.
This dashboard presents students and staff with a series of screens displaying core student progress data. It shows how much summative assessment students have completed, what their progress looks like in a series of graphics and how attainment compares to goals that students can (optionally) set. It sits within RISIS and is accessible to Academic Tutors and support staff.
The Student Progress Dashboard supports other strategic projects such as the introduction of the Academic Tutor System. It also lays the foundations for thinking about innovative new ways to support students in their learning. Using assessment data in combination with other datasets, such as attendance and Blackboard access, a future learning analytics solution could enable better understanding and improvement of student engagement. This has been explored within the EMA Programme and may be developed in the future outside of the Programme.
The EMA Programme was created to deliver the University’s long-term vision for online assessment in order to enable a consistently good student and staff assessment experience and reduce the administrative burden for the University.
The Curriculum Framework (CF) acts as a catalyst for programme teams working in partnership with students to: take a holistic programme level approach; share existing good practice; and explore creative ways for further enhancing our programmes. The CF defines a set of pedagogic principles, including those underpinning assessment.
EMA and the CF are therefore complementary. For example, moving to online management of assessment provides an ideal opportunity for programme teams to reflect on assessment across a programme. Please click here to access an online tutorial on the new assessment reporting screens on RISIS which provide assessment information at University/School/Departmental level on the volume, timing and range of assessments.
Reflection on assessment design can also strengthen the alignment between learning outcomes of programmes and of associated modules, and the ways in which these feed into assessment criteria
The EMA Programme and Curriculum Framework teams were working collaboratively to ensure that coordinated support is available to schools.
The EMA Programme and the Academic Tutor System were working collaboratively to support student success and progression.
A significant piece of work within the EMA Programme was to record individual summative assessment marks for each piece of assessment (or ‘sub modular marks’) on RISIS. This was driven, in part, by the need to provide timely and more detailed assessment data throughout the year to Academic Tutors. Previously only overall module marks were available on RISIS and they were only accessible at the end of the academic year. This was too late for Tutors to understand their tutees' progression and to identify any concerns surrounding academic performance. Tutors are now more able to assist students in reflecting on their academic progress to date, working with students to identify their strengths and weaknesses in relation to academic study and encouraging students to use all opportunities to develop a broad range of skills.
The development of the Student Progress Dashboard provides additional graphical and tabular information to Academic Tutors to enable further discussion around student expectations, student engagement, analysis of feedback trends and personal goal setting. This will further enhance the quality of academic support available to all students.
Currently Blackboard allows all staff who are enrolled on a Blackboard module access to student submissions and feedback. This includes colleagues involved in marking and moderating assessment. Under the EMA Programme, Academic Tutors now have access to their tutee’s individual assessment marks, as they become available throughout the year, in RISIS. But, currently, Academic Tutors do not have access to their tutee’s assessment feedback in Blackboard unless they are also involved in marking or moderating each module that their tutees are enrolled on.
We looked at a range of potential solutions to address this. Enrolling all Academic Tutors on all modules, both those owned and not owned by their department, that their tutees might be enrolled on would not be practical or scalable so we began a consultation process with Blackboard to explore how their current system might meet this requirement. Having looked at a number of different features, we have found that, unfortunately, Blackboard is not currently able to meet this need. Having raised this issue we hope that they will be able to adapt existing functionality or develop new functionality to meet this need in the future.
During discussions with colleagues regarding electronic marking and feedback, it emerged that having a second screen can make it easier when marking online. Split-screen working (having two different screens opened and available at the same time) when using Blackboard or Turnitin can make the task of marking and providing feedback more fluid, reducing the need to scroll and move between screens. This may be particularly helpful, for example, when working with rubrics and grading forms or when marking assessments with multiple appendices.
The EMA Programme provided a second screen to anyone who felt it would assist them in the electronic marking process, at no cost to schools,. This scheme closed in July 2019.
The EMA Programme supported the majority of Schools to increase electronic submission, feedback and grading. The University has been using the systems and tools for electronic management of assessment to varying degrees prior to commencing the EMA Programme. For many Schools and Departments, the changes will be centred around programme administrative processes. A higher level of change may be felt in Schools or Departments where electronic marking is a new practice.
It can take time to adjust to new ways of working and new system tools, and it will become quicker and easier over time. The TEL team are providing training and one-to-one support, and guidance materials are available on our Resource Map. There are also useful video experiences from colleagues on our Benefits page, explaining how they have adjusted to increased electronic assessment.
It is recognised that some colleagues may have concerns in relation to increased screen time or specific health related or medical problems. In order to be thorough and ensure good practice, a Disability and Accessibility Advisor was brought in to conduct an inclusivity review.
For Students, the review identified that electronic management of assessment will increase accessibility options for submission of assessment and receipt of feedback, and that the structures are well embedded to identify needed adjustments and ensure that all students have the same access to teaching, learning and university activities.
For staff, it was identified that the existing University support processes are suitable for assessing and identifying if any reasonable adjustments are required. Colleagues are asked to follow this process by initially undertaking a Display Screen Equipment Assessment. This may identify actions that can be taken, and if further support or advice is required then staff can speak to any of the following:
- DSE Advisor
- Line Manager (who can arrange for an Occupational Health referral, if required)
- HR Partner (who can advise on other available options for support)
There are a number of accessibility features that you can switch on or adjust in Blackboard, Turnitin, your internet brower and your computer’s operating system. More information can be found on the TEL site. Please contact the TEL team via firstname.lastname@example.org if you need assistance.
There are a number of features, preferences and extensions built into to your computer and available in the browser (e.g. Chrome) that might improve your experience of working online. These include magnifying the screen, changing screen background colours, contrast and brightness and speech to text.
If you have started marking online and increased the time you spend using a computer, then you should undertake a DSE assessment as an essential first step to identify any actions to take or if further support and advise is required. You can also speak directly with your:
- DSE Advisor
- Line Manager (who can arrange for an Occupational Health referral, if required)
- HR Partner (who can advise on other available options for support)
It is also worth trying out some of the in-built accessibility features in the computer and browser to see if any of these can help your experience of using a computer when marking online. For example:
- The font or cursor is too small – use the zoom to or change the background colour of the screen. Check your settings to personalise the mouse pointer size and contrast.
- The screen is too bright – Reverse the colour of your screen or choose a high contract theme
- Reduce the amount of typing by using speech recognition or keyboard shortcuts
Links to the different accessibility features available in the main browsers can be found on the TEL Blog page – Accessibility Resources.
The Library’s Inclusive technology page also describes how make the screen easier to see and explains how to use Text to speech options.