An institution-wide evaluation of the university move to Blackboard Ultra, and what the data surfaced about course design, student belonging and inclusion.
By: Jacqueline Fairbairn SFHEA and Emma Herrod SFHEA
Technology Enhanced Learning, Centre for Quality, Support and Development
Excerpt
“Rather than simply evaluating a platform, we found ourselves exploring how everyday Blackboard practices shape the student experience.”
Overview
The University moved every programme onto Blackboard Ultra in 2024/25. The Technology Enhanced Learning (TEL) team used the switch to Ultra as an opportunity to create a more accessible and engaging digital learning experience for students by introducing a new institutional course template, promoting more consistent course design in Blackboard, and encouraging new approaches to teaching and learning across the University.
After the first academic year of use, we wanted to know if the anticipated benefits were realised, so we ran a mixed-methods evaluation with staff and students.
We found it useful to think about our evaluation using the Five Levels of Impact model. Our surveys and focus groups gave us plenty of evidence around Reaction and Learning, whether staff and students found the new features helpful, and whether they felt more confident using Blackboard.
What surprised us was how far the qualitative data pushed us up the evaluation model towards Behaviour. These were changes in confidence, independence, and sense of belonging, particularly for widening participation (WP) students already navigating other barriers. Students talked about greater confidence, independence and belonging, particularly those from groups already navigating additional barriers to learning.
Objectives
- To understand how staff and students were experiencing Blackboard Ultra and the new institutional template, beyond simply whether people could use the system.
- To find out whether the move to a clearer and more consistent course structure was landing as intended, particularly for WP students.
- To use what we learned to shape our Technology Enhanced Learning (TEL) support and guidance, and to feed evidence-informed findings into wider institutional conversations about the role of Blackboard in teaching and learning.
Context
The University is a medium-sized HE institution, so the move to Blackboard Ultra was a significant undertaking with over 27,000 students, around 1,300 academic staff, and colleagues teaching across the UK, Malaysia and China; from a TEL perspective we support more than 2,300 Blackboard courses. All 14 academic Schools moved to Blackboard Ultra at the same time. We supported this with over 160 staff development sessions (650+ staff trained), roadshows across Schools, and a network of Ultra Champions.
Alongside the migration, we introduced a standard institutional course template, built around
- a flat structure that limited folder nesting
- Ultra Documents to help sequence learning.
These choices aimed to reduce cognitive load, make things easier to find, and support more intentional course design, rather than Blackboard functioning as a file repository.
Implementation
For our evaluation data, we took a mixed-methods approach because we wanted both the breadth of a survey and the depth of user discussions. 
- Focus group: we met with ten of the university’s student Inclusion Consultants, with a scribe on hand, and asked them to walk through their own Blackboard courses live, narrating what was helpful and what wasn’t.
- Written reflections: the same students followed this up with independent written reflections via Padlet, which gave us further material for thematic analysis.
- Staff and student surveys: we ran an institutional survey (195 staff, 195 students), combining Likert-scale and open-response questions. We piloted the questions with student partners and our TEL practitioner forum first, which led to some useful revisions around terminology and question ordering.
- Platform and accessibility data: we drew on Blackboard’s Ally statistics and worked with our digital accessibility officer and student volunteers to test a sample of courses, focusing specifically on instructor practice rather than the system’s built-in accessibility.
We were interested in three things in particular: how helpful people found Ultra’s features; how well the new template supported navigation and wayfinding, and what people thought of our TEL support and guidance. The aim was to produce a descriptive, exploratory piece of work to surface insights about how Blackboard Ultra was being adopted.
Impact
For instructors; the main headline was that staff confidence in using Blackboard was consistently higher than their confidence in designing Digitally Enabled Learning within it. Under time pressure, a lot of modules were understandably starting out as content repositories rather than properly sequenced learning spaces. Feature usage backed this up:
- two-thirds of staff weren’t using Bulk Edit;
- half weren’t using the course search bar,
- a quarter weren’t using Ultra Documents at all, often because the priority was simply getting content moved across rather than redesigning it.
For students; the picture was more nuanced once we broke it down by widening participation group:
Black, Asian and other minority ethnic students reported particularly strong positive perceptions of the new template, around ten percentage points above the overall baseline on questions about navigation and wayfinding. Their comments suggested Blackboard was helping them navigate academic expectations more broadly, not just locate files.
“From a learner’s perspective, Blackboard functioned as a centralised academic hub, enhancing the coherence and self-regulation of my study practices.”
Commuter students particularly valued features that supported quick, flexible, just-in-time engagement such as announcement pop-ups, a consistent mobile view and the course search bar. For these students, clear structure wasn’t just convenient, it helped them be more strategic with their time. We were struck by how much they wanted to see active lecturer presence in their courses; they liked feeling connected to the module as it unfolded.
“Being able to use my phone and still have the same layout really helps me stay on track when I’m commuting.”
Students reporting a long-term health condition or learning difference told a different story. This group was furthest from positivity averages and showed the greatest sensitivity to inconsistency in layout, file naming and assessment visibility. They valued the system update itself, but inconsistent instructor practice within the course template was amplifying anxiety rather than easing it.
“Overall, I believe the update of Blackboard Ultra to have been a significant factor in terms of easing my anxiety about my studies and examinations this year.”
“Sometimes the assignment info was a bit all over the place, across different [template] sections, which made it difficult to make sure you’d read all the information needed.”
Taken together, the message was consistent (even where the experiences differed); clear, predictable course design changes how confident and connected students feel, and the effect is more strongly experienced by students who are already navigating additional barriers to learning.
Outcomes: what can teaching colleagues do?
The evaluation surfaced a set of small, low-effort changes instructors can make that have a real, positive impact on belonging and the learning journey. None of these require redesigning a module from scratch:
- Use meaningful file naming. Keep titles predictable, short and distinct, so students can wayfind easily, including on mobile.
- Use the file description field. A brief description helps students orientate themselves and understand what they’ll find in a resource before they open it.
- Give a sense of ‘you’. Students value knowing their lecturer is present in the course; a conversational tone, or a short video or audio update, goes a long way.
- Combine files in Ultra Documents. Avoid long lists of separate files; use Ultra Documents to guide students through content in sequence.
- Aim for anti-scroll Ultra Documents. Keep them short – they should ‘feel’ like an A4 sheet of paper, not an endless scroll.
- Uphold consistency. Stick to consistent course structures and templates so students can find information quickly, week to week and module to module.
- Schedule release of content and announcements. Timed releases help guide students through the module as it unfolds, rather than leaving everything visible at once.
Reflections
What made the evaluation project useful was resisting the urge to treat it as a checklist of requirements met. The questions that mattered most we not “do people like the updated system”, but “how do everyday practices affect user’s experience”.
If we were doing this again, we would like to find a more systematic way of following up with the WP students whose experiences diverged most, particularly students with a health condition or learning difference, where the gap between intention and instructor practice was most distinct.
One thing we didn’t anticipate was how far the findings would travel. We’d planned this as a local, TEL-focused evaluation piece; however, it opened a wider conversation about institutional approaches to engaging colleagues with Blackboard Ultra practices and ultimately supporting student learning.
Follow up
The findings have since been shared with the University’s Teaching and Learning and Infrastructure Committees, the Awarding Gaps steering group and Reading’s Commuter Student working group, among others. Colleagues presented the work externally at the Blackboard User Group and at the 2026 Durham Blackboard Conference.
Locally, in TEL we will be looking towards:
- Developing metrics to monitor adherence to the Ultra template and file-naming conventions, working with School Directors of Teaching and Learning to identify courses needing support.
- Shifting more of our TEL support towards course design and digital pedagogy, rather than technical training alone.
- Continuing to work with the university groups to shape thinking about how the VLE can better support belonging and connection for students
None of this would have carried the same weight without the evaluation planning behind it. It gave us a thorough evidence base and a student voice to bring into conversations. That’s probably the bigger lesson here; evaluation isn’t just about checking whether a project worked. Done well, it gives you the legitimacy to keep having the conversation afterwards.





























