Evaluation and Impact

This site is designed to help you with evaluation, whether you are new to it or looking for ways to build on an existing evaluation. If you click through each tab, it will guide you through the key points of planning and running an evaluation.

Introduction to evaluation

Why evaluate?

It is important to evaluate the impact of our teaching, learning and student support activities consistently and systemically in order to:

identify, share and build on good practice across the institution
use resources effectively to maximise the benefit for both students and staff, particularly under-represented students
confidently report the impact of our work to key stakeholders and within quality assurance and enhancement processes
meet our obligations to the Office for Students with regard to our Access and Participation Plan.

Evaluation skills can also help you in your professional career, both in being able to generate strong evidence and then improving your practice based on that evidence. Being able to confidently demonstrate the impact of your work can also help in applications for awards and funding.

Definitions

Evaluation:

“A process of data gathering leading to reflection on what has been done against predefined objectives. An opportunity to consider impact and to rethink how things can be done in the future. An assessment of the effectiveness and impact as a result of an activity.” (Amended from AdvanceHE)

Impact:

“An effect of an initiative, or intervention, whether it be positive or negative, direct or indirect, intended or unintended, on an outcome." (Gabarino and Holland, 2009)

Evaluation for Impact Cycle

The diagram below shows the key steps in the evaluation cycle (click to open larger in a new tab).

Levels of impact

When evaluating your activity, it is important to think about the level of impact. Put simply, these levels encourage you to set objectives that go beyond numbers of participants (Level 1) and whether they enjoyed the activity or not (Level 2) to considering what has been learnt or how behaviours and attitudes have been impacted (Level 3), how the learning, knowledge and new skills are applied in different contexts (Level 4) and ultimately demonstrate how your work supports our institutional strategic priorities (Level 5). Click the diagram to open it in full screen.

Standards of Evidence

The Office for Students have produced guidance on Standards of Evidence.

Type 1 - The impact evaluation provides a narrative or a coherent theory of change to motivate its selection of activities in the context of a coherent strategy. For example, a single time point survey.

Type 2 - The impact evaluation collects data on impact and reports evidence that those receiving an intervention have better outcomes, though does not establish any direct causal effect. For example, pre and post surveys, or a comparison of a group who did an activity and a group who did not.

Type 3 - The impact evaluation methodology provides evidence of a causal effect of an intervention. For example, a Randomised Control Trial.

Policy and guidance

Policy on student evaluation of teaching and learning- This policy covers three forms of student evaluation: mid-modular; modular; and, programme evaluation.

Guidance on Programme Evaluation- This guidance aims to outline what Programme Evaluation is, why it is important and how you can put the Policy on Student Evaluation of Teaching and Learning into practice for your Programme. This guidance is recommended as good practice and will help you to align your evaluation with University strategic priorities and provide evidence to meet the needs of other internal and/or external Quality Assurance processes.

Guidance on running a student survey- This guidance note sets out the process for approval to run a survey to seek feedback from students.

Preparing for School Teaching Enhancement Action Plans (STEAP) (Online resources) - These resources aim to support colleagues preparing to complete the School Teaching Enhancement Action Plans (STEAP) QA/QE process. The screencasts explore key elements that UBTLSE have identified as priorities for an effective STEAP submission.

[we are aware that some of these links are broken; please bear with us whilst we fix them!]

For Access and Participation areas

OfS- Evaluation and effective practice guidance- This page links to OfS resources which promote effective practices that can strengthen impact evaluation within access and participation areas.

Resource Bank

Introduction to the resource bank

We have designed the resource bank to guide you through each step of doing your evaluation. If there are other resources you need, please see the Support for Evaluation Guide for who to contact.

Impact by Design steps

Click to open the framework in a new tab.

Creating an evaluation plan

From the diagram above, you can see that planning an evaluation alongside the activity is crucially important. We have several templates and guidance on creating an evaluation plan.

For Teaching & Learning evaluation: Evaluation Planning and Reflecting Template; this mirrors the planning and post-evaluation reflection stages of the Impact by Design Steps.

If your activity relates to under-represented groups: APP Evaluation Planning Template

We also recommend creating a Theory of Change as the first step - the Research and Evaluation team can help with this.

Data collection methods

When planning your evaluation, you will need to decide what data to collect and how. Here you can explore types of data collection methods, with guidance as well as pros and cons.

Focus Groups Guidance

Surveys Guidance

Interviews Guidance

See also validated scales from TASO and the Toolkit for Access and Participation Evaluation for APP-related outcomes.

Top Tips for informal light touch evaluation and closing the feedback loop in an online environment - This guide is designed for teaching staff. It provides tips, tools and a conversation starter you could use to help you evaluate and respond to feedback in an online environment.

Making use of Blackboard Learn data - This guide contains tips and tools to make use of data available through Blackboard Learn. It aims to help you to gauge if students are engaging with your course content which in turn can help you to tailor your teaching and provide further support or guidance as necessary.

Making use of institutional data that is already collected, such as module scores, NSS, graduate outcomes or continuation rates can be a great way to collect data without over-burdening students. Contact PSO to request the data you need for your evaluation.

Ethics

In any evaluation, it is important to uphold ethical standards. The following guides will help you to do so. There is specific wording used for APP ethics statements on surveys, which we have included below.

UoR Ethics Guidance

British Educational Research Association Ethical Guidelines

TASO research ethics guidance

Ethics wording for APP evaluations (please check with the Research & Evaluation Team for specific revisions for your evaluation):

The personal information you supply on this form will be used to activity-relevant information and for quality assurance and data analytics purposes. The contact information you supply is confidential and will be treated in accordance with the General Data Protection Regulation (2016), the UK Data Protection Act 2018 and all applicable Data Protection laws. Please refer to the University of Reading Privacy Policy for more information. The data collected may also be used to evaluate the impact of the activity. Only organisations that require the information for the funding, delivery, evaluation and tracking of these activities will be given access to the data.  This data will be anonymised for reporting purposes.  Your data will be stored on a secure database for future tracking.

Data analysis

Once you have decided on your data collection, you need to decide how it will be analysed. This may be quantitative analysis, such as t-tests or regression analysis, or qualitative thematic analysis. If you are unsure on the best method of data analysis, contact CQSD or the Research & Evaluation team who will either be able to support or point you in the direction of further resources.

Evaluation reporting and dissemination

Once you have completed your evaluation, sharing the findings is really important - even if you found a null result. There are several ways you could disseminate your findings.

Submit an entry to the T&L Exchange. This is useful to take ownership of your activity and to share your findings with others.
If your activity relates to the APP, it can be published on the Evaluation and Reports web page and the evaluation blog. Contact the Research & Evaluation team for more advice on disseminating APP-linked evaluations.
Share at relevant conferences with colleagues from across the sector.
Share with University of Reading colleagues at the relevant committees or community of practice.

FAQs

Q: Do you have any examples of evaluations?

A: Yes, there are some examples of T&L Impact Case Studies, now hosted on the T&L Exchange.

There are examples of APP evaluations that can be found on the Evaluation and Reports page.

Q: Is there anyone who can help me evaluate my activity/initiative?

A: Yes, whilst it is your responsibility to evaluate your work, there are lots of people within the institution who are able to support you. It will depend on what you are evaluating or need help with as to who is best suited to supporting you. The Support for Evaluation Guide sets out the various support which is available.

Q: I'm worried about over surveying my students. Should I use a survey? What else can I do to evaluate?

A: It is a concern shared by many across the institution. There are a number of other methods which can be used to gather new data such as focus groups or interviews, and you could also consider making use of existing data rather than gathering new data.

Q: Can I see a full example report?

A: Here is an APP Evaluation Report FULL example.

Q: Who do I contact if I have feedback about this website?

A: Please contact Lydia Fletcher (lydia.fletcher@reading.ac.uk) to give us any feedback.

If you have any other questions please contact CQSD or the Research & Evaluation team (APP). We will do our best to answer your question directly or identify who is best able to answer your query.