Evaluation Master-class Training Session Programme Details
Session Introduction
This will provide an overview of the training session, outlining the overall objectives of the programme and bringing together the experiences of the participants of the event in terms of developing and implementing qualitative evaluation strategies. Participants at the event are also asked at this point to list any specific points that they want to get out the day and if they are not covered this will be added to the event later in the day wherever possible for the group work exercise. Examples of where specific queries have been raised in earlier sessions in 2009 / 2010 include:
- How to evaluate neighbourhood management activities.
Session A – Introduction to Specialist Evaluation Processes and Trends in Evaluation Practice
This introductory session will look at the importance of developing specialist evaluation techniques that measure more than just the quantitative effects of a neighbourhood renewal programme and will directly look at:
What are the key benefits to an organisation and programme of undertaking qualitative evaluation processes and practices? These include:
- A summary of the key differences between qualitative and quantitative evaluation processes and practices?
- General benefits: Hard outcomes incldues indicators such as jobs obtained, qulaificiatons gained, numbers progressing onto further education and training (though useful in some cases), do not show the success of the project as a whole. They are an insufficient indicator of a beneficiary’s increased employability. Target groups that are facing multiple barriers to employment may be a long way from being able to acquire a qualification or employment. Consideration of soft outcomes for such groups is a crucial indicator of success. Measuring soft outcomes can also help with the national level evaluation to provide a fuller picture of the impact of the programme as a whole.
Benefits at a project level: Many projects may have specific objectives that relate to the acquisition of soft outcomes. It is vital that projects have the systems in place to be able to measure progress towards these objectives. Soft outcomes can provide a further indicator as to the additionality of the intervention, ie positive results that may not have occurred in the absence of the activity in question.
Benefits at a beneficiary level: Employers are particularly interested in soft skills and abilities, and have been found greatly to value key skills, and personal attributes and attitudes. The beneficiary will therefore be at a distinct advantage in the labour market if they are able to demonstrate possession of such skills and attributes through evidence gained during the project. Working with the beneficiary to record and monitor soft outcomes and distance travelled, and involving the beneficiary in the assessment process, can be a very empowering experience. A project worker may be able to demonstrate to the beneficiary that they have pre-existing skills and attributes of which they were unaware. In addition, if beneficiaries are made aware of the distance they have travelled it can be an enormous confidence boost.
Session B - The Language of Qualitative Evaluation
This will look at some of the definitions that are used in specialist evaluation that include:
(a) Soft outcomes. These are outcomes from training, support or guidance interventions, which unlike hard outcomes, such as qualifications and jobs, cannot be measured directly or tangibly. Soft outcomes may include achievements relating to:
Interpersonal skills, for example: social skills and coping with authority;
Organisational skills, such as: personal organisation, and the ability to order and prioritise;
Analytical skills, such as: the ability to exercise judgement, managing time or problem solving, and
Personal skills, for example: insight, motivation, confidence, reliability and health awareness.
(b) Soft Indicators: There is interplay between indicators and outcomes, in that indicators are the means by which we can measure whether the outcomes have been achieved. The term soft indicators therefore can be used when referring to the achievements which may ‘indicate’ acquisition or progress towards an outcome. A project may wish, for example, to explore whether an individual’s motivation has increased over the length of the project. However, this is, to all intents and purposes, a subjective judgement, indicators (or measures) such as improved levels of attendance, improved time keeping and improved communication skills, can suggest strongly that motivation has increased. Not all indicators will be suitable for all target groups, and some will be target group specific.
(c) Distance travelled by beneficiaries. The term distance travelled refers to the progress that a beneficiary makes towards employability or harder outcomes, as a result of the project intervention. The acquisition of certain soft outcomes may seem insignificant, but for certain individuals the leap forward in achieving these outcomes is immense. A consideration of distance travelled is very important in contextualising beneficiaries’ achievements.
Session C - How toprepare and undertake a specialist evaluation programme
This will look at issues such as:
(a) Drawing up the Evaluation Brief – Deciding why ‘soft’ outcomes should be measured. This will provide advice and guidance on the key details that need to be in a brief when commissioning evaluation strategies.
(b) Managing the Evaluation Process – This will provide a range of pointer on the how to manage an evaluation process, whether it is undertaken externally or internally to the organisation in question.
(c) Key Issues to Remember Concerning Qualitative Evaluation – This covers this issue of managing user expectations of an evaluation processes and highlighting the drawbacks of implementing an evaluation strategy that may be well meaning but is not appropriate to the programme in question.
(d) Guide to the use of Information Technology – This is a new part of this presentation and highlights how developments in Social Networking such as ‘Facebook’ and ‘Twitter’ are now starting to be used to assist with evaluation activity. It also looks at the range of programme development and management software programmes that are in use to assist with evaluation activity.
Session D - General Evaluation Guidance
Why bother? How do organisations benefit from evaluating the direct benefits of a programme? This look as the range of benefits that accrue from effective qualitative evaluation that include learning lessons and acting as a basis for bidding for future resources.
Who needs to be involved in the qualitative evaluation within a Project? – This will look at the range of stakeholders that are required to be involved in any evaluation strategy. Who actually carries out an assessment of soft outcomes and distance travelled will depend on your particular project. Beneficiaries are clearly the mainstay of self-assessment, usually in conjunction with, and perhaps supplemented by, assessments from project staff and external individuals. Those involved in assessment should be those who have regular contact with the beneficiary and whose professional judgements may add to the stock of knowledge about their progression.
When to undertake a qualitative evaluation within a Project? In order to measure progress and distance travelled, it is essential that assessment takes place at the very least at the beginning and end of your project. If time and resources permit, it is advisable to carry out assessment at regular intervals during the project. Not only does this allow for a more responsive service to changing needs, it also provides a more thorough and reliable evaluation of whether true progress is being made.
Session E - What to do with an Evaluation Report when it have been completed
As the session begins to draw to an end for the day this element of the session provides some advice on how to produce an effective Evaluation Report and how to communicate its findings to a range of interested parties. Advice is provided here on a number of topics that include:
Session F - Evaluation Group Work Exercises
This is an interactive element of the programme where participants on the training day work in groups of four to see how they would evaluate some real life examples of regeneration programmes and then compare their answers to what actually took place. We have about 25 examples that can be used and on each training day four or five are used which are of relevance to those attending the training programme.
Session G - Summary and Wrap Up Session
The training day concludes by summing up the key messages that have disseminated during the day and answering and debating any points that have been raised by participants.
