Evaluation in WIL: Themes across case studies

In the afternoon session of the NSW/ACT Chapter Forum a number of case studies were analysed by participants in relation to evaluation in work integrated learning (WIL). 

Blair Slater, Senior Careers Consultant (International), UNSW Sydney, has collated the themes across the case studies, and presented a snapshot of those themes below.

It would be great to hear in the comments section below, your views in response to the challenges and tensions identified by the groups. And, more importantly, how have you dealt with these challenges and tensions at your university?

1. Stakeholders:

• Predominate Stakeholders in WIL evaluations:
  Students (current and prospective), academics, designers/Administrators of program or course (careers office, Faculty office), senior executives (decision makers), employers, recipients of work performed (client), 3^rd party providers, university or organisation recruitment team, future employers within the industry since they may potentially benefit from WIL graduates.

  2. Desired Outcomes:

•  Acquire a better understanding of impact to achieve a change.
• Understand the value of a WIL initiative and inform program/course development. “The purpose should be to improve, not to prove.”
• To show positive results in the form of benefits for stakeholders, however, negative results also need to be considered.
• Demonstrate value to increase funding
• To formulate a “toolkit” for success for stakeholders.
• Alignment between WILl and graduate attributes and employability.
• Increase perception of return on investment for students.

3. Steps in the evaluation:

•  Develop research questions and examine if there is any existing theory or literature to help inform questions and area of focus. Ask, will evaluation need to be longitudinal? Is it summative or formative?
• Develop methodology, ethics application if considering research publication.
• Data collection, online surveys, student self-evaluation.  In person interviews are effective for qualitative data.
• Analysis and dissemination to stakeholders.
• All the above need to be assessed in terms of feasibility and cost before starting.

 4. Opportunities, Challenges and Tensions:

 Opportunities

• Evaluation can potentially be applied to other disciplines. For example, faculty to faculty.
• Opportunities for curriculum enhancement/development. Receive greater academic support/financing.
• Evaluation can create a good baseline upon which to base future development and evaluation.
• If evaluation is done well, it could be used for, or extended, into research.

 Challenges

• When evaluating outcomes, be open to the possibility of learning about an unintentional outcome which perhaps can warrant further evaluation or research. Also need to be cautious of censoring negative findings and handling this challenge.
• Not much published on benefits for employers. What evaluations can be done in this area to help bring on more employers?
• Importance of defining/listing the variables in addition to defining and understanding the assumptions being made prior to undertaking evaluation.
• Potential for evaluations to be too broad if variables cannot me limited or identified. For example, students’ background, academics, industry and other factors that may be hard to control or know.
• Not being as successful as originally thought. Do you still report the negative? Does this change the dissemination?
• Tracking alumni in any longitudinal evaluation.

Tensions

• Balancing research/evaluation with program agenda. Often need someone dedicated to the research component. Investment of time and resources in undertaking a good quality evaluation. Cost versus strategic value needs to be assessed.
• Diverse conceptions of “value” depending on stakeholder. Stakeholder who is administrating the evaluation may have different values in mind from those who will be making decisions based on the evaluation.
• Employability skills need to have a clear definition for both the evaluator and subject.
• Employability is also often hard to measure and must be defined. Is it measured through satisfaction (assess against student expectations), learning outcomes, or engagement and activity level of learning longitudinally.
• If students are aware that their reflections will be used for evaluation, will this change what they write or give them self in any self-ratings?
• Need to be aware of comparing same WIL format but with different employers.