At the risk of seeming overly negative, here are some of the challenges I see:
- The experiences we’re evaluating are complex and involve not only many factors, but also many avenues in which they are integrated – as knowledge, attitudes, values and more. Moving to a different country and culture can profoundly affect our worldview, our sense of identity, and our understanding of the world around us.
- The learning and integration happen over time, including (and importantly) after the experience. I may not realize what I’ve learned for years. (Don’t get me started on the relative worthlessness of end-of-program evaluation forms!)
· Stakeholders value different approaches and have varying priorities. Some press for “SMART” objectives and clear (and quantifiable) measures of whether (and to what extent) those objectives were met. Others look for “stories” of growth and development. Some are focused on academic learning, while others look at new personal awareness. Retention and graduation rate are key indicators for many, while some look at changes on majors (which can slow down graduation).
· There is a temptation to measure what can be easily accessed rather than what is most meaningful. I can give a student a language test before and after a study abroad program, for example, but I may well not be measuring all aspects of his or her increasing fluency – as some studies have shown. Grades in courses overseas are an easy measure to pinpoint, but they may not begin to measure deeper academic growth.
I tell my students to speak with stakeholders to develop common understandings; to balance limited resources and meaningful design; to triangulate not only methods but also participants; to work ethically; to consider intervening factors, and more. But even doing all of that, how do we accomplish what we’ve set out to do, what we’ve been told to do – to measure learning outcomes?
It’s a humbling process. But an important one.