Route One Evaluation

Leave a comment

Should All Evaluators Be Ethnographers?

I have recently engaged in evaluation conversations that included talk of the importance of understanding what it is you are evaluating—really understanding—the kind of understanding driven by insatiable curiosity about a program, its place and people. You read everything you can find that helps you understand, from websites and program documents to research literature. And you become an ethnographer of sorts. Ethnography is defined as “the study and systematic recording of human cultures.” Ethnographers study, observe, participate, and talk to people to gain deep understanding of a phenomenon of interest. They immerse themselves. As evaluators, regardless of evaluation approach, design, and methods, the more we immerse ourselves for understanding, the better the evaluation will be.

Many years ago, I was leading the evaluation of the NAEP (National Assessment of Educational Progress) State Service Center. The Center provided training and support for NAEP state coordinators across the US and its jurisdictions. The Center provided 1- or 2-day trainings in Washington, DC throughout the year and an annual conference that spanned several days, held on the west coast. I was fortunate to be able to attend the trainings and conferences. My purpose for attending was to collect data—training surveys and an occasional focus group.

What I knew at the time was that the evaluation was better because I attended the trainings and conferences and got to know the NAEP state coordinators, as well as key NAEP and Center staff. This immersion alongside participants and staff increased my understanding of what I was evaluating.

What I know now is that I should have journaled my observations and reflections. Although this would not have been part of official data collection, it would have furthered my understanding. I would have been an evaluator-ethnographer, an important role for all who want to do this work well.

Leave a comment

Evaluation Approaches: You Don’t Need to Choose Just One

by Bernadette Wright, Guest Blogger

If you’re conducting an evaluation for your thesis or dissertation, I recommend Tamara Walser’s post, “Developing a Framework for a Program Evaluation Thesis or Dissertation,” here on Route One Evaluation. This informative piece highlights the essentials of graduate school writing on evaluation. I’d just like to expand on one thing, based on my evaluation experience.

You don’t have to choose one evaluation approach to follow like a template. Indeed, one way to demonstrate your competency as a researcher is to show your understanding of multiple methods by combining them. Yes: you can develop your own approach to fit your project, as long as you explain what you are doing to conduct the evaluation and how your approach is based on effective practices.

Off-the-Shelf Approaches Don’t Always Fit

People sometimes use different words to describe the same thing. Approach, design, or method, all may be used to describe your unique framework that guides all aspects of how you’ll go about conducting your research.

Within that overarching framework, you can choose from a bunch of theoretical models/approaches. You can find many examples in books, articles, and websites such as the BetterEvaluation, Research Methods Knowledge Base, and Free Resources for Program Evaluation and Social Research Methods websites. Some approaches that evaluators frequently use are case study, mixed methods, qualitative evaluation, field experiment, summative evaluation, and formative evaluation.

When you get into the real world, however, you will find that program managers want answers to many questions at once. In order to get meaningful information to help answer all the questions, you often need to take what you learn from the existing approaches to design an approach to fit the specific situation.

Three Ways to Construct Your Own Evaluation Approach

Here are three ways that you can create your own evaluation approach to fit the context and questions at hand.

Using Multiple Approaches

For my own dissertation, I followed the excellent advice of my advisor, Dr. Marv Mandell, for studying dialogue/action circles program on racism in the Baltimore Region. Instead of specifying a particular “approach,” I described what I would do to answer the questions and how what I was doing was based on effective evaluation practices. I guess you could call it a kind of “two-phase approach.”

  • In the first phase, I used what you could call a qualitative approach. I conducted individual and focus group interviews with stakeholders to identify evaluable goals that were relevant to stakeholders. This was based on general practices for conducting an effective evaluation, which recognize understanding the program you’re evaluating as an important step. (For more on this, see my recent com article on Understanding Your Program.)
  • In the next phase, I examined the impacts of the circles in terms of the goals identified in phase one. For that phase, I used what you could call a mix of two research methods. I 1) analyzed qualitative data from participant feedback forms and 2) conducted a case study of two circles. The case study involved a mix of unobtrusive observation and interviews with participants.

It was a lot of work, but I believe I got more useful results—and learned a lot more—than if I’d just followed one off-the-shelf approach.

For other evaluations that I’ve conducted and managed for customers since, we’ve used different approaches and methods and done things in a different order. Because each evaluation should fit the timeline, scope, and focus of the unique situation.

Using Parts of Different Approaches

As with most evaluations that I’ve been involved with, the evaluation that I conducted for my dissertation used parts of what evaluators often call “impact evaluation,” “process evaluation,” and “formative evaluation.” I examined what impacts the circles had, the goals and the most helpful and least helpful parts of the program, and stakeholders’ recommendations for forming future changes to the program.

Create a Customized Evaluation Approach

It is even possible to create a completely new approach. For his dissertation, my business partner, Steve Wallis, ended up inventing a new approach to evaluating and integrating theories, which led to the technique of Integrative Propositional Analysis. The technique builds on related streams of research across philosophy, studies on conceptual systems and systems thinking, and the technique of Integrative Complexity.

The same kinds of innovation are certainly possible for evaluating programs and policies.

Devise Your Own Evaluation Approach

Design a customized approach for the specific situation of your study. Remember, you are the expert on your specific study. And that’s what experts do. When you use a custom-fit approach, you’re more likely to get evaluation results that are highly useful for benefiting the program and the people it serves and filling knowledge gaps in your field.

Bernadette Wright, PhD is Director of Research and Evaluation at Meaningful Evidence, LLC, where she helps non-profit organizations to get the meaningful and reliable research they need to increase their program results and make a bigger difference.