I have recently engaged in evaluation conversations that included talk of the importance of understanding what it is you are evaluating—really understanding—the kind of understanding driven by insatiable curiosity about a program, its place and people. You read everything you can find that helps you understand, from websites and program documents to research literature. And you become an ethnographer of sorts. Ethnography is defined as “the study and systematic recording of human cultures.” Ethnographers study, observe, participate, and talk to people to gain deep understanding of a phenomenon of interest. They immerse themselves. As evaluators, regardless of evaluation approach, design, and methods, the more we immerse ourselves for understanding, the better the evaluation will be.

Many years ago, I was leading the evaluation of the NAEP (National Assessment of Educational Progress) State Service Center. The Center provided training and support for NAEP state coordinators across the US and its jurisdictions. The Center provided 1- or 2-day trainings in Washington, DC throughout the year and an annual conference that spanned several days, held on the west coast. I was fortunate to be able to attend the trainings and conferences. My purpose for attending was to collect data—training surveys and an occasional focus group.

What I knew at the time was that the evaluation was better because I attended the trainings and conferences and got to know the NAEP state coordinators, as well as key NAEP and Center staff. This immersion alongside participants and staff increased my understanding of what I was evaluating.

What I know now is that I should have journaled my observations and reflections. Although this would not have been part of official data collection, it would have furthered my understanding. I would have been an evaluator-ethnographer, an important role for all who want to do this work well.