EVALUATION TEACHING CASE
Evaluation of the
David and Lucile Packard Foundation’s
Preschool for California’s Children
Grantmaking Program
By Susan Parker
Clear Thinking Communications
September 2011
Teaching Case: Evaluation of Preschool for California’s Children
2
Teaching Evaluation Using the Case Method
[From: Patton, M. Q. & Patrizi, P. (2005). Teaching evaluation using the case method. San
Francisco, CA: Jossey Bass. pp. 5-‐14.]
“[Traditional] evaluation training…relies mainly on traditional didactic teaching in the
classroom to ground students in the scientific approaches that are the cornerstone of the field. But
methods are only the beginning of what [students] need to understand in order to succeed. Once
students have mastered the basics of evaluation options, designs, and methods, the challenge of
professional practice becomes matching actual evaluation design and processes to the nature of the
situation, as well as hearing and mediating the opposing opinions that often surface.
In mature professions like law, medicine, and business, case teaching has become
fundamental to professional development. Once one has learned the basic knowledge of a field,
higher-‐level applications require judgment, astute situational analysis, critical thinking, and often
creativity. Professional practice does not lend itself to rules and formulas. Decisions are seldom
routine. Each new client, patient, or customer presents a new challenge. How does one teach
professionals to do situational analysis and
exercise astute judgment? The answer from these
established professions is the case method.”
“Cases take us beyond the reality of the individual and plunge the learner into a plot with
multiple perspectives, strong disagreements, and avid articulation of fully plausible yet fully
divergent views. Just as in real life, learners hear from others who may have conflicting opinions, but
unlike reality, learners can step out of vested interests, remove blinders that can hinder learning,
and experiment with new skills and approaches in a secure environment.”
Best teaching case practices include:
• The core decision points throughout the case should have enough tension (and enough
factual information and context leading up to them) that you could reasonably argue
competing perspectives about the decisions made. In other words, the case should not just
be a narrative about what worked or did not work. There must be clear moments where
decisions could have gone different ways. Choices have different benefits and costs.
• The author's voice should be neutral, with no "drawing of conclusions." The tension between
the choices at the decision points can, for example, be presented through direct quotes of
the participants. The case itself does not do any diagnosing or give commentary on the
success or failure of a particular decision, nor does it frame or summarize the questions for
discussion.
• The facilitator should be able to ask questions like: "What is the main tension at play here?"
"What do you think about the way the group decided to proceed?" What are the practical
implications of the decision for grantees?" "What did they give up by going that route?"
"What else could they have done and at what cost/to what benefit?"
Teaching Case: Evaluation of Preschool for California’s Children
3
Background on the Case
This teaching case was written for evaluators and funders. The case focuses on a real-‐world
evaluation supported by a private foundation that used a strategic learning approach to evaluation.
Strategic learning means using evaluation to help organizations or groups learn in real-‐time and
adapt their strategies to the changing circumstances around them. It means integrating evaluation
and evaluative thinking into strategic decision making and bringing timely data to the table for
reflection and use. It means making evaluation a part of the intervention—embedding it so that it
influences the process.
Evaluation focused on strategic learning is different from more traditional evaluation
approaches in some important ways. For example, it is fundamentally different from summative
evaluation, which judges the overall merit or worth of an effort for the purpose of concluding
whether that effort should be continued or discontinued.
1
As Michael Patton says, summative
evaluation is not even possible with emergent strategies because they will not “hold still long
enough for summative review.”
2
Strategic learning is also different from formative evaluation, which
focuses on improving a program or effort, often so that a later summative evaluation can be done.
While strategic learning certainly aims to help strategies improve or move in a positive direction, in
reality the “right” direction is not always known. Strategic learning means helping strategies adapt
based on what information is known or can be collected at the time. It does not necessarily mean
making judgments that what was done before was ineffective. Finally, strategic learning is different
from evaluation focused on accountability, which aims to ensure that efforts are doing what they
said they would do and that resources are being managed well. Strategic learning has a much
broader purpose that goes well beyond oversight and compliance.
Specifically, this case focuses on the evaluation of the David and Lucile Packard
Foundation’s Preschool for California’s Children grantmaking program. It is intended to promote a
critical analysis of the evaluation and its evolving interaction with the grantmaking program and
strategy, rather than an analysis of the grantmaking strategy itself. The case chronicles the
evaluation’s nine-‐year evolution, and identifies key points at which it switched course because
methods were not working, or because the Foundation’s strategy shifted. It highlights several
questions/challenges, all of which are relevant to strategic learning approaches, such as:
• How to ensure the evaluation is useful to multiple audiences (board, funder, grantees)
• How to “embed” the evaluator in a reasonable way while maintaining role boundaries
• How to manage often competing learning versus accountability needs
• How to time data collection so it is “just in time” but also reliable and credible
• How to get information that does not just verify what program officers already know.
1
Scriven, M. (1991). Evaluation thesaurus. 4
th
edition. Newbury Park, CA: Sage.
2
Patton, M.Q. (2008). Utilization-‐focused evaluation. Thousand Oaks, CA: Sage Publications. p. 118.