Teaching Case: Evaluation of Preschool for California’s
Children
5
Case Teaching Questions
Evaluation Points to Elicit During Questioning
4. Flexibility and Timing: How can
evaluators simultaneously meet
demands for flexibility and quick
timing while also maintaining data
collection quality and rigor? What
are the implications for evaluation
contracts and resources with
strategic learning approaches?
With a strategic learning approach, timing is everything. To ensure that
evaluation is used, evaluators who aim to support real-‐time learning and
decision making must deliver the right data at the right time. They must be
flexible—willing to adjust their data collection plans according to how the
environment or strategy is shifting; and timely—providing data and
facilitating learning at the right time to inform strategic decisions.
Evaluators who support strategic learning often must have the capacity to
work rapidly—quick to design, implement, and analyze data; and
responsively—able to provide useful and trustworthy strategy-‐level data
when the need arises. All of the data and intelligence needs of an evolving
strategy cannot be anticipated at the outset of a social change effort, and
many arise at a moment’s notice. Fast-‐moving policy environments, for
example, often present unexpected windows of opportunity or quick and
unexpected changes in momentum. This case offers the opportunity to
discuss issues around the communication structure, resource investments,
and other practical concerns that are needed to make a strategic learning
evaluation work for both evaluators and funders.
5. Learning: What does it mean for a
strategic learning evaluation to be
successful? How do we know when
strategic learning has occurred?
What structure and processes are
important to ensure such learning
occurs? What is the evaluator’s
responsibility in the learning
process?
While the quality of data collection and design is crucial, the ultimate
criterion for the success of a strategic learning approach is the extent to
which the client uses the information generated through data collection
and reflection to answer strategy-‐related questions. Evaluation for
strategic learning is necessarily utilization-‐focused, and is therefore
committed to actionable data. However, evaluators supporting strategic
learning cannot assume that learning necessarily happens simply because
data are available, even when data are actionable and timely. To help
support use, evaluators must collect and then frame data in a way that
clarifies its connection to strategy and tactics, surfacing its implications.
Evaluators must also build in intentional learning and reflection processes
so that its users—the strategic decision makers—become sensemakers,
too, interpreting the meaning of data and uncovering the implications
together. This case allows for a discussion about how the evaluators
attempted to foster learning and use of data, and how the Foundation
responded. In some cases, the Foundation chose to go in a different
direction than the evaluation data suggested. Does this mean that strategic
learning did not occur?
TEACHING CASE
Evaluation of the David and Lucile Packard Foundation’s
Preschool for California’s Children Grantmaking Program
by Susan Parker
Clear Thinking Communications
Introduction
This case tells the story of how external evaluators and program staff at the David and Lucile Packard
Foundation took a risk on a nontraditional approach to evaluation. Called “real-‐time evaluation” or
“developmental evaluation,” this approach aims to promote strategic learning by providing regular,
timely information about how a strategy is unfolding, which organizations then use to inform future
work and make course corrections.
This type of evaluation relies heavily on skills that are often not required of evaluators and program
officers. To be successful, a specific organizational culture must also be present. The evaluators had
tried this approach at another foundation and experienced several challenges. While they had faith
in this newer evaluation approach, they were also wary from their earlier experience.
Packard program officers, meanwhile, had mixed experiences with evaluation. Some were
unconvinced that evaluations were useful. The Packard Board of Trustees, while open to employing
a new approach to evaluation, also had more traditional evaluation questions about long-‐term
impacts. What’s more, the context was challenging—the Packard Foundation had recently suffered a
devastating loss of assets and needed to make large cuts in its grantmaking programs.
During this time, Packard made a major investment in a strategy to ensure that all three-‐ and four-‐
year-‐olds in California would have access to preschool within 10 years. The preschool grantmaking
program was among the largest and most audacious programs that Packard had ever funded. Hand
in hand with this bold strategy, Packard contracted for an evaluation approach that had potential
but was yet to be fully tested.
The Beginning
In March 2002, Lois Salisbury came to work at the David and Lucile Packard Foundation at one of
the most difficult periods in the Foundation’s nearly 40-‐year history. Packard had hired Salisbury to
head its Children, Families, and Communities program, one of six grantmaking program areas.
Salisbury, who had worked for 20 years as a class action civil rights litigator and later led Children
Now, California’s largest child advocacy organization in the state, is at heart an advocate, a fighter
and a risk taker. She was taking a calculated risk coming to Packard.