Microsoft Word Packard Teaching Case revised docx

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

17  

Atkin  added,  “We  have  moved  to  a  report  that  is  more  streamlined  and  reflective.  It’s  more  valuable  

to  us  to  spend  time  thinking  about  the  work  we  did  versus  spending  a  bunch  of  time  gathering  lots  

of  data.”  

 

The  grantee  reporting  form  experience  was  a  wakeup  call  that  led  to  a  significant  shift  in  the  

evaluation,  Coffman  said.  

 

“It  was  a  big  turning  point  for  me,”  she  said.  “If  the  purpose  of  evaluation  is  for  strategic  learning  

then  we  should  be  focused  on  capturing  information  that  the  foundation  is  not  capturing.  We  

[eventually]  stopped  collecting  data  from  grantees.  It  is  extremely  nontraditional  that  evaluators  

don’t  collect  data  from  grantees.  Instead,  we  decided  that  we  would  collect  data  from  audiences  

that  Packard  wasn’t  already  systematically  tracking.  We  determined  that’s  where  we  could  add  the  

most  strategic  learning  value.”  

 

“For  me,  that  is  a  key  point  of  strategic  learning,”  Coffman  continued.  “We  should  be  focusing  on  

questions  that  other  people  aren’t  answering.  Packard  was  much  less  interested  in  having  us  

validate  what  they  already  knew."    

 

More  “Ah  Ha”  Moments  are  Desired  

 

If  a  goal  of  this  evaluation  was  to  provide  insights  or  data  that  Packard  program  staff  and  grantees  

didn’t  already  know,  for  some  Foundation  staff,  the  evaluators  sometimes  fell  short  in  meeting  it.  

 

“A  lot  of  times  if  you  talk  to  the  program  team  after  they  get  the  evaluation  reports,  they  say  that  

there  are  no  big  ‘aha  moments’,  or  ‘we  had  no  idea’  moments,”  Berkowitz  said.  “That’s  something  

our  program  people  always  want  from  evaluations—they  want  to  be  surprised,  they  want  to  find  out  

something  they  didn’t  know.  The  [Children,  Families  and  Communities]  program  has  more  or  less  

accepted  that.  What  they  get  from  an  external  report  is  validity  and  credibility  rather  than  just  

relying  on  their  own  intuitions.”  

 

Salisbury  added,  “I  wish  we  could  find  from  an  evaluation  more  of  what  you  don’t  know.  There  were  

not  a  lot  of  surprises.  I  don’t  know  what  to  say  about  how  one  fixes  that.  It’s  where  this  evaluation,  

as  most  evaluations,  falls  down.”  

 

Coffman  acknowledges  that  staying  on  top  of  new  strategy  developments  and  figuring  out  how  and  

when  to  add  value  without  being  involved  in  every  strategy-­‐related  conversation  that  happens  at  the  

Foundation  can  be  challenging.    

 

“Making  sure  we  are  always  relevant  and  our  data  are  always  fresh  is  really  hard  unless  this  one  

evaluation  is  the  only  thing  you  are  focused  on  or  doing,”  she  said.  “And  most  evaluators  don’t  have  

that  kind  of  luxury  to  focus  on  one  evaluation  at  a  time;  it’s  not  a  viable  business  model.  But  this  is  

not  like  other  approaches  to  evaluation  where  you  can  go  and  come  back  and  expect  that  most  

things  haven’t  changed  much.  If  you’re  not  paying  attention,  or  if  you’re  not  quick  enough,  especially  

if  advocacy  is  involved,  then  you  are  behind  and  what  you  are  doing  to  inform  the  strategy  is  bound  

to  be  less  relevant.”  

 

---

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

18  

Added  Arron  Jiron,  who  joined  Packard’s  Children,  Families  and  Communities  program  in  2006,  “A  

big  challenge  for  the  Harvard  evaluators  is  how  they  get  inside  the  head  of  program  officers.  We  

have  a  lot  of  rich  conversations  in  the  Foundation  around  strategy.  That  is  often  hard  to  get  at  for  

the  evaluators.  It  also  hard  for  us  to  know  when  to  pull  the  evaluators  in.  It  tends  to  be  more  ad  hoc,  

or  when  we  think  about  it.”  

 

The  Bellwether  Methodology  Emerges  as  a  Response  to  an  Early  Strategy  Challenge  

 

The  evaluators  had  another  early  opportunity  to  show  how  their  approach  to  strategic  learning  

could  provide  timely,  critical  information  to  Packard  about  its  strategy.    

 

A  hallmark  of  a  strategic  learning  approach  to  evaluation  is  for  evaluators  to  be  nimble  and  

responsive  as  the  landscape  changes  in  which  funders  and  grantees  are  trying  to  make  an  impact.  A  

case  in  point  is  the  bellwether  methodology—an  early  innovation  of  the  evaluation  and  an  example  

of  the  collaborative  work  possible  between  evaluators  and  foundation  staff.    

 

As  the  Packard  strategy  was  getting  underway,  Rob  Reiner  had  set  the  wheels  in  motion  for  a  

ballot  initiative  that,  if  passed,  would  fund  voluntary  full-­‐year  preschool  for  all  California  four-­‐

year-­‐olds,  provided  mostly  by  local  school  districts.    

 

Salisbury  was  not  happy  with  the  prospect  of  a  ballot  initiative  being  filed  anytime  soon.  She  felt  that  

it  was  happening  too  quickly—not  enough  groundwork  had  been  laid  to  put  this  initiative  in  front  of  

voters.  Packard  staff  had  developed  a  detailed  logic  model  outlining  the  steps  they  foresaw  that  

needed  to  happen  before  California  would  adopt  universal  preschool.  Few  of  them  were  in  place.  As  

a  foundation  official,  however,  there  was  nothing  Salisbury  could  do  to  influence  the  ballot’s  timing.    

 

The  ballot  initiative  also  gave  rise  to  a  larger  issue.  Packard  staff  needed  to  find  a  way  to  gauge  

whether  the  importance  of  universal  preschool  was  breaking  through  with  influential  leaders  in  

California—a  group  Salisbury  dubbed  “bellwethers.”

10

 

 

“Whether  [universal  preschool]  was  going  to  be  addressed  through  a  ballot  initiative,  a  legal  

strategy,  or  a  local  effort,  we  needed  to  know  whether  this  issue  was  moving,”  Salisbury  said.    

 

Packard  did  not  have  a  way  to  find  out  that  information.  So  the  staff  turned  to  the  HFRP  evaluators  

to  figure  out  how  to  get  it.  Working  together,  Packard  staff  and  the  evaluators  developed  a  new  tool  

to  gauge  the  level  of  support  for  universal  preschool  by  interviewing  40  influential  leaders  or  

“bellwethers”  in  California.  These  structured  interviews  were  unique  in  that  half  of  the  bellwethers  

were  individuals  who  were  not  expected  to  have  any  prior  background  or  knowledge  on  the  

preschool  issue  (therefore  if  they  did  know  the  issue,  it  was  probably  because  of  advocacy  efforts).  

In  addition,  bellwethers  did  not  know  in  advance  that  the  interview  would  focus  on  preschool.    

 

                                                                                                                         

10

 “Bellwethers”  refer  to  leaders  or  “influentials”  in  California,  whose  positions  require  that  they  track  state-­‐level  issues  and  

politics.  Bellwethers  were  not  funded  by  the  Packard  Foundation  and  could  provide  a  key  external  perspective  on  the  

progress  and  status  of  efforts  to  promote  a  universal  preschool  policy.  

 

---