Microsoft Word Packard Teaching Case revised docx

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

19  

In  June  2005,  Reiner  filed  the  Preschool  for  All  ballot  initiative  (Proposition  82).  It  would  be  

financed  by  a  1.7  percent  tax  on  individual  incomes  higher  than  $400,000  or  couples  earning  more  

than  $800,000.  

 

When  Reiner  announced  the  ballot  initiative,  the  Harvard  evaluators  had  already  begun  the  

bellwether  interviews.  They  had  not  yet,  however,  conducted  interviews  with  a  key  subsection  of  

the  group—policymakers.  Those  interviews,  coincidentally,  took  place  around  the  height  of  

optimism  about  the  ballot’s  chances  for  success,  which  had  implications  for  the  eventual  findings.  

 

“The  ballot  initiative’s  timing  affected  [the  bellwether  interviews],”  Coffman  said.  “When  we  

originally  planned  them,  the  idea  was  that  they  would  provide  more  of  a  baseline.  All  of  a  sudden,  in  

the  middle  of  our  interviews,  there  was  a  lot  of  press  and  a  lot  happening  in  the  background  that  

wasn’t  related  to  Packard’s  grantmaking.  As  a  result,  our  earlier  interviews  took  place  in  a  very  

different  context  than  our  later  ones.  The  ballot  initiative  was  announced  and  then  opposition  

ramped  up.  In  order  for  our  data  to  be  useful,  it  had  to  be  carefully  timed.  We  missed  some  

opportunities  because  we  didn’t  time  it  quite  right.  Some  of  that  we  couldn’t  have  predicted,  but  

some  of  it  we  probably  should  have.”  

 

In  keeping  with  their  promise  of  offering  “real-­‐time  evaluation,”  one  month  after  completing  the  

bellwether  interviews—in  August  2005—the  evaluators  prepared  the  first  of  their  “learning  reports,”  

which  was  on  their  bellwether  survey  findings.  A  week  later,  they  held  their  first  learning  meeting  to  

discuss  the  findings  and  implications  for  Packard’s  strategy.    

 

Among  the  key  findings  of  the  bellwether  report:  

 

•  Some  88%  of  the  bellwethers  said  they  were  familiar  or  very  familiar  with  the  issue  of  

universal  preschool.    

•  More  than  half  of  the  bellwethers  supported  adopting  a  universal  preschool  policy  now  or  in  

the  near  future  and  only  a  small  percentage  (13%)  was  clearly  opposed.  

•  Bellwethers  identified  Packard  grantees  and  the  Packard  Foundation  as  the  main  advocates  

for  universal  preschool  in  California,  findings  that  supported  the  organizational  investments  

made  by  the  Packard  team,  particularly  for  high-­‐profile  grantees  such  as  Preschool  California  

and  Children  Now.    

•  Bellwethers  did  not  cite  business  or  Latinos  as  key  advocates  for  universal  preschool—two  

groups  that  the  preschool  initiative  had  specifically  targeted  to  serve  as  leaders.  In  addition,  

bellwethers  did  not  see  the  preschool  issue  as  one  with  a  strong  grassroots  movement.    

•  Bellwethers  raised  a  number  of  concerns  about  the  specifics  of  a  universal  preschool  policy,  

primarily  related  to  the  cost  and  California’s  capacity  and  readiness  to  implement  it.    

 

The  Ballot  Initiative  is  Defeated    

 

Not  long  after  the  evaluators  completed  the  bellwether  report,  opposition  to  the  ballot  initiative  

mobilized  and  leaders  carried  out  a  fierce  attack  on  the  initiative  and  on  Reiner  himself.  In  June  

2006  voters  overwhelmingly  rejected  Proposition  82  with  60.9%  opposed  and  39.1%  in  favor.    

---

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

20  

 

Salisbury  does  not  attribute  specific  change  in  Packard  strategy  to  information  gained  from  the  

bellwether  interviews.  Because  of  its  timing,  the  bellwether  interviews  provided  cause  for  optimism  

that  the  ballot  initiative  could  pass.  She  said  the  results  of  the  ballot  initiative  did  make  staff  think  

about  whether  they  could  control  the  timing  of  the  bellwether  interviews  more.    

 

On  the  grantee  side,  Preschool  California’s  Atkins  remembered  that  the  bellwether  report  caused  

some  anxiety.  “It  was  early  on  in  our  work  and  I  remember  Julia  presenting  this  to  the  grantees,”  

she  said.  “I  remember  people  were  getting  upset.  You  can’t  talk  to  a  few  business  people  in  

California  and  make  generalizations  about  whether  the  business  community  supported  preschool.  It  

was  a  tension  to  the  extent  that  people  were  feeling  that  [the  report’s  findings]  could  be  a  reflection  

of  whether  they  were  doing  a  good  job  as  a  grantee.  We  have  grappled  with  this.  Over  time,  Packard  

has  taken  steps  to  be  careful  about  how  you  disseminate  this  kind  of  information  to  grantees.”  

 

Coffman  said  that  her  work  on  the  bellwether  methodology  helped  her  begin  to  make  a  shift  in  her  

thinking  about  timing  and  the  importance  of  context.  

 

“Even  though  our  proposal  used  all  the  right  language  about  learning,  I  think  it  was  still  kind  of  

traditional  in  some  ways,  especially  in  the  beginning,”  she  said.  “To  achieve  strategic  learning  you  

have  to  design  evaluation  around  the  other  person’s  timeline,  not  your  own.  You  also  have  to  

consider  what  is  happening  or  what  is  likely  to  happen  in  the  broader  context.  I  set  up  the  evaluation  

with  traditional  reporting  timelines—we’re  going  to  do  this  report  in  March  and  this  report  in  June.  

That  didn’t  map  on  to  when  they  needed  certain  information.”  

 

Still,  Reich  remembers  the  bellwether  report  as  a  key  moment  when  she  began  to  see  the  utility  of  

the  evaluation  to  her  daily  work.  “The  bellwether  report  was  enormously  informative,”  she  said.  “It  

gave  us  candid  feedback  on  how  preschool  was  perceived  by  important  policymakers  and  leaders  in  

the  state.  We  got  feedback  that  we  were  not  getting  traction  among  business  leaders  and  Latino  

leaders.  It  confirmed  some  gut  feelings  that  we  were  not  hitting  our  mark.”  The  findings  also  helped  

prompt  “significant  changes  to  our  grant  making  to  Latino  leaders  and  businesses.”    

 

Jiron  said  it  was  also  helpful  to  learn  that  Preschool  California  and  other  Packard  grantees  were  

recognized  as  leaders  on  the  preschool  issue.  “It’s  a  data  point  when  you  have  a  grantee  that  says  

‘we’re  going  to  target  the  audience  of  grass  tops’  and  then  you  talk  to  the  grass  tops  and  they  

mention  two  or  three  of  our  leading  children  advocates.  It’s  a  proximate  indication  that  we  are  

headed  in  the  right  direction.”

     

 

“Real-­‐Time”  Evaluation  can  be  an  Elusive  Goal  

 

Reflecting  on  the  overall  evaluation,  several  participants  said  that  providing  “real-­‐time”  evaluation  

that  program  staff  can  use  to  inform  strategy  can  be  difficult  to  do  in  practice.    The  bellwether  

report  is  a  case  in  point.  According  to  Salisbury,  “One  of  the  challenges  [in  this  approach  to  

evaluation]  was  timing.  In  a  field  where  there  was  so  much  rapid  change  and  decisions  being  made  

not  within  our  tapestry,  could  the  evaluators  give  us  feedback  that  was  timely?”  

 

“The  real  time  stuff  sounds  better  on  paper,”  Sunshine  said.  “Reports  don’t  come  in  on  time,  they’re  

late,  or  they  capture  a  static  moment.  They  were  interesting  but  they  did  not  help  tweak  strategy.  

---