Microsoft Word Packard Teaching Case revised docx

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

25  

“Oh  God,”  Sunshine  said,  remembering  that  time.  “The  midcourse  review  is  a  very  big  deal  at  

Packard.  We  take  extra  care  and  attention  to  make  sure  that  we  make  cogent  presentations  to  the  

Board  that  hit  the  mark.  We  have  rigorous  reviews  and  rehearsals  of  our  presentations.  It’s  an  

intensive  process.”  

 

 “There  was  a  lot  of  anxiety  on  [Packard’s]  part  around  the  midcourse  review,”  Coffman  said.  “They  

weren’t  exactly  sure  how  the  Board  was  going  to  react  to  the  fact  they  had  made  progress  but  

hadn’t  any  big  wins  [on  universal  preschool].”  

 

“There  was  some  anxiety  on  our  part  too,”  Coffman  added.  “We  had  to  turn  what  we  had  been  

doing  in  getting  information  back  to  the  [Children,  Families,  and  Communities]  team  to  information  

that  would  be  useful  and  relevant  to  the  Board.  Sometimes  that  wasn’t  a  completely  great  fit.”  

 

To  prepare  a  report  for  a  Board  that  wanted  more  outcome  data,  Coffman  drew  on  the  data  she  had  

collected  already,  but  HFRP  had  to  collect  additional  data  as  well.    

 

“Some  of  information  we  presented  to  the  Board  wasn’t  data  we  had  collected  for  the  

evaluation,”  Coffman  said.  “The  Board  was  asking  impact  questions,  like  ‘are  kids  better  off?’  or  

‘how  has  the  quality  of  preschool  changed  in  California?’  Our  evaluation  was  not  designed  to  

answer  those  questions  and  we  weren’t  sure  Packard  could  or  really  should  answer  those  questions.  

They  wanted  data  on  long-­‐term  impacts  that  hinged  on  earlier  changes  in  the  policy  arena.  We  were  

trying  to  collect  information  to  inform  advocacy  strategy  to  get  to  those  policy  changes,  not  long-­‐

term  impact.  It  was  a  bit  of  a  dilemma.  Ultimately  we  reported  data  from  external  sources  that  

addressed  those  questions,  even  though  the  questions  were  not  directly  relevant  to  where  Packard’s  

strategy  was  at  the  time.”    

 

This  was  also  a  point  when  the  decision  not  to  collect  data  from  Packard  grantees  was  a  

disadvantage,  as  Salisbury  and  her  team  felt  that  the  Trustees  would  want  to  know  grantee  

perspectives  on  Packard’s  role  and  strategy  and  those  data  were  missing.  Consequently,  right  before  

the  midcourse  report  was  written,  HFRP  added  a  set  of  qualitative  interviews  with  grantees  and  

other  preschool  “insiders”  to  collect  that  data.  

 

The  Second  Bellwether  Report  Shows  Progress,  Clarifies  Focus  

 

As  part  of  getting  ready  for  the  midcourse  review,  HFRP  wrote  a  report  on  the  latest  round  of  

bellwether  interviews  conducted  in  2008.  Reich  had  been  prepared  for  the  possibility  that  the  

preschool  program  might  not  continue  because  of  the  lack  of  major  wins  to  point  to.  At  first  glance,  

the  evaluation  report  bolstered  her  feeling  that  issue  of  universal  preschool  in  California  had  not  

made  much  headway.  Reich  remembers  feeling  downcast  after  she  read  the  report.  

 

Shortly  after  sending  their  report  Coffman  and  Weiss  flew  out  to  Packard,  as  they  had  been  doing  

regularly  since  the  evaluation  began  in  2004,  to  discuss  the  findings  with  the  Packard  staff.  

“I  was  down  in  the  dumps  before  the  meeting,”  Reich  remembers.  “But  Julia  and  Heather  began  

providing  more  qualitative  information  about  what  they  heard  in  the  interviews,  particularly  

compared  to  the  first  bellwether  report.  They  said  ‘something  is  changing,  particularly  in  the  

legislature.  We  think  it’s  a  more  fertile  climate  for  preschool.’  That  advice  came  at  a  critical  time.  It  

---

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

26  

put  to  bed  whether  we  should  continue.  From  that  meeting  forward,  we  were  invested  in  

continuing.”  

 

Evaluators  Play  a  Role  that  Raises  Larger  Questions  about  Boundaries  

 

The  Preschool  team  started  their  preparation  for  the  December  2008  Board  of  Trustees  meeting  in  

early  2008.  Over  several  months,  they  discussed  and  debated,  often  with  Coffman  present,  what  

made  most  sense  to  do  to  recalibrate  Packard’s  preschool  program.  

 

By  this  time,  HFRP  had  worked  closely  with  the  Packard  staff  for  four  years.  “They  knew  the  

preschool  program  as  well  as  anyone,”  Salisbury  said.  “As  HFRP  helped  to  prepare  for  the  midcourse  

review,  they  brought  a  depth  of  knowledge  rare  for  evaluators  who  might  just  ‘parachute  in.’”    

 

The  HFRP  evaluators  “understood  the  strengths  and  weaknesses  of  where  the  issues  sat.  They  had  

an  ear  for  the  dynamics.  They  knew  the  story  and  could  pick  up  the  threads  that  otherwise  just  

might  pass  you  by,”  Salisbury  said.  

 

This  illustrates  a  larger  point  about  the  skills  needed  for  this  approach  to  evaluation.    

 

“This  isn’t  for  the  newcomer  to  evaluate,”  Berkowitz  said.  “To  do  this  well,  evaluators  have  to  

come  in  with  a  full  tool  box  of  tools,  experience  and  people  skills.  There  is  a  lot  of  emotional  

intelligence  involved.”  

 

Jiron  added,  “Evaluators  who  do  this  have  to  be  excellent  writers.  They  have  to  be  very  clear  in  

explaining  often  complicated  ideas.  We’ve  seen  other  evaluators  that  are  ponderous—classically  

academic.  You  also  want  an  evaluator  who  knows  how  to  ask  really  good  questions  to  get  a  clear  

understanding  of  how  you  are  thinking  of  strategy  and  can  see  where  the  holes  are.  You  want  

evaluators  who  have  several  methodologies  at  their  disposal.  Evaluators  tend  to  lean  toward  a  

specific  style  of  evaluation.  It  would  be  nice  to  have  folks  who  can  draw  from  a  number  of  different  

ways  of  evaluating.  They  can’t  rely  on  a  single  methodology.”  

 

Packard  staff  said  that  the  evaluators’  contribution  to  the  midcourse  review  is  one  of  the  strongest  

examples  of  how  this  approach  to  evaluation  can  help  in  strategic  learning.  Some  also  said  that  the  

work  brought  up  a  larger  issue  on  the  role  that  an  evaluator  plays  in  this  approach  to  evaluation.  

Reich  and  Coffman  remember  a  sometimes  arduous  process  in  getting  the  midterm  evaluation  

report  to  strike  the  “right  tone”  for  the  Board.  

 

“I  really  pushed  Julia  to  give  the  Board  more  strategic  advice,”  Reich  said.  “I  said  ‘don’t  be  afraid  to  

do  it.’  It  brings  up  the  question  of  at  what  point  is  an  evaluator  an  evaluator  and  at  what  point  do  

you  use  them  as  a  strategic  advisor?  It  doesn’t  happen  with  every  evaluator  and  it  doesn’t  happen  

overnight.  By  the  time  Julia  found  herself  in  that  role  [of  strategic  advisor],  she  had  been  evaluating  

this  program  for  four  years.  Her  role  as  strategic  advisor  evolved.  We  trusted  her  data  and  opinions.”  

 

For  her  part,  Coffman  recall’s  Reich’s  advice:  “The  [Packard  team]  felt  like  the  Board  is  sick  of  hearing  

‘this  program  is  great’  at  these  midcourse  reviews.  They  want  to  know  what  is  not  working,  what  

needs  to  be  different.  Kathy  pushed  us  to  more  to  say  to  the  Trustees.”    

 

---