Microsoft Word Packard Teaching Case revised docx

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

15  

 

A  Key  Ally  Comes  to  Packard  

 

In  2005,  about  a  year  after  the  evaluation  got  underway,  the  Packard  Foundation  hired  Gale  

Berkowitz  as  its  new  evaluation  director.  Berkowitz  was  eager  to  find  new  ways  that  evaluation  

could  be  useful  to  program  staff  at  Packard.  

 

“One  of  the  things  I  was  trying  to  achieve  at  Packard  was  to  have  evaluation  seen  as  a  strategic  tool  

and  not  just  an  accountability  function,”  Berkowitz  said.  “Here,  the  preschool  evaluation  was  trying  

to  do  just  that.  Staff  seemed  pretty  engaged  with  the  evaluators.  They  were  making  use  of  the  

information  they  got  and  they  seemed  comfortable  discarding  information  if  it  wasn’t  useful.”    

 

Berkowitz  soon  became  a  strong  advocate  within  Packard  of  the  HFRP  evaluation  team’s  work  and  

approach.  She  also  helped  prepare  the  ground  for  program  officers  to  work  with  evaluators  in  the  

new  ways  required  by  this  approach  to  evaluation—work  that  was  critical,  Reich  said.  

 

“We  were  lucky  to  have  Gale  Berkowitz,”  Reich  said.  “She  did  a  lot  of  training  for  senior  foundation  

staff  on  how  real-­‐time  evaluation  works  and  how  to  manage  it.  If  you  are  going  to  manage  

something  as  complex  as  strategic  evaluation,  you  do  need  some  training.  Gale  brought  in  not  just  

Julia,  but  also  Michael  Quinn  Patton  to  discuss  the  difference  between  traditional  evaluation  and  

real  time  or  developmental  evaluation.  My  training  was  all  around  randomized  controlled  trials.  I  

had  a  real  stereotype  and  perhaps  skepticism  about  its  utility  in  the  real  world.  Gale  helped  us  [see  

the  potential  value  of  real-­‐time  evaluation]  through  holding  a  series  of  senior  staff  meetings.  She  

developed  a  learning  team.  She  helped  sow  the  seeds  to  look  at  evaluation  in  a  different  way.”  

 

Berkowitz  helped  the  evaluators  as  well,  Weiss  said.  “She  challenged  us  by  raising  good  questions  

about  how  [this  approach]  was  working,”  Weiss  said.  “At  one  meeting,  she  convened  all  of  Packard’s  

evaluators  to  talk  about  evaluation  and  start  cross  program  and  cross  evaluation  conversations  and  

she  featured  our  evaluation.  At  that  meeting,  she  also  had  each  of  the  teams  go  off  with  their  

evaluators  and  talk  candidly  about  how  the  evaluation  was  working.  Lois  laid  out  some  things  that  

she  wished  we  had  done  differently.  Gale  created  some  space  for  challenges  and  honest  feedback.”  

 

A  Grantee  Reporting  Innovation  Hits  Some  Bumps  

 

Among  the  first  innovations  that  the  evaluators  tried  was  developing  a  grantee  reporting  form.  

The  Packard  Foundation  places  a  premium  on  reducing  grantee  burden,  and  that  was  an  explicit  

evaluation  goal.  But  as  at  other  foundations,  Packard  grantees  had  to  fill  out  annual  reports  for  the  

foundation  and  were  often  also  expected  to  provide  information  in  separate  reports  for  external  

evaluators.  As  a  way  of  reducing  work  for  the  grantees,  Packard  and  HFRP  decided  to  develop  a  

grantee  reporting  form  that  would  fulfill  both  roles.  

 

“It  was  an  innovation  at  the  time,”  Coffman  said.  “We  wanted  to  bring  the  grantee  annual  report  

together  with  the  reporting  requirements  for  the  evaluation.  We  wanted  to  make  the  information  

more  relevant  to  decision  making  in  the  Foundation.  For  example,  the  report  used  to  come  in  after  

each  one-­‐year  grant  was  completed.  This  meant  that  information  in  those  reports  could  not  be  used  

to  inform  decisions  about  the  next  grant.  To  remedy  this,  the  decision  was  made  to  have  the  reports  

come  in  at  nine  months  into  12-­‐month  grants  so  program  officers  could  use  them  in  developing  

---

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

16  

future  grants.  The  grantee  report  would  then  come  to  us  and  we  would  use  the  information  [for  our  

evaluation.]”  

 

Packard  staff,  grantees,  and  evaluators  devoted  a  large  amount  of  time  in  2004  and  2005  to  develop  

a  single  reporting  form.  The  eventual  report  asked  grantees  to  fill  out  detailed  information  

describing  their  activities  and  outputs  as  well  evidence  that  they  had  achieved  outcomes.  As  it  

turned  out,  in  spite  of  good  intentions  to  streamline  the  reporting  process,  the  report  was  extremely  

time  consuming  for  the  grantees  to  complete.  In  addition,  the  evaluators  found  that  aggregating  the  

data  in  a  meaningful  way  once  it  was  received  was  more  difficult  than  anticipated.  Outcomes  like  

increased  collaboration  or  political  will  are  much  less  easier  to  standardize  and  aggregate  than  

outcomes  like  number  of  children  served,  for  example.  

 

“Early  on,  the  grantee  forms  had  us  report  every  time  preschool  was  mentioned,”  said  Catherine  

Atkin,  president  of  grantee  Preschool  California.  “We  ended  up  amassing  a  lot  of  information.  Our  

feedback  was  ‘we  are  giving  you  a  lot  of  data  points  but  this  is  not  helping  us.’”  

 

For  all  that  work,  the  evaluators  gained  less  useful  information  than  intended.  What’s  more,  because  

Packard  program  officers  are  in  regular  contact  with  their  grantees,  the  one  report  that  the  

evaluators  did  that  summarized  the  grantee’s  work  did  not  tell  the  Packard  staff  significantly  more  

about  their  grantees  than  the  staff  already  knew,  Coffman  said.  

 

The  Foundation  Says:  “Tell  Me  Something  I  Don’t  Know  ”  

 

“We  submitted  our  report  [compiling  the  grantee  information]  to  Packard,”  Coffman  said.  “After  the  

team  at  Packard  read  it,  we  had  a  meeting  with  them  to  discuss  it.  We  sat  down  and  Lois  started  out  

the  meeting.  In  her  characteristically  direct  and  frank  way,  she  said,  ‘so,  thank  you  for  this,  it’s  

thorough  and  nicely  packaged.  But  honestly,  it  doesn’t  tell  me  anything  I  wasn’t  aware  of  from  my  

own  experiences  and  observations.  What  I  really  want  is  for  the  evaluation  to  tell  me  something  that  

I  don’t  know  already,’”  Coffman  recalled.  

 

“The  grantee  reporting  data  was  not  compelling,”  Coffman  added.  “The  report  didn’t  tell  them  

enough  about  strategy.  It  just  told  them  what  the  grantees  had  been  doing,  which  they  felt  they  

already  knew.  It  didn’t  help  for  us  to  sum  it  up.”  

 

Eventually,  when  Meera  Mani  took  over  the  role  of  directing  the  preschool  subprogram  from  Kathy  

Reich,  Mani  worked  with  Coffman  to  revise  the  report  to  a  much  simpler  form.  They  did,  however,  

retain  the  nine-­‐month  timeline,  as  program  officers  found  that  timing  to  be  more  useful  than  if  

reports  came  in  after  grants  were  completed.  

 

“The  feedback  we  were  getting  from  the  grantees  was  that  the  reporting  was  onerous,”  Mani  said.  

“We  were  getting  binders  that  were  two  and  a  half  inches  thick  and  that  included  agendas  for  every  

meeting  they  attended.  Julia  and  I  decided  to  streamline  the  grantee  reports.  The  interim  reports  are  

now  no  more  than  five  pages  and  final  reports  no  more  than  ten  pages.  The  grantees  really  have  to  

distill  what  they  have  achieved.”  

 

---