Microsoft Word Packard Teaching Case revised docx

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

23  

HFRP  developed  a  policymaker  rating  tool  that  had  Preschool  California  assess  California  

policymakers’  support  for  preschool  (all  120  state  legislators  and  a  set  of  defined  local  policymaker  

preschool  program  target  counties)  on  three  scales  that  ranked  policymakers  on:    (1)  their  support  

for  preschool,  (2)  their  influence  in  moving  preschool  policy,  and  (3)  Preschool  California’s  

confidence  that  the  first  two  ratings  were  accurate  and  reliable.    

 

Starting  in  2007,  Preschool  California  has  done  these  ratings  once  a  year.  

 

“I’m  a  real  fan  of  this  tool,”  Salisbury  said.  “It’s  a  more  nuanced  tool  [than  report  cards].  You  can  see  

what  policymakers  did  in  their  districts,  whether  they  wrote  op-­‐eds,  or  went  to  visit  a  local  preschool  

center  or  spoke  about  preschool  at  the  Rotary  or  Chamber  of  Commerce.  It  has  a  much  more  

textured  quality  of  the  activities  that  lawmakers  are  engaged  in.”  

 

Preschool  California  staff,  meanwhile,  can  use  the  ratings  to  see  where  they  need  to  do  more  work  

with  specific  lawmakers,  Salisbury  said.  For  example,  Preschool  California  might  want  to  work  more  

closely  with  lawmakers  rated  as  highly  influential  to  get  them  to  write  op-­‐eds  in  support  of  

preschool,  or  bring  them  in  to  see  a  high-­‐quality  preschool  program.  “The  ratings  became  a  work  

plan  for  Preschool  California,”  Salisbury  said.  “It  upped  their  game.”    

 

Sunshine  said  that  the  policymaker  ratings  “enabled  Preschool  California  to  have  a  thoughtful  

conversation  on  how  they  were  going  to  focus  on  to  educate  on  preschool.  It  enabled  them  to  

develop  a  list  of  who  are  friends,  allies  and  foes  and  how  to  amass  votes  and  support.”    

 

Atkins  said  that  policymaker  ratings  were  especially  valuable  in  noting  changes  over  time  in  support  

from  legislators.  “It  didn’t  make  us  change  strategy,”  she  said.  “It  showed  us  where  we  could  have  

lost  ground  with  people  or  where  we  were  making  progress.  The  policymaker  rating  helped  us  see  

where  we  were  going  to  have  to  be  more  intentional.”  

 

Atkin  also  noted  that  the  policymaker  ratings  were  just  one  tool  that  advocates  used  in  gauging  

support  for  preschool.  They  also  relied  on  their  years  of  experience  working  with  legislators  and  a  

certain  amount  of  “gut  instinct”  on  where  lawmakers  stood.  

 

Policymaker  Ratings  Raise  Transparency  Questions  

 

Policymaker  ratings  were  never  made  public  and  Packard  did  not  want  the  outside  world  to  know  

that  they  were  doing  these  ratings  or  for  legislators  to  try  and  influence  them.  As  a  result,  the  annual  

analyses  of  the  ratings  have  only  been  shared  with  the  Foundation  (in  aggregate),  and  with  

Preschool  California.  They  have  not  been  shared  with  other  grantees.  

 

Reich,  who  in  2009  moved  to  the  Organizational  Effectiveness  and  Philanthropy  Program  at  the  

Foundation  where  a  key  value  in  her  job  is  promoting  transparency  in  philanthropy,  said,  “I’m  not  

comfortable  that  we  were  not  entirely  transparent  on  this  evaluation.”  

 

“It  created  unease,”  Sunshine  added,  “Preschool  California  has  this  information.  Should  they  be  

sharing  this  with  other  people?  How  should  we  be  using  this  and  whose  hands  should  it  be  in?”    

 

---

Teaching  Case:  Evaluation  of  Preschool  for  California’s  Children

 

 

24  

Challenges  Surface  in  Developing  Another  New  Tool  

 

Packard’s  shift  in  focus  after  the  ballot  initiative  defeat  brought  up  a  dilemma  for  the  evaluation.  In  

addition  to  focusing  on  the  state  legislature,  Packard  wanted  to  put  more  emphasis  on  cultivating  

support  in  key  communities.  The  evaluators  did  not  have  a  tool  to  collect  information  on  the  

effectiveness  of  that  work.  

   

Evaluators  struggled  for  a  long  time  with  what  to  collect  in  Packard’s  target  communities  that  

would  have  strategic  value.  They  tried  and  then  scrapped  a  number  of  options  that  failed  to  meet  

the  Foundation’s  learning  needs.  

   

“The  bigger  challenge  [than  the  policymaker  ratings]  was  with  Packard’s  strategic  shift  after  the  

ballot  when  they  focused  more  on  communities,”  Coffman  said.  “We  had  to  figure  out  how  to  

capture  what  was  happening  in  the  communities…their  local  strategy  was  a  little  uncertain  at  the  

time.  They  were  focused  in  how  to  better  develop  champions  in  key  constituencies  such  as  

businesses  and  K-­‐12.  It  took  a  while  to  figure  out  how  exactly  that  was  going  to  happen,  and  how  we  

could  add  value.”    

 

Eventually,  in  2009,  the  evaluators  began  developing  a  “champion  tracking”  tool,  similar  to  the  

policymaker  tool.  But  the  tool  took  a  long  time  to  develop,  went  through  much  iteration  and  at  

times,  did  not  receive  much  attention  as  other  priorities  took  precedence.  The  tool  did  not  hit  the  

field  until  in  2010.  Evaluators  are  still  collecting  data.  

 

“We  were  too  late  in  the  champion  tracking,”  Mani  said.  “It  took  a  while  to  get  it  out  there.  Just  as  

we  were  on  top  of  the  policymaker  rating  if  we  had  done  the  same  with  the  champion  ratings  on  

businesses  we  may  have  had  better  results  in  engaging  the  business  community.  The  policymaker  

ratings  drove  Preschool  California  to  be  so  targeted  in  its  work.  I  wish  we  had  that  with  the  

champion  ratings.  We  really  believe  that  we  can’t  move  the  needle  if  we  don’t  have  the  business  

community  engaged.”  

 

The  Foundation,  grantees,  and  evaluators  remained  convinced,  however,  that  champion  tracking  

could  be  a  useful  process.  As  a  result,  they  worked  together  to  revise  the  method  to  be  a  more  

useful  planning  and  evaluation  tool  for  grantees.  “Just  because  it  does  not  work  the  first  time,  

doesn’t  mean  you  should  abandon  it,”  Coffman  said.  “Champions  were  still  an  integral  part  of  the  

strategy,  and  we  needed  to  find  a  way  to  capture  progress  on  champion  development.  Strategic  

learning  often  involves  this  type  of  trial  and  error.  The  trick  is  to  follow  it  up  with  continuous  

improvement.”  

 

The  Midcourse  Review  Creates  Anxiety    

 

In  2008,  Children,  Families  and  Communities  program  staff  had  to  begin  preparing  for  a  midcourse  

review  of  the  preschool  program  before  Packard’s  Board  of  Trustees.  Midcourse  reviews  of  

programs  by  Packard’s  Trustees  are  rigorous,  according  to  several  program  staff.  During  the  

midcourse  review,  the  Board  decides  whether  a  program  should  continue  and  if  so,  whether  it  

should  in  its  current  form  or  take  another  approach.  

 

---