Aes conference Sydney 2011 Ralph g straton Murdoch University

AES Conference Sydney 2011 Ralph G Straton Murdoch University

• Evaluation Design for Differential Effects Evaluation

Evaluation Design for Differential Effects Evaluation Overview

• Nature of Evaluation Design

• Evaluation Questions

• Inference Issues

• Evaluation Design for Differential Effects

• Qualitative Evaluation and Differential Effects

• Conclusion

Evaluation Design for Differential Effects Evaluation Nature of Evaluation Design

• Blueprint for the Study

• What information to be gathered
• From which Sources
• When it is to be gathered
• How will it be Obtained
• Limits Nature and Extent of Information Obtained

Evaluation Design for Differential Effects Evaluation Evaluation Questions

• Does the program have an effect? and
• What are the effects of the program?
• Does the program achieve its goals? and
• To what extent does the program achieve its goals?
• Is the program better than the present program (or
• some other specific program)? and
• Is the program the best program?

Evaluation Design for Differential Effects Evaluation Evaluation Questions

• To what extent are program outcomes related to status on Characteristic X?
• Do NESB students perform as well as ESB students on the new ‘Culture Free’ maths program?
• Is performance among NESB students related to their culture of heritage?

Evaluation Design for Differential Effects Evaluation Inference Issues

• Idealised View of Evaluation Inference

• Inference to the Population Sampled From utos > UTOS

• Reality of Inference in Evaluation

• Inference to ‘Other’ Populations utos > *UTOS

• Maximising Utility of Evaluation Inferences

• Choose Sample to Reflect ‘Other’ Populations
• Observe Relevant Person, Program and Setting Characteristics

Evaluation Design for Differential Effects Evaluation Evaluation Design for Differential Effects

• Identify and Observe Relevant Characteristics Variables

• Build Program Variations into the Design

• Observe ad hoc Program Variations

• Determine and Report Diversity on Relevant Variables

• Determine and Report Program Interactions

Evaluation Design for Differential Effects Evaluation Qualitative Evaluation and Differential Effects

• Include Qualitative Descriptions of Program Variations

• Include Qualitative Descriptions of Program Settings