Literature search

Literature search

• Literature search

• study quality assessment

• Selection criteria

• Statistical analysis

• Heterogeneity

• Publication bias

Narrative procedure (conventional critical review method)

• Narrative procedure (conventional critical review method)

• Vote-counting method (significant results marked “+”, converse “–” and no significant results “neutral”)

• Combined tests (combining the probabilities obtain from two or more independent studies)

Systematic review is the entire process of collecting, reviewing and presenting all available evidence

• Systematic review is the entire process of collecting, reviewing and presenting all available evidence

• Meta-analysis is the statistical technique involved in extracting and combining data to produce a summary result

A meta-analysis is also possible without doing a systematic review

• A meta-analysis is also possible without doing a systematic review

• With no attempt to be systematic about the particular studies were chosen

To increase power

• To increase power

• To improve precision

• To answer questions not posed by the individual studies

• To settle controversies arising from apparently conflicting studies or

• To generate new hypothesis

Assessment of strength of evidence

• Assessment of strength of evidence

• To determine whether an effect exists in a particular direction

• Statistical pooling of results

• To obtain a single summary result

• Investigation of heterogeneity

• To examine reasons for different results

A meta-analysis is a two-stage process

• A meta-analysis is a two-stage process

• Stage 1
• Extraction of data from individual study
• Calculation of a result for that study (point estimate)
• Estimation of chance variation (confidence interval)
• Stage 2
• Deciding if it is appropriate to calculate and pool average results across studies
• If so, calculate and present the results.

What are the main comparisons in your view?

• What are the main comparisons in your view?

• How will you summarise the results of the outcomes for each study?

• How will you decide whether to combine the results of the separate studies?

• Do you plan any subgroup or sensitivity analyses?

Dichotomous data (e.g. dead or live)

• Dichotomous data (e.g. dead or live)

• Counts of events (e.g. no. of pregnancies)

• Short ordinal scales (e.g. pain score)

• Long ordinal scales (e.g. quality of life)

• Continuous data (e.g. cholesterol con.)

• Censored data or survival data (e.g. time to 1st service)

Continuous data

• Continuous data

• Calculation of overall effect size (standardised mean difference)

• Rate data

• Measures of effect (difference between incidence in the population of exposed vs not exposed)
• Relative risk
• Odds ratio
• Risk difference

Fixed effect models

• Fixed effect models

• Mantel-Haenszel (MH)
• Has optimal statistical power
• Softwares are available for the analysis
• Peto test (modified MH method)
• Recommended for non-experimental studies

• Random effect models

• DerSimonian & Laird method
• Bayesian method

• Regression models (Mixed model)

This model is based on a mathematical assumption that every study is evaluating a common treatment effect

• This model is based on a mathematical assumption that every study is evaluating a common treatment effect

• In this model, the true treatment difference is considered to be the same for all trials

• The SE of each trial estimate is based on sampling variation within the trial

• The summary results are specific to the trials included

• The summary results can not be generalised to the population

Mantel-Haenszel approach

• Mantel-Haenszel approach

• Odd ratio
• Risk ratio
• Risk difference
• Not recommended in review with sparse data (trials with zero events in treatment or control group)

• Peto method

• Odds ratio
• Used in studies with small treatment effect and rare events
• Not a very common method
• Used when the size of groups within trial are balanced

In this model, the assumption is that the true treatment effects in the individual studies may be different from each other

• In this model, the assumption is that the true treatment effects in the individual studies may be different from each other

• In this model, the true treatment difference in each trial is itself assumed to be a realisation of random variable, which is usually assumed to be normally distributed

• The SE of each trial estimate is increased due to the addition of this between-trial variation

Odd ratio

• Odd ratio

• Risk ratio

• Risk difference

Fixed effects assumption

• Fixed effects assumption

• “did the treatment produce benefit on average in the studies in hand”?
• “what is the best estimate of the treatment effect”?

• Random effects assumption

• “will the treatment produce benefit on average”?
• “what is the average treatment effect”?

• Choice between fixed and random effects may be decided

• By a formal chi-square test of homogeneity
• That is whether the between study variance component is zero or not

Risk

• Risk

• A chance or probability of having a specific event (no of participants having the event in a group divided the total no. of participants)

• Odds

• The ratio of events to not-events (risk of having an events divided by the risk of not having it)

• Odds Ratio (OR)

• The odds of the event occurring in one group divided by the odds of the event occurring in the other group

• Relative risk or Risk Ratio (RR)

• The risk of the events in one group divided by the risk of the event in the other group

• Risk difference (RD; -1 to +1)

• Risk in the experimental group minus risk in the control group

• Confidence interval (CI)

• The level of uncertainty in the estimate of treatment effect
• An estimate of the range in which the estimate would fall a fixed percentage of times if the study repeated many times

Odds ratio (OR) will always be further from the point of no effect than a risk ratio (RR)

• Odds ratio (OR) will always be further from the point of no effect than a risk ratio (RR)

• If event rate in the treatment group

• OR & RR > 1, but
• OR > RR

• If event rate in the treatment group

• OR & RR < 1, but
• OR < RR

When the event is rare

• When the event is rare

• OR and RR will be similar

• When the event is common

• OR and RR will differ

• In situations of common events, odd ratio can be misleading

1. metan

• 1. metan

• 2. labbe

• 3. metacum

• 4. metap

• 5. metareg

• 6. metafunnel

• 7. confunnel

• 8. metabias

• 9. metatrim

Relative Risk (Fixed and Random effect model)

• Relative Risk (Fixed and Random effect model)

• Fixedi= Fixed effect RR with inverse variance method

• Fixed= M-H RR method

• metan evtrt non_evtrt evctrl non_evctrl, rr fixed second(random)

• favours(reduces pregnancy rate # increases pregnancy rate)

• lcols(names outcome dose) by(status) sortby(outcome) force

• astext(70) textsize(200) boxsca(80) xsize(10) ysize(6)

• pointopt( msymbol(triangle) mcolor(gold) msize(tiny)

• mlabel() mlabsize(vsmall) mlabcolor(forest_green) mlabposition(1))

• ciopt( lcolor(sienna) lwidth(medium)) rfdist rflevel(95) counts

• Saving the graph in different formats

• graph export "D:\Forest plot.gph", replace

• graph export "D:\Forest plot.gph".png", replace

• graph export "D:\Forest plot.gph".eps", replace

Meta-analysis should only be considered when a group of trials is sufficiently homogeneous in terms of participations, interventions and outcomes to provide a meaningful summary

• Meta-analysis should only be considered when a group of trials is sufficiently homogeneous in terms of participations, interventions and outcomes to provide a meaningful summary

Examination for “heterogeneity” involves determination of whether individual differences between study outcomes are greater than could be expected by chance alone.

• Examination for “heterogeneity” involves determination of whether individual differences between study outcomes are greater than could be expected by chance alone.

• Analysis of “heterogeneity” is the most important function of MA, often more important than computing an “average” effect.

By different investigators

• By different investigators

• In different settings

• In different countries

• In different ways

• For different length of time

• To look at different outcomes

• Etc.

Clinical diversity: Variability in the participants, interventions and outcomes studied

• Clinical diversity: Variability in the participants, interventions and outcomes studied

• Methodological diversity: Variability in the trial design and quality

• Statistical heterogeneity: Variability in the treatment effects being evaluated in the different trials. This is a consequence of clinical and/or methodological diversity among the studies

Study location and setting

• Study location and setting

• Age, sex, diagnosis and disease severity of cases

• Timing of the treatments

• Dose and density of the intervention

• Definition of the outcomes

Conventional chi-square (χ2) analysis (P>0.10)

• Conventional chi-square (χ2) analysis (P>0.10)

• I2= [(Q-df)/Q x 100% (Higgins et al. 2003), where

• Q is the chi-squared statistic; df is its degrees of freedom

• Graphical test-forest plots (OR or RR and confidence intervals)

• L’Abbe plots (outcome rates in treatment and control groups are plotted on the vertical and horizontal axes)

• Galbraith plot

• Regression analysis

• Comparing the results of fixed and random effect models (a crude assessment of heterogeneity)

Check again that the data are correct

• Check again that the data are correct

• Do not do a meta-analysis

• Ignore heterogeneity (fixed effect model)

• Perform a random effects meta-analysis

• Change the effect measure (e.g. different scale or units)

• Split studies into subgroups

• Investigate heterogeneity using meta-regression

• Exclude studies

A process for re-analysing the same data set

• A process for re-analysing the same data set

• A range of principles used, depends on

• Choice of statistical test
• Inclusion criteria
• Inclusion of both published and unpublished

To investigate whether heterogeneity among results of multiple studies is related to specific characteristics of the studies (e.g. dose rate)

• To investigate whether heterogeneity among results of multiple studies is related to specific characteristics of the studies (e.g. dose rate)

• To investigate whether particular covariate (potential ‘effect modifier’) explain any of the heterogeneity of treatment effect between studies

• Can find out if there is evidence of different effects in different subgroups of trials

• It is appropriate to use meta-regression to explore sources of heterogeneity even if an initial overall test for heterogeneity is non-significant

metareg _ES bcalving acalving full_lact monen_other bstcode apcode, wsse(_seES) bsest(reml)

• metareg _ES bcalving acalving full_lact monen_other bstcode apcode, wsse(_seES) bsest(reml)

• Meta-regression Number of obs = 23

• REML estimate of between-study variance tau2 = .04357

• % residual variation due to heterogeneity I-squared_res = 65.24%

• Proportion of between-study variance explained Adj R-squared = 51.05%

• Joint test for all covariates Model F(6,16) = 3.50

• With Knapp-Hartung modification Prob > F = 0.0209

• -------------------------------------------------------------------------------------------------------------

• _ES | Coef. Std. Err. t P>|t| [95% Conf. Interval]

• -------------+-----------------------------------------------------------------------------------------------

• bcalving | -.0028578 .0031445 -0.91 0.377 -.0095239 .0038083

• acalving | -.0007429 .0013228 -0.56 0.582 -.0035472 .0020613

• full_lact | .3517979 .2544234 1.38 0.186 -.1875556 .8911513

• other s| .3403943 .1278838 2.66 0.017 .0692928 .6114959

• bstcode | -.0333014 .1370641 -0.24 0.811 -.3238642 .2572614

• apcode | .4589506 .1391693 3.30 0.005 .1639249 .7539762

• _cons | -.3564435 .2521411 -1.41 0.177 -.8909586 .1780717

• -----------------------------------------------------------------------------------------------------------

metareg _ES full_lact monen_other apcode, wsse(_seES) bsest(reml)

• metareg _ES full_lact monen_other apcode, wsse(_seES) bsest(reml)

• Meta-regression Number of obs = 23

• REML estimate of between-study variance tau2 = .04134

• % residual variation due to heterogeneity I-squared_res = 66.02%

• Proportion of between-study variance explained Adj R-squared = 53.55%

• Joint test for all covariates Model F(3,19) = 6.63

• With Knapp-Hartung modification Prob > F = 0.0030

• ---------------------------------------------------------------------------------------------------------

• _ES | Coef. Std. Err. t P>|t| [95% Conf. Interval]

• -------------+-------------------------------------------------------------------------------------------

• full_lact | .3138975 .117573 2.67 0.015 .0678144 .5599807

• others | .3640601 .124601 2.92 0.009 .1032672 .6248529

• apcode | .4385834 .1253647 3.50 0.002 .1761921 .700974

• _cons | -.4478162 .1598295 -2.80 0.011 -.7823432 -.1132892

• -------------------------------------------------------------------------------------------------------

+ve results more likely

• +ve results more likely

• To be published (publication bias)
• To be published rapidly (time lag bias)
• To be published in English (language bias)
• To be published more than once (multiple publications bias)
• To be cited by others (citation bias)

Bias arising from the studies included in the review

• Bias arising from the studies included in the review

• Bias arising from the way the review is done

• Publication bias is only one of the possible reasons for asymmetrical funnel plot

• Funnel plot should been seen as a means of examining “small study effect”

Funnel plot

• Funnel plot

• Publication bias exists (asymmetrical)
• Publication bias doesn’t exists (symmetrical)
• For continuous data- Effect size plotted vs SE or sample size
• For dichotomous data- LogOR or RR vs logSE or sample size

• Fail Safe Number (F)

• Z= (∑ ES/1.645)2-N: (where N= no of papers; ∑ ES is summed of effect size over all studies)-
• for calculation of unpublished studies that would be required to negate the results of a significantly positive ES analysis.

Cochrane group suggests that that tests for funnel plot asymmetry should be used in only a minority of meta-analyses (Ioannidis 2007)

• Cochrane group suggests that that tests for funnel plot asymmetry should be used in only a minority of meta-analyses (Ioannidis 2007)

• Begg’s rank correlation test (adjusted rank correlation-low power)

• This test is NOT recommended with any type of data

• Eggers linear regression test (regression analysis-low power)

• This test is mainly recommended for continuous data

Peters (2006) & Harbord (2006) tests

• Peters (2006) & Harbord (2006) tests

• These tests are suitable for dichotomous data with odds ratios

• False-positive results may occur in the presence of substantial between-study heterogeneity

• For dichotomous outcomes with risk ratios (RR) or risk differences (RD)

• Firm guidance is not yet available

Trim and fill method (tail of the side of the funnel plot with smaller trials chopped off)

• Trim and fill method (tail of the side of the funnel plot with smaller trials chopped off)

• Fail safe N (required studies to overturn positive results)

• Modelling for the probability of studies not published

• Conclusion: there is no definite answer for assessing the presence of publication bias

www.stata.com/support/faqs/stat/meta.html

• www.stata.com/support/faqs/stat/meta.html

• Cochrane Collaboration Open learning material for reviewers (2002)

• Higgins et al. (2001). BMJ 327: 557-560

• Sterne et al. (2001). BMJ 323: 101-105

• Whitehead A (2002). Meta-analysis of Controlled Clinical Trials