A monte carlo experiment to study the curse of dimensionality in the multivariate probit model
DESIGN OF THE SIMULATION EXPERIMENT
Yüklə 294,82 Kb.
|
- Bu səhifədəki naviqasiya:
- square root of mean squared error (RMSE)
- PERFORMANCE EVALUATION RESULTS
- 4.1 Accuracy of Parameter Recovery
- 4.2 Precision in Estimation Across Approaches
- 4.3 Root Mean Squared Error (RMSE) and Coverage Probability (CP)
- ACKNOWLEDGEMENTS
DESIGN OF THE SIMULATION EXPERIMENT We consider MNP models with five alternatives and five independent variables (and five random coefficients, one on each of the independent variables) for the following four data generation processes: (a) Cross-sectional data without correlation among random parameters, (b) Cross-sectional data with correlation among random parameters, (c) Panel data without correlation among random parameters, and (d) Panel data with correlation among random parameters. Consider a “true” or underlying probit model according to the utility function: 
Finally, each kernel error term For the cross-sectional datasets, we generate a sample of 2500 realizations of the five independent variables corresponding to 2500 individuals. For the panel datasets, we generate a sample of 2500 realizations of the five independent variables corresponding to a situation where 500 individuals each have five choice occasions for a total of 2500 choice occasions. These are combined with different realizations of For each of the above 200 datasets, we estimate the MNP using each of the estimation approaches. For the MACML approach, we used a single random permutation as discussed in Section 2.1.1. For estimating models with the GHK-Halton approach, we used 500 Halton draws. For the GHK-SGI approach, to keep the estimation times similar to the GHK-Halton approach, we computed the MVN integral over 351 and 589 supports points for cross-sectional and panel cases, respectively. For the Bayesian approach, we used 50,000 MCMC draws for all four cases with a burn-in of the first 500 elements of the chain.4 Finally, for all the three freqentist methods, standard errors of parameter estimates (for each dataset) were computed using the Godambe (1960) sandwich estimator ( To measure the performance of each estimation method, we computed performance metrics as described below.
 .
If a true parameter value is zero, the APB is computed by taking the difference of the mean estimate from the true value (= 0), dividing this difference by the value of 1 in the denominator, and multiplying by 100.
 ,
where, CP is the coverage probability,
Table 1 presents an overall summary of the performance of all the estimation approaches considered in this study – MACML, GHK-Halton, GHK-SGI, and MCMC – for all four cases of the MNP data generation process – cross-sectional uncorrelated, cross-sectional correlated, panel uncorrelated, and panel correlated.5 Note that two different columns are reported for the GHK-Halton method for panel data settings. One of them corresponds to the traditional GHK-Halton-FIML approach and the other corresponds to the GHK-Halton-CML approach. For each of the estimation methods and data settings, the first block of rows in Table 1 presents the average APB value (across all parameters), as well as the average APB value computed separately for the mean (the b vector) parameters and the covariance matrix (the 4.1 Accuracy of Parameter Recovery The APB measures in the first block of Table 1 provide several important insights. The MACML approach outperforms other inference approaches for all the four cases of data generation. This underscores the superiority of the MACML approach in accurately recovering model parameters. In all inference approaches, the overall APB increases as we move from the cross-sectional to panel case, and from the uncorrelated to the correlated case. But, even here, the MACML shows the least APB dispersion among the many data generation cases, while the MCMC and GHK-SGI approaches show the highest dispersions among the data generation cases. The most striking observation is the rapid degradation of the MCMC approach between the uncorrelated and correlated random coefficients cases, for both cross-sectional and panel data sets. The MCMC has the worst APB of all inference approaches (and by a substantial margin) in the correlated random coefficients setting in the cross-sectional case, and the second worst APB in the correlated random coefficients in the panel case. In terms of the performance of the GHK-SGI approach, the most striking observation is the substantially poor performance of the GHK-SGI approach (in combination with the CML approach) in the panel cases relative to the performance of the GHK-SGI approach in the cross-sectional cases.6 The APB values from the GHK-Halton approach for the cross-sectional cases are higher than (but in the same order of APB) as the other two frequentist (MACML and GHK-SGI) approaches for the cross-sectional cases. For the panel cases, as already discussed, we implement both an FIML version (labeled as GHK Halton-FIML in Table 1) as well as a CML version (labeled as GHK Halton-CML) of the GHK-Halton approach. Both these GHK Halton versions provide an APB that is higher than the MACML approach, but are superior to the GHK-SGI and MCMC approaches in terms of recovering parameters accurately. Between the FIML and CML versions of this GHK-Halton approach, the latter approach recovers the parameters more accurately; the APB for the GHK-Halton-FIML simulator is 30-50% higher than the GHK-Halton CML simulator. This is a manifestation of the degradation of simulation techniques to evaluate the MVNCD function as the number of dimensions of integration increases. The results clearly show the advantage of combining the traditional GHK simulator with the CML inference technique for panel data, although the MACML approach still dominates over the GHK-Halton CML approach. The split of the APB by the mean and covariance parameters follow the overall APB trends rather closely. Not surprisingly, except for the MCMC approach with uncorrelated cross-sectional data, it is more difficult to recover the covariance parameters accurately relative to the mean parameters. For the frequentist methods, this is a reflection of the appearance of the covariance parameters in a much more complex non-linear fashion than the mean parameters in the likelihood function, leading to a relatively flat log-likelihood function for different covariance parameter values and more difficulty in accurately recovering these parameters. But the most noticeable observation from the mean and covariance APB values is the difference between these for the MCMC method with correlated random coefficients. In fact, it becomes clear now that the substantially higher overall APB for the MCMC approach (relative to the MACML and GHK-Halton approaches) for the correlated random coefficients case is primarily driven by the poor MCMC ability to recover the covariance parameters, suggesting increasing difficulty in drawing efficiently from the joint posterior distribution of parameters (through a sequence of conditioning mechanisms) when there is covariance in the parameters. In summary, from the perspective of recovering parameters accurately, the MACML outperforms other approaches for all the four data generation cases. The GHK-Halton also does reasonably well across the board, with the GHK-Halton-CML doing better than the GHK-Halton-FIML for the panel cases. The GHK-SGI is marginally better than the GHK-Halton for the cross-sectional cases, but, when combined with the CML approach, is the worst in the panel cases. The MCMC approach’s ability to recover parameters is in the same range as the approaches involving the GHK-Halton for the uncorrelated random coefficients cases, but deteriorates substantially in the presence of correlated random coefficients (note also that 50,000 iterations are used in the MCMC approach in the current paper, more than the 5,000-15,000 iterations typically used in earlier MCMC estimations of the MNP; see, for example, Chib et al., 1998; Johndrow et al., 2013; Jiao and van Dyk, 2015).
We now turn to standard errors. The FSSE values are useful for assessing the empirical (finite-sample) efficiency (or precision) of the different estimators, while the ASE values provide efficiency results as the sample size gets very large. The ASE values essentially provide an approximation to the FSSE values for finite samples. Table 1 indicates that the MCMC estimator has the advantage of good efficiency (lowest FSSE and ASE) for the cross-sectional, uncorrelated random coefficients case, but the GHK-SGI wins the finite-sample efficiency battle (lowest FSSE) for all the remaining three cases.7 In terms of ASE, the MACML has the lowest value for the cross-sectional correlated case, while the GHK-SGI has the lowest value for the panel cases. Such a high precision in the estimates in the GHK-SGI, however, is not of much use because of the rather high finite sample bias (APB) in the parameter estimates of the GHK-SGI approach. In all cases, the MACML does very well too in terms of closeness to the approach with the lowest FSSE and ASE. Of particular note is that the MACML estimator’s efficiency in terms of both FSSE and ASE is better than the traditional frequentist GHK-Halton simulator for all cases, except in the panel data-uncorrelated random coefficients case. Additionally, the MACML estimator’s efficiency, while not as good as that of the MCMC in the two uncorrelated coefficients cases, is better than the MCMC for the two correlated coefficients cases. For all the frequentist methods, the FSSE and ASE values across all parameters are smaller in the presence of correlation among random parameters than without correlation. As can be observed from the third rows of the tables under FSSE and ASE in Table 1, this pattern is driven by the smaller FSSE and ASE values for the covariance parameters in the correlated case relative to the non-correlated case. As discussed in Bhat et al. (2010), it may be easier to retrieve covariance parameters with greater precision at higher values of covariance because, at lower correlation values, the likelihood surface tends to be flat, increasing the variability in parameter estimation. This trend, however, reverses for the MCMC method, with the FSSE and ASE values being higher in data settings with correlated random parameters than those with non-correlated random parameters, presumably for the same reason that the APB values in the MCMC method are very high in the correlated coefficients case relative to the uncorrelated coefficients case. Across all inference approaches, a consistent result is that the FSSE and ASE are smaller for the mean parameters than the covariance parameters. Also, the closeness of the FSSE and ASE values for the frequentist approaches suggest that the inverse of the Godambe sandwich estimator serves as a good approximation to the finite sample efficiency for the sample size considered in this paper. The FSSE and ASE are also close for the MCMC approach. Overall, in terms of estimator efficiency, it appears that all inference approaches do reasonably well. There are also some more general takeaways from the examination of the FSSE and ASE values. First, while the full-information maximum likelihood approach is theoretically supposed to be more asymptotically efficient than the limited-information composite marginal likelihood approach (see a proof for this in Bhat, 2015), this result does not necessarily extend to the case when there is no clear analytically tractable expression for the probabilities of choice in a discrete choice model. This is illustrated in the FSSE/ASE estimates from the GHK Halton-FIML and GHK Halton-CML approaches for panel data in Table 1, with the latter proving to be a more efficient estimator than the former. At a fundamental level, when any kind of an approximation is needed (either through simulation methods or analytically) for the choice probabilities, the efficiency results will also depend on how accurately the objective function (the log-likelihood function in FIML and the composite log-likelihood in CML) can be evaluated. The CML approach has lower dimensional integrals, which can be evaluated more accurately than the higher dimensional integrals in the FIML approach, and this can lead to a more efficient CML estimator (as is the case in Table 1). Second, the MACML estimator’s efficiency is consistently better than that of the GHK-Halton based simulator for the range of data settings considered in this paper. In combination with the superior performance of the MACML in terms of parameter recovery, this lends reinforcement to our claim that accuracy of evaluating the objective function (as a function of the parameters to be estimated) does play a role in determining estimator efficiency. Third, while Bayesian estimators are typically invoked on the grounds of good small sample inference properties in terms of higher efficiency relative to frequentist estimators in finite samples, our results indicate that, at least for the sample size considered in this paper, this all depends on the context and is certainly not a foregone conclusion empirically. For instance, while the MCMC approach leads to lower FSSE/ASE values than the MACML approach for the uncorrelated coefficients cases, the MACML leads to lower FSSE/ASE values than the MCMC approach for the correlated coefficients cases. 4.3 Root Mean Squared Error (RMSE) and Coverage Probability (CP) The RMSE measure combines the bias and efficiency considerations into a single metric, as discussed in Section 3. The results indicate that the MACML approach has the lowest RMSE values for all the four data generation cases. The GHK-SGI approach is the next best for the cross-sectional cases, but is the worst (and by a substantial margin) for the panel cases. The MCMC approach and the GHK-Halton approach are comparable to each other in the cross-sectional uncorrelated coefficients (first) case in Table 1, and both of these are also comparable to the performance of the GHK-SGI approach in this first case. For the panel uncorrelated coefficients (third) case, the MCMC has an RMSE value comparable to the FIML version of the GHK-Halton, but fares clearly worse than the CML version of the GHK-Halton. Of course, for both the correlated coefficients cases (second and fourth cases), the MCMC is not a contender at all based on our analysis. The coverage probability (CP) values help assess how the parameter estimates spread about the true parameter value. As one may observe from Table 1, all approaches provide good empirical coverage of the 80% nominal confidence interval in the cross-sectional uncorrelated case (all the values are above 80%). The MCMC falls short in the cross-sectional correlated random coefficients case. For the panel cases, the MACML and the GHK-Halton CML approaches are the only two that cover or come very close to covering the 80% confidence interval, with the MACML clearly providing better coverage than the GHK-Halton CML. These results are generally in line with the RMSE value trends. Overall, based on the RMSE and CP values, the MACML approach is the clear winner across all data generation cases. In terms of stability in performance across all cases, the GHK-Halton turns out to be the second best inference approach in our results (when used in combination with the CML approach in the two cases of panel data). 4.4 Computation Time The last block of Table 1 provides model estimation times (or run times) for different estimation methods explored in this study. The total run time for the frequentist methods include both the time taken for parameter convergence as well as for the computation of asymptotic standard errors (ASEs) using the Godambe sandwich estimator. The run times reported for the MCMC approach does not include ASE computation; instead, it involves a simple standard deviation of the posterior distribution. The computer configuration used to conduct these tests is: Intel Xeon® CPU E5-1620 @3.70GHz, Windows 7 Enterprise (64 bit), 16.0 GB RAM. Also, all the estimations were performed using codes written in the Gauss matrix programming suite to ensure comparability. The results in Table 1 shows that the convergence times for the MACML approach is the lowest for the cross-sectional datasets, with the time for ASE computation being about half of the convergence time for the uncorrelated random coefficients case and a fifth of the convergence time for the correlated random coefficients case. The other two frequentist approaches take about the same time as the MACML. However, the time for the Bayesian MCMC approach is substantially higher in the cross-sectional cases (about five times the MACML estimation time for the uncorrelated coefficients case and 2.5 times the MACML estimation time for the correlated coefficients case). As also observed by Train (2009), we found little change in the MCMC estimation time between the uncorrelated and correlated coefficients cases. For the panel cases, the MACML is the fastest approach in terms of convergence time, though the GHK-Halton implemented with the CML approach has a comparable convergence time. The other two frequentist approaches (GHK-Halton with FIML and the GHK-SGI CML) have a much higher convergence time relative to the MACML and GHK-Halton CML approaches. The MCMC convergence times are in the same range as the MACML and GHK-Halton-CML. However, the MCMC has the advantage that the ASE estimates of parameters are obtained directly from the posterior distribution of parameters at convergence. For the frequentist methods, however, the ASE computation involves the computation of the inverse of the Godambe information matrix, which itself involves the computation of the Hessian matrix that is time consuming (the ASE computation time for the MACML approach, for example, is about twice the time needed for parameter convergence in the two panel cases). When the ASE computation time is added in for the frequentist methods, the MCMC has a speed advantage by a factor of about 2.5 relative to the MACML. The problem, though, is that the MCMC fares much more poorly compared to the MACML (and GHK-Halton CML) approaches for panel data in terms of parameter recovery accuracy and precision, as evident in the RMSE and CP measures. 5. Summary and Conclusions Multinomial Probit (MNP) models are gaining increasing interest for choice model estimation in transportation and other fields. This paper has presented an extensive simulation experiment to evaluate different estimation techniques for MNP models. While one cannot make conclusive statements applicable for all possible data generation settings in terms of number of choice alternatives, correlation structures, and sample sizes, the simulations we have undertaken do provide some key insights that could be used to guide MNP estimation. Overall, taking all the three metrics (accuracy and precision of parameter recovery and estimation time) into consideration, the MACML approach provided the best performance for the data generation settings examined in this study. These results indicate the promise of this approach for estimating MNP models in different settings. The GHK-Halton simulation, when used in conjunction with the CML approach (for panel models), yielded the second best performance in recovering the parameters. On the other hand, the bias in parameter estimation was more than double that of the MACML approach when the GHK-Halton simulator was used in its original FIML form for panel data models. In fact, the GHK-Halton when combined with the FIML estimator for panel data sets was also less efficient than the GHK-Halton in conjunction with the CML estimator, highlighting the fact that the FIML estimator’s theoretical efficiency superiority over the CML estimator may not get manifested in empirical samples when the objective function to be maximized is analytically intractable. In such cases, the accuracy of evaluating the objective function is also important. In the current paper, the CML involves lower-dimensional integrals than the FIML, and the ability to evaluate the lower dimensional integrals more accurately leads to more precision in the CML estimator relative to the FIML estimator. These results highlight the potential for gainful applicability of the CML approach with the traditional GHK simulator. The GHK-based sparse grid integration approach performed well in the cross-sectional cases, but very poorly for panel datasets when combined with the CML approach. These results suggest that the approach may not be applicable for settings with higher than 5 dimensional integrals or panel data settings (see Abay, 2015 for a similar finding). The MCMC approach performed very well for the cross-sectional data without correlation in the parameters and appears to be a good alternative approach to use for such a data setting. But, even in this case, the MACML approach dominates in terms of accuracy and precision of parameter recovery, as well as has a speed advantage by a factor of about five relative to the MCMC approach. Our simulations also indicate a notable limitation of the MCMC approach in recovering MNP parameters in cases where the random coefficients are correlated (both in the cross-sectional and panel settings). This finding needs to be further investigated to examine ways to improve the MCMC method in the presence of correlated random coefficients. The results in this paper are encouraging in that the emerging methods – MACML, CML, and MCMC – are making the estimation of MNP models easier and faster than before. But there is a need for continued simulation experimentation with these alternative methods to provide more general guidance under a wider variety of data settings, including different numbers of alternatives, different sample sizes, different numbers of repeated choice occasions in the panel case, a range of correlation structures across coefficients and choice occasions, and different numbers of exogenous variables and types of exogenous variables (including discrete and binary variables). Also, there are a variety of potential ways to improve upon the MACML and MCMC approaches in particular, such as alternative analytic approximations for the MVNCD function (see Trinh and Genz, 2015) in the MACML, and reducing MACML computation time by sampling pairings for an individual rather than using the full set of pairings as done here. Finally, future research needs to investigate ways to improve the MCMC performance in correlated random coefficients cases and consider Imai and van Dyk’s (2005) method of scaling utilities at the beginning of the estimation.
Two anonymous reviewers provided useful comments on an earlier version of this paper. References Abay, K. A. (2015). Evaluating simulation-based approaches and multivariate quadrature on sparse grids in estimating multivariate binary probit models. Economics Letters, 126, 51-56. Bhat, C. R. (2001). Quasi-random maximum simulated likelihood estimation of the mixed multinomial logit model. Transportation Research Part B, 35(7), 677-693. Bhat, C. R. (2003). Simulation estimation of mixed discrete choice models using randomized and scrambled Halton sequences. Transportation Research Part B, 37(9), 837-855. Bhat, C.R., (2011). The maximum approximate composite marginal likelihood (MACML) estimation of multinomial probit-based unordered response choice models. Transportation Research Part B, 45 (7), 923-939. Bhat, C.R. (2014). The composite marginal likelihood (CML) inference approach with applications to discrete and mixed dependent variable models. Foundation and Trends in Econometrics, 7(1), 1-117.
Burgette, L. F., and Reiter, J. P. (2013). Multiple-shrinkage multinomial probit models with applications to simulating geographies in public use data. Bayesian Analysis, 8(2), 453-478. Chib, S., Greenberg, E., and Chen, Y. (1998). MCMC methods for fitting and comparing multinomial response models. Economics Working Paper Archive, Econometrics, 9802001.
Daganzo, C. (1979). Multinomial Probit: the Theory and its Application to Demand Forecasting. Academic Press, New York. Gelman, A., and Shirley, K. (2011). Inference from simulations and monitoring convergence. In Brooks, S. P., Gelman, A., Jones, G., and Meng, X-L. (Eds.) Handbook of Markov Chain Monte Carlo, Ch. 6, 163-174. Chapman Hall, Boca Raton, FL.
Hajivassiliou, V., McFadden, D., and Ruud, P. (1996). Simulation of multivariate normal rectangle probabilities and their derivatives: Theoretical and computational results. Journal of Econometrics, 72, 85-134. Heiss, F. (2010). The panel probit model: Adaptive integration on sparse grids. In Greene, W.H. and Hill, R.C. (Eds.) Advances in Econometrics: Maximum Simulated Likelihood Methods and Applications, Vol. 26, 41-64, Emerald Group Publishing Limited. Heiss, F., and Winschel, V. (2008). Likelihood approximation by numerical integration on sparse grids. Journal of Econometrics, 144(1), 62-80. Imai, K., and van Dyk, D. A. (2005). A Bayesian analysis of the multinomial probit model using marginal data augmentation. Journal of Econometrics, 124(2), 311-334. Jiao, X., and van Dyk, D. A. (2015). A corrected and more efficient suite of MCMC samplers for the multinomal probit model. arXiv preprint arXiv:1504.07823. Joe, H. (1995). Approximations to multivariate normal rectangle probabilities based on conditional expectations. Journal of the American Statistical Association, 90(431), 957-964. Johndrow, J., Dunson, D. B., and Lum, K. (2013). Diagonal orthant multinomial probit models. Proceedings of the 16th International Conference on Artificial Intelligence and Statistics (AISTATS), Scottsdale, AZ, Journal of Machine Learning Research (JMLR) Workshop and Conference Proceedings, Vol. 31, 29-38. McCulloch, R. R., Polson, N. G., and Rossi, P. E. (2000). Bayesian analysis of the multinomial probit model with fully identified parameters. Journal of Econometrics, 99, 173-193. McFadden, D., and Train, K. (2000). Mixed MNL models for discrete response. Journal of Applied Econometrics, 15(5), 447-470. Richard, J. F., and Zhang, W. (2007). Efficient high-dimensional importance sampling. Journal of Econometrics, 141(2), 1385-1411. Sándor, Z., and András, P. (2004). Alternative sampling methods for estimating multivariate normal probabilities. Journal of Econometrics, 120(2), 207-234. Solow, A. R. (1990). A method for approximating multivariate normal orthant probabilities. Journal of Statistical Computation and Simulation, 37(3-4), 225-229. Smolyak, S. A. (1963). Quadrature and interpolation formulas for tensor products of certain classes of functions. Soviet Mathematics Doklady, 4, 240-243. Train, K. E. (2009). Discrete Choice Methods with Simulation, 2nd ed. Cambridge University Press, Cambridge. Trinh, G., and Genz, A.(2015). Bivariate conditioning approximations for multivariate normal probabilities. Statistics and Computing, 25, 989-996. Varin, C. (2008). On composite marginal likelihoods. AStA Advances in Statistical Analysis, 92(1), 1-28. Wang, Y., Kockelman, K. M., and Damien, P. (2014). A spatial autoregressive multinomial probit model for anticipating land-use change in Austin, Texas. The Annals of Regional Science, 52(1), 251-278. Zhang, X., Boscardin, W. J., and Belin, T. R. (2008). Bayesian analysis of multivariate nominal measures using multivariate multinomial probit models. Computational Statistics & Data Analysis, 52(7), 3697-3708. Yüklə 294,82 Kb. Dostları ilə paylaş:
|
as in Equation (1). For all the datasets generated in this study, the values of each of the five independent variables 
for the alternatives are drawn from a standard normal distribution. We allow random coefficients on all the five independent variables in 
That is, 
is a vector of normally distributed coefficients with mean 
and covariance matrix 
. For the case of uncorrelated random parameters, we assume a diagonal covariance matrix with all the diagonal elements set to a value of 1, entailing the estimation of five diagonal elements (albeit all are of value 1). For the case of correlated random parameters, the matrix 
(q = 1, 2, …, Q; t = 1, 2, 3, ..., T; i = 1, 2, …, I) is generated from a univariate normal distribution with a variance of 0.5. 
, where H is the Hessian matrix and J is the sandwich matrix). The Hessian and sandwich matrices were computed at the convergent parameters using analytic expressions that a. For the MCMC method, the standard errors of the parameter estimates (for each dataset) were calculated as the standard deviation of the parameter’s posterior distribution at convergence.
is the estimated value of the parameter in dataset r, 
is the true value of the parameter, 
is the asymptotic standard error (ASE) of the parameter in the dataset r, 
is an indicator function which takes a value of 1 if the argument in the bracket is true (otherwise 0), N is the number of datasets (200), and 
is the t-statistic value for a given confidence level 
We compute CP values for 80% nominal coverage probability (i.e., 
). CP is the empirical probability that a confidence interval contains the true parameter (i.e., the proportion of confidence intervals across the 200 datasets that contain the true parameter). CP values smaller than the nominal confidence level (80% in our study) suggest that the confidence intervals do not provide sufficient empirical coverage of the true parameter.