The Annals of Applied Statistics
- Ann. Appl. Stat.
- Volume 7, Number 1 (2013), 51-80.
Variable selection and sensitivity analysis using dynamic trees, with an application to computer code performance tuning
Robert B. Gramacy, Matt Taddy, and Stefan M. Wild
Full-text: Open access
Abstract
We investigate an application in the automatic tuning of computer codes, an area of research that has come to prominence alongside the recent rise of distributed scientific processing and heterogeneity in high-performance computing environments. Here, the response function is nonlinear and noisy and may not be smooth or stationary. Clearly needed are variable selection, decomposition of influence, and analysis of main and secondary effects for both real-valued and binary inputs and outputs. Our contribution is a novel set of tools for variable selection and sensitivity analysis based on the recently proposed dynamic tree model. We argue that this approach is uniquely well suited to the demands of our motivating example. In illustrations on benchmark data sets, we show that the new techniques are faster and offer richer feature sets than do similar approaches in the static tree and computer experiment literature. We apply the methods in code-tuning optimization, examination of a cold-cache effect, and detection of transformation errors.
Article information
Source
Ann. Appl. Stat. Volume 7, Number 1 (2013), 51-80.
Dates
First available in Project Euclid: 9 April 2013
Permanent link to this document
https://projecteuclid.org/euclid.aoas/1365527190
Digital Object Identifier
doi:10.1214/12-AOAS590
Mathematical Reviews number (MathSciNet)
MR3086410
Zentralblatt MATH identifier
06171263
Keywords
Sensitivity analysis variable selection Bayesian methods Bayesian regression trees CART exploratory data analysis particle filtering computer experiments
Citation
Gramacy, Robert B.; Taddy, Matt; Wild, Stefan M. Variable selection and sensitivity analysis using dynamic trees, with an application to computer code performance tuning. Ann. Appl. Stat. 7 (2013), no. 1, 51--80. doi:10.1214/12-AOAS590. https://projecteuclid.org/euclid.aoas/1365527190
References
- Asuncion, A. and Newman, D. J. (2007). UCI machine learning repository. Available at http://www.ics.uci.edu/~mlearn/MLRepository.html.
- Balaprakash, P., Wild, S. M. and Hovland, P. D. (2011). Can search algorithms save large-scale automatic performance tuning? Procedia Computer Science 4 2136–2145.
- Balaprakash, P., Wild, S. M. and Norris, B. (2012). SPAPT: Search problems in automatic performance tuning. Procedia Computer Science 9 1959–1968.
- Bastos, L. S. and O’Hagan, A. (2009). Diagnostics for Gaussian process emulators. Technometrics 51 425–438.
- Bayarri, M. J., Berger, J. O., Kennedy, M. C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J. A., Lin, C.-H. and Tu, J. (2009). Predicting vehicle crashworthiness: Validation of computer models for functional and hierarchical data. J. Amer. Statist. Assoc. 104 929–943.Mathematical Reviews (MathSciNet): MR2750226
Digital Object Identifier: doi:10.1198/jasa.2009.ap06623 - Blackford, L. S., Demmel, J., Dongarra, J., Duff, I., Hammarling, S., Henry, G., Heroux, M., Kaufman, L., Lumsdaine, A., Petitet, A., Pozo, R., Remington, K. and Whaley, R. C. (2002). An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Software 28 135–151.
- Breiman, L. (1995). Better subset regression using the nonnegative garrote. Technometrics 37 373–384.Mathematical Reviews (MathSciNet): MR1365720
Digital Object Identifier: doi:10.1080/00401706.1995.10484371 - Breiman, L., Friedman, J. H., Olshen, R. and Stone, C. (1984). Classification and Regression Trees. Wadsworth, Belmont, CA.
- Cantoni, E., Flemming, J. M. and Ronchetti, E. (2011). Variable selection in additive models by non-negative garrote. Stat. Model. 11 237–252.Mathematical Reviews (MathSciNet): MR2857594
Digital Object Identifier: doi:10.1177/1471082X1001100304 - Carvalho, C. M., Johannes, M. S., Lopes, H. F. and Polson, N. G. (2010). Particle learning and smoothing. Statist. Sci. 25 88–106.Mathematical Reviews (MathSciNet): MR2741816
Digital Object Identifier: doi:10.1214/10-STS325
Project Euclid: euclid.ss/1280841735 - Chipman, H. A., George, E. I. and McCulloch, R. E. (1998). Bayesian CART model search (with discussion). J. Amer. Statist. Assoc. 93 935–960.Mathematical Reviews (MathSciNet): MR1631325
Zentralblatt MATH: 1072.62650
Digital Object Identifier: doi:10.1080/01621459.1998.10473707 - Chipman, H. A., George, E. I. and McCulloch, R. E. (2002). Bayesian treed models. Machine Learning 48 303–324.Zentralblatt MATH: 0820.68098
- Chipman, H. A., George, E. I. and McCulloch, R. E. (2010). BART: Bayesian additive regression trees. Ann. Appl. Stat. 4 266–298.Mathematical Reviews (MathSciNet): MR2758172
Zentralblatt MATH: 1189.62066
Digital Object Identifier: doi:10.1214/09-AOAS285
Project Euclid: euclid.aoas/1273584455 - Farah, M. and Kottas, A. (2011). Bayesian inference for sensitivity analysis of computer simulators, with an application to radiative transfer models. Technical Report UCSC-SOE-10-15, Univ. California, Santa Cruz.
- Friedman, J. H. (1991). Multivariate adaptive regression splines. Ann. Statist. 19 1–141.Mathematical Reviews (MathSciNet): MR1091842
Zentralblatt MATH: 0765.62064
Digital Object Identifier: doi:10.1214/aos/1176347963
Project Euclid: euclid.aos/1176347963 - Friedman, J. H., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33 1–22.
- George, E. I. and McCulloch, R. E. (1993). Variable selection via Gibbs sampling. J. Amer. Statist. Assoc. 88 881–889.
- Gramacy, R. B. and Polson, N. G. (2011). Particle learning of Gaussian process models for sequential design and optimization. J. Comput. Graph. Statist. 20 102–118.
- Gramacy, R. B. and Polson, N. G. (2012). Simulation-based regularized logistic regression. Bayesian Anal. 7 1–24.
- Gramacy, R. B. and Taddy, M. A. (2010). Categorical inputs, sensitivity analysis, optimization and importance tempering with tgp version 2, an R package for treed Gaussian process models. Journal of Statistical Software 33 1–48.Mathematical Reviews (MathSciNet): MR2578072
Digital Object Identifier: doi:10.1007/s11222-008-9108-5 - Gramacy, R. B. and Taddy, M. A. (2011). dynaTree: Dynamic trees for learning and design. R package version 2.0.
- Gu, C. (2002). Smoothing Spline ANOVA Models. Springer, New York.Mathematical Reviews (MathSciNet): MR1876599
- Haaland, B. and Qian, P. Z. G. (2011). Accurate emulators for large-scale computer experiments. Ann. Statist. 39 2974–3002.Mathematical Reviews (MathSciNet): MR3012398
Zentralblatt MATH: 1246.65172
Digital Object Identifier: doi:10.1214/11-AOS929
Project Euclid: euclid.aos/1327413775 - Hartono, A., Norris, B. and Sadayappan, P. (2009). Annotation-based empirical performance tuning using orio. In Proceedings of the IEEE International Symposium on Parallel Distributed Processing, 2009 (IPDPS 2009) 1–11. IEEE, New York.
- Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, New York.Mathematical Reviews (MathSciNet): MR2722294
- Huang, J., Horowitz, J. L. and Wei, F. (2010). Variable selection in nonparametric additive models. Ann. Statist. 38 2282–2313.Mathematical Reviews (MathSciNet): MR2676890
Zentralblatt MATH: 1202.62051
Digital Object Identifier: doi:10.1214/09-AOS781
Project Euclid: euclid.aos/1278861249 - Jones, D. R., Schonlau, M. and Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. J. Global Optim. 13 455–492.
- Krishnapuram, B., Carin, L., Figueiredo, M. and Hartemink, A. (2005). Sparse multinomial logistic regression: Fast algorithms and generalization bounds. IEEE Transactions on Pattern Analysis and Machine Intelligence 27 957–969.
- Lee, H. K. H., Sansó, B., Zhou, W. and Higdon, D. M. (2008). Inference for a proton accelerator using convolution models. J. Amer. Statist. Assoc. 103 604–613.Mathematical Reviews (MathSciNet): MR2523997
Zentralblatt MATH: 05564514
Digital Object Identifier: doi:10.1198/016214507000000833 - Linkletter, C., Bingham, D., Hengartner, N., Higdon, D. and Ye, K. Q. (2006). Variable selection for Gaussian process models in computer experiments. Technometrics 48 478–490.Mathematical Reviews (MathSciNet): MR2328617
Digital Object Identifier: doi:10.1198/004017006000000228 - Maity, A. and Lin, X. (2011). Powerful tests for detecting a gene effect in the presence of possible gene-gene interactions using garrote kernel machines. Biometrics 67 1271–1284.Mathematical Reviews (MathSciNet): MR2872377
Digital Object Identifier: doi:10.1111/j.1541-0420.2011.01598.x - Marrel, A., Iooss, B., Laurent, B. and Roustant, O. (2009). Calculations of Sobol indices for the Gaussian process metamodel. Reliability Engineering and System Safety 94 742–751.
- Morgan, J. N. and Sonquist, J. A. (1963). Problems in the analysis of survey data, and a proposal. J. Amer. Statist. Assoc. 58 415–434.
- Morris, R. D., Kottas, A., Taddy, M., Furfaro, R. and Ganapol, B. (2008). A statistical framework for the sensitivity analysis of radiative transfer models. IEEE Transactions on Geoscience and Remote Sensing 12 4062–4074.
- Oakley, J. E. and O’Hagan, A. (2004). Probabilistic sensitivity analysis of complex models: A Bayesian approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 66 751–769.Mathematical Reviews (MathSciNet): MR2088780
Zentralblatt MATH: 1046.62027
Digital Object Identifier: doi:10.1111/j.1467-9868.2004.05304.x - Patterson, D. A. and Hennessy, J. L. (2007). Computer Organization and Design—the Hardware / Software Interface, 3rd ed. Morgan Kaufmann, Boston.
- Reich, B. J., Storlie, C. B. and Bondell, H. D. (2009). Variable selection in Bayesian smoothing spline ANOVA models: Application to deterministic computer codes. Technometrics 51 110–120.
- Saltelli, A. (2002). Making best use of model evaluations to compute sensitivity indices. Comput. Phys. Comm. 145 280–297.
- Saltelli, A., Chan., K. and Scott, E. M., eds. (2000). Sensitivity Analysis. Wiley, Chichester.Mathematical Reviews (MathSciNet): MR1886391
- Saltelli, A. and Tarantola, S. (2002). On the relative importance of input factors in mathematical models: Safety assessment for nuclear waste disposal. J. Amer. Statist. Assoc. 97 702–709.Mathematical Reviews (MathSciNet): MR1973688
Zentralblatt MATH: 1073.62602
Digital Object Identifier: doi:10.1198/016214502388618447 - Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M. and Tarantola, S. (2008). Global Sensitivity Analysis. The Primer. Wiley, Chichester.
- Sang, H. and Huang, J. Z. (2012). A full scale approximation of covariance functions for large spatial data sets. J. R. Stat. Soc. Ser. B Stat. Methodol. 74 111–132.Mathematical Reviews (MathSciNet): MR2885842
Digital Object Identifier: doi:10.1111/j.1467-9868.2011.01007.x - Santner, T. J., Williams, B. J. and Notz, W. I. (2003). The Design and Analysis of Computer Experiments. Springer, New York.
- Storlie, C. B., Swiler, L. P., Helton, J. C. and Sallaberry, C. J. (2009). Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models. Reliability Engineering & System Safety 94 1735–1763.
- Taddy, M. A., Gramacy, R. B. and Polson, N. G. (2011). Dynamic trees for learning and design. J. Amer. Statist. Assoc. 106 109–123.Mathematical Reviews (MathSciNet): MR2816706
Digital Object Identifier: doi:10.1198/jasa.2011.ap09769 - Taddy, M. A., Lee, H. K. H., Gray, G. A. and Griffin, J. D. (2009). Bayesian guided pattern search for robust local optimization. Technometrics 51 389–401.
- Yi, G., Shi, J. Q. and Choi, T. (2011). Penalized Gaussian process regression and classification for high-dimensional nonlinear data. Biometrics 67 1285–1294.Mathematical Reviews (MathSciNet): MR2872378
Digital Object Identifier: doi:10.1111/j.1541-0420.2011.01576.x - Ziehn, T. and Tomlin, A. S. (2009). GUI-HDMR—a software tool for global sensitivity analysis of complex models. Environmental Modelling and Software 24 775–785.
- You have access to this content.
- You have partial access to this content.
- You do not have access to this content.
More like this
- Nonlinear Dynamics in a Cournot Duopoly with Different Attitudes towards Strategic Uncertainty
Fanti, Luciano, Gori, Luca, and Sodini, Mauro, Abstract and Applied Analysis, 2013 - Power-Expected-Posterior Priors for Generalized Linear Models
Fouskakis, Dimitris, Ntzoufras, Ioannis, and Perrakis, Konstantinos, Bayesian Analysis, 2018 - Faithful variable screening for high-dimensional convex regression
Xu, Min, Chen, Minhua, and Lafferty, John, The Annals of Statistics, 2016
- Nonlinear Dynamics in a Cournot Duopoly with Different Attitudes towards Strategic Uncertainty
Fanti, Luciano, Gori, Luca, and Sodini, Mauro, Abstract and Applied Analysis, 2013 - Power-Expected-Posterior Priors for Generalized Linear Models
Fouskakis, Dimitris, Ntzoufras, Ioannis, and Perrakis, Konstantinos, Bayesian Analysis, 2018 - Faithful variable screening for high-dimensional convex regression
Xu, Min, Chen, Minhua, and Lafferty, John, The Annals of Statistics, 2016 - Estimating Transmission from Genetic and Epidemiological Data: A Metric to Compare Transmission Trees
Kendall, Michelle, Ayabina, Diepreye, Xu, Yuanwei, Stimson, James, and Colijn, Caroline, Statistical Science, 2018 - Longitudinal functional principal component analysis
Greven, Sonja, Crainiceanu, Ciprian, Caffo, Brian, and Reich, Daniel, Electronic Journal of Statistics, 2010 - Recent Development in Rigorous Computational Methods in Dynamical Systems
Arai, Zin, Kokubu, Hiroshi, and Pilarczyk, Paweł, Japan Journal of Industrial and Applied Mathematics, 2009 - Simple approximate MAP inference for Dirichlet processes mixtures
Raykov, Yordan P., Boukouvalas, Alexis, and Little, Max A., Electronic Journal of Statistics, 2016 - Distributional results for thresholding estimators in high-dimensional Gaussian regression models
Pötscher, Benedikt M. and Schneider, Ulrike, Electronic Journal of Statistics, 2011 - Efficient, adaptive cross-validation for tuning
and comparing models, with application to drug discovery
Shen, Hui, Welch, William J., and Hughes-Oliver, Jacqueline M., The Annals of Applied Statistics, 2011 - Nonparametric independence screening and structure identification for ultra-high dimensional longitudinal data
Cheng, Ming-Yen, Honda, Toshio, Li, Jialiang, and Peng, Heng, The Annals of Statistics, 2014