In this paper, we review state-of-the-art methods for feature selection in statistics with an application-oriented eye. Indeed, sparsity is a valuable property and the profusion of research on the topic might have provided little guidance to practitioners. We demonstrate empirically how noise and correlation impact both the accuracy—the number of correct features selected—and the false detection—the number of incorrect features selected—for five methods: the cardinality-constrained formulation, its Boolean relaxation, $\ell _{1}$ regularization and two methods with non-convex penalties. A cogent feature selection method is expected to exhibit a two-fold convergence, namely the accuracy and false detection rate should converge to $1$ and $0$ respectively, as the sample size increases. As a result, proper method should recover all and nothing but true features. Empirically, the integer optimization formulation and its Boolean relaxation are the closest to exhibit this two properties consistently in various regimes of noise and correlation. In addition, apart from the discrete optimization approach which requires a substantial, yet often affordable, computational time, all methods terminate in times comparable with the glmnet package for Lasso. We released code for methods that were not publicly implemented. Jointly considered, accuracy, false detection and computational time provide a comprehensive assessment of each feature selection method and shed light on alternatives to the Lasso-regularization which are not as popular in practice yet.

In exciting recent work, Bertsimas, King and Mazumder (Ann. Statist. 44 (2016) 813–852) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes than what was thought possible in the statistics community. They presented empirical comparisons of best subset with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset consistently outperformed both methods in terms of prediction accuracy. Here, we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows:

•neither best subset nor the lasso uniformly dominate the other, with best subset generally performing better in very high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes;

•for a large proportion of the settings considered, best subset and forward stepwise perform similarly, but in certain cases in the high SNR regime, best subset performs better;

•forward stepwise and best subsets tend to yield sparser models (when tuned on a validation set), especially in the high SNR regime;

•the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen (Comput. Statist. Data Anal. 52 (2007) 374–393)) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and nearly as well as best subset in high SNR scenarios.

Sparse linear regression is a vast field and there are many different algorithms available to build models. Two new papers published in Statistical Science study the comparative performance of several sparse regression methodologies, including the lasso and subset selection. Comprehensive empirical analyses allow the researchers to demonstrate the relative merits of each estimator and provide guidance to practitioners. In this discussion, we summarize and compare the two studies and we examine points of agreement and divergence, aiming to provide clarity and value to users. The authors have started a highly constructive dialogue, our goal is to continue it.

We congratulate the authors Bertsimas, Pauphilet and van Parys (hereafter BPvP) and Hastie, Tibshirani and Tibshirani (hereafter HTT) for providing fresh and insightful views on the problem of variable selection and prediction in linear models. Their contributions at the fundamental level provide guidance for more complex models and procedures.

Exponential-family Random Graph Models (ERGMs) constitute a large statistical framework for modeling dense and sparse random graphs with short- or long-tailed degree distributions, covariate effects and a wide range of complex dependencies. Special cases of ERGMs include network equivalents of generalized linear models (GLMs), Bernoulli random graphs, $\beta $-models, $p_{1}$-models and models related to Markov random fields in spatial statistics and image processing. While ERGMs are widely used in practice, questions have been raised about their theoretical properties. These include concerns that some ERGMs are near-degenerate and that many ERGMs are non-projective. To address such questions, careful attention must be paid to model specifications and their underlying assumptions, and to the inferential settings in which models are employed. As we discuss, near-degeneracy can affect simplistic ERGMs lacking structure, but well-posed ERGMs with additional structure can be well-behaved. Likewise, lack of projectivity can affect non-likelihood-based inference, but likelihood-based inference does not require projectivity. Here, we review well-posed ERGMs along with likelihood-based inference. We first clarify the core statistical notions of “sample” and “population” in the ERGM framework, separating the process that generates the population graph from the observation process. We then review likelihood-based inference in finite, super and infinite population scenarios. We conclude with consistency results, and an application to human brain networks.

J. Stuart (Stu) Hunter has been an inspiration and mentor to a generation of statisticians, especially to those working in industry. Born on June 3, 1923 in Holyoke, Massachusetts, Stu moved to Linden, New Jersey at the age of 2 where he spent the rest of his childhood, graduating from high school at age 16. After receiving a bachelor’s degree in Electrical Engineering in 1947, he went on to receive a master’s degree in applied mathematics in 1949 and a PhD in statistics in 1954, all from North Carolina State. His research centered on experimental design, in particular the study of fractional factorial designs and response surface methods. He was the founding editor of Technometrics. Stu joined the faculty at Princeton University as an assistant professor in the Engineering School in 1961. He was a first-rate teacher, and his courses at Princeton were often rated among the top courses at the University. The interviewer had the good fortune to take his Engineering Statistics course in 1970 which began a life-long friendship. Stu was a consultant for many companies and the co-author of the influential book Statistics for Experimenters with George Box and William Hunter. His short courses in industry were legendary. He served as the 1993 president of the American Statistical Association (ASA) and has received many honors and awards from the ASA, the ASQ and other organizations. In 2005, he was named as a fellow to the National Academy of Engineering. The Stu Hunter Research Conference was established in 2012 to “honor one of the pioneers in applied statistics.” Stu retired from Princeton in 1984, but remains active consulting, mentoring and traveling to this day.