1.3. Generalized Linear Model (GLM) Coefficient

Assuming that the data is generated from a Linear Model,

\[Y = \sum \beta_j X^j+ \epsilon, \quad \text{ with }\varepsilon\sim\mathcal{N}(0, \sigma^2),\]

the most natural way of defining the importance of each feature is through the absolute/squared value of the corresponding coefficient \(\beta_j\) [1]:

\[\psi^j_{GLM} = \beta_j^2.\]

This definition has the advantage of being simple and interpretable. Moreover, it can be directly extended to classification for example by considering the coefficients of a Generalized Linear Model (GLM).

Note

Misspecification of the underlying distribution

This definition relies on the strong assumption that the data is generated from a (Generalized) Linear Model. In practice, this assumption is often violated, which can lead to misspecifying the underlying data distribution and therefore to misleading importance measures ([2]). See Variable Selection Under Model Misspecification.

They can be estimated using standard statistical techniques for fitting GLMs. There are standard penalization procedures, such as Lasso or Ridge, that address estimation issues in high-dimensional or highly correlated settings. Moreover, there are debiased alternatives to these procedures (see Desparsified Lasso).

Behind this simple idea lie several core procedures of statistical inference. For instance, the popular Model-X Knockoff (Model-X Knockoffs, [3]) and the Distilled Conditional Randomization Test (Distilled Conditional Randomization Test, [4]) usually rely on GLM coefficients to perform variable selection with FDR and type-I error control, respectively.

1.3.1. References

[1]

Isabella Verdinelli and Larry Wasserman. Feature importance: a closer look at shapley values and loco. Statistical Science, 39(4):623–636, 2024.

[2]

Christoph Molnar, Gunnar König, Julia Herbinger, Timo Freiesleben, Susanne Dandl, Christian A. Scholbeck, Giuseppe Casalicchio, Moritz Grosse-Wentrup, and Bernd Bischl. General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models, pages 39–68. Springer International Publishing, Cham, 2022.

[3]

Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:’model-x’ knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):551–577, 2018.

[4]

Molei Liu, Eugene Katsevich, Lucas Janson, and Aaditya Ramdas. Fast and powerful conditional randomization testing via distillation. Biometrika, 109(2):277–293, 2022.