2. Sparse Linear Models

The methods presented in this section assume that the dependencies between variables can be captured by a linear model. They are specifically designed to tackle feature importance and selection in high-dimensional settings by leveraging sparse linear models (SLM).

2.1. Desparsified Lasso

Desparsified Lasso, also known as debiased Lasso, is a method that aims at estimating the regression coefficients Hastie et al.[1]. To do so, it uses coefficients obtained from a Lasso regression and corrects the bias induced by the L1-penalty. This method is particularly useful in high-dimensional settings where the number of features exceeds the number of samples.

2.1.1. Regression example

Desparsified Lasso can be used as follows:

>>> from sklearn.datasets import make_regression
>>> from sklearn.linear_model import LassoCV
>>> from hidimstat import DesparsifiedLasso


>>> X, y = make_regression(n_features=2)

>>> dl = DesparsifiedLasso(estimator=LassoCV(), n_jobs=n_jobs, random_state=0)
>>> dl.fit(X, y)
>>> features_importance = dl.importance(X, y)

>>> # Selection based on FDR control
>>> selected_features = dl.fdr_selection(fdr=0.05)
>>> # Selection based on FWER control
>>> selected_features = dl.fwer_selection(fwer=0.05)

2.1.2. Target quantity

This method assumes a linear model of the form: \(Y = X \beta^\star + \epsilon\), where \(Y\) is the response variable, \(X\) is the design matrix, \(\beta\) is the vector of regression coefficients, and \(\epsilon\) is an additive noise term. The target quantity estimated by Desparsified Lasso is the vector of regression coefficients \(\beta^\star\). Denoting \(\hat{\beta}_\lambda\) the Lasso estimator with regularization parameter \(\lambda\), and \(\Theta\) an approximate inverse of the empirical covariance matrix \(\hat{\Sigma} = \frac{1}{n} X^T X\), the Desparsified Lasso estimator is given by:

\[\hat{\beta}_{DL} = \hat{\beta}_\lambda + \frac{1}{n} \Theta X^T (Y - X \hat{\beta}_\lambda)\]

2.1.3. Estimation procedure

The provided implementation hidimstat.DesparsifiedLasso is based on the one proposed by Zhang and Zhang[2]. It first fits a Lasso regression to obtain the initial estimator \(\hat{\beta}_\lambda\), and then estimates nodewise Lasso regressions, each predicting a feature \(X^j\) using all other features \(X^{-j}\).

2.1.4. Inference

As described in Chevalier et al.[3], under some sparsity assumptions, it can be obtained that the Desparsified Lasso estimator is asymptotically normal. This property allows to derive confidence intervals and p-values for the regression coefficients.

2.1.5. Extensions to spatially structured data

When the features have a known spatial structure, hidimstat.DesparsifiedLasso can be suboptimal, identifying scattered elements of the support and making false discoveries far from the support. Methods like hidimstat.CluDL and hidimstat.EnCluDL address this issue by leveraging the spatial structure of the data. Read more in the User Guide.

2.1.6. Examples

Ensemble Clustered Inference on 2D Data

Ensemble Clustered Inference on 2D Data

Pixel-wise inference on digit classification

Pixel-wise inference on digit classification

Coefficient estimates with Desparsified Lasso on the diabetes dataset

Coefficient estimates with Desparsified Lasso on the diabetes dataset

Support Recovery on fMRI Data

Support Recovery on fMRI Data

Source localization of somatosensory MEG data

Source localization of somatosensory MEG data

2.1.7. References

[1]

Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity. Monographs on statistics and applied probability, 143(143):8, 2015.

[2]

Cun-Hui Zhang and Stephanie S Zhang. Confidence intervals for low dimensional parameters in high dimensional linear models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):217–242, 2014.

[3]

Jérôme-Alexis Chevalier, Tuan-Binh Nguyen, Joseph Salmon, Gaël Varoquaux, and Bertrand Thirion. Decoding with confidence: statistical control on decoder maps. NeuroImage, 234:117921, 2021.

2.2. Model-X Knockoffs

The Model-X Knockoffs method Candes et al.[4] is a method for variable selection that controls the false discovery rate (FDR). It is particularly useful in high-dimensional settings since its default implementation leverages sparse regression with the Lasso.

2.2.1. Regression example

Model-X Knockoffs can be used as follows:

>>> from sklearn.datasets import make_regression
>>> from sklearn.linear_model import LassoCV
>>> from hidimstat import ModelXKnockoff


>>> X, y = make_regression(n_features=2)

>>> ko = ModelXKnockoff(estimator=LassoCV(), random_state=0)
>>> ko.fit(X, y)

>>> # Selection based on FDR control
>>> selected_features = ko.fdr_selection(fdr=0.05)

2.2.2. Target quantity

Model-X Knockoffs is a variable selection method that is not meant to estimate any particular measure of importance. The ModelXKnockoff.importance() method still returns the so-called Knockoff statistics, such as the Lasso coefficient difference statistics, which can be used to rank the selected variables. The main goal of the method is to select a set of features denoted \(\hat{S} \subset \{1, \ldots, p\}\) such that the false discovery rate (FDR) is controlled at a target level.

2.2.3. Estimation procedure

The implementation of hidimstat.ModelXKnockoff relies on two main ingredients. First, a procedure to generate knockoff variables, that we denote \(\tilde{X}\). Second, a knockoff statistic.

2.2.3.1. Knockoff variables construction

Given the original design matrix \(X = [X^1, \ldots, X^p]\), knockoff variables are a new set of variables \(\tilde{X} = [\tilde{X}^1, \ldots, \tilde{X}^p]\) that have two key properties: 1. they are exchangeable with the original variables. That is, swapping any subset of variables with their knockoff counterparts does not change the joint distribution. For example, for \(p=3\),

\[(X^1, X^2, X^3, \tilde{X}^1, \tilde{X}^2, \tilde{X}^3) \overset{d}{=} (\tilde{X}^1, \tilde{X}^2, X^3, X^1, X^2, \tilde{X}^3)\]

where \(\overset{d}{=}\) denotes equality in distribution, where the subset of swapped variables is \(\{1, 2\}\).

In hidimstat.ModelXKnockoff, the knockoff variables are generated using the equicorrelated construction proposed by Candes et al.[4].

2.2.3.2. Knockoff statistics

To perform variable selection, a test statistic needs to be used in order to provide evidence against the null hypothesis, which for the \(j^{th}\) feature is \(X^j \perp\!\!\!\!\perp Y | X^{-j}\). This statistic needs to satisfy the so-called flip-sign property, which ensures that swapping a variable with its knockoff counterpart results in a sign change of the statistic. hidimstat.ModelXKnockoff implements the popular Lasso Coeffcient-Difference (LCD), which given a Lasso model fitted to predict the target \(Y\) using both the original and knockoff variables \([X, \tilde{X}]\), is defined as:

\[w_j = |\hat{\beta}_j| - |\hat{\beta}_{j + p}|\]

where \(\hat{\beta}_j\) is the Lasso coefficient associated to the original variable \(X^j\), and \(\hat{\beta}_{j + p}\) is the coefficient associated to its knockoff counterpart \(\tilde{X}^j\). intuitively, a large positive value of \(w_j\) indicates that the original variable contains information that is not captured by its knockoff counterpart, and thus provides additional information about the response \(Y\), that is not explained by other variables.

2.2.4. Inference

The variable selection set \(\hat{S}\) is obtained by choosing a threshold \(\tau\) on the knockoff statistics \(w_j\), such that all variables with a statistic larger than \(\tau\) are selected. For a target FDR level \(\alpha\), choosing the threshold as:

\[\tau = \min \left\{ t > 0 : \frac{1 + \#\{j : w_j \leq -t\}}{\#\{j : w_j \geq t\}} \leq \alpha \right\}\]

guarantees that the FDR is controlled at level \(\alpha\).

2.2.5. De-randomization of Knockoffs

The generation of knockoff variables introduces randomness in the selection procedure. Indeed, the sample of knockoff variables \(\tilde{X}\) corresponds to a single draw and repeating the entire procedure, with a different set of knockoff variables, may lead to a different selection set. To mitigate this source of variability, the selection can be de-randomized by aggregating the results of multiple runs of the Knockoff procedure. This can be done using the n_repeats parameter, for instance, ModelXKnockoff(n_repeats=10).

2.2.6. Examples

Knockoff aggregation

Knockoff aggregation

Controlled multiple variable selection on the Wisconsin breast cancer dataset

Controlled multiple variable selection on the Wisconsin breast cancer dataset

2.2.7. References

[4] (1,2,3,4)

Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:’model-x’ knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):551–577, 2018.

2.3. Distilled Conditional Randomization Test

The Distilled Conditional Randomization Test (dCRT) is a method for variable selection that tests whether each feature \(X^j\) is conditionally independent of the target \(Y\) given all other features \(X^{-j}\). It was introduced by Liu et al.[5] as a computationally efficient version of the Conditional Randomization Test (CRT) from Candes et al.[4]. The key idea is to replace the expensive resampling step of the CRT with a distillation procedure that decomposes the problem into simple residual computations.

2.3.1. Target quantity

The dCRT tests the conditional independence hypothesis for each feature \(j\):

\[\mathcal{H}_0^j: Y \perp\!\!\!\!\perp X^j \mid X^{-j}.\]

Unlike Model-X Knockoffs, which control the False Discovery Rate (FDR) across the full set of selected variables, the dCRT produces a p-value for each individual feature. This allows conditional independence testing at the single-feature level with type-I error control.

Under a linear model \(Y = X \beta^* + \varepsilon\), the null hypothesis is equivalent to testing \(\beta^*_j = 0\). The test statistic produced by dCRT follows a standard normal distribution under the null, enabling direct p-value computation without permutations.

2.3.2. Estimation procedure

The original CRT (Candes et al.[4]) requires, for each feature, fitting the predictive model many times on resampled data to build the null distribution of the test statistic. This makes it prohibitively expensive when the number of features is large. The dCRT replaces this resampling loop with a distillation procedure: instead of repeatedly refitting the model, it reduces the problem to computing residuals from two regressions. For each feature \(j\):

1. Distillation. The key idea is to decompose the problem into two regression sub-problems that remove the effect of all other features:

• X-distillation: Regress \(X^j\) on \(X^{-j}\) to obtain the residual \(\hat{\epsilon}_j^X = X^j - \hat{\nu}_j(X^{-j})\). This residual captures the part of \(X^j\) that cannot be predicted from the other features.

• Y-distillation: Regress \(Y\) on \(X^{-j}\) to obtain the residual \(\hat{\epsilon}_j^Y = Y - \hat{\mu}_{-j}(X^{-j})\). This residual captures the part of \(Y\) that cannot be predicted from the other features.

While the X-distillation step currently uses a linear model, the Y-distillation model can be any supervised learner (e.g., random forests, gradient boosting). This makes the dCRT applicable beyond purely linear settings: the test remains valid as long as the X-distillation model is correctly specified.

2. Test statistic. The test statistic is the normalized correlation between the two residuals:

\[T_j = \frac{\hat{\epsilon}_j^{Y\top} \hat{\epsilon}_j^X} {\sqrt{n \, \hat{\sigma}_j^2 \, \|\hat{\epsilon}_j^Y\|^2 / n}}\]

where \(\hat{\sigma}_j^2\) is the estimated variance of the X-residuals. Under the null hypothesis, \(T_j \sim \mathcal{N}(0, 1)\) asymptotically.

Note

Screening (optional)

When the number of features is large, an optional screening step can be enabled via the lasso_screening parameter. A Lasso regression is fitted on the full data and features with coefficients below a percentile threshold are discarded before running the distillation. This reduces computation but can be turned off by setting lasso_screening=None.

Note

Logistic regression variant (dCRT-logit)

When the estimator is a logistic regression, hidimstat.D0CRT automatically switches to the dCRT-logit approach from Nguyen et al.[6], which adapts the distillation and test statistic to the logistic loss. The null distribution remains standard normal. See the dCRT-logit example for an illustration.

2.3.3. Inference

Since the test statistic \(T_j\) follows a standard normal distribution under the null, p-values are obtained directly as \(p_j = 2 \, \Phi(-|T_j|)\), where \(\Phi\) is the standard normal CDF.

These p-values can then be used for variable selection with type-I error control (via pvalue_selection) or FDR control (via fdr_selection).

2.3.4. Regression example

The following example illustrates the use of dCRT on a regression task:

>>> from sklearn.datasets import make_regression
>>> from sklearn.linear_model import LassoCV
>>> from hidimstat import D0CRT

>>> X, y = make_regression(n_samples=200, n_features=20, n_informative=5)

>>> dcrt = D0CRT(estimator=LassoCV(), random_state=0)
>>> dcrt.fit(X, y)
>>> features_importance = dcrt.importance(X, y)

>>> # Selection based on p-value threshold
>>> selected_features = dcrt.pvalue_selection(threshold_max=0.05)

2.3.5. Classification example

For classification with logistic regression, the dCRT-logit variant is used automatically:

>>> import numpy as np
>>> from sklearn.linear_model import LogisticRegressionCV
>>> from hidimstat import D0CRT

>>> rng = np.random.default_rng(0)
>>> X = rng.standard_normal((200, 20))
>>> beta = np.zeros(20)
>>> beta[:2] = 10.0
>>> y = rng.binomial(1, 1 / (1 + np.exp(-X @ beta)))

>>> dcrt = D0CRT(
...     estimator=LogisticRegressionCV(penalty="l1", solver="liblinear"),
...     lasso_screening=LogisticRegressionCV(penalty="l1", solver="liblinear"),
...     random_state=0,
... )
>>> dcrt.fit(X, y)
>>> features_importance = dcrt.importance(X, y)

2.3.6. Examples

Distilled Conditional Randomization Test (dCRT) using Lasso vs Random Forest learners

Distilled Conditional Randomization Test (dCRT) using Lasso vs Random Forest learners

Conditional Randomization Test for Sparse Logistic Regression

Conditional Randomization Test for Sparse Logistic Regression

Variable Selection Under Model Misspecification

Variable Selection Under Model Misspecification

2.3.7. References

[5]

Molei Liu, Eugene Katsevich, Lucas Janson, and Aaditya Ramdas. Fast and powerful conditional randomization testing via distillation. Biometrika, 109(2):277–293, 2022.

[6]

Binh T Nguyen, Bertrand Thirion, and Sylvain Arlot. A conditional randomization test for sparse logistic regression in high-dimension. Advances in Neural Information Processing Systems, 35:13691–13703, 2022.