Penalized Precision Matrix Estimation in grasps

Preliminary

Consider the following setting:

• Gaussian graphical model (GGM) assumption:
  The data X_n × d consists of independent and identically distributed samples X₁, …, X_n ∼ N_d(μ, Σ).

• Disjoint group structure:
  The d variables can be partitioned into disjoint groups.

• Goal:
  Estimate the precision matrix Ω = Σ⁻¹ = (ω_ij)_d × d.

Sparse-Group Estimator

where:

• $S = n^{-1} \sum_{i=1}^n (X_i-\bar{X})(X_i-\bar{X})^\top$ is the empirical covariance matrix.

• λ ≥ 0 is the global regularization parameter controlling overall shrinkage.

• α ∈ [0, 1] is the mixing parameter controlling the balance between element-wise and block-wise penalties.

• γ is the additional parameter controlling the curvature and effective degree of nonconvexity of the penalty.

• P_α, γ(Ω) is a generic bi-level penalty template that can incorporate convex or non-convex regularizers while preserving the intrinsic group structure among variables.

• P_γ^idv(Ω) is the element-wise individual penalty component.

• P_γ^grp(Ω) is the block-wise group penalty component.

• p_γ(⋅) is a penalty kernel parameterized by γ.

• Ω_gg^′ is the submatrix of Ω with the rows from group g and columns from group g^′.

• The Frobenius norm ‖Ω‖_F is defined as ‖Ω‖_F = (∑_i, j|ω_ij|²)^1/2 = [tr(Ω^⊤Ω)]^1/2.

Penalties

1. Lasso: Least absolute shrinkage and selection operator (Tibshirani 1996; Friedman, Hastie, and Tibshirani 2008)

λp(ω_ij) = λ|ω_ij|.

2. Adaptive lasso (Zou 2006; Fan, Feng, and Wu 2009)

$$ \lambda p_\gamma(\omega_{ij}) = \lambda\frac{\vert\omega_{ij}\vert}{v_{ij}}, $$ where V = (v_ij)_d × d = (|ω̃_ij|^γ)_d × d is a matrix of adaptive weights, and ω̃_ij is the initial estimate obtained using penalty = "lasso".

3. Atan: Arctangent type penalty (Wang and Zhu 2016)

$$ \lambda p_\gamma(\omega_{ij}) = \lambda(\gamma+\frac{2}{\pi}) \arctan\left(\frac{\vert\omega_{ij}\vert}{\gamma}\right), \quad \gamma > 0. $$

4. Exp: Exponential type penalty (Wang, Fan, and Zhu 2018)

$$ \lambda p_\gamma(\omega_{ij}) = \lambda\left[1-\exp\left(-\frac{\vert\omega_{ij}\vert}{\gamma}\right)\right], \quad \gamma > 0. $$

5. Lq (Frank and Friedman 1993; Fu 1998; Fan and Li 2001)

λp_γ(ω_ij) = λ|ω_ij|^γ,  0 < γ < 1.

6. LSP: Log-sum penalty (Candès, Wakin, and Boyd 2008)

$$ \lambda p_\gamma(\omega_{ij}) = \lambda\log\left(1+\frac{\vert\omega_{ij}\vert}{\gamma}\right), \quad \gamma > 0. $$

7. MCP: Minimax concave penalty (Zhang 2010)

$$ \lambda p_\gamma(\omega_{ij}) = \begin{cases} \lambda\vert\omega_{ij}\vert - \dfrac{\omega_{ij}^2}{2\gamma}, & \text{if } \vert\omega_{ij}\vert \leq \gamma\lambda, \\ \dfrac{1}{2}\gamma\lambda^2, & \text{if } \vert\omega_{ij}\vert > \gamma\lambda. \end{cases} \quad \gamma > 1. $$

8. SCAD: Smoothly clipped absolute deviation (Fan and Li 2001; Fan, Feng, and Wu 2009)

$$ \lambda p_\gamma(\omega_{ij}) = \begin{cases} \lambda\vert\omega_{ij}\vert & \text{if } \vert\omega_{ij}\vert \leq \lambda, \\ \dfrac{2\gamma\lambda\vert\omega_{ij}\vert-\omega_{ij}^2-\lambda^2}{2(\gamma-1)} & \text{if } \lambda < \vert\omega_{ij}\vert < \gamma\lambda, \\ \dfrac{\lambda^2(\gamma+1)}{2} & \text{if } \vert\omega_{ij}\vert \geq \gamma\lambda. \end{cases} \quad \gamma > 2. $$

Illustrative Visualization

Figure 1 illustrates a comparison of various penalty functions λp(ω) evaluated over a range of ω values. The main panel (right) provides a wider view of the penalty functions’ behavior for larger |ω|, while the inset panel (left) magnifies the region near zero [−1, 1].

library(grasps) ## for penalty computation
library(ggplot2) ## for visualization

penalties <- c("atan", "exp", "lasso", "lq", "lsp", "mcp", "scad")

pen_df <- compute_penalty(seq(-4, 4, by = 0.01), penalties, lambda = 1)
plot(pen_df, xlim = c(-1, 1), ylim = c(0, 1), zoom.size = 1) +
  guides(color = guide_legend(nrow = 2, byrow = TRUE))

Figure 2 displays the derivative function p^′(ω) associated with a range of penalty types. The Lasso exhibits a constant derivative, corresponding to uniform shrinkage. For MCP and SCAD, the derivatives are piecewise: initially equal to the Lasso derivative, then decreasing over an intermediate region, and eventually dropping to zero, indicating that large |ω| receive no shrinkage. Other non-convex penalties show smoothly diminishing derivatives as |ω| increases, reflecting their tendency to shrink small |ω| strongly while exerting little to no shrinkage on large ones.

deriv_df <- compute_derivative(seq(0, 4, by = 0.01), penalties, lambda = 1)
plot(deriv_df) +
  scale_y_continuous(limits = c(0, 1.5)) +
  guides(color = guide_legend(nrow = 2, byrow = TRUE))

Reference