Compute variable importance with p-values. For high dimensional data, the fast method of Janitza et al. (2016) can be used. The permutation approach of Altmann et al. (2010) is computationally intensive but can be used with all kinds of data. See below for details.
Usage
importance_pvalues(
x,
method = c("janitza", "altmann"),
num.permutations = 100,
formula = NULL,
data = NULL,
...
)Arguments
- x
rangerorholdoutRFobject.- method
Method to compute p-values. Use "janitza" for the method by Janitza et al. (2016) or "altmann" for the non-parametric method by Altmann et al. (2010).
- num.permutations
Number of permutations. Used in the "altmann" method only.
- formula
Object of class formula or character describing the model to fit. Used in the "altmann" method only.
- data
Training data of class data.frame or matrix. Used in the "altmann" method only.
- ...
Further arguments passed to
ranger(). Used in the "altmann" method only.
Details
The method of Janitza et al. (2016) uses a clever trick: With an unbiased variable importance measure, the importance values of non-associated variables vary randomly around zero. Thus, all non-positive importance values are assumed to correspond to these non-associated variables and they are used to construct a distribution of the importance under the null hypothesis of no association to the response. Since only the non-positive values of this distribution can be observed, the positive values are created by mirroring the negative distribution. See Janitza et al. (2016) for details.
The method of Altmann et al. (2010) uses a simple permutation test: The distribution of the importance under the null hypothesis of no association to the response is created by several replications of permuting the response, growing an RF and computing the variable importance. The authors recommend 50-100 permutations. However, much larger numbers have to be used to estimate more precise p-values. We add 1 to the numerator and denominator to avoid zero p-values.
References
Janitza, S., Celik, E. & Boulesteix, A.-L., (2016). A computationally fast variable importance test for random forests for high-dimensional data. Adv Data Anal Classif doi:10.1007/s11634-016-0276-4
.
Altmann, A., Tolosi, L., Sander, O. & Lengauer, T. (2010). Permutation importance: a corrected feature importance measure, Bioinformatics 26:1340-1347.
Examples
## Janitza's p-values with corrected Gini importance
n <- 50
p <- 400
dat <- data.frame(y = factor(rbinom(n, 1, .5)), replicate(p, runif(n)))
rf.sim <- ranger(y ~ ., dat, importance = "impurity_corrected")
importance_pvalues(rf.sim, method = "janitza")
#> importance pvalue
#> X1 -2.051051e-02 0.768496420
#> X2 1.197972e-02 0.336515513
#> X3 -1.075553e-03 0.515513126
#> X4 1.055714e-02 0.357995227
#> X5 -2.486442e-02 0.797136038
#> X6 3.107964e-02 0.159904535
#> X7 -8.180821e-02 0.971360382
#> X8 1.589186e-02 0.305489260
#> X9 -1.813397e-02 0.737470167
#> X10 1.354363e-02 0.324582339
#> X11 -2.668973e-02 0.821002387
#> X12 1.412068e-02 0.319809069
#> X13 1.696665e-03 0.470167064
#> X14 7.841233e-03 0.398568019
#> X15 3.717469e-02 0.131264916
#> X16 -6.563636e-03 0.587112172
#> X17 -1.027739e-02 0.637231504
#> X18 -2.983235e-02 0.840095465
#> X19 -1.851218e-02 0.747016706
#> X20 1.563859e-02 0.310262530
#> X21 -8.496508e-03 0.618138425
#> X22 -1.403984e-02 0.680190931
#> X23 3.894714e-02 0.116945107
#> X24 -2.951008e-02 0.837708831
#> X25 5.114879e-02 0.090692124
#> X26 -3.147212e-03 0.541766110
#> X27 -5.174413e-02 0.918854415
#> X28 -1.790556e-02 0.730310263
#> X29 -7.884411e-03 0.603818616
#> X30 -5.921955e-03 0.582338902
#> X31 5.348606e-02 0.076372315
#> X32 2.986667e-03 0.463007160
#> X33 4.418477e-03 0.436754177
#> X34 -6.601004e-02 0.949880668
#> X35 1.698864e-02 0.288782816
#> X36 4.056382e-02 0.112171838
#> X37 -1.173499e-02 0.663484487
#> X38 1.020749e-02 0.365155131
#> X39 -1.803509e-02 0.732696897
#> X40 7.118667e-02 0.040572792
#> X41 3.363081e-02 0.150357995
#> X42 3.521276e-02 0.145584726
#> X43 -3.409154e-02 0.852028640
#> X44 8.927035e-03 0.377088305
#> X45 6.392344e-03 0.415274463
#> X46 -2.630064e-02 0.816229117
#> X47 6.992723e-02 0.042959427
#> X48 2.017607e-02 0.233890215
#> X49 -1.235261e-02 0.665871122
#> X50 -3.703853e-03 0.548926014
#> X51 1.081280e-02 0.353221957
#> X52 -2.658269e-03 0.536992840
#> X53 -1.959099e-02 0.761336516
#> X54 -1.681788e-02 0.706443914
#> X55 3.127032e-02 0.159904535
#> X56 -1.082466e-02 0.649164678
#> X57 -2.068614e-02 0.770883055
#> X58 -1.984159e-02 0.763723150
#> X59 4.272348e-03 0.441527446
#> X60 5.424179e-03 0.424821002
#> X61 2.955992e-02 0.162291169
#> X62 -1.269663e-02 0.668257757
#> X63 3.854827e-03 0.451073986
#> X64 -1.468207e-02 0.684964200
#> X65 -1.070456e-01 0.990453461
#> X66 5.987992e-03 0.417661098
#> X67 2.321526e-02 0.212410501
#> X68 4.634963e-02 0.097852029
#> X69 -1.269234e-03 0.520286396
#> X70 7.714306e-02 0.033412888
#> X71 3.803097e-02 0.126491647
#> X72 -5.257539e-02 0.921241050
#> X73 -1.360999e-03 0.522673031
#> X74 1.703210e-02 0.288782816
#> X75 -1.616833e-02 0.696897375
#> X76 3.273681e-03 0.458233890
#> X77 1.444228e-02 0.319809069
#> X78 -1.760955e-02 0.725536993
#> X79 3.128435e-02 0.159904535
#> X80 -3.986181e-02 0.887828162
#> X81 -1.038095e-02 0.639618138
#> X82 -1.165712e-01 0.992840095
#> X83 -1.530322e-04 0.508353222
#> X84 4.566149e-02 0.097852029
#> X85 1.005095e-01 0.011933174
#> X86 -9.866295e-03 0.627684964
#> X87 -5.429808e-02 0.935560859
#> X88 5.897426e-03 0.422434368
#> X89 -5.090823e-03 0.570405728
#> X90 1.613748e-02 0.305489260
#> X91 3.160424e-02 0.157517900
#> X92 7.107071e-04 0.489260143
#> X93 -8.069290e-03 0.610978520
#> X94 3.084983e-02 0.159904535
#> X95 9.201634e-03 0.377088305
#> X96 -2.358672e-02 0.789976134
#> X97 3.723956e-02 0.128878282
#> X98 2.667616e-02 0.181384248
#> X99 4.644018e-03 0.434367542
#> X100 -1.011255e-02 0.632458234
#> X101 -1.752536e-02 0.720763723
#> X102 3.120815e-02 0.159904535
#> X103 3.165812e-03 0.458233890
#> X104 6.369735e-03 0.415274463
#> X105 4.141066e-02 0.107398568
#> X106 1.295238e-03 0.479713604
#> X107 1.077745e-01 0.009546539
#> X108 -1.149961e-05 0.503579952
#> X109 -6.716498e-02 0.954653938
#> X110 7.969636e-02 0.033412888
#> X111 1.588947e-02 0.305489260
#> X112 -2.158427e-02 0.778042959
#> X113 -7.037370e-02 0.959427208
#> X114 -3.316999e-03 0.544152745
#> X115 8.218917e-02 0.026252983
#> X116 5.864077e-02 0.054892601
#> X117 -5.820975e-03 0.577565632
#> X118 -7.491049e-02 0.964200477
#> X119 -1.651190e-02 0.699284010
#> X120 4.077290e-02 0.112171838
#> X121 2.470198e-02 0.205250597
#> X122 -6.113058e-03 0.584725537
#> X123 -3.351300e-02 0.849642005
#> X124 3.161178e-02 0.157517900
#> X125 -5.021585e-02 0.909307876
#> X126 -2.831474e-02 0.835322196
#> X127 -4.338365e-04 0.510739857
#> X128 3.846154e-03 0.451073986
#> X129 2.179527e-02 0.221957041
#> X130 -1.750325e-02 0.715990453
#> X131 1.012192e-01 0.011933174
#> X132 2.228510e-02 0.219570406
#> X133 -2.013171e-02 0.766109785
#> X134 -1.108481e-02 0.656324582
#> X135 -9.483494e-02 0.980906921
#> X136 1.457227e-02 0.319809069
#> X137 -2.515201e-02 0.804295943
#> X138 2.728787e-02 0.171837709
#> X139 -5.330081e-03 0.575178998
#> X140 7.346734e-03 0.403341289
#> X141 7.973283e-03 0.393794749
#> X142 -5.374345e-02 0.926014320
#> X143 2.009170e-02 0.236276850
#> X144 -5.129982e-02 0.911694511
#> X145 -2.572708e-02 0.811455847
#> X146 8.440875e-03 0.384248210
#> X147 -3.143071e-02 0.842482100
#> X148 -3.665435e-02 0.868735084
#> X149 2.533766e-03 0.465393795
#> X150 3.802952e-02 0.126491647
#> X151 1.310725e-02 0.329355609
#> X152 1.434775e-02 0.319809069
#> X153 4.211921e-02 0.107398568
#> X154 1.085646e-02 0.350835322
#> X155 -3.662476e-02 0.863961814
#> X156 6.030887e-02 0.054892601
#> X157 1.239864e-03 0.484486874
#> X158 1.444968e-02 0.319809069
#> X159 3.467248e-02 0.147971360
#> X160 -5.223149e-03 0.572792363
#> X161 -8.133410e-02 0.968973747
#> X162 -7.634971e-02 0.966587112
#> X163 -1.555699e-02 0.689737470
#> X164 6.234936e-03 0.415274463
#> X165 2.309444e-02 0.212410501
#> X166 8.929877e-02 0.023866348
#> X167 -4.359708e-02 0.899761337
#> X168 -5.165124e-02 0.916467780
#> X169 -2.631580e-02 0.818615752
#> X170 -1.071429e-02 0.644391408
#> X171 -1.898981e-02 0.754176611
#> X172 2.037851e-02 0.233890215
#> X173 4.715177e-02 0.097852029
#> X174 -8.617095e-02 0.976133652
#> X175 1.301880e-02 0.329355609
#> X176 3.500583e-02 0.147971360
#> X177 1.451197e-02 0.319809069
#> X178 -1.372640e-02 0.677804296
#> X179 8.139498e-03 0.386634845
#> X180 -7.914379e-03 0.606205251
#> X181 -4.252414e-03 0.558472554
#> X182 -8.483884e-05 0.505966587
#> X183 5.464431e-03 0.424821002
#> X184 1.141214e-02 0.341288783
#> X185 4.410016e-02 0.100238663
#> X186 3.671312e-02 0.131264916
#> X187 2.017173e-02 0.233890215
#> X188 3.514781e-02 0.147971360
#> X189 -5.656302e-02 0.940334129
#> X190 -3.236942e-02 0.844868735
#> X191 3.289945e-02 0.152744630
#> X192 2.257497e-02 0.219570406
#> X193 -3.718088e-02 0.871121718
#> X194 -6.902048e-02 0.957040573
#> X195 3.180227e-02 0.157517900
#> X196 -4.333049e-03 0.563245823
#> X197 5.869094e-02 0.054892601
#> X198 2.019064e-02 0.233890215
#> X199 -1.246498e-03 0.517899761
#> X200 4.932036e-02 0.093078759
#> X201 -2.617344e-02 0.813842482
#> X202 -2.069645e-02 0.773269690
#> X203 -1.937495e-02 0.756563246
#> X204 -1.751985e-02 0.718377088
#> X205 -4.610335e-03 0.565632458
#> X206 -1.515247e-01 0.995226730
#> X207 4.123576e-02 0.107398568
#> X208 -4.082170e-02 0.890214797
#> X209 1.296675e-02 0.329355609
#> X210 7.471878e-02 0.038186158
#> X211 -8.113160e-03 0.613365155
#> X212 9.105831e-03 0.377088305
#> X213 2.226429e-02 0.219570406
#> X214 8.736846e-03 0.377088305
#> X215 -5.904079e-03 0.579952267
#> X216 -1.946622e-02 0.758949881
#> X217 4.136364e-03 0.446300716
#> X218 -1.014667e-02 0.634844869
#> X219 -3.000180e-03 0.539379475
#> X220 -1.460329e-02 0.682577566
#> X221 3.035824e-02 0.159904535
#> X222 1.105526e-02 0.346062053
#> X223 -2.505651e-02 0.801909308
#> X224 -4.305371e-02 0.894988067
#> X225 -2.691698e-02 0.823389021
#> X226 -8.695490e-03 0.622911695
#> X227 -1.688611e-02 0.711217184
#> X228 8.633682e-04 0.486873508
#> X229 -6.698905e-02 0.952267303
#> X230 3.461502e-02 0.147971360
#> X231 -5.332378e-02 0.923627685
#> X232 -3.532680e-02 0.856801909
#> X233 -1.591792e-03 0.527446301
#> X234 -8.435291e-03 0.615751790
#> X235 -7.726930e-03 0.601431981
#> X236 4.286913e-03 0.439140811
#> X237 -2.273125e-02 0.785202864
#> X238 4.052850e-03 0.446300716
#> X239 4.501534e-02 0.097852029
#> X240 -1.600863e-01 0.997613365
#> X241 -9.071139e-02 0.978520286
#> X242 -3.428571e-03 0.546539379
#> X243 -5.852312e-02 0.945107399
#> X244 -7.506587e-03 0.599045346
#> X245 4.886767e-03 0.434367542
#> X246 5.349290e-02 0.076372315
#> X247 -4.027309e-03 0.553699284
#> X248 -4.094893e-02 0.892601432
#> X249 -2.133483e-03 0.534606205
#> X250 -1.858427e-02 0.751789976
#> X251 -1.569987e-02 0.692124105
#> X252 2.076947e-02 0.226730310
#> X253 -7.985187e-03 0.608591885
#> X254 -5.158427e-02 0.914081146
#> X255 2.487967e-02 0.202863962
#> X256 -1.683231e-02 0.708830549
#> X257 2.087660e-02 0.226730310
#> X258 3.654937e-02 0.140811456
#> X259 -6.933188e-03 0.591885442
#> X260 -2.216667e-02 0.780429594
#> X261 8.360801e-03 0.386634845
#> X262 1.418283e-02 0.319809069
#> X263 -7.415807e-02 0.961813842
#> X264 -1.840832e-02 0.744630072
#> X265 3.054545e-03 0.460620525
#> X266 1.035442e-01 0.011933174
#> X267 -5.397136e-02 0.928400955
#> X268 -1.050716e-02 0.642004773
#> X269 3.940000e-03 0.448687351
#> X270 -2.777470e-02 0.832935561
#> X271 3.651993e-02 0.140811456
#> X272 -1.079545e-02 0.646778043
#> X273 1.053269e-03 0.486873508
#> X274 2.755886e-02 0.169451074
#> X275 -2.516708e-02 0.806682578
#> X276 -3.520793e-02 0.854415274
#> X277 -3.244882e-02 0.847255370
#> X278 -2.261850e-02 0.782816229
#> X279 -2.067532e-03 0.532219570
#> X280 9.435466e-03 0.377088305
#> X281 8.851405e-02 0.023866348
#> X282 4.542832e-02 0.097852029
#> X283 -1.521183e-02 0.687350835
#> X284 -1.136075e-02 0.658711217
#> X285 -3.663447e-02 0.866348449
#> X286 -1.805455e-02 0.735083532
#> X287 -6.645794e-03 0.589498807
#> X288 -1.781465e-02 0.727923628
#> X289 1.364834e-02 0.324582339
#> X290 9.326229e-03 0.377088305
#> X291 8.144796e-05 0.496420048
#> X292 -1.852477e-02 0.749403341
#> X293 4.179443e-02 0.107398568
#> X294 -1.663159e-02 0.704057279
#> X295 -2.718620e-02 0.828162291
#> X296 -2.406965e-02 0.794749403
#> X297 -3.871688e-03 0.551312649
#> X298 3.497201e-03 0.453460621
#> X299 1.056012e-01 0.011933174
#> X300 4.508135e-02 0.097852029
#> X301 -5.527866e-02 0.937947494
#> X302 7.443223e-04 0.489260143
#> X303 2.296718e-02 0.212410501
#> X304 1.980275e-02 0.238663484
#> X305 3.082547e-02 0.159904535
#> X306 -8.675570e-03 0.620525060
#> X307 -1.755036e-02 0.723150358
#> X308 -2.490476e-02 0.799522673
#> X309 -7.097838e-03 0.594272076
#> X310 8.401495e-02 0.026252983
#> X311 1.764710e-03 0.470167064
#> X312 2.304444e-02 0.212410501
#> X313 -3.864065e-02 0.880668258
#> X314 -3.908439e-02 0.885441527
#> X315 -3.655404e-02 0.861575179
#> X316 5.528468e-03 0.424821002
#> X317 1.226094e-02 0.336515513
#> X318 -1.100409e-02 0.651551313
#> X319 4.122694e-02 0.107398568
#> X320 -2.138120e-02 0.775656325
#> X321 1.083507e-02 0.350835322
#> X322 2.372741e-03 0.465393795
#> X323 3.737834e-02 0.128878282
#> X324 -9.607773e-03 0.625298329
#> X325 1.242729e-02 0.334128878
#> X326 -4.745988e-02 0.904534606
#> X327 7.885335e-02 0.033412888
#> X328 2.028077e-02 0.233890215
#> X329 3.049174e-02 0.159904535
#> X330 -1.319444e-02 0.673031026
#> X331 1.923608e-03 0.470167064
#> X332 1.610030e-02 0.305489260
#> X333 -5.043955e-03 0.568019093
#> X334 0.000000e+00 0.501193317
#> X335 3.893994e-02 0.116945107
#> X336 -4.273283e-03 0.560859189
#> X337 -1.149850e-02 0.661097852
#> X338 -2.293736e-02 0.787589499
#> X339 6.218093e-03 0.415274463
#> X340 1.197168e-02 0.336515513
#> X341 -4.809878e-02 0.906921241
#> X342 -2.569281e-02 0.809069212
#> X343 1.724123e-02 0.286396181
#> X344 -3.817804e-02 0.875894988
#> X345 1.188392e-01 0.007159905
#> X346 9.604376e-03 0.377088305
#> X347 5.532460e-02 0.062052506
#> X348 2.824571e-02 0.167064439
#> X349 -1.286425e-02 0.670644391
#> X350 7.688804e-03 0.400954654
#> X351 1.628455e-02 0.303102625
#> X352 -1.104611e-02 0.653937947
#> X353 -9.698458e-02 0.983293556
#> X354 9.998387e-03 0.369928401
#> X355 -1.818745e-02 0.739856802
#> X356 -5.414402e-02 0.930787589
#> X357 -4.333487e-02 0.897374702
#> X358 -2.387227e-02 0.792362768
#> X359 6.230357e-03 0.415274463
#> X360 -9.812736e-02 0.988066826
#> X361 -9.912950e-03 0.630071599
#> X362 -6.255550e-02 0.947494033
#> X363 -2.736729e-02 0.830548926
#> X364 -3.771475e-02 0.873508353
#> X365 -4.234848e-03 0.556085919
#> X366 -5.419727e-02 0.933174224
#> X367 -3.868504e-02 0.883054893
#> X368 -4.486848e-02 0.902147971
#> X369 1.643539e-02 0.303102625
#> X370 1.779955e-02 0.274463007
#> X371 -8.217757e-02 0.973747017
#> X372 -1.612362e-03 0.529832936
#> X373 -1.341699e-02 0.675417661
#> X374 -1.829091e-02 0.742243437
#> X375 -5.665986e-02 0.942720764
#> X376 -1.583762e-02 0.694510740
#> X377 -3.839596e-02 0.878281623
#> X378 4.068396e-02 0.112171838
#> X379 1.084108e-02 0.350835322
#> X380 -3.625195e-02 0.859188544
#> X381 -8.154749e-04 0.513126492
#> X382 -2.355239e-01 1.000000000
#> X383 1.740183e-02 0.286396181
#> X384 1.176765e-01 0.007159905
#> X385 -9.771103e-02 0.985680191
#> X386 2.078593e-02 0.226730310
#> X387 -1.708295e-02 0.713603819
#> X388 2.652071e-02 0.181384248
#> X389 3.110088e-02 0.159904535
#> X390 6.996492e-02 0.042959427
#> X391 -1.661072e-02 0.701670644
#> X392 2.838191e-02 0.164677804
#> X393 3.933750e-03 0.448687351
#> X394 -7.172922e-03 0.596658711
#> X395 1.170896e-02 0.338902148
#> X396 3.973847e-02 0.114558473
#> X397 -2.693836e-02 0.825775656
#> X398 9.628561e-03 0.374701671
#> X399 -1.415261e-03 0.525059666
#> X400 3.711603e-02 0.131264916
## Permutation p-values
if (FALSE) { # \dontrun{
rf.iris <- ranger(Species ~ ., data = iris, importance = 'permutation')
importance_pvalues(rf.iris, method = "altmann", formula = Species ~ ., data = iris)
} # }