BH
-friendly, lfsr
-hostile Correlated \(N(0, 1)\) ScoresLast updated: 2017-12-21
Code version: 6e42447
We take a look at those data sets where BH
produces few (false) discoveries, and at the same time, ASH
’s lfsr
produces many (false) discoveries. For each data set, \(z\) scores, \(p\) values, and diagnostic \(\hat F_j\) are plotted.
library(ashr)
z = read.table("../output/z_null_liver_777.txt")
p = read.table("../output/p_null_liver_777.txt")
pihat0 = as.numeric(unlist(read.table("../output/pihat0_z_null_liver_777.txt")))
fd.bh = as.numeric(unlist(read.table("../output/fd.bh.0.05_null_liver_777.txt")))
fd.lfsr = as.numeric(unlist(read.table("../output/fd.lfsr.0.05.z.normal_null_liver_777.txt")))
BH
didn’t produce false discoveries but lfsr
does.In this simulation, BH
used the FDR level \(\alpha = 0.05\), whereas ASH
declared a discovery when lfsr
\(\leq 0.05\).
N0. 1 : Data Set 693 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1890 ; pihat0 = 0.01296388
N0. 2 : Data Set 33 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1111 ; pihat0 = 0.01048871
N0. 3 : Data Set 537 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1060 ; pihat0 = 0.01174394
N0. 4 : Data Set 885 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1012 ; pihat0 = 0.01285109
N0. 5 : Data Set 338 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 991 ; pihat0 = 0.01832042
N0. 6 : Data Set 942 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 844 ; pihat0 = 0.02694257
N0. 7 : Data Set 627 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 771 ; pihat0 = 0.02034137
N0. 8 : Data Set 176 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 763 ; pihat0 = 0.03005686
N0. 9 : Data Set 41 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 671 ; pihat0 = 0.03013827
N0. 10 : Data Set 407 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 648 ; pihat0 = 0.02274312
N0. 11 : Data Set 467 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 648 ; pihat0 = 0.0513426
N0. 12 : Data Set 404 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 642 ; pihat0 = 0.01757131
N0. 13 : Data Set 649 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 630 ; pihat0 = 0.02077128
N0. 14 : Data Set 411 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 484 ; pihat0 = 0.05617472
N0. 15 : Data Set 341 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 463 ; pihat0 = 0.04295436
N0. 16 : Data Set 538 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 449 ; pihat0 = 0.04976013
N0. 17 : Data Set 562 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 416 ; pihat0 = 0.05014307
N0. 18 : Data Set 705 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 365 ; pihat0 = 0.03966321
N0. 19 : Data Set 937 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 346 ; pihat0 = 0.04736585
N0. 20 : Data Set 893 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 309 ; pihat0 = 0.0422105
N0. 21 : Data Set 324 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 252 ; pihat0 = 0.04925775
N0. 22 : Data Set 915 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 217 ; pihat0 = 0.03371413
N0. 23 : Data Set 511 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 204 ; pihat0 = 0.04832825
N0. 24 : Data Set 3 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 193 ; pihat0 = 0.04912382
N0. 25 : Data Set 476 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 193 ; pihat0 = 0.06738787
N0. 26 : Data Set 985 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 182 ; pihat0 = 0.04389071
N0. 27 : Data Set 133 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 168 ; pihat0 = 0.04729238
N0. 28 : Data Set 495 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 167 ; pihat0 = 0.04841265
N0. 29 : Data Set 976 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 161 ; pihat0 = 0.06447058
N0. 30 : Data Set 809 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 156 ; pihat0 = 0.03198968
N0. 31 : Data Set 499 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 155 ; pihat0 = 0.07477045
N0. 32 : Data Set 999 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 154 ; pihat0 = 0.04802764
N0. 33 : Data Set 83 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 151 ; pihat0 = 0.06797304
N0. 34 : Data Set 780 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 145 ; pihat0 = 0.05197813
N0. 35 : Data Set 182 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 141 ; pihat0 = 0.04835373
N0. 36 : Data Set 982 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 129 ; pihat0 = 0.04228667
N0. 37 : Data Set 800 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 124 ; pihat0 = 0.0352261
N0. 38 : Data Set 972 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 118 ; pihat0 = 0.06009152
N0. 39 : Data Set 40 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 117 ; pihat0 = 0.04225311
N0. 40 : Data Set 895 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 113 ; pihat0 = 0.05313983
N0. 41 : Data Set 968 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 109 ; pihat0 = 0.05946434
N0. 42 : Data Set 644 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 108 ; pihat0 = 0.06256917
N0. 43 : Data Set 232 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 96 ; pihat0 = 0.07981507
N0. 44 : Data Set 15 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 94 ; pihat0 = 0.047921
N0. 45 : Data Set 251 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 89 ; pihat0 = 0.1130911
N0. 46 : Data Set 685 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 89 ; pihat0 = 0.04238147
N0. 47 : Data Set 659 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 85 ; pihat0 = 0.04122881
N0. 48 : Data Set 734 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 82 ; pihat0 = 0.05525044
N0. 49 : Data Set 484 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 73 ; pihat0 = 0.03296313
N0. 50 : Data Set 638 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 71 ; pihat0 = 0.05038079
N0. 51 : Data Set 268 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 70 ; pihat0 = 0.07180346
N0. 52 : Data Set 425 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 67 ; pihat0 = 0.1239249
N0. 53 : Data Set 781 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 65 ; pihat0 = 0.04563832
N0. 54 : Data Set 379 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 64 ; pihat0 = 0.03824364
N0. 55 : Data Set 303 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 59 ; pihat0 = 0.06374509
N0. 56 : Data Set 871 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 59 ; pihat0 = 0.07502045
N0. 57 : Data Set 606 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 56 ; pihat0 = 0.08713022
N0. 58 : Data Set 259 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 48 ; pihat0 = 0.06952575
N0. 59 : Data Set 526 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 45 ; pihat0 = 0.06426254
N0. 60 : Data Set 831 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 44 ; pihat0 = 0.08150764
N0. 61 : Data Set 853 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 37 ; pihat0 = 0.07850494
N0. 62 : Data Set 402 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 34 ; pihat0 = 0.07096583
N0. 63 : Data Set 190 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 33 ; pihat0 = 0.08695344
N0. 64 : Data Set 977 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 29 ; pihat0 = 0.06351771
N0. 65 : Data Set 452 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 24 ; pihat0 = 0.04829788
N0. 66 : Data Set 477 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 23 ; pihat0 = 0.05702149
N0. 67 : Data Set 633 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 22 ; pihat0 = 0.1001185
N0. 68 : Data Set 505 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 21 ; pihat0 = 0.09021379
N0. 69 : Data Set 953 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 21 ; pihat0 = 0.06148864
N0. 70 : Data Set 653 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 19 ; pihat0 = 0.09313401
N0. 71 : Data Set 907 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 18 ; pihat0 = 0.03578098
N0. 72 : Data Set 296 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 17 ; pihat0 = 0.08233874
N0. 73 : Data Set 845 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 16 ; pihat0 = 0.09543107
N0. 74 : Data Set 857 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 16 ; pihat0 = 0.07138568
N0. 75 : Data Set 423 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 15 ; pihat0 = 0.08643177
N0. 76 : Data Set 434 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 14 ; pihat0 = 0.0835438
N0. 77 : Data Set 349 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 13 ; pihat0 = 0.08685835
N0. 78 : Data Set 588 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 13 ; pihat0 = 0.2121708
N0. 79 : Data Set 622 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 12 ; pihat0 = 0.0782977
N0. 80 : Data Set 986 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 12 ; pihat0 = 0.2194227
N0. 81 : Data Set 866 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 11 ; pihat0 = 0.1081335
N0. 82 : Data Set 231 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 9 ; pihat0 = 0.1099259
N0. 83 : Data Set 532 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 9 ; pihat0 = 0.1006315
N0. 84 : Data Set 241 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 8 ; pihat0 = 0.2951308
N0. 85 : Data Set 433 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 8 ; pihat0 = 0.06930144
N0. 86 : Data Set 949 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 8 ; pihat0 = 0.09631564
N0. 87 : Data Set 618 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 6 ; pihat0 = 0.07763429
N0. 88 : Data Set 711 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 6 ; pihat0 = 0.09199754
N0. 89 : Data Set 44 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 5 ; pihat0 = 0.07141674
N0. 90 : Data Set 153 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 5 ; pihat0 = 0.05761454
N0. 91 : Data Set 671 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 5 ; pihat0 = 0.1067464
N0. 92 : Data Set 849 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 4 ; pihat0 = 0.1557612
N0. 93 : Data Set 32 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1531939
N0. 94 : Data Set 71 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1989475
N0. 95 : Data Set 266 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.2153199
N0. 96 : Data Set 353 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1153508
N0. 97 : Data Set 487 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.3311973
N0. 98 : Data Set 799 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1880441
N0. 99 : Data Set 868 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1153411
N0. 100 : Data Set 877 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1894112
N0. 101 : Data Set 39 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1052546
N0. 102 : Data Set 132 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.109358
N0. 103 : Data Set 424 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1877424
N0. 104 : Data Set 502 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1171748
N0. 105 : Data Set 552 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1703334
N0. 106 : Data Set 27 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.2143327
N0. 107 : Data Set 70 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.30275
N0. 108 : Data Set 97 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.184987
N0. 109 : Data Set 180 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1318437
N0. 110 : Data Set 188 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.2319604
N0. 111 : Data Set 199 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1978562
N0. 112 : Data Set 280 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1960971
N0. 113 : Data Set 291 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1825971
N0. 114 : Data Set 393 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1925067
N0. 115 : Data Set 475 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.06974273
N0. 116 : Data Set 584 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.0723416
N0. 117 : Data Set 721 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1049651
N0. 118 : Data Set 892 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.2868517
lfsr
is more conservative than BH
.N0. 1 : Data Set 901 ; Number of False Discoveries by BH: 2 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.9997685
N0. 2 : Data Set 187 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.6662317
N0. 3 : Data Set 285 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.5178013
N0. 4 : Data Set 483 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.9998828
N0. 5 : Data Set 817 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 1
N0. 6 : Data Set 837 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.828704
N0. 7 : Data Set 955 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.9998902
BH
and lfsr
To get a more comprehensive picture, we plot as follows the numbers of false discoveries by BH
and lfsr
for each data set. These data sets were selected out of \(1000\) of them, because they produced at least one false discoveries by either method. Note that for every data set there are \(10K\) correlated null statistics.
sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.2
Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.4.3 backports_1.1.2 magrittr_1.5 rprojroot_1.3-1
[5] tools_3.4.3 htmltools_0.3.6 yaml_2.1.16 Rcpp_0.12.14
[9] stringi_1.1.6 rmarkdown_1.8 knitr_1.17 git2r_0.20.0
[13] stringr_1.2.0 digest_0.6.13 evaluate_0.10.1
This R Markdown site was created with workflowr