VirusTotal APK Malware Detection Data 20220101-20220831

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

We publish the detection results and zip the CSV files to AWS S3. For this test, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20220101_20220831.zip

The monthly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

Vendor TPR FPR TP FP TN FN
K7GW 99.61% 0.19% 347789 4780 2506813 1374
ESET-NOD32 99.47% 0.09% 347327 2283 2509310 1836
Fortinet 98.76% 0.13% 344834 3335 2508258 4329
Trustlook 98.49% 0.02% 343874 400 2511193 5289
Avira 98.46% 0.00% 343773 118 2511475 5390
AhnLab-V3 96.45% 0.01% 336760 281 2511312 12403
Kaspersky 95.67% 0.02% 334061 441 2511152 15102
Avast-Mobile 90.59% 0.89% 316324 22373 2489220 32839
Ikarus 90.39% 0.14% 315599 3455 2508138 33564
Microsoft 89.95% 0.02% 314082 478 2511115 35081
McAfee 89.17% 0.03% 311346 701 2510892 37817
Sophos 87.92% 0.01% 306982 268 2511325 42181
DrWeb 86.38% 0.09% 301601 2277 2509316 47562
Tencent 86.17% 0.12% 300868 2960 2508633 48295
McAfee-GW-Edition 84.37% 0.07% 294604 1703 2509890 54559
CAT-QuickHeal 83.07% 0.02% 290046 533 2511060 59117
SymantecMobileInsight 80.29% 2.08% 280327 52256 2459337 68836
Alibaba 74.36% 0.01% 259624 281 2511312 89539
Symantec 66.21% 0.01% 231184 163 2511430 117979
MAX 62.09% 0.00% 216791 36 2511557 132372
NANO-Antivirus 61.75% 0.03% 215604 824 2510769 133559
Cyren 58.51% 0.00% 204286 116 2511477 144877
AVG 54.84% 0.02% 191491 578 2511015 157672
Avast 54.80% 0.02% 191337 576 2511017 157826
Antiy-AVL 54.61% 0.14% 190680 3617 2507976 158483
ZoneAlarm 54.40% 0.01% 189946 353 2511240 159217
Comodo 24.09% 0.05% 84102 1316 2510277 265061
Zillya 22.71% 0.08% 79284 2106 2509487 269879
F-Secure 15.71% 0.00% 54868 30 2511563 294295
Rising 15.28% 0.02% 53335 441 2511152 295828
GData 14.34% 0.00% 50078 23 2511570 299085
Jiangmin 12.91% 0.36% 45086 8967 2502626 304077
Kingsoft 12.83% 0.03% 44807 665 2510928 304356
BitDefender 10.99% 0.00% 38388 15 2511578 310775
Emsisoft 10.69% 0.00% 37333 14 2511579 311830
ClamAV 10.07% 0.23% 35167 5817 2505776 313996
TrendMicro 8.06% 0.02% 28147 442 2511151 321016
TrendMicro-HouseCall 7.03% 0.01% 24543 214 2511379 324620
Arcabit 6.40% 0.00% 22340 42 2511551 326823
MicroWorld-eScan 5.70% 0.00% 19897 12 2511581 329266
Yandex 5.67% 0.00% 19811 56 2511537 329352
VIPRE 4.74% 0.00% 16557 12 2511581 332606
VBA32 1.46% 0.02% 5099 466 2511127 344064
Ad-Aware 1.23% 0.00% 4308 9 2511584 344855
Panda 1.03% 0.00% 3592 25 2511568 345571
SentinelOne 0.49% 0.00% 1708 56 2511537 347455
Baidu 0.21% 0.01% 734 249 2511344 348429
ViRobot 0.14% 0.00% 500 5 2511588 348663
ALYac 0.07% 0.00% 244 5 2511588 348919
Malwarebytes 0.05% 0.00% 188 13 2511580 348975
Zoner 0.05% 0.00% 171 32 2511561 348992
K7AntiVirus 0.03% 0.00% 117 0 2511593 349046
SUPERAntiSpyware 0.00% 0.00% 10 0 2511593 349153
CMC 0.00% 0.00% 7 0 2511593 349156
Bkav 0.00% 0.03% 1 809 2510784 349162
TotalDefense 0.00% 0.00% 0 0 2511593 349163
nProtect 0.00% 0.00% 0 0 2511593 349163
CrowdStrike 0.00% 0.00% 0 0 2511593 349163
TheHacker 0.00% 0.00% 0 0 2511593 349163
eScan 0.00% 0.00% 0 0 2511593 349163
Babable 0.00% 0.00% 0 0 2511593 349163
Invincea 0.00% 0.00% 0 0 2511593 349163
F-Prot 0.00% 0.00% 0 0 2511593 349163
Endgame 0.00% 0.00% 0 0 2511593 349163
Webroot 0.00% 0.00% 0 0 2511593 349163
AegisLab 0.00% 0.00% 0 0 2511593 349163
AVware 0.00% 0.00% 0 0 2511593 349163
Qihoo-360 0.00% 0.00% 0 0 2511593 349163
TotalGoodware 2511593
TotalMalware 349163
TotalSample 2860756

Please send an email to lxu@trustlook.com if you have any comments. Thanks.