VirusTotal APK Malware Detection Data 2021-07

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a monthly basis, we publish the detection results and zip the CSV files to AWS S3. For this month, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210701_20210731.zip

The monthly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

endor TPR FPR TP FP TN FN
ESET-NOD32 99.62% 0.01% 23237 20 230432 88
K7GW 99.31% 0.06% 23163 133 230319 162
Fortinet 99.25% 0.01% 23150 20 230432 175
Trustlook 99.22% 0.12% 23142 269 230183 183
Avira 98.24% 0.01% 22914 13 230439 411
Ikarus 98.20% 0.03% 22904 74 230378 421
DrWeb 97.48% 0.05% 22737 120 230332 588
Kaspersky 96.55% 0.01% 22520 13 230439 805
Avast-Mobile 96.21% 0.35% 22441 802 229650 884
CAT-QuickHeal 94.41% 0.02% 22022 40 230412 1303
McAfee 93.53% 0.01% 21817 23 230429 1508
AhnLab-V3 92.31% 0.02% 21531 44 230408 1794
Microsoft 88.28% 0.01% 20592 28 230424 2733
Sophos 87.70% 0.02% 20457 38 230414 2868
McAfee-GW-Edition 85.77% 0.01% 20007 30 230422 3318
NANO-Antivirus 81.11% 0.02% 18919 38 230414 4406
Cyren 79.37% 0.01% 18513 22 230430 4812
Symantec 72.24% 0.04% 16849 88 230364 6476
SymantecMobileInsight 68.14% 3.41% 15893 7861 222591 7432
Antiy-AVL 67.58% 0.18% 15763 423 230029 7562
ZoneAlarm 57.52% 0.01% 13417 25 230427 9908
Alibaba 52.47% 0.01% 12239 16 230436 11086
Tencent 51.80% 0.26% 12083 595 229857 11242
AVG 37.98% 0.03% 8859 73 230379 14466
Avast 37.90% 0.03% 8841 72 230380 14484
MAX 37.87% 0.00% 8833 1 230451 14492
Zillya 24.93% 0.19% 5815 433 230019 17510
Comodo 24.16% 0.65% 5636 1497 228955 17689
ClamAV 16.59% 0.16% 3870 371 230081 19455
Kingsoft 16.40% 0.02% 3825 57 230395 19500
AegisLab 12.40% 0.01% 2893 17 230435 20432
Jiangmin 10.94% 0.95% 2551 2200 228252 20774
GData 9.11% 0.00% 2126 8 230444 21199
F-Secure 8.07% 0.00% 1883 3 230449 21442
BitDefender 7.72% 0.00% 1800 8 230444 21525
Emsisoft 7.61% 0.00% 1776 8 230444 21549
Yandex 7.19% 0.01% 1678 14 230438 21647
TrendMicro-HouseCall 6.02% 0.05% 1405 118 230334 21920
TrendMicro 5.98% 0.06% 1394 133 230319 21931
Zoner 5.09% 0.02% 1188 39 230413 22137
Rising 4.71% 0.01% 1098 30 230422 22227
MicroWorld-eScan 4.21% 0.00% 981 8 230444 22344
Arcabit 4.18% 0.01% 976 13 230439 22349
Qihoo-360 3.24% 0.03% 756 61 230391 22569
VBA32 3.11% 0.03% 725 74 230378 22600
Ad-Aware 1.81% 0.00% 423 8 230444 22902
Panda 1.71% 0.00% 398 3 230449 22927
Baidu 1.30% 0.01% 304 22 230430 23021
ViRobot 0.94% 0.01% 220 14 230438 23105
SentinelOne 0.32% 0.00% 74 3 230449 23251
Malwarebytes 0.27% 0.00% 62 2 230450 23263
Babable 0.25% 0.02% 59 49 230403 23266
ALYac 0.24% 0.00% 56 0 230452 23269
K7AntiVirus 0.08% 0.00% 19 0 230452 23306
F-Prot 0.08% 0.00% 18 0 230452 23307
VIPRE 0.04% 0.00% 10 0 230452 23315
SUPERAntiSpyware 0.01% 0.00% 3 0 230452 23322
CMC 0.01% 0.00% 2 0 230452 23323
Bkav 0.00% 0.00% 0 0 230452 23325
TotalDefense 0.00% 0.00% 0 0 230452 23325
nProtect 0.00% 0.00% 0 0 230452 23325
CrowdStrike 0.00% 0.00% 0 0 230452 23325
TheHacker 0.00% 0.00% 0 0 230452 23325
eScan 0.00% 0.00% 0 0 230452 23325
Invincea 0.00% 0.00% 0 0 230452 23325
Endgame 0.00% 0.00% 0 0 230452 23325
Webroot 0.00% 0.00% 0 0 230452 23325
AVware 0.00% 0.00% 0 0 230452 23325
TotalGoodware 230452
TotalMalware 23325
TotalSample 253777

Please send an email to lxu@trustlook.com if you have any comments. Thanks.