VirusTotal APK Malware Detection Data 2021-08

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a monthly basis, we publish the detection results and zip the CSV files to AWS S3. For this month, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210801_20210831.zip

The monthly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.52% 0.03% 17282 322 1025420 83
Fortinet 99.50% 0.03% 17279 310 1025432 86
K7GW 98.89% 0.10% 17173 1063 1024679 192
Avira 98.63% 0.00% 17127 42 1025700 238
Ikarus 98.46% 0.06% 17098 565 1025177 267
Trustlook 97.54% 0.04% 16938 459 1025283 427
Avast-Mobile 97.24% 0.24% 16885 2504 1023238 480
DrWeb 96.17% 0.15% 16700 1521 1024221 665
McAfee 95.53% 0.01% 16588 67 1025675 777
CAT-QuickHeal 92.73% 0.02% 16102 217 1025525 1263
AhnLab-V3 92.31% 0.02% 16030 209 1025533 1335
Kaspersky 92.02% 0.00% 15979 41 1025701 1386
Microsoft 88.21% 0.03% 15318 292 1025450 2047
McAfee-GW-Edition 86.97% 0.02% 15102 177 1025565 2263
Sophos 81.46% 0.01% 14145 138 1025604 3220
NANO-Antivirus 79.31% 0.04% 13772 459 1025283 3593
Symantec 76.69% 0.03% 13318 315 1025427 4047
Cyren 75.97% 0.00% 13192 44 1025698 4173
Antiy-AVL 72.74% 0.10% 12631 1069 1024673 4734
SymantecMobileInsight 72.54% 4.63% 12596 47501 978241 4769
Alibaba 59.01% 0.01% 10247 70 1025672 7118
Tencent 54.87% 0.10% 9528 1069 1024673 7837
ZoneAlarm 54.49% 0.01% 9462 106 1025636 7903
AVG 41.22% 0.02% 7158 158 1025584 10207
Avast 40.99% 0.02% 7118 158 1025584 10247
MAX 38.65% 0.00% 6712 34 1025708 10653
Comodo 33.20% 0.21% 5765 2113 1023629 11600
Zillya 20.48% 0.08% 3557 771 1024971 13808
Jiangmin 20.14% 1.12% 3497 11510 1014232 13868
ClamAV 17.33% 0.20% 3010 2005 1023737 14355
GData 13.41% 0.00% 2328 16 1025726 15037
Kingsoft 12.53% 0.01% 2176 128 1025614 15189
Yandex 10.78% 0.00% 1872 20 1025722 15493
BitDefender 10.45% 0.00% 1814 13 1025729 15551
Emsisoft 10.31% 0.00% 1790 15 1025727 15575
TrendMicro 8.79% 0.07% 1527 765 1024977 15838
TrendMicro-HouseCall 8.72% 0.04% 1515 370 1025372 15850
Rising 8.07% 0.02% 1402 175 1025567 15963
VBA32 7.66% 0.02% 1330 230 1025512 16035
F-Secure 6.59% 0.00% 1144 1 1025741 16221
MicroWorld-eScan 6.38% 0.00% 1108 12 1025730 16257
Zoner 6.13% 0.01% 1065 89 1025653 16300
Arcabit 5.00% 0.00% 869 20 1025722 16496
Ad-Aware 3.33% 0.00% 579 11 1025731 16786
Panda 2.06% 0.00% 358 7 1025735 17007
Baidu 1.30% 0.01% 226 125 1025617 17139
ViRobot 0.98% 0.00% 170 31 1025711 17195
Qihoo-360 0.88% 0.00% 152 4 1025738 17213
ALYac 0.44% 0.00% 77 1 1025741 17288
SentinelOne 0.35% 0.00% 61 9 1025733 17304
K7AntiVirus 0.29% 0.00% 50 0 1025742 17315
Malwarebytes 0.21% 0.00% 36 6 1025736 17329
VIPRE 0.21% 0.00% 36 0 1025742 17329
Babable 0.12% 0.02% 20 176 1025566 17345
F-Prot 0.05% 0.00% 8 0 1025742 17357
SUPERAntiSpyware 0.03% 0.00% 5 0 1025742 17360
CMC 0.01% 0.00% 1 1 1025741 17364
Bkav 0.00% 0.00% 0 0 1025742 17365
TotalDefense 0.00% 0.00% 0 0 1025742 17365
nProtect 0.00% 0.00% 0 0 1025742 17365
CrowdStrike 0.00% 0.00% 0 0 1025742 17365
TheHacker 0.00% 0.00% 0 0 1025742 17365
eScan 0.00% 0.00% 0 0 1025742 17365
Invincea 0.00% 0.00% 0 0 1025742 17365
Endgame 0.00% 0.00% 0 0 1025742 17365
Webroot 0.00% 0.00% 0 0 1025742 17365
AegisLab 0.00% 0.00% 0 0 1025742 17365
AVware 0.00% 0.00% 0 0 1025742 17365
TotalGoodware 1025742
TotalMalware 17365
TotalSample 1043107

Please send an email to lxu@trustlook.com if you have any comments. Thanks.