VirusTotal APK Malware Detection Data 2021-11

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a monthly basis, we publish the detection results and zip the CSV files to AWS S3. For this month, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20211101_20211130.zip

The monthly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

Vendor TPR FPR TP FP TN FN
Fortinet 99.73% 0.05% 17964 154 317562 48
ESET-NOD32 99.66% 0.02% 17951 53 317663 61
K7GW 99.01% 0.05% 17834 171 317545 178
Avast-Mobile 98.92% 0.38% 17818 1207 316509 194
Avira 98.63% 0.00% 17766 9 317707 246
DrWeb 97.72% 0.05% 17602 165 317551 410
Ikarus 95.35% 0.03% 17175 83 317633 837
AhnLab-V3 94.96% 0.01% 17104 38 317678 908
McAfee 93.86% 0.01% 16906 18 317698 1106
Trustlook 93.69% 0.02% 16876 66 317650 1136
CAT-QuickHeal 93.65% 0.01% 16868 22 317694 1144
Kaspersky 92.99% 0.01% 16749 17 317699 1263
Microsoft 91.13% 0.02% 16414 61 317655 1598
McAfee-GW-Edition 86.62% 0.01% 15602 39 317677 2410
SymantecMobileInsight 80.61% 1.55% 14519 4938 312778 3493
Tencent 78.95% 0.08% 14220 245 317471 3792
Sophos 78.53% 0.01% 14145 26 317690 3867
Symantec 75.87% 0.01% 13665 18 317698 4347
NANO-Antivirus 64.73% 0.01% 11660 46 317670 6352
Alibaba 64.50% 0.00% 11617 7 317709 6395
Cyren 61.87% 0.01% 11144 23 317693 6868
Antiy-AVL 59.12% 0.08% 10649 245 317471 7363
AVG 52.25% 0.02% 9412 53 317663 8600
Avast 52.17% 0.02% 9396 53 317663 8616
MAX 48.91% 0.00% 8809 1 317715 9203
Comodo 33.40% 0.05% 6016 164 317552 11996
Zillya 22.51% 0.05% 4054 169 317547 13958
Jiangmin 14.43% 0.47% 2599 1502 316214 15413
Kingsoft 14.35% 0.01% 2585 43 317673 15427
ClamAV 12.54% 0.09% 2258 273 317443 15754
GData 12.22% 0.00% 2201 0 317716 15811
BitDefender 11.11% 0.00% 2002 0 317716 16010
Emsisoft 10.90% 0.00% 1963 0 317716 16049
Yandex 7.80% 0.01% 1405 17 317699 16607
F-Secure 7.60% 0.00% 1369 1 317715 16643
ZoneAlarm 7.47% 0.00% 1346 3 317713 16666
Arcabit 6.16% 0.00% 1110 1 317715 16902
MicroWorld-eScan 6.12% 0.00% 1103 0 317716 16909
Rising 5.06% 0.01% 911 16 317700 17101
TrendMicro-HouseCall 3.09% 0.01% 556 41 317675 17456
TrendMicro 2.65% 0.02% 477 58 317658 17535
Ad-Aware 2.17% 0.00% 390 0 317716 17622
Zoner 1.47% 0.00% 264 15 317701 17748
VBA32 0.82% 0.01% 148 30 317686 17864
Panda 0.52% 0.00% 94 4 317712 17918
Baidu 0.52% 0.00% 93 15 317701 17919
SentinelOne 0.32% 0.00% 57 5 317711 17955
ViRobot 0.30% 0.00% 54 2 317714 17958
Malwarebytes 0.10% 0.00% 18 1 317715 17994
ALYac 0.08% 0.00% 14 0 317716 17998
K7AntiVirus 0.04% 0.00% 7 0 317716 18005
VIPRE 0.01% 0.00% 2 0 317716 18010
CMC 0.01% 0.00% 1 0 317716 18011
SUPERAntiSpyware 0.01% 0.00% 1 0 317716 18011
Bkav 0.00% 0.00% 0 0 317716 18012
TotalDefense 0.00% 0.00% 0 0 317716 18012
nProtect 0.00% 0.00% 0 0 317716 18012
CrowdStrike 0.00% 0.00% 0 0 317716 18012
TheHacker 0.00% 0.00% 0 0 317716 18012
eScan 0.00% 0.00% 0 0 317716 18012
Babable 0.00% 0.00% 0 0 317716 18012
Invincea 0.00% 0.00% 0 0 317716 18012
F-Prot 0.00% 0.00% 0 0 317716 18012
Endgame 0.00% 0.00% 0 0 317716 18012
Webroot 0.00% 0.00% 0 0 317716 18012
AegisLab 0.00% 0.00% 0 0 317716 18012
AVware 0.00% 0.00% 0 0 317716 18012
Qihoo-360 0.00% 0.00% 0 0 317716 18012
TotalGoodware 317716
TotalMalware 18012
TotalSample 335728

Please send an email to lxu@trustlook.com if you have any comments. Thanks.