December 1, 2021

VirusTotal APK Malware Detection Data 2021-10

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a monthly basis, we publish the detection results and zip the CSV files to AWS S3. For this month, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20211001_20211031.zip

The monthly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.63% 0.04% 7517 66 165131 28
Fortinet 99.48% 0.10% 7506 159 165038 39
K7GW 99.44% 0.10% 7503 164 165033 42
Avast-Mobile 98.91% 0.82% 7463 1352 163845 82
DrWeb 98.22% 0.09% 7411 142 165055 134
Ikarus 98.14% 0.06% 7405 101 165096 140
Avira 97.80% 0.00% 7379 4 165193 166
CAT-QuickHeal 96.39% 0.05% 7273 85 165112 272
AhnLab-V3 94.98% 0.02% 7166 29 165168 379
Kaspersky 94.13% 0.01% 7102 20 165177 443
Trustlook 93.76% 0.00% 7074 7 165190 471
McAfee 92.60% 0.02% 6987 34 165163 558
Microsoft 87.28% 0.03% 6585 52 165145 960
McAfee-GW-Edition 84.96% 0.02% 6410 35 165162 1135
Sophos 83.46% 0.02% 6297 27 165170 1248
NANO-Antivirus 78.52% 0.02% 5924 41 165156 1621
Antiy-AVL 74.16% 0.10% 5595 173 165024 1950
SymantecMobileInsight 74.04% 2.22% 5586 3675 161522 1959
Tencent 69.65% 0.11% 5255 184 165013 2290
Cyren 64.15% 0.02% 4840 30 165167 2705
Symantec 57.99% 0.00% 4375 3 165194 3170
Alibaba 56.30% 0.00% 4248 6 165191 3297
AVG 55.26% 0.01% 4169 22 165175 3376
Avast 55.14% 0.01% 4160 22 165175 3385
MAX 52.72% 0.00% 3978 0 165197 3567
ZoneAlarm 48.32% 0.01% 3646 19 165178 3899
Comodo 42.32% 0.07% 3193 120 165077 4352
Zillya 30.88% 0.06% 2330 93 165104 5215
ClamAV 17.81% 0.07% 1344 113 165084 6201
Jiangmin 15.79% 0.43% 1191 713 164484 6354
Kingsoft 15.40% 0.01% 1162 24 165173 6383
GData 15.06% 0.00% 1136 1 165196 6409
BitDefender 14.25% 0.00% 1075 0 165197 6470
Emsisoft 14.12% 0.00% 1065 0 165197 6480
Yandex 10.40% 0.00% 785 7 165190 6760
F-Secure 9.94% 0.00% 750 1 165196 6795
Arcabit 8.56% 0.00% 646 1 165196 6899
MicroWorld-eScan 8.40% 0.00% 634 0 165197 6911
Zoner 7.20% 0.00% 543 4 165193 7002
Rising 6.03% 0.01% 455 10 165187 7090
TrendMicro-HouseCall 5.99% 0.01% 452 21 165176 7093
TrendMicro 4.53% 0.02% 342 34 165163 7203
Ad-Aware 2.92% 0.00% 220 0 165197 7325
VBA32 1.02% 0.01% 77 9 165188 7468
Panda 0.34% 0.00% 26 0 165197 7519
Baidu 0.28% 0.01% 21 15 165182 7524
SentinelOne 0.28% 0.00% 21 0 165197 7524
ViRobot 0.19% 0.00% 14 0 165197 7531
Malwarebytes 0.05% 0.00% 4 0 165197 7541
ALYac 0.04% 0.00% 3 0 165197 7542
SUPERAntiSpyware 0.04% 0.00% 3 0 165197 7542
VIPRE 0.04% 0.00% 3 0 165197 7542
CMC 0.03% 0.00% 2 0 165197 7543
K7AntiVirus 0.03% 0.00% 2 0 165197 7543
Bkav 0.00% 0.00% 0 0 165197 7545
TotalDefense 0.00% 0.00% 0 0 165197 7545
nProtect 0.00% 0.00% 0 0 165197 7545
CrowdStrike 0.00% 0.00% 0 0 165197 7545
TheHacker 0.00% 0.00% 0 0 165197 7545
eScan 0.00% 0.00% 0 0 165197 7545
Babable 0.00% 0.00% 0 0 165197 7545
Invincea 0.00% 0.00% 0 0 165197 7545
F-Prot 0.00% 0.00% 0 0 165197 7545
Endgame 0.00% 0.00% 0 0 165197 7545
Webroot 0.00% 0.00% 0 0 165197 7545
AegisLab 0.00% 0.00% 0 0 165197 7545
AVware 0.00% 0.00% 0 0 165197 7545
Qihoo-360 0.00% 0.00% 0 0 165197 7545
TotalGoodware 165197
TotalMalware 7545
TotalSample 172742

Please send an email to lxu@trustlook.com if you have any comments. Thanks.