January 25, 2021

VirusTotal APK Malware Detection Data - Week 4: 20210118-20210124

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210118_20210124.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.31% 0.04% 57329 50 122113 398
Trustlook 98.95% 1.24% 57119 1518 120645 608
Fortinet 98.83% 0.03% 57050 32 122131 677
Avira 98.56% 0.01% 56896 9 122154 831
AhnLab-V3 97.07% 0.03% 56033 42 122121 1694
Avast-Mobile 96.65% 1.29% 55791 1576 120587 1936
ZoneAlarm 95.78% 0.10% 55291 117 122046 2436
K7GW 94.52% 0.28% 54561 342 121821 3166
Ikarus 93.28% 0.14% 53846 171 121992 3881
Kaspersky 90.27% 0.01% 52109 7 122156 5618
DrWeb 88.97% 0.17% 51358 211 121952 6369
Sophos 87.46% 0.01% 50489 12 122151 7238
F-Secure 78.80% 0.00% 45491 2 122161 12236
McAfee 75.14% 0.00% 43378 4 122159 14349
McAfee-GW-Edition 74.24% 0.16% 42857 192 121971 14870
CAT-QuickHeal 70.41% 0.01% 40645 18 122145 17082
Microsoft 69.59% 0.03% 40170 37 122126 17557
Qihoo-360 68.00% 0.02% 39254 26 122137 18473
Cyren 64.04% 0.01% 36969 10 122153 20758
Antiy-AVL 58.71% 1.08% 33893 1325 120838 23834
AegisLab 57.13% 0.52% 32977 634 121529 24750
NANO-Antivirus 56.83% 0.03% 32807 35 122128 24920
AVG 48.90% 0.02% 28226 28 122135 29501
Avast 48.89% 0.02% 28225 28 122135 29502
Tencent 44.47% 0.07% 25672 89 122074 32055
Symantec 33.18% 0.02% 19154 29 122134 38573
Comodo 26.75% 0.29% 15442 351 121812 42285
Alibaba 26.56% 0.06% 15335 73 122090 42392
ClamAV 25.48% 0.59% 14711 715 121448 43016
Jiangmin 20.03% 0.93% 11562 1141 121022 46165
MAX 19.16% 0.00% 11059 3 122160 46668
Zillya 14.01% 0.09% 8089 116 122047 49638
Yandex 11.65% 0.04% 6725 50 122113 51002
Rising 11.11% 0.02% 6411 25 122138 51316
Kingsoft 10.64% 0.01% 6142 14 122149 51585
GData 10.12% 0.00% 5844 0 122163 51883
VBA32 8.42% 0.03% 4858 42 122121 52869
TrendMicro 8.30% 0.09% 4793 107 122056 52934
TrendMicro-HouseCall 8.21% 0.13% 4741 153 122010 52986
Emsisoft 7.41% 0.00% 4276 0 122163 53451
BitDefender 7.40% 0.00% 4269 0 122163 53458
Arcabit 7.10% 0.00% 4101 6 122157 53626
Zoner 3.58% 0.03% 2067 39 122124 55660
MicroWorld-eScan 2.50% 0.00% 1443 0 122163 56284
Panda 2.05% 0.00% 1183 0 122163 56544
Ad-Aware 0.71% 0.00% 410 0 122163 57317
TotalDefense 0.50% 0.00% 286 0 122163 57441
K7AntiVirus 0.10% 0.00% 58 3 122160 57669
Baidu 0.08% 0.05% 47 55 122108 57680
SentinelOne 0.07% 0.00% 39 4 122159 57688
ViRobot 0.04% 0.00% 23 1 122162 57704
ALYac 0.04% 0.00% 22 1 122162 57705
Malwarebytes 0.02% 0.01% 11 10 122153 57716
VIPRE 0.00% 0.00% 1 0 122163 57726
Bkav 0.00% 0.03% 0 32 122131 57727
nProtect 0.00% 0.00% 0 0 122163 57727
CMC 0.00% 0.00% 0 1 122162 57727
CrowdStrike 0.00% 0.00% 0 0 122163 57727
SymantecMobileInsight 0.00% 0.00% 0 0 122163 57727
TheHacker 0.00% 0.00% 0 0 122163 57727
eScan 0.00% 0.00% 0 0 122163 57727
Babable 0.00% 0.00% 0 0 122163 57727
SUPERAntiSpyware 0.00% 0.00% 0 0 122163 57727
Invincea 0.00% 0.00% 0 0 122163 57727
F-Prot 0.00% 0.00% 0 0 122163 57727
Endgame 0.00% 0.00% 0 0 122163 57727
Webroot 0.00% 0.00% 0 0 122163 57727
AVware 0.00% 0.00% 0 0 122163 57727
TotalGoodware 122163
TotalMalware 57727
TotalSample 179890

Please send an email to lxu@trustlook.com if you have any comments. Thanks.