April 14, 2021

VirusTotal APK Malware Detection Data - Week 15: 202100405-20210411

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210405_20210411.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

Vendor TPR FPR TP FP TN FN
K7GW 99.89% 0.06% 35167 67 106197 40
CAT-QuickHeal 99.59% 0.00% 35062 5 106259 145
Trustlook 99.33% 0.07% 34970 76 106188 237
ESET-NOD32 99.32% 0.01% 34969 14 106250 238
Avira 99.11% 0.00% 34893 0 106264 314
Fortinet 98.75% 0.01% 34766 13 106251 441
AhnLab-V3 97.15% 0.02% 34205 17 106247 1002
McAfee 96.60% 0.01% 34010 6 106258 1197
Avast-Mobile 95.39% 0.26% 33584 279 105985 1623
SymantecMobileInsight 95.03% 2.00% 33456 2123 104141 1751
McAfee-GW-Edition 94.47% 0.00% 33260 5 106259 1947
Symantec 92.80% 0.19% 32673 200 106064 2534
Tencent 90.42% 0.06% 31833 67 106197 3374
Kaspersky 90.22% 0.00% 31765 1 106263 3442
AegisLab 87.61% 0.05% 30844 48 106216 4363
Ikarus 86.43% 0.07% 30430 78 106186 4777
Qihoo-360 85.83% 0.01% 30217 13 106251 4990
DrWeb 85.66% 0.09% 30158 91 106173 5049
Alibaba 84.97% 0.00% 29915 5 106259 5292
NANO-Antivirus 84.72% 0.03% 29826 31 106233 5381
Microsoft 83.07% 0.02% 29247 17 106247 5960
Sophos 78.38% 0.01% 27594 6 106258 7613
ZoneAlarm 66.96% 0.00% 23575 2 106262 11632
MAX 60.10% 0.00% 21160 1 106263 14047
Cyren 54.53% 0.01% 19200 6 106258 16007
Avast 52.92% 0.01% 18631 11 106253 16576
AVG 52.92% 0.01% 18631 11 106253 16576
Zillya 35.15% 0.13% 12375 136 106128 22832
Comodo 34.42% 0.06% 12120 68 106196 23087
ClamAV 23.80% 0.13% 8378 139 106125 26829
Jiangmin 14.59% 0.25% 5137 264 106000 30070
Yandex 13.31% 0.00% 4687 4 106260 30520
GData 12.73% 0.00% 4483 0 106264 30724
Rising 11.17% 0.00% 3934 4 106260 31273
BitDefender 9.13% 0.00% 3214 0 106264 31993
Emsisoft 9.01% 0.00% 3173 0 106264 32034
Antiy-AVL 8.17% 0.05% 2878 54 106210 32329
Kingsoft 7.67% 0.01% 2699 6 106258 32508
VBA32 6.74% 0.01% 2374 9 106255 32833
TrendMicro-HouseCall 6.50% 0.00% 2290 5 106259 32917
TrendMicro 5.43% 0.01% 1913 11 106253 33294
Arcabit 5.42% 0.00% 1908 0 106264 33299
Zoner 5.37% 0.00% 1889 4 106260 33318
F-Secure 4.83% 0.00% 1701 0 106264 33506
MicroWorld-eScan 2.46% 0.00% 867 0 106264 34340
Panda 2.45% 0.01% 864 6 106258 34343
Ad-Aware 0.67% 0.00% 237 0 106264 34970
Babable 0.24% 0.01% 86 15 106249 35121
Baidu 0.05% 0.01% 18 11 106253 35189
F-Prot 0.05% 0.00% 17 0 106264 35190
ViRobot 0.03% 0.00% 12 0 106264 35195
K7AntiVirus 0.01% 0.00% 5 0 106264 35202
SentinelOne 0.01% 0.00% 2 0 106264 35205
ALYac 0.00% 0.00% 1 0 106264 35206
Malwarebytes 0.00% 0.00% 1 2 106262 35206
VIPRE 0.00% 0.00% 1 0 106264 35206
Bkav 0.00% 0.00% 0 1 106263 35207
TotalDefense 0.00% 0.00% 0 0 106264 35207
nProtect 0.00% 0.00% 0 0 106264 35207
CMC 0.00% 0.00% 0 0 106264 35207
CrowdStrike 0.00% 0.00% 0 0 106264 35207
TheHacker 0.00% 0.00% 0 0 106264 35207
eScan 0.00% 0.00% 0 0 106264 35207
SUPERAntiSpyware 0.00% 0.00% 0 0 106264 35207
Invincea 0.00% 0.00% 0 0 106264 35207
Endgame 0.00% 0.00% 0 0 106264 35207
Webroot 0.00% 0.00% 0 0 106264 35207
AVware 0.00% 0.00% 0 0 106264 35207
TotalGoodware 106264
TotalMalware 35207
TotalSample 141471

Please send an email to lxu@trustlook.com if you have any comments. Thanks.