February 8, 2021

VirusTotal APK Malware Detection Data - Week 6: 20210201-20210207

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210201_20210207.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.55% 0.03% 51064 20 66733 230
Trustlook 99.24% 0.16% 50904 107 66646 390
Avast-Mobile 99.24% 0.27% 50902 183 66570 392
Fortinet 98.60% 0.02% 50575 15 66738 719
Avira 98.29% 0.00% 50416 1 66752 878
DrWeb 97.92% 0.15% 50226 100 66653 1068
K7GW 96.38% 0.09% 49436 61 66692 1858
ZoneAlarm 95.97% 0.02% 49229 16 66737 2065
Ikarus 94.93% 0.11% 48691 74 66679 2603
Kaspersky 93.91% 0.01% 48171 10 66743 3123
Sophos 93.91% 0.02% 48168 13 66740 3126
AhnLab-V3 92.49% 0.02% 47444 16 66737 3850
F-Secure 91.78% 0.00% 47076 2 66751 4218
Microsoft 81.52% 0.04% 41813 29 66724 9481
CAT-QuickHeal 81.25% 0.01% 41676 7 66746 9618
Cyren 81.05% 0.01% 41574 5 66748 9720
McAfee 79.58% 0.00% 40818 3 66750 10476
NANO-Antivirus 78.37% 0.03% 40201 20 66733 11093
Qihoo-360 77.64% 0.01% 39826 8 66745 11468
McAfee-GW-Edition 74.68% 0.01% 38304 9 66744 12990
AegisLab 70.43% 0.24% 36126 159 66594 15168
Avast 69.33% 0.02% 35561 14 66739 15733
AVG 69.33% 0.02% 35561 14 66739 15733
SymantecMobileInsight 67.82% 2.77% 34786 1851 64902 16508
Antiy-AVL 56.52% 0.19% 28993 127 66626 22301
Tencent 55.55% 0.08% 28492 51 66702 22802
Symantec 36.83% 0.03% 18894 17 66736 32400
Comodo 36.00% 0.14% 18468 95 66658 32826
MAX 25.84% 0.00% 13252 1 66752 38042
ClamAV 24.91% 0.19% 12779 126 66627 38515
Alibaba 24.68% 0.02% 12657 12 66741 38637
Rising 19.61% 0.01% 10058 6 66747 41236
Zillya 17.84% 0.08% 9150 52 66701 42144
Jiangmin 13.98% 0.39% 7171 258 66495 44123
TrendMicro 13.77% 0.15% 7062 102 66651 44232
TrendMicro-HouseCall 13.64% 0.16% 6998 109 66644 44296
Yandex 13.42% 0.01% 6885 4 66749 44409
VIPRE 13.21% 0.00% 6777 0 66753 44517
GData 12.51% 0.00% 6416 0 66753 44878
Kingsoft 8.62% 0.00% 4421 1 66752 46873
VBA32 6.90% 0.02% 3538 15 66738 47756
BitDefender 6.81% 0.00% 3495 0 66753 47799
Emsisoft 6.76% 0.00% 3466 0 66753 47828
Arcabit 6.73% 0.00% 3450 0 66753 47844
Zoner 3.17% 0.01% 1624 8 66745 49670
MicroWorld-eScan 3.16% 0.00% 1622 0 66753 49672
Panda 1.75% 0.00% 896 0 66753 50398
Ad-Aware 1.12% 0.00% 572 0 66753 50722
TotalDefense 0.58% 0.00% 297 0 66753 50997
K7AntiVirus 0.25% 0.00% 129 0 66753 51165
Baidu 0.17% 0.01% 86 10 66743 51208
ViRobot 0.12% 0.00% 60 0 66753 51234
SentinelOne 0.08% 0.00% 42 0 66753 51252
ALYac 0.03% 0.00% 13 0 66753 51281
Malwarebytes 0.02% 0.00% 10 0 66753 51284
Bkav 0.00% 0.00% 0 0 66753 51294
nProtect 0.00% 0.00% 0 0 66753 51294
CMC 0.00% 0.00% 0 0 66753 51294
CrowdStrike 0.00% 0.00% 0 0 66753 51294
TheHacker 0.00% 0.00% 0 0 66753 51294
eScan 0.00% 0.00% 0 0 66753 51294
Babable 0.00% 0.00% 0 0 66753 51294
SUPERAntiSpyware 0.00% 0.00% 0 0 66753 51294
Invincea 0.00% 0.00% 0 0 66753 51294
F-Prot 0.00% 0.00% 0 0 66753 51294
Endgame 0.00% 0.00% 0 0 66753 51294
Webroot 0.00% 0.00% 0 0 66753 51294
AVware 0.00% 0.00% 0 0 66753 51294
TotalGoodware 66753
TotalMalware 51294
TotalSample 118047

Please send an email to lxu@trustlook.com if you have any comments. Thanks.