January 19, 2021

VirusTotal APK Malware Detection Data - Week 3: 20210111-20210117

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210111_20210117.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.10% 0.03% 97692 24 80967 883
Trustlook 98.68% 0.07% 97270 57 80934 1305
Fortinet 98.45% 0.01% 97049 10 80981 1526
Avira 98.19% 0.00% 96791 0 80991 1784
ZoneAlarm 96.84% 0.00% 95463 3 80988 3112
Ikarus 95.94% 0.10% 94577 80 80911 3998
K7GW 92.92% 0.07% 91594 56 80935 6981
Kaspersky 92.35% 0.00% 91032 1 80990 7543
AhnLab-V3 92.14% 0.02% 90827 16 80975 7748
Avast-Mobile 90.59% 0.20% 89304 162 80829 9271
F-Secure 86.92% 0.00% 85686 0 80991 12889
CAT-QuickHeal 86.50% 0.01% 85265 8 80983 13310
Sophos 84.86% 0.01% 83655 9 80982 14920
DrWeb 77.19% 0.10% 76089 85 80906 22486
Qihoo-360 77.03% 0.01% 75937 9 80982 22638
McAfee 73.63% 0.00% 72585 3 80988 25990
Microsoft 72.23% 0.02% 71205 14 80977 27370
McAfee-GW-Edition 71.12% 0.03% 70105 21 80970 28470
AVG 67.32% 0.01% 66356 10 80981 32219
Avast 66.42% 0.01% 65475 9 80982 33100
AegisLab 61.28% 0.16% 60410 129 80862 38165
NANO-Antivirus 61.21% 0.02% 60333 14 80977 38242
Tencent 57.10% 0.05% 56290 41 80950 42285
Cyren 56.89% 0.00% 56076 3 80988 42499
Symantec 50.62% 0.01% 49897 10 80981 48678
Alibaba 44.39% 0.00% 43758 3 80988 54817
Antiy-AVL 42.75% 0.11% 42145 90 80901 56430
Comodo 32.72% 0.03% 32258 24 80967 66317
MAX 29.27% 0.00% 28857 0 80991 69718
ClamAV 27.63% 0.18% 27240 147 80844 71335
Jiangmin 16.77% 0.19% 16531 156 80835 82044
Rising 12.99% 0.00% 12800 4 80987 85775
GData 12.98% 0.00% 12798 0 80991 85777
Zillya 12.97% 0.03% 12786 24 80967 85789
TrendMicro 11.81% 0.04% 11643 29 80962 86932
TrendMicro-HouseCall 11.67% 0.03% 11508 26 80965 87067
VBA32 10.53% 0.01% 10380 5 80986 88195
BitDefender 7.77% 0.00% 7657 0 80991 90918
Emsisoft 7.70% 0.00% 7588 0 80991 90987
Arcabit 7.58% 0.00% 7471 0 80991 91104
Yandex 6.63% 0.00% 6538 1 80990 92037
Kingsoft 5.90% 0.01% 5813 5 80986 92762
Panda 3.90% 0.00% 3849 0 80991 94726
Zoner 3.53% 0.00% 3477 3 80988 95098
MicroWorld-eScan 3.11% 0.00% 3063 0 80991 95512
Ad-Aware 1.45% 0.00% 1428 0 80991 97147
TotalDefense 0.73% 0.00% 718 0 80991 97857
K7AntiVirus 0.08% 0.00% 74 0 80991 98501
Baidu 0.04% 0.00% 36 4 80987 98539
ALYac 0.02% 0.00% 24 0 80991 98551
ViRobot 0.02% 0.00% 19 0 80991 98556
VIPRE 0.01% 0.00% 11 0 80991 98564
SentinelOne 0.00% 0.00% 3 0 80991 98572
Bkav 0.00% 0.00% 0 0 80991 98575
nProtect 0.00% 0.00% 0 0 80991 98575
CMC 0.00% 0.00% 0 0 80991 98575
Malwarebytes 0.00% 0.00% 0 0 80991 98575
CrowdStrike 0.00% 0.00% 0 0 80991 98575
SymantecMobileInsight 0.00% 0.00% 0 0 80991 98575
TheHacker 0.00% 0.00% 0 0 80991 98575
eScan 0.00% 0.00% 0 0 80991 98575
Babable 0.00% 0.00% 0 0 80991 98575
SUPERAntiSpyware 0.00% 0.00% 0 0 80991 98575
Invincea 0.00% 0.00% 0 0 80991 98575
F-Prot 0.00% 0.00% 0 0 80991 98575
Endgame 0.00% 0.00% 0 0 80991 98575
Webroot 0.00% 0.00% 0 0 80991 98575
AVware 0.00% 0.00% 0 0 80991 98575
TotalGoodware 80991
TotalMalware 98575
TotalSample 179566

Please send an email to lxu@trustlook.com if you have any comments. Thanks.