VirusTotal APK Malware Detection Data 2021-09

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a monthly basis, we publish the detection results and zip the CSV files to AWS S3. For this month, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210901_20210930.zip

The monthly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

Vendor TPR FPR TP FP TN FN
K7GW 99.73% 0.10% 10871 207 205774 29
ESET-NOD32 99.65% 0.03% 10862 54 205927 38
Fortinet 99.61% 0.09% 10858 180 205801 42
Avast-Mobile 98.89% 0.72% 10779 1484 204497 121
Ikarus 98.28% 0.06% 10712 132 205849 188
DrWeb 98.07% 0.09% 10690 182 205799 210
Avira 98.06% 0.01% 10688 14 205967 212
Trustlook 96.25% 0.00% 10491 10 205971 409
Kaspersky 94.47% 0.02% 10297 31 205950 603
McAfee 94.14% 0.03% 10261 54 205927 639
AhnLab-V3 93.65% 0.02% 10208 35 205946 692
CAT-QuickHeal 92.50% 0.05% 10082 104 205877 818
Microsoft 90.21% 0.02% 9833 44 205937 1067
McAfee-GW-Edition 86.45% 0.03% 9423 56 205925 1477
Sophos 80.59% 0.01% 8784 25 205956 2116
NANO-Antivirus 76.53% 0.02% 8342 43 205938 2558
SymantecMobileInsight 73.94% 2.69% 8059 5536 200445 2841
Cyren 68.58% 0.02% 7475 41 205940 3425
Tencent 67.96% 0.12% 7408 251 205730 3492
Symantec 66.94% 0.00% 7296 4 205977 3604
Alibaba 63.95% 0.00% 6971 8 205973 3929
Antiy-AVL 63.48% 0.09% 6919 185 205796 3981
MAX 55.36% 0.00% 6034 4 205977 4866
ZoneAlarm 48.06% 0.02% 5239 36 205945 5661
AVG 42.71% 0.02% 4655 31 205950 6245
Avast 42.56% 0.02% 4639 31 205950 6261
Comodo 28.25% 0.09% 3079 186 205795 7821
Zillya 26.37% 0.08% 2874 157 205824 8026
GData 16.16% 0.00% 1761 1 205980 9139
Yandex 14.40% 0.00% 1570 4 205977 9330
ClamAV 14.33% 0.08% 1562 174 205807 9338
BitDefender 14.19% 0.00% 1547 1 205980 9353
Emsisoft 14.02% 0.00% 1528 0 205981 9372
Jiangmin 11.43% 0.48% 1246 991 204990 9654
Kingsoft 9.48% 0.01% 1033 27 205954 9867
MicroWorld-eScan 9.45% 0.00% 1030 0 205981 9870
F-Secure 9.39% 0.00% 1024 1 205980 9876
Arcabit 8.06% 0.00% 879 1 205980 10021
Ad-Aware 4.09% 0.00% 446 0 205981 10454
Zoner 4.08% 0.00% 445 8 205973 10455
TrendMicro-HouseCall 3.80% 0.02% 414 36 205945 10486
TrendMicro 3.25% 0.03% 354 65 205916 10546
Rising 1.98% 0.01% 216 14 205967 10684
VBA32 0.94% 0.01% 103 28 205953 10797
Panda 0.58% 0.00% 63 1 205980 10837
Baidu 0.30% 0.01% 33 27 205954 10867
ViRobot 0.26% 0.00% 28 1 205980 10872
SentinelOne 0.08% 0.00% 9 0 205981 10891
VIPRE 0.04% 0.00% 4 0 205981 10896
K7AntiVirus 0.03% 0.00% 3 0 205981 10897
Malwarebytes 0.02% 0.00% 2 1 205980 10898
SUPERAntiSpyware 0.02% 0.00% 2 0 205981 10898
ALYac 0.01% 0.00% 1 0 205981 10899
Bkav 0.00% 0.00% 0 0 205981 10900
TotalDefense 0.00% 0.00% 0 0 205981 10900
nProtect 0.00% 0.00% 0 0 205981 10900
CMC 0.00% 0.00% 0 0 205981 10900
CrowdStrike 0.00% 0.00% 0 0 205981 10900
TheHacker 0.00% 0.00% 0 0 205981 10900
eScan 0.00% 0.00% 0 0 205981 10900
Babable 0.00% 0.00% 0 0 205981 10900
Invincea 0.00% 0.00% 0 0 205981 10900
F-Prot 0.00% 0.00% 0 0 205981 10900
Endgame 0.00% 0.00% 0 0 205981 10900
Webroot 0.00% 0.00% 0 0 205981 10900
AegisLab 0.00% 0.00% 0 0 205981 10900
AVware 0.00% 0.00% 0 0 205981 10900
Qihoo-360 0.00% 0.00% 0 0 205981 10900
TotalGoodware 205981
TotalMalware 10900
TotalSample 216881

Please send an email to lxu@trustlook.com if you have any comments. Thanks.