January 4, 2021

VirusTotal APK Malware Detection Data - Week 1: 20201228-20210103

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20201228_20210103.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.75% 0.01% 28348 3 44573 70
K7GW 99.65% 0.06% 28318 25 44551 100
Avira 99.23% 0.00% 28199 0 44576 219
Fortinet 99.23% 0.01% 28198 5 44571 220
Trustlook 99.07% 0.06% 28155 25 44551 263
Avast-Mobile 98.90% 0.22% 28104 100 44476 314
DrWeb 98.03% 0.16% 27859 70 44506 559
ZoneAlarm 97.07% 0.01% 27584 5 44571 834
CAT-QuickHeal 96.42% 0.01% 27401 4 44572 1017
Ikarus 96.17% 0.16% 27331 71 44505 1087
Sophos 90.91% 0.02% 25836 9 44567 2582
McAfee 90.20% 0.01% 25632 3 44573 2786
SymantecMobileInsight 89.07% 1.64% 25311 732 43844 3107
Qihoo-360 88.62% 0.01% 25183 3 44573 3235
McAfee-GW-Edition 84.65% 0.03% 24056 12 44564 4362
NANO-Antivirus 84.51% 0.02% 24015 10 44566 4403
Microsoft 83.13% 0.02% 23623 11 44565 4795
AegisLab 82.25% 0.07% 23374 30 44546 5044
Antiy-AVL 79.60% 0.14% 22620 62 44514 5798
Symantec 77.36% 0.01% 21985 3 44573 6433
AhnLab-V3 76.84% 0.01% 21836 5 44571 6582
Tencent 74.83% 0.08% 21266 36 44540 7152
F-Secure 70.99% 0.00% 20173 1 44575 8245
Alibaba 69.19% 0.00% 19663 2 44574 8755
Kaspersky 65.02% 0.00% 18476 0 44576 9942
Comodo 60.39% 0.06% 17162 25 44551 11256
AVG 47.18% 0.01% 13409 6 44570 15009
MAX 47.10% 0.00% 13386 0 44576 15032
Cyren 45.15% 0.01% 12830 3 44573 15588
Avast 43.57% 0.01% 12381 5 44571 16037
ClamAV 16.88% 0.14% 4796 61 44515 23622
Jiangmin 12.78% 0.28% 3631 126 44450 24787
Rising 9.64% 0.01% 2739 6 44570 25679
Zillya 8.51% 0.03% 2417 13 44563 26001
GData 8.41% 0.00% 2389 0 44576 26029
TrendMicro 7.20% 0.07% 2047 31 44545 26371
TrendMicro-HouseCall 6.89% 0.04% 1959 20 44556 26459
Kingsoft 6.70% 0.00% 1904 1 44575 26514
Yandex 6.32% 0.00% 1795 0 44576 26623
BitDefender 6.14% 0.00% 1746 0 44576 26672
Emsisoft 6.10% 0.00% 1733 0 44576 26685
Arcabit 5.61% 0.00% 1593 0 44576 26825
Zoner 5.49% 0.00% 1559 0 44576 26859
VBA32 3.84% 0.00% 1092 2 44574 27326
MicroWorld-eScan 2.04% 0.00% 581 0 44576 27837
Panda 1.12% 0.00% 319 0 44576 28099
TotalDefense 1.10% 0.00% 313 0 44576 28105
Ad-Aware 0.78% 0.00% 221 0 44576 28197
Baidu 0.19% 0.00% 54 2 44574 28364
ViRobot 0.12% 0.00% 35 0 44576 28383
K7AntiVirus 0.05% 0.00% 15 0 44576 28403
SentinelOne 0.05% 0.00% 15 0 44576 28403
ALYac 0.03% 0.00% 8 0 44576 28410
Malwarebytes 0.01% 0.00% 4 0 44576 28414
VIPRE 0.00% 0.00% 1 0 44576 28417
Bkav 0.00% 0.00% 0 0 44576 28418
nProtect 0.00% 0.00% 0 0 44576 28418
CMC 0.00% 0.00% 0 0 44576 28418
CrowdStrike 0.00% 0.00% 0 0 44576 28418
TheHacker 0.00% 0.00% 0 0 44576 28418
eScan 0.00% 0.00% 0 0 44576 28418
Babable 0.00% 0.00% 0 0 44576 28418
SUPERAntiSpyware 0.00% 0.00% 0 0 44576 28418
Invincea 0.00% 0.00% 0 0 44576 28418
F-Prot 0.00% 0.00% 0 0 44576 28418
Endgame 0.00% 0.00% 0 0 44576 28418
Webroot 0.00% 0.00% 0 0 44576 28418
AVware 0.00% 0.00% 0 0 44576 28418
TotalGoodware 44576
TotalMalware 28418
TotalSample 72994

Please send an email to lxu@trustlook.com if you have any comments. Thanks.