February 3, 2020

VirusTotal APK Malware Detection Data - Week 5: 20200127-20200202

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200127_20200202.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.85% 0.03% 30947 27 104564 45
K7GW 99.61% 0.17% 30872 183 104408 120
Trustlook 99.40% 0.83% 30807 863 103728 185
DrWeb 98.81% 0.11% 30623 112 104479 369
ZoneAlarm 98.74% 0.00% 30602 3 104588 390
McAfee 98.37% 0.01% 30487 7 104584 505
Kaspersky 97.84% 0.00% 30323 3 104588 669
Avira 97.72% 0.00% 30285 0 104591 707
AhnLab-V3 97.66% 0.03% 30267 34 104557 725
F-Secure 96.97% 0.01% 30052 6 104585 940
Ikarus 96.36% 0.20% 29864 208 104383 1128
Sophos 94.45% 0.03% 29272 32 104559 1720
Fortinet 88.78% 0.01% 27514 8 104583 3478
Qihoo-360 87.36% 0.02% 27075 22 104569 3917
Tencent 82.52% 0.07% 25574 77 104514 5418
CAT-QuickHeal 81.34% 0.64% 25208 668 103923 5784
Avast-Mobile 75.99% 0.16% 23551 166 104425 7441
McAfee-GW-Edition 68.38% 0.00% 21193 4 104587 9799
NANO-Antivirus 68.29% 0.03% 21165 27 104564 9827
AVG 60.13% 0.05% 18634 56 104535 12358
Avast 57.51% 0.05% 17824 48 104543 13168
Cyren 56.74% 0.00% 17586 2 104589 13406
MAX 55.25% 0.00% 17122 1 104590 13870
Symantec 52.17% 0.02% 16167 20 104571 14825
Antiy-AVL 4.20% 0.00% 1302 4 104587 29690
TrendMicro-HouseCall 4.03% 0.01% 1249 8 104583 29743
Rising 3.72% 0.02% 1153 16 104575 29839
BitDefender 2.70% 0.00% 838 0 104591 30154
Ad-Aware 0.05% 0.00% 14 0 104591 30978
Baidu 0.01% 0.01% 3 11 104580 30989
Babable 0.00% 0.00% 0 0 104591 30992
TotalGoodware 104591
TotalMalware 30992
TotalSample 135583

Please send an email to lxu@trustlook.com if you have any comments. Thanks.