January 27, 2020

VirusTotal APK Malware Detection Data - Week 4: 20200120-20200126

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200120_20200126.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.87% 0.04% 35840 32 81299 48
Fortinet 99.54% 0.01% 35722 5 81326 166
K7GW 99.29% 0.14% 35634 112 81219 254
ZoneAlarm 98.50% 0.00% 35350 1 81330 538
Kaspersky 98.15% 0.00% 35223 0 81331 665
DrWeb 97.89% 0.09% 35129 70 81261 759
Ikarus 97.44% 0.12% 34968 99 81232 920
F-Secure 97.13% 0.01% 34857 5 81326 1031
Avira 96.31% 0.00% 34563 0 81331 1325
AhnLab-V3 95.19% 0.02% 34160 16 81315 1728
McAfee 92.78% 0.01% 33296 7 81324 2592
Trustlook 92.40% 0.51% 33161 413 80918 2727
Sophos 92.13% 0.03% 33062 28 81303 2826
Qihoo-360 85.75% 0.02% 30774 17 81314 5114
Tencent 83.46% 0.08% 29951 69 81262 5937
Avast-Mobile 80.77% 0.17% 28986 136 81195 6902
CAT-QuickHeal 80.56% 0.03% 28911 23 81308 6977
NANO-Antivirus 73.15% 0.03% 26252 23 81308 9636
McAfee-GW-Edition 68.95% 0.01% 24745 6 81325 11143
AVG 64.10% 0.04% 23005 32 81299 12883
Avast 60.99% 0.03% 21889 27 81304 13999
Cyren 59.30% 0.00% 21281 2 81329 14607
Symantec 53.13% 0.02% 19069 17 81314 16819
MAX 42.74% 0.00% 15337 1 81330 20551
TrendMicro-HouseCall 6.89% 0.01% 2474 11 81320 33414
Rising 5.99% 0.01% 2150 8 81323 33738
Antiy-AVL 3.79% 0.00% 1359 3 81328 34529
BitDefender 2.87% 0.00% 1029 0 81331 34859
Ad-Aware 0.11% 0.00% 39 0 81331 35849
Baidu 0.01% 0.01% 5 5 81326 35883
Babable 0.00% 0.00% 0 0 81331 35888
TotalGoodware 81331
TotalMalware 35888
TotalSample 117219

Please send an email to lxu@trustlook.com if you have any comments. Thanks.