August 31, 2020

VirusTotal APK Malware Detection Data - Week 35: 20200824-20200830

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200824_20200830.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.70% 0.02% 13185 11 46006 40
K7GW 99.19% 0.11% 13118 52 45965 107
Trustlook 98.53% 0.13% 13030 61 45956 195
CAT-QuickHeal 98.52% 0.01% 13029 5 46012 196
Fortinet 98.04% 0.05% 12966 24 45993 259
DrWeb 97.92% 0.22% 12950 102 45915 275
Avira 97.33% 0.00% 12872 0 46017 353
Avast-Mobile 97.29% 0.21% 12866 97 45920 359
McAfee 97.04% 0.00% 12833 1 46016 392
AhnLab-V3 96.56% 0.04% 12770 19 45998 455
ZoneAlarm 94.95% 0.02% 12557 7 46010 668
Kaspersky 94.33% 0.00% 12475 0 46017 750
Ikarus 93.49% 0.23% 12364 104 45913 861
F-Secure 90.42% 0.00% 11958 1 46016 1267
NANO-Antivirus 78.50% 0.06% 10381 28 45989 2844
Sophos 69.63% 0.04% 9208 17 46000 4017
Symantec 67.28% 0.01% 8898 6 46011 4327
Qihoo-360 64.85% 0.02% 8576 10 46007 4649
AVG 51.61% 0.10% 6825 46 45971 6400
Ad-Aware 0.60% 0.00% 79 0 46017 13146
McAfee-GW-Edition 0.00% 0.00% 0 0 46017 13225
TotalGoodware 46017
TotalMalware 13225
TotalSample 59242

Please send an email to lxu@trustlook.com if you have any comments. Thanks.