August 24, 2020

VirusTotal APK Malware Detection Data - Week 34: 20200817-20200823

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200817_20200823.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.81% 0.03% 14904 12 45743 29
K7GW 99.54% 0.10% 14865 44 45711 68
Fortinet 99.15% 0.02% 14806 7 45748 127
Trustlook 98.73% 0.10% 14743 47 45708 190
DrWeb 97.87% 0.20% 14615 90 45665 318
CAT-QuickHeal 97.77% 0.02% 14600 10 45745 333
AhnLab-V3 97.70% 0.05% 14590 22 45733 343
Avast-Mobile 97.56% 0.35% 14569 161 45594 364
Avira 97.40% 0.00% 14544 0 45755 389
McAfee 96.94% 0.00% 14476 2 45753 457
ZoneAlarm 95.53% 0.01% 14266 5 45750 667
Kaspersky 94.89% 0.00% 14170 1 45754 763
Ikarus 94.23% 0.10% 14072 45 45710 861
F-Secure 92.38% 0.00% 13795 2 45753 1138
NANO-Antivirus 87.26% 0.03% 13031 13 45742 1902
Sophos 73.66% 0.04% 11000 20 45735 3933
Symantec 70.78% 0.01% 10570 6 45749 4363
Qihoo-360 68.47% 0.02% 10225 11 45744 4708
AVG 58.26% 0.17% 8700 77 45678 6233
Ad-Aware 0.62% 0.00% 92 0 45755 14841
McAfee-GW-Edition 0.00% 0.00% 0 0 45755 14933
TotalGoodware 45755
TotalMalware 14933
TotalSample 60688

Please send an email to lxu@trustlook.com if you have any comments. Thanks.