March 24, 2020

VirusTotal APK Malware Detection Data - Week 12: 20200316-20200322

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200316_20200322.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.79% 0.03% 13205 18 54325 28
Trustlook 99.37% 0.34% 13150 184 54159 83
AhnLab-V3 99.24% 0.05% 13133 27 54316 100
Avira 99.10% 0.00% 13114 0 54343 119
Avast-Mobile 98.79% 0.24% 13073 130 54213 160
ZoneAlarm 98.25% 0.02% 13002 9 54334 231
Kaspersky 97.91% 0.01% 12957 5 54338 276
F-Secure 97.71% 0.01% 12930 7 54336 303
DrWeb 97.45% 0.21% 12896 115 54228 337
Ikarus 96.60% 0.16% 12783 85 54258 450
K7GW 96.14% 0.17% 12722 90 54253 511
CAT-QuickHeal 95.84% 0.09% 12682 49 54294 551
Sophos 91.64% 0.05% 12127 28 54315 1106
Qihoo-360 88.54% 0.04% 11716 22 54321 1517
McAfee 82.81% 0.02% 10958 9 54334 2275
NANO-Antivirus 80.62% 0.07% 10669 37 54306 2564
AVG 66.28% 0.10% 8771 55 54288 4462
Fortinet 60.60% 0.01% 8019 4 54339 5214
McAfee-GW-Edition 56.68% 0.00% 7501 2 54341 5732
Symantec 55.41% 0.01% 7332 7 54336 5901
Ad-Aware 0.07% 0.00% 9 0 54343 13224
TotalGoodware 54343
TotalMalware 13233
TotalSample 67576

Please send an email to lxu@trustlook.com if you have any comments. Thanks.