August 3, 2020

VirusTotal APK Malware Detection Data - Week 31: 20200727-20200802

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200727_20200802.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.89% 0.06% 40284 34 52531 46
Fortinet 99.67% 0.02% 40196 11 52554 134
K7GW 99.65% 0.12% 40190 61 52504 140
Trustlook 99.38% 0.22% 40080 114 52451 250
AhnLab-V3 99.07% 0.04% 39955 21 52544 375
Avast-Mobile 99.06% 0.13% 39952 69 52496 378
DrWeb 98.70% 0.13% 39804 69 52496 526
Avira 98.66% 0.00% 39789 0 52565 541
McAfee 97.40% 0.00% 39282 2 52563 1048
F-Secure 96.89% 0.00% 39076 1 52564 1254
Ikarus 95.30% 0.12% 38433 65 52500 1897
ZoneAlarm 91.00% 0.01% 36699 3 52562 3631
Kaspersky 90.60% 0.00% 36538 2 52563 3792
Sophos 85.92% 0.02% 34651 12 52553 5679
NANO-Antivirus 84.88% 0.05% 34234 28 52537 6096
Symantec 84.70% 0.02% 34158 10 52555 6172
Qihoo-360 82.92% 0.02% 33440 9 52556 6890
AVG 82.81% 0.03% 33396 17 52548 6934
CAT-QuickHeal 55.76% 0.01% 22486 7 52558 17844
Ad-Aware 0.21% 0.00% 86 0 52565 40244
McAfee-GW-Edition 0.00% 0.00% 0 0 52565 40330
TotalGoodware 52565
TotalMalware 40330
TotalSample 92895

Please send an email to lxu@trustlook.com if you have any comments. Thanks.