April 27, 2020

VirusTotal APK Malware Detection Data - Week 17: 20200420-20200426

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200420_20200426.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
Trustlook 99.54% 0.33% 240330 362 107741 1119
ESET-NOD32 99.44% 0.06% 240088 67 108036 1361
AhnLab-V3 98.84% 0.05% 238654 57 108046 2795
ZoneAlarm 98.59% 0.08% 238055 86 108017 3394
Avast-Mobile 98.23% 0.32% 237166 347 107756 4283
Qihoo-360 97.53% 0.03% 235480 29 108074 5969
Kaspersky 96.36% 0.01% 232651 7 108096 8798
K7GW 96.31% 0.21% 232537 226 107877 8912
McAfee 95.69% 0.02% 231043 17 108086 10406
Ikarus 95.16% 0.38% 229766 416 107687 11683
Avira 94.80% 0.00% 228884 0 108103 12565
CAT-QuickHeal 94.63% 0.27% 228483 291 107812 12966
NANO-Antivirus 93.98% 0.13% 226925 140 107963 14524
DrWeb 93.81% 0.69% 226514 744 107359 14935
Sophos 93.64% 0.17% 226096 187 107916 15353
F-Secure 84.15% 0.02% 203177 19 108084 38272
Fortinet 82.68% 0.00% 199624 3 108100 41825
McAfee-GW-Edition 81.50% 0.00% 196788 3 108100 44661
AVG 79.45% 0.08% 191840 85 108018 49609
Symantec 72.23% 0.14% 174398 146 107957 67051
Ad-Aware 0.13% 0.00% 309 0 108103 241140
TotalGoodware 108103
TotalMalware 241449
TotalSample 349552

Please send an email to lxu@trustlook.com if you have any comments. Thanks.