April 20, 2020

VirusTotal APK Malware Detection Data - Week 16: 20200413-20200419

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200413_20200419.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
K7GW 99.71% 0.26% 42722 293 110979 125
Trustlook 99.51% 0.30% 42635 335 110937 212
ESET-NOD32 99.47% 0.07% 42618 74 111198 229
AhnLab-V3 99.12% 0.03% 42469 31 111241 378
ZoneAlarm 98.82% 0.02% 42341 27 111245 506
Avast-Mobile 98.55% 0.29% 42226 321 110951 621
Kaspersky 98.00% 0.02% 41991 17 111255 856
DrWeb 97.23% 0.26% 41661 288 110984 1186
Avira 96.80% 0.00% 41476 2 111270 1371
Sophos 96.77% 0.03% 41462 33 111239 1385
Ikarus 95.73% 0.50% 41018 551 110721 1829
F-Secure 93.76% 0.01% 40172 11 111261 2675
NANO-Antivirus 92.92% 0.04% 39812 50 111222 3035
McAfee 91.12% 0.02% 39041 17 111255 3806
Qihoo-360 90.39% 0.02% 38728 27 111245 4119
CAT-QuickHeal 90.03% 0.13% 38574 146 111126 4273
AVG 89.32% 0.08% 38272 92 111180 4575
Fortinet 77.43% 0.01% 33176 10 111262 9671
McAfee-GW-Edition 51.84% 0.00% 22210 0 111272 20637
Symantec 32.15% 0.01% 13777 15 111257 29070
Ad-Aware 0.31% 0.00% 132 0 111272 42715
TotalGoodware 111272
TotalMalware 42847
TotalSample 154119

Please send an email to lxu@trustlook.com if you have any comments. Thanks.