March 16, 2020

VirusTotal APK Malware Detection Data - Week 11: 20200309-20200315

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200309_20200315.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.78% 0.04% 9692 24 64017 21
K7GW 99.59% 0.14% 9673 88 63953 40
Trustlook 99.45% 0.30% 9660 189 63852 53
ZoneAlarm 98.54% 0.01% 9571 5 64036 142
AhnLab-V3 98.40% 0.05% 9558 29 64012 155
Kaspersky 97.83% 0.00% 9502 3 64038 211
DrWeb 97.45% 0.14% 9465 92 63949 248
Ikarus 96.97% 0.13% 9419 86 63955 294
McAfee 96.49% 0.01% 9372 5 64036 341
Avira 96.30% 0.00% 9354 0 64041 359
F-Secure 94.65% 0.00% 9193 2 64039 520
CAT-QuickHeal 92.85% 0.04% 9019 23 64018 694
Avast-Mobile 92.50% 0.20% 8985 131 63910 728
Sophos 89.95% 0.04% 8737 23 64018 976
NANO-Antivirus 82.28% 0.05% 7992 31 64010 1721
Qihoo-360 79.95% 0.03% 7766 21 64020 1947
Symantec 70.58% 0.03% 6855 19 64022 2858
McAfee-GW-Edition 67.47% 0.00% 6553 2 64039 3160
Tencent 60.36% 0.12% 405 8 6433 266
AVG 48.05% 0.09% 4667 57 63984 5046
Fortinet 39.59% 0.00% 3845 1 64040 5868
Cyren 38.00% 0.02% 255 1 6440 416
Avast 30.40% 0.03% 204 2 6439 467
MAX 20.86% 0.00% 140 0 6441 531
Rising 8.05% 0.08% 54 5 6436 617
Antiy-AVL 7.00% 0.00% 47 0 6441 624
BitDefender 4.62% 0.00% 31 0 6441 640
TrendMicro-HouseCall 3.58% 0.03% 24 2 6439 647
Baidu 0.60% 0.00% 4 0 6441 667
Ad-Aware 0.07% 0.00% 7 0 64041 9706
Babable 0.00% 0.00% 0 0 6441 671
TotalGoodware 64041
TotalMalware 9713
TotalSample 73754

Please send an email to lxu@trustlook.com if you have any comments. Thanks.