August 17, 2020

VirusTotal APK Malware Detection Data - Week 33: 20200810-20200816

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200810_20200816.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.76% 0.01% 18184 4 43069 43
K7GW 99.62% 0.11% 18158 46 43027 69
Trustlook 99.32% 0.11% 18103 49 43024 124
Fortinet 99.31% 0.04% 18101 18 43055 126
Avast-Mobile 98.27% 0.13% 17911 56 43017 316
DrWeb 98.18% 0.22% 17896 94 42979 331
AhnLab-V3 97.84% 0.04% 17833 18 43055 394
McAfee 97.74% 0.00% 17815 2 43071 412
Avira 96.94% 0.00% 17670 1 43072 557
Ikarus 95.34% 0.11% 17378 49 43024 849
F-Secure 94.64% 0.00% 17250 1 43072 977
CAT-QuickHeal 94.57% 0.01% 17237 4 43069 990
Kaspersky 89.63% 0.00% 16337 1 43072 1890
Symantec 84.68% 0.00% 15435 2 43071 2792
Sophos 83.20% 0.03% 15164 12 43061 3063
NANO-Antivirus 81.61% 0.04% 14875 19 43054 3352
Qihoo-360 78.55% 0.02% 14318 7 43066 3909
AVG 73.22% 0.03% 13345 12 43061 4882
ZoneAlarm 70.89% 0.01% 12922 5 43068 5305
Ad-Aware 0.81% 0.00% 147 0 43073 18080
McAfee-GW-Edition 0.00% 0.00% 0 0 43073 18227
TotalGoodware 43073
TotalMalware 18227
TotalSample 61300

Please send an email to lxu@trustlook.com if you have any comments. Thanks.