November 9, 2020

VirusTotal APK Malware Detection Data - Week 45: 20201102-20201108

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20201102_20201108.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.40% 0.00% 10422 5 109286 63
Trustlook 99.15% 0.49% 10396 532 108759 89
Avast-Mobile 98.05% 0.27% 10281 290 109001 204
Fortinet 97.39% 0.01% 10211 14 109277 274
Avira 97.30% 0.00% 10202 2 109289 283
DrWeb 96.71% 0.18% 10140 199 109092 345
AhnLab-V3 95.74% 0.02% 10038 25 109266 447
CAT-QuickHeal 95.70% 0.01% 10034 12 109279 451
ZoneAlarm 95.58% 0.01% 10022 12 109279 463
Kaspersky 94.79% 0.00% 9939 2 109289 546
Ikarus 94.16% 0.14% 9873 149 109142 612
K7GW 92.76% 0.10% 9726 104 109187 759
Qihoo-360 81.55% 0.01% 8551 13 109278 1934
F-Secure 81.13% 0.00% 8507 2 109289 1978
McAfee 76.63% 0.00% 8035 0 109291 2450
McAfee-GW-Edition 73.92% 0.01% 7751 8 109283 2734
Sophos 67.76% 0.02% 7105 17 109274 3380
NANO-Antivirus 67.45% 0.02% 7072 26 109265 3413
Symantec 62.13% 0.01% 6514 12 109279 3971
AVG 54.76% 0.02% 5742 23 109268 4743
Ad-Aware 0.81% 0.00% 85 0 109291 10400
TotalGoodware 109291
TotalMalware 10485
TotalSample 119776

Please send an email to lxu@trustlook.com if you have any comments. Thanks.