February 24, 2020

VirusTotal APK Malware Detection Data - Week 8: 20200217-20200223

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200217_20200223.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.80% 0.03% 28761 22 87961 59
K7GW 99.69% 0.10% 28732 87 87896 88
Trustlook 99.46% 0.63% 28664 556 87427 156
ZoneAlarm 98.85% 0.00% 28488 4 87979 332
McAfee 98.72% 0.01% 28450 7 87976 370
Kaspersky 98.23% 0.00% 28309 3 87980 511
Avira 97.98% 0.00% 28239 0 87983 581
DrWeb 97.05% 0.10% 27971 84 87899 849
Ikarus 96.99% 0.16% 27953 137 87846 867
F-Secure 96.38% 0.00% 27778 2 87981 1042
AhnLab-V3 95.59% 0.02% 27548 15 87968 1272
Sophos 93.37% 0.03% 26908 30 87953 1912
Qihoo-360 92.46% 0.02% 26646 19 87964 2174
Tencent 87.69% 0.12% 25272 103 87880 3548
Avast-Mobile 85.83% 0.20% 24736 173 87810 4084
CAT-QuickHeal 79.59% 0.02% 22937 17 87966 5883
AVG 73.45% 0.08% 21167 68 87915 7653
Avast 72.24% 0.07% 20819 65 87918 8001
NANO-Antivirus 70.55% 0.03% 20333 23 87960 8487
Symantec 61.75% 0.04% 17797 31 87952 11023
Cyren 57.77% 0.00% 16648 3 87980 12172
McAfee-GW-Edition 52.12% 0.00% 15021 0 87983 13799
MAX 50.93% 0.00% 14678 3 87980 14142
Fortinet 50.69% 0.00% 14609 0 87983 14211
Rising 10.22% 0.01% 2946 13 87970 25874
TrendMicro-HouseCall 7.55% 0.01% 2177 11 87972 26643
Antiy-AVL 4.74% 0.00% 1365 2 87981 27455
BitDefender 3.42% 0.00% 985 0 87983 27835
Baidu 0.05% 0.00% 13 1 87982 28807
Ad-Aware 0.04% 0.00% 11 0 87983 28809
Babable 0.00% 0.00% 0 0 87983 28820
TotalGoodware 87983
TotalMalware 28820
TotalSample 116803

Please send an email to lxu@trustlook.com if you have any comments. Thanks.