February 18, 2020

VirusTotal APK Malware Detection Data - Week 7: 20200210-20200216

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200210_20200216.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.92% 0.03% 27175 24 85937 21
K7GW 99.68% 0.07% 27108 59 85902 88
Trustlook 99.61% 0.54% 27091 460 85501 105
ZoneAlarm 98.81% 0.01% 26872 6 85955 324
AhnLab-V3 98.78% 0.02% 26864 14 85947 332
Kaspersky 98.20% 0.00% 26706 3 85958 490
DrWeb 97.83% 0.09% 26606 75 85886 590
Avira 97.72% 0.00% 26577 0 85961 619
Ikarus 97.67% 0.12% 26563 107 85854 633
McAfee 97.28% 0.01% 26455 10 85951 741
F-Secure 96.72% 0.00% 26305 2 85959 891
Sophos 94.61% 0.03% 25730 23 85938 1466
Qihoo-360 92.34% 0.03% 25112 22 85939 2084
Tencent 86.68% 0.09% 23573 79 85882 3623
CAT-QuickHeal 71.65% 0.01% 19487 12 85949 7709
Symantec 65.91% 0.03% 17926 30 85931 9270
Avast-Mobile 55.46% 0.26% 15082 224 85737 12114
Cyren 54.56% 0.00% 14839 0 85961 12357
McAfee-GW-Edition 47.60% 0.00% 12946 1 85960 14250
NANO-Antivirus 43.20% 0.02% 11749 20 85941 15447
AVG 40.25% 0.08% 10946 73 85888 16250
Avast 39.51% 0.08% 10746 73 85888 16450
MAX 33.66% 0.00% 9153 1 85960 18043
Fortinet 27.44% 0.00% 7462 0 85961 19734
Rising 5.44% 0.03% 1479 25 85936 25717
Antiy-AVL 5.36% 0.00% 1457 4 85957 25739
TrendMicro-HouseCall 5.33% 0.02% 1450 21 85940 25746
BitDefender 2.56% 0.00% 695 0 85961 26501
Ad-Aware 0.15% 0.00% 42 0 85961 27154
Baidu 0.04% 0.00% 11 4 85957 27185
Babable 0.00% 0.00% 0 0 85961 27196
TotalGoodware 85961
TotalMalware 27196
TotalSample 113157

Please send an email to lxu@trustlook.com if you have any comments. Thanks.