January 7, 2020

VirusTotal APK Malware Detection Data - Week 1: 20191230-20200105

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20191230_20200105.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.71% 0.02% 20650 13 57904 60
K7GW 99.60% 0.08% 20628 47 57870 82
ZoneAlarm 99.31% 0.00% 20568 1 57916 142
Fortinet 99.12% 0.00% 20528 1 57916 182
AhnLab-V3 99.11% 0.03% 20525 18 57899 185
Kaspersky 99.08% 0.00% 20520 0 57917 190
DrWeb 98.75% 0.11% 20452 64 57853 258
Ikarus 97.69% 0.13% 20231 76 57841 479
Avira 97.63% 0.00% 20220 1 57916 490
Trustlook 97.37% 0.22% 20166 127 57790 544
F-Secure 96.89% 0.01% 20065 3 57914 645
McAfee 95.95% 0.01% 19872 7 57910 838
Symantec 82.83% 0.03% 17155 20 57897 3555
Sophos 78.32% 0.01% 16221 3 57914 4489
Avast-Mobile 77.58% 0.16% 16067 92 57825 4643
Qihoo-360 73.37% 0.04% 15195 23 57894 5515
CAT-QuickHeal 65.89% 0.04% 13645 22 57895 7065
McAfee-GW-Edition 55.72% 0.00% 11540 0 57917 9170
NANO-Antivirus 52.84% 0.04% 10943 24 57893 9767
Cyren 48.42% 0.00% 10027 1 57916 10683
MAX 41.52% 0.00% 8599 0 57917 12111
AVG 37.47% 0.04% 7760 23 57894 12950
Avast 36.80% 0.04% 7622 22 57895 13088
Tencent 25.19% 0.00% 5216 0 57917 15494
Antiy-AVL 5.91% 0.01% 1224 4 57913 19486
Rising 5.35% 0.02% 1108 11 57906 19602
TrendMicro-HouseCall 3.73% 0.03% 773 15 57902 19937
BitDefender 2.73% 0.00% 565 0 57917 20145
Ad-Aware 0.09% 0.00% 19 0 57917 20691
Baidu 0.06% 0.00% 12 0 57917 20698
Babable 0.00% 0.00% 0 0 57917 20710
TotalGoodware 57917
TotalMalware 20710
TotalSample 78627

Please send an email to lxu@trustlook.com if you have any comments. Thanks.