January 21, 2020

VirusTotal APK Malware Detection Data - Week 3: 20200113-20200119

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200113_20200119.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.78% 0.03% 25653 25 74052 56
K7GW 99.54% 0.10% 25592 74 74003 117
Fortinet 99.32% 0.01% 25534 11 74066 175
DrWeb 98.40% 0.12% 25297 86 73991 412
Trustlook 97.75% 0.26% 25131 195 73882 578
Ikarus 97.50% 0.19% 25066 142 73935 643
McAfee 97.35% 0.01% 25027 7 74070 682
ZoneAlarm 97.14% 0.00% 24975 1 74076 734
Avira 96.60% 0.00% 24836 0 74077 873
AhnLab-V3 96.58% 0.03% 24829 22 74055 880
Kaspersky 96.39% 0.00% 24782 1 74076 927
F-Secure 96.17% 0.01% 24724 5 74072 985
Avast-Mobile 83.69% 0.19% 21516 142 73935 4193
Sophos 80.35% 0.04% 20657 26 74051 5052
Tencent 79.28% 0.12% 20382 89 73988 5327
Qihoo-360 76.44% 0.03% 19652 24 74053 6057
CAT-QuickHeal 74.64% 0.04% 19189 28 74049 6520
McAfee-GW-Edition 68.51% 0.00% 17612 0 74077 8097
Symantec 63.89% 0.06% 16426 45 74032 9283
NANO-Antivirus 61.54% 0.03% 15822 23 74054 9887
AVG 50.36% 0.07% 12947 53 74024 12762
Avast 48.82% 0.06% 12551 48 74029 13158
Cyren 47.19% 0.01% 12132 4 74073 13577
MAX 38.52% 0.00% 9902 0 74077 15807
Rising 5.76% 0.03% 1481 19 74058 24228
Antiy-AVL 4.83% 0.01% 1243 7 74070 24466
TrendMicro-HouseCall 4.30% 0.03% 1106 19 74058 24603
BitDefender 2.63% 0.00% 676 0 74077 25033
Baidu 0.07% 0.00% 17 1 74076 25692
Ad-Aware 0.07% 0.00% 17 0 74077 25692
Babable 0.00% 0.00% 0 0 74077 25709
TotalGoodware 74077
TotalMalware 25709
TotalSample 99786

Please send an email to lxu@trustlook.com if you have any comments. Thanks.