January 11, 2021

VirusTotal APK Malware Detection Data - Week 2: 20210104-20210110

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210104_20210110.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
ESET-NOD32 99.47% 0.05% 48234 45 87945 255
Trustlook 99.23% 0.14% 48115 119 87871 374
Avira 98.77% 0.00% 47892 1 87989 597
Fortinet 98.34% 0.04% 47682 39 87951 807
Avast-Mobile 95.56% 0.22% 46335 195 87795 2154
ZoneAlarm 94.52% 0.01% 45834 12 87978 2655
CAT-QuickHeal 94.15% 0.02% 45651 16 87974 2838
K7GW 93.73% 0.35% 45450 308 87682 3039
Ikarus 93.41% 0.41% 45294 359 87631 3195
DrWeb 91.80% 0.81% 44514 715 87275 3975
AhnLab-V3 89.33% 0.04% 43313 33 87957 5176
Qihoo-360 84.31% 0.03% 40880 28 87962 7609
Kaspersky 81.92% 0.01% 39721 5 87985 8768
Sophos 80.34% 0.02% 38956 14 87976 9533
McAfee 78.77% 0.00% 38193 2 87988 10296
McAfee-GW-Edition 77.87% 0.02% 37758 17 87973 10731
NANO-Antivirus 76.75% 0.09% 37217 79 87911 11272
Microsoft 75.87% 0.03% 36789 29 87961 11700
Antiy-AVL 70.52% 0.13% 34196 113 87877 14293
AegisLab 69.83% 0.54% 33859 473 87517 14630
F-Secure 65.92% 0.02% 31963 15 87975 16526
Tencent 65.57% 0.12% 31792 104 87886 16697
Symantec 64.34% 0.01% 31198 11 87979 17291
Alibaba 61.82% 0.00% 29974 4 87986 18515
AVG 49.29% 0.02% 23902 20 87970 24587
Cyren 48.98% 0.01% 23749 7 87983 24740
MAX 47.29% 0.00% 22932 1 87989 25557
Avast 46.88% 0.02% 22733 20 87970 25756
Comodo 38.65% 0.09% 18739 79 87911 29750
GData 17.18% 0.00% 8329 0 87990 40160
ClamAV 15.76% 0.17% 7640 147 87843 40849
BitDefender 15.01% 0.00% 7276 0 87990 41213
Emsisoft 14.96% 0.00% 7252 0 87990 41237
Arcabit 13.96% 0.00% 6770 1 87989 41719
Jiangmin 11.79% 0.40% 5715 348 87642 42774
Rising 11.29% 0.01% 5475 6 87984 43014
SymantecMobileInsight 10.86% 3.67% 5266 3229 84761 43223
Zoner 7.88% 0.00% 3822 1 87989 44667
TrendMicro-HouseCall 6.99% 0.03% 3390 24 87966 45099
TrendMicro 6.58% 0.07% 3193 59 87931 45296
Zillya 5.27% 0.04% 2554 32 87958 45935
VBA32 4.98% 0.01% 2417 8 87982 46072
Kingsoft 4.72% 0.01% 2291 6 87984 46198
Yandex 4.08% 0.00% 1976 1 87989 46513
MicroWorld-eScan 4.06% 0.00% 1969 0 87990 46520
Ad-Aware 0.86% 0.00% 416 0 87990 48073
TotalDefense 0.75% 0.00% 364 0 87990 48125
Panda 0.28% 0.00% 138 0 87990 48351
Baidu 0.05% 0.01% 25 6 87984 48464
ViRobot 0.02% 0.00% 12 2 87988 48477
K7AntiVirus 0.02% 0.00% 11 0 87990 48478
ALYac 0.01% 0.00% 4 0 87990 48485
VIPRE 0.01% 0.00% 4 0 87990 48485
SentinelOne 0.01% 0.00% 4 1 87989 48485
CMC 0.00% 0.00% 1 0 87990 48488
SUPERAntiSpyware 0.00% 0.00% 1 0 87990 48488
Bkav 0.00% 0.00% 0 0 87990 48489
nProtect 0.00% 0.00% 0 0 87990 48489
Malwarebytes 0.00% 0.00% 0 0 87990 48489
CrowdStrike 0.00% 0.00% 0 0 87990 48489
TheHacker 0.00% 0.00% 0 0 87990 48489
eScan 0.00% 0.00% 0 0 87990 48489
Babable 0.00% 0.00% 0 0 87990 48489
Invincea 0.00% 0.00% 0 0 87990 48489
F-Prot 0.00% 0.00% 0 0 87990 48489
Endgame 0.00% 0.00% 0 0 87990 48489
Webroot 0.00% 0.00% 0 0 87990 48489
AVware 0.00% 0.00% 0 0 87990 48489
TotalGoodware 87990
TotalMalware 48489
TotalSample 136479

Please send an email to lxu@trustlook.com if you have any comments. Thanks.