April 27, 2021

VirusTotal APK Malware Detection Data - Week 17: 202100419-20210425

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its detection results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20210419_20210425.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative

Vendor TPR FPR TP FP TN FN
K7GW 99.69% 0.09% 14583 93 108609 45
ESET-NOD32 99.26% 0.01% 14520 15 108687 108
Trustlook 98.58% 0.02% 14421 19 108683 207
Avast-Mobile 98.45% 0.78% 14401 853 107849 227
AhnLab-V3 97.56% 0.03% 14271 31 108671 357
DrWeb 97.26% 0.14% 14227 149 108553 401
Avira 97.08% 0.01% 14201 6 108696 427
Fortinet 96.70% 0.01% 14146 8 108694 482
CAT-QuickHeal 93.71% 0.01% 13708 14 108688 920
McAfee 91.24% 0.02% 13347 21 108681 1281
Ikarus 90.84% 0.13% 13288 136 108566 1340
Kaspersky 90.56% 0.01% 13247 9 108693 1381
McAfee-GW-Edition 87.79% 0.02% 12842 20 108682 1786
Microsoft 86.81% 0.01% 12699 11 108691 1929
AegisLab 85.89% 0.08% 12564 91 108611 2064
Sophos 83.98% 0.01% 12285 12 108690 2343
ZoneAlarm 66.19% 0.01% 9682 8 108694 4946
Symantec 63.10% 0.18% 9230 199 108503 5398
SymantecMobileInsight 62.72% 2.69% 9175 2926 105776 5453
Alibaba 58.81% 0.01% 8602 10 108692 6026
Tencent 57.40% 0.14% 8396 153 108549 6232
NANO-Antivirus 50.64% 0.02% 7407 23 108679 7221
Cyren 50.43% 0.00% 7377 3 108699 7251
MAX 49.03% 0.00% 7172 0 108702 7456
AVG 48.39% 0.01% 7079 13 108689 7549
Avast 48.07% 0.01% 7032 13 108689 7596
ClamAV 25.27% 0.12% 3696 127 108575 10932
Zillya 25.14% 0.13% 3678 141 108561 10950
Comodo 12.07% 0.06% 1765 68 108634 12863
Kingsoft 12.01% 0.02% 1757 21 108681 12871
GData 11.25% 0.00% 1645 0 108702 12983
Yandex 10.58% 0.00% 1548 4 108698 13080
BitDefender 9.90% 0.00% 1448 0 108702 13180
Emsisoft 9.82% 0.00% 1436 0 108702 13192
Qihoo-360 9.70% 0.00% 1419 2 108700 13209
F-Secure 6.55% 0.00% 958 1 108701 13670
Zoner 6.03% 0.01% 882 6 108696 13746
Jiangmin 6.02% 0.93% 880 1014 107688 13748
Arcabit 4.22% 0.00% 617 0 108702 14011
MicroWorld-eScan 4.06% 0.00% 594 0 108702 14034
Rising 3.23% 0.00% 472 5 108697 14156
Ad-Aware 2.28% 0.00% 334 0 108702 14294
Antiy-AVL 1.91% 0.02% 280 19 108683 14348
TrendMicro 1.70% 0.01% 248 16 108686 14380
TrendMicro-HouseCall 1.50% 0.01% 220 6 108696 14408
VBA32 0.58% 0.01% 85 15 108687 14543
Babable 0.37% 0.05% 54 53 108649 14574
Panda 0.22% 0.01% 32 7 108695 14596
Baidu 0.11% 0.01% 16 7 108695 14612
ALYac 0.07% 0.00% 10 0 108702 14618
ViRobot 0.06% 0.00% 9 0 108702 14619
SentinelOne 0.05% 0.00% 7 0 108702 14621
K7AntiVirus 0.03% 0.00% 5 0 108702 14623
F-Prot 0.02% 0.00% 3 0 108702 14625
Malwarebytes 0.01% 0.00% 2 1 108701 14626
Bkav 0.00% 0.00% 0 0 108702 14628
TotalDefense 0.00% 0.00% 0 0 108702 14628
nProtect 0.00% 0.00% 0 0 108702 14628
CMC 0.00% 0.00% 0 0 108702 14628
CrowdStrike 0.00% 0.00% 0 0 108702 14628
TheHacker 0.00% 0.00% 0 0 108702 14628
eScan 0.00% 0.00% 0 0 108702 14628
SUPERAntiSpyware 0.00% 0.00% 0 0 108702 14628
VIPRE 0.00% 0.00% 0 1 108701 14628
Invincea 0.00% 0.00% 0 0 108702 14628
Endgame 0.00% 0.00% 0 0 108702 14628
Webroot 0.00% 0.00% 0 0 108702 14628
AVware 0.00% 0.00% 0 0 108702 14628
TotalGoodware 108702
TotalMalware 14628
TotalSample 123330

Please send an email to lxu@trustlook.com if you have any comments. Thanks.