March 31, 2020

VirusTotal APK Malware Detection Data - Week 13: 20200323-20200329

At Trustlook, we monitor live feed from VirusTotal (VT). On a daily basis, we collect APK samples from VT along with detection results from Anti-Virus (AV) vendors hosted on VT. Using a conservative labeling policy, we are able to select thousands of benign and malicious APK samples from millions of live feed samples. Then we look at detection results from AV vendors and rate them by how many malware they have detected and how many benign samples they have misclassified.

We generate a CSV file recording the detection results everyday. In the CSV file, from left to right, the columns are MD5 hash of the APK, label where 1 means positive (malicious) and 0 means negative (benign), and one column for each vendor showing its deteciton results where 1 means positive and 0 means negative.

On a weekly basis, we publish the detection results and zip the CSV files to AWS S3. For this week, you can download the detection data from:

https://virustotal-results.s3-us-west-1.amazonaws.com/VirusTotal_Results_20200323_20200329.zip

The weekly results are summarized in the table below and here is a simple explanation of the columns in the table:

  • Vendor: AV engine vendor
  • TPR: True Positive Rate, percentage of positive (malware) samples being correctly classified as positive
  • FPR: False Positive Rate, percentage of negative (goodware) samples being misclassified as positive
  • TP: True Positive, number of positive (malware) samples being correctly classified as positive
  • FP: False Positive, number of negative (goodware) samples being misclassified as positive
  • TN: True Negative, number of negative (goodware) samples being correctly classified as negative
  • FN: False Negative, number of positive (malware) samples being misclassified as negative
Vendor TPR FPR TP FP TN FN
Trustlook 99.58% 0.31% 10638 209 68066 45
ESET-NOD32 99.54% 0.02% 10634 13 68262 49
K7GW 99.02% 0.11% 10578 72 68203 105
ZoneAlarm 98.65% 0.00% 10539 1 68274 144
AhnLab-V3 98.58% 0.05% 10531 32 68243 152
Avast-Mobile 98.34% 0.27% 10506 184 68091 177
Kaspersky 98.14% 0.00% 10484 0 68275 199
Avira 96.67% 0.00% 10327 0 68275 356
DrWeb 96.41% 0.14% 10299 93 68182 384
F-Secure 96.38% 0.00% 10296 3 68272 387
Ikarus 96.25% 0.14% 10282 98 68177 401
CAT-QuickHeal 95.55% 0.03% 10208 20 68255 475
Sophos 93.84% 0.03% 10025 23 68252 658
Qihoo-360 91.42% 0.03% 9766 19 68256 917
McAfee 89.33% 0.01% 9543 6 68269 1140
NANO-Antivirus 88.13% 0.04% 9415 28 68247 1268
AVG 79.57% 0.11% 8500 77 68198 2183
Fortinet 77.63% 0.00% 8293 0 68275 2390
McAfee-GW-Edition 73.72% 0.00% 7876 0 68275 2807
Symantec 57.38% 0.02% 6130 16 68259 4553
Ad-Aware 0.24% 0.00% 26 0 68275 10657
TotalGoodware 68275
TotalMalware 10683
TotalSample 78958

Please send an email to lxu@trustlook.com if you have any comments. Thanks.