yashsingh1/xdaforums huggingface dataset:
[Android Apps and Games] DiamondJohn:
To be clear, the pgl.yoyo.org list is one of the default lists of AdAway, not "mine". It Peter Lowe's list. Spoiler content hidden..

I ran AdAway twice, and neither runs had this host. I actually found it was also missing another seemingly random host. zy16eoat1w.com from his same hosts source/list.

In a check, I ran AdAway once (it wasnt there), ran my other host builder (it was there in the resultant host and also in the cached copy of the actual downloaded file) ran AdAway again ()knowing it should be there now and it wouldn't be a new/transient host that just moved in) and it wassnt6 there. IKK also checked the AdAway database and it isnt there either. I then manually checked the source, and (As you have found) it is in the list.

I havealso checked my allow list, which is empty, and it does not include the above hosts.

I checked my explicit user block list (some DNS logging of IMDB for mobile) and neither opf the two are in that either (just in case)

api.graphql.imdb.com
d1zcggttmijv1z.cloudfront.net
images-na.ssl-images-amazon.com
m.media-amazon.com
unagi.amazon.com

Attached are the host files and the dB's ( the resulrtant host files are the exact same, apart from the header date comment).Not that it should make any difference, I am not using systemless hosts on this device.

I just ran it all again this morning, just now,and there are no missing hosts in this test run. What can I say, it happened, and the fact it appears to have resolved itself this morning, kinda does confirm my method isn't obviously flawed. What I will be doing in my following test is clearing out my custom block lists, just to have a clean starting point. ie wipe AdAway data and restart it from a clean install.

To be clear, these test were actually done to check what the OTHER host blocking is failing on.
[Android Apps and Games] DiamondJohn:
I just checked another host list. The examples I was checking for were for hosts with an underscore in the name; which I very vaguely remember reading are not valid (but this specific example by chance does not), but are present in many lists I have seen and do resolve and respond to ping events (so obviously I misread or misunderstood). So, I loaded one local host list file that I had saved off; but of course could equally of been a web source, and was originally a web source, but in the wrong format.

One of the hosts that do not get loaded by AdAway into the resultant hosts file is ftp-.hiveon.ne. A ping of the host (With my firewall turned off) not only resolves to an IP address, but it also responds. I did not check for the same on a mobile device.

>ping ftp-.hiveon.net

Pinging ftp-.hiveon.net [34.90.112.127] with 32 bytes of data:
Reply from 34.90.112.127: bytes=32 time=297ms TTL=100
Reply from 34.90.112.127: bytes=32 time=297ms TTL=100
Reply from 34.90.112.127: bytes=32 time=297ms TTL=100
Reply from 34.90.112.127: bytes=32 time=297ms TTL=100

Ping statistics for 34.90.112.127:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 297ms, Maximum = 297ms, Average = 297ms

There are multiple addresses in the file I have that do not make it into an AdAway hosts file; many do not resolve, and those that I have checked and do, are variations of the above and go to the same IP. The above is just the first I checked that did get resolved and a response. You can duplicate yourself by creating a new local source blocking file with the above host. I have attached the list for anyone to check my checks if they are interested.

FYI: The above host was only found in a single source I had when blocking about 3M from 4M non unique hosts (but the AdAway test was ONLY this file), and is from a "Coin blocker" list I had, so would obviously be more attuned for desktop PC than mobile devices. But thats just this one specific case, and I would estimate its the host file name format that excludes it from the list even though it does resolve and respond.
[Android Apps and Games] zgfg:
You said you don't use Systemless hosts. If I remember correctly from your previous discussions, you use a large hosts sources list, generating a huge blocked list.

All that has to be written directly to /system/etc/hosts

If there is no ebough space left on the System, creating such a huge file will fail, and maybe you could have such 'random' results (depending on various factors) that certain domains sometimes find a space in the hosts file and sometimes not

I could only suggest you to try using Systemless hosts. In that case AdAway should have enough space on Data to create the (huge) hosts file and Magisk will mount it to System (but upon reboot!!!)

Otherwise, I don't know (but I don't believe to) if AdAway have glitches in creating the hosts file.

My hosts source list has 11 sources, resulting with the blocked list of cca 165K domains.

My SAR is read-only (A11) and of course I use Systemless hosts.

I randomly checked for about 10 domains from various of the hosts sources I use, and each and every of them was properly blocked
[Android Apps and Games] koberulz:
I've apparently got an AdAway update available, but when I download it and then click "Install" there's a progress bar and then an "App not installed" message.

Currently running 5.7.0 on Android 11 on a Pixel 3, rooted.
[Android Apps and Games] DiamondJohn:
the last test case was only the single file I attached to the post.

The previous test case was only the 3 stock/default host lists that AdAway creates on initialisation.

Before I was doing these latest tests on small targetted files for very specific test cases, I was testing the max capacity of the other system on low-end systems (worst case) by having something like 45+ host sources. I cannot compare that with AdAway as that many hosts either cause AdAway to fail or just take forever, even on my newer Pixel 4A. That 3M unique hosts from 4M total hosts results in a 95MB file, and has no problem fitting on the system partition of the old phone (2013 S4). The old S4 phone has a 2.2GB system partition with 1GB free. ie plenty of space, so space is not the issue. And again, the last test was only the single file. But I will (eventually) switch to systemless hosts just for future test purposes, because in reality, thats how I will run it on my daily. Direct System hosts just enables me to quickly test the resultant blocking, as I do not have to reboot to experience/test the effects.

On the Pixel 4A running stock A11, I am obviously using Magisk systemless hosts.

With my checks, I am using the app dB of both AdAway and the other one and quickly cross check the list in the dB with some SQL, and then manually check the file that they match to whats in the dB. Although, with these smaller files, I run the two to independent locations and then can actually use winmerge on my desktop PC to see the diff as well. Not an option on the 3M host-file.

Again, to be clear, the driver for these tests is to see where the other system fails, as it is less tried and tested compared to AdAway. But I am seeing these missing hosts in the AdAway results.

PS: You could script an automated check of many many more hosts. In the app TeMeFI, there is also a host file check that gives an indication of the size and number of hosts currently mounted (or written) to system and whats awaiting in the systemless location for the next boot. It also has some AdAway checks showing things such as the number of hosts per source, unique hosts per source (ie not in any other source) and more.

edit: oooo update available...
[Android Apps and Games] stingbandel:
I just upgrade my phone to Android 11 and now I think adaway is not getting ads…
[Android Apps and Games] zgfg:
AdAway does not get (process or something) ads, it only readresses IP addresses to onvalid ones for the sites listed in your hosts sources

Anyway, if you use root (when asking/reporting always clearly describe do you use root or non-root) you have to have properly enabled Systemless hosts in Magisk app (settings)
[Android Apps and Games] Serpico27:
New Adaway user here.

Details: Xiaomi Pocophone F1 - Rooted

Android 10 - MIUI version 11.0.9

Adaway 5.8.0

Systemless hosts enabled in Magisk

Adaway has been blocking banner ads properly but it's not blocking ads on Google searches.

There's also some 'annoyances' I'd like to block that Adguard did successfully and I hope Adaway can do so too.

The annoyances are:

Cookie requests

"Install app" requests

I've attached screenshots of these issues.

Some help would be awesome.
[Android Apps and Games] Nergal di Cuthah:
It doesn't block the showing of the google search ads but should block the clicking on it. That's the best you can ask for in a hosts blocker, the text of the ad comes straight from the search engine and blocking their display would block the search itself. I switched to duckduckgo because you can turn ads of in the search settings.
[Android Apps and Games] DiamondJohn:
another option is to use a different search engine as your default search engine, which does not have ads before the start of the results. www.duckduckgo.com is the one I use, but there are many other options.
[Android Apps and Games] Gunnlaugg:
Hi,

i want to use Custom Rules, but cant find a manual for the syntax.

Can anyone please help?

I want to use something like *google* to block every URL with google in it.

This is just an example.
[Android Apps and Games] DiamondJohn:
I could be wrong, but wildcards cannot be used in blocklist, only in allow lists. This is a feature of the host file. The wildcards in the allow list simply excludes matching blocked hosts, when building the host file.
[Android Apps and Games] manojcitc:
How do I know adaway is working?

I am rooted and magisk sysytemless hosts enable but after starting adaway I am not seeing notification of it ,though is not blocked .

Also not a single ad is getting removed in some site like jalopnik.

What's the issue here ,?
[Android Apps and Games] DiamondJohn:
just another thought on the 127.0.0.1 OR 0.0.0.0 issue, by someone who would possibly know more than us. i.e. Stephen Black.

hosts/alternates/fakenews-gambling-porn-social at master · StevenBlack/hosts

🔒 Consolidating and extending hosts files from several well-curated sources. Optionally pick extensions for porn, social media, and other categories. - StevenBlack/hosts

github.com
[Android Apps and Games] DiamondJohn:
open the host file, get a blocked host name. Open a shell, and try doing a ping to that host. It should fail with an IP of 127..0.1(or 0.0.0.0), NB You also have to reboot after applying the systemless hosts

There are many posts in the thread about different browsers bypassing the host file or cache the ads.

If it is being blocked (by the above test) you may need to add additional host sources for the apps or sites you visit.
[Android Apps and Games] zgfg:
Open Block list, press + and add: edition.cnn.com

Maybe add then also cnn.com

And accept to apply the new configuration (pop-up will be asking you)

In browser, you should still be able to open:

https://edition.cnn.com/

Then reboot and try again to open in browser:

https://edition.cnn.com/

It should fail like on the screenshot (from Chrome).

Check also as @DiamondJohn suggested, if edition cnn.com is now in /system/etc/hosts
[Android Apps and Games] zgfg:
Btw, I have strange results with ping (which: /system/bin/ping)

Although edition.cnn.com is resolved in /etc/system/hosts to 0.0.0.0, it pings to 127.0.0.1

Btw, maybe that's why @ipdev (I think it was him) opposed to use 0.0.0.0 and said it's better to block to 127.0.0.1 (rationale: even if redirected to invalid address 0.0.0.0, it also eventually ends at localhost 127.0.0.1)

Screenshots below

Additionally I forced BusyBox ping:

/system/xbin/ping

Apparently, it ignores the hosts file: it resolved and pinged edition.cnn.com to 151.101.1.67 (for more about that address, scroll down to my next post)

PS: This post was edited/corrected upon the important observation from @guhvanoh (see his subsequent post)
[Android Apps and Games] guhvanoh:
From the 1st screenshot, the cnn site is resolved to 127.0.0.1 which is your localhost. Therefore the pinging doesn't fail. If the cnn site site had been resolved to an external ip address then I'd say you have a problem. But then what do I know eh?
[Android Apps and Games] zgfg:
Yes. You're absolutely right - I didn't pay attention.

I was only puzzled why it's pinging although in the hosts file it is resolved to 0.0.0.0 (and not to 127.0.0.1 how it pings)

But Busy Box ping (/system/xbin/ping) resolves to 151.101.1.67 (I just checked from RIPE and by opening that address in browser - that's not CNN) and pings

PS: I have edited/corrected my original post

However, later on I removed edition.cnn.com from the Blocked list, restarted and used nslookup - it turned up that DNS servers do resolve edition.cnn.com to 151.101.1.67, meaning that BusyBox ping indeed ignored redirection to 0.0.0.0 for edition cnn.com from the local hosts file and went out to resolve the address (?!)
[Android Apps and Games] scarlion:
Am trying to install on rooted device but on the second step "Syncing" it has an error "Failed to sync: Not enough space" and then something else I can't read. It's a new device with 128 GB storage so there is plenty of space, any ideas what the real issue is? Thanks
[Android Apps and Games] zgfg:
Root or non root?

Root: New device, certainly new Android.

Did you enable

Systemless hosts in Magisk?
[Android Apps and Games] scarlion:
Yeah root, android 11... there isn't a "systemless hosts" in Magisk... just "systemless TZdata"
[Android Apps and Games] DiamondJohn:
which version of Magisk?
[Android Apps and Games] zgfg:
In Magisk app Settings!

And you must reboot then. Then go back to AdAway and let it sync.

Then reboot again
[Android Apps and Games] scarlion:
Thanks, I was looking in the wrong place
[Android Apps and Games] Wolfcity:
@PerfectSlayer : As it is asked over and over again here in the thread I would recommend putting a sticky to the OP about the need of activating systemless hosts in Magisk and also to mention it in the wiki.

(Yeah, I know not many people seem to read OP's these days....)
[Android Apps and Games] zgfg:
Better a toast in the app (IMO, the app should know and can check if /data/adb/modules/hosts/system/etc/hosts file exists) to warn the user to enable Systemless hosts - maybe only when it fails to write to the /system/etc/hosts (i.e., in case when it is really needed)
[Android Apps and Games] gallegonovato:
Good afternoon @DiamondJohn:

I've been looking at the host you put in https://xdaforums.com/t/app-root-nonroot-official-adaway-v5-8-0.2190753/post-85559583, do you have the raw url of the host ?. And I don't know if it's just me but the files from https://xdaforums.com/t/app-root-nonroot-official-adaway-v5-8-0.2190753/post-85558781 won't let me open them. I don't know if you can comment on what host lists you have in Adaway?

@zgfg opened a topic at https://github.com/AdAway/AdAway/issues/2907 because I don't know what happens, but it doesn't allow me to make suggestions for the translation.

Also I do not know if you know any host of Huawei, qualcomm, Lg. Or some interesting and good host.

A hug and a thousand thanks for your help
[Android Apps and Games] DiamondJohn:
I am no longer using AdAway. one of the reasons is that it isnt so good at looooong blocking lists.

I have a pixel 4a.. However, just checking for Huawei in the host name, for one of my recent host file rebuilds, gives me the following hosts and the sources I got them from

NOTE: NONE OF THE FOLLOWING IS A RECOMMENDATION OR EVEN A SUGGESTION.

I dont know if any of the sources are any "good" I do have a lot of duplication. For current method, its not such a problem. But, I have not seen an ad in anything for a loooong time. Using the default adaway lists alone let ads get through, in apps I used often.

Spoiler content hidden.

And a more complete list of host sources are

Spoiler content hidden.
[Android Apps and Games] DiamondJohn:
FYI: and slightly OT for the AdAway dev thread.

I have been testing host building, and have kept on seeing the same host source downloading a fresh version every single time! So I looked into why and what was possibly failing.

Turns out it was working correctly. This one host source gets updated every 5 minutes!!!

https://urlhaus.abuse.ch/downloads/hostfile/

They also have a lot of stats. I think they mostly aim to block malware. In testing, with over 2.5M hosts being blocked, from over 47 URL sources, they still manage to get 11 hosts not in any of the other 46 sources.
[Android Apps and Games] gallegonovato:
Good afternoon @DiamondJohn

Thank you very much for your help and for the tremendous list of hosts you have given me. I now have a few to look at so I can look at the ones I have in Adaway. And change for the better.

By the way the deny coinblocker link that you put in your previous post, do you have it?

This https://urlhaus.abuse.ch/downloads/hostfile/ , which you have provided me. Does Adaway get it right?

I don't know how to thank you for your help. Thanks a lot mate.

By the way, I think I have some bad news. I was reading that the creator of Magisk now works for Google. So I think he is going to leave Magisk. I read it https://topjohnwu.medium.com/state-of-magisk-2021-fe29fdaee458

I hope some developer is still with him. Because if not he can give us a hard time for Adaway.

Best regards
[Android Apps and Games] DiamondJohn:
Sorry to all, I know lists are basically OT, as this thread is about AdAway.And source lists are contentions at best. So this will be the last OT post on host sources. At least for now. ;)

But here are ALL my local file lists. To be clear, a lot would be more targeted for desktops than android devices, and are also quite old. The attached Zip of a report shows the unique hosts in each source, and as such, you can see that most of the hosts in my static files must be covered by another web source. But not all the files add nothing, and I do have a LOT of web sourced hosts list; ie drop a few, and the files will become more prevalent. The fact that current web sources do have the same hosts, gives me some confidence they are still active/valid, even though I saved them off a while ago. Also, not all my host sources are compatible with adaway. Some are simply only lists of host names.
[Android Apps and Games] DiamondJohn:
Thius is kinda embarrassing, and also a little off-topic.

The following is not on my daily, but an older test phone, running Oreo. AdAway IS building my host file, and it is appearing in /system/etc/hosts (using systemless hosts), and I have rebooted (more than once), but when I take a line from the host file and ping it, it resolves to the correct IP, not to 127.0.0.1 (see image below)

Any ideas what I could try?

Host blocking was working a few days ago, The phone has an old prepaid SIM, that also expired for the opportunity to recharge it (a few days ago as well). The credit actually expired many months ago, and I only use it on WiFI. I was using systemless hosts and have tried switching the module off (and changing the host file) but that made no difference (after a reboot). I also tried reducing the host file to that single server (And local hosts), and that didnt help either. A few years back, I had an issue which eas fixed by changing the DNS's of various inet interfaces, but that did not help now. Im at a loss.
[Android Apps and Games] GTMonster:
On occasion I need to pause the app as some banking apps rely on info peddlers cookies!

Often I try to pause AdAway, only to find it's 'starting' which disables the pause function. This seems to last ages, so I have to turn of Adaway in VPN settings in the phone.

I've tried stopping all Adaway updates, only to find nothing helps.

Firstly why is it starting several times a day for minutes on end when it's not been shut down, paused or updated?

Secondly why can't it be paused during these incessant periods while it's starting?

Galaxy S21 Ultra

Latest UK Stock firmware - Sim free

No root

Thanks. James
[Android Apps and Games] zgfg:
There are some problems with the non-root VPN mode when loosing and chaning the bearer (WLAN, mobile Internet). They were reported here and on Github.

I also remember having the similar issues like two years ago (but for me it was only showing Starting although it already restarted and worked and there was no problem to pause it) but I'm no more using that second, not rooted phone
[Android Apps and Games] joefuf:
Just updated to the most recent version for the first time in a number of years. I added a whitelist of sites that I want to be able to access. For instance, when I browse SlickDeals, the websites they pass you through seem to be blocked even though they are part of my whitelist. I see that some are part of the Blocked sites, but I assumed that Whitelist overrode Block.

Is there a way to permit these sites even though they are allowed on the Whitelist?
[Android Apps and Games] gallegonovato:
Good evening @PerfectSlayer @zgfg @DiamondJohn

First of all I apologize for the inconvenience caused and thank you for your time and help.

I happened to find an interesting host list. That is https://github.com/Zelo72/hosts . More specifically https://raw.githubusercontent.com/Zelo72/hosts/main/big.txt . Which contains several sources reaching more than a million entries.

But my question is, does Adaway recognize it and loads well. That is to say if you have other sources, this source of Zelo72 puts you updated for x time. And everything ok?

I say this because I don't know why I think that Adaway can't handle such big lists, that's why I'm asking if you are ok with it.

Thanks a lot for your help and your time.
[Android Apps and Games] zgfg:
You can't put a Web page with the list of hosts sources to AdAway but the hosts sources one by one

Big.txt will likely work, you should try - but frankly I don't like and don't need that big hosts files (and won't try the 1M of blocked domains).

Currently I run less than 18K of blocked domains (from 12 but lean and mean hosts sources) and I never ever observe any problem with ads on the Web pages I read and in the apps I use
[Android Apps and Games] DiamondJohn:
I am a user of huge lists. And AdAway does have issues when the list gets too long.

Some things you need to watch out for / understand with using big lists, is that many of the hosts are actually desktop specific, and would (*possibly*) never be used on an android device. Another thing to also note when using a big list, you will most likely need to manually maintain your own whitelist.

I had a look at your sources, and in particular, this sources whitelist. There were a few very suspicious (ie advertising hosts) in their white list. for example:

...snip...
# Referrer (Clicks)
a.nonstoppartner.net
ad.doubleclick.net
adservice.google.com
adservice.google.de
clickserve.dartsearch.net
googleadservices.com
m.exactag.com
mydealz.digidip.net
pagead46.l.doubleclick.net
pixel.everesttech.net
promo.mobile.de
t.myvisualiq.net
tp-emea.exactag.com
trk.klclick.com
www.googleadservices.com
...snip...