yashsingh1/xdaforums huggingface dataset:
[G1 Apps and Games] delta_foxtrot2:
Don't feel too bad, took me a while to think of doing it too! ;)
[G1 Apps and Games] delta_foxtrot2:
Basically.

I stopped doing wget cause it was failing, but more or less, as for the server side it's pretty simple too, download a hosts file, add any hostnames into the DB missing, tell apps if there the timestamp of the last host added.
[G1 Apps and Games] delta_foxtrot2:
More hostnames added to the system.

Also does anyone know if it's possible to have an app that checks updates periodically that doesn't run in the background?
[G1 Apps and Games] jashsu:
Sounds like a job for the AlarmManager

http://developer.android.com/reference/android/app/AlarmManager.html
[G1 Apps and Games] delta_foxtrot2:
ta for that, everything else I searched for didn't seem to fit.
[G1 Apps and Games] delta_foxtrot2:
I don't even know where to begin with this comment:

Using hosts files to block ad hostnames does nothing like that.

The only other person this happened to, had screwed up their phone and lost root access, if you are reading this install the 'terminal' app from the market and type in 'su' and see if it crashes.
[G1 Apps and Games] Smokuevo:
i wrote a comment on the market that it wont download for me.. root and everything works.. tried downloading on wifi and 3g
[G1 Apps and Games] delta_foxtrot2:
I added extra error reporting can you upgrade to the latest version and see if it reports why it doesn't work?
[G1 Apps and Games] Smokuevo:
got version 0.0.19 and it finds updates for the host files but fails to download.
[G1 Apps and Games] delta_foxtrot2:
I'll post a version shortly that dumps a ton of logging information to a file and see if we can get to the bottom of this.
[G1 Apps and Games] delta_foxtrot2:
v0.1.0 is now up, not only does it show a lot more information if/when failures occur, but it also now has an update check once a day.
[G1 Apps and Games] Smokuevo:
allright i got the 0.1.0 version and i get

Error!

java.io.IOExceptions: Hostname <www.bigtincan.com> was not verified.
[G1 Apps and Games] bmdixon:
Just thought i'd add my current script to this thread. After discovering several domains that weren't blocked i got fed up with constantly editing the script file on my phone so i've not got my own hosts file additions on my dropbox account. I can add domains to this list and they get added to the hosts file. Credit goes to "XlAfbk" for the original script.

<MY id> and <myfile> would obviously need to be changed (just get the public url for the file via dropbox)

/system/xbin/curl -so /data/data/hoststmp. http://hosts-file.net/ad_servers.asp
/system/xbin/curl -s http://winhelp2002.mvps.org/hosts.txt >> /data/data/hoststmp
/system/xbin/curl -s http://dl.dropbox.com/u/<MY id>/<myfile>.txt >> /data/data/hoststmp
cat /data/data/hoststmp | cut -d'#' -f 1 | grep -v gravatar.com | grep -v wordpress.com | awk NF | awk '{sub(/\r$/,"")};1' | sed 's/ \|\t/ /g' | sort | uniq -u > /system/etc/hosts
rm /data/data/hoststmp
[G1 Apps and Games] SubMatrix:
OK, I spoke too soon. Today I tried Angry Birds again and was getting ads again. I went in to /data/data and deleted everything except the two .lua files in /files and the /lib directory. I rechecked my hosts file and I have the permissions set to rw-rw-rw, with Owner and Group both set to root.

The ads I get are the blue ones again as in the screenshots I posted a few posts back.

Just to make sure, once again the only thing I'm doing to block ads is modifying the hosts file in /system/etc. I'm not doing any of this curl stuff or any linking. Did I forget anything?
[G1 Apps and Games] XlAfbk:
curl is a requirement for my script (as said in the original post), without curl the hosts files aren't downloaded
[G1 Apps and Games] larrycl:
Thanks to XlAfbk for his script!

I've 'front-ended' it with gscript so I can just run it with a single click from launcher.

It's now just as easy to run as the original adblock app!
[G1 Apps and Games] wraithdu:
EDIT:

Still a slight WIP, working on removed end-of-line comments.

EDIT2:

Ok, got it. First it strips anything that is a comment, so from a # char to the end of the line. Then if anything is left, it converts the line ending and removes extra white space. egrep then removes any html that snuck in and any other lines you want to except (gravatar, wordpress, etc).

I've made some improvements to the script, added an additional source, and fixed a few errors. I got some good info from this site about awk: http://www.pement.org/awk/awk1line.txt

Fixed:

1) Correctly removes leading/trailing/duplicate white space, allowing for proper stripping and de-duplication.

2) Removes comment lines and embedded html lines

3) Fixed uniq command, which was totally removing any duplicate lines. Now it correctly leaves one copy of the duplicate line.

Add any additional excpetions (like gravatar.com) to the egrep line.

#!/system/bin/sh

/system/xbin/curl -so /data/hoststmp.tmp -G -d hostformat=hosts -d showintro=0 -d mimetype=plaintext http://pgl.yoyo.org/adservers/serverlist.php
/system/xbin/curl -s http://hosts-file.net/ad_servers.asp >> /data/hoststmp.tmp
/system/xbin/curl -s http://winhelp2002.mvps.org/hosts.txt >> /data/hoststmp.tmp
/system/xbin/curl -s http://dl.dropbox.com/s/<ID>/<file>.txt >> /data/hoststmp.tmp
/system/xbin/busybox mount -o rw,remount /system
cat /data/hoststmp.tmp | awk '(NF){sub(/\#.*/,"")};(NF){sub(/\r$/,"");$1=$1;print}' | egrep -v '[\<\>]|gravatar\.com|wordpress\.com' | sort | uniq > /system/etc/hosts
chown 0:0 /system/etc/hosts
chmod 644 /system/etc/hosts
/system/xbin/busybox mount -o ro,remount /system
ls -l /system/etc/hosts
rm /data/hoststmp.tmp
[G1 Apps and Games] XlAfbk:
that still leaves you with some html as the site doesn't sent the pure list but a html file with the actual list in <pre>

ooops

that's just combing the 2 awk commands into 1, isn't it?

cut -d'#' -f 1 should have taken care of all of them, at least i didn't find any remaining ones in my tests.
[G1 Apps and Games] wraithdu:
Ha, cross-posted.

- I updated the first URL to only grab the plaintext page, mimetype=plaintext

- There's an additional awk command, $1=$1, which removes leading/trailing/duplicate white space.

- I added an extra awk command to delete the comments, both leading comment lines, and any line trailing comments (breaks de-duplication if you don't).

Credit where credit is due - I couldn't have modified the script without somewhere to start :)
[G1 Apps and Games] XlAfbk:
you can even drop the egrep completly:

awk '(NF){sub(/\#.*/,"")};(NF){sub(/.*<.*/,"")};(NF){sub(/\r$/,"");$1=$1;print}'

with wordpress:

awk '(NF){sub(/\#.*/,"")};(NF){sub(/.*<.*/,"")};(NF){sub(/.*wordpress.*/,"")};(NF){sub(/\r$/,"");$1=$1;print}'

edit: even better, just kill anything that contains chars other than a-z, 0-9, ., space

edit2: ignore that, code removed
[G1 Apps and Games] wraithdu:
Nice. I'm curious about performance of an all-awk command versus leaving the one egrep line in there. Based on file size I'm sure it's negligible. Good to have options though.
[G1 Apps and Games] XlAfbk:
here's my updated version:

#!/system/xbin/bash
/system/xbin/curl -so /data/data/hoststmp -G -d hostformat=hosts -d showintro=0 -d mimetype=plaintext http://pgl.yoyo.org/adservers/serverlist.php
/system/xbin/curl -s http://hosts-file.net/ad_servers.asp >> /data/data/hoststmp
/system/xbin/curl -s http://winhelp2002.mvps.org/hosts.txt >> /data/data/hoststmp
/system/xbin/curl -s http://content.wuala.com/contents/XXX/webaccess/hosts.txt?key=XXX\&dl=1 >> /data/data/hoststmp
cat /data/data/hoststmp | awk '(NF){sub(/\#.*/,"")};(NF){sub(/\r$/,"");$1=$1;print}' | sort | uniq > /data/data/hosts
rm /data/data/hoststmp

added some of wraithdus changes. not removing stuff like wordpress cause I don't need it, using wuala instead of dropbox

edit: removed the command for removing "<" as there shouldn't be any html comments left in the file
[G1 Apps and Games] wraithdu:
Maybe need to escape the < char, ie \< ? Not sure if awk or the shell treats that specially it this context.
[G1 Apps and Games] XlAfbk:
nah, was an inconsistency with my local awk version. but I just completly removed that awk command as there's no html left in the file.
[G1 Apps and Games] SubMatrix:
XIA are you using Tasker to automate the launching of this script?

Otherwise what are the steps for actually executing the script once I've copied the script code into a file and placed it somewhere on my phone?
[G1 Apps and Games] wraithdu:
I use Script Manager, I like it better than GScript. Push the script file to /sdcard/scripts and set the folder as your home location in the app. Make sure you are using Unix line endings in the script file (LF, and not CRLF) or it will not execute properly.
[G1 Apps and Games] XlAfbk:
You can use Tasker or SL4A or ... or just use the command line ;-) ("sh script.sh")
[G1 Apps and Games] AAccount:
Haven't checked on how my angry birds black lists were doing. I noticed that too and figured out there were rovioXXX.appads.com where XXX is a 1 to 3 digit number. I attached my new one.

And now I need your help. Has anybody figured out how to remove the ads on the new maps 5.7.0 or the google mobile website (the pink highlighted ones)? My standard method of "wiretaping" my phone through wire shark and looking for DNS queries has failed this time.
[G1 Apps and Games] dhostetter:
Is there a way to get this to work on CM7.1?
[G1 Apps and Games] magillos:
Thanks. It removed ads from AB.
[G1 Apps and Games] AAccount:
Glad it worked for you. Has anybody figured this out yet: "And now I need your help. Has anybody figured out how to remove the ads on the new maps 5.7.0 or the google mobile website (the pink highlighted ones)? My standard method of "wiretaping" my phone through wire shark and looking for DNS queries has failed this time."

Please don't tell me that the IP address of the ad server is hard coded or something awful like that:mad::mad:
[G1 Apps and Games] XlAfbk:
not seeing any ads in google maps. where are they supposed to be?
[G1 Apps and Games] AAccount:
See the screenshot. It seems more likely to show up for vague searches. I just put in "electronics store". I am using the latest hosts file with my angry birds addition. You can see it on a previous post. If I just need to clear some cache that would be great but the "electronics store" search I did for this screenshot is the first time I've ever done it. I have google maps 5.7.0.
[G1 Apps and Games] XlAfbk:
not getting those. am not in the us though
[G1 Apps and Games] creeve4:
How do you set ownership on a file?
[G1 Apps and Games] XlAfbk:
chown, chmod or use Root Explorer (dunno about other file managers)
[G1 Apps and Games] creeve4:
I've tried Root Explorer, I can set permissions, but I can't find where/how to set ownership
[G1 Apps and Games] SubMatrix:
should be at the bottom if you long press on the file. Scroll all the way down.
[G1 Apps and Games] XlAfbk:
or use this script to create the symlink and set proper permissions and owner on the file. run as root of course

#!/system/xbin/sh
/system/xbin/busybox mount -o rw,remount /system
ln -s /data/data/hosts /system/etc/hosts
chown 0:0 /system/etc/hosts
chmod 644 /system/etc/hosts
/system/xbin/busybox mount -o ro,remount /system
[G1 Apps and Games] creeve4:
Here are all the options available when I long press on hosts.

Nothing for ownership:
[G1 Apps and Games] XlAfbk:
uhm, as you can see from the scrollbar that's not all the way down. there's 2 more menu items: create tar and change owner
[G1 Apps and Games] creeve4:
I'm using Root Explorer 2.14.2

I did miss the create TAR, but there is no change owner:
[G1 Apps and Games] XlAfbk:
you're using an outdated version of RE (which probably means you warezd it from the net...)
[G1 Apps and Games] creeve4:
Yep, usenet
[G1 Apps and Games] Crox:
I tried this too - but it won't work. If I use AB 1.5.3 there's no ad, with the new version ad appears. At the moment I went back to 1.5.3 by Titanium Backup. Why can't Adfree block ad in the new Version of Angry Birds and some other Apps?
[G1 Apps and Games] roccoadam32:
some newer android markets have a setting in the menu to get interest based adds i would check and see if this is checked in your market
[G1 Apps and Games] filou75:
Yes ! I can play without ads, thanks :)
[G1 Apps and Games] Sergey2011:
Doesn't work with Battery Monitor Windget v.1.7.1. When you open BMW ad is shown and popup saying 'Please disable your adblocker before using this application' is displayed.
[G1 Apps and Games] XlAfbk:
It works just fine with BMW. Just that BMW checks for the existance of ad blocking hosts file. It works just fine though.
[G1 Apps and Games] Helgaiden:
Broke my MMA News (by sempixel) RSS reader app. Not sure how to fix it. Uninstalling didn't help, reinstalling and clicking 'revert' didn't help. Not sure what to do...
[G1 Apps and Games] Sergey2011:
What version of blocker and BMW do you use?
[G1 Apps and Games] XlAfbk:
latest. BMW says "please disable ad blocker" but works just fine, showing the default ad
[G1 Apps and Games] BXM500:
stats.wordpress.com is filtered.

Cannot view stats with Wordpress blog app.
[G1 Apps and Games] Sergey2011:
If ad is shown when how is this adblocker can be called 'working'?
[G1 Apps and Games] XlAfbk:
the default 'ad' is included in the BMW app itself. it can't be blocked from loading as it doesn't have to be loaded. If you're that annoyed by it just buy the app.
[G1 Apps and Games] dominiksch:
new Ad blocker

Hi folks,

I have released my own hosts-based ad blocker named AdAway, it can be found on the market with the following features:

AdAway is Open Source

AdAway lets you select your own sources of hosts files

You can add exceptions to your Whitelist if some apps are not working with ads disabled

You can define extra hostnames in your own Blacklist

Market: market.android.com/details?id=org.adaway

Website: code.google.com/p/ad-away

Source: gitorious.org/adaway

I would like to hear suggestions and ideas for possible future releases. Please look into the Issue tracker before and add your suggestions there: code.google.com/p/ad-away/issues/list
[G1 Apps and Games] XlAfbk:
looks nice. output is pretty much identical to my script.

doesn't seem to support https:// though, didn't test ftp.

suggestion: add update-check that sends a head-request and shows the Last-Modified-header in the UI. that way you don't have to update all the time but can check whether the lists were updated and only then do a full update.
[G1 Apps and Games] dominiksch:
Thanks for the suggestions, i added them to my issue tracker.

They will be implemented in future versions
[G1 Apps and Games] bobbyphoenix:
Works like a charm. Thanks.
[G1 Apps and Games] cephraim:
Can this be run in parallel with AdFree? Or, is it one or the other?

Also, what is the difference between the two?
[G1 Apps and Games] larrycl:
Thanks!

Trying to use the QR code tool to add hosts to the blocklist, I keep getting malformed host error.

I realized I was trying to add a list of hosts, and you only accept one at a time.

Would be nice to be able to add a list.

[Edit: Realized I can add a list by creating my own list, hosting it on Dropbox, then adding the public URL to the list of sources.]
[G1 Apps and Games] dominiksch:
When you would run them both, everytime you apply AdAway the whole hosts-file got overwritten by it so all hosts added by AdFree are gone.

The differences are the feature sets.

The main difference in my opinion (I am the dev of AdAway ;) ) is that you can choose your own sources of blocklists (hosts sources) in AdAway. AdFree uses only one source but has the advantage that it only adds new hosts, meaning it is faster because it only updates.
[G1 Apps and Games] dkmurphys:
As the Dev you are planning to implement the feature of adding new hosts, correct?

I am currently using your app and I very much appreciate it and like it. I haven't had a problem with AdFree missing any ads, and AdAway has yet to miss any as well!

Also you should start your own thread over here to offer a place for support, suggestions, and praise.
[G1 Apps and Games] XlAfbk:
It's already there.

Feature-request for AdAway: Put the file in /data/data/ and hardlink it instead of writing directly to /system/etc/
[G1 Apps and Games] dkmurphys:
I meant that while updating the App only downloads new hosts and not the entire hosts file.
[G1 Apps and Games] XlAfbk:
that'd require some server that compares the files and sends diffs. the amount of such files would grow huge after a few days, it could only be supported for some servers (as you can add custom urls) and considering how small those hosts files are it's just not worth it.
[G1 Apps and Games] dominiksch:
At the moment I am not allowed to post there because my post count is below 10 :D

As already said, implementing an update function like in AdFree requires to make diffs on server side. My idea was to rely totally on exisiting infrastructure, meaning I don't want to host hosts-files and diffs by myself. Because of that the whole files are downloaded everytime you update.

But there will be a new feature to show how old your current hosts-file is against the ones from the sources. This will be added in the future.
[G1 Apps and Games] dominiksch:
Thanks for that suggestion. It will come in a future release!
[G1 Apps and Games] Gamones:
i usually use adblockers
[G1 Apps and Games] XlAfbk:
If one download fails (server not available or stuff) you get a dialog informing you about the failed download. But when orientation changes that dialog disappears and there's no way to make it appear again and the app doesn't continue downloading, it's simply stuck. Had to kill it with a task killer to make it work again.
[G1 Apps and Games] magillos:
Great job dominiksch. Thanks.

It would be useful if that was possible to add custom hosts from file. Adding every single one manually in pop up window on mobile is bit frustrating. I still get adds in angry birds and some apps with Addaway that I managed to get rid with manually created hosts file.
[G1 Apps and Games] dominiksch:
Thanks for the bug report. I will look into it

Thanks for the suggestion, i added it to my enhancments for a future release.
[G1 Apps and Games] HamsterHam:
Your host file worked for me to remove adds from angry birds :D
[G1 Apps and Games] quickquest88:
It seems to be missing the ads in the latest very of free handcent sms.

Otherwise, its working very well. I've gotten used to ad free. They really POP out at me now. If we could only figure out how to skip those between play ads in words with friends, I'd be set.
[G1 Apps and Games] bbernardini:
Is there any way possible to use this to block video ads sites show before main content? I've found that I can't watch any Viddler-hosted videos on my phone (basically, anything from the Cheezburger family of blogs), and I suspect part of the problem is the ads they try to run before the main video.
[G1 Apps and Games] Taomyn:
Ads in AppBrain continue to show, though everything else I use gets theirs blocked.

---
[G1 Apps and Games] atrich0608:
I was directed to this thread by my host's file. I update my file with the AdFree app.

I was unable to use the My Verizon or NFL Mobile application due to the hosts file. I removed 2 Entries:

127.0.0.1 ehg-verizon.hitbox.com

127.0.0.1 oascentral.verizononline.com

And was able to log into and use both applications.

Please let me know if I am posting this in the wrong location and please advise me as to ware to share this information.
[G1 Apps and Games] motopig:
Just installed AdFree. Ad gone from speedtest.net and Angry Birds! :D Thanks!

Note that I have to remount /system as read-write using a terminal before the hosts file would go in. This is on a Motorola XT300. Block device to be remounted is /dev/mtdblock1.

EDIT: AdMob has returned on Angry Birds. Is there some way I can log the DNS requests going out of my phone?
[G1 Apps and Games] smog92:
error

Hello

sorry for my english :D

i am with JVR stock and Galaxian Kernel. The root is ok but i'have this error : "error , unable to locate a partition to remount to write the hoste file to.".

Before with JVR and kernel semaphore, i did'nt have that !!!!!:mad:

Thanks
[G1 Apps and Games] val5:
I'm playing japan life, and the ads keep showing, how to get rid it? It's said ad by inmobi
[G1 Apps and Games] crackedvenom2:
New adfree is getting a lot of hate. Any other good ad blocking apps out there?
[G1 Apps and Games] bobbyphoenix:
AdAway is much better. https://market.android.com/details?id=org.adaway&feature=search_result
[G1 Apps and Games] crackedvenom2:
Ah, cool. Thanks! I am going to get this one now.
[G1 Apps and Games] robdood:
Hmm well neither adfree or ad away are working for me any more. How strange. I'm running cyanogenmod 7 on a SGS2, no idea if that has any bearing, it was working fine for me a couple of weeks ago..
[G1 Apps and Games] crackedvenom2:
I havent used ad away since i got it, but will try now and see how it works.

Also, im on the thunderbolt. Just found this link via google. Will update soon
[G1 Apps and Games] tatsit:
neither adaway nor adfree works for me anymore

i'm on cm7 nexus one
[G1 Apps and Games] BeenAndroidized:
I am currently on Froyo ROM and I know AdFree DO NOT work on it. Haven't try AdAway yet, but from the sound of it, it does not look promising...
[G1 Apps and Games] Tx Redneck:
AdAway works fabulously, especially since the additions of Webserver on the device and more host sources.

Saved and Texan by the grace of God, Redneck by choice.
[G1 Apps and Games] Dragonn90:
Hello,

(Sorry for my bad english ^^)

How can i find the hostname of the advertising of this application : https://market.android.com/details?id=com.allocine.androidapp

Thank you ;)

Ps: i use AdAway
[G1 Apps and Games] JordanElliott:
u mean how to find what the hostnames its reaching that arent block?

download alogcat. and search http in the filter option

---------- Post added at 04:05 PM ---------- Previous post was at 04:00 PM ----------

heres the ones i found:

redirect108.tagcommander.com

logc188.xiti.com

regie-prod.sofialys.com

---------- Post added at 05:04 PM ---------- Previous post was at 04:05 PM ----------

127.0.0.1 c.w.inmobi.com

127.0.0.1 i.w.inmobi.com

127.0.0.1 r.w.inmobi.com

---------- Post added at 05:39 PM ---------- Previous post was at 05:04 PM ----------

more ads in angry birds:

rovio.appads.com

rovio1.appads.com

rovio2.appads.com

rovio3.appads.com
[G1 Apps and Games] BeenAndroidized:
Can I add that to the Hosts sources? Just wondering.

Also, Anyone got the list for WWF?
[G1 Apps and Games] JordanElliott:
i have a huge host list if anybody wants it. i added my own findings and merged together multiple files from other sources including the domains mentioned in this thread. and is wwf an app?

---------- Post added at 03:45 PM ---------- Previous post was at 03:43 PM ----------

if anybody has any findings of more ads let me know. ill ad them to my list. actually. even tho i dont have this phone ill make a thread for you guys.

---------- Post added at 03:54 PM ---------- Previous post was at 03:45 PM ----------

posted here:

http://xdaforums.com/showthread.php?t=1300529

just post your findings there or any apps that you have ads in and ill take a look.
[G1 Apps and Games] Tx Redneck:
No, but here's an ad free version from here on XDA that I was turned on to.

http://db.tt/EbDIlZqa

Saved and Texan by the grace of God, Redneck by choice.
[G1 Apps and Games] Dragonn90:
Thanks a lot, you're the best :)
[G1 Apps and Games] BeenAndroidized:
Thanks a bunch, now I don't have to wait in between game. :D And nice color tone (Black) too. ;) LOL!
[G1 Apps and Games] fatspirit:
On the one side I want fresh ad-free hosts, on the other side I need to constantly exclude some entries from it. Is there any non-manual way to do it?
[G1 Apps and Games] Ke11ett:
AdAway supports white lists. Then you'll only need to exclude them once.
[G1 Apps and Games] Racle90:
Adfree gets error msg after updated to newest version, here is logcat for that

http://paste.dy.fi/jXB/plain
[G1 Apps and Games] delta_foxtrot2:
0.8.4+ should have fixed that particular bug... PM me if you are still having problems with 0.8.5.