I'm doing a project for my school in which I would like to compare scam mails. I found this website: http://www.419scam.org/emails/
Now what I would like to do is to save every scam in apart documents then later on I can analyse them.
Here is my code so far:

import BeautifulSoup, urllib2

address='http://www.419scam.org/emails/'
html = urllib2.urlopen(address).read()
f = open('test.txt', 'wb')
f.write(html)
f.close()


This saves me the whole html file in a text format, now I would like to strip the file and save the content of the html links to the scams:

&lt;a href="2011-12/01/index.htm"&gt;01&lt;/a&gt; 
&lt;a href="2011-12/02/index.htm"&gt;02&lt;/a&gt; 
&lt;a href="2011-12/03/index.htm"&gt;03&lt;/a&gt;


etc.

If i get that, I would still need to go a step further and open save another href. Any idea how do I do it in one python code?

Thank you!


Question: URLs with final slash trigger hotlink protection on my server
Answer: Are you sure it's the hotlink protection?  If your website is laid out something like this:

/somepage.htm
/images/a.jpg


Then if somepage.htm has relative paths to the image like &lt;img src="images/a.jpg"&gt; and you go to http://example.com/somepage.htm/, then your web browser will think it's in the "somepage.htm" directory and try to load the image http://example.com/somepage.htm/images/a.jpg.  Apache will probably send you another copy of somepage.htm if you go there, which will cause your browser to show a broken image.

I'm not sure if Apache has a setting to turn this off.   It doesn't make much sense on plain HTML files but the reason Apache (and other servers) allow this is for sites that use "pretty URLs" (like serverfault's URLs!) where the URL extends past the actual file.  There isn't a file on serverfault called "urls-with-final-slash-trigger-hotlink-protection-on-my-server", that's just there for people (and Google) to read. You can load this question just fine with http://serverfault.com/questions/197935

While I don't know how serverfault was made (there are many ways to do this) it's almost certain that there's no file named "197935" either.  It's possible that there's a script named "questions" though, and the server could run that script, and the script would see the entire URL, look up the number in the database, and make this page.