Let me start by saying that I am a sadochistic scraper (yeah I just made up that word) but I will get your database if I want it. This goes the same for other scrapers who I am sure are more persistent than even I am.

You don't have to read any futher, but you should realise that...

* People will get your data if they want it *

The only way you can try and prevent it, is to have a [1] whitelist of scrapers and blacklist useragents who are hitting you faster than you deem possible. You should also paywall if the information is that valuable to you. Or work on your business model so that you can work on providing it free.... so that reuse doesn't effect you.

---------------------------------

I thought I would provide an account of the three reasons why I scrape data.

There are lots of different types of data that I scrape for and it falls into a few different categories. I'll keep it all vague so I can explain in as much detail as possible.

[1] User information (to generate leads for my own services)...

This can be useful for a few reasons. But often it's to find people who might find my service useful.... So many sites reveal their users information. Don't do this unless you have good reason to do so.
If I'm just looking for contact information of users, I'll run something like httrack and then parse the mirrored site for patterns. (I'm that paranoid that check out how I write my email address in my user profile on this site).

[2] Economically valuable data that I can resuppose....

A lot of the data that I scrape I won't use directly on sites. I'm not going to cross legal boundaries.. and I certainly don't want to be slapped with a copyright notice (I might scrape content, but I'm not going to so willfully break the law). But, for example, there is a certain very popular website that collects business information and displays it on their network of websites. They also display this information in Google Maps as Markers.
One of my most successful scrapes of all time, was to pretend to be a user and constantly request different locations to their "private API". It took over a month to stay under the radar, but I got the data. I got banned regularly, but would just spawn up a new server with a new IP.
I'm not going to use this data anywhere on my sites. It's their database that they have built up. But, I can use this data to make my service better to my users.

[3] Content...

Back in day... I used to just scrap content. I don't do this any more since I'm actually working on what will hopefully be a very succesul startup... however, I used to scrape articles/content written by people. I created my own content management system that would publish entire websites for specific terms. This used to work fantastically when the search engines weren't that smart. I would guess it would fail awfully now. But I would quite easily be able to generate a few hundred uniques per website. (This would be considerable when multiplied out to lots of websites!!!).

Anyway, content would be useful to me, because I would spin in into new content, using a very basic markov chain. I'd have thousands of websites up and running all on different .info domains, (bought for 88cents each) and running advertisements on them. The domains would eventually get banned from Google and you'd throw the domain away. You'd make more than 88 cents through affiliate systems and commission junction and the likes that this didn't matter, and you were doing it on such a large scale that it would be quite prosperous.

------------------------------------

I honestly couldn't really offer you any advice on how to prevent scraping.
The best you can do is slow us down.

And the best way to do that is the figure out who is hitting your pages in such a methodical manner and rate limiting them. If you are smart enough, you might also try to "hellban" us, by serving up totally false data. I really would have laughed, if the time I scraped 5million longitude and latitudes over a period of a few months, if at the end of the process, I noticed that all of the lats were wrong.

Resistance is futile. You will be assimilated. </geek>