Once you have one or two scrapers in your back pocket you will find it very quick and easy to convert any current scrapers into one that is useful for your next big project.

Do you scrape the names of the blogs or the titles of the new blog posts? If you are scraping weblogs then it seems you are just taking the blog name right?

Would you link direct to a WH money site? Should an inbetween site be use to protect it from a bad neighbourhood?

Do you filter titles for adult etc (unless you want that vertical)

One thing that makes me giggle every time I do some scraping is to set my agent to the agent string for GoogleBot *grin*. That way as I’m ripping their content the webmaster likely feels all warm and fuzzy because it looks like GoogleBot is making sweet sweet love to his site

(I use PHP and Snoopy for scraping… that makes setting things like agent and referrer really easy.)

This will forward the user to the url you specify. Browsers don’t pass the referrer header if you use meta refresh so the original website owner will never find out where the user is coming from.

I will also add to redirect.php and disallow redirect.php in my robots.txt.

Can you guys show me some examples or something?

Cant wait to hear from Aur as well. Good stuff.

I am currently writing a lexical tool that takes one web page to recreate different sentences and words, with the same meaning, and of course still readable! My goal is to be able to pass the plagiarism test on a tool such as iThenticate.

But it is some work, it is just starting, but the first results I did to test are not too bad!
By the way I said in another post I was working in my company on a powerful automation tool, that could help Blue/Black Hatters, unfortunately, this project is on hold for some time (christmas time brought other priorities to the company!) so I think I’m gonna concentrate on scraping and my lexical tool.
I’ll keep you more informed about it when I have made it work well enough (and hopefully earned some money with it ), I can even release it to the public if I feel it is good enough.

One problem about scraping is that you get links from the original source. What is wrong then is that when somebody clicks on the link, usually the link goes to the original scraped website. Then the original website owner may discover in his stats logs that a lot of visitors are coming from your website. He goes there and he sees that you just scraped his content, so he may complain to your hostsng company, google, whoever could bring your site to shut down. (It seems that it is one of the main reason why scraped site get often banned from google after some weeks).

So what is the solution ?
Usually, remove all the links from your scraped content, or change them to point to your own website
Easy but not very good for the user. He clicks and thinks he’s gonna go on another article and he may go to nowhere…

So I had this idea : Set up a website you could call “www.my-search-engine.com”
with only one main php file that you would invoke this way:
http://www.my-search-engine.com/index.php?url=www.scraped-website.com/whatever-article.html
This main file just takes one parameter: a url, and should redirect you to this url.

Then you only need to modify all links in your scraped content, adding your “search engine invokation” to the links…

e.g.
blablablaclick here blabla

would be transformed into :

Then when an user clicks on the link he goes to your “search engine” website and he is redirected to the page he wanted to read.
The scraped website owner will see on his logs only some people coming from your fake search engine, so he will even be happy thinking “cool, I’m indexed in a new search engine”.

Now you’re thinking… eh, how do you do such a redirect page in php?
Hopefully most of you already know, but here is the code anyway, just copy the following into a file called “index.php”: