- Blue Hat SEO-Advanced SEO Tactics - https://www.bluehatseo.com -

Hot New List of Places To Scrape

Posted By Eli On November 4, 2006 @ 4:20 am In General Articles | 58 Comments

You’re not still scraping RSS feeds are you? *Shakes his head in shame* I am so disappointed in you. How about we review a few places where only the trekies are brave enough to scrape. Its a huge content filled Internet world out there. I’m sure we can do a little better than crappy RSS feeds.

Encarta Encyclopedia- WTF?! Yeah you heard right. The articles and content are held in huge datafiles on the CDs. Go fuckin’ grab ‘em already! No one else is.

YouTube- This one is too easy. They even have a feed you can use to grab the videos, descriptions, and titles.

IMDB- Same as YouTube. There is even an example of how to grab and parse IMDB content on the LWP module example code.

Newsgroups- A classic and too easy to pass up.

Drudge Report- Nothing is more beautiful than snagging big news that has popularity but isn’t already stolen by CNN and MSN. Also consider who your competitor in the SERPS is. Drudge Report may have a ton of links but the site itself is SEO’d to shit. My little sister could kick his ass in the SERPS.

Craigslist- Same as Drudge Report but I’m going to stay out of this one because I have a ton of respect for Craig Newmark. It is also a bit harder to beat him in the SERPS, but the vast volume of new content being added every day more than makes up for it.

IRC- I’ve beatin this technique to death so I’m not even going to bother talking about it.

Froogle- I couldn’t help but mention this one. However please respect when I say, stay off my turf! Seriously…

Forums- One of the easiest way to build millions of pages of content quickly. The quality tends to suck but you hit such a high range of topics in such a short amount of data it really helps bring in traffic from those odd phrases.

Looksmart and Article Finder- Their templates make it way too easy to scrape the content. The articles are also long which makes it nice.

User Contributed(Check Comments)
Google News- Uhhhg yes.

Public Libraries- Simply fuckin brilliant!

Ebay- That one is news to me. I’m all over that one. Ever thought of scraping Ebay and then feeding it into froogle. Ebay does the same damn thing, but why not go through your aff links? Its worth a shot and there has got to be a good way to make some cash off it.

University Data- Such an asshole thing. I love it!

This is great. Keep em comin!

58 Comments To "Hot New List of Places To Scrape"

#1 Comment By Aur On November 4, 2006 @ 9:54 am

May I add Flickers in case you’re interested in having dynamic pictures content…
I don’t know if the value is good for SEO, but for users it is always nice to have some changing pictures on your website…

You can even display pictures from their static urls (urls of pictures are in their rss)… Which means no bandwidth from your website

Another favorite one of mine : Google News, you can get RSS for one specific search in the news, nice way to have dynamic content for one specific subject

#2 Comment By Will On November 4, 2006 @ 3:53 pm

I wonder if you could explain how scraping works for a newbie. I did a google search but I don’t think I understand the process.

Thanks and great blog.

#3 Comment By Vito On November 4, 2006 @ 7:55 pm

RSS feed? LOL

I am scraping the public libary and MAKING RSS feeds. How about you?


#4 Comment By Eli On November 4, 2006 @ 9:43 pm

vito- Got your email(haven’t had a chance to respond yet) awesome idea on the scraping public library database. Absolutely brilliant.

Will- I’ll see what I can do. Thanks for the compliment.

Aur- You are right text is definitely not the only content available. Its always a good idea to combine the two. Like I mentioned in the Youtube idea.

#5 Comment By bradlee On November 5, 2006 @ 3:48 am

I was checking out some black-hatters spammy site (he or she had linked to a page of mine that was #1 in the serps) and do you know what this spammer scraped a lot? EBAY! Lots of different categories but tons of ebook auctions. They got a gazillion categories. Lots of text of varying lengths. Sprinkle in a couple extra keywords, markov, BAM! very unique content!

#6 Comment By phil On November 6, 2006 @ 1:20 pm

Great post, I would add that you can scrap university data, like course ciriculum, study guides and other online resources and tie that into related aff. programs and offers to kill.

#7 Comment By john On November 8, 2006 @ 1:11 pm

so what do you use to scrape when you can’t use RSS?

#8 Comment By bradlee On November 8, 2006 @ 2:53 pm

You scrape with fairly simple .php scripts, if that is what you are asking. They are often found in larger page generator packages. There are a few free ones out there. Some of the scripts are well “commented” also, and are great aids in learning php.

#9 Comment By roy On November 9, 2006 @ 11:13 am

bradlee,can u give a example on how to?

#10 Comment By George On November 10, 2006 @ 1:47 pm

What’s your favorite scraping software? I tried WebSpinner but I haven’t really evaluated any other software packages.

#11 Comment By bradlee On November 10, 2006 @ 3:30 pm

Roy, I can’t give a how to here, but you should search for RSSGM, and MYGEN. Both are content generators that are free and both scrape for content. A little investigating in the code and you can find the scraping part. MYGEN is actually built off of rssgm, so maybe MYGEN would be better to start with.

#12 Comment By Eli On November 10, 2006 @ 7:05 pm

Fairly soon I will be releasing a beginner-advanced guide to scraping. So hopefully that will clear up some pending questions on the topic.

#13 Comment By Sam On November 11, 2006 @ 8:07 pm

thats great Eli. Looking forward to your guide to scraping.


#14 Comment By Aur On November 14, 2006 @ 1:13 pm

Hi Eli,
I was wondering, how would you advise to organize scraped contents?

- either having one domain, with one blog containing 10000 scraped articles,

- or one domain with 100 subdomains, and one blog per subdomain, each containing about 100 scraped articles?

#15 Comment By John On December 5, 2006 @ 3:01 pm

What is the point of all this?

#16 Comment By neil strauss On January 25, 2008 @ 4:26 am

So what is the best and easiest to use scraping software??

#17 Comment By rider On February 16, 2009 @ 12:30 pm

This list is sweet thanks

#18 Comment By new air max On August 9, 2010 @ 6:01 pm

thanks for giving me such useful and great technique,this will help me in future.

thanks again!

#19 Comment By Peter Dunin On September 13, 2010 @ 5:26 am

useful list,cheers for sharing it!

#20 Comment By India Tour Packeges On October 9, 2010 @ 9:28 pm

This is Sweet Thank you so much for this!

I had some idea’s of trackback links but this sealed the deal.

I look for more…

#21 Comment By wolanlw On November 7, 2010 @ 10:48 pm

The well-being of our environment is a big 2social bridesmaid dresses,bridesmaid dresses and all companies should strive to do their part in bridesmaid dresses uk it.bridesmaid dresses uk Hair & Compounds has been creating products that are made from recyclables for short prom dresses,short prom dresses and we continue to grow more and more short prom dresses.
Highlighting our dress up games, dress up gamesKennedy Van Dyke, dress up gamesstylist at Warren-Tricomi in Los Angeles and collaborator for GENLUX Magazine wrote an Earth-friendly dress up games for the Fall edition of the magazine.

#22 Comment By imleme On November 11, 2010 @ 5:48 am

you never cease to amaze me. thanks admin.

#23 Comment By abercrombie New York On December 27, 2010 @ 12:43 am

When you think of happy, unhappy or think of you when you smile in my mind wander

#24 Comment By hollister uk On December 31, 2010 @ 8:48 am

i feel so good

#25 Comment By Wireless Networking On January 15, 2011 @ 3:59 pm

This is a quality blog,thank you for passing on your knowledge with us all. Many Thanks!

#26 Comment By Freelance SEO India On February 2, 2011 @ 2:52 am

India’s leading Freelance SEO India services provider with main competency in Search Engine Optimization of websites.

#27 Comment By Güncel Blog On February 18, 2011 @ 4:47 am


#28 Comment By Güncel Blog On February 18, 2011 @ 4:47 am

I like..

#29 Comment By Güncel Blog On February 18, 2011 @ 4:47 am


#30 Comment By Güncel Blog On February 18, 2011 @ 4:48 am


#31 Comment By Arthur V On April 2, 2011 @ 6:16 pm

Very useful list and great info!!

#32 Comment By abercrombie milano On May 16, 2011 @ 11:16 pm

qThere is noticeably a bundle to learn about this. I assume you made sure nice points in features also.

#33 Comment By abercrombie london On May 17, 2011 @ 2:59 am

66I believe this really is excellent information. Most of men and women will concur with you and I ought to thank you about it.

#34 Comment By flash game On May 20, 2011 @ 4:37 am

great one!!

#35 Comment By kadın On July 29, 2011 @ 5:00 am

I do agree with all of the ideas you have presented in your post. They’re really convincing and will definitely work. Still, the posts are too short for newbies. Could you please extend them a bit from next time? Thanks for the post.

#36 Comment By Wireless Networking On August 3, 2011 @ 10:26 am

I think its plain luck. This article is extremely fruitful for me and other visitors.

#37 Comment By Rowe Cohen On September 7, 2011 @ 12:05 am

Now we’re really scraping the barrel.

#38 Comment By Property law On September 28, 2011 @ 7:49 am


#39 Comment By Solicitors in Bournemouth On September 29, 2011 @ 11:16 am

one two theree

#40 Comment By Microsoft Outlook 2010 On November 25, 2011 @ 8:26 pm

This article is GREAT it can be EXCELLENT JOB and what a great tool!

#41 Comment By Moncler On December 21, 2011 @ 8:23 pm

Although this was a time of Moncler Winter Jackets – think Wall Street, “greed is good” and so on, one of the Moncler Coats Women was, of course, the oil industry, so this part of Moncler Jackets Men was indubitably where the money was. It was a decade dedicated to conspicuous consumption, Moncler Coats Women and branding yourself with designer labels. Moncler Women went from wanting to marry the millionaire to wanting to be the millionaire, and so shows such as Moncler Boots weren’t just television fiction, they reflected the attitude and aesthetic of the time, as well as the financial power wielded

#42 Comment By Summer Holidays On January 6, 2012 @ 8:43 pm

Thanks for the informative post. Keep it up.

#43 Comment By Nitish On January 8, 2012 @ 7:43 am

Great Post Eli

#44 Comment By Nitish On January 8, 2012 @ 7:43 am

Four Five Six

#45 Comment By Nitish On January 8, 2012 @ 7:44 am

It surely is xD

#46 Comment By Nitish On January 8, 2012 @ 7:44 am

Yep a great one

#47 Comment By Ismat Zahra On January 8, 2012 @ 11:12 am

yeah ryt . realy very gr8 blog :)

#48 Comment By Ismat Zahra On January 8, 2012 @ 11:14 am

gr8 info :)

#49 Comment By Ismat Zahra On January 8, 2012 @ 11:15 am

yeah true #8

#50 Comment By Ismat Zahra On January 8, 2012 @ 11:16 am

yeah it is :@

#51 Comment By Ismat Zahra On January 8, 2012 @ 11:16 am

Seven Eight Nine

#52 Comment By Ismat Zahra On January 8, 2012 @ 11:17 am

yeah awsome… Eli..

#53 Comment By Property Marbella On April 25, 2012 @ 12:03 am

I did a google search but I don’t think I understand the process.

#54 Comment By FixCleaner review On April 29, 2012 @ 9:13 am

Thank you it is great!

#55 Comment By Güncel blog On June 25, 2012 @ 5:35 pm

I wonder if you could explain how scraping works for a newbie. I did a google search but I don’t think I understand the process.

Thanks and great blog.

#56 Comment By hut be phot On September 2, 2012 @ 6:24 pm

This list is sweet thanks

#57 Comment By thong cong On September 3, 2012 @ 12:30 am

you never cease to amaze me. thanks admin fod sharing it

#58 Comment By chong tham On September 8, 2012 @ 6:19 am

thats great Eli. Looking forward to your guide to scraping.

Article printed from Blue Hat SEO-Advanced SEO Tactics: https://www.bluehatseo.com

URL to article: https://www.bluehatseo.com/hot-new-list-of-places-to-scrape/

Click here to print.