The "Crawler" link...

So this mythical "crawler" the cnet article mentions can be downloaded here. http://download.cnet.com/downloads/0...0_103_1.lst.td Haven't used it yet, but...ahh, I don't know if it's worth it. I don't even know if this is an "honourable" endeavor, but I've always been one for free thought. Hey, here it is, make your own decision. __________________ this is very intriguing. i wonder how many hidden pages we can find?? how many secrets we can unravel?? __________________ It's not exactly a star quality program. VERY hard to use. If anyone has anything better, let's hear it. __________________ what does this do? __________________ i guess this is a spider right? __________________ I thought it would index all the pages on a particular site, by following each link to a page, and then following all those links, etc. It doesn't. We just need to look for pages that contain the word "promo". If you do the same in Google, you'll get outdated ones, so I know it works. We just need an up to date crawl. Clueless, I guess it's a spider. I don't even know what that is. __________________ seems we are teetering on an ethical fine line here. What are some of your thoughts as to whether it would be right for us to use these types of programs to "seek out and capture" unannounced and/or undeveloped coupons and promos. Tim __________________ I don't know if ethical but i know that companys don't like them. not only do they find outdated or unannounced pages. it also may exploit security holes also it majorly lags the server. Many big companys use different protections agianst spiders but they are hard to keep out. hehe there is one technque that i heard of that is widley used in hotline which is to make a loop page haha those confuse the heck out of low level spiders =) __________________ This may be useful for future contests here? __________________ quote: Originally posted by tongyang: This may be useful for future contests here? true. true. __________________ Ethical fine line? Oh yeah, you bet. I can't even think of a B&M equivalent...maybe going and asking each employee if there are any deals? I mean, say we find all sorts of coupons at, say , pets.com. We get $15 off stuff there all the time, right? Ever read the coupon page? "THanks for shopping, Women's Health subscriber." "Thanks for shopping, AOL customer." None of those apply to me, yet it never raises any moral qualms with us, right? Should it? I dunno. So if we go and look for coupon pages at like buy or yes, pets, I'm not so sure we're wading into territory we've not already been in. __________________ so go for it! start using it, tell us how it is..... __________________ I'm pretty disappointed with this particular program, but I'm sure there are better ones out there. Anyone know of any? Mavi forum