Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Maybe now, I don't remember I had do anything sneaky back then (8-10 years ago)


In 2004 I attempted to do some automatic crawling of Google for my masters thesis and was astonished to get an unfriendly server response saying it was disallowed and "don't even bother asking for an exception for research, it won't be granted."

So at least 11 years ago it was blocked.

(I didn't know about spoofing a user agent back then, so it might not have been as easy as that to get around it.)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: