# http://en.wikipedia.org/wiki/Robots.txt Sitemap: http://www.deminy.net/comments.rss2 Sitemap: http://www.deminy.net/index.rss2 Sitemap: http://www.deminy.net/rss.guestbook.php Sitemap: http://www.deminy.net/sitemap-index-deminy_wiki.xml Sitemap: http://www.deminy.net/sitemap User-agent: * Disallow: /zpq/ Disallow: /tools/ Disallow: /test/ Disallow: /temp/ Disallow: /taolige/ Disallow: /taolige.bak/ Disallow: /qingqing/ Disallow: /priv/ Disallow: /prebyte/ Disallow: /msg/china/ Disallow: /msg/first/ Disallow: /msg/firstlove/ Disallow: /msg/joke/ Disallow: /msg/love/ Disallow: /msg/note/ Disallow: /msg/politics/ Disallow: /msg/qingqing_update/ Disallow: /msg/settings/ Disallow: /mp3/ # Disallow: /oncemore/ Disallow: /love.bak/ Disallow: /love/ Disallow: /includes/ Disallow: /egw/ Disallow: /download/ Disallow: /dingnan/ Disallow: /digest/ Disallow: /cam/ Disallow: /calendar/ Disallow: /articles/data/ Disallow: /articles/dingding/ Disallow: /articles/flash/ Disallow: /articles/materials/ Disallow: /abcit/ Disallow: /cgi-bin/ Disallow: /japanmake/ Disallow: /rss/ Disallow: /bookmark/ Disallow: /bookmark/tag/ Disallow: /bookmark/rss/ Disallow: /bookmark/rss92/ Disallow: /blog/authors/ Disallow: /blog/categories/ # Disallow: /blog/feeds/ Disallow: /blog/unsubscribe/ Disallow: /blog/approve/ Disallow: /blog/delete/ Disallow: /blog/admin/ Disallow: /blog/entries/ Disallow: /blog/plugin/ Disallow: /blog/search/ # Disallow: /blog/archives/ Disallow: /articles/page/00000008.php # added at 2007-07-15 17:34:12 # Disallow: /maintenance/ #remove this, otherwise Wiki's sitemap won't work 2007-07-16 21:36:13 Disallow: /photos/ Disallow: /todolist/ Disallow: /unfuddle/ Disallow: /backup/ # http://devpit.org/wiki/Installing_Mediawiki#Robots.txt # http://en.wikipedia.org/robots.txt # advertising-related bots: User-agent: Mediapartners-Google* Disallow: / # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / # # Sorry, wget in its recursive mode is a frequent problem. # Please read the man page and use it properly; there is a # --wait option you can use to set the delay between hits, # for instance. # User-agent: wget Disallow: / # # The 'grub' distributed client has been *very* poorly behaved. # User-agent: grub-client Disallow: / # # Doesn't follow robots.txt anyway, but... # User-agent: k2spider Disallow: / # # Hits many times per second, not acceptable # http://www.nameprotect.com/botinfo.html User-agent: NPBot Disallow: / # A capture bot, downloads gazillions of pages with no public benefit # http://www.webreaper.net/ User-agent: WebReaper Disallow: / # Don't allow the wayback-maschine to index user-pages #User-agent: ia_archiver #Disallow: /wiki/User User-agent: * Disallow: /w/ Disallow: /trap/ Disallow: /wiki/Special:Random Disallow: /wiki/Special%3ARandom Disallow: /wiki/Special:Search Disallow: /wiki/Special%3ASearch # Folks get annoyed when VfD discussions end up the number 1 google hit for # their name. See bugzilla bug #4776 # en: Disallow: /wiki/Wikipedia:Articles_for_deletion/ Disallow: /wiki/Wikipedia%3AArticles_for_deletion/ Disallow: /wiki/Wikipedia:Votes_for_deletion/ Disallow: /wiki/Wikipedia%3AVotes_for_deletion/ Disallow: /wiki/Wikipedia:Pages_for_deletion/ Disallow: /wiki/Wikipedia%3APages_for_deletion/ Disallow: /wiki/Wikipedia:Miscellany_for_deletion/ Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/ Disallow: /wiki/Wikipedia:Miscellaneous_deletion/ Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/ Disallow: /wiki/Wikipedia:Copyright_problems Disallow: /wiki/Wikipedia%3ACopyright_problems Disallow: /wiki/Wikipedia:Protected_titles/ # en.wikinews: # https://bugzilla.wikimedia.org/show_bug.cgi?id=5340 Disallow: /wiki/Portal:Prepared_stories/ Disallow: /wiki/Portal%3APrepared_stories/