# Reglas generales para todos los crawlers # - Allow explícito de JS/CSS (evita bloqueos accidentales) # - Bloqueo de index.php, /*/1000, PDFs, URLs con query, búsquedas, PPC # - Bloqueo de áreas privadas/utilidades User-agent: * Allow: /*.js$ Allow: /*.css$ Disallow: /*index.php* Disallow: /*/1000 Disallow: /*.pdf$ Allow: /img?* Allow: /*?page= Disallow: /*? Disallow: /*?s= Disallow: /%24slug Disallow: /null Disallow: /template Disallow: /search$ Disallow: /search? Disallow: /ppc/* Disallow: /admin/ Disallow: /login/ Disallow: /registro Disallow: /comparador # Bots permitidos explícitamente User-agent: Googlebot-Image Allow: / User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: SemrushBot Allow: / User-agent: SemrushBot-SA Allow: / # Lista de bots bloqueados (scrapers) User-agent: AhrefsBot Disallow: / User-agent: UbiCrawler Disallow: / User-agent: BUbiNG Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: wget Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: / User-agent: psbot Disallow: / User-agent: Exabot Disallow: / User-agent: Speedy Disallow: / User-agent: dotbot Disallow: / User-agent: Bloglines/3.1 Disallow: / User-agent: Jyxobot/1 Disallow: / User-agent: cityreview Disallow: / User-agent: CrazyWebCrawler-Spider Disallow: / User-agent: Domain Re-Animator Bot Disallow: / User-agent: Vegi Disallow: / User-agent: rogerbot Disallow: / User-agent: Baiduspider Disallow: / User-agent: GurujiBot Disallow: / User-agent: hl_ftien_spider Disallow: / User-agent: sogou spider Disallow: / User-agent: Yeti Disallow: / User-agent: YodaoBot Disallow: / User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / # Bots de IA permitidos (consentimiento implícito por Allow: /) User-agent: GPTBot Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / Sitemap: https://www.comparasoftware.com.ve/sitemap.xml