Adding additional directive to your Robots.txt file

Adding additional directive to your Robots.txt file

Overview

The robots.txt file controls how search engines access and index pages on your website. Vacation Labs provides a default robots.txt to maintain functionality, but you can add custom rules to refine search engine behavior.

Why Add Custom Rules?

Adding custom directives helps optimize SEO by guiding search engines to focus on important pages while avoiding unnecessary ones. This improves crawl efficiency and protects sensitive information.

Default Robots.txt in Vacation Labs

The default robots.txt file in Vacation Labs includes essential rules to prevent indexing of pages that should not appear in search results.

Default Robots.txt Content:

User-agent: *
Disallow: /itineraries/bookings
Disallow: /search
Disallow: /*/dates_and_rates
Disallow: /*/date_based_upcoming_departures

Sitemap: https://www.econatuarls.life/sitemap_index.xml

Default Rules and Their Purpose:

  • Disallow: /itineraries/bookings – Prevents search engines from indexing itinerary booking pages to avoid cluttering search results with transactional pages.

  • Disallow: /search – Blocks search result pages to prevent indexing of duplicate content that can dilute SEO value.

  • Disallow: /*/dates_and_rates – Stops indexing of real-time pricing pages, which can change frequently and are not useful for search engine results.

  • Disallow: /*/date_based_upcoming_departures – Prevents indexing of date-based departure pages to avoid outdated or irrelevant pages appearing in search results.

WarningNote: The default robots.txt file cannot be modified or removed. However, you can add additional directives to customize search engine behavior.

How to Add Custom Rules

To refine the robots.txt file further, follow these steps:

  1. Go to the Vacation Labs dashboard.

  2. Navigate to SEO & Analytics Settings.

  3. Locate the Additional Content for Robots.txt section.

  4. Add your custom rules.

  5. Save changes. Search engines will update during their next crawl.

Example Custom Rules

  • Blocking unwanted crawlers:

    User-agent: BadBot  
Disallow: /pricing

  • Allowing specific search engines:

    User-agent: Googlebot  
Allow: /  
User-agent: Bingbot  
Allow: /
NotesTip: Avoid blocking critical pages, such as product listings or key content, to maintain a strong SEO presence. Blocking important pages may reduce visibility in search results and impact traffic.


    • Related Articles

    • Adding sliders to the homepage banner

      Overview Vacation Labs lets you add multiple banner slides to your website (Homepage, Static Page, etc.) so you can showcase products in the first fold of the page. You can display different types of sliders, such as: Tour Slide – Shows tour details ...

    • Adding Multiple Sections to your Website

      You add multiple sections on any static page to show more content with different styles and layouts. For example: you may want to display more than 2 carousels for the Tour cards, or collection cards, etc. Please follow the below steps to add ...

    • Connecting Google Analytics to your Vacation Labs Website

      Quick Overview Optional: Create Google Analytics account and Property Get tracking code from Google Analytics Install Tracking Code Confirm that you can see the tracking code on your website Confirm that you can see visitor activity & reports on ...

    • Configure your account settings

      The SETTINGS tab, also know as Global Settings is divided into multiple parts and which allow you to control many important aspects of your Vacation Labs account. Let's go through them all: Payment Processors Payment Processors allow you to collect ...

    • Banner images and focus points

      Overview The "banner" image that you pick for a tour, collection, or static page, is shown at a number of places in a variety of sizes, aspect ratios, and orientations. For example: On the desktop, as a full sized "horizontal" banner image (what is ...