How to Edit Robots.txt in CranSEO #

The robots.txt file controls how search engines crawl and index your site. CranSEO makes it easy to edit and customize your robots.txt file for optimal SEO performance.

Step 1: Access Robots.txt Settings #

1. Log in to your WordPress dashboard.
2. Navigate to CranSEO > Settings > Robots.txt.
3. Click Enable Custom Robots.txt if it is not already enabled.

Step 2: Edit Your Robots.txt File #

• In the Robots.txt Editor, you will see the current rules applied to your site.
• You can modify or add new rules depending on your needs.

Example Robots.txt Configurations #

Allow All Search Engines #

User-agent: *
Disallow:

Block All Search Engines #

User-agent: *
Disallow: /

Block Specific Pages or Directories #

User-agent: *
Disallow: /private-page/
Disallow: /wp-admin/

Allow Googlebot but Block Others #

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

Step 3: Save Changes #

1. After making edits, click Save Changes.
2. Test your robots.txt file by visiting https://yourwebsite.com/robots.txt.

Step 4: Test with Google Search Console #

1. Go to Google Search Console.
2. Select your website and navigate to the Robots.txt Tester.
3. Enter your new robots.txt rules and click Test to ensure they are valid.

Step 5: Monitor & Update as Needed #

• Periodically review your robots.txt file to ensure it is not blocking important pages from being indexed.
• Update rules as needed based on your SEO strategy.

---

#