Test your robots.txt file
- Open the tester tool for your site, and scroll through the robots.
- Type in the URL of a page on your site in the text box at the bottom of the page.
- Select the user-agent you want to simulate in the dropdown list to the right of the text box.
- Click the TEST button to test access.
You need to remove both lines from your robots. txt file. The robots file is located in the root directory of your web hosting folder, this normally can be found in /public_html/ and you should be able to edit or delete this file using: FTP using a FTP client such as FileZilla or WinSCP.
In practice, robots. txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website. These crawl instructions are specified by “disallowing” or “allowing” the behavior of certain (or all) user agents.
The robots exclusion standard, also known as the robots exclusion protocol or simply robots. txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
Follow these simple steps:
- Open Notepad, Microsoft Word or any text editor and save the file as 'robots,' all lowercase, making sure to choose . txt as the file type extension (in Word, choose 'Plain Text' ).
- Next, add the following two lines of text to your file:
The robots. txt file must be located at the root of the website host to which it applies. For instance, to control crawling on all URLs below example.com/ , the robots. txt file must be located at example.com/robots.txt .
Place your robots. txt file in the top-level directory of your website code to simplify crawling and indexing. Structure your robots. txt properly, like this: User-agent → Disallow → Allow → Host → Sitemap.
Most websites don't need a robots. txt file. That's because Google can usually find and index all of the important pages on your site. And they'll automatically NOT index pages that aren't important or duplicate versions of other pages.
Stick to lowercase, always when it comes to SEO. If you want your robots. txt file to be found, you have to place it in the main directory of your site. The disallow instructions are required so that search engine bots understand your intent.
You can
enable the
editors by changing your wp-config. php
file.
If you use a Managed WordPress installation, enabling the theme and plugin editor is currently not supported.
- Use SFTP and go to the location of your WordPress site.
- Open wp-config.
- Search for:
- Change true to false:
You need to remove both lines from your robots. txt file. The robots file is located in the root directory of your web hosting folder, this normally can be found in /public_html/ and you should be able to edit or delete this file using: FTP using a FTP client such as FileZilla or WinSCP.
Robots. txt is a text file which allows a website to provide instructions to web crawling bots. Search engines like Google use these web crawlers, sometimes called web robots, to archive and categorize websites.
Create or edit robots.txt in the WordPress Dashboard
- Log in to your WordPress website. When you're logged in, you will be in your 'Dashboard'.
- Click on 'SEO'. On the left-hand side, you will see a menu.
- Click on 'Tools'.
- Click on 'File Editor'.
- Make the changes to your file.
- Save your changes.
To fix this, you should audit your robots. txt file to identify rule that's blocking the pages. You should edit or remove the rules effecting the pages you want to fix. After you have fixed the file, click the “Validate Fix” button in Google Search Console.
The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.
Mostly it takes around 12 to 24 hours if your website is a new it make take more than that. If your site is new then submit it to all search engines manually, this will help you get indexed faster. If your site is old and you have blocked only a few pages, it takes 12 to 24 hours as said earlier.
In a nutshell
Web site owners use the /robots. txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "Disallow: /" tells the robot that it should not visit any pages on the site.Robots.txt files
- Go to cPanel → File Manager → Your site's root folder.
- Click New File.
- Name the file 'robots. txt' and click Create. Check if it shows up on the list. The icon is similar to the Windows Notepad icon or Mac TextEdit icon.
- Select the robots. txt file and click Edit on the panel.
- Type:
Well including sitemap in robots. txt is good praticle but isn't required. You can add sitemap in SearchConsole (or Bing WMT) and bots will index site again. Even bots can index site w/o sitemap but sitemap helps them.
