What to create Robots.txt file and searching on web for this then you are at right place In this article we will cover all the topics about How to create robots.txt file perfectly.
First of all we should know about what is Robots.txt file
Robots.txt is a file which tells search engines like Google, Yahoo, Bing etc to crawl and index specific article. It generally tells search engines that these articles or pages are for indexing and other those are not for indexing.
Are robots.txt file necessary for blog
It is difficult to tell that your website need a robots.txt file or not because there are some websites that wants to index all their contents in search engines. But if you want to hide some contents like your WordPress admin panel, some pages , tags , categories then you need this.
Finding your robots.txt file
It is very easy For you to find your robots.txt file by following this easier method it works with any website what you have to do is to type the url of your website for example https://tricksmaffia.com Then add /robots.txt onto the end.
- You will get your robots.txt file
- Or You will get a empty page
If you gets your robots.txt file this means that your website already have a Robots.txt file which was probably created by some SEO plugin like rank math or yoast SEO. And you need to improve your Robots.txt File.
And if you didn’t find your robots.txt File then you need to create Robots.txt file perfectly by following some steps which We will discuss with you later
Basic Rules for Setting Up Robots.txt file
To create Robots.txt perfectly you need to know about it perfectly because a single mistake in it can remove your site completely from search engines so it is very important for your to learn some important terms about it like What to hide form from search engines.
Here are some pages you should hide using a robots.txt file:
- Pages with duplicate content
- Pagination pages
- Dynamic product and service pages
- Account pages
- Admin pages
- Shopping cart
- Thank-you pages
Here are some examples through which you can Create Robots.txt File Perfectly
For BloggerUser-agent: * Allow: / Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
For WordPressUser-Agent: * Disallow: /wp-admin/ Sitemap: https://example.com/sitemap_index.xml
For custom websiteUser-agent: * Allow: / Allow: /sitemap.htm Sitemap: https://example.com/sitemap.xml
what you have to do is to change only example.com to your domain name
Also read – Thrive Architect Free Download