Recently I published many tutorials on SEO’s topic on this blog as well as on My Youtube Channel. Many guys emailed Me & contacted me. Some of them appreciated my SEO efforts & many of them asked many questions about SEO. Few of them asked about Robots.txt file; that what is this file and what does it do?. Therefore today I created an urgent video tutorial in Urdu about Robots.txt file and describe it with examples. You’ll learn how to create this file and how to carefully use this file in Urdu, if anything went wrong then it might very harmful for your website, so you should be doing things very carefully. However, in case something goes wrong, you can instantly remove this file from your website to avoid bad experience.
What is Robot.txt File?
Robot.txt file is a text file which instructs the search engines robots/crawlers/spiders which folder or page of any a website should not be accessed by them. You can disallow any search robot from some of your website’s specific folders or pages. They will not be able to access those pages which you’ve specified in the robots.txt file.
How to create a robots.txt file?
The syntax of creating this file is quite simple, just open notepad & save the file, give it a name as robots.txt and write
following two lines:
The above example is the simplest syntax for creating a robots.txt file, and the the first line “User-agent*” means that all crawlers/robots are allowed to access all the pages/folders of the website, however you can specify a folder/page in the below line “Disallow:/” you just need to write that page/folder after the / backslash.
if you wanna disallow search robots from any of your pages or folders then just add that page address after the back slash of Disallow line. Like this: Disallow: /kaka.html. Upload that txt file to your root directory of your website and you’re all done! in Blogger you can find this file by default by visiting setting page & then search preferences. Also if you want to disallow many pages or folders in the txt file then just do like this :
In above txt file we disallowed some pages and folders of our site, so now Search robots/crawlers can not access them, and they will neither be indexed nor be crawled.
Video is also here in Urdu to help you better understanding of it.