Brizy.cloud : Robots.txt - sadly nofollow?

Smart Suburbs

• August 25, 2020 06:15

Hello team,

For my brizy.cloud listed hinjawadi.in website, i noticed that the robots.txt has disallow. Probably that is the reason why my urls are not getting indexed even though sitemap has 28 urls. Why is the User-agent marked Disallow? It should be Allow:  correct?

# www.robotstxt.org/
# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

User-agent: *
Disallow:

regards,

parag

0

• 

  Sandra Prunici

  • August 26, 2020 08:08

  Hi,

  At the moment we don't have specified this file in Brizy Cloud project and in this case, it is set with default settings and allow all Serch Engine Crawlers to crawl files, web site and directories. In your screen, it isn't said that the user-agent is disallowed. The "User-agent" and "Disallow" are terms used in the robots.txt file.
  - user-agent: denotes the name of the crawler (the names can be found in the Robots Database);
  - disallow: prevents crawling of certain files, directories or web pages;
  - *: stands for any number of character

  More details here.

  You said that your site isn't indexed, could you please give us more details regarding this? Did you set the Site Settings of the project? Could you send us the URL link of the project?

  Best regards,
  Sandra

  0
• 

  Smart Suburbs

  • August 26, 2020 11:47

  Thank you Sandra for your speedy response.

  Yes, i have over 10 websites on wordpress and have a fair grip on robots & sitemaps.

  I think the disallow tag without classification will disallow every content on the brizy cloud site. That is my understanding.

  Checkout my website: https://hinjawadi.in/sitemap.xml and

  https://hinjawadi.in/robots.xml

  While i have submitted the sitemaps in google search console, i don't see the traffic being recorded.

  Please check and let me know if i am reading wrong.

  regards,

  parag

  0
• 

  Smart Suburbs

  • August 26, 2020 12:21

  The default robots.txt i think should be like below for all agents to crawl and index.

   

  User-agent: *
  Allow: /

  0
• 

  Sandra Prunici

  • August 26, 2020 15:39

  Google algorithms are very complicated and it is a bit complicated to understand why it indexes some sites very fast and others a bit slowly. Mostly it depends on the content you have on the site. After this, Google decides when to index your site.

  The default or standard form of the robots.txt that allow all full access is this one:

  User-agent: *
  Disallow:

  See this article.

  Best regards,
  Sandra

  0

Please sign in to leave a comment.