Better Robots.txt Pro may be a game changer. This plugin, supported thousands of SEO analysis, was specifically made to spice up your site ranking and proper a recurrent problematic of content indexation (seen on quite 95% WordPress websites).
Better Robots.txt Pro helps you create a virtual Robots.txt file with simple clicks, add in specific instructions for many popular web robots (including a Crawl-Delay feature) also as your sitemap URL (generated from Yoast SEO plugin) or the other sitemap. Better robots also will assist you block bad bots known for scraping (illegally) great deal of knowledge .
This plugin was also designed to form direct modifications inside the Robots.txt and increase your SEO by taking advantage of a natural a part of every website that rarely gets talked about.
The robots.txt file: a source of SEO juice just waiting to be unlocked.
Better Robots.txt :
Detects your Sitemap index (sitemap-index.xml) from Yoast SEO plugin (or the other Sitemap generator) and adds it automatically to your Robots.txt
Allows the addition of your own Sitemaps (manually)
Allows the edition of your robots.txt to feature custom rules
Allows to define a selected crawl-delay for program bots
Blocks bad bots from crawling your website
Blocks most SEO crawlers so as to form backlinks undetectable.
Boosts your online store while blocking useless links from Woocommerce
Avoids Crawler traps generating Crawl-Budget issues
Provides 150+ Growth Hacking tools