Modify WordPress auto-generated robots.txt without plugin

WordPress generates automatically its own robots.txt and the best way to add your own rules in it, is to use the robots_txt filter.

Instead of using plugins or manually creating your robots.txt file, you can use WordPress robots_txt filter to modify the auto-generated robots.txt and add your rules.

Edit your theme (or child theme) functions.php and add the following snippet:

function add_to_robotstxt($output, $public) {
    $my_rules = "
# Added by robots_txt filter in functions
# Add your rules below, e.g.: 
Disallow: /wp-content/themes/
";
    return $output . $my_rules;
}

add_filter('robots_txt', 'add_to_robotstxt');

That’s it. That’s the preferred way to do it, since you don’t want a whole plugin just for a few lines of code and you don’t really want to create your own robots.txt file and miss any of the rules added by WordPress 😉