Oh Dear is the all-in-one monitoring tool for your entire website. We monitor uptime, SSL certificates, broken links, scheduled tasks and more. You'll get a notifications for us when something's wrong. All that paired with a developer friendly API and kick-ass documentation. O, and you'll also be able to create a public status page under a minute. Start monitoring using our free trial now.

Say goodbye to manually creating a robots.txt file

Original – by Freek Van der Herten – 1 minute read

If you don't want a site to be indexed by search engines you must place a robots.txt file. Typically you don't want anything indexed except production sites.

Today Spatie released a new package called laravel-robots-middleware. It was coded up by my colleague Sebastian. Instead of you having to create a robots.txt this Laravel package will add a x-robots-tag to every request. If you want to limit a search bot to only crawl your production enviroment you can extend the Spatie\RobotsMiddleware\RobotsMiddleware-class like this:

// app/Http/Middleware/MyRobotsMiddleware.php

use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class MyRobotsMiddleware extends RobotsMiddleware
{
    /**
     * @return string|bool
     */
    protected function shouldIndex(Request $request)
    {
        return app()->environment('production');
    }
}

To learn all the options read the documentation on GitHub. Like this package? Then maybe some of our other Laravel packages might be useful to you as well.

Stay up to date with all things Laravel, PHP, and JavaScript.

You can follow me on these platforms:

On all these platforms, regularly share programming tips, and what I myself have learned in ongoing projects.

Every month I send out a newsletter containing lots of interesting stuff for the modern PHP developer.

Expect quick tips & tricks, interesting tutorials, opinions and packages. Because I work with Laravel every day there is an emphasis on that framework.

Rest assured that I will only use your email address to send you the newsletter and will not use it for any other purposes.

Comments

What are your thoughts on "Say goodbye to manually creating a robots.txt file"?

Comments powered by Laravel Comments
Want to join the conversation? Log in or create an account to post a comment.