If you want to allow robots/spiders to index all of the content it finds on your webpage then you can use the following in your robots.txt file .
User-agent: *
Disallow:
It is preferred for Google that you have either a blank robots.txt file, or you have the content above. If the request for the robots.txt file on your domain returns an error or a redirect status, your domain is at risk of being removed from the index.

News