What command in Robots.txt exactly works for blocking Dynamic URLs??
We have been hitting on this thing back and forth so many times in the forums and there seems to be varied answers to this question.
Some say that:
user-agent: *
disallow: /filename.php
Will block not only the filename.php file but also any other query strings/parameters which it attached to it.
While some say that
user-agent: *
disallow: /filename.php*
is the one which works.
So I thought of starting a small test to see which command is really effective in blocking Dynamic URLs through the robots.txt file.
We'll take a domain that has dynamic pages in it. Then we'll try to block those dynamic pages with both the above mentioned commands PLUS any other which anyone of you can suggest. We'll apply those commands one at a time and see which one really blocks those dynamic pages. I believe that this test will be an eye-opener for all those like me who are still in this dilemma.
Any Suggestions?

News