Joomla! Tips: customize robots.txt
One of the poorest sides of Joomla! is that it does allow multiply URLs be the very same page. Which for search engines usually means duplicated content.
The effects from this are not very clear, however it is better if this can be avoided.
What I usually do with robots.txt and some other tricks, in order at least to have my page look more clear in the webmasters tools.
1. Very common thing is that some template designers put a link to the logo to site.com/index.php - which basically means duplication with site.com. Therefore, if I use a ready template, I look at its code and change all links to index.php just to site.com.
2. If I use SEF (the built-in), I always in fact forbid search engines to index anything that starts with index.php, because this is duplicated content in 99% of the cases. I do this by simply adding in robots.txt:
Disallow: /index.php
3. I add the following line in robots.txt, if I want google to index my images:
Allow: /images/stories
This line should go, BEFORE the Disallow: /images
4. I use only www or totally forbid www in my web sites (it depends in every case, to use www or not), by redirect in htaccess
Add this code, to remove www:
Use this code to always force www:
component/COMPONENTNAME/VIEW/SOMETHING/SOMETHING
This happens, when certain item is not linked in any way through a menu item. Well, if I haven't link something via such item, it is either a duplicated content, or something I don't want to show. Therefore, I see no reason, why Google should index it. By adding to robots.txt:
Disallow: /component
It very well solves the issues with duplicated content.
Share your experience in making your web sites more SEO friendly.
The effects from this are not very clear, however it is better if this can be avoided.
What I usually do with robots.txt and some other tricks, in order at least to have my page look more clear in the webmasters tools.
1. Very common thing is that some template designers put a link to the logo to site.com/index.php - which basically means duplication with site.com. Therefore, if I use a ready template, I look at its code and change all links to index.php just to site.com.
2. If I use SEF (the built-in), I always in fact forbid search engines to index anything that starts with index.php, because this is duplicated content in 99% of the cases. I do this by simply adding in robots.txt:
Disallow: /index.php
3. I add the following line in robots.txt, if I want google to index my images:
Allow: /images/stories
This line should go, BEFORE the Disallow: /images
4. I use only www or totally forbid www in my web sites (it depends in every case, to use www or not), by redirect in htaccess
Add this code, to remove www:
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.domain.com$ [NC]
RewriteRule ^(.*)$ http://domain.com/$1 [R=301,L]
Use this code to always force www:
RewriteEngine on5. The most important, according to me. Joomla! has a big issue by having routing like:
RewriteCond %{HTTP_HOST} !^www.your_domain.com$
RewriteRule ^(.*)$ http://www.your_domain.com/$1 [R=301]
component/COMPONENTNAME/VIEW/SOMETHING/SOMETHING
This happens, when certain item is not linked in any way through a menu item. Well, if I haven't link something via such item, it is either a duplicated content, or something I don't want to show. Therefore, I see no reason, why Google should index it. By adding to robots.txt:
Disallow: /component
It very well solves the issues with duplicated content.
Share your experience in making your web sites more SEO friendly.
Views: 399
Tags: content, duplicated, google, htaccess, indexing, joomla, robots.txt, seo
-
Comment by Amy Stephen on February 6, 2010 at 8:05pm
-
Excellent advice! Many times, people do not address your point #5. That's important.
-
Comment by Ivo Apostolov on February 6, 2010 at 8:14pm -
One more... a hidden secret in Joomla! (don't know if someone mention it).
If anyone wants to have custom alias to certain page (article) , but does not want to put it in a menu as menu item, this could be done easily by creating a new menu, which corresponding module is unpublished. And then placing the links to these pages (articles, or other components) in this menu. The alias is still valid and usable (which avoids component/ANYTHING...) and can go easily in sitemaps (i.e. XMAP can use it) and still to not be directly linked in the site.
-
Comment by Mark W. Bender on February 6, 2010 at 8:52pm -
@Ivo - I use the "hidden menu" trick on several websites. Also a great menu to put "landing pages".
-
Comment by Ivo Apostolov on February 7, 2010 at 2:47am
-
Leo
It is components in the standard. And it blocks from access to the physical folder. Blocking component is something different.
-
Comment by Ivo Apostolov on February 7, 2010 at 6:11am
-
@Nicholas, I doubt my articles will be accepted in the Joomla! Community Magazine.
-
Comment by Ivo Apostolov on February 7, 2010 at 6:31am
-
Nicholas, I don't mind trying. ;)
-
Comment by Mustaq Sheikh on February 7, 2010 at 9:31am -
A valuable posting indeed Ivo, thanks, and I also want to echo the suggestion made by @Nicholas K. Dionysopoulos . Let us know if you encounter any obstacles in that regard.
-
Comment by Gobezu Sewu on February 17, 2010 at 11:16am -
more of a performance additions but as of .... speed is as well taken into considerations of seo, so here you go
start by turning off etags (redundant and unnecessary, last modified handles this well and consitently) and then head on with caching static contents for 2 months
FileETag none
ExpiresActive On
ExpiresDefault A3600
ExpiresByType image/x-icon A5184000
ExpiresByType application/x-javascript A5184000
ExpiresByType text/css A5184000
ExpiresByType text/javascript A5184000
ExpiresByType image/gif A5184000
ExpiresByType image/png A5184000
ExpiresByType image/jpeg A5184000
ExpiresByType text/plain A86400
ExpiresByType application/x-shockwave-flash A5184000
ExpiresByType video/x-flv A5184000
ExpiresByType application/pdf A5184000
ExpiresByType text/html A3600
ExpiresByType application/xhtml+xml A3600
ExpiresByType application/xml A3600
-
Comment by Gobezu Sewu on February 17, 2010 at 11:18am
-
hmmm this forum editor is just wonderful, stripping out any thing that resembles to code, my second one today, i am sorry but there are some condition tags for each part that are removed
Comment
© 2012 Created by Amy Stephen.
You need to be a member of All Together, As A Whole to add comments!
Join All Together, As A Whole