1) robots.txt file
The robots.txt is a simple text file that is placed on our root directory. Search Bots, crawl each URL and the first thing they search on an URL root is the robots.txt file. A simple robots.txt looks like:
- User-agent: *
“User-agent: *” means this section applies to all robots, the wildcard “*” means all bots. The “Disallow: ” tells the robots that they can go anywhere they want.
To exclude all robots from the server:
- User-agent: *
- Disallow: /
robots.txt should have the same place where you put your web site’s main “index.html”.Also remember to use all lower case for the filename: “robots.txt”, not “Robots.TXT.
2) Google Webmaster guidelines :
- Design and content guidelines
- Every page should be reachable from at least one static text link.
- Offer a site map to your users with links that point to the parts of your site.
- Think about the words users would type to find your pages, and make sure that your site actually includes those words within it.
- Try to use text instead of images to display important names, content, or links. The Google crawler doesn’t recognize text contained in images. Using the “ALT” attribute to include a few words of descriptive text.
- Technical guidelines
- Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would.
- If your company buys a content management system, make sure that the system creates pages and links that search engines can crawl.
- Test your site to make sure that it appears correctly in different browsers.
3) Add your website on Google Webmaster which gives you following information:
- Site configuration: Sitemaps, Crawler access, Sitelinks, Change of address, Settings
- Your site on the web: Top search queries, Links to your site, Keywords, Internal links, Subscriber stats
- Diagnostics: Crawl errors, Crawl stats, HTML suggestions
4) Google Analytics
- Google Analytics is the enterprise-class web analytics solution that gives you rich insights into your website traffic and marketing effectiveness. You can check the following information:
- Traffic Sources
5) Bing Webmaster:
- Bing Webmaster Tools is to troubleshoot the crawling and indexing of your site, submit sitemaps and view statistics about your sites.
- Robots.txt validator
- HTTP verifier
- Keyword research tool
- Webmaster resources