Manual indexing consists of sending a series of files to search engines so that they know the structure and content of a web page. It is the recommended method because it is faster and more controllable.
Below we are going to list the steps to index a website manually.
How to create sitemap
The first step to create and verify a sitemap is to know what is a sitemap, an XML file that stores, in a structured way, the content of our website.
Although there are several types of sitemap, the best known are:
- XML Sitemap, the one used by robots like Google to index the content of a website. This will be the one that we will use, generate and send to search engines like Google.
- HTML sitemap, used by and for humans so that they know the content of a web page.
- RSS or ATOM sitemap, used by feed readers that allow people to subscribe to the content of a website.
After knowing this information, the next step is to create the sitemap, this point depends on what type of website you have:
- If you have a website created with a content manager, it will most likely be capable of generating it automatically. For example, WordPress and Blogger are capable of performing this task easily and automatically.
- If you have a website without a content manager, that is, you have a generic web page, you can generate the sitemap manually or automatically using an Internet tool such as, for example, XML-Sitemaps.
Finally, and as a last step, we will need to verify that the sitemap is correct using a website such as, for example, XML validator XSD.
How to send sitemap
Once the file has been generated through any of the options set out above, it is necessary to send the sitemap to the search engines so that they know the content of the site. This step depends on where we want to send it and, therefore, the steps to send the sitemap to Google and Bing are different:
- To send the sitemap to Google we will use Google Search Console. This tool not only allows submission, it also allows you to verify that the sitemap submission is correct, know the number of indexed pages and know some indexing errors in Google such as, for example, alternative page with appropriate canonical tag.
- To send the sitemap to Bing we will use Bing Webmaster Tools, the Microsoft tool that allows us to manage issues related to the SEO of a website. Again, this tool allows us to verify that the shipment has been made correctly and shows us a list of the indexing errors on our website.
Finally we will see how to know what we have indexed in Google and Bing through the tools previously exposed and through the search, both in Google or Bing, for our website with the command “site:https://myWeb.com”.
What is robots.txt
The robots.txt TXT file is a text file that is used to tell spiders which content is traceable and which is not. It is interesting to know how the robots.txt file works, it is a blacklist where what is not explicitly prohibited is allowed.
A simple example of robots.txt file is the following where, for the spider with the name psbot, we deny tracing the /cgi-bin/ folder and, since we have nothing else, we allow that specific spider to trace the rest of the web, and we allow all spiders that do not have that name to trace the entire web:
User-agent: psbot
Disallow: /cgi-bin/The main difference between the sitemap and robots.txt is:
- The sitemap allows search engines to indicate the structure or content of our website.
- The robots.txt file is used to prevent web spiders from tracing certain parts of a website.
In addition to the above, it is important to know that the robots.txt file is not used to remove a page from search engines, there are other techniques for this, such as:
- Delete the specific page from the website.
- Mask the content behind a login so that Google cannot access it.
- Use the noindex HTML meta tag.
Although, as we mentioned previously, the sitemap and the robots.txt file are different files, it is possible to indicate in the robots.txt file where the sitemap is by simply adding the following line to the end of the file:
Sitemap: http://myWeb/sitemap.xmlFinally, we must take into account that we can tell the main search engines such as Google or Bing, through Google Search Console and Bing Webmaster Tools, where the robots.txt file is located (it is important to always place it at the root of the web page) and check and correct, if they exist, errors in the robots.txt file.
