Google Custom Search Engine

Finally after much work I have a Google Custom Search Engine working for a public site I am working on for a friend. In the past I have worked on many internal web sites or private forums so I did not need for them to be located in a public web search. I have many times wanted a way to search thru this content without having to write my own custom search. So this week brought me to the place where I wanted to try to create a custom search using Google.

This is where I started my adventure in googling. I also needed to know the format to allow image, video, audio and PDF files to be indexed in the search. I also need a nice summary that helps me thru the process so this article was so helpful. I then thought it would be nice to have a tool that just does all of this for me and google provides a list of validators and generators. I tried a couple of them and was not happy with the results. I found myself referring to video format page as it was the hardest one to solve. Since all of the videos are stored publicly, I found this page that explains how to add them from Vimeo correctly. The nice thing about that page is that it describes how to get this working in other video formats also.

I first started by creating my own Google Custom Search Engine and then add the top level URL as the main index page. I waited a full 24 hours to see what the Google found by default. The results were disappointing as it never found more than the initial 9 pages I had at the time when I defined the search engine. I tried and tried and could not find a way to get the new pages to show up.

My next attempt that to create a list of URLs by using an on-line site list generator. That gave me a list and after waiting 24 hours that gave me all of the pages showing up in my search, including content inside my text based PDFs.

My next attempt was to add a site map directly in the Google Custom Search Engine  control panel. But after waiting yet another 24 hours I found that it did absolutely nothing. That is when I realized I was not doing something right. I clicked on the grey question mark next to "Identified Sitemaps" and it had a linked to "Submit a Sitemap". That is when I learned about Google Webmaster Tools and the ability to validate my site map. The odd thing is that I had already validated it on a couple of sites and they both said the file was valid. When I loaded my site map to Google Webmaster Tools in order test it, I saw all kinds of odd errors. The list of error codes is always helpful as there is no way to get the site map XML correct the first time as I was creating it manually myself.

Finally I have a working site map, but it took me over a week working part time on weekends and at night. I have to wait until the video and images are processed so by tomorrow I should be done with this whole affair. I have to honestly say that it is not a love affair as I did not really enjoy the process and life is all about enjoying the journey so I need to find a another adventure that is more fun.