It has been what seems an extraordinarily long time since I wrote in this blog. No doubt when I come to check the date of the last post, that will prove to be the case, but the last few weeks, even months have been unusually busy for many reasons.

Perhaps this is a good then to reflect and report back on the progress regarding the drop-shipping osCommerce site I mention in an earlier post. This site has had Supertracker installed since it was first installed and I have been using it as a means to keep tabs on not just what potential customers are doing, but also the behaviour of search engine bots.

As an aside, I note that one of the people arriving at this site in the last 48hrs did so using the following search query: “how to get the google bot to index you every 48hrs”, and perhaps by the end of this article I provide my own thoughts on that Utopia!

Now, I know of late that some people have requested that I introduce a means to filter out search engine robot traffic from the Supertracker logs and this is something that I will introduce, but this information has actually been quite useful for me over recent months. For example, I have seen that pretty much from day 1 of the new osCommerce site being made live, Inktomi Slurp (aka Yahoo) has been spending time on and off every single day indexing every product page on my site (over and over and over again). This of course is good news, and brings in relevant search traffic to the site from Yahoo as a result.

I also see that MSN’s robot makes regular visits too, but perhaps is not quite so diligent in indexing all the pages. Googlebot however, has been a different story…. for weeks and weeks I didn’t see it at all. I have the google sitemaps contribution installed and the sitemap registered with google (though as it says on the tin – this isn’t a guarantee of anything!). So, eventually when Googlebot turns up a few weeks ago, I think, great, now it’s going to do its thing and check out the whole of my site…. But alas no, it just indexes index.php and leaves it at that. Every few days it would do this and nothing more. So, I would see a few very specific google searches arriving at my site, where people have searched for items that happen to be featured on the homepage, but not the level of traffic I was hoping for.

Then, about a month or so ago, Googlebot came back and started doing the job properly and indexing all the products and categories pages. It has made regular re-appearances every few days to do some more. However, when I search for the site by name on Google, these pages are clearly not yet available in the search results. So what could be the reasons for this?

1. It has always taken some time for new pages to show up in google search results – they will probably show up eventually.
2. My Page Rank is low because I haven’t put a lot of effort in to getting good quality links pointing to the site. Even if I had done this, what constitutes a “good” link is constantly changing as google’s algorithms get smarter, so I might be wasting my time in some cases. Also, there is some evidence that google now subjects such links to an “ageing delay” of several months, which means it is difficult to fix low pagerank overnight. With a low pagerank google may be putting a low priority on including my site.

In essence, experience tells me that this is still a waiting game. Looking at other sites I have responsibility for, they have got to, and stay at the top for a number of reasons:

1. They add new products quite often, so when search engine spiders visit, there is something new for them to store away
2. They offer great customer experiences and focus on 100% customer satisfaction. This means that they receive quality links to their site without even having to request them as their customers recommend their site to others through links posted on their own websites and internet forums.

So, what is the answer to the “how do I get google to keep coming back?” question? Well first of all, make sure your robots.txt file is set up to encourage it (there’s plenty of info on this out there, so I won’t explain here). Secondly, provide great content that is regularly updated. That’s the purpose of this weblog and I can’t believe how often the spiders come back to check it for something new. When it comes to content though, right it with the reader in mind, rather than the search engine spiders. If you write content that people actually are interested in, everything else should fall into place.

That’s the end of this article, which is something of a catch-up/gap filler I will admit, but is really intended as a way of getting myself back into the habit of writing regularly again.

I have some ideas of interesting topics I would like to share on this blog soon, but if anyone has requests of topics they would like to see covered, please feel free to drop me a line at mark at phpworks dot co dot uk.