The crawlability of your site is vital for improving your indexation; however, it’s also the easiest part of SEO. Most sites that have a logical structure will have no problem being crawled by Google — after all, Google was built to be able to crawl sites. The vast majority of SEOs don’t have to worry about crawlability at all.
However, there are site designers out there who manage to do some truly spectacularly bone-headed things that prevent Google from seeing the site.
The Dear God Don’t List
Before we get into the stuff that you should be doing, here are the truly horrible things that you absolutely do not want to be doing. These are the spectacular SEO fails of site design. Unfortunately I have indeed seen sites that do each of these things — any one of them will prevent Google from crawling your site, which means you won’t ever show up in Google at all.
- Don’t design your site in Flash or within a single ajax frame. If you do this Google cannot see any of your content. There are technically ways to design a single page application (SPA) that doesn’t totally destroy your SEO, but even those are proven to do worse than a real site time after time.
- Don’t block all bots in your robots.txt file — doing so is explicitly telling Google not to crawl any page of your site.
- Don’t use the noindex tag on every page of your site — doing this explicitly tells Google not to index those pages.
Okay, got that out of the way. Now we can sit back and have a pleasant conversation about SEO and crawlability.