Despite the increasing use of search engine friendly URLs, custom meta tags and cleaner navigation, there’s a lot of web content out there that is inaccessible to the search engines.
Perhaps it is in the “too hard basket” for many designers. In which case, perhaps you need a new designer! After all (repeat after me) You Are the Customer.
There’s a nice article from SearchEnginePosition.com that discusses the topic: The Invisible Web Still Exists
Too many websites are not search engine friendly or are not properly optimized; in other words, are active members of the Invisible Web Club.
How do you get out of this club? I’m glad you asked:
- Rewrite “dynamic” URLs to remove question marks, ampersands, and equals signs from them
- Remove frames if you have them
- Don’t have links that rely on JavaScript to function
- Don’t embed navigation elements in Flash or Java
- Provide alternate link-based navigation to content that is behind search boxes or fill-in forms (that includes JavaScripted pulldown lists!)
SIGN UP FOR EXCLUSIVE WEEKLY CONTENT 
Charles, in general, search engine robots have a hard time indexing pages with too many dynamic variables, such as = ; + ; ? ; ids and so on. My client has a site with too many = signs (2 is too many for me) and Google never indexed these pages.
It’s so easy to do a mod rewrite that it’s really worth it.