The search functionality on the Michigan Tech website is powered by Google. It works the same way as a search on google.com, except it only searches within the mtu.edu domain, subdomains, and sites that we manually tell Google are also owned by Michigan Tech (such as superiorideas.org or michigantechhuskies.com).
Crawling, Indexing, and Ranking
In order for webpages to show up in search results, they must be crawled by the search engine’s bot. The bot navigates pages it has already crawled and follows links to find new pages. The new pages found are added to an index that the search engine pulls results from.
For any given webpage, that crawl by Google’s bot will take place between 4 and 30 days, with more popular or frequently updated pages being crawled more often. Because the bot finds new pages using links from pages it already knows about, pages that are not linked anywhere on the Internet are not crawled, and thus not available in search results. Likewise, password-protected pages are not able to be crawled, so they aren’t available in search results.
Once a new webpage is indexed, Google must determine how to rank it in search results. An initial ranking typically starts out low and, over time, may rise as Google tries to understand how useful the webpage is to end users.
Removing Search Results
When you remove a page or file from the internet (recycle in Modern Campus CMS) or change a URL (rename or move files or folders), the search engine will still have the old information in the index to show in search results. The next time the bot crawls the URL it will find the 404 error, but will not remove the search result in case there is a temporary problem. It will take at least a few crawls for the bot to confirm that the page is really gone before it will be removed from the index/results, which could take anywhere from a few weeks to a few months.
This is another reason that redirects are so important! By putting a redirect in place any time a URL changes or is removed, users who come across the link in search results will be directed to the correct location and the search engine bot will see that the URL is redirected (not just broken) and update the index sooner.
While we are not able to immediately change or manipulate search results on mtu.edu or google.com, using best practices can help speed up the updating process. Digital Services does a lot of technical SEO to help keep bot crawling moving quickly. Keeping your CMS and non-CMS websites lightweight (remove unnecessary and legacy webpages or websites) helps as well.