A search engine works simply by crawling or spidering the internet, indexing every digital document it finds in the process and storing them in a database.
- Crawling
- spidering
- indexing
Sounds pretty complicated?
Relax! It's simple.
SEO is a combination of art and science. And there are no hard and fast rules to it.
But that DOESN'T make it rocket science. So don't let that scare you.
Although, your job is not to study and know everything about the search engines. But that still doesn't mean you shouldn't learn the basics that apply to you. Like finding answers to questions like...
What’s the first step in the search engine optimization process for a website?
As an SEO, you need to focus on the competitive landscape happening on SERPs - to identify opportunities and increase your site's visibility.
Why learning a few things about search engines is important
I know many people try to ignore this part, and they struggle to rank their websites on the first page of SERP.
Search engines constantly change and update how they present information on their result pages to provide a better search experience to their users.
And those updates sometimes affect some websites.
Plus, you should know how search engines:
- Crawl the entire web?
- Extract information from your website?
- Store the information in its databases?
- Apply the search engine algorithms?
- Republish the information when you search?
I'm going to walk you through these step by step.
We'll take Google as an example but, whatever you learn here, applies to all search engines.
And remember, this does not mean you can use the same approach to rank first on all search engines. Their algorithms work differently, and when it comes to ranking factors, it may look general, but the weight focus is not the same. More on this later on.
Crawling/Spidering and Indexing.
These two basic processes are the core functionality of search engines.
Although there are some processes between these two ends (crawling and indexing), such as Rendering, which can be something difficult on JavaScript websites for crawlers.
Crawling or Spidering
It is the process of finding new information, additional information, or existing information updated on the web and storing them in a database.
A web crawler or spider is a software program that systematically browses the world wide web by following links from one web page to the other - and discovers new web pages in the process.
Indexing
Is storing the information returned from the process of crawling and organizing them in a database.
What happens when I type a query into Google?
When you enter a query into the search bar of Google, you might be thinking that Google has gone to search the whole internet for your search terms.
I used to think that too… But that's not how it works.
Google crawls to discover new or updated web pages to add to its databases.
When this information is returned, during the process of indexing, Google then analyzes the content to know what it's about and catalogues them based on the document type:
- Webpages
- PDFs
- Images
- Videos
- Files
Google does this from time to time.
So that when you enter a query, it just goes to its database and fetches the information right from there.
How does Google know the correct answer for my query?
When you submit a query, Google applies many factors to determine the most relevant and best answer to your question.
Since it now has updated resources in its storeroom (database), it is easier to apply a set of algorithms to determine the most relevant and best answer to your query.
Over the years, Google has stored and analyzed vast amounts of data in its core processor, and when you search for something, it already knows all related information to your query and then brings it to you.
However, this process is a continuous thing, and Google is improving every day with machine learning to deliver the best results.
Learn more about how Google's search engine works from Google itself.
There are many factors - algorithm factors that play into this. You might have heard someone saying "ranking factors," right? That's what we're going to discuss in our next article.
So that at the end you look at search engines process as:
- Crawling - indexing - serving/ranking
A lot of people will stop talking at the crawling and indexing level. But Google doesn't stop there. After you clicked on any of the results, it returned; it still tries to know whether you're pleased with the result or not.
Why does Google care?
Google wants its user's problems solved.
Many people try to deceive search engines with their tricks, typically with what we usually refer to as black hat in SEO, but Google is more clever.
If a user lands on your website and hates what he/she sees - and leaves your webpage unsatisfied, Google knows!
So when you're Optimizing your web pages, don't try to circumvent any of their guidelines.
All their algorithms are based on human factors - when visitors like your website, Google loves you and rewards you.
What do search engines want from me?
The saying that Google is biased or hates a particular website is a myth. You don't want to believe that!
Every search engine has its guidelines that they want you to follow. And these guidelines are written to satisfy their users - who're their primary customers.
In the case of Google, it's popularly referred to as a Google Webmaster Guidelines. I recommend that you read it. It's about 200 pages, but you don't have to read from page to page.
And once you think human first, build your website, services, or products around your customers - and deliver a quality user experience, you win!
Providing valuable and relevant answers to searchers' questions in the most helpful format is what search engines want.
So take this from me…
If you take the time to understand the objectives and goals of these search engines, you'll not have problems ranking your website.
How Search Engines Discover New Webpages
Search engines find new websites and pages by following links from pages that already exist in the database.
This means, if you want search engines to crawl your newly launched website, you may have to get links from a page that's already existing and has been indexed by any of the search engines.
Or, submit a Sitemap instructing Google to crawl the URLs of your website.
Here's how it works...
Take Google bot, for instance; it updates its database from time to time to keep supplying the users with relevant and up-to-date information.
During this process, the crawler can find your new website by following links from other pages and then adding them to the database.
This is where link building comes in. And is the reason why you hear that getting "links back" (back-links) to your website will improve your SEO scores.
Backlinks are an essential ranking factor.
What if I don't find any website to link back to me?
Search engines like Google, Bing, Yandex provide web admins with an option to submit a list of pages (a Sitemap) on their website for them to crawl.
A sitemap is an XML file containing a list of all the pages on your website, which you can submit to search engines to assist them in crawling your website and indexing it quickly.
Pro tips: If your website has less than 1,000 pages, all you need is to submit the homepage URL (e.g., https://example.com) to Google. And with proper internal linking of your web pages, Google will reach to crawl and index even the deepest pages on your site.
How do I know my site is indexed?
You've done all you can by getting a link back to your website or submitting a sitemap to Google, for instance, but you aren't sure whether your page is indexed in their database.
Here's what you need to do…
Head over to the Google search bar and enter the following query - putting your domain name in "site:yourdomain.com."
This will return all the pages on your website.
You can then check to see if any page is missing. In case of a missing page, you might want to check your Robot.txt file - if you have one - at the root directory of your website, or visit the search engine console and submit the URL of the missing page.
What if I don't want the search engine to index a particular page on my website?
There are times you might want to prevent certain web pages from being indexed. All you need to do is disallow it using a robot.txt I mentioned earlier.
I know you might be wondering what a robot.txt file is.
It's a file that is located in the root directory of a website. It tells search engines which parts of your site it should and shouldn't crawl.
To know whether your website has one or not, use this format to verify; "yourdomain.com/robots.txt."
Google will crawl your site usually, even if you don't have a robots.txt file on your website.
Further reading: Everything you need to know about robots.txt file.
Top 6 search engines in 2021
- Bing
- Yahoo
- Yandex
- Baidu
- DuckDuckGo
Like I said earlier, the basic principle behind when you enter any query on a search engine, and it then returns some millions of results for your question works around:
- Crawling/spidering - going out to find documents
- Indexing - storing into database
- Ranking - applying algorithms
The difference is in how these algorithms are applied in their index. The most important ranking factor in Google may not be as crucial in Bing.
For instance, Bing places more weight on social signals - signals from your social media handles than backlinks which is one of the most vital ranking factors on Google.
Google currently does not emphasize social signals - but expects that to change in the coming years.
Before optimizing for any of these search engines, my advice is to make sure you check out their guidelines.
We're about to ramp up. But before we do, let's take a look at the aim and goals of these search engines.
You'll be surprised that they have different goals. And it's critical to your SEO success.
Google Search Engine
Google's mission is to "organize the world's information and make it universally accessible and useful."
Bing Microsoft Search Engine
Bing's mission is to help you search less and do more - constantly looking for ways to make your search experience more efficient.
Yahoo Search Engine
Yahoo's mission is to make the world's daily habits inspiring and entertaining even while you search through their engine.
Baidu (China) Search Engine
If you're looking to rank your website in China's market, consider Baidu. China's internet space is rising. And as of 2016, they have over 730 million internet users - the largest in the world.
Baidu's goal is to connect users to relevant information online, including web pages, news, images, documents, multimedia files, and services, through links provided on their website, apps, and skills store, as well as native-app-like experiences via our innovative mini program.
Yandex (Russia) Search Engine
The term "google" might have become completely synonymous with "internet search" and a few others like Yahoo or Bing – you might be thinking are the only search engines.
But one search engine that's playing a significant role in the online landscape is the Russian-founded Yandex. Yandex's innovation is not only in search algorithms but also in AI, analytics, and web development.
If you want to rank for search terms in Russia and other neighbouring countries like Ukraine, consider Yandex.
Founded in 1997, the goal of Yandex is to help its users navigate both the online and offline world.
DuckDuckGo (Privacy) Search Engine
While privacy is a concern in Google, DuckDuckGo pride itself on protecting your data from being tracked gives you a secured online browsing experience such that whatever you search for it's your own business. Everything is private to you.
Founded in 2008 with over 1.8 billion searches per month, DuckDuckGo is a search engine you might want to try if you want some search privacy.
The goal is to provide a better search experience across topics and interests, continents and countries, languages, and devices by working together to ensure every contributor's individual interests/experience can be represented in their search engine.
"Our community's mission is to provide a better search experience across topics and interests, continents and countries, languages and devices, by working together to ensure every contributor's individual interests/experience can be represented in our search engine."
Summing it up on How Search Engines Work...
You can now agree with me that there's nothing rocket science in this. As we proceed, we shall come across more technical terms, but at least you can put your mind at rest now that the rest will not be any more difficult.
Here's a recap of what we have learned so far…
- Search engines work by crawling and indexing web pages and other documents in their databases.
- Crawling is finding new web pages by following links from one page to the other.
- Indexing is storing the information returned from crawling into the database and cataloging them.
- The ranking is done by applying a set of algorithms to match any submitted query.
- The order of ranking is usually based on relevancy, quality, and freshness.
- Google wants to satisfy its customers and so expect you to have the users in mind.
- To rank on Google, you need to understand the ranking factors.
You must get this right… and I want you to.
So next in our discussion is Google Ranking Factors.
Got questions? Feel free to drop it in the comment below.