WordPress robots.txt example for great SEO

The robots.txt file is a very powerful file if you’re working on a site’s SEO. At the same time, it also has to be used with care. It allows you to deny search engines access to certain files and folders, but that’s very often not what you want to do. Over the years, especially Google changed a lot in how it crawls the web, so old best practices are no longer valid. This post explains what the new best practices are for your WordPress robots.txt and why.

Google fully renders your site

No longer is Google the dumb little kid that just fetches your sites HTML and ignores your styling and JavaScript. It fetches everything and renders your pages completely. This means that when you deny Google access to your CSS or JavaScript files, it doesn’t like that at all. This post about Google Panda 4 shows an example of this.
The old best practices of having a robots.txt that blocks access to your wp-includes directory and your plugins directory are no longer valid. This is why, in WordPress 4.0, I opened the issue and wrote the patch to remove wp-includes/.* from the default WordPress robots.txt.
A lot of themes also use asynchronous JavaScript requests, so-called AJAX, to add content to the page. By default, WordPress used to block these. So I created the ticket for WordPress to allow Google to crawl the admin-ajax.php URL in wp-admin. This was fixed in WordPress 4.4.

Robots.txt denies links their value

Something else is very important to keep in mind. If you block a URL with your site’s robots.txt, search engines will not crawl those pages. This also means that they cannot distribute the link value pointing at those URLs. So if you have a section of your site that you’d rather not have showing in the search results, but does get a lot of links, don’t use the robots.txt file. Instead, use a robots meta tag with a value noindex, follow. This allows search engines to properly distribute the link value for those pages across your site.

Our WordPress robots.txt example

So, what should be in your WordPress robots.txt? Ours is very clean now. We no longer block anything! We don’t block our /wp-content/plugins/ directory, as plugins might output JavaScript or CSS that Google needs to render the page. We also do not block our /wp-includes/ directory, as the default JavaScripts that come with WordPress, which many themes use, come from these directories.
We also do not block our /wp-admin/ folder. The reason is simple: if you block it, but link to it somewhere by chance, people will still be able to do a simple [inurl:wp-admin] query in Google and find your site. This type of query is the type of query malicious hackers love to do. If you don’t do anything, WordPress has (by my doing) a robots meta x-http header on the admin pages that prevents search engines from showing these pages in the search results, a much cleaner solution.

What you should do with your robots.txt

You should log into Google Search Console and under Crawl → Fetch as Google, use the Fetch and Render option:

If it doesn’t look like what you’re seeing when you browse your site, or it throws errors or notices: fix them by removing the lines that block access to those URLs from your robots.txt file.

Should you link to your XML Sitemap from your robots.txt?

We’ve always felt linking to your XML sitemap from your robots.txt is a bit nonsense. You should be adding them manually to your Google and Bing Webmaster Tools and make sure you look at their feedback about your XML sitemap. This is the reason our Yoast SEO plugin doesn’t add it to your robots.txt. Don’t rely on them to find out about your XML sitemap through your robots.txt

How to use the content analysis of Yoast SEO

The Content Analysis Tool in the Yoast SEO plugin measures many aspects of the text you’re writing. These checks run real-time, so you’ll receive feedback while writing! The content analysis helps you to make your text SEO-friendly. In this post, I’ll first describe the most important features of the Content Analysis Tool. After that, I’ll explain how to use and interpret these features.

Most important features

1.  The plugin allows you to formulate a meta description. This description has to be a short text describing the main topic of the page. If the meta description contains the search term people use, the exact text will be shown by Google below your URL in the search results.
2.  The plugin analyzes the text you write. It calculates the Flesch reading-ease score, which indicates the readability of your article. The Flesch reading-ease score takes into account sentence length, for example. In the future, we’ll add more checks on readability. This will allow you to check the SEO and readability of your text simultaneously.
3. The plugin does numerous content checks on your page. It checks whether you use your focus keyword in:

• the title of the page;
• a heading;
• the URL;
• the content of the article;
• the meta description;
• the alt tag of the images.

The plugin also checks the presence of links and images in the article. It calculates the number of words and the density of usage of the focus keyword in the article. Moreover, the plugin checks whether you’re using the same focus keyword on other pages of your website. This should prevent you from competing with yourself.
If you write a relatively SEO-friendly text (based on the aspects mentioned above) the plugin will indicate this with a green bullet. Writing pages that are rewarded with green bullets will help you improve the ranking of those pages.

Want to optimize a text for multiple keywords, maybe for multiple synonyms? Yoast SEO Premium allows for multiple focus keywords! »

Two warnings before you start!

When you optimize your post for a certain keyword, keep two things in mind:

• The first thing is that in this phase (the final, optimizing phase) you shouldn’t change any major things in your article. If you’ve put effort into writing an attractive, structured and readable text, the optimization process should in no way jeopardize that.
• The second thing is that you shouldn’t change your keyword strategy in this phase. If you’ve done your keyword research properly and you’ve written your post or your article with a focus keyword in mind, don’t go change your focus keyword now! Read The temptation of the green bullet for more in-depth information about that.

7 simple steps to optimize your text

Step 1: Put your text in the WordPress backend

Distraction free writing

WordPress has a distraction-free writing mode that enables you to write in the WordPress backend without being distracted by the menu, the toolbar, the categories box, etc.

You’ve written your article or your blog post. You can write directly in the backend of WordPress or write in any kind of text editor and copy your text into the WordPress backend. Do whatever you like!
If you choose to copy your text in the WordPress backend, copy without the layout. You should adapt the layout in the backend, as otherwise you might run into some layout problems. Make sure to set subheadings into heading 2, sub-subheadings to heading 3 and so on. Then put the title of your post in the title box.

Step 2: Enter your focus keyword

Scroll down to the Content Analysis Tool in the WordPress backend. Enter your focus keyword in the appropriate field of the Yoast SEO Metabox. Your focus keyword is the keyword you would like your post to rank for. Ideally, this should be a keyword which emerged from your keyword research and which you have kept in mind during the entire writing process.

Yoast SEO premium offers the possibility to optimize one article for more than one focus keyword. Optimizing your post for more than one search term allows you to rank for more keywords and to gain traffic to your site through more keywords.

Step 3: Write a meta description

Enter the meta description of your post. Describe clearly what your post or article is about. And make sure you use the exact phrase of your focus keyword. The meta description will be shown by Google below the URL if people search for your focus keyword.

It’s important that the meta description contains the focus keyword. Not because it will improve your rankings, but because otherwise Google usually won’t show your meta description in the search results. Google will try to match the search query with the description. If the focus keyword isn’t mentioned in the meta description, Google will just grab a random piece of content from your page containing the keyword.
The meta description shouldn’t be too long. On the other hand, there’s no ‘penalty’ for having too long meta descriptions either. What you should pay attention to is: 1. the logical bits of it are of the right length and, 2. when it’s cut in half, it still makes sense and still entices people to click.

Step 4: Fine-tune your headings

Look critically at your title, the headings and subheadings of your article. Do these contain your focus keyword? If not, can you alter them (without changing the structure or content of your article) in such a way that they will contain your focus keyword? Don’t put your focus keyword in all of your headings though! That is too much. Using your focus keyword in one heading and in your title should be enough. You can read more about headings in one of Michiel’s posts.

Step 5: Fine-tune your body text

You should also mention the focus keyword in your text a couple of times. Make sure to mention it in the first paragraph. Throughout the text, you should mention it again. As a general rule of thumb: try to use your search terms in about 1 to 2 percent of your text. Say your article has 300 words, that means you should mention your search terms 3 to 6 times. 300 words isn’t the exact goal, nor is the amount of keyword mentions. However, 300 is a decent minimum for the number of words of an article that needs to show authority.

Step 6: Check your bullets!

Clicking on the Content Analysis tab will allow you to see which aspects of the search engine optimization process were successful. The green bullets show which aspects are good. Orange and red bullets indicate where you can improve your SEO strategy. You don’t have to keep on optimizing until all of the bullets are green. Posts on Yoast.com, often have a few orange bullets and sometimes even one or two red bullets. The important thing is that the overall bullet (the one on the upper right in the backend of your post) should be green. The overall bullet will become green if the majority of your SEO aspects are covered.

Overall SEO score in the publish box

Want to know what your posts will look like when shared on Facebook or Twitter? Yoast SEO Premium now has Social Previews! »

Step 7: Fill out the Social data

The final step to take in the Yoast SEO meta box is filling the Social data. If you fill out a description or title for a social network on this tab, it’s shown in the metadata for the page. This means this description, title or image will be shown when the page is shared on the respective social network. These descriptions basically have the same requirement as the meta description (which is what they fall back to), but usually can be longer. They should tell people what to expect and why they should click.

Social previews

The preview screenshots on the left are taken from Yoast SEO Premium, in free, you won’t see those previews, just the fields!

  

Keyword Research Tool

Earlier, I wrote a post about the use of long tail keywords. Focusing on long tail keywords could be a good strategy, especially when trying to rank in a highly competitive market. But how do you decide on which (long tail) keywords you want to rank? This post will give you some handy tips and keyword research tools to make your keyword research a bit easier.

Repeat keyword research regularly!

Keyword research is an activity you undertake every now and then. If you have a clear definition about the product or service you want to ‘sell’ with your website, you should be able to come up with keywords, related keywords and even more related keywords to make your awesome website (more) findable. As your product and the market will evolve, your keyword strategy should do the same.

Resemble the vocabulary of your audience

The keywords you want to focus your SEO on, should closely resemble the vocabulary of your audience. In order to come up with the proper keywords you really have to get inside the heads of the people who search for your website. What terms will people use? How do people search? Which question does your website answer? You should create a list of all search terms people could use and think of combinations and nuances within these search terms.

Keyword research tools to use

Making a list remains hard. And up until a few years ago, doing your keyword research was much easier. You could simply check Google Analytics to see on which terms people found your website. That is no longer possible. So you’re pretty much left in the dark about the terms people use in search engines to end up at your website. Luckily, there are some other tools which can make your keyword research a bit easier:

Google Adwords Keyword Planner

Use the Google Adwords Keyword Planner to find new and related keywords, but ignore the search volume data! The search volume data in the planner is really only useful for keywords that you’re actually spending money to advertise on. Otherwise, these volumes are not reliable. While not really helpful to decide which keyword is most used by your potential audience, Google Adwords Keyword Planner makes a useful tool in coming up with ideas for potential keywords!

Yoast suggests

Joost developed his own keyword research tool to come up with keywords as well! Yoast Suggests uses the Google Suggest functionality you know from searching in Google. It finds the keyword expansions Google gives and then requests more of them. So if you type ‘example‘, it’ll also give you the expansions for ‘example a…’ till ‘example z…’ etc. Just go on and try it and fill out some of your potential keywords. It’s a great way to quickly find more long tail keywords you can focus on.

Google Trends

Google Trends allows you to compare the traffic for sets of keywords. You can even see the difference for numerous geographical regions. It’s very important to check Google Trends if you expect that some of your keywords are seasonal, for instance due to regulations, holiday seasons etc.

Your internal search engine

What are people looking for on your site? These terms are keywords in the vocabulary of your actual audience and should definitely be added to your keyword list. Do not forget to look at the keywords people filled in that didn’t get any results: this was stuff people were expecting but didn’t find. You can look into the results of your internal search engine with our Google Analytics for WordPress plugin. 

Conclusion

These keyword research tools should make it easier to create a list of relevant search terms. You should make sure to create awesome landing pages for keywords you want to be found on. You should also think about cornerstone content articles and a great internal linking structure in order to make your SEO strategy complete. In our Content SEO eBook we dive munch deeper into keywords, landing pages and long tail keywords.

How to choose the perfect focus keyword

Adding quality content to your website on a regular basis is a very good SEO tactic. Google sees that your website is active because new pieces of information are added. On top of that, you increase the volume of your content. If your keyword strategy has been crafted properly and your content is nicely optimized for the right keywords, adding content will increase your findability. But how to choose the perfect focus keyword? Especially when you’re blogging and focussing on long tail keywords, it can be quite hard to decide which keywords to optimize for. In our view, there are at least three things you should do before writing an amazing (high quality) text.

Choose a focus keyword that is used to search

Your keyword strategy should have given you some idea what you want to write about. For blog posts, you will usually aim for a long tail keyword (containing multiple words). In our Yoast SEO plugin you will find a drop down menu which supplements your entry with suggestions. These suggestions are actually based on Google Suggest. This is exactly the same as what you’d see when you type the search term into a Google search box.
Update August 2015: unfortunately we had to drop the suggestion drop down of our Yoast plugin, since Google shut down the autocomplete API that was needed for this feature.
The terms you will find in the suggest drop down menu are thus terms and combinations of words that are logical and used by actual people to search on the web. This tool can be very helpful in giving you some first ideas about the search behavior of people in the area you want to write about. Our Yoast Suggest tool uses the same data to find the first 10 keywords and then expands on that. Also, Marieke has recently written about more keyword research tools.
When we searched for the term “focus keyword” in February of last year, this was the suggest output:

Right now, the output shows this:

Suggest changes based on the problems people have, so monitoring it for important keywords makes sense. This particular new result gave us some extra input for this post.

Discover some information about search volume

Once you have found a long tail search term you would like to start ranking for, you should put some effort into discovering whether or not the search volume of your chosen focus keyword is high. We will be the first to admit, Google has made this really hard. The only way to know ‘for sure’ how often a search term is used, is by having an active and alive AdWords account and by bidding on the search term of your choice. We understand this is a bit too difficult and expensive for most of you (we honestly hardly ever do this).
Not to worry, using Google Trends should give at least some idea, in a creative way, about search volume. Google Trends allows you to compare the search volume between two search terms over time. This will give some insights in the volume of the search terms people use (always relative to another term).
If you already have some (blog)posts that rank well for good terms, you will know how many visitors these posts attract. Using Google Trends to compare the focus keywords of older posts (of which at least the number of visitors to your website is known) with the focus keyword you have in mind for your new post, could give you some idea about the potential of traffic this new focus keyword could have. Make sure to choose older posts that are most similar to the post you’re planning to write: if you’re planning to choose a long tail keyword, compare posts with long tail focus keywords as well.
For instance, this post about focus keywords could be compared to a post about snippet previews, a very related feature of the WordPress SEO plugin we wrote about before:

As you can see the traffic is comparable, we know the search traffic to our snippet preview post is reasonably good, so we know it’s worth optimizing for.
Using Google Trends to compare between your old focus keywords and the one you would like to choose will give you some insights about the prospects for your focus keyword.
Another way to use Google Trends is when you are doubting between a number of (long tail) focus keywords. Google Trends will easily show you what search term will have the highest search volume (compared to another). Google Trends will help you decide which long tail keyword is most common in the search engines.

Google your proposed focus keyword!

Apart from knowing which search terms are actually used by people, you need to know whether or not your idea for your post or page fits the desires and expectations of the people who use the search terms. The best way to find out whether or not your content fits these desires is to Google your proposed (sets of) keywords yourself.
Take the time to look at the first two result pages. Are the articles Google shows of the same character that your article will be like? Does your website fit between the results shown in these result pages? If you decide to write your blogpost or page, while optimizing for this exact focus keyword, you are aiming for a display of your page amongst these.
For instance, when we wrote this post and Googled our chosen focus keyword, we saw we’d be competing with ourselves:

We also saw lots of questions on the WordPress forums, giving us all the more reason to write this post.
Note that we looked at the old post and decided it wasn’t good and complete enough, so we decided we would delete it and replace it with this one.
Make sure to use the content of the result pages as an inspiration for your blogpost. Are there any useful ideas (we are NOT encouraging to copy content, merely to see whether you perhaps missed some information or arguments for your own blogpost)? But more importantly: how can you make sure your post will stand out? In what way could your post be better, funnier, more original than the post presently displayed in the result pages. Try to think of content that will make the audience click and share!

 “Should every page have a focus keyword?”
We often get this question: should our about page or our contact page have a focus keyword? What should it be?
The answer is easy: not every page needs a focus keyword. Your contact page should be easily reachable, it might for instance need to rank for “<company name> address”. That probably doesn’t make sense as a focus keyword though and it’s perfectly fine to leave it empty.

Conclusion: picking a focus keyword is not easy

Choosing a perfect focus keyword is not an exact science. You should aim for a combination of words that are actually used by a search audience. Aim for a focus keyword that is relatively high on volume and aim for one that will fit your audience.

10 tips for an awesome and SEO-friendly blog post

Writing a blog post -like all other writing- is a skill. To keep your reader interested, you should think about structuring your text and writing in an appealing style. You can help your readers to grasp the main idea of your post by providing headings, subheadings and clear paragraphs. If people understand and like your text, they are much more inclined to share, like, tweet and link to your post. And that will increase your rankings! So, in order to improve your ranking in Google, you should definitely try to maximize your writing skills!
For some, writing for SEO purposes and writing to attract and keep attracting your audience could appear as two contradictory goals. I totally disagree with this. Indeed, if you not only want a good but also an SEO-friendly blog post, the words you want to be found on should be in a very prominent place. But, using your keywords too often severely damages the readability of your text. So, you definitely shouldn’t do that!
In this post, I would like to give some tips on writing blog posts that are both very readable as well as SEO-friendly. I genuinely think those two goals should (and easily can!) go hand in hand.

Elementary writing tips for good blog posts

Before anything, your blog post just has to be a good piece of writing! A lot of bloggers just begin to write when creating a new blog post. They just type what comes to mind. For some, this may be sufficient, because they are natural writing talents. Others might need some help. I always follow the next set of ‘rules’ myself.

1. Think before you write!

Think hard about the message of your text. What do you want to tell your readers? And what’s the purpose of your text? What do you want your readers to do at the end of the page? Write down the answers to these questions before you begin writing.

2. Write down the structure of your blog post.

Start your post with creating a clear structure. Every post should have some sort of introduction (in which you introduce your topic), a body (in which the main message is written) and a conclusion (which should summarize the most important ideas or deduce some new idea). Write down what you want to write in all these three sections. You now have a kind of summary of your post. The real writing can begin!

3. Use paragraphs.

Everybody uses paragraphs, but make sure to use paragraphs that make sense. Do not start a new sentence on a new line, just because it looks nice. There should be a reason for making a new paragraph. Every paragraph should have a main idea or a main subject. Ask yourself what the main idea of each paragraph is. You should be able to grasp that main idea in only one sentence. If you need more sentences, you simply need more paragraphs!

4. Use headings.

Headings structure the entire page, so you should use headings. They’re important for readability, but for SEO as well. Headings help Google to grasp the main topics of a long post and therefore can help in your ranking. If you want people to find their way in your articles, you should use subheadings. Subheadings will lead people, help them scan your page, and clarify the structure of your articles. Make sure you use your keywords in some subheadings. Not in each and everyone of them though, as it will make the text unreadable.

5. Use signal words.

Signal words help people to scan through your text and help people to grasp the main idea. Let’s say, for instance, that there are three reasons for people to buy your product. You should use signal words like: ‘first of all’; ‘secondly’ and ‘finally’. Also, words as ‘nevertheless’, ‘surely’ and ‘indeed’ give a clear signal to your readers. Readers will instantly get that a conclusion will follow after words like ‘consequently’, ’so’ or ‘for this reason’. Signal words are therefore very important to structure your text.

6. Let other people read your post.

Before publishing your post, let someone else read your post first. Ask him/her whether he understands the main idea of your post. Correct typos and sentences that aren’t formulated correctly.

7. Optimize the length of your article.

Make sure your articles have a minimum of 300 words. Google likes long articles, if your article is too long though it might scare away users. So try to stop at around 700 words. And, as a general rule of thumb: try to put your search terms in about 1 to 2 percent of your text. So in an article of 300 words, you should mention your search terms 3 to 6 times.

8. Link to previous content.

If you already wrote some content about the topic of your current post, don’t forget to link to these posts. It will make your post stronger because you show some authority on the subject. Next to that, your link structure is important for your ranking in Google as well. You should read Joost his post about cornerstone articles if you want to read more about this.

9. Add content regularly.

Adding actual and functional information to your website will give Google the idea that your website is alive. If it’s not an active website, Google will crawl it less often and it might become less appealing to Google to include the page in the search results.

10. Use our Yoast SEO plugin.

Our Yoast SEO plugin helps you write an SEO-friendly blog post. You start by choosing your focus keyword. This is the most important search term you want people to find this particular page for. Our plugin checks your post to see whether you use the keyword in the right places, and it measures many other aspects of the text. These are the most important ones:

• The plugin allows you to formulate a meta description.
• The plugin analyzes the text you write. It calculates a Flesch reading ease score, which indicates the readability of your article.
• It checks whether or not you used your keyword in 5 important locations: the article-heading, the title of the page, the URL of the page, the content of the article and the meta-description.
• The plugin also checks the presence of links in your article and the presence of images in the article.
• It calculates the number of words and the density of usage of the focus keyword in the article.
• Above that, the plugin also checks whether or not other pages on your website use the same focus keyword, to prevent you from competing with yourself.

If you write a relatively SEO-friendly blog post (based on the aspects mentioned before) the plugin will indicate this with a green bullet. Writing pages with green bullets will help you improve the ranking of the pages on your website.
Note that not every dot has to be green for the overall score to be “good”. For instance, these are the results of this post, which does have a “good” score:

If you’re not on WordPress, we also have this Page Analysis tool available on our site. It’s still in beta, so we’d love your feedback!

Conclusion

The era in which some SEO tricks were sufficient to get your website to rank high in Google has long ended. Nowadays, good content has the highest likelihood to result in a higher positions in Google. And good content also leads to more Facebook likes and shares, tweets and return visitors to your website. Of course, you can do some extra things to maximize the SEO friendliness of your post, but most important is: just write a very, very good post!