March 21 2014

Robots in Disguise: 5 Creative Uses of Robots.txt and Source Code

Today’s post is a light-hearted story about two common playgrounds for SEO tomfoolery; Source Code and Robots.txt. We’re going to look at some of the most creative uses of both and how, in some cases, they’ve led to some successful link acquisition!

Robots dot what?

Okay, so let’s first cover off exactly what we mean by robots.txt and not just assume everyone knows what it is. Google’s explanation of a robots.txt file is pretty detailed.

“A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password-protecting confidential information.)”

The robots.txt file is essentially the polite gatekeeper of your website that deals with bots requesting access. Unfortunately, polite gatekeepers can be bypassed more easily and this is quite often the case when it comes to different tools and crawlers sidestepping this particular minder.

Sorting the Wheat from the Chaff

One of the best uses of robots.txt we’ve seen for some time was by the MailOnline. Robots.txt is something often observed by SEOs and the bods at the MailOnline had obviously decided that they wanted to hire someone who had real SEO chops. Naturally, they thought the best way to find such a person was to advertise the job opportunity within their robots.txt file.

The Daily Mail take a different approach to recruitment.

Not only did the MailOnline sort the wheat from the chaff in terms of SEO applicants, they also bagged themselves a few links on the way.

Number of Links: 142

Linking Root Domains: 52

Obviously a few decent domains from the Tech and SEO industries;

All in all, a good effort!

 

Good Advice

Our second example is from Last FM, which instructed its robots.txt file to carry out some pretty essential orders. http://www.lastfm.com/robots.txt.

Last FM making sure their robots follow orders!

Unfortunately, these directories don’t seem to exist on the domain, and though it’s definitely good advice for any robot to heed, this effort didn’t receive much in the way of link love.

Robot Revolution

YouTube has joined the robot revolution with some jovial attempts of its own. Deserved or not, robots seem to have a bad reputation and there’s a slight Terminator feel to this one. https://www.youtube.com/robots.txt

YouTube Terminator Robots.txt

Links-wise we’re looking at the following;

Number of Links: 26

Linking Root Domains: 26

Source Code: HTML Playground

It’s not just robots.txt that gets used as a developer/SEO playground; there are plenty of opportunities to use source code to show some creativity – and again, generate some quality links. The following is a selection of domains that have done just that.

Spot the Wren

Wren Living decided to take their branding to the next by adding their beloved Wren to their source code.

The Mighty Wren

 

Even the US President is doing it…

The World’s most important man also recognises the potential link opportunities from this HTML trick. You’ll also notice the not-so-subtle use of only three letters in this particular example. http://www.barackobama.com

The President’s Source Code

 

Over to You…

Have you seen any other brands or personalities dabbling in their robots.txt or source code? If so let us know in the comments below – can anyone beat the MailOnline’s haul of links for their job advertisement?

You may also like …

August 26 2024

The Importance of UX in Your Creative Strategy

At Mediaworks, we recognise that exceptional UX is not just an addition to your strategy but a pillar of your digital brand. So, why should UX be embedded into your creative strategy?
August 12 2024

Bridging the Gap Between Web Development and Creative Design: Overcoming Collaboration Challenges

Integrating creative design with web development is key for stellar web experiences. Mediaworks explains how to foster a collaborative approach. Read on.