Check your own web estate

The two problems, which are not the same

The first is an address you no longer use. An organisation moves, merges, rebrands or tidies up, and lets a domain go. Somebody buys it, because the inbound links and the search standing built over twenty years come with it. The address still carries your name to everyone who follows an old link, and nothing about the change of hands is visible to them.

The second is an address you still own, where somebody has put content on your site that you did not write. That is not a lapsed domain, it is an intrusion, and it is often invisible in a browser because the pages are shown to search engines rather than to visitors, or sit below the rendered end of a page where nobody scrolls.

The four steps below find both. Do all four, because each catches something the others miss.

Step one: list every address you have ever held

Not the ones you use. All of them. Most organisations cannot produce this list, and that is the root of the whole problem.

• Your registrar account, which shows what you currently pay for.
• Your DNS provider, which often still holds zones for domains nobody has renewed.
• Old letterheads, printed material, conference stands, vehicle livery and email signatures.
• Campaign and project microsites, usually registered by an agency on your behalf and rarely handed back.
• Addresses that arrived with a company you acquired, which keep their own renewal dates in somebody else’s account.
• Your email domains, including any you stopped using when you migrated.

Mark each one as in use, deliberately released, or unknown. The unknown column is the one that matters.

Step two: ask a search engine what it holds for you

Search engines index pages you do not know about. This is the fastest way to see them and it needs nothing but a browser.

Replace the domain with your own and run them one at a time. Use the operators’ own phrasing rather than the word casino, which is why “non gamstop” and “slot gacor” are on that list. GamStop is the British gambling self-exclusion scheme, so sites advertising themselves as outside it are aimed at people who have asked to be blocked. “Slot gacor”, “daftar” and “togel” are the Indonesian-market equivalents and turn up a different population entirely.

Run the first query on its own as well. You are looking for any page you do not recognise, in any language, on any subject.

Step three: look in Search Console

Google Search Console shows considerably more than a public search, including when a page was first seen and which pages are receiving traffic. It also lets you request removal quickly, which a member of the public cannot do. It needs you to prove you own the domain, which takes a few minutes and is worth doing once.

• Check the Pages report for anything your organisation did not publish.
• Check Users and permissions for accounts that should not be there. A domain that has changed hands is often verified by its new owner, and that is how they inherit your ranking history.
• Check the Security issues and Manual actions reports, which will tell you whether Google has already noticed.

Step four: open the addresses you no longer use

Take the unknown and released entries from step one and visit each in a browser. Do not rely on a glance at the home page, because several of these operations serve different content to different visitors.

• View the page source and look at where the outbound links actually go.
• Scroll to the bottom of the rendered page, and then keep going. Links are routinely placed below the point where anyone stops reading.
• Try the address with and without www, and request a path that should not exist, such as /zzz. A site that returns the home page for every path is running a cloaking script rather than a website.

What to do with what you find

If the address is still yours and the content is not. Treat it as a compromise of the site or its hosting rather than as a content problem, and have whoever maintains it look for pages and user accounts nobody created. Check whether the site has a public registration form, because several of the cases I have documented were not break-ins at all: somebody signed up through the organisation’s own form and published through it. Removing the pages is not the end of it, because they stay in the search index until they are removed there too.

If the address is no longer yours. You cannot have the content taken down, and in most circumstances nor can anyone else. What you can do is find every link still pointing at it from your own site and from your partners’ sites, and remove or redirect them, so that your organisation is not the thing sending people there. If the address matters enough it can sometimes be bought back, and that is a decision to take on its merits rather than in a hurry.

If what you find is illegal rather than merely unwanted. Material that appears to involve the sexual abuse of children goes to the Internet Watch Foundation, who have the legal authority to assess it. Do not forward it, screenshot it, or send it to a journalist, including me. Report it and let them look.

Then stop it happening again

Almost every organisation has a policy covering who may register a web address. Very few have anything covering what happens when one is given up, which is the moment the value walks out of the door. The Domain Release Clause is a short clause to add to a policy you already have. Six paragraphs, no new system, free to use.

Published under a Creative Commons Attribution 4.0 licence. Copy it, adapt it, put it on your intranet, send it to your suppliers. Attribution is appreciated and not required for internal use. If it finds you something I would be glad to know, at hello@jharbourne.com, though I am not able to investigate individual cases.