This is usually a troublesome state of affairs to treatment if you find yourself unable to seek out any earlier sitemap.xml recordsdata or older Screaming Frog crawls.

If some pages had excessive visitors, gross sales, or lead era worth, then they might be misplaced altogether. If some pages had a excessive whole of inbound hyperlinks, then the worth of these hyperlinks — measured in PageRank, Hyperlink Fairness, Belief Move, and so forth. — can be misplaced solely too.

With out full data of the web site’s former web site construction and the URLs inside it, there might be loads of worth misplaced to dead-end 404 pages.

Having run into the identical state of affairs ourselves lately, we had to determine — because of a big serving to hand from Liam Delahunty. Thanks, Liam! — an answer which we’d now wish to go on to you.

Utilizing Archive.org Knowledge

Archive.org, or the Wayback Machine because it’s extra generally know, is an internet crawler and indexing system for the web’s net pages for historic archiving. It’s a cool device which permits us to take a peek at what Google seemed like when it was nonetheless in Beta again in 1998, for instance.

Because it crawls a big proportion of the web, it’s extremely probably that your web site has been crawled by their net crawler. By retrieving this publically out there information, we are able to piece collectively a tough concept what the pre-migrated web site’s web site construction might have been.

The information is freely out there to make use of, and Archive.org have a short define of how the API could also be accessed and used out there right here.

Not being an API-wielding specialist myself, within the following course of, I’ll be falling again on a basic copy-and-paste strategy which Search Engine Optimisation Specialists of any talent stage can use.

How To Extract Previous URLs from Archive.org

1. Find your web site’s JSON or TXT file

Begin by navigating to the next URL, altering the holding root area to your web site’s personal root.

For JSON file:
http://net.archive.org/cdx/search/cdx?url=instance.com*&output=json

For TXT format:
http://net.archive.org/cdx/search/cdx?url=instance.com*&output=txt

If you could restrict the timeframe of the crawl, then you may add the next parameters to the top to slim the vary.

yyyyMMddhhmmss

Instance:
http://net.archive.org/cdx/search/cdx?url=instance.com*&output=txt&from=2010&to=2018

You too can lower or enhance the restrict to match your wants.

Instance:
http://net.archive.org/cdx/search/cdx?url=instance.com*&output=txt&restrict=999999

You’ll find a full rundown of the out there filtering choices right here:
https://github.com/internetarchive/wayback/tree/grasp/wayback-cdx-server#filtering

2. Paste into your spreadsheet and separate into columns

Copy all the textual content of the loaded web page and paste the outcomes right into a spreadsheet. On this occasion, we’re utilizing Google Sheets.

Choose all the vary of knowledge and use the “Cut up textual content into columns…” possibility of the “Knowledge” menu within the toolbar. As we’re utilizing the TXT formatting, we use the “area” delimiter to separate our information.

3. Take away columns leaving solely the URLs

Delete all the unrequired columns to go away solely the URLs. This can normally be Column C.

4. Use Discover and Exchange to take away:80 from URLs

Choose the column of URLs and use the “Discover and Exchange” operate to find the textual content “:80” and change it with nothing (depart the alternative textual content field empty). This can tidy up all the URLs, typically eradicating tens of 1000’s of situations of “:80”

5. Use =UNIQUE method to take away duplicates

In a separate column use the UNIQUE method — i.e, =UNIQUE(A:A) — to take away the duplicates from the primary column, leaving solely singular URLs to verify for 3XX, 4XX, and 5XX standing codes.

6. Crawl URLs utilizing Screaming Frog and extract report for evaluate

Copy your ultimate listing of URLs, open Screaming Frog and swap it to Record mode, then paste in your gathered URLs.

Export your accomplished crawl as a CSV and replica/paste the info into one other tab of your spreadsheet. At this level, you may both take away all columns apart from the URL and Standing Code columns, or you are able to do a VLOOKUP to populate the correlating statuses on your unique listing.

Now you can filter this entire listing of URLs to seek out 404 pages or redirect chains.

Different benefits and suggestions

This course of may be enhanced additional by gathering URLs by way of Google Analytics for way back to you may, ensuring to verify any former URLs which can have been excessive visitors or excessive changing gross sales pages previously.

Taking it one other step additional, you could find extra URLs by way of different net crawlers reminiscent of Majestic — which additionally retains a log of URLs crawled — which you may as well obtain and add to your whole listing earlier than eradicating all of the duplicates and crawling them.

Moreover essential is to run this listing of URLs by way of a device like Majestic to see whether or not there are any backlinks to the pages with 3XX, 4XX, and 5XX standing codes the place hyperlink fairness could also be subtle or misplaced solely.

This course of can be used for link-building. By following the identical course of on your competitor’s web sites, you might discover pages with 4XX standing codes with backlinks to them. You may use the Wayback Machine to see what these pages was once after which recreate and enhance on their outdated content material — with out copying something of their unique — earlier than reaching out to the linking domains to recommend your new content material as a alternative for that damaged hyperlink.

And that’s it—a easy course of for gathering the URLs of an outdated web site lengthy forgotten or lately migrated.

Once more, because of Liam Delahunty for steerage by way of this course of. It’s solely his brainchild.