View Categories

Use Local Storage Variable to scrape Do-follow Links

3 min read

Use Case #

When studying the competition or even doing an analysis of your own website, you may want to find out all the links of a given page that are using do-follow. Because the do-follow element cannot be captured using a CSS locator, we need to use a JavaScript that would capture all the do-follow URLs found on a given page and save them as an internal variable called Local Storage Variable. The role of this type of variable is to be able to feed or capture data that is manipulated by a JavaScript code and not an RTILA command. In this use case we cover the flow whereby it is the JavaScript Execute Command code that captures and saves the do-follow URLs into an Local Storage Variable. And then we will use a Dynamic Command which will allow us to run a one line JavaScript code to capture the value of the Local Storage Variable (list of do-follow URLs) and “transfer it an RTILA Variable Command (here the Dynamic Variable). This allows us then to easily re-use and call the Dynamic Variable value in any other commands or properties in the datasets.

Import a list of URLs to crawl for do-follow links #

First step is to load a txt file with all the URLs for which you want to find out and scrape their respective do-follow links. See below screenshot and make sure you use a txt file that has the ANSI format and that you have one URL per line.

Use Execute JavaScript command to catch do-follow links #

As explained earlier we need to use a JavaScript code that will go through all the links (href) in the source code and find out those using do-follow as an attribute and is saving all those urls seperated by a comma into a Local Storage Variable called “do-Follow-Links”. The JavaScript code is shared below.

Copy Code
const links = document.querySelectorAll('a');
const doFollowLinks = [];
for (let link of links) {
  if (link.rel === '' || !link.rel.includes('nofollow')) {
    doFollowLinks.push(link.getAttribute('href'));
  }
}
console.log(doFollowLinks);
window.localStorage.setItem('do-Follow-Links', doFollowLinks);

Transfer Local Storage value into a Dynamic Variable value #

Once the URLs are identified and saved by our Execute JavaScript command into the ” Local Storage Variable called do-Follow-Links”, we then use a Dynamic Variable to execute an other line of JavaScript to “transfer” the value of “do-Follow-Links” into the Dynamic Variable Command VALUE.

The one liner JavaScript code used to “transfer” the Local Storage value into the Dynamic Variable Value is shared below

Copy Code
VALUE=window.localStorage.getItem('do-Follow-Links');

We can now use the syntax {{VariableName}} to print and share this value with other parts of RTILA Studio, in our case we want to save the list of URLs into a DataSet property, see below how we configure the properties. For the Source URL we are using the JavaScript action “FIELD_VALUE=window.location.href” to capture current URL being visited by RTILA. As for the Do-Follow URLs property we are using the syntax that calls the current value of our Dynamic Variable, in our case this syntax: {{Dynamic}}.

Download, edit and improve #

You can download this Public Template file as well as the txt file example by clicking here. Feel free to replace the URLs or add other commands and actions or properties to the flow.

    Powered by BetterDocs