Link analysis sample set

In 2022 and 2023 two researchers started working with the KB web collection for the first time in the history of our web collection. This triggered the need for analysing the web-data within the WARC container files we hold. The result of this: datasets with hyperlinks. On this page you can find an example of the kind of dataset we used for their research. As example a kb.nl harvest was used. 

Analysing hyperlinks in the KB web collection

The first time extracting hyperlinks was done by myself in preparation for the arrival of Researcher in Residence Karin the Wild back in 2022. You can read how I extracted the hyperlinks and worked with them in Gephi in the blog series ‘Analysing hyperlinks in the KB web collection’. 

I learned a lot from this first experiment. The main thing I discovered was that I wanted to have the tags as well as the hyperlink in my dataset. They tell a lot about the type of hyperlink you extract and are needed if you want to do a thorough analysis. I also learned that the way you crop your hyperlink is very important for the outcome of your hyperlink analysis. I took those experiences and used them when Jesper Verhoef became our Researcher in Residence in 2023. Changes to our original extraction script improved the dataset. 

Fast forward to the end of 2022 and&nbsp;</em><a href="https://www.kb.nl/en/news/kbs-xs4all-web-collection-unesco-world-heritage-list"><em>this collection was recognized as UNESCO heritage</em></a><em>. But… Was Kees right? 

Example how a hyperlink is embedded in a website (and therefore in a WARC file). Source: No Longer XS4ALL. 

The dataset

For those interested I made a sample set based on a crawl from the website of kb.nl:

Table 1. Metadata from the kb.nl harvest.

This target has four seeds, but the harvest was cut short by a quality assurance officer. This means only a part of the selected seeds were harvested. Because kb.nl is the primary seed, the harvester started with that seed and did not come around to the other before it was manually stopped. 

Result

You can find the whole dataset in csv under the ‘examples’ tab. But here (table 1) is a little sneak peek. In the datasets I used you first have the ‘source’ website: the website on which the link was found. After this comes the ‘target’: the website to which the link refers. The third column is the weight: how many times the source website refers to the ‘target’ website. Lastly we have the type of URL. As mentioned above in the beginning this was only anchor or embedded. After updating the extraction script we now find the specific tag which surrounds the link. In the KB dataset it is mostly ‘a’ for anchor link. But you can also find some images and frames in there. 

Table 2. Example of the data found inside the kb.nl_2024_links dataset.

It is important to note that this dataset does not reflect the kb.nl website as a whole. As mentioned above the harvest was stopped manually. But even if the harvest was completed, it only shows what we have been managed to archive with Heritrix and WCT and what we managed to extract with our WARC-link-extraction script.