I'm working on a personal project where I crawl a website and check for WCAG Level AA accessibility issues.

I would like to first crawl the website to find all links, not just on the page but the entire website.

I want those stored in a column in my database.

Then I want to scrape each page.

For scraping I plan to use HTMLAgilityPack but I can't seem to find a web crawler...if one even exists.

I currently have a controller using HTMLAgilityPack for scraping all links on a single web page.

public class ScraperController : Controller { public IActionResult Index() { var site = @"https://www.harvard.edu/"; HtmlWeb web = new HtmlWeb(); var htmlDoc = web.Load(site); List<ScraperViewModel> hrefTags = new List<ScraperViewModel>(); hrefTags = ExtractAllAHrefTags(htmlDoc, site); return View(hrefTags); } private List<ScraperViewModel> ExtractAllAHrefTags(HtmlDocument htmlDoc, string site) { List<ScraperViewModel> hrefTags = new List<ScraperViewModel>(); var linksOnPage = from lnks in htmlDoc.DocumentNode.Descendants() where lnks.Name == "a" && lnks.Attributes["href"] != null && lnks.InnerText.Trim().Length > 0 select new { Url = lnks.Attributes["href"].Value, Text = lnks.InnerText }; foreach (var link in linksOnPage) { string url = link.Url; if (link.Url.StartsWith("/")) { url = site + link.Url; } if (link.Url.StartsWith("#")) { url = site + link.Url; } hrefTags.Add(new ScraperViewModel() { Name = link.Text, Url = url}); } return hrefTags; } } 

But then I need to scrape any internal links that it found and add them to the table as long as they don't already exists and keep repeating until everything has been crawled. Is there a Crawler package that already exists?

I can't figure out the logic of how to do this myself....loop through initial list of links, scrape and add to database if they don't exist and then add to list of links that are being looped through if it finds some more unique links..