Web Crawling

An automated technique that systematically retrieves web‑published resources (HTML pages, linked files and machine‑readable feeds) by following link structures, sitemaps or harvest manifests to support indexing, discovery or capture of scholarly and cultural materials.