"robots.txt" is a file which allows the website owner to disallow bots, crawlers, scrapers, and other potentially malicious or unwanted behaviour on their website.
