What are the laws surrounding web scraping?

Web scraping has been around for a long time and is a fundamental pillar of the internet. Bad bots pull content from a website in order to use it for purposes that are beyond the control of the website owner. Malicious bots make up to 20% of all web traffic and are used to perform a wide variety of malicious activities, such as, competitive data mining, online fraud, account theft, data theft, intellectual property theft, unauthorized vulnerability scans. However, the law does not grant web crawlers the freedom to use the scraped data for unlimited commercial purposes.

For example, a web crawler could search Youtube for video titles but not republish Youtube videos on their website because the videos are copyrighted.

The websites that require authentication of users that they must accept the website’s terms of use before they can access the website. These terms of use usually prohibit activities such as automatic data collection. For publicly accessible websites, since a user cannot be required to accept the terms of use before accessing the data, users can use web crawlers to collect data from the website.