Create a multi threaded web tool to scrape via proxies + save visual/text output

We want to bulk check thousands of domain names per day in Wayback Machine to look for irregularities and spam factors. ([login to view URL]) ** P.S - we've included a very detailed request of what we want to do, but the project is relatively simple scraping and output. So don't be put off :) ** We want to create a web script that can connect and scrape the content + take screenshots of the pages in bulk, via hundreds of proxies. On average, we will want to check 10,000 domains per day with an average of 20-50 captures per domain. Wayback Machine has a max requests limit of 10-15 per minute, per IP. The strategy behind this project is to check for spam/adult content on the websites across the history of the domain. We will have a list of blacklist keywords we will use to mark the domains as good/bad. If a blacklisted word appears on the page in wayback history, we will have the tool mark the domain as bad. This tool will be used to remove the bulk of the spam and cut down 10,000 domains to 500-1000 that we can manually analyze. We will want to extract the text output of each capture page so we can check for blacklisted keywords, and also a screenshot of the page once it’s fully loaded. Wayback is quite slow and takes a while to load, so we’ll need to factor that in as well. We will ensure we have fast dedicated proxies. Layout Interface We will want to utilize a web based interface where we can have each project and individual domains along with their associated data. We can use a web template like this: [login to view URL] We currently have a tool that does a % of what we want to build here, but relies on third party solutions. This is the current layout > [login to view URL] As you can see, we can create a project, and paste the domains in that we want to analyze, then we click inside a project and load each domain individually. We will want to show each capture as a screenshot like this > [login to view URL] But with each screenshot, we want to also include the following. A link direct to the page URL on wayback machine (as pictured) A list of what blacklisted keywords have appeared on the page. Essentially we want to be able to see a screenshot of the entire page, and then a list of keywords that appear, so we can click to see a larger image of the screenshot and check that the blacklisted keywords were not by mistake. This process will not be bulletproof and the tool will show false positives, which is to be expected. As an example we will want to filter any adult content, and will have a list of adult keywords in the blacklist, but for example a political or legal blog will mention certain words that will trigger a red flag. We will be able to manually vet that the domain is fine and not spammy though. Currently we use [login to view URL] which gives us screenshots, but does not deliver the full length website screenshot, which is what we need. So you will need to locate and implement a full page length screenshot software. It will be fine to show a square screenshot of the top of the page in the interface, to make it all look uniform, but then if you click the screenshot, it will load a full screen scrollable version. We can discuss specifics of this later to come. In the settings page we will want to be able to load in proxies, as well as an updateable blacklist keyword list. So in summary, it’s a fairly simple process. Import a list of domains Process them all via wayback machine, scraping 1 capture per month per domain Pull the plain text (not HTML) + a screenshot of each capture Output the screenshot + link to direct page + list which blacklisted keywords show up We need this to be reliable, strong tech that works great as a multi threaded process at speed. We will provide whatever server/tech/hardware is necessary, speed is the utmost highest factor here.