A few of them had warnings about how their scripts no longer worked because StreetEasy started using Distil Networks to protect them from unwanted bots and web scrapers. So now I decided to take a closer look at the scraping scripts I saw on )( ) earlier. However, I quickly realized neither was able to give me the result I wanted, I was still getting the same html page that said “Pardon Our Interruption”. I could use the new requests-html library or I could use selenium.

My initial thought was since I was using the requests library that I wasn’t rendering JavaScript so after doing some googling I came across two ways to render JavaScript in Python. NoScript, is preventing JavaScript from running.Īdditional information is available in thisĪfter completing the CAPTCHA below, you will immediately You've disabled JavaScript in your web browser Ī third-party browser plugin, such as Ghostery or You're a power user moving through this website with This quickly led me to receive the following htmlĪs you were browsing, something about your browser made us However, the first task in my iterative approach was to just get a listing page for a StreetEasy search. It had been a while since I had scraped sites and I wanted to do it all (mostly) on my own. However, I thought it was simple enough I’d prefer to do it myself. At first glance, there appeared to be a few different StreetEasy scraping scripts on Github. So first let me quickly describe the reason I wasn’t able to scrape StreetEasy. Spoiler alert I wasn’t able to do this for a reason I will explain in more detail, but this led me to use a lot of old tools I hadn’t used in a while and come up with a script for scraping through Tor and switching IP’s between requests. So I had a thought, I used to web scrape sites using Python, why not try it on StreetEasy and filter the apartments myself. After a quick search I realized that people have been requesting this feature for years (since Oct 2015 to be exact), and it was nowhere in sight and there didn’t seem to be any services out there that did it either.

The other day I was starting the search for a new aparment in New York City, which I have done a couple of times now, and was frustrated that StreetEasy doesn’t allow you to filter apartments that are available after a certain date. Another thing to note is that some sites are able to automatically block IP’s that are Tor exit nodes, so this may not work for some sites that go to these measures. This was more an informational exercise and I wanted to share it with others. I do not condone the use of this information for creating illegal web crawlers.

Tor proxy github