Web Scraping – Discovering Hidden APIs – Ian London’s Blog

I was trying to help someone with a web scraping task today, and stumbled upon an interesting technique to find hidden APIs to scrape data from certain websites. Some sites use frontend frameworks which render dynamic content by loading a JSON or XML file from their backend to populate the user-facing site. I’m going to show you how to find the URL to access that dynamic content so you can easily scrape data from a site, without lxml or BeautifulSoup or CSS selectors or XPath. Just nice, simple JSON.

Source : Web Scraping – Discovering Hidden APIs – Ian London’s Blog