The semiconductor shortage led to one of my most successful projects: ilox, a product stock tracking bot. It monitors changes in product stock and sends out notifications whenever it changes.
This guide will show you how to write a monitoring bot yourself using NodeJS and cover the following topics:
- setting up a scalable architecture for a tracking bot
- different crawling methods: source code parsing, using APIs and automated browsers with Puppeteer
- advanced tips (avoiding bot detection, logging, proxies and more)
Keep in mind that this is not a guide on how to crawl a specific shop - we're going to use a fake "shop" for this which can be found here.
As with all my guides, the entire source code will be downloadable at the end of the article.