What's happening on the Framebuffer?

To make the browser seem more “real” it might be better to actually launch a browser window into a framebuffer.

sudo apt-get update -q
sudo apt-get install xvfb

Xvfb :99 -screen 0 1024x768x16 &

export DISPLAY=:99

Then run the scraper that launches a browser. You could also use xvfb-run.

Xpra

For debugging purposes it can be useful to see what’s being rendered onto the framebuffer. You can do this using Xpra. Once you’ve installed Xpra launch the server.

xpra shadow :99 --bind-tcp=0.0.0.0:14500

Then launch the client.

xpra attach tcp:127.0.0.1:14500