12
Benefits of using scrapy over requests/selenium (lemmy.world)
submitted 1 year ago by IceSea@lemmy.world to c/python@programming.dev
3 comments fedilink hide all child comments

When I'm writing webscrapers I mostly just pivot between selenium (because the website is too "fancy" and definitely needs a browser) and pure requests calls (both in conjunction with bs4).

But when reading about scrapers, scrapy is often the first mentioned Python package. What am I missing out on if I'm not using it?

you are viewing a single comment's thread
view the rest of the comments
[-] Wats0ns@programming.dev 2 points 1 year ago

Yep try scrapy. And also it handles for you the concurrency of your pipelines items, configuration for every part,...

this post was submitted on 18 Jul 2023
12 points (92.9% liked)

Python

6187 readers
1 users here now

Welcome to the Python community on the programming.dev Lemmy instance!

📅 Events

October 2023

November 2023

PastJuly 2023

August 2023

September 2023

🐍 Python project:
💓 Python Community:
✨ Python Ecosystem:
🌌 Fediverse
Communities
Projects
Feeds

founded 1 year ago
MODERATORS
erlingur@programming.dev
jnovinger@programming.dev