Small dockerized tool to regularly scrape http://knack.news and write all posts to an sqlite database.
|
|
||
|---|---|---|
| .gitignore | ||
| Dockerfile | ||
| main.py | ||
| Makefile | ||
| README.md | ||
| requirements.txt | ||
Knack-Scraper does exacly what its name suggests it does. Knack-Scraper scrapes knack.news and writes to an sqlite database for later usage.
Example for .env
NUM_THREADS=8
NUM_SCRAPES=100
DATABASE_LOCATION='./data/knack.sqlite'
Run once
python main.py