linus/Knack-Scraper
1
0
Fork 0
forked from lukaszett/Knack-Scraper
Code Pull requests Activity
Small dockerized tool to regularly scrape http://knack.news and write all posts to an sqlite database.
5 commits 1 branch 0 tags 81 KiB
Find a file
quorploop bcd210ce01 Dockerized Scraper 
- Implements Dockerized Version of Scraper
- Atomized tags and categories columns
2025-12-20 20:55:04 +01:00
.gitignore initial commit 2023-10-19 22:53:36 +02:00
Dockerfile Dockerized Scraper 2025-12-20 20:55:04 +01:00
main.py Dockerized Scraper 2025-12-20 20:55:04 +01:00
Makefile initial commit 2023-10-19 22:53:36 +02:00
README.md Dockerized Scraper 2025-12-20 20:55:04 +01:00
requirements.txt Dockerized Scraper 2025-12-20 20:55:04 +01:00

README.md

Knack-Scraper does exacly what its name suggests it does. Knack-Scraper scrapes knack.news and writes to an sqlite database for later usage.

Example for .env

NUM_THREADS=8
NUM_SCRAPES=100
DATABASE_LOCATION='./data/knack.sqlite'

Run once

python main.py