forked from lukaszett/Knack-Scraper
Dockerized Scraper
- Implements Dockerized Version of Scraper - Atomized tags and categories columns
This commit is contained in:
parent
7cc3d1b7e4
commit
bcd210ce01
5 changed files with 201 additions and 42 deletions
18
README.md
18
README.md
|
|
@ -0,0 +1,18 @@
|
|||
Knack-Scraper does exacly what its name suggests it does.
|
||||
Knack-Scraper scrapes knack.news and writes to an sqlite
|
||||
database for later usage.
|
||||
|
||||
## Example for .env
|
||||
|
||||
```
|
||||
NUM_THREADS=8
|
||||
NUM_SCRAPES=100
|
||||
DATABASE_LOCATION='./data/knack.sqlite'
|
||||
```
|
||||
|
||||
## Run once
|
||||
|
||||
```
|
||||
python main.py
|
||||
```
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue