Python - how to setup Scrapy + psycopg2 (postgresql)

Hello, please advise me how to setup the python library Scrapy + psycopg2 (postgres database driver) on linux ubuntu server. I want to use it for web scraping, web harvesting. Thank you

REPLY

Hi,

how to run Python library Scrapy + psycopg2 on Ubuntu Server? I would proceed as follows:

apt install python3-pip
pip3 install scrapy

POSSIBLE ERROR: ModuleNotFoundError: No module named 'psycopg2'
-------------------
apt install postgresql
apt install python-psycopg2
apt install libpq-dev
pip3 install psycopg2

POSSIBLE ERROR: scrapy: command not found

The script scrapy is installed in '/home/scrapy/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
scrapy: command not found

-------------------
FIX:
#call absolute path (temporarily):
/home/scrapy/.local/bin/scrapy crawl hydro

#or permanently perform pip3 install scrapy as root, then it works via the scrapy command straight away