Scrapy - Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

Scrapy - Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

Dobrý den,

mám Linux server, dosud vše běželo na Ubuntu 19, upgradem jsem teď přešel na Ubuntu 20 LTS. Mám tam skripty na stahování dat z webu přes python scrapy. Po upgradu to přestalo fungovat, hlavní je asi chyba:
twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

Celé chybové hlášení je:

scrapy shell 'https://www.up4.cz/'
2020-05-16 11:40:35 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: daily)
2020-05-16 11:40:35 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 18.9.0, Python 3.8.2 (default, Apr 27 2020, 15:53:34) - [GCC 9.3.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1f 31 Mar 2020), cryptography 2.8, Platform Linux-5.4.0-29-generic-x86_64-with-glibc2.29
2020-05-16 11:40:35 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2020-05-16 11:40:35 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'daily',
'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
'LOGSTATS_INTERVAL': 0,
'NEWSPIDER_MODULE': 'daily.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['daily.spiders']}
2020-05-16 11:40:35 [scrapy.extensions.telnet] INFO: Telnet Password: 7ff16f13eb3675ee
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage']
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-05-16 11:40:35 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-05-16 11:40:35 [scrapy.core.engine] INFO: Spider opened
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 1 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 2 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying (failed 3 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading : []
Traceback (most recent call last):
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived: []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 1 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 2 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying (failed 3 times): []
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 8, in
sys.exit(execute())
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/cmdline.py", line 98, in _run_print_help
func(*a, **kw)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/cmdline.py", line 151, in _run_command
cmd.run(args, opts)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/commands/shell.py", line 74, in run
shell.start(url=url, redirect=not opts.no_redirect)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/shell.py", line 45, in start
self.fetch(url, spider, redirect=redirect)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/shell.py", line 112, in fetch
response, spider = threads.blockingCallFromThread(
File "/usr/lib/python3/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
result.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in raiseException
raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseNeverReceived: []


PS: Skutečná cílová doména byla nahrazena up4.cz

ODPOVĚĎ


Dobrý den,

chybu "twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]" ve scrapy vyřešíte upgradem:

pip3 install scrapy --upgrade --force Show english version

Komentáře

Vaše reakce na Scrapy - Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

Související obsah

linux

python

scrapy

webscraping

Reference

Podívejte se na naše reference

Prohlédnout

Aplikace

Podívejte se na naše aplikace

Prohlédnout

Co umíme?

Podívejte se co umíme

Prohlédnout

Co umíme?

Vytváříme sofistikované aplikace pro náročné

Od webových aplikací přes android až po převodové můstky či složité informační systémy.

Podívejte se k nám

Máte ještě čas? Podívejte se na další rubriky

Tento web používá soubory cookie. Dalším procházením tohoto webu vyjadřujete souhlas s jejich používáním.. Více informací zde.