Scrapy - Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

Scrapy - Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

Hello,

I have a Linux server, so far everything was running on Ubuntu 19, I have now switched to Ubuntu 20 LTS with an upgrade. I have scripts there to download data from the web via python scraps. It stopped working after the upgrade, the main one is probably a bug:
twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

The whole error message is:

scrapy shell 'https://www.up4.cz/'
2020-05-16 11:40:35 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: daily)
2020-05-16 11:40:35 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 18.9.0, Python 3.8.2 (default, Apr 27 2020, 15:53:34) - [GCC 9.3.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1f 31 Mar 2020), cryptography 2.8, Platform Linux-5.4.0-29-generic-x86_64-with-glibc2.29
2020-05-16 11:40:35 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2020-05-16 11:40:35 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'daily',
'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
'LOGSTATS_INTERVAL': 0,
'NEWSPIDER_MODULE': 'daily.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['daily.spiders']}
2020-05-16 11:40:35 [scrapy.extensions.telnet] INFO: Telnet Password: 7ff16f13eb3675ee
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage']
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-05-16 11:40:35 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-05-16 11:40:35 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-05-16 11:40:35 [scrapy.core.engine] INFO: Spider opened
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 1 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 2 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying (failed 3 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading : []
Traceback (most recent call last):
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived: []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 1 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 2 times): []
2020-05-16 11:40:35 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying (failed 3 times): []
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 8, in
sys.exit(execute())
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/cmdline.py", line 98, in _run_print_help
func(*a, **kw)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/cmdline.py", line 151, in _run_command
cmd.run(args, opts)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/commands/shell.py", line 74, in run
shell.start(url=url, redirect=not opts.no_redirect)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/shell.py", line 45, in start
self.fetch(url, spider, redirect=redirect)
File "/home/scrapy/.local/lib/python3.8/site-packages/scrapy/shell.py", line 112, in fetch
response, spider = threads.blockingCallFromThread(
File "/usr/lib/python3/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
result.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in raiseException
raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseNeverReceived: []


PS: The actual target domain has been replaced by up4.cz

REPLY


Hello,

error "twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]" in scrapy you can fix by upgrade:

pip3 install scrapy --upgrade --force

Související obsah

linux

python

scrapy

webscraping

Komentáře

Vaše reakce na Scrapy - Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_ske_dhe', 'dh key too small')]

Reference

Podívejte se na naše reference

Prohlédnout

Aplikace

Podívejte se na naše aplikace

Prohlédnout

Co umíme?

Podívejte se co umíme

Prohlédnout

Co umíme?

Vytváříme sofistikované aplikace pro náročné

Od webových aplikací přes android až po převodové můstky či složité informační systémy.

Podívejte se k nám

Máte ještě čas? Podívejte se na další rubriky

Tento web používá soubory cookie. Dalším procházením tohoto webu vyjadřujete souhlas s jejich používáním.. Více informací zde.