Waton Developer Cloud Python SDK sometimes returnes NewConnectionError
My customer uses Watson Developer Cloud Python SDK and calls NLC services for evaluation classifiers' performance. When they called NLC many times in loop, sometimes NLC returned NewConnectionError and stopped python script.
Error message is as follows.
Traceback (most recent call last): File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connection.py", line 138, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\util\connection.py", line 98, in create_connection raise err File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\util\connection.py", line 88, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] 接続済みの呼び出し先が一定の時間を過ぎても正しく応答しなかったため、接続できませんでした。または接続済みのホストが応答しなかったため、確立された接続は失敗しました。
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 594, in urlopen chunked=chunked) File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 350, in _make_request self._validate_conn(conn) File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 835, in _validate_conn conn.connect() File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connection.py", line 281, in connect conn = self._new_conn() File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connection.py", line 147, in _new_conn self, "Failed to establish a new connection: %s" % e) requests.packages.urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [WinError 10060] 接続済みの呼び出し先が一定の時間を過ぎても正しく応答しなかったため、接続できませんでした。または接続済みのホストが応答しなかったため、確立された接続は失敗しました。
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Anaconda3\lib\site-packages\requests\adapters.py", line 423, in send timeout=timeout File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 643, in urlopen _stacktrace=sys.exc_info()[2]) File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\util\retry.py", line 363, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gateway.watsonplatform.net', port=443): Max retries exceeded with url: /natural-language-classifier/api/v1/classifiers/ff1b44x158-nlc-617/classify (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 100 60] 接続済みの呼び出し先が一定の時間を過ぎても正しく応答しなかったため、接続できませんでした。または接続済みのホストが応答しなかったため、確立された接続は失敗しました。',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "NLC.py", line 24, in classes = natural_language_classifier.classify(classifier_id[i], text) File "C:\Anaconda3\lib\site-packages\watson_developer_cloud\natural_language_classifier_v1.py", line 63, in classify json={'text': text}) File "C:\Anaconda3\lib\site-packages\watson_developer_cloud\watson_developer_cloud_service.py", line 298, in request **kwargs) File "C:\Anaconda3\lib\site-packages\requests\api.py", line 56, in request return session.request(method=method, url=url, **kwargs) File "C:\Anaconda3\lib\site-packages\requests\sessions.py", line 488, in request resp = self.send(prep, **send_kwargs) File "C:\Anaconda3\lib\site-packages\requests\sessions.py", line 609, in send r = adapter.send(request, **kwargs) File "C:\Anaconda3\lib\site-packages\requests\adapters.py", line 487, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gateway.watsonplatform.net', port=443): Max retries exceeded with url: /natural-language-classifier/api/v1/classifiers/ff1b44x158-nlc-617/classify (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10060] 接続済みの 呼び出し先が一定の時間を過ぎても正しく応答しなかったため、接続できませんでした。または接続済みのホストが応答しなかったため、確立された接続は失敗しました。',))
I found similar issue posted here, but the issue doesn't seems to be resolved. https://developer.ibm.com/answers/questions/292691/is-data-transfer-to-watson-vision-reco-with-python/?smartspace=watson
Sometimes they can succeed to complete script without errors, so I think this is not caused by user program logic. Could anyone give me solutions about it?
Thanks.
I am guessing that as you are getting errors in and whilst repeating invoking methods on urllib3 then you are hitting a resource limitation in Python on your machine. This could either be a thread limit or an open connection limit, which will be further affected if you are making async calls. Either way you need to batch up the calls so that thread or connection resources are cleanly freed up allowing them to be reused.
It looks like you are trying a fresh call on the exception handling which is failing as the resources you need haven't been freed up yet.
Thank you for your comment and I'm sorry to respond too late.
I doubted that this was caused by limitation of sockets so I already checked socket status by using both netstat command and netsh command while program was running. (Client program runs on Windows PC(Win 7).) As a result, program used different sockets for each call but calls were serialized and use sockets one by one. So I think this is not caused by any limits(thread limit, open connection limit and so on).
As you say it is one solution to add exception handling and I will add to my source code. But I also want to know route causes of this issue and underlying solutions. If anyone have ideas, I'm happy to hear them.
Thank you!