PRTG integration (Python script)
Asked by Richard Norman:

Hi community,  
I am trying to get acknowledgement of alerts in Opsgenie to trigger (via HTTP API) acknowledgement in PRTG of the initial alert.   
I have everything else in terms of the HTTP PRTG Opsgenie integration functional.  

The issue I seem to be having revolves around the local OEC service on the PRTG server having issues running the python script. The script in use is the one referred to from the Atlassian PRTG integration article and available from Github.

The error I am getting is pasted below and seems that certain parts of the ActionExecutor script are not understood.  
I am not a python coder so there is every chance I have completely misunderstood what manipulation I need to do with the script.  

Any insight would be great  

Thanks

Rich

Python errors:  

\[37mDEBUG\[0m\[2021-03-24T13:42:13.199198Z\] Job\[a658b2e9-7884-41b8-a631-fbc3982971df\] has been processed by worker\[d15b95b2-b14e-448b-8a83-1e34eeb67723\].  
\[37mDEBUG\[0m\[2021-03-24T13:42:13.3519763Z\] Message\[289159ad-e8f5-4456-88c6-3d158e31d534\] is deleted from the queue\[us-west-2\].  
\[37mDEBUG\[0m\[2021-03-24T13:42:13.3880999Z\] Successfully sent result of message\[a658b2e9-7884-41b8-a631-fbc3982971df\] to OpsGenie and it took 0.188902 seconds.  
\[37mDEBUG\[0m\[2021-03-24T13:42:13.4901267Z\] Action\[acknowledgeSensor\] execution of message\[289159ad-e8f5-4456-88c6-3d158e31d534\] with entityId\[5f8b3b61-70a1-45bc-af3f-b636af93e590-1616592719477\] failed: exit status 1 Stderr: Traceback (most recent call last):  
File "C:\\opsgenie\\scripts\\actionExecutor.py", line 6, in \<module\>  
import requests  
ModuleNotFoundError: No module named 'requests'


Answers:
author: Connor Eyles
body: Hi Richard,

Would also like to inform you that I am not a python coder but can help with pointing you in the right direction in getting this working :)

The error message you are receiving is 'ModuleNotFoundError: No module named 'requests'

To resolve this you will need to install the 'requests' module in your environment, check out this document here <https://stackoverflow.com/questions/17309288/importerror-no-module-named-requests>

This should help you with running the right command to install this module, let me know how you go!

Thanks,  
Connor

comments: author: Richard Norman
body: Thanks Connor for your help on this one. I have only just got around to having a good read through that article and installing the modules. I am not in my comfort zone here but normally can apply enough savvy to get there in the end, so im still keen on getting it working.  

Now I have applied the modules I dont seem to get any errors but when acknowledging an alert on Opsgenie the alert doesnt seem to get acknowledged on PRTG. I have pasted below the output from the PRTG OEC integration service installed on the PRTG server. I am grateful for your help so far. If you have anything else you might be able to throw in to the ring I would be grateful as I want to see this through :-) Thanks

\[37mDEBUG \[0m\[2021-04-13T15:15:58.5768097+01:00\] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0  
\[37mDEBUG \[0m\[2021-04-13T15:16:13.5831573+01:00\] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0  
\[37mDEBUG \[0m\[2021-04-13T15:16:28.574219+01:00\] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0  
\[37mDEBUG \[0m\[2021-04-13T15:16:43.158568+01:00\] Received 1 messages from the queue\[us-west-2\].  
\[37mDEBUG \[0m\[2021-04-13T15:16:43.1595668+01:00\] Job\[0b128283-ff33-4302-a104-5ffbb8c09ee2\] is being submitted  
\[37mDEBUG \[0m\[2021-04-13T15:16:43.1605601+01:00\] Job\[0b128283-ff33-4302-a104-5ffbb8c09ee2\] is submitted to worker\[cc6ae6d4-8108-4311-bf25-cdd940047d92\]  
\[37mDEBUG \[0m\[2021-04-13T15:16:43.5248695+01:00\] Message\[0b128283-ff33-4302-a104-5ffbb8c09ee2\] is deleted from the queue\[us-west-2\].  
\[37mDEBUG \[0m\[2021-04-13T15:16:43.5748759+01:00\] Current Worker: 4, Idle Worker: 3, Queue Size: 0, Queue load: 0  
\[37mDEBUG \[0m\[2021-04-13T15:16:44.4010725+01:00\] Action\[acknowledgeSensor\] execution of message\[0b128283-ff33-4302-a104-5ffbb8c09ee2\] with entityId\[80f53b00-ecf1-4f79-84a5-e77511b8a6d6-1618323335455\] has been completed and it took 0.876208 seconds.  
\[37mDEBUG \[0m\[2021-04-13T15:16:44.4010725+01:00\] Job\[0b128283-ff33-4302-a104-5ffbb8c09ee2\] has been processed by worker\[cc6ae6d4-8108-4311-bf25-cdd940047d92\].  
\[37mDEBUG \[0m\[2021-04-13T15:16:44.6028558+01:00\] Successfully sent result of message\[0b128283-ff33-4302-a104-5ffbb8c09ee2\] to OpsGenie and it took 0.192868 seconds.  
\[37mDEBUG \[0m\[2021-04-13T15:16:58.5817922+01:00\] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0  
\[37mDEBUG \[0m\[2021-04-13T15:17:13.5746428+01:00\] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0  
\[37mDEBUG \[0m\[2021-04-13T15:17:28.5740414+01:00\] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0


author: Connor Eyles
body: Completely understand that! Well the log messages state "Action\[acknowledgeSensor\] execution of message\[0b128283-ff33-4302-a104-5ffbb8c09ee2\] with entityId\[80f53b00-ecf1-4f79-84a5-e77511b8a6d6-1618323335455\] has been completed"  

Which tells me that it executed correctly! The next thing I would suggest is that there are no networking issues between your OEC server and your PRTG server and check the PRTG server for any logs to see if it has contacted PRTG correctly.

Another thing to check that the PRTG credentials are set correctly within the conf.json so that the script can correctly authenticate with PRTG.

Thanks,  
Connor


author: Richard Norman
body: Thanks Connor,

I have done a quick check along these lines and I have actually installed the OEC service on the PRTG server so network connectivity shouldnt be an issue. I don't see a firewall issue either as I assume if the PRTG http interface is available from the web then opsgenie should be able to speak to it OK. Something I now think might be an issue is the account on PRTG opsgenie is using to try and acknowledge the alerts, and whether that account has the right access or permissions within PRTG to do so? I have as such made the account a member of a group with full visibility of all devices and retested without luck. Any further thoughts?

cheers

Rich


author: Connor Eyles
body: Hi [@Richard Norman](/t5/user/viewprofilepage/user-id/4319740)

I think the next best action is that you raise a support ticket with us so we can gain access to your Opsgenie to review the logs from the Opsgenie point of view. To ensure that it is executing correctly!

<https://support.atlassian.com/opsgenie/>

Thanks,

Connor


author: Richard Norman
body: Hey Connor.

I thought maybe having OEC run on the PRTG server might be causing issues accessing the PRTG web page (HTTPS offloaded via a load balancer) To get around that I created an entry in the PRTG server local hosts file for the loopback to resolve to the PRTG web URL. At this point I got a lot of errors as the OEC service didnt like the fact the PRTG server doesnt have a cert (that is why I offloaded the SSL as putting a cert on PRTG isnt straightforward.

Having proved that the OEC was now speaking internally to the apache server (to eliminate any networking issues or load balancer issues) I decided to go through the antiquated process of putting a cert on the server itself to get ris of the OEC SSL errors now seen. I successfully managed to get rid of the errors afterward and success messages came back as before (pasted above) After all that, the back acknowledge to PRTG still isnt functional. I have not raised a ticket with you guys before but I think now is the time. Can you please guide me through the process if you wouldnt mind?

Many thanks for all your help so far and moving forward.

Rich


author: Richard Norman
body: FYI errors when cert didnt work  

File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\connectionpool.py", line 699, in urlopen  
httplib_response = self._make_request(  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\connectionpool.py", line 382, in _make_request  
self._validate_conn(conn)  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\connectionpool.py", line 1010, in _validate_conn  
conn.connect()  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\connection.py", line 411, in connect  
self.sock = ssl_wrap_socket(  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\util\\ssl_.py", line 428, in ssl_wrap_socket  
ssl_sock = _ssl_wrap_socket_impl(  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\util\\ssl_.py", line 472, in _ssl_wrap_socket_impl  
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\ssl.py", line 500, in wrap_socket  
return self.sslsocket_class._create(  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\ssl.py", line 1040, in _create  
self.do_handshake()  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\ssl.py", line 1309, in do_handshake  
self._sslobj.do_handshake()  
ssl.SSLCertVerificationError: \[SSL: CERTIFICATE_VERIFY_FAILED\] certificate verify failed: self signed certificate (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\requests\\adapters.py", line 439, in send  
resp = conn.urlopen(  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\connectionpool.py", line 755, in urlopen  
retries = retries.increment(  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\urllib3\\util\\retry.py", line 574, in increment  
raise MaxRetryError(_pool, url, error or ResponseError(cause))  
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='prtgservice.surrey.ac.uk', port=443): Max retries exceeded with url: /index.htm/api/acknowledgealarm.htm?id=15856\&ackmsg=Richard+Norman+acknowledged+alert%3A+%22%5BPRTG%5D+%5BPRTG+Network+Monitor+%28PRTGAPP01V%29%5D+AMONbalancer+Throughput+%28SNMP+Custom%29+Down%22+https%3A%2F%2Fopsg.in%2Fa%2Ft%2Funiversityofsurrey%2F9099\&username=netops%40universityofsurrey.opsgenie.net\&passhash=1408463140 (Caused by SSLError(SSLCertVerificationError(1, '\[SSL: CERTIFICATE_VERIFY_FAILED\] certificate verify failed: self signed certificate (_ssl.c:1123)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):  
File "C:\\opsgenie\\scripts\\actionExecutor.py", line 89, in \<module\>  
main()  
File "C:\\opsgenie\\scripts\\actionExecutor.py", line 78, in main  
response = requests.post(result_url, params=params, timeout=timeout)  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\requests\\api.py", line 119, in post  
return request('post', url, data=data, json=json, \*\*kwargs)  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\requests\\api.py", line 61, in request  
return session.request(method=method, url=url, \*\*kwargs)  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\requests\\sessions.py", line 542, in request  
resp = self.send(prep, \*\*send_kwargs)  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\requests\\sessions.py", line 655, in send  
r = adapter.send(request, \*\*kwargs)  
File "C:\\Users\\rn0017\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\requests\\adapters.py", line 514, in send  
raise SSLError(e, request=request)  

<br />

After cert installed on PRTG backend web server

\[37mDEBUG\[0m\[2021-04-21T10:14:54.3774497+01:00\] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0  
\[37mDEBUG\[0m\[2021-04-21T10:15:08.6839897+01:00\] Received 1 messages from the queue\[us-west-2\].  
\[37mDEBUG\[0m\[2021-04-21T10:15:08.6849936+01:00\] Job\[73e6790f-02e0-40e8-bcc0-9c0655557924\] is being submitted  
\[37mDEBUG\[0m\[2021-04-21T10:15:08.6849936+01:00\] Job\[73e6790f-02e0-40e8-bcc0-9c0655557924\] is submitted to worker\[1794559e-8622-4395-8dc5-b02d18ba107c\]  
\[37mDEBUG\[0m\[2021-04-21T10:15:09.0391197+01:00\] Message\[73e6790f-02e0-40e8-bcc0-9c0655557924\] is deleted from the queue\[us-west-2\].  
\[37mDEBUG\[0m\[2021-04-21T10:15:09.3645657+01:00\] Current Worker: 4, Idle Worker: 3, Queue Size: 0, Queue load: 0  
\[37mDEBUG\[0m\[2021-04-21T10:15:10.2086108+01:00\] Action\[acknowledgeSensor\] execution of message\[73e6790f-02e0-40e8-bcc0-9c0655557924\] with entityId\[17551ce8-e333-4998-88d6-3e102af38049-1618996472332\] has been completed and it took 1.169483 seconds.  
\[37mDEBUG\[0m\[2021-04-21T10:15:10.2086108+01:00\] Job\[73e6790f-02e0-40e8-bcc0-9c0655557924\] has been processed by worker\[1794559e-8622-4395-8dc5-b02d18ba107c\].  
\[37mDEBUG\[0m\[2021-04-21T10:15:10.3924782+01:00\] Successfully sent result of message\[73e6790f-02e0-40e8-bcc0-9c0655557924\] to OpsGenie and it took 0.183866 seconds.


author: Connor Eyles
body: Hi [@Richard Norman](/t5/user/viewprofilepage/user-id/4319740)

You can start by going to this URL <https://support.atlassian.com/opsgenie/>

Then click on the green "login to contact support" which will direct you to log into your Atlassian account.

Then start filling out the form for Opsgenie and support details!

I will keep my eye out on the support Queue so I can pick up the case :smile:

Thanks,

Connor


Tags: cloud

integration

python