Repository: matrix-org/synapse

Issue #2514: Pintobyte.com regularly receives thundering herds of presence EDUs which badly overloads it.
Created: 2017-10-09T15:57:46Z
The box apparently spikes CPU when this happens, and the logs look like:
2017-10-09 09:46:54,078 - synapse.access.https.8448 - 91 - INFO - PUT-608361- 137.74.150.193 - 8448 - {maunium.net} Processed request: 15875ms (12ms, 0ms) (20ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1507147021275/ HTTP/1.1" "Synapse/0.23.1"
2017-10-09 09:46:54,079 - synapse.access.https.8448 - 91 - INFO - PUT-608362- 5.196.33.160 - 8448 - {unixpimps.net} Processed request: 15873ms (8ms, 0ms) (31ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1506285847368/ HTTP/1.1" "Synapse/0.22.1"
2017-10-09 09:46:54,080 - synapse.access.https.8448 - 91 - INFO - PUT-608363- 83.136.253.55 - 8448 - {status.im} Processed request: 15381ms (11ms, 0ms) (30ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1507209450899/ HTTP/1.1" "Synapse/0.23.1 (b=develop,5de926d)"
2017-10-09 09:46:54,080 - synapse.access.https.8448 - 91 - INFO - PUT-608360- 74.15.30.197 - 8448 - {otonokizaka.is-a-linux-user.org} Processed request: 15881ms (0ms, 0ms) (40ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1507329008642/ HTTP/1.1" "Synapse/0.22.1"
2017-10-09 09:46:54,087 - synapse.access.https.8448 - 91 - INFO - PUT-608364- 178.255.151.168 - 8448 - {err.no} Processed request: 15245ms (0ms, 0ms) (29ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1507308264601/ HTTP/1.1" "Synapse/0.22.1"
2017-10-09 09:46:54,089 - synapse.access.https.8448 - 91 - INFO - PUT-608365- 62.210.203.126 - 8448 - {thekinrar.fr} Processed request: 15122ms (16ms, 0ms) (61ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1502572424591/ HTTP/1.1" "Synapse/0.22.1"
2017-10-09 09:46:54,098 - synapse.access.https.8448 - 91 - INFO - PUT-608366- 5.9.31.99 - 8448 - {matrix.crypto.koeln} Processed request: 15115ms (12ms, 0ms) (60ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1507500334310/ HTTP/1.1" "Synapse/0.22.1"
2017-10-09 09:46:54,101 - synapse.access.https.8448 - 91 - INFO - PUT-608370- 5.45.104.21 - 8448 - {msgs.tk} Processed request: 9335ms (0ms, 0ms) (56ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1507217392729/ HTTP/1.1" "Synapse/0.23.1 (b=msgs.tk,d7e7bbe8)"
2017-10-09 09:46:54,103 - synapse.access.https.8448 - 91 - INFO - PUT-608371- 109.74.200.146 - 8448 - {sw1v.org} Processed request: 8830ms (15ms, 0ms) (40ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1507307336083/ HTTP/1.1" "Synapse/0.23.1 (b=rav/federation_server_meltdown,37048b8)"
2017-10-09 09:46:54,112 - synapse.access.https.8448 - 91 - INFO - PUT-608372- 163.172.150.247 - 8448 - {thewordnerd.info} Processed request: 8205ms (8ms, 0ms) (40ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1504875069277/ HTTP/1.1" "Synapse/0.22.1"
2017-10-09 09:46:54,120 - synapse.access.https.8448 - 91 - INFO - PUT-608373- 35.185.61.183 - 8448 - {pfe.io} Processed request: 8204ms (4ms, 0ms) (47ms/2) 11B 200 "PUT /_matrix/federation/v1/send/1506311318218/ HTTP/1.1" "Synapse/0.22.1"

However, in the preceding 15s, there are pauses of 2-3s in the logs.  It looks like the reactor is getting wedged on something, but no idea what, nor does it seem to be covered by a log context.  It's not getting stuck behind a linearizer lock seemingly.
I'm wondering if there's some queuing going on in processing the requests such that they suddenly all end up getting processed at the same point.  I vaguely remember seeing this when investigating the Christmas meltdown - that Twisted has some kind of internal queue where requests start to stack up if the reactor is too busy?

Duplicate issue #3490: I don't care that you're online (or: presence EDUs from strangers)
Created: 2018-07-06T18:07:43Z
Version: 28.1
Installed via 'python2-matrix-synapse' package on openSUSE Leap 15.0
Let me preface this by saying a) I've only recently setup a homeserver, and b) I don't know if this is expected behavior but c) I wouldn't think so.
I have noticed that my homeserver gets a steady flow of m.presence EDUs (2-3 per second every few seconds on average) from remote homeservers that I assume were joined to busy rooms that I also joined, but subsequently left. It seems they did not get the 'leave' notification, and continue to let me know that @bob@someserver.tld is offline (or online, whatever). I'm sure Bob's a swell guy, but I don't really care. These EDUs bump the CPU and eat a little bandwidth each time they chat up my server to inform me of Bob's whereabouts. If this trend continues, it seems like that hail of unsolicited presence updates might add up over time, accumulating each time I join and then leave a room. Eventually it might represent a noticeable impact to CPU and network traffic.
EDIT: I assumed this presence data was from rooms I had joined, but see also #1429 and #1532. User reported getting status from people they never shared a room with. Maybe same here?
So first question: is this normal, expected behavior for Synapse? Join a busy room, leave, but still continue to get presence updates about some of the people who were in that room? Is this just an unfortunate side effect of decentralization? My (admittedly uninformed) presumption would be that if no users on my homeserver are joined to any rooms, then the only traffic hitting my server should be client connections. I shouldn't see federation traffic if I'm not actively federating. Am I just misinformed on that point?
Further to that point, from a design perspective shouldn't presence requests expire after some period unless renewed? Or perhaps there should be two classes of presence subscription - implicit and explicit. If a user joins a room, that join implicitly requests presence info of the users in the room - but that implicit request expires after some period (perhaps every reconnection to the room performs a renewal?). If a user subscribes to another user's presence, that explicit request might not expire. Either that or leave it to the client to periodically renew presence subscriptions for users in an address book or something (just spitballing). It just seems like there should be a finite life to these things to cover the "didn't get the message" case.
Second question: is there a way to ask Synapse to explicitly send an 'unpoll: @bob@somewhere.tld' message to the sending server to let it know that my server no longer wants to see presence updates for that user (or even better, all users from that server) unless or until the next poll. I have tcpdump running on my homeserver and can see the user IDs being informed about, I could probably let that run for a bit and then parse out the usernames to send to a script that sends 'unpoll' messages. I just don't know if there is some facility in Synapse to do that, or if such a script would have to be created whole cloth (and I'm guessing would have to use the same keys, etc. to authenticate the unpoll request?).
I asked around in #matrix:matrix.org and didn't get any bites, so I thought I would try here to see if someone might have an informed opinion on this case. Thanks!

Closing comment (2019-05-24T13:56:16Z): this is probably a dup of #2514

and/or #3962