Live removal of issue comments using htmx websocket

[anbraten]

First step into "realtime" updating UI using htmx with ws extension as discussed in bigskysoftware/htmx#2287 and #25661

Concept

By using htmx we use a websocket to send dom elements matched by #id to clients to update specific parts of the UI.

Some examples where this could be used

• Issue comment updates: A dom element of the issue comment is send when the content was changed or a reaction was added etc. The client would just replace that element and the new comment would be visible.
• User got a new notification: We render the template for the notification bell & counter and send it to the user. In addition we send a list item for the notification page if the user is currently on that page.
• Issue comment removed: We send a special empty dom element that removes a comment from the issue page.

Note

As rendering comments was changing a few templates I started implementing the concept for live updates
for removing a comment for now to focus on the websocket logic. Other updating logic will be straightforward to add > in followup PRs.

How it works in detail:

• a tab / client connects via websocket to the server /-/ws
• the client sends on dom.load: { action: "subscrib", data: { url: "http://localhost:3000/anbraten/test/issues/1"}} to the server to let it know it is interested in updates relevant for this page
• another client triggers some action (for now deletes a comment)
• the notifier kicks and checks for all locally connected sessions:
  • if their url is relevant for the issue comment and
  • if the user of that session is allowed to receive that notification data
• the websocket_notifier sends the updated dom element (matched based on #id) via websocket

Screencast.from.2024-02-16.17-50-20.webm

Using a pub/sub system the notifier events are also sent to other servers which forward them again to all relevant & locally connected clients:

TODO

• listen to events like comment adding (with permission handling)
• send new issue comments templates through websocket
• send sth like url from client to server via websocket to filter necessary events (the url could be used to detect that only updates for a specific issue and general updates like new notifications are relevant)

websocket logic based on #26679 and #27806

[anbraten]

I will wait with further development until #28880 has a final decision for now. Just wanted to do this POC to see and show the team if it could be used for the discussed realtime topics as well.

[lunny]

Wouldn't we say htmx will be used on simple situation? I think this page is very complex.

[yardenshoham]

I agree with lunny, this is more complex than I imagined but hey, if it works that's awesome. Do we really have 2 way communication here though? Maybe SSE fits better for this use case

[silverwind]

I agree with lunny, this is more complex than I imagined but hey, if it works that's awesome. Do we really have 2 way communication here though? Maybe SSE fits better for this use case

We need a Websocket in any case as SSE is too limiting:

• max. 6 connections limitation per browser is a big problem when opening many tabs
• no 2-way communication
• SSE is seldomly used and therefor likely buggy depending on browser, websocket is a more stable and well-supported API

[anbraten]

Is it fine to hook into services/notify to get updates about changes like the added comments?

[lunny]

Is it fine to hook into services/notify to get updates about changes like the added comments?

I think it should be. You need to implemente a new notifier like notification.

[lunny]

Maybe a notification icon or notification list is a better start to introduce the WebSocket.

[wxiaoguang]

Maybe a notification icon or notification list is a better start to introduce the WebSocket.

To be honest, I agree with it.

And share my thoughts here:

1. Is it fine to introduce websocket now and still keep the legacy "/user/events"?

   • IMO there should be only one, if we'd like to use websocket, we should refactor the existing code first before introducing duplicate mechanisms.

2. Is it possible to use websocket to update the comment UI?

   • At the moment, my answers is somewhat possible but very difficult to make it really work
   • Take GitHub as an example, their "UI refreshing" sometimes works, for many cases not, so I always have to "reload" the page to read new contents.
   • As a large team as GitHub, they haven't made it right yet, do we have the manpower to make it better than GitHub?

[hiifong]

Regarding this problem, I found a good project. I don’t know if it will help us achieve it.

https://github.com/lesismal/arpc

[hiifong]

First step into "realtime" updating UI using htmx with ws extension as discussed in #2287 and #25661

the websocket extension not found

[silverwind]

Is it fine to introduce websocket now and still keep the legacy "/user/events"?
IMO there should be only one, if we'd like to use websocket, we should refactor the existing code first before introducing duplicate mechanisms.

Imho we should just nuke all the EventSource/SSE code. It's messy, broken and I don't think it ever worked correctly. I'd also appreciate if we had a simple way to test server-sent messages on the devtest pages.

Take GitHub as an example, their "UI refreshing" sometimes works, for many cases not, so I always have to "reload" the page to read new contents.

GitHub has the complexity that their WebSocket is in SharedWorker, maybe that is a source of bugs. I don't recommend for the first implementation to use SharedWorker. It seems like a half-baked browser API.

[anbraten]

Maybe a notification icon or notification list is a better start to introduce the WebSocket.

As explained before I would suggest small steps & PRs here. Adjusting the notification area or the bell icon would require touching frontend code that is not really related to a functional websocket update system.

Is it fine to introduce websocket now and still keep the legacy "/user/events"?

I don't think it is a problem. Both parts do not conflict and actually it could be pretty smart to test and stabilize the websocket first with UI parts that wont break the exisitng UX for the beginning. Worst case some hiccup would currently not update the UI and the user would not even notice it.

At the moment, my answers is somewhat possible but very difficult to make it really work

@wxiaoguang I've updated the issue description to explain the concept / idea again. Seems like it wasn't clear enough. Adding it for several events (issue comment added, PR description updates etc) takes time, but in general I can't seem why this could not be done. Main task would be to align templates so that you can easily render parts of them. For example to update an issue comment it would be required to be able to render only one comment. I had the comment updating / adding working already, mainly removed it again to keep changes low and focus on the concept.

the websocket extension not found

@hiifong Updated the link.

Imho we should just nuke all the EventSource/SSE code.

That would be the next step / PR I would suggest.

[silverwind]

I guess I can make a PR to nuke EventSource. I don't usually like PRs that only remove functionality, but we have to take this step by step and now is a good time to do it given that the 1.24 release is a while off.

[wxiaoguang]

I don't think it is a problem. Both parts do not conflict and actually it could be pretty smart to test and stabilize the websocket first with UI parts that wont break the exisitng UX for the beginning. Worst case some hiccup would currently not update the UI and the user would not even notice it.

I would suggest to only use one approach to avoid unnecessary technical debt. At least, eventsource + websocket uses one more connection to the server, which consumes more resources.

You could take a look at my PRs: https://github.com/go-gitea/gitea/pulls?q=is%3Apr+author%3Awxiaoguang+is%3Aclosed , many of them are "fix this", "fix that", "refactor xxxx". I have spent too much time on legacy unmaintained code. That's why I strongly prefer to improve the whole code quality and improve maintainability.

Adding it for several events (issue comment added, PR description updates etc) takes time, but in general I can't seem why this could not be done. Main task would be to align templates so that you can easily render parts of them.

That's the real challenge: how to correctly "align", there are too many details (UI component init, markdown math render, PR conversation layout, etc), it is somewhat possible, but I do not think we have that manpower, and it makes it much difficult (or blocks) to improve/refactor the issue page in the future: every small change needs to keep the WS module working correctly. In other words, "Return on Investment (ROI)" is poor.

---

Overall I think introducing websocket is good, while I think these problems could be addressed/commented at the moment:

1. Replace legacy EventSource system.
2. At the moment only do "comment deleting" and "UI notification (could be another PR)", do not touch the "newly added or updated comments" unless there is a well-designed complete solution.

What do you think?

---

Oh, one more thing: should we really use htmx websocket? Does it have limits? Will it fully satisfy our requirements in the future? It seems that it is difficult to make htmx call our JS code properly.

My intuition tells me that Gitea is too large and complex, maybe it needs its own websocket frontend module (handler).