Distributed and Scalable WebSocket Server using Goakt

[civilizeddev]

I am planning to implement a WebSocket server composed of multiple distributed instances using Goakt.

This is a very interesting topic, and I am going to present an idea on how to solve it with Goakt.

I would like to hear what others think about it.

Readings
https://medium.com/lumen-engineering-blog/how-to-implement-a-distributed-and-auto-scalable-websocket-server-architecture-on-kubernetes-4cc32e1dfa45

Steps

1. Browser client requests create a WebSocket connection to one of the multiple instances via a load balancer.
2. In response to the WebSocket connection, a Member actor is created and holds the WebSocket connection as its state.
3. The Member actor is associated with the Members state of one or more Room actors.
4. A singleton service connected to the messaging system finds the Room to route messages and broadcasts the message to its Member actors.
5. When a Member actor receives a message, it sends the message through the WebSocket connection.

This way, a distributed environment WebSocket server can be implemented.

[civilizeddev]

#207

This question is also related. I need a way to send messages to an actor from a message source that is not part of the goakt cluster.

[Tochemey]

@civilizeddev This is an interesting project. I will be glad to help in a any way possible.

[civilizeddev]

I understand that the messages of the actor defined as protocol buffer should be serializable.

So I don't know how to send and receive complex objects such as websocket.Conn through messages.

These can only be injected through the New-() constructor when creating an actor. Is that right?

[Tochemey]

@civilizeddev Every message sent to an actor in Go-Akt is a protocol buffer message. One thing I think you can do is to wrap the complex message into a protobuf as follows:

message WebSocketConnection {
  bytes connection_payload = 1;
}

So you just have to use tools like gob to serialize the websocket.Conn into bytes array and pass it to the message. When receiving the message just deserialize the payload into websocket.Conn using tools like gob.

I hope this helps.

[civilizeddev]

Thank you for your answer. But the socket connection object is generally not a target for serialization. It has many internal fields and is exclusive to its node(pod). Therefore, it can only be injected through the constructor when the actor is created. However, there's one question. When cluster rebalancing occurs, does the actor move to another node? How does rebalancing work with objects like this that cannot be serialized? Must the state of the object consist entirely of fields that can be serialized? If so, am I misunderstanding something?

[Tochemey]

At the moment, sadly, when a node dies or leaves the cluster, actors on that node are not moved to another node. Even though the partition system gets redistributed among the rest of the cluster, we lose those actors because they can not be found. I have an issue to track that feature. The challenge I am having is providing a serialisation of the underlying implementation of Actor.

[civilizeddev]

• https://socket.io/docs/v4/redis-adapter/
• https://github.com/socketio/socket.io-redis-adapter

The functionality I am trying to implement with Goakt from other platforms is like this.

The difference is that it has a repository like redis on the outside, and Goakt is that the cluster itself has information without redis.

The benefits of using Goakt are:

1. it can reduce the cost of adding Redis
2. for scaling, simply increase the nodes in the Goakt cluster.

Does it seem feasible to implement?

[Tochemey]

@civilizeddev I will take a look and advise if necessary.

[Tochemey]

• https://socket.io/docs/v4/redis-adapter/
• https://github.com/socketio/socket.io-redis-adapter

The functionality I am trying to implement with Goakt from other platforms is like this.

The difference is that it has a repository like Redis on the outside, and Goakt is that the cluster itself has information without Redis.

The benefits of using Goakt are:

1. it can reduce the cost of adding Redis
2. for scaling, simply increase the nodes in the Goakt cluster.

Does it seem feasible to implement?

From what I can see based upon the description of the project, an actor model may not be necessary if it is only scalability you are looking at.

[Tochemey]

@civilizeddev I am going to provide an example on how to use Go-Akt to build a websocket server.

[Tochemey]

@civilizeddev I have started some work here

[civilizeddev]

Thank you. This is an excellent example that's easy to understand.

1. I would like to ask for one more concept to be added, please. Adding a concept similar to socket.io's rooms might make it even clearer for people. A room refers to creating a group of multiple WebSocket connections. And messages can be sent to all connections that have joined this room. The participants of the group are sessions scattered across various nodes.

2. One thing I'm curious about is why the term 'broadcast' is used here, and I'm interested in the intention behind it. From what I see, it seems like simply emitting WebSocket messages, so why is it called broadcast?

[Tochemey]

This example is just basic. I hope and expect developer can learn from it add more features or get inspired by it and implement their own web socket server using the actor model. I am not intending to extend it any longer. That will be a separate project. The term broadcast is just to send messages to session. There is no intention behind it. I could have called anything else.

[Tochemey]

@civilizeddev The example has some design floors. I have closed the PR and removed the branch. I will look into a better example to publish. This may take some time because I am swamped with my day work at the moment.

[Tochemey]

@civilizeddev closing this discussion. I hope to a complete implementation of websocket once I have some spare time from day work. :)