Thursday, February 12, 2015

Cheating Latency



In our last posts we have been dedicating our efforts to reduce the client perceived latency. Let’s check what   we have  focused:
-          We have configured TCP to flush data as soon as it has been generated.
-          We decided to mix TCP & UDP whenever is possible.
-          We create control layers to sync client and server to get a smooth gameplay.
-          We made our own Object Serialization .
All points above are about  technical issues. Now, let’s turn inside our app and let's check if we can enhance its performance.
Summarizing how the server works  will show us some clues about which point can be improved. Let’s look inside:
-        The server and the remote clients wait, initially,  the same gap of time (tick) for each turn.
-          The server wakes up each time tick and consolidates the players movements sent previously by the remote clients.
-        The server sends back a message containing the consolidated movements and tells every client if they have advanced or delayed related to the objective lag between them.
-          The remote clients make some adjustments on the client side to compensate changing latency related to the server.
-          In order to make this work, clients begin their tick before the server in order to compensate each own latency plus certain buffer.
Now, let’s think what we can do to improve this algorithm:

First Enhancement: Try to send  consolidated movements once all player movements have arrived.
As we have seen, the  server  waits until the end of the tick to send back consolidated movements to the client. That’s how it should be, you may think. Sure? Is it strictly mandatory to wait until the end of the server tick if we already have all movements?
Let’ see... We have nothing to do once all movements are received, at server side, except waiting to the conclusion of the tick. So, if we advance the message containing the consolidated movements,  remote  clients will receive sooner that movements and they will paint non own movements before.
As a result, remote clients will see a more fluid and dynamically game and that’s one of our main goals.
            Risks? If the end client  tick time is so close  to the server end tick , we will be in risk of arriving late for the following movements. That’s where our algorithm, described in our last post, works. The following client tick will be shorter than usual in order to keep the same gap related to the server. If it happens very often  we will only feel a faster game and, in fact, this is what we want. Anyway, we always can wait a minimum of time before sending back consolidated movements. In our case, Snake the Net, we have not implemented this guarantee, except at LAN modes,  because we want a fast game and unfortunately, the latency itself introduces more time that we wanted at minimum.
We implemented this approach with sensible results and I recommend to apply similar strategies wherever  is possible.

Second Enhancements: Extra time for the lazy clients
  Let’s think in the opposite way. What if we have lazy clients? What if we add some extra milliseconds to the server tick and then we tell him to recover that time by reducing his tick client? By doing this, we avoid to correct our clients despite he is a really lazy client. We make him to do a fast tick but, in the other hand, we avoid to correct him. In our case this enhancement  help  us by reducing corrections and improving stability. Of course, you have to control if a lazy client is actually a disconnected client. Again, sense common. If he is late several consecutive times, we will disconnect him and replaced by a bot, that they always arrive at time ;-)

2 comments:

  1. Hi,

    I see you have made a lot of improvements to get rid of the latency issue.

    I want to propose a change to your architecture, to reduce the latency even more.

    Assumptions taken by reading what you have described in the history of posts:

    - There is a complete and synchronized model of the status of the game in the server side.

    - The game is a multi-player on-line game.

    - Every user movement is being sent from the client to the server, then the collision algorithm runs in the server and the updates broadcast to every client.

    - The latency you are trying to improve is in fact, the client perspective perceived latency, so you want to improve the user experience.

    (Note I might be over-simplifying the problem)

    Having the assumptions above as valid, the type of latency improvements could be grouped in three types.

    Reducing the size of data sent to the server.

    - Here you have made a lot of improvements by making your own Object serialization for example

    Tweaking the transmission protocol used depending on the type of message.

    - This has been also improved by mixing UDP and TCP.

    Reducing the number of times your message is transmitted.

    - This is by design always two (client perspective latency), one transmission to send the movement, and a second transmission to broadcast the updates to all your clients once you have executed the collision algorithm.

    Proposal: Broadcast every movement from every client to all the connected clients directly.

    Server responsibility:
    - Maintain the list of connected clients/users.
    - Synchronize the speed of the game (using a music simile, acts as a drummer in a band).

    Client responsibility:
    - Apply collision algorithm.
    - Broadcast movements to all connected clients.
    - Keep a cached list of connected clients/users once downloaded from the server when joining the game.
    - Apply speed tweaks when the player gets a “turbo” (this is something might be delegate to the server, perhaps a deeper understanding of the design might help)

    I think I need more time to develop a bit more my proposal, and for sure a more detailed read of the posts from my side, so I apologize if I have over-simplified too much or taken wrong assumptions.

    To close my proposal, I want to thank you for sharing your architecture design and challenges, awesome job guys!!!.

    ReplyDelete
  2. Thank you, so much for your comments. It's a nice approach to study and we are going to think about it.

    ReplyDelete