TCP Packets

[elias4444]

post deleted.

[Riven]

Understand that ANY packet can be lost (yes, even a TCP packet). Location and positioning packets should be sent regularly (typically via UDP, but even TCP can handle it for most games these days), and if one is lost, forget about it and grab the next one.

Heck NO!

You can never 'grab the next one' in TCP. Never! (unless you reconnect..)

[Nate]

First off, if you're getting out of sync, you're doing it wrong. This was a hard lesson for me, as I thought that by doing predictive movement packets and transmitting as little information as possible, that the game would act smoother. Smoother, perhaps it was, but playable? No, it was not.

Predictive movement packets? Also, I don't see how sending a little or a lot of information has much to do with the game being playable. Finally, what did you do different that fixed the problem?

Understand that ANY packet can be lost (yes, even a TCP packet).

Lost TCP packets are automatically resent. At the application level, nothing has to be done for this to happen. If you send something over TCP, it will be delivered and in the order sent, or the connection will die.

Build and use custom packet formats (string a bunch of integer or floats together) - don't even THINK about using serialization for an action game.

Well, stringing a bunch of integers or floats together IS serialization. If you meant not to use a serialization framework, the difference between a fast serialization framework and hand written serialization code (eg java.io.Externalizable) is negligible:
http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking
If you mean not to use the built-in Java serialization, then I agree.

Make sure you use some kind of error checking as well (even if it's just sticking a packet identifier on the front).

"Some kind of error checking" is pretty vague. Error checking for what? A corrupt payload is not something you normally have to worry about.

[Riven]

"Some kind of error checking" is pretty vague. Error checking for what? A corrupt payload is not something you normally have to worry about.

Indeed, there is however a tiny chance that the corrupted checksum matches the corrupted payload. On a busy server, a few monster packets per year are to be expected.

[elias4444]

post deleted.

[Momoko_Fan]

Considering downloads go through TCP, how come files I download from the internet never get corrupted then? If I remember correctly, UDP uses the same checksum system so by that idea, UDP packets should never get corrupted as well. UDP packets are allowed however, to arrive at different order, to be duplicated, or to never arrive at all- that you do need to check for.

[Nate]

The checksum is 16 bit, so isn't fool proof, it is just very unlikely that multiple errors occur in a corrupt packet in such a fashion that doesn't invalidate the checksum. So unlikely, you never need to normally worry about including your own checksum. However, that would be too easy so apparently they made the protocol's checksum optional!

UDP already has the packet length in the header, so no point in also sending it in the payload. It does make sense to send the packet length with TCP because it is stream based, not to detect corrupt packets.

I agree with elias4444 that you can't rely on the network to be fast, and you should definitely limit the speed at which you periodically send. Keeping that in mind, sending has overhead, so you should send a reasonable amount of data all at once (if needed and if possible without unacceptable latency). Eg, and someone please correct me if I'm wrong, sending just a few bytes and sending a thousand bytes is going to take roughly the same time. When you go past the MTU size (typically ~1400 bytes) then TCP has to break your data into multiple pieces, and only then do you get really penalized for including more data. See:
http://en.wikipedia.org/wiki/IPv4#Fragmentation_and_reassembly

[Riven]

I ran into troubles with some TCP packets getting corrupted by a user's router. Luckily, I send a packet-size bit at the front, so I was able to detect this and manually drop the packet. Does that help explain better?

Well, no. Split/merged TCP packets are allowed by the spec, and have nothing to do with a 'corrupting router'. I fear most of your observations are from a general misunderstanding of how TCP works.

If you want to send N bytes, and receive N bytes, you basically have to apply your own logic in your TCP handling code. You never have to drop packets (received bytes) in a TCP stream, because you are guaranteed to receive any bytes that are sent. The only thing that is not guaranteed is the amount of bytes in each bulk read-operation.

[Riven]

And finally, make sure you're using NIO, and not the old network stuff that blocks.

Could you elaborate? Hardly anybody uses NIO, because it's so darn hard. Blocking I/O is 'good enough' for just about every application.

[Orangy Tang]

I ran into troubles with some TCP packets getting corrupted by a user's router. Luckily, I send a packet-size bit at the front, so I was able to detect this and manually drop the packet. Does that help explain better?

I think it shows that you really don't know what you're talking about when it comes to TCP. TCP is a stream protocol, not a packet protocol.

[elias4444]

post deleted.

[Riven]

... big chunk of text ...

I realize you are serious about this, and have put a lot of effort into this, but what you're saying just doesn't make sense. I'm not trying to be offensive, nor arrogant, or whatever, but what you have been experiencing must have been the result of bugs in your own code.

If that 'corrupt router' was as bad as you say it was, it would have been useless, as every application that makes TCP connections, simply assumes the stream is not corrupted, so a router that bad would have made any TCP traffic go bezerk, making things like browsing/mailing/messenging impossible. I really wonder how he could have downloaded your game, with such a bad piece of hardware.

I really think there is some syndrome of 'inexperienced' people that stumble upon a bug and if they can't find the bug in their own code, they tend to blame the libraries that are in use throughout the world. It's like saying there is a bug in ArrayList -- it's ridiculous.

With 12-16 users there is no need whatsoever to use NIO for performance. NIO is actually quite a bit slower than blocking-IO. NIO only shines due to requiring less threads, which you only notice when dealing with >100 of concurrent connections.

Just saying... I run server software with 300 new tcp connections per second, and I've never ever had to drop a 'PAYLOAD' or encountered any corrupted data in a TCP stream. Surely connections get dropped, but no corruption, let alone causing serious problems in a tiny multiplayer session of 16 clients.

Trust me, every problem you had was caused by your own code. You should be grateful too, because that means you can fix it.

[Demonpants]

How about you guys move this into PMs instead of further derailing CyanPrime's Showcase post? I can also split this topic if you so desire.

[elias4444]

...post deleted.

Sorry about your thread Cyan. Hopefully the moderators can clean up my mess.

I learned in college that the blame for miscommunication must fall on the shoulders of the speaker, so I take full responsibility. I've deleted my relevant posts.

[Riven]

How about you guys move this into PMs instead of further derailing CyanPrime's Showcase post? I can also split this topic if you so desire.

Splitting would be nice. Thanks!

[Demonpants]

It has been split. Now go ahead and debate all you want!

[elias4444]

Let the public flogging continue!!!!!! 

[Alan_W]

Could you elaborate? Hardly anybody uses NIO, because it's so darn hard. Blocking I/O is 'good enough' for just about every application.

The nice thing about NIO is the ability to process the data in the main game thread without blocking it.  However it takes a whole bunch of classes and lots of trial (and especially error) to make it work.  All that selector stuff is hard work.  Conventional blocking I/O is easier to set up, but the resulting data needs careful handling to avoid synchronisation problems as you need a receive thread as well as the main thread.  More trial (and once again copious amounts of error) on my part.

[DzzD]

The nice thing about NIO is the ability to process the data in the main game thread without blocking it.  However it takes a whole bunch of classes and lots of trial (and especially error) to make it work.  All that selector stuff is hard work.  Conventional blocking I/O is easier to set up, but the resulting data needs careful handling to avoid synchronisation problems as you need a receive thread as well as the main thread.  More trial (and once again copious amounts of error) on my part.

plz dont be off topic this thread is about chiken, nothing else, no ?

[Riven]

Yeah.

So Demonpants, could you please split this thread right where Alan_W started?

Offtopic:
Alan_W, you're absolutely right, too. It was simply my advise because not only is NIO a can of worms, the architecture also hard to grasp for most people. Doing all I/O on threads and pushing all 'i/o-events' in a queue, where one threads pops off events, is a simple way to ensure there are no synchronization problems.

[ddyer]

Remember that the tcp guarantee of reliable delivery is only operative at the tcp stream
level  where the checksum is checked or generated.  There's a lot of software between your
application and the low level TCP driver, and your only guarantee there is hope and best
wishes.

I run an application that keeps its own checksum at the application level, and still get
checksum errors at rare intervals - currently running at one error per 200 million transactions.
Note that this is an "all causes" count, not limited to transmission errors.  Still, it's sobering
to consider that even "reliable" channels are not.

[Riven]

I run an application that keeps its own checksum at the application level, and still get
checksum errors at rare intervals - currently running at one error per 200 million transactions.

Indeed, there is however a tiny chance that the corrupted checksum matches the corrupted payload. On a busy server, a few monster packets per year are to be expected.

Don't forget RAM and disk errors.

[i30817]

Why things like  ZFS are important no.

Please don't remove my simplicity plz...

Anymore than it already is.

[DzzD]

Don't forget RAM and disk errors.

this is something that have recently afraying me when I discover the "ressource monitor" showing material memory error on window... so many error , up to 12/min for some process ... so much fear that I decided to close this monitor window suddently !  while it stay closed cant see any error and all seems to be ok

[Nate]

The nice thing about NIO is the ability to process the data in the main game thread without blocking it.

Do you guys think it is OK to do NIO selects on the game's render thread? I have been running a separate thread to do network read/writes, then queuing any data read to be processed in the game thread. Which approach would be preferred? I was worried that the network IO might affect the framerate.

[Riven]

Do you guys think it is OK to do NIO selects on the game's render thread? I have been running a separate thread to do network read/writes, then queuing any data read to be processed in the game thread. Which approach would be preferred? I was worried that the network IO might affect the framerate.

In 'C land' network code is (or used to be?) part of the main loop. Everything is easier with only one thread.

It certainly makes code simpler... but networking I/O is rather 'expensive' due to interacting with the kernel, so why not give it its own thread? In the end it doesn't really matter, as you probably have one low-traffic  connection to the server, in the typical multiplayer game.