Official Everybody Edits Forums

drunkbnu

Formerly HG

Joined: 2017-08-16

Posts: 2,306

A small implementation to reduce block message size

I just decided to experiment with bit-wise operators, and thought of a way to reduce the amount of elements from the server's "b" message.

Ignoring the arguments, each block within "b" has 4 important elements: Layer, X, Y and ID. Each element is handled by the server as a signed integer (32 bits). 4 integers would then equal to 128 bits. However, if the attributes are not to reach such high values, we can compact them into 1 unique integer.

Assuming that none of them will ever get past 65536, we can save those elements in a single, unsigned long integer (64 bits).

00000000 00000000   00000000 00000000   00000000 00000000   00000000 00000000
Layer               X                   Y                   ID

Each 16 bits represent a block attribute. We can read specific sections of the long integer to get the desired block attribute, using the shift operators to move the bits to our advantage.

This is how we can save the block data into the unsigned integer:

public static ulong SaveBlock (int layer, int x, int y, int id) {
  return ((ulong) layer << 48) + ((ulong) x << 32) + ((ulong) y << 16) + (ulong) id;
}

Here's a method that easily allows getting the attributes from the integer:

public static void ParseBlock (ulong block, out int layer, out int x, out int y, out int id) {
  layer = (int) (block >> 48);
  x = (int) (block << 16 >> 48);
  y = (int) (block << 32 >> 48);
  id = (int) (block << 48 >> 48);
}

This might initially look like a micro-optimization. But when it comes to bot-assisted levels that depend on placing blocks very fast, this size reduction plays an important factor. Users with connection issues should be able to handle block placement spam, instead of losing connection.

▼Block Save Example

int layer = 0, x = 400, y = 200, id = 2500;
ulong block = SaveBlock (layer, x, y, id);
Console.WriteLine("Block: " + block);

Output should be:

Block: 1718000028100

▼Block Parse Example

ulong block = 1718000028100;
ParseBlock (block, out layer, out x, out y, out id);
Console.WriteLine("Layer: " + layer);
Console.WriteLine("X: " + x);
Console.WriteLine("Y: " + y);
Console.WriteLine("ID: " + id);

Output should be:

Layer: 0
X: 400
Y: 200
ID: 2500

MartenM

Member

From: The Netherlands

Joined: 2016-03-31

Posts: 978

Website

Re: A small implementation to reduce block message size

What about the additional arguments?

---

Ingame: marten22        My steam: MartenM

drunkbnu

Formerly HG

Joined: 2017-08-16

Posts: 2,306

Re: A small implementation to reduce block message size

What about the additional arguments?

They would be handled normally, after the block.

The message would be like:

{ "b", block, arguments }

LukeM

Member

From: England

Joined: 2016-06-03

Posts: 3,009

Website

Re: A small implementation to reduce block message size

In theory this is all good, but I doubt that it's the size of the message that's the problem, there's probably orders of magnitude more overhead from the Internet protocols, as well as whatever PlayerIO does with messages, so the overall performance increase would probably be barely noticeable.

den3107

Member

From: Netherlands

Joined: 2015-04-24

Posts: 1,025

Re: A small implementation to reduce block message size

Personally I'd prefer actually sending 4 numbers (also make the API more consisted vs compressing it into something like this).
You could achieve the same result with 4 unsigned shorts (or just make it a short and have a range of -32,768 to 32,767, in case you might want a couple negative numbers).

I personally feel like this would also work way better (4 separate vs 1 long) when taking special blocks in account. What would you want to do in that case?
Have 1 argument as "meta" (or something) containing this int64, or have everything just parsed into one 'big' byte array?
In case of the latter, parsing things like signs becomes a lot more tedious that it should be.

Just my 10 cents, ignore if you don't like it.

drunkbnu

Formerly HG

Joined: 2017-08-16

Posts: 2,306

Re: A small implementation to reduce block message size

As arguments are unpredictable, they can be sent without any special handling.

The reason for the suggestion, was that it's impossible to send integers other than 32-bit and 64-bit ones. I assume they could easily send short integers if they were possible to send, to save bandwidth.
And as I mentioned and showed with the examples, it's actually very easy to work with those long integers. I doubt it would become an issue for future programmers, unless they rely on old tutorials instead of the protocol.

Zumza

Member

From: root

Joined: 2015-02-17

Posts: 4,656

Re: A small implementation to reduce block message size

I can't see any benefit on doing what you proposed.

Let's track down with facts:

it's impossible to send integers other than 32-bit and 64-bit ones

PlayerIO has some mechanisms that compact the message length in case of integers.

Taking from your example, a message made of ("b", 0, 400, 200, 2500) would be sent as 84 c1 62 80 05 01 90 04 c8 05 09 c4, so the packet length is 12 bytes.

A message made of ("b", 1718000028100) would take 10 bytes, 81 c1 62 05 01 90 00 c8 09 c4.

So taking your example your solution saved 2 bytes of data.

▼How did I get this result / How can you test it by your own?

I used PyPlayerIO(https://github.com/Zumza123/PyPlayerIO) a minimalistic rewrite of the PlayerIOClient in Python.

This code produced the mentioned output.

from playerio.message import Message
from playerio._serializer import Serializer

from binascii import hexlify

data = Serializer.serialize_message(Message('b', 0, 400, 200, 2500))

hexlify(data)
len(data)

HOWEVER BigDB can't store shorts indeed. That's why world data positions are stored in a ByteArray. A position is stored in 4 bytes(int sized), the same way you're proposing.

In conclusion, your idea is already implemented where is needed.

---

Everybody edits, but some edit more than others

Zumza

Member

From: root

Joined: 2015-02-17

Posts: 4,656

Re: A small implementation to reduce block message size

Also not related to the idea but with it's implementation instead of:

public static void ParseBlock (ulong block, out int layer, out int x, out int y, out int id) {
  layer = (int) (block >> 48);
  x = (int) (block << 16 >> 48);
  y = (int) (block << 32 >> 48);
  id = (int) (block << 48 >> 48);

}

This is more elegant (in my opinion) and makes easier to understand the memory layout you're using:

public static void ParseBlock (ulong block, out int layer, out int x, out int y, out int id) {
    id = block & 0xFFFF;

    block >>= 16;
    y = block & 0xFFFF;

    block >>= 16;
    x = block & 0xFFFF;

    block >>= 16;
    layer = block & 0xFFFF;
}

---

Everybody edits, but some edit more than others

XxAtillaxX

Member

Joined: 2015-11-28

Posts: 4,202

Re: A small implementation to reduce block message size

I'd strongly suggest doing an actual comparison with the serialized messages.

The amount of bits you're allocating at the moment is far too many, and exceeds the serialized length of a typical block packet.

Which is to say, the approach you listed would be a larger packet than the current one.
If you decrease the amount of bits allocated, you could perhaps accomplish your task.

I don't know how much of an impact it'd be though, probably not enough to be worth it.

---

*u stinky*

John

Member

Joined: 2019-01-11

Posts: 2,009

Re: A small implementation to reduce block message size

The Stone brothers and I did this with BlockWorks.

---

Gosha

Member

From: Russia

Joined: 2015-03-15

Posts: 6,211

Re: A small implementation to reduce block message size

I used this system in my eejs,  cool

benedani

Member

Joined: 2015-02-19

Posts: 98

Re: A small implementation to reduce block message size

How about a 6-byte array? Something like this:

0000000l
xxxxxxxx
xxxxyyyy
yyyyyyyy
iiiiiiii
iiiiiiii

l = layer
x = x position
y = y position
i = block id

Heck, could probably make it a 5 byte array by doing this:

xxxxxxxx
xxxxlyyy
yyyyyyyy
iiiiiiii
iiiiiiii

Use AND and multiplication to get the correct values using the byte array and to create a byte array with necessary information. Anyways, the real issue with "b" is... a lot of packets being sent repeatedly, instead of bigger, fewer packets. (Which I'm 100% positive is the cause of the disconnections, as I have no problems sending huge byte arrays in my testing, and the message is "There were a lot of outstanding packets") Something like "bb" could be made:

"bb", repeat for all blocks [<byte array>, <necessary args, either client figures this out or ends with a separator>]

(bb stands for bulk blocks, but I guess you could give it your own name.) It would solve so many problems.

Edit: alternative solution if you really need the 16-bit ranges (6 bytes):

xxxxxxxx
xxxxxxxx
yyyyyyyy
yyyyyyyy
liiiiiii
iiiiiiii

(Gets rid of block IDs 32768 to 65535, which are unused anyway.)

Gosha

Member

From: Russia

Joined: 2015-03-15

Posts: 6,211

Re: A small implementation to reduce block message size

iirc there are no blocks that can be placed both on background and foreground.

So why are we even sending layer? Server can calculate block's layer based on id (and it already does it)

if 500 <= x < 1000
then it's background
otherwise it's foreground

Xfrogman43

Member

From: need to find a new home

Joined: 2015-02-15

Posts: 4,174

Re: A small implementation to reduce block message size

iirc there are no blocks that can be placed both on background and foreground.

So why are we even sending layer? Server can calculate block's layer based on id (and it already does it)

if 500 <= x < 1000
then it's background
otherwise it's foreground

but there are fg blocks that are in the 1,100+

---

thanks zoey aaaaaaaaaaaand thanks latif for the avatar

Gosha

Member

From: Russia

Joined: 2015-03-15

Posts: 6,211

Re: A small implementation to reduce block message size

iirc there are no blocks that can be placed both on background and foreground.

So why are we even sending layer? Server can calculate block's layer based on id (and it already does it)

if 500 <= x < 1000
then it's background
otherwise it's foreground

but there are fg blocks that are in the 1,100+

just like i said

500 <= x < 1000

aka

function getLayer(id) {
   if(id >= 500 && id < 1000) return "background";
   return "foreground";
}

LukeM

Member

From: England

Joined: 2016-06-03

Posts: 3,009

Website

Re: A small implementation to reduce block message size

if 500 <= x < 1000
then it's background
otherwise it's foreground

but there are fg blocks that are in the 1,100+

That's why Gosha said 500 <= x < 1000

Xfrogman43

Member

From: need to find a new home

Joined: 2015-02-15

Posts: 4,174

Re: A small implementation to reduce block message size

if 500 <= x < 1000
then it's background
otherwise it's foreground

but there are fg blocks that are in the 1,100+

That's why Gosha said 500 <= x < 1000

i just read everything wrong, excuse me while i go die

---

thanks zoey aaaaaaaaaaaand thanks latif for the avatar