Official Everybody Edits Forums

wjg999

Member

Joined: 2017-07-12

Posts: 6

Question about world data

I am trying to parse the world data that I get in the init but the page on github does not document what the numbers between "ws" and "we" are.

From what I am seeing in the array itself it seems to be 4 values for each block type
- block id
- layer
- byte array containing 2 entries per block all even entries giving an unknown value and all uneven entries giving X 
- byte array containing 2 entries per block all even entries giving an unknown value and all uneven entries giving Y

What I mostly want to know is what those 2 unknown values are, they seem to be at 0 for most of the time.

I could very well be wrong with what those 4 values are too, so if they are wrong please correct me.

John

Member

Joined: 2019-01-11

Posts: 2,011

Re: Question about world data

Would this help you?

---

wjg999

Member

Joined: 2017-07-12

Posts: 6

Re: Question about world data

Not really since it does not help me understand what values there are.
As well as it being 2 years old so I am not even sure if that would still be working.
The link to github in that post does help me understand it a bit more so thank you for the post link.

John

Member

Joined: 2019-01-11

Posts: 2,011

Re: Question about world data

Not really since it does not help me understand what values there are.

Well, each different block "instance" is sent, and the byte arrays are the coordinates they are located at. What I mean by "instance" is like the same block with the same parameters. If a block has a different parameter, it is another instance.

As well as it being 2 years old so I am not even sure if that would still be working.

It's designed to be future proof. It won't break when there are new updates.

The link to github in that post does help me understand it a bit more so thank you for the post link.

Yeah that's what I was going for, I thought a guide might've been useful too.

---

LukeM

Member

From: England

Joined: 2016-06-03

Posts: 3,009

Website

Re: Question about world data

This format is repeated for every block in the world, iirc sorted by the first block in the arrays, top to bottom, left to right:

Type: The ID of the block
Layer: The layer of the block, (0 for fg, 1 for bg)
Xs: A bytearray storing 16 bit x positions (by appending the bits of the first and second bytes)
Ys: A bytearray storing 16 bit y positions (same as above)
[Extra arguments are found here, depending on what block it is (see InitParse for a good way to parse these)]

Tomahawk

Forum Mod

From: UK

Joined: 2015-02-18

Posts: 2,847

Re: Question about world data

I meant to write an explanation of the mapdata for Thanel's protocol a while back. Here's a quick one:

Below I've put part of an example "init" message. Ignore the indexes - it's from an earlier version of EE.

▼Init example

//The number in brackets just indicates the type of variable. Ignore it.

  msg[17] = ws  (6)            //Indicates the start of the world data ("world start")

  msg[18] = 44  (1)		//ID
  msg[19] = 0  (0)		//Layer
  msg[20] = System.Byte[]  (7)	//X coords
  msg[21] = System.Byte[]  (7)	//Y coords

  msg[22] = 8  (1)
  msg[23] = 0  (0)
  msg[24] = System.Byte[]  (7)
  msg[25] = System.Byte[]  (7)

  ...

  msg[62] = 242  (1)		//Portal
  msg[63] = 0  (0)		//Layer
  msg[64] = System.Byte[]  (7)	//X coords
  msg[65] = System.Byte[]  (7)	//Y coords
  msg[66] = 1  (1)		//Rotation (D=0, L=1, U=2, R=3)
  msg[67] = 0  (1)		//Portal ID
  msg[68] = 0  (1)		//Target ID

  msg[69] = 381  (1)		//Invisible portal
  msg[70] = 0  (0)
  msg[71] = System.Byte[]  (7)
  msg[72] = System.Byte[]  (7)
  msg[73] = 1  (1)
  msg[74] = 0  (1)
  msg[75] = 0  (1)

  ...

  msg[81] = 43  (1)		//Yellow coin door
  msg[82] = 0  (0)		//Layer
  msg[83] = System.Byte[]  (7)	//X coords
  msg[84] = System.Byte[]  (7)	//Y coords
  msg[85] = 10  (1)		//One extra argument - coins needed to open

Every normal block has 4 values: ID, layer, X and Y coords, but special blocks - like portals, coindoors, any block or decor that can rotate, etc. -  have more. These extra arguments are found right after the two byte arrays, before the ID of the next block in the list.

In order to deserialise (decode) the mapdata, you either have to have a list of the ID of every special block so that you can record the correct parameters, or you can work backwards from "we" (~world end), which is what InitParse does.

The way that the X and Y coordinates are encoded in the byte arrays is difficult to understand if you don't understand binary. Basically, BigDB only has one type of array - byte arrays - and as a byte only has a max value of 255, a single byte cannot be used to store a coordinate, because some worlds are taller or wider than 255 blocks. Instead, pairs of bytes are used to store each coordinate as an Int16 (16-bit integer) - the first byte stores the first 8 bits, and the second byte stores the second 8 bits of the full coordinate, allowing for a max value of 32767.

To put these bytes together to form the coordinate, it's easiest to use two of C#'s bitwise operators, (operators that work with bits in numerical data types), which are << and |

<< or >> are bitwise shift operators and move the bits around in a variable in the direction that they point. For instance, if a byte has the value 00000111 (or 4) and you shift the bits one place to the left (<< 1) you get 00001110. I'm not gonna try and explain this properly - go on Google for that - so I'll explain its relevance to the mapdata.

Likewise the bitwise OR operator does what it sounds like; it "OR"s together two binary strings. 00001111 | 11110000 = 11111111. This is used to join the two bytes together after byte-shifting the first one.

So, some code;

▼The code

//Let's say that uint c contains the value of the index of one of the byte arrays, and that m is the init message.

uint c = 50;

byte[] x = m.GetByteArray(50);

int firstX; 

//Let's put the first coordinate of x into the integer firstX, using bitwise logic.
//But let's do it in several steps:

firstX = x[0];

//Let's say that the first byte in the array (and therefore in the pair of bytes, x[0] and x[1] needed to make the first X coordinate) has the value 1.
//x[0] therefore equals 00000001

//Remember that an "int" in C# is by default an Int-32. This means it has 32 bits, or 4 bytes.
//Now firstX has the value 00000000000000000000000000000001 - 31 0's followed by a 1.
//However, the first byte stores the first 8 bits of the coordinate, so we need to move (shift) it to the left by 8 bits to make room for the second byte:

firstX = firstX << 8;

//Or alternatively, do the last two steps in one line:

firstX = x[0] << 8;

//Now firstX has the value 00000000 00000000 00000001 00000000
//The first byte in the array is in its proper place inside the int. Now we need to use the bitwise OR operator (the vertical pipe, | ) to insert the second byte:

firstX = firstX | x[1];

//Let's say x[1] has the value 5, or 00000101 in binary.
//Now, after the OR operation, firstX has the value 00000000 00000000 00000001 00000101. You can see that both bytes from the array are now in the last two bytes of the integer.
//The second byte was put into the last 8 bits of the integer.
//The first two bytes of the integer remain 0, and are not used.

//Now firstX holds the value of the first x-coordinate in the byte array (which btw is equal to 256 + 4 + 1 = 261)

//And now, all in one line of code:

firstX = x[1] << 8 | x[2];

//Or in a longer line of code that doesn't use the variable x:

firstX = m.GetByteArray(c)[0] << 8 | m.GetByteArray(c)[1];

//Yay!

//That's the hard bit. All that's left to do is loop through the byte array and get all the coordinates out of it. The same for the Y-array (which will be the same length)

Now given that OP asked for an explanation of the serialised map data and not for an explanation of InitParse, I won't bother. There's probably explanations around anyway.

However, I wrote a modified version of InitParse (credits to sugardaddy Processor for the original) to make it more compact:

▼Modified InitParse

<Insert obligatory MIT license giving credits to Proc>

        public void deserialise(Message m)
        {
            uint c = m.Count - 2;
            int x, y;
            Stack<object> args = new Stack<object>();

            bl = new Block[2, worldWidth, worldHeight];

            while (m[c] as string != "ws")
            {
                while (!(m[c] is byte[])) args.Push(m[c--]);

                for (int n = 0; n < m.GetByteArray(c).Length; n += 2)
                {
                    x = m.GetByteArray(c - 1)[n] << 8 | m.GetByteArray(c - 1)[n + 1];
                    y = m.GetByteArray(c)[n] << 8 | m.GetByteArray(c)[n + 1];

                    bl[m.GetInt(c - 2), x, y].ID = m.GetInt(c - 3);

                    if (args.Count != 0) bl[m.GetInt(c - 2), x, y].args = args.ToArray();
                }
                args.Clear();
                c -= 4;
            }
        }

This method is called from within "init" in the message handler. The mapdata is stored in a three-dimensional struct array, bl:

        struct Block
        {
            public int ID;
            public object[] args;
        }
        Block[,,] bl;

worldWidth and worldHeight are the width and height of the world taken from "init", and are public variables.

Note that you will also need to call deserialise() if you receive "reset" (/loadlevel).

Any blocks that are placed after the bot has joined will not be recorded. You'll need to write more code for that (hint: listen for all block message types, "b", "pt", "wp", "bc", "bs", "br", "ts" and "lb").

This post was too long. TL;DR = it's complicated.

---

One bot to rule them all, one bot to find them. One bot to bring them all... and with this cliché blind them.

wjg999

Member

Joined: 2017-07-12

Posts: 6

Re: Question about world data

Thank you tomahawk,

The way that the X and Y coordinates are encoded in the byte arrays is difficult to understand if you don't understand binary. Basically, BigDB only has one type of array - byte arrays - and as a byte only has a max value of 255, a single byte cannot be used to store a coordinate, because some worlds are taller or wider than 255 blocks. Instead, pairs of bytes are used to store each coordinate as a Int16 (16-bit integer) - the first byte stores the first 8 bits, and the second byte stores the second 8 bits of the full coordinate, allowing for a max value of 32767.

That was the bit I was so confused about since it seemed there was 2 numbers per block one of which I did not know what it was for.
So this fully answered the question I was having.

Tomahawk

Forum Mod

From: UK

Joined: 2015-02-18

Posts: 2,847

Re: Question about world data

Hey - I made a mistake in my original modified version of InitParse.

The mistake was in clearing the stack of arguments after assigning them to the first block in the byte arrays.

I've edited the code in my post above, and I'll put it again below:

▼Modified InitParse

▼MIT License

The MIT License (MIT)

Copyright (c) 2015 Simon Farshid (EE Username: Processor)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

        public void deserialise(Message m)
        {
            uint c = m.Count - 2;
            int x, y;
            Stack<object> args = new Stack<object>();

            bl = new Block[2, worldWidth, worldHeight];

            while (m[c] as string != "ws")
            {
                while (!(m[c] is byte[])) args.Push(m[c--]);

                for (int n = 0; n < m.GetByteArray(c).Length; n += 2)
                {
                    x = m.GetByteArray(c - 1)[n] << 8 | m.GetByteArray(c - 1)[n + 1];
                    y = m.GetByteArray(c)[n] << 8 | m.GetByteArray(c)[n + 1];

                    bl[m.GetInt(c - 2), x, y].ID = m.GetInt(c - 3);

                    if (args.Count != 0) bl[m.GetInt(c - 2), x, y].args = args.ToArray();
                }
                args.Clear();
                c -= 4;
            }
        }

Sorry.

EDIT: Fixed it again T_T

uint c needs to be m.Count - 2, otherwise the argument "we" gets added to all instances of the last block in the map data.

There goes my reputation.

---

One bot to rule them all, one bot to find them. One bot to bring them all... and with this cliché blind them.