How many blocks, portals, and switches can be placed in a world?

shadowda · 2017-10-28 18:08:42

I have a world that refuses to save and i believe it has to do with the number of blocks, portals, and switches because the world can not be saved even when placed into another world via ee editor.

Xfrogman43 · 2017-10-28 18:40:04

count all the blocks, then delete 1 by 1 until you get your answer

Gosha · 2017-10-28 18:55:40

world can't be saved when it's weight is more than 500 KB

Blocks are not a big deal. switches are a bit worse, they have to hold 1 extra number

Portals are the worst. They have to hold 3 extra numbers (rotation, id, target)
that's why i suggest you to make as portal-free level as possible

Swarth100

Xfrogman43 · 2017-10-28 19:20:49

Gosha wrote:

world can't be saved when it's weight is more than 500 KB
Blocks are not a big deal. switches are a bit worse, they have to hold 1 extra number
Portals are the worst. They have to hold 3 extra numbers (rotation, id, target)
that's why i suggest you to make as portal-free level as possible

how much is 1 block? 1kb?

Gosha · 2017-10-28 20:06:40

Xfrogman43 wrote:

how much is 1 block? 1kb?

You wouldn't be able to fill 25x25 world then.

Each block stores:
layer, x, y, id. 4 numbers.

Portals store 7 numbers
layer, x, y, id, pid, ptarget, rotation.

portals weight twice as much as original blocks.
I might create a program which allows you to see how much space does the world take. (please tell me if somebody has already done that)

hummerz5, Xfrogman43, mutantdevle, Swarth100, Jedpogi

N1KF · 2017-10-28 20:38:27

Xfrogman43 wrote:

how much is 1 block? 1kb?

To add onto what Gosha wrote:

Since the layer is (or should be) a simple on/off detection system, that's 1 bit. Since X and Y go up to 299 (plus 0) in a huge world, that's nine bits each after rounding up. So we have 18 bits, 19 bits total so far. I don't know how many blocks we have but I'm going to be on the safe side and assume we have over 512. That means another 10 bits, for a total of 29. Also, let's assume that the block is some door/gate set at 999. That's an extra 10 bits.

39 bits is far fewer than a single kilobyte, which is 1000 bits.

Now let's look at portals: if we assume that portals have the same layer, coordinates, and ID system that gives 29 bits. There are four rotations, taking up 2 more bits. If I recall correctly, portal IDs go up to 99999. That's 17 bits, or a whopping 34 bits accounting for both portal ID and target (not that you'd need so many IDs).

That's still only 75 bits, meaning you would need over 10 portals to take up a single kilobyte assuming that block data is decently compressed. However, I can only imagine how beastly signs are at sucking up data.

John · 2017-10-28 21:02:41

Gosha wrote:

Portals are the worst

I'd say signs are the worst.

hummerz5, Latif, mutantdevle, LukeM, Dencc, Swarth100, Raon, Jedpogi

SirJosh3917 · 2017-10-28 21:31:55

what's the world ID out of curiosity?

hummerz5 · 2017-10-28 21:54:55

just popping in to say that n1kf might be taking a right proper analytical approach except if the world system hasn't changed in the previous two years then you're wrongly assuming that the storage method is streamlined. Technically you could store 0-299 as so many bits, except the PlayerIO system is only going to allow Int32 or Int64 so you'll have overhead there.

Further, this information isn't stored simply as byte data that you have defined in a certain order. World data has a key/value sort of structure, meaning that if nothing else you know that there is more space given to making the data more robust (?) and comprehensible. So your points are interesting and theoretical but not really applicable.

Also, while each block is stored individually as an entry of its own... the exact same block is not. So while having three portals of different IDs will give you three different entries, three of the exact same portals (rotation, target, ID, not position obv) will actually store more efficiently in the system.

So in short, the rule of thumb that complicated blocks make the worlds more bulky isn't quite correct. Unique blocks stored in worlds make bigger bulk.

N1KF, LukeM, Tomahawk, Swarth100

LukeM · 2017-10-29 01:26:07

Just summarising, and adding a few extra points:

Each instance of a block takes 2 bytes if in the top left 256x256 blocks in the world, otherwise it takes 4 bytes

Each normal type of block used takes an extra 8 bytes
Each number of number block or morph of morphable used (switches, spikes, etc) takes an extra 12 bytes
Each portal ID / rotation / type combo used takes an extra 20 bytes
Signs use a lot (more than anything else), to know how much though I would probably need to do either a lot of testing, or have insider PIO info

The limit is 500 KB (500 000 bytes), although there is some overhead for world settings / info / whatever, so the limit for blocks is probably slightly below that

Just going to put a big citation needed here, as most of this is just from some experimentation I did a while ago

Edit: Corrected some values

Edit 2: As pointed out by Tomahawk, this is just in theory, and in practice there is a lot more overhead than I thought
A rough estimate for the size of each type of block should probably be roughly 4x what I say here.

N1KF, hummerz5

Gosha · 2017-10-29 06:07:38

N1KF wrote:

Since the layer is (or should be) a simple on/off detection system, that's 1 bit.

Nah, it's a legit int32
In fact, you can place blocks on other layers (but I won't say how [might crash some bots if abused] and they won't be visible)

N1KF · 2017-10-29 06:14:41

Gosha wrote:

N1KF wrote:
Since the layer is (or should be) a simple on/off detection system, that's 1 bit.
Nah, it's a legit int32
In fact, you can place blocks on other layers (but I won't say how [might crash some bots if abused] and they won't be visible)

That seems really inefficient...

edit:

hummerz5 wrote:

Technically you could store 0-299 as so many bits, except the PlayerIO system is only going to allow Int32 or Int64 so you'll have overhead there.

If we have silly PlayerIO limitations, why not get rid of layer data altogether and put all blocks, regardless of layer into a single variable? I suppose that theoretically, you could have all different number variations of doors and gates as separate block IDs though since I'm not a programmer that might just make the universe implode on itself.

hummerz5 · 2017-10-29 16:09:43

N1KF wrote:

If we have silly PlayerIO limitations, why not get rid of layer data altogether and put all blocks, regardless of layer into a single variable? I suppose that theoretically, you could have all different number variations of doors and gates as separate block IDs though since I'm not a programmer that might just make the universe implode on itself.

so I'll skip a step for you and perhaps the next guy can answer this question:

why don't we just put everything into one byte array where we control all the serialization?

Well that kinda defeats the purpose of some BigDB tools. But there is a good point there. I don't think anyone really checks out a BigDB entry to just get one piece of information about blocks. They're there for the whole world data so it wouldn't be a terrible loss if access there was more complicated.

But yeah, the layer thing would save a small amount of space. That's not really the issue here. Again, we don't literally have just seven or so int32 spaces for portals. You have that and whatever other overhead is required. But yes, there's only one block that I know of that we actually have in both layers, the empty block. Since that's not encoded into world data we could do that I suppose. Then you'd just have to remember to keep track of layers whenever accessing the data. A bit more of a pain I guess.

Tomahawk · 2017-10-29 18:51:19

As Hummerz said, we can guess at the file sizes all day but really we have no idea how the values are padded, indexed and so on; a PlayerIO byte array might have 3 bytes of real data and 30 more bytes in the back.

It would be interesting to have access to a BigDB world object in plain 0's and 1's, which you could treat as a black box and test how the total size changes when you add certain blocks to the level.

To answer OP's question with some bad calculations:

A completely empty world has no mapdata whatsoever. Adding a single normal block will create an Int32 (4 bytes) for the block ID, an Int32 for the layer, and 2 byte arrays with 2 bytes of data in each, for the coordinate.

That's 12 bytes of useful data.

Adding more blocks of the same ID will add only 4 more bytes per block to the mapdata.

Therefore, 500KB = 124,998 normal blocks of the same ID, which would almost fill a 500x250 world. Given that BigDB has trouble with 400x200 worlds, it's safe to say that normal blocks aren't the issue. What about portals and switches?

A switch needs 16 bytes. 500KB = 31,250 unique switches, but there can only be 1000 unique switches (ID 0 to 999). Yes there's other switches, doors and gates but to really stack those bytes, we need portals.

A portal needs 24 bytes. 500KB = 20,833 unique portals, and there can be 4,000,000 unique portals (1000[ID] * 1000[Target ID] * 4[Rotation]). 8 million I suppose, because invisible portals.

20,833?!! That's smaller than a 145x145 square! Let's try it!

*5 mins later*

Can confirm that an empty world with 21,025 (that's 145x145) unique portals in it, will not save. So how many will save?

*20 mins later*

50x50 = 2,500 unique portals = 60KB, saves fine.
100x100 = 10,000 unique portals = 240KB, does not save! Or takes longer than a minute to save; I got bored waiting.
75x75 = 5,625 unique portals = 135KB, will not save.
60x60 = 3,600 unique portals = 86.4KB, will save! Takes a good few seconds though.
65x65 = 4,225 unique portals = 101.4KB, will save.
70x70 = 4,900 unique portals = 117.6KB, will not save. Waiting longer doesn't seem to matter; it'll either save within 5 seconds or not at all, for me.
67x67 = 4,489 unique portals = 107.736KB, will save.
68x68 = 4,624 unique portals = 110.976KB, will save. It's taking 2-3 seconds to deserialise this mapdata.
69x69 = 4,761 unique portals = 114.264KB, will save.
70x70 again, will not save.
69x69 again, will save. Kek.

So there it is. Somewhere between 114.264KB and 117.6KB of front-end, accessible mapdata is the limit. Maybe. That's a significant amount of overhead on portals if the maximum size in BigDB is 500KB. [I doubt EE would have trouble saving a world with one of every normal block in it, so the overhead added by those three extra portal parameters must be big when you've got a lot of portals.]

Maybe on a rainy day I'll revisit this and narrow it down until a single portal is the difference between saving and not saving. Then I'll add a letter to the world title or change the edit key or do something else unrelated to the mapdata, and maybe it'll change the outcome. Who knows? I wonder what the raw size of an empty world is.

We can blame Benjaminsen for designing an encoding with so much redundancy, but I suppose EE had smaller worlds, fewer normal blocks and far fewer special blocks in his era.

Post the world ID, Shadowda. I wanna see what you've filled it with.

hummerz5

Gosha · 2017-10-29 19:05:20

Tomahawk wrote:

A portal needs 24 bytes. 500KB = 20,833 unique portals, and there can be 4,000,000 unique portals (1000[ID] * 1000[Target ID] * 4[Rotation]). 8 million I suppose, because invisible portals.

that's false.

numbers don't just stop at 1000.
little secret: portals ids and targets use Int64. Each one requires 16 bytes. (portal ids can be from 0 up to 18446744073709551614. and if ee uses unsigned int (which it does i believe))
(even if you set your portal's id to 0, it will still use 16 bytes. they all will just be empty)

just like i said, portals are horrible.

also, ee uses chunks. (i believe they are 16x16, but i might not be right)

when you place a block in the empty world to 5x5. it will belong to 0x0-15x15 chuck area.
it will create a chunk for you and place a block there.
so
when you place 2 block next to each other in the same chunk - in total they will weight less than those 2 same blocks placed on other parts of the world because in the second case world would have to create another chunk.

also also, because of the chunks each blocks requires only 1 byte. (from 0 to 15)
then you can just take chunk's position and add block's position to get block's position in the world

drunkbnu · 2017-10-29 19:44:26

Friendly reminder that kilobytes on Player.IO are actually kibibytes (KiB). The server can save objects with a maximum size of 500 KiB, which equals to 512.000 bytes.

And another friendly reminder that decimal kilobytes use the symbol kB, with a lower-case k. If you see KB, chances are they're binary, not decimal.

Tomahawk · 2017-10-29 20:13:46

Gosha wrote:

little secret: portals ids and targets use Int64. Each one requires 16 bytes. (portal ids can be from 0 up to 18446744073709551614. and if ee uses unsigned int (which it does i believe))

LIES!! But it's weird.

The Portal ID and Target ID values you receive in "init" are Uint32's, but the highest portal ID you can place with a bot is 2,147,483,647, which is the max value of an Int32.

For some reason EE also lets you place negative portal IDs, up to -2,147,483,648. So you can place portals with Int32 values.

A portal with ID -1 will show up as -1 when you hover over it, but in "init" this seems to wrap around to the maximum Uint32 value, 4,294,967,295.

Negative portals work fine in-world, so someone's messed up their data types. Maybe the client uses Int32 while the server uses Uint32.

I thought portal IDs and switch IDs had the same 999 limit, but it doesn't change my maths. The KiB/KB does if it's true, but not by much - by 1 portal, in fact!

Gosha · 2017-10-29 20:28:23

yea, in fact it is a limit. i was wrong

it's fun that after resetting the world negative numbers drop to 0

it's weird.
ee lets you to place a negative value portal
but it doesn't save negative value nor warps around to positive (2,147,483,647 < x < 4,294,967,295)

LukeM · 2017-10-30 00:13:41

So I guess BigDB entries have much more overhead than we thought? The byte array serialisation is starting to look more attractive

I have no idea why I didnt notice that earlier For some reason I was thinking EE saved something similar to the init message, but I guess it probably stores all the names of the blocks properties, as well as the actual data...

XxAtillaxX · 2017-10-30 07:14:22

I'll likely contact Player.IO and inquire about increasing the (from what I can gather) artificial BigDB limitations.

I don't really feel like writing a vector level format, and the serialization that exists currently is relatively good although obviously still full of redundancies.
In the event that they're unable to, I'd much rather run it through the deflate algorithm, which would entail a fraction of the current size (at the very least, a ~50% reduction.)

hummerz5 wrote:

why don't we just put everything into one byte array where we control all the serialization?

You could very well do so, and that'd definitely be the appropriate approach towards it, rather than the current one which involves individually partitioning block types when property differences aside from x/y/x1/y1 arise.
In an optimal approach, you'd effectively be forced to design the algorithm to account for separate properties whilst maintaining a low overhead (i.e. message(-pack)) and a low redundancy, thereby involving compression.

I think you'd get away with solely implementing the latter; it'd be good enough™ in the vast majority of worlds, although likely not so much for the really verbose story-driven worlds with a metric ****-ton of signs with unique text in.

destroyer123 wrote:

So I guess BigDB entries have much more overhead than we thought?

Tomahawk wrote:

We can blame Benjaminsen for designing an encoding with so much redundancy,

Tomahawk wrote:

It would be interesting to have access to a BigDB world object in plain 0's and 1's

I wouldn't blame him entirely for this one, for the same reasons you've mentioned.

I wouldn't really measure with Protocol Buffers, they seem to have a ton of overhead with nested dictionaries from what I've experienced.
If you're talking about the internal storage of documents though, I can only imagine what horrors lie beneath the surface.
I've suspected it of being similar to the BigDB exports; essentially, modified JSON encoding which attempts to achieve similar goals as BSON in storing byte arrays, although in a very poorly thought through implementation.

If you're curious, I dare you to play around with it - the issue is pretty **** obvious and it's infuriatingly moronic in yet a highly amusing way.

Charlie59876EE · 2017-10-30 17:03:33

Gosha wrote:

I might create a program which allows you to see how much space does the world take. (please tell me if somebody has already done that)

if you use EEditor and load a world, then save the world as a .eelevel file, you could then find the space of that file on your computer

mutantdevle · 2017-10-30 17:05:09

Charlie59876EE wrote:

Gosha wrote:
I might create a program which allows you to see how much space does the world take. (please tell me if somebody has already done that)
if you use EEditor and load a world, then save the world as a .eelevel file, you could then find the space of that file on your computer

That wouldn't be accurate though as some of the file size is specific to EEditor. I'm also not entirely sure that they would store blocks in the same way.

Gosha · 2017-10-30 17:11:01

Charlie59876EE wrote:

Gosha wrote:
I might create a program which allows you to see how much space does the world take. (please tell me if somebody has already done that)
if you use EEditor and load a world, then save the world as a .eelevel file, you could then find the space of that file on your computer

EEditor uses different method of storing.

mutantdevle, Offensive Ray, Latif

Official Everybody Edits Forums

#1 2017-10-28 18:08:42

How many blocks, portals, and switches can be placed in a world?

#2 2017-10-28 18:40:04

Re: How many blocks, portals, and switches can be placed in a world?

#3 2017-10-28 18:55:40

Re: How many blocks, portals, and switches can be placed in a world?

#4 2017-10-28 19:20:49

Re: How many blocks, portals, and switches can be placed in a world?

#5 2017-10-28 20:06:40

Re: How many blocks, portals, and switches can be placed in a world?

#6 2017-10-28 20:38:27

Re: How many blocks, portals, and switches can be placed in a world?

#7 2017-10-28 21:02:41

Re: How many blocks, portals, and switches can be placed in a world?

#8 2017-10-28 21:31:55, last edited by SirJosh3917 (2017-10-28 21:37:08)

Re: How many blocks, portals, and switches can be placed in a world?

#9 2017-10-28 21:54:55

Re: How many blocks, portals, and switches can be placed in a world?

#10 2017-10-29 01:26:07, last edited by LukeM (2017-10-30 00:16:09)

Re: How many blocks, portals, and switches can be placed in a world?

#11 2017-10-29 06:07:38, last edited by Gosha (2017-10-29 06:08:36)

Re: How many blocks, portals, and switches can be placed in a world?

#12 2017-10-29 06:14:41, last edited by N1KF (2017-10-29 06:30:08)

Re: How many blocks, portals, and switches can be placed in a world?

#13 2017-10-29 16:09:43

Re: How many blocks, portals, and switches can be placed in a world?

#14 2017-10-29 18:51:19

Re: How many blocks, portals, and switches can be placed in a world?

#15 2017-10-29 19:05:20, last edited by Gosha (2017-10-29 19:15:56)

Re: How many blocks, portals, and switches can be placed in a world?

#16 2017-10-29 19:44:26, last edited by drunkbnu (2017-10-29 19:50:07)

Re: How many blocks, portals, and switches can be placed in a world?

#17 2017-10-29 20:13:46

Re: How many blocks, portals, and switches can be placed in a world?

#18 2017-10-29 20:28:23

Re: How many blocks, portals, and switches can be placed in a world?

#19 2017-10-30 00:13:41

Re: How many blocks, portals, and switches can be placed in a world?

#20 2017-10-30 07:14:22, last edited by XxAtillaxX (2017-10-30 07:15:18)

Re: How many blocks, portals, and switches can be placed in a world?

#21 2017-10-30 17:03:33

Re: How many blocks, portals, and switches can be placed in a world?

#22 2017-10-30 17:05:09

Re: How many blocks, portals, and switches can be placed in a world?

#23 2017-10-30 17:11:01

Re: How many blocks, portals, and switches can be placed in a world?

Board footer