[Alpha Library] WorldSaver

LukeM · 2017-01-20 15:22:53

Just thought I would post a topic on saving worlds, as I dont think there is an accpeted format yet.
If we all agreed on a single format, it would allow for so many things, like using a bot like eeditor to create minigames for a shift type bot, which would make bot making soooo much easier!
For this to work, we would need the input of many bot-makers so that it would suit all of their needs, so if anyone has a particular change, or feature they would want, then please say.

Ive now created a library that impliments this format in a fairly user friendly way, which can be found here.
The download page also contains some addons:
ConnectedWorld can be used to more easily keep track of blocks etc.
WorldArea can be used to section a world into parts, and use each of those parts as if it was a normal world.
The source code for this library, along with a small test bot, can be found here

Here is a short list of requirements that I would like in this format:

Be able to edit without any other program, (probably be openable by text editors e.g. notepad++)
Be easily convertable to block messages
Be easily convertable from block messages
Allow for block arrays of any size to be saved
Allow for all types of block
Be likely to work in future updates
Have a fairly small file size

Here is a short explanation of a possible format (im open to changes):
Saves to json files, so that it can be opened using text editors
The format:

{
  "format": "standard_1",    A key to allow for possible backwards compatability, and different formats
  "fglist": [        A list of blocks, in the format that they would be sent to playerIO
    9,              Can just use an id for simple blocks
    [9],            Would be the same as the block above
    [385, "hi", 0], A sign saying 'hi'
    [242, 0, 1, 2]  A 1-2 portal
  ],
  "bglist": [
    500             A background with id 500
  ],
  "foreground": [
    [1,1,1,1,1],    A 5x5 hollow square of basic blocks
    [1,0,0,0,1],    with a portal in the right wall, and a
    [1,0,0,0,4],    sign in the bottom middle of the centre
    [1,0,3,0,1],    
    [1,1,1,1,1]     Note: 0 = air, 1+ = items of blist
  ],
  "background": [
    [1,1,1,1,1],    Filled with bg 500
    [1,1,1,1,1],    
    [1,1,1,1,1],    If there were more than 10 block types,
    [1,1,1,1,1],    then there would be spaces padding the
    [1,1,1,1,1]     array, so it is easy to read
  ],
  "data": {         Optional data table that can be used to store bot-specific data
    "bot key": {    A seperate table for each bot, so that extra data can be added, and will not be overwritten
      json data
    },
    "another bot key": {
      more (seperate) json data
    }
  }
}

Formats:

standard_1: The format shown above
simple_1: As above, but using ids in the foreground and background arrays instead of referances to a block table (not yet implimented)

The files for this are not terribly big, but are bigger than just storing it as its own file type, however, they have the advantage of being highly compressable (a 90%+ reduction in file size for large structures), so would be a fairly good way to store large numbers of block arrays

So, if you have any ideas, have a better way to do it, or think doing this is just a bad idea altogether, please reply, so I can make any changes necessary

Changes: (all times are in GMT)
20/1/17 15:00 - den3107 asked about extra data, e.g. entities stored with the world so I added a data table
20/1/17 15:00 - also added a simpler format for easier use
24/1/17 22:00 - replaced the format 'basic' with 'standard', as basic and simple could lead to confusion
25/1/17 20:00 - uploaded an alpha version of this library
26/1/17 08:40 - replaced blist with the more logical fglist (as den3107 suggested)
26/1/17 08:40 - also replaced blocks with foreground, and backgrounds with background

kubapolish, SirJosh3917

den3107 · 2017-01-20 15:35:51

As I assume this concept is supposed to work with every bot:
What if a bot requires certain extra data? For example requires structures, or "entities", where with those entities should be specified what parts can or can't be overwritten.

Just a thought.

And no, I personally have no idea as of yet how to allow such kind of extra data, although I haven't given it a lot of thought either.

LukeM

XxAtillaxX · 2017-01-20 15:37:13

Well, for a start, there's the immediate issue with storing blocks in a matrix, that being the fact that some blocks contain additional properties, e.g. portals, coin doors, and so on.
Additionally, you don't exactly need any definition on the proper property order, they already are in the DatabaseObject by default, and even if they weren't, they could be permutated easily.

The past and current administration does not comply to standards and if you were to look at the client sources, it's been maintained and developed by amateurs at best.
This is to say that there is not, and very likely never will be an accepted and standardised format for anything.

Swarth100

LukeM · 2017-01-20 15:50:46

den3107 wrote:

As I assume this concept is supposed to work with every bot:
What if a bot requires certain extra data? For example requires structures, or "entities", where with those entities should be specified what parts can or can't be overwritten.
Just a thought.
And no, I personally have no idea as of yet how to allow such kind of extra data, although I haven't given it a lot of thought either.

I was thinking of this just being a way to save the actual blocks, but now you say it, it would probably be a good idea to include that too.
I think adding an additional extra data table that can be stored with the world would be a good idea, and would solve most problems, allowing this to work with more bots

Edit: Done , tell me what you think about the data table idea

XxAtillaxX wrote:

Well, for a start, there's the immediate issue with storing blocks in a matrix, that being the fact that some blocks contain additional properties, e.g. portals, coin doors, and so on.
Additionally, you don't exactly need any definition on the proper property order, they already are in the DatabaseObject by default, and even if they weren't, they could be permutated easily.
The past and current administration does not comply to standards and if you were to look at the client sources, it's been maintained and developed by amateurs at best.
This is to say that there is not, and very likely never will be an accepted and standardised format for anything.

The additional properties is solved by using the block definition table at the start, rather than just using the ids, as this means that a number in the matrix represents an index in the table, which can contain extra values
However, it might be an idea to also support ids directly in the array, because that might simplify it for new users
Also, im not suggesting this as a feature of EE, this would just be a library that can be used to help with making bots, (and could possibly could convert the serialised world data to a C# version of this format)

drunkbnu · 2017-01-20 19:52:21

What about saving the world serialized data you receive with "init" or "reset" on a file, and then de-serialize it with Krock's world deserializer when loading the file.

LukeM · 2017-01-20 23:21:08

HG wrote:

What about saving the world serialized data you receive with "init" or "reset" on a file, and then de-serialize it with Krock's world deserializer when loading the file.

That would work, but it would be hard to edit without a specific program to do it, this way you can edit it even without a program to do so, for example, if you wanted to use a part of a world in this format, for a shift level or something, and couldnt be bothered to download a program to allow you to edit it.

Edit: also, it might be difficult to detect the types of extra block data stored as text (you could create a file type to save it properly, but that would be annoying). This would mean you would have to have hard-coded block types, which would probably break often.

SirJosh3917 · 2017-01-23 21:31:33

Should add a format to this current setup that allows for massive data compression, to avoid the nasty amount of space it takes up when it's in plain text.

LukeM

medeia · 2017-01-23 21:35:18

SirJosh3917, kubapolish

SirJosh3917 · 2017-01-23 22:10:02

I made a library that contains ( so far ) everything specified.
The output of the library is this:

{"format":"basic_1","blist":[9,[9],[385,"hi",0],[242,0,1,2]],"bglist":[500],"blocks":[[1,1,1,1,1],[1,0,0,0,1],[1,0,0,0,4],[1,0,3,0,1],[1,1,1,1,1]],"backgrounds":[[1,1,1,1,1],[1,1,1,1,1],[1,1,1,1,1],[1,1,1,1,1],[1,1,1,1,1]],"data":{"bot key":{"json data":"ahdiowahdioahwiod"},"another bot key":{"more json data":"..."}}}

Neatified would be this:

{
  "format":"basic_1",
  "blist": [
  9,
  [9],
  [385,"hi",0],
  [242,0,1,2]
  ],
  "bglist": [
  500
  ],
  "blocks": [
  [1,1,1,1,1],
  [1,0,0,0,1],
  [1,0,0,0,4],
  [1,0,3,0,1],
  [1,1,1,1,1]
  ],
  "backgrounds":[
  [1,1,1,1,1],
  [1,1,1,1,1],
  [1,1,1,1,1],
  [1,1,1,1,1],
  [1,1,1,1,1]
  ],
  "data": {
   "bot key": {
     "json data":"ahdiowahdioahwiod"
    },
    "another bot key": {
     "more json data":"..."
    }
  }
}

The github link is herewill be here shortly

LukeM

LukeM · 2017-01-24 16:17:45

ninjasupeatsninja wrote:

Should add a format to this current setup that allows for massive data compression, to avoid the nasty amount of space it takes up when it's in plain text.

As this is currently, just zipping it compresses it massively, as standard gzip removes most memory used by repeated strings.
I managed to get a 96% compression rate on my files for mars mine which use a similar format.
This would also mean that it would be very easy to compress and uncompress files, possibly even directly from the program

EDIT:
If this is not enough, there could also be a format that is not easily readable, but stores the same data, for those who would need this (we could possibly get incredibly compact if we packed the bytes, and made the length of the tags inversely proportional to the frequency that they are used, but im not sure it would be needed)

ninjasupeatsninja wrote:

I made a library that contains ( so far ) everything specified.
...
The github link is herewill be here shortly

Thanks , ill probably make my own version at some point that will allow for a simpler interface with bots, possibly with some automatic neatification as well, but this works very well as an example of how this format would work

SirJosh3917

LukeM · 2017-01-25 21:05:52

I've finished a (sort of) working version of this library, it can be found here
This is in no way finished, so expect there to be many bugs, and even crashes, but I thought I'd upload it anyway to get feedback

I have uploaded the source code to github, along with a quick test program I wrote, so you can look through it and see some examples if you want.

In the test, I connected a bot to my home world, and passed the init message to the library, (printed the top left block to show that it is the init parseing that is the problem), and saved the world. Here is the output json file:

{
	"format": "standard_1",
	"blist": [
		[119],
		[9],
		[35],
		[16],
		[210],
		[208],
		[311],
		[374,"PWAIjKWOiLbEI"],
		[385,"Learn how to play in the tutorial!",0],
		[34],
		[255],
		[385,"Welcome to your home!",0],
		[96],
		[281],
		[20],
		[388],
		[97],
		[389],
		[21],
		[120],
		[36],
		[100],
		[385,"Use the block building menu in the bottom to change your world",0],
		[300],
		[121]
	],
	"bglist": [
		[576],
		[531],
		[566],
		[570],
		[507],
		[511],
		[512],
		[567],
		[571],
		[574]
	],
	"blocks": [
		[1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,25,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,3,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,13,13,13,13,13,17,17,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,11,0,0,14,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,12,0,0,0,0,0,16,18,18,18,18,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,10,3,3,10,3,3,3,3,3,3,3,21,20,0,0,0,0,0,0,10,3,3,21,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,4,4,4,4,4,4,4,4,4,19,0,20,0,22,0,0,0,0,15,4,4,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,15,4,4,4,4,19,0,0,20,0,23,0,0,0,0,0,15,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,4,4,4,4,0,20,0,10,3,3,3,21,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0,20,10,4,4,4,4,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0,20,0,4,15,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,0,8,0,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,5,5,5,6,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
		[2,0,6,6,6,6,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20,0,0,0,0,0,0,0,24,24,24,24,24,24,24,24,0,2],
		[2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,1,1,1,1,1,1,1,1,3,2],
		[2,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,1,1,1,1,1,1,1,4,2],
		[2,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,1,1,1,1,1,4,4,2],
		[2,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,2],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]
	],
	"backgrounds": [
		[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,4,4,4,4,4,9,9,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,4,4,4,4,4,9,9,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,4,4,4,4,4,9,9,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0,0,0,0,0,0,0,0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,0,0,0,0,0,0,0,0,0,0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,0,5,0,0,0,0,0,0,2,2,2,2,2,2,2,2,2,2,6,6,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,5,0,0,7,0,5,5,2,2,2,2,2,2,2,2,2,2,6,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0,7,5,5,2,2,2,0,0,0,0,2,2,2,2,2,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0,5,2,2,2,2,2,6,0,6,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0,5,2,2,2,2,2,2,6,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,6,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,10,10,10,10,10,10,10,10,2,0],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,10,1,10,10,10,1,10,2,2],
		[0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,10,10,10,10,10,2,2,2],
		[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],
		[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
	],
	"data": {
	}
}

This, as far as I can tell, is in the correct format, apart from no whitespace padding the array, and a 1 in the top left corner, which im unsure of the cause of (if you know, please tell me) - This is fixed now, it was caused by world positions now taking 2 bytes, instead of 1

SirJosh3917

den3107 · 2017-01-26 00:35:51

Just a thought, not sure if it's relevant or even a problem...
I understand that you first create an indexed object which you can refer to in the actual final 2D array, to prevent duplicate data with stuff like portals.
However, simple structures, like background or anything that really is just the block ID now just get an unnecessary extra reference.
Meaning (in the form of an example)...
Instead of straight out knowing on position [5, 10], is the gray basic block. You see the id 1. Which you in term then first have to translate to the gray basic block.

Once again I don't have a solution for this problem (at least none that satisfies me), but isn't it problem to, indeed, use an indexer/reference of some sort for the complex tiles, and just use the direct required simple data for the simple tiles.
A (what I think is) bad solution, would be a prefix. if it has prefix "c" (or any prefix at all), it's a complex tile, thus it's a reference. If it has prefix "s" (or maybe no prefix works too), it's a simple tile, which already contains all the data.
Now I think this is a bad idea since you'd first have to check for certain things that possibly could be done in a more self-explanatory way (see next solution).

Another possible solution would be to separate the 2d array into 2 2d arrays (so 2 arrays for fg and 2 for bg).
One array would only hold reference tiles (empty slots in the array mean it's either simple or just an empty tile) and the other would only hold simple tiles, where again, empty slots either mean it's in the other array, or is simply empty.
Now obviously the problem with this implementation would be that you effectively iterate over 2 arrays, instead of just one. You could check both arrays in the same 2 for-loops, but still you have to check 2 values instead of just a single one.

Anyway...
This suggestion would probably ruin the human readability of the file, which I assume is something you aim to keep (even though I'd prefer writing an editing tool if I find I often have to change those save files, instead of editing it manually all the time).
I'm also probably just thinking about unnecessary micro-optimizations and thinking way too complicated and feeling way too smart after 2 youtube videos about object-oriented programming...

Anyways: good catch on adding the format parameter for backwards "compatibility".
And sorry for the late mention: "blist" would more logically be called "fglist", as it's in the same style as "bglist" and thus "blocks" also be called "foreground".

LukeM · 2017-01-26 09:38:59

den3107 wrote:

Just a thought, not sure if it's relevant or even a problem...
I understand that you first create an indexed object which you can refer to in the actual final 2D array, to prevent duplicate data with stuff like portals.
However, simple structures, like background or anything that really is just the block ID now just get an unnecessary extra reference.
Meaning (in the form of an example)...
Instead of straight out knowing on position [5, 10], is the gray basic block. You see the id 1. Which you in term then first have to translate to the gray basic block...

Although it would be nice to have a sory of hybrid format, im not sure that there is an easy way to do that. The current solution is to just have 2 seperate formats, one if you want to store blocks by id, and one if you want to store them by reference.

den3107 wrote:

...Once again I don't have a solution for this problem (at least none that satisfies me), but isn't it problem to, indeed, use an indexer/reference of some sort for the complex tiles, and just use the direct required simple data for the simple tiles.
A (what I think is) bad solution, would be a prefix. if it has prefix "c" (or any prefix at all), it's a complex tile, thus it's a reference. If it has prefix "s" (or maybe no prefix works too), it's a simple tile, which already contains all the data.
Now I think this is a bad idea since you'd first have to check for certain things that possibly could be done in a more self-explanatory way (see next solution)...

Another problem with this would be that you would have to then store the blocks as strings, if you wanted to comply with the json specification.

den3107 wrote:

...Another possible solution would be to separate the 2d array into 2 2d arrays (so 2 arrays for fg and 2 for bg).
One array would only hold reference tiles (empty slots in the array mean it's either simple or just an empty tile) and the other would only hold simple tiles, where again, empty slots either mean it's in the other array, or is simply empty.
Now obviously the problem with this implementation would be that you effectively iterate over 2 arrays, instead of just one. You could check both arrays in the same 2 for-loops, but still you have to check 2 values instead of just a single one...

The problem with this would be that there would be a lot of redundant data, so any advantage you'd get from using a block ID rather than a reference, you'd probably lose from having to store 2 arrays.

den3107 wrote:

...Anyway...
This suggestion would probably ruin the human readability of the file, which I assume is something you aim to keep (even though I'd prefer writing an editing tool if I find I often have to change those save files, instead of editing it manually all the time).
I'm also probably just thinking about unnecessary micro-optimizations and thinking way too complicated and feeling way too smart after 2 youtube videos about object-oriented programming...
Anyways: good catch on adding the format parameter for backwards "compatibility".
And sorry for the late mention: "blist" would more logically be called "fglist", as it's in the same style as "bglist" and thus "blocks" also be called "foreground".

The library already internally uses fg and bg, so I should probably change it in the file... But that shouldnt be a problem, because we have the version string, so (even if somebody is already using this bot) there shouldnt be any compatability problems

LukeM · 2017-01-26 10:49:51

I just added GetArea and SetArea functions for the World format in my version of the library, to allow for easier getting and setting of areas (and therefore saving and loading of areas)

XxAtillaxX · 2017-01-26 14:08:47

How does your format outperform World? World is completely agnostic about additional block parameters and does not require them to be specified.

Jesse

LukeM · 2017-01-26 15:22:48

XxAtillaxX wrote:

How does your format outperform World? World is completely agnostic about additional block parameters and does not require them to be specified.

Im not exactly sure how World works (I haven't heard of it before), but from quickly looking at the GitHub page, and forum post, it seems like it is mostly for saving a whole world (as a backup etc), wheras, although this should be able to do that (but won't save settings), it is more aimed towards saving just parts of a world, and having optional bot data stored with that. I'll look into World more later, this is just my first impression.

This library doesnt require additional block parameters either, im not sure how World does it, but this is also fairly futureproof, and, given some good programming skills, should allow most bots using it to be fairly futureproof.

This also doesnt do anything to do with the EE connection, other than parsing the init and load messages, so leaves the rest up to whoever is making the bot.

Edit: also, could you give an example output of World, so I can more easily compare it to this

Tomahawk · 2017-01-26 15:58:48

XxAtillaxX wrote:

How does your format outperform World?

Except that yours only reads from the DB and not from init, so the map that you download won't always be exactly what's in the world at the time of downloading.

You could probably shoehorn whatever Initparse outputs into that Block class, but until then I'd have to check that the world isn't open before trusting whatever the DB has on it.

XxAtillaxX · 2017-01-26 16:07:10

Tomahawk wrote:

XxAtillaxX wrote:
How does your format outperform World?
Except that yours only reads from the DB and not from init, so the map that you download won't always be exactly what's in the world at the time of downloading.
You could probably shoehorn whatever Initparse outputs into that Block class, but until then I'd have to check that the world isn't open before trusting whatever the DB has on it.

That's correct, however I'm explicitly asking about the structure of the data within the format, rather than the data contained within. Apologies if that wasn't clear enough.

destroyer123 wrote:

Im not exactly sure how World works (I haven't heard of it before), but from quickly looking at the GitHub page, and forum post, it seems like it is mostly for saving a whole world (as a backup etc), wheras, although this should be able to do that (but won't save settings), it is more aimed towards saving just parts of a world, and having optional bot data stored with that. I'll look into World more later, this is just my first impression.

It certainly does save the entirety of a world, of which I've embraced from Git versioning.
The benefits are that data can be contrasted without requiring a large index to rebuild the entire structure, with the comparably insignificant cost of storage capacity.

I'll attempt to explain what I had said about property agnosticism in a less abstract way.
For example, let's say that to send a portal, you must supply the rotation, the id and the target.

Within the Worlds structure, this is laid out with defined property names, whereas in your format they are supplied as purely integer values.
How would you guarantee that these integer values are being supplied in the correct order, if, and when, the property order of a block changes and it is no longer succinct?

If you are going down the route of proposing versioning to the library itself to handle these changes, it will be awfully tedious to upgrade and then contrast older formats, sooner than later you'll find yourself doing this constantly, and people will be required to update the library accordingly, each time Everybody Edits updates.

destroyer123 wrote:

This library doesnt require additional block parameters either, im not sure how World does it, but this is also fairly futureproof, and, given some good programming skills, should allow most bots using it to be fairly futureproof.
This also doesnt do anything to do with the EE connection, other than parsing the init and load messages, so leaves the rest up to whoever is making the bot.
Edit: also, could you give an example output of World, so I can more easily compare it to this

The very structure of Init has changed much more often than the database structure. The ladder certainly isn't unexpected, nor is it as tedious as ensuring functionality with the former.
Sure thing, here's an example. As is evident, the structure remains nearly identical to the database object, the only differences being valid JSON and inheriting the internal versioning number from PlayerIO.

The raw output is 468KB, 44KB compressed.

LukeM · 2017-01-26 18:21:40

XxAtillaxX wrote:

...
The very structure of Init has changed much more often than the database structure. The ladder certainly isn't unexpected, nor is it as tedious as ensuring functionality with the former.
Sure thing, here's an example. As is evident, the structure remains nearly identical to the database object, the only differences being valid JSON and inheriting the internal versioning number from PlayerIO.
The raw output is 468KB, 44KB compressed.

Ok...
So, ill split this into two parts, one about compatability, and one about the actual format.

Compatability:
Yes, old worlds would be broken after a change to the order of parameters, but this library should still work for new worlds.
It seems to me, that in your Sync class, you are also relying on the parameters being in the same order in the database as they would be in the 'b' message, so both would share the same problem where older worlds would no longer work (I could be completely wrong)
The init parsing works similar to InitParse, but works forwards, checking for a sequence of types the same as the id, layer, and arrays of Xs, Ys. This means that as long as these 4 types are not changed, and as long as there isnt a similar sequence of types in any of the blocks data values, then this will continue to work, even if there is an update.
The library also leaves all the things that are likely to change to the person using the library, e.g. world width and height, which are passed to the init parsing function seperately to the message

Format:
The main advantages of this format over the World format are as follows:

This format is more easily readable, as your's converts a list of Xs and Ys to a string, which is not in a readable format, this also shows the data in an array format, so you can more easily visualise how the world would look
This formats file sizes are smaller, as your's seems to contain a lot of redundant data
This format allows for arbitrary data to be stored with the world, which can be used by bots using this library

The main disadvantages are as follows:

This format requires most of the file to be deserialised before you can even start placing blocks, so requires a moderate amount of memory, while your's could place blocks while reading the file, which would require very little memory
Your's contains all of the world data, so contains settings etc.

den3107 · 2017-01-27 09:57:18

Do mind that in your format (destroyer), you can still save the world data in the extra Data tag, although there would be no "official" format for that.

LukeM · 2017-01-27 16:13:30

den3107 wrote:

Do mind that in your format (destroyer), you can still save the world data in the extra Data tag, although there would be no "official" format for that.

Well... I guess you could, but that would be more specific to certain bots, as most bots wouldn't need to, so probably wouldn't need an official format

LukeM · 2017-01-28 21:08:26

Made a few simple changes to the library:

Fixed loading of worlds
Added Width and Height properties to the world
Changed World from a struct to a world, to allow for null values, and passing of referances instead of values
Fixed GetArea, as it added the width instead of height to the y position to check if it fits
Removed unused (empty) class

LukeM · 2017-02-24 00:24:54

Ive just made an addon for the library, which can be found here
It is currently in alpha, so expect there to be bugs.

There is a short tutorial on how to use it above the download link, but im going to make a better one soon, so dont worry.
This basically contains code to place blocks in EE, and code to handle blocks that are placed, to keep the World object updated. You do have to make some of the OnMessage yourself, but this had to be the case to prevent a potential memory leak, and to make the library work in future updates. Here is the code you need to add (there is another copy above the download link):

Put this code at the top of your namespace:

using PlayerIOClient;
using WorldSaver;
using ConnectedWorld;

Put this code at the top of your class (you may not need static):

static World world;
static int botID;

Put this code into your OnMessage switch:

case "init":
    botID = m.GetInt(5);
    world = new CWorld(con, botID, m, m.GetInt(18), m.GetInt(19));
    break;
case "reset":
    world.Dispose();
    world = new CWorld(con, botID, m, world.Width, world.Height);
    break;
case "b":
    ((CWorld)world).BlockPlaced(m, 0);
    break;
case "br":
    ((CWorld)world).BlockPlaced(m, 4);
    break;
case "bc":
case "bs":
case "lb":
case "pt":
case "ts":
case "wp":
    ((CWorld)world).BlockPlaced(m);
    break;

'world' can then be used as previously, apart from you dont need to worry about locks, placing blocks, (most) timing issues, or keeping 'world' updated

LukeM · 2017-02-26 13:35:22

Ive now fixed the memory leak in ConnectedWorld.
The only change is that you need to call world.Dispose() after you finish using it.
If you forget to dispose the world, it will continue to place all the blocks in the queue, and once it does, should be automatically disposed by the GC.

Ive also created a new plugin called WorldArea
This can be used to section a WorldSaver World into parts, and use each of these parts as if it were its own world.
This works with ConnectedWorld, so can be used for minigames that only take up a small area of the world, and could possibly allow you to have multiple of the minigame running in the same world, without complicated threading / offset issues.
How to use it:

1. Reference PlayerIOClient, WorldSaver, and WorldArea
2. If you want to use a section of the world for something, put:
    World section = new AWorld(world, x, y, width, height);
    (replacing each with the area wanted)
3. You can now use 'section' as you would a normal World, apart from instead of changing itself, it changes its parent World.
4. This also works with ConnectedWorlds, so you could create a minigame in a certain part of the world, and not need to worry about an offset.

LukeM · 2017-03-02 15:09:46

Just fixed a load of problems with each of the libraries:

WorldSaver- Fixed booleans in the json data
WorldSaver- Exposed the json data reader / saver so it can be used if you need it
ConnectedWorld- Fixed placing areas (didnt work before)
ConnectedWorld- Added a event for when all blocks in the queue have been placed
WorldArea- Fixed a few small problems (nothing much, but it should be mentioned that there is a new update)

Official Everybody Edits Forums

#1 2017-01-20 15:22:53, last edited by LukeM (2017-02-26 13:28:11)

[Alpha Library] WorldSaver

#2 2017-01-20 15:35:51

Re: [Alpha Library] WorldSaver

#3 2017-01-20 15:37:13

Re: [Alpha Library] WorldSaver

#4 2017-01-20 15:50:46, last edited by LukeM (2017-01-20 16:02:06)

Re: [Alpha Library] WorldSaver

#5 2017-01-20 19:52:21

Re: [Alpha Library] WorldSaver

#6 2017-01-20 23:21:08, last edited by LukeM (2017-01-21 00:07:50)

Re: [Alpha Library] WorldSaver

#7 2017-01-23 21:31:33

Re: [Alpha Library] WorldSaver

#8 2017-01-23 21:35:18

Re: [Alpha Library] WorldSaver

#9 2017-01-23 22:10:02, last edited by SirJosh3917 (2017-01-23 22:20:17)

Re: [Alpha Library] WorldSaver

#10 2017-01-24 16:17:45, last edited by LukeM (2017-01-24 16:48:18)

Re: [Alpha Library] WorldSaver

#11 2017-01-25 21:05:52, last edited by LukeM (2017-02-25 12:20:51)

Re: [Alpha Library] WorldSaver

#12 2017-01-26 00:35:51

Re: [Alpha Library] WorldSaver

#13 2017-01-26 09:38:59

Re: [Alpha Library] WorldSaver

#14 2017-01-26 10:49:51

Re: [Alpha Library] WorldSaver

#15 2017-01-26 14:08:47, last edited by XxAtillaxX (2017-01-26 14:09:04)

Re: [Alpha Library] WorldSaver

#16 2017-01-26 15:22:48, last edited by LukeM (2017-01-26 15:23:42)

Re: [Alpha Library] WorldSaver

#17 2017-01-26 15:58:48

Re: [Alpha Library] WorldSaver

#18 2017-01-26 16:07:10, last edited by XxAtillaxX (2017-01-26 16:31:56)

Re: [Alpha Library] WorldSaver

#19 2017-01-26 18:21:40

Re: [Alpha Library] WorldSaver

#20 2017-01-27 09:57:18

Re: [Alpha Library] WorldSaver

#21 2017-01-27 16:13:30

Re: [Alpha Library] WorldSaver

#22 2017-01-28 21:08:26, last edited by LukeM (2017-01-28 21:10:37)

Re: [Alpha Library] WorldSaver

#23 2017-02-24 00:24:54, last edited by LukeM (2017-02-26 13:28:38)

Re: [Alpha Library] WorldSaver

#24 2017-02-26 13:35:22, last edited by LukeM (2017-02-26 13:38:21)

Re: [Alpha Library] WorldSaver

#25 2017-03-02 15:09:46

Re: [Alpha Library] WorldSaver

Board footer