Official Everybody Edits Forums

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

[Discussion] EE Standardization -- let's all get on the same page

This post will be periodically updated so as to not unnecessarily bump the topic.

Current stage: 1. Preliminary

Stages
1. Preliminary   -- preliminary work item
2. Proposal         -- new work item proposal
3. Preparatory -- working drafts
4. Committee   -- committee drafts
5. Enquiry           -- draft standard
6. Approval        -- final draft standard
7. Publication   -- the standard is published, and ready for use

These stages have been directly copied from https://www.youtube.com/watch?v=UgvDeWFBNNc with minor modifications

Purpose of this document

The purpose of this document is to allow applications, bots, and developers to freely interchange, modify, adapt, and read data created by others. Furthermore, this document aims to produce a uniform API (application programming interface, like commands) so that users do not need to re-learn the core set of commands for every world that they choose to join, and interact with. For example, if one wants to find out what the version of a bot is, they may type "!version" (or something equivalent) and it will always return the version, for every standardized world.

The aim is to reduce isolation among data formats, and to bring the community together to create a universal standard, incorporating advantages and reducing disadvantages. You are not forced to follow this standard.

Committees

There's a possibility that discussion about a particular feature could be enhanced if there are committee's. I propose that there are the following committees. More committees may be added. Note: you do not have a join a committee, and you may join more than one if you wish.

- Bugfix. Fix or report the problems that the testers find.
- Testers. Test everything, try to break something.
- Documentation & Definitions. How should these terms be used?
- Commands (writers, resolvers, and synthesization). What core commands are important to the community?

Current questions

- How should the commands be prefixed? Common prefixes are (separated by commas): !, ., $
- The init message is somewhat messy. Should this be standardized? If so, how, to maintain backwards and forwards compatibility?
- How should long chat messages be displayed? Some ways are:
       - Prefixing each messages with [current page/total pages]
       - Split the message into two separate messages with no prefix
       - Attempt to make the message more concise
- How can one disambiguate which messages is for them?
       - Make each message contain the recipient's username
       - Give each recipient a number and use that to address them

1. Scope

The open EE standards project aims to formalize how rooms are saved, commands used, and what information is available. Everything is up for discussion, feel free to give your input! If you wish to not voice your opinions publically, you may anonymously submit feedback here: http://goo.gl/forms/j2epEj4lzw.

The project is currently underway at https://github.com/Decagon/ee-standardization.

2. Normative References

This document's structure and wording is loosely based on the format in ISO/IEC 8824-1.

3. Definitions

Some of the terms are synonyms of each other, and so only one term should be used in that case. These terms, once disambiguated, must be given a formal definition.

- Level, room, world, place, creation
- Bot, program, app, application, script, command, computer application
3.1 room: ?
3.2 bot: an autonomous computer program

4. Abbreviations

4.1 EE Everybody Edits
4.2 BAL Bot Assisted Level

5. Conventions

5.1 File type (for room saves/archival, and loading rooms)

The current schema is available at https://github.com/Decagon/ee-standardi … d/World.md

Possible downsides:
- The 7-Zip algorithm, while giving extremely good compression, is somewhat slow on older hardware. It may be an issue if one has to compress/decompress many levels at once.

5.2 World Specification (mirrored from GitHub)

5.2.1 Scope
This file defines the specifications for retrieving, saving, and addressing worlds. The schema and file format to save and load room save files is listed below. An easy way to adhere to the standards is to use the included example application for storing and retriving data.

5.2.2 Creation of the world save file

5.2.2.1 The file type

5.2.2.1.1 Protobuf format must be used.

5.2.2.1.2 It must be compressed using 7-Zip, using the default compression settings.

5.2.2.1.3 No whitespace should exist, except otherwise required by the type of data being stored.

5.2.2.1.4 The file extension is ".ee_save"

5.2.2.1.5 The line endings are UNIX

5.2.2.2 The file contents

5.2.2.2.1 The file shall contain only UTF-8 or UTF-16 characters, before compression.

5.2.2.2.2 It should not exceed a total file size of 1.0mb after compression

5.2.2.2.3 All data in the file should adhere to the attached schema.

5.2.2.2.4 No extra data shall be added to the file, unless specified by the world encoder.

5.2.2.2.4.1 If extra data is added, it should either (a) be in a separate file within the same folder as the save file or (b) within the same file, using a compression or tar technique to bind the two files together. It is your responsibility to decode the files prior to processing.

5.2.2.2.5 The file must contain all of the necessary data to rebuild the world. It should be error free.

5.2.2.2.6 The file shall not be edited by any application once it is saved. It may only be read.

5.2.2.2.6.1 If you wish to edit the file, the original copy shall not be modified.

5.2.2.2.7 The file must be able to detect if it was modified.

5.2.2.3 File compatibility

5.2.2.3.1 The file must be compatible with all valid parsers.

5.2.2.3.2 The files should not contain viruses, malware, or exploits.

5.2.2.4 File adaptability

5.2.2.4.1 All files must ensure future compatibility.

5.2.2.4.1.1 Fields such as new blocks should automatically populate without the user updating their application.

5.2.2.4.1.2 In the case that a feature was removed in a later version, it may be ignored. The user must be notified of this change.

5.2.2.4.2 The files should not cause other valid parsers to malfunction.

5.2.2.4.3 If an error is encountered when parsing the file, abort and warn the user.

5.2.2.4.4 The file is greater than 1.0mb

5.2.2.4.4.1 A preprocessing technique to reduce the file size may be used. However, your application is entirely responsible for repackaging the file in the proper format. It must be error-free.

5.2.2.4.4.2 The file may be compressed using a higher compression option using the 7-Zip algorthim.

5.2.2.4.5 The file must be compatible with all major operating systems.

5.2.2.4.6 The file must be verified before the program quits.

5.2.2.5 Parsing

5.2.2.5.1 The parsers must adhere to all standards in this document.

5.2.2.5.2 The parsers must pass all tests* in this repository.

5.2.2.5.3 The parsers must always assume that the data will be in the correct format.

5.2.2.5.4 The parsers must produce valid, readable data that can be read by other parsers.

5.2.2.5.5 If the parser encounters an error, it must stop.
5.2.2.6 Uploading

5.2.2.6.1 The world data should be an exact copy of the contents of the file.

5.2.2.6.1.1 If the user does not have certain blocks that are described in the file, notify the user.

5.2.2.6.1.2 The program may allow the user to choose blocks that are of a similar color or functionality.

When creating the file, it must be in protobuf format, and must be compressed using gzip.

5.3 Commands:
The examples below are in the following format: Description command1,command2... (author1,author2...; empty if default)

These commands are still under discussion (as they can be added to or removed from) and are considered the "core" commands.

There are also commands that begin with / (slash) and so I'm not sure if they should have a synonym.

5.3.1 Administrative tasks
- Ban !ban (Angelbot)
- Unban !unban (Angelbot)
- Clear room !clear (Angelbot)
- Turn on god mode !god (Angelbot)
- Turn off god mode !ungod (Angelbot)
- Kick a user !kick (Angelbot)

5.3.1.1 Levels
- Saving a level !save, .save (DigBot, EE Troll Catcher)
- Loading a level !load
- Deleting a level !delete
- Renaming a level
- Modifying an existing level
- View the save date

5.3.2 Languages
- Changing a language !lang
- Viewing the currently selected language !viewlang
- Reset the language to defaults !resetlang
5.3.3 Guides and help
- Asking for help !help (DigBot)
- Access a guide or tutorial
- Get name of admin
- Check whether a user is in the level
5.3.4 Toggle idle status
- Turn idle on
- Turn idle off
- Check how long someone has been idle

5.3.5 Sending feedback
- Send feedback
- Changing feedback*
5.3.6 Bot specifics
- Version of bot
- How a bot should speak (use {prefix} for bot prefix)
- How much text a bot should say
- When a bot should respond
     - Level saved
     - Errors
     - Debug information
     - Send feedback
     - Placed a block
     - Idle has been disabled
     - Idle has been enabled
- When a bot should not respond
- Homepage of bot
- Where to download bot*
- Where to email or contact the admin

A list of features have been synthesized from the following applications (and not limited to): Titles: Troll Catcher C, {912468} Background, Troll Catcher [BETA] , Online Check, Auto Builder , EEBuilder v1.8, EE Bot by Krock, AngelBot, Super-Jump Auto Kick by AngelBoy, Guest Bomb, Special Account Bomb, EE RPG Bot, IceBot, Icepegasus command bot, Moving Bot, [ELC] Botz!, WriteBot, invisiblocks, [EC]botz!-Account-Bomb!, FillBot, Stalker Bricks, Ban List , Owner-Blocks, [ELC] - EESaver, gravitybot, Make Your Bot, Trolls Be Gone,9crew MAZEBOT, 9crew Dig, watterfall bot, Floddy DigBot "I Can Dig It", EE World Viewer,   Troller Light, EEArtist, EEViewer, Stalker Pro, Fill Tool, Block Replacer, Smileys, WPE Packet filler, Select Packets, EEAnimator - Block Tool, EE Pics, EE Colors, Banlist, CoinCollector, MagicCollector, Abrar11, bot, ZUMZA BOT, Zerk, Templewormbot, and NBot bot. Data courtsey of http://everybodyedits.wikia.com/wiki/Ev … Edits_Bots.

Synthesized list of features:

- Auto edit
- Auto kick level 3 or under; ninjas
- Infinite background changer
- Black List, white list
- Trust list (no draw speed limit)
- Auto removeedit trollers
- Select a player who's god/in your world and give/remove his edit or kick him.
- Troll catcher
- Friend online check
- Level saving, clearing, password changing
- Text writing
- Multi-block snake
- Auto-save
- There is an EE RPG Bot that has commands such as "!fight", "!wear sword", "!buy", "!tutorial", "!start fight (username)" among others.
- Place completely invisible blocks
- Fill words and can replace blocks.
- EESaver
- Floddy Dig Bot "I Can Dig It"
- EE Minimap
- Upload pictures to EE.
- Debug the EE messages.
- Stalker Pro is a bot made to follow you or someone else.
- Allows players to send "broadcast" messages
- "!sendto" command able to use, get the command information with "!whatdoes {command}".
NOTE: some of these features may be used wholly or in part from the original text in the cited link.

Complying to standards

If one chooses to comply with the standards, how does one verify that all of the requirements are met?

- Invite their bot to an empty room, and have the bot compute a checksum for the level (which would involve the cooperation and validation of multiple components at once). If the checksum is valid, then the bot complies with all standards.

Should there be multiple variations of the standard? For example, one may follow the strict standard, while others, who do not wish to implement the entire standard may choose to follow a relaxed one.

Credits:

Thank you to all who have and are helping with this project.

lrussell -- formalization and creation of initial file format

Last edited by Hexagon (Jan 4 2015 7:16:16 pm)

---

???

Vitalijus

Member

From: Lithuania

Joined: 2015-02-15

Posts: 1,384

Website

Re: [Discussion] EE Standardization -- let's all get on the same page

U mean somthing like if someone save the level +1 is someone save level again +1

Hexagon: !levelsaves
Bot: This world was saved 2 times.

?

---

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

You mean something like if someone save the level +1 is someone save level again +1
Command: !levelsaves
Output: This world was saved 2 times.
(Adapted from original text)

I haven't thought about that, but that's actually a really good idea.

---

For example, when I save an EE level (using an application on my computer) it saves to a file. This file is usually only readable by the program that generated it. This means that if I use a different application made by someone else to load my level again using that file, it probably won't work because the file formats are different. I'm proposing that programs may choose to save their level in a specific format, so that it can be read by others.

Of course, this idea is not limited to level saves, but standardization could exist within commands (as you said).

---

???

XxAtillaxX

Member

Joined: 2015-11-28

Posts: 4,202

Re: [Discussion] EE Standardization -- let's all get on the same page

People will save world data in all kinds of idiotic formats, there's no standard and I doubt there ever will be one.

---

*u stinky*

Different55

Forum Admin

Joined: 2015-02-07

Posts: 16,574

Re: [Discussion] EE Standardization -- let's all get on the same page

You could probably adapt the bmp format to your purposes. Set the red value as the block id, and the green value as additional information like portal direction or instrument note or number of coins. The blue value could be used for the background layer. Image processing functions are available for pretty much every programming language and the images and individual color components can be accessed with x y coordinates just like worlds.

---

"Sometimes failing a leap of faith is better than inching forward"
- ShinsukeIto

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

You could probably adapt the bmp format to your purposes. Set the red value as the block id, and the green value as additional information like portal direction or instrument note or number of coins. The blue value could be used for the background layer. Image processing functions are available for pretty much every programming language and the images and individual color components can be accessed with x y coordinates just like worlds.

That's a very good idea. I'm going to scrap my original level save thing and use that, actually.

---

???

lrussell

Member

From: Saturn's Titan

Joined: 2015-02-15

Posts: 843

Website

Re: [Discussion] EE Standardization -- let's all get on the same page

I have developed my own WorldFormat that I'm standardizing in all my projects. This one is slightly modified to support Protobuf (System.Drawing.Point to Tuple<int,int> and System.Drawing.Color to a Hex String). You can of course GZip it (or some other compression format) if you wish.

Live Json Version (using Point and Color): http://api.everybodyedits.info/WorldDat … dt06vvya0I
Example project using my class: http://raspberry.lrussell.ga/~lrussell/ … Example.7z

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

I have developed my own WorldFormat that I'm standardizing in all my projects. This one is slightly modified to support Protobuf (System.Drawing.Point to Tuple<int,int> and System.Drawing.Color to a Hex String). You can of course GZip it (or some other compression format) if you wish.

Live Json Version (using Point and Color): http://api.everybodyedits.info/WorldDat … dt06vvya0I
Example project using my class: http://raspberry.lrussell.ga/~lrussell/ … Example.7z

Hmm, that's quite interesting as well. Assuming that your format has undergone some testing, I suppose it's better to use yours than create a "new" file format from scratch. The downside though is that the files might be a tad big (even after gzip) but probably not too big of an issue.

EDIT: the use of 7zip compressed the files to a very manageable size.

Last edited by Hexagon (Dec 29 2014 8:12:38 pm)

---

???

Vitalijus

Member

From: Lithuania

Joined: 2015-02-15

Posts: 1,384

Website

Re: [Discussion] EE Standardization -- let's all get on the same page

Woudnt it be easier to add private int and when someone loads or saves level it would add +1 int and save that every second to file format u want?

---

Zumza

Member

From: root

Joined: 2015-02-17

Posts: 4,645

Re: [Discussion] EE Standardization -- let's all get on the same page

My format is this:
layer x y block_id s1 s2 s3 '/n'
Where s1, s2, s3 can or cannot be depending on block_id.
Its the most simple format I think and easy to read/edit for humans, also for extension i use ".ee"
Maybe in one day i hope, for compressed ee format everybody use this: https://atil.la/public_projects/EEWorldFormat/

Last edited by The Doctor (Dec 30 2014 2:58:13 am)

---

Everybody edits, but some edit more than others

XxAtillaxX

Member

Joined: 2015-11-28

Posts: 4,202

Re: [Discussion] EE Standardization -- let's all get on the same page

I'm completely removing everything in public_projects and the like.

EEWorldFormat isn't bad for saving files but it could be improved. I'm sure others can find an equally useful format to store worlds in.

---

*u stinky*

Zumza

Member

From: root

Joined: 2015-02-17

Posts: 4,645

Re: [Discussion] EE Standardization -- let's all get on the same page

I'm completely removing everything in public_projects and the like.

Sorry if I'm rude, but why?

---

Everybody edits, but some edit more than others

XxAtillaxX

Member

Joined: 2015-11-28

Posts: 4,202

Re: [Discussion] EE Standardization -- let's all get on the same page

I'm completely removing everything in public_projects and the like.

Sorry if I'm rude, but why?

Very messy directories and files. None of the projects listed there should be needed at all for future Everybody Edits projects.

EEWorldFormat isn't the most ideal format that could be strived for. I'm sure others could make much better ones, with or without protocol buffers.

---

*u stinky*

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

Woudnt it be easier to add private int and when someone loads or saves level it would add +1 int and save that every second to file format u want?

I'm really sorry, but I'm not sure what what you're trying to say. Would you be able to clarify that?

My format is this:
layer x y block_id s1 s2 s3 '/n'
Where s1, s2, s3 can or cannot be depending on block_id.
Its the most simple format I think and easy to read/edit for humans, also for extension i use ".ee"

That is certainly readable, and in the case of corrupting the file, might be easier to recover. Furthermore, the file extension chosen is (to my knowledge) not associated with another windows program, preventing users from opening the file and possibly corrupting it. It's a good idea (as you did) to include the layer and block id and not rely on a block lookup table, because then if more blocks are added with layers and ids, the block lookup table doesn't have to be updated (and subsequently the program) which increases extensibility.

However, we'd like to save signs, portals, and world portals into the save file as well. It might be complicated when the sign contains spaces or quotations, because it appears that the parser relies on spaces to differentiate every element. A parser would have to be created for parsing this data, while JSON parsers have already been ported to many programming languages.

---

Some things to think about when designing a "standard" are: future compatibility, extensibility (the program is automatically able to adapt to minor new changes, and is able to write the file without requiring a program update), file size, and recoverability (among many others).

What are your thoughts on using JSON (possibly compressed) for this standard generated in a format similar to lrussell's, http://api.everybodyedits.info/WorldDat … dt06vvya0I?

Last edited by Hexagon (Dec 30 2014 8:57:54 am)

---

???

Different55

Forum Admin

Joined: 2015-02-07

Posts: 16,574

Re: [Discussion] EE Standardization -- let's all get on the same page

However, we'd like to save signs, portals, and world portals into the save file as well. It might be complicated when the sign contains spaces or quotations, because it appears that the parser relies on spaces to differentiate every element. A parser would have to be created for parsing this data, while JSON parsers have already been ported to many programming languages.

You could base64 encode sign messages.

---

"Sometimes failing a leap of faith is better than inching forward"
- ShinsukeIto

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

You could base64 encode sign messages.

I haven't thought about that. Downside is that it'd have to be decoded twice, though.

---

???

lrussell

Member

From: Saturn's Titan

Joined: 2015-02-15

Posts: 843

Website

Re: [Discussion] EE Standardization -- let's all get on the same page

However, we'd like to save signs, portals, and world portals into the save file as well. It might be complicated when the sign contains spaces or quotations, because it appears that the parser relies on spaces to differentiate every element. A parser would have to be created for parsing this data, while JSON parsers have already been ported to many programming languages.

You could base64 encode sign messages.

That all seems a bit overkill though when standards such as Protobuf and Json have already been created. Json still provides human-readableness and edit-ability while Protobuf isn't but is even more compact.

ewoke

Member

Joined: 2015-02-20

Posts: 412

Re: [Discussion] EE Standardization -- let's all get on the same page

also if you guys add every block id to a file including 0 you dont need the x and y coords as you increase it while looping through it

just an idea

---

if you can read this....good for you

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

also if you guys add every block id to a file including 0 you dont need the x and y coords as you increase it while looping through it

That's an interesting idea, ewoke. I'll have to see whether using an implied index (using the blocks as they appear in the file as coordinates) would be beneficial. I'll consider that when tweaking the file format.

EDIT: since your idea involves the consideration that the room is contiguous, it sparked the idea of segmenting the room into n rectangles that all contain (or almost contain) a specific block (rectilinear polygons separated into chords https://www.cise.ufl.edu/~sahni/papers/part.pdf). Using your idea, the coordinate can begin at point p and end at o without needing to put in all of the coordinates for each block, and just rely on the rectangle size to know when to start the next row. This means that the data can be compressed very well. Thanks for your idea!

Last edited by Hexagon (Dec 30 2014 3:20:13 pm)

---

???

lrussell

Member

From: Saturn's Titan

Joined: 2015-02-15

Posts: 843

Website

Re: [Discussion] EE Standardization -- let's all get on the same page

also if you guys add every block id to a file including 0 you dont need the x and y coords as you increase it while looping through it

That's an interesting idea, ewoke. I'll have to see whether using an implied index (using the blocks as they appear in the file as coordinates) would be beneficial. I'll consider that when tweaking the file format.

EDIT: since your idea involves the consideration that the room is contiguous, it sparked the idea of segmenting the room into n rectangles that all contain (or almost contain) a specific block (rectilinear polygons separated into chords https://www.cise.ufl.edu/~sahni/papers/part.pdf). Using your idea, the coordinate can begin at point p and end at o without needing to put in all of the coordinates for each block, and just rely on the rectangle size to know when to start the next row. This means that the data can be compressed very well. Thanks for your idea!

Oh my... this is becoming scary.

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

Oh my... this is becoming scary.

Yeah, it's getting a little complicated. Depending on what the file size needs of the community are, the compression (and subsequently my idea) might not be as relevant.

---

A question to everyone: What would be the maximum file size of a room that would be considered manageable in your opinion (for example, a 300x300 room)?

Last edited by Hexagon (Dec 30 2014 4:28:51 pm)

---

???

lrussell

Member

From: Saturn's Titan

Joined: 2015-02-15

Posts: 843

Website

Re: [Discussion] EE Standardization -- let's all get on the same page

Oh my... this is becoming scary.

Yeah, it's getting a little complicated. Depending on what the file size needs of the community are, the compression (and subsequently my idea) might not be as relevant.

---

What would be the maximum file size of a room that would be considered manageable in your opinion (for example, a 300x300 room)?

Well, in my format the largest world in EE (with the most block variety basically, it's a 400x200 world but it has quite the pattern [I forgot the World Id]) is a bit over 200 KB. If you're going super compressed and efficient, perhaps 100 KB? But frankly as long as its not over 250 KB that's perfectly manageable.

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

Well, in my format the largest world in EE (with the most block variety basically, it's a 400x200 world but it has quite the pattern [I forgot the World Id]) is a bit over 200 KB. If you're going super compressed and efficient, perhaps 100 KB? But frankly as long as its not over 250 KB that's perfectly manageable.

Just recently I found a somewhat uncompressable world with a variety of blocks: http://everybodyedits.com/games/PWABy_vFGra0I. However, I'm not sure if that's the one that you are referring to, though. It's able to be compressed down to about 52kb, which should be manageable.

Last edited by Hexagon (Dec 31 2014 4:37:05 pm)

---

???

TiKen

Member

Joined: 2015-02-24

Posts: 298

Re: [Discussion] EE Standardization -- let's all get on the same page

What I did for EEPainter was the same thing as the "init" message you receive:
"block ID + param
x,y;x,y;x,y;..."

It's in clear text, easy to read an modify instead of opening the soft again.
It's nice when the world doesn't have many different blocks, and tends to grow a bit when many portals/coin doors& are used. But it won't grow too big anyway, and I don't think the space use will be consequent enough to bother. Except if you work on a potato.

Hexagon

Member

Joined: 2015-04-22

Posts: 1,213

Re: [Discussion] EE Standardization -- let's all get on the same page

What I did for EEPainter was the same thing as the "init" message you receive:
"block ID + param
x,y;x,y;x,y;..."

It's in clear text, easy to read an modify instead of opening the soft again.
It's nice when the world doesn't have many different blocks, and tends to grow a bit when many portals/coin doors& are used. But it won't grow too big anyway, and I don't think the space use will be consequent enough to bother. Except if you work on a potato.

I didn't think about standardizing the init messages (as the world is just copied directly from the database then serialized). Your method looks like it could be very human-readable. Also, since the init messages are in a different format than the database, it might be interesting to consider how to merge the two formats.

However, with all formats they need to be forward-compatible and can't operate on a fixed schema (to an extent). Do you mind if you could post the syntax for encoding portals and coin doors? Thanks!

---

???