EE Analytics torrent [Feb 2016 available]

XxAtillaxX · 2016-01-31 16:20:00

Move events are unnecessary, not only do they have the potential of being deprecated, but years worth of move events can be generated in a day.
If someone were to make an AI (think NEAT (neuralevolution of augmenting topologies)), then they could generate tons of datasets with simply using time and/or coins as a fitness variable.

The only potentially valuable data is init, godmode, chat, trophy and saved room packets, essentially anything that can't be easily computated in the future.
There are large privacy implications with storing chat logs in the first place, regardless of them being in publicly available rooms.
The godmode would be essential for knowing who the owners' friends are by seeing who they grant access to the most, and in what timespan and worlds, etc.
I'd think anonymised friendship graphs would be neat. Though, with anything nowadays, you could just simply request an admin to make it for you.

Though, it would be quite neat to have a repository of Everybody Edits related material.
For this, I would recommend something like BT Sync (if only it weren't proprietary...), which is highly related to torrents, only with the addition of being synchronised in realtime.

In any case, storing a large amount of data has its complications.
When it comes to replicating data, using torrents as a means of backing up, you have to be aware of that torrents aren't by default synchronised with realtime data, so data redundancy is a factor as well.
That being said, nobody is going to allocate tons of digital storage space and bandwidth for things they don't use or find useful.

Koya

Different55 · 2016-01-31 16:25:19

Hexagon wrote:

lrussell wrote:
By base set I mean the most important data. 107 GB is still out of reach for people who want to help keep the data safe. I'd say it should be no more than 15 GB. Knowing what block someone placed, when, and where or if they used the Grinch smiley isn't all that important. It might be fun if someone made a "player" for it, showing everything as it happened. But did you store the timing between each message to do even that?
I agree that 107GB is still very large, and even if I could compress it down to 1GB, nobody would be able to do anything with it because it would expand to 1.5TB in order to read it. What I could is split them into months, or even into days in gzip (as each month compressed is about 10GB, days compressed are about 250MB, which is manageable.) As long as I get enough volunteers to host a section of the data, the entire data set can be reassembled.
Most of the data is move events, and I could remove them (or compressed them separately in a numerical format) to see if I can get down the file sizes a bit more. As with the timing between each message, it was recorded however a fatal error in my implementation for at least a couple of months means that some timings are off by a lot (when played back, it will look like time is slowing down during high activity and then suddenly speed up when the activity stops.)

Do the month thing. I like month things.

Hexagon · 2016-02-01 19:39:11

Combining a few ideas together from the community, the mirrors (hosters) could just host the last three months of data, and then the rest lives on archive.org. Three months of data will be about 14GB, which is much less than 150GB. If those files are still too large, I can do with Atilla and lrussell are saying, which is to provide a base set with only the essentials.

Right now I'm currently sanitizing the data and getting things compressed on ultra 7z settings, and I'll let you know once the data is available for download.

EDIT: I remembered that open worlds were not recorded from about six months ago to now, so those might be an inconsistency if you see them in the lobby data.
- also the lobby data is very, well... granular as it recorded in chunks of 30 minutes and only user ids and world ids.

Koya · 2016-02-02 13:46:39

Zumza wrote:

I bet you have some kind of google glass and record every moment of your life.
Or you are more interested in others?

Recorded on a minute-to-minute basis though

Hexagon · 2016-02-02 17:40:18

Later day we'll be releasing the first full data archive from December 2015 via torrent, and if no issues are detected will be accessible to everyone (I'll update this post with a magnet link.) The smaller base archives will take some time to generate, as it takes around 2 hours for each removing-events pass (6 hours for sanitization.)

Nou · 2016-02-02 18:43:38

Hexagon wrote:

Later day we'll be releasing the first full data archive from December 2015 via torrent, and if no issues are detected will be accessible to everyone (I'll update this post with a magnet link.) The smaller base archives will take some time to generate, as it takes around 2 hours for each removing-events pass (6 hours for sanitization.)

Leave out the chatlogs. We've expressed disapproval of this before: http://forums.everybodyedits.com/viewto … 35#p556035

Hexagon · 2016-02-02 18:44:44

Nou wrote:

Leave out the chatlogs. We've expressed disapproval of this before: http://forums.everybodyedits.com/viewto … 35#p556035

Chat logs aren't going to be included. Sorry, I meant to say that the "full" set is the sanitized one, and the base one is even more condensed.

EDIT: magnetic link available, for 2015-12 data:

magnet:?xt=urn:btih:45d8d6ee3877011960c5346f4eaf7273a9ec6cd1&dn=2015-12&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969&tr=udp%3a%2f%2ftracker.leechers-paradise.org%3a6969&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a80&tr=udp%3a%2f%2fopen.demonii.com%3a1337

lrussell

XxAtillaxX · 2016-02-02 20:14:57

I will definitely seed with all my 10kbps speeds.

Hexagon

Zumza · 2016-02-02 20:40:34

I have an avarange of 200 KB/s

Hexagon · 2016-02-02 20:49:08

Zumza wrote:

I have an avarange of 200 KB/s

It takes a long time to start up (like 10 minutes or so.) I know that another peer has been downloading the file successfully so far.

Zumza · 2016-02-02 21:08:04

Hexagon wrote:

Zumza wrote:

I have an avarange of 200 KB/s
It takes a long time to start up (like 10 minutes or so.) I know that another peer has been downloading the file successfully so far.

It was just a joke.
Also I will seed this file while people are interested.
It seems that the wireless router wormed up and gives me almost 1 MB/s I'm kinda disappointed that the uploading speed is 3 times garter than download speed.

Hexagon · 2016-02-02 23:18:24

We fully understand why chat messages shouldn't be in the data, however we're considering about re-adding the add messages because otherwise it's very difficult to reconstruct a user based on when the first move event occurs. This means that usernames will be in the data, and add messages are just messages that someone joins a room.

We'd really want them to be back in, but we're asking if you'd feel that's okay if they were in.

lrussell

lrussell · 2016-02-04 20:48:01

Hexagon wrote:

We fully understand why chat messages shouldn't be in the data, however we're considering about re-adding the add messages because otherwise it's very difficult to reconstruct a user based on when the first move event occurs. This means that usernames will be in the data, and add messages are just messages that someone joins a room.
We'd really want them to be back in, but we're asking if you'd feel that's okay if they were in.

I don't see a problem in having the "add" messages included. It'd be a lot more useful for statistics and rather harmless since chat isn't being recorded.

Hexagon

XxAtillaxX · 2016-02-04 20:50:38

Where do you draw the line when it comes to anonymising the data? It should be obvious that some users won't want the time and places they've joined worlds tracked.

Hexagon · 2016-02-07 03:09:01

XxAtillaxX wrote:

Where do you draw the line when it comes to anonymising the data? It should be obvious that some users won't want the time and places they've joined worlds tracked.

I draw the line where the community says where they wouldn't feel comfortable sharing a certain piece of data. For example, the community doesn't want chat messages in there, so chat messages won't be public.

---

November 2015 has been posted, and is free to download.

Zumza

Hexagon · 2016-03-01 16:06:18

The data for February 2016 has been released, and is now available for download.

Koya · 2016-03-01 16:51:58

Hexagon wrote:

For example, the community doesn't want chat messages in there, so chat messages won't be public

Hexagon wrote:

public

I'm not one to care about chat tracking but you should have said not tracked and kept it secret; ignorance is bliss

Hexagon · 2016-03-17 00:49:20

We understand that there are some users who feel that their privacy has been reduced when EE Analytics joins their world. If you'd like to have the bot not join your world, PM me and I'll give you a form that you can fill out. The form must be completed in order to opt-out from EE Analytics (a PM won't suffice.)

SmittyW · 2016-03-17 02:00:49

I have to fill out a form to opt-out of an event I never signed up for.

Good one.

Hexagon · 2016-03-17 02:16:20

SmittyW. wrote:

I have to fill out a form to opt-out of an event I never signed up for.
Good one.

Everyone is opted in by default, as it follows internet indexing patterns (i.e you have to opt-out your website from Google.)

Tomahawk · 2016-03-17 17:08:58

Hexagon wrote:

Everyone is opted in by default, as it follows internet indexing patterns (i.e you have to opt-out your website from Google.)

The problem is that most people won't know they're being tracked unless they happen to read this thread.

Or does the bot tell you you're being message stalked?

Hexagon · 2016-04-05 02:12:37

Tomahawk wrote:

Hexagon wrote:
Everyone is opted in by default, as it follows internet indexing patterns (i.e you have to opt-out your website from Google.)
The problem is that most people won't know they're being tracked unless they happen to read this thread.
Or does the bot tell you you're being message stalked?

The bot acts like a typical bot would, making it intentionally easy to detect it (i.e just idles, doesn't send any messages on death or anything.) Using this behaviour, a large portion of the EE population has detected my bots.

Tomahawk · 2016-04-05 13:57:57

Hexagon wrote:

The bot acts like a typical bot would, making it intentionally easy to detect it

Well there's a lot of cases where it won't be detected when just idling. It's still secretly stalking anyone who doesn't know, and although what they don't know doesn't hurt them, they do have a right to know.
Maybe have it say something like "[EEAnalytics] Connected." and add some messages that explain the bot or link players to this thread. You could add chat commands for people to opt out of tracking for a certain world or time period, and have a vote kick thing like zerk bot had.

I'm not entirely sure I care, but if it's no longer entirely anonymous with "add" being recorded, and chat messages are logged, you should be careful that this doesn't happen again.

den3107 · 2016-04-05 15:50:53

Tomahawk wrote:

Hexagon wrote:
Everyone is opted in by default, as it follows internet indexing patterns (i.e you have to opt-out your website from Google.)
The problem is that most people won't know they're being tracked unless they happen to read this thread.
Or does the bot tell you you're being message stalked?

Being completely honest, barely anybody knows their facebook page is stalked by Google, Yahoo, etc.
I also do website spidering as a livign myself, and you'd be surprised how many sites don't have a robots.txt (a text file that tells bots that they don't want to be spidered (indexed))
Google won't tell every site they're being spidered, they'll have to find it out themselves and see how to prevent that.

Hexagon · 2016-04-05 17:03:47

Tomahawk wrote:

Hexagon wrote:
The bot acts like a typical bot would, making it intentionally easy to detect it
Well there's a lot of cases where it won't be detected when just idling. It's still secretly stalking anyone who doesn't know, and although what they don't know doesn't hurt them, they do have a right to know.
Maybe have it say something like "[EEAnalytics] Connected." and add some messages that explain the bot or link players to this thread. You could add chat commands for people to opt out of tracking for a certain world or time period, and have a vote kick thing like zerk bot had.
I'm not entirely sure I care, but if it's no longer entirely anonymous with "add" being recorded, and chat messages are logged, you should be careful that this doesn't happen again.

We would be more than willing to work with anyone if they feel that their privacy is being invaded or feel like their being stalked. Feel free (and anyone else) to PM me if you have any concerns with your world being spidered, and we'll see if we what we can do (we don't normally reach an agreement here because it might bump the post too many times.)

If there is enough demand, we can put in more controls as to what data is being collected from your world, and how it is published too.

Official Everybody Edits Forums

#26 2016-01-31 16:20:00, last edited by XxAtillaxX (2016-01-31 16:21:30)

Re: EE Analytics torrent [Feb 2016 available]

#27 2016-01-31 16:25:19

Re: EE Analytics torrent [Feb 2016 available]

#28 2016-02-01 19:39:11, last edited by Hexagon (2016-02-01 19:48:41)

Re: EE Analytics torrent [Feb 2016 available]

#29 2016-02-02 13:46:39

Re: EE Analytics torrent [Feb 2016 available]

#30 2016-02-02 17:40:18

Re: EE Analytics torrent [Feb 2016 available]

#31 2016-02-02 18:43:38

Re: EE Analytics torrent [Feb 2016 available]

#32 2016-02-02 18:44:44, last edited by Hexagon (2016-02-02 20:10:22)

Re: EE Analytics torrent [Feb 2016 available]

#33 2016-02-02 20:14:57

Re: EE Analytics torrent [Feb 2016 available]

#34 2016-02-02 20:40:34

Re: EE Analytics torrent [Feb 2016 available]

#35 2016-02-02 20:49:08

Re: EE Analytics torrent [Feb 2016 available]

#36 2016-02-02 21:08:04, last edited by Zumza (2016-02-02 21:28:12)

Re: EE Analytics torrent [Feb 2016 available]

#37 2016-02-02 23:18:24

Re: EE Analytics torrent [Feb 2016 available]

#38 2016-02-04 20:48:01

Re: EE Analytics torrent [Feb 2016 available]

#39 2016-02-04 20:50:38, last edited by XxAtillaxX (2016-02-04 20:50:47)

Re: EE Analytics torrent [Feb 2016 available]

#40 2016-02-07 03:09:01

Re: EE Analytics torrent [Feb 2016 available]

#41 2016-03-01 16:06:18

Re: EE Analytics torrent [Feb 2016 available]

#42 2016-03-01 16:51:58

Re: EE Analytics torrent [Feb 2016 available]

#43 2016-03-17 00:49:20

Re: EE Analytics torrent [Feb 2016 available]

#44 2016-03-17 02:00:49

Re: EE Analytics torrent [Feb 2016 available]

#45 2016-03-17 02:16:20

Re: EE Analytics torrent [Feb 2016 available]

#46 2016-03-17 17:08:58

Re: EE Analytics torrent [Feb 2016 available]

#47 2016-04-05 02:12:37

Re: EE Analytics torrent [Feb 2016 available]

#48 2016-04-05 13:57:57, last edited by Tomahawk (2016-04-05 14:00:34)

Re: EE Analytics torrent [Feb 2016 available]

#49 2016-04-05 15:50:53

Re: EE Analytics torrent [Feb 2016 available]

#50 2016-04-05 17:03:47

Re: EE Analytics torrent [Feb 2016 available]

Board footer