Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

Hexagon · 2015-09-15 00:08:41

Hello,

So we have a lot of data, and there's no way that we'll be able to go through it all in our lifetimes. We're trying to crowdsource whether or not a certain action is considered vandalism to get training data for a neural network so that everyone can benefit from troll protection. We need tonnes and tonnes of data and lots and lots of training because trolling is very complex at times -- are they just shading it a different color or are they wreaking someone's picture of an apple? There have been many attempts to solve trolling, including detecting if the user is drawing too fast, but they appear to be too sensitive or not sensitive enough. Again, we don't know if we can do any better.

So what will happen is that you can visit a website on a voluntary basis, you'll see a short video of someone editing a world (no usernames), and you'll have the option to mark it as vandalism, not sure, or constructive editing. When you've rated that one, another one will appear.

The reason why we're asking if people will volunteer is because if there isn't enough people who can volunteer, we'll try to do it another way (because it will take a lot of time and investment to convert the event data into videos, and to get everything working in the backend.)

Any amount of videos that you can rate would be awesome.

What do you think?

Cyral, hummerz5, Swarth100, Xfrogman43, skullz17, Anch, SPT, shadowda, SirJosh3917, Krosis

shadowda · 2015-09-15 01:22:01

cool. im in if you need me. just pm me. or post the link to what ever vid when its up.

SirJosh3917

STD · 2015-09-15 01:25:57

Sounds like a great thing

I'd like to help if I'm needed. Just PM me.

P.S. This kinda sounds like the overwatch system for CS: GO.

SirJosh3917

Xfrogman43 · 2015-09-15 01:29:57

Sounds great, I'd love to see the absence of trollers.

SirJosh3917

SirJosh3917 · 2015-09-15 01:56:45

make sure you give them a normal face - sometimes players have a constant smiley

Anch · 2015-09-15 03:01:28

I can rate some videos.

SPT · 2015-09-15 03:15:42

sounds interesting. i'd like to see and rate some videos.

Zumza · 2015-09-15 10:52:15

What if is abstract art?

Hexagon · 2015-09-15 12:20:48

Zumza wrote:

What if is abstract art?

When it's abstract art, it's really hard to tell. In this case, you can answer "indeterminate" and we might be able to send you a longer video, which might help you decide. It's certainly a grey area though.

There might be a feature where you can see the rating of the user (i.e how many times they vandalized and what percentage of EE thought it was vandalism, whether or not they got kicked for it.) It's sort of like a background check. This might help you make your own decision, but might introduce bias.

Krosis · 2015-09-15 16:39:48

maybe i can help

Swarth100 · 2015-09-15 16:55:11

Up for it!

Zumza · 2015-09-15 17:56:40

▼Hidden text

Ill volunteer.

skullz17 · 2015-09-15 18:09:37

Do you count something as trolling only if it is vandalising someone else's work? Or does scribbling in an open space count as well (Perosnally I don't think it should)?

Anyway, I'll volunteer.

Hexagon

Abelysk · 2015-09-15 19:19:06

I'll volunteer

some woman · 2015-09-15 19:40:44

might as well volunteer

Hexagon · 2015-09-15 22:53:39

skullz17 wrote:

Do you count something as trolling only if it is vandalising someone else's work? Or does scribbling in an open space count as well (Personally I don't think it should)?

This is where it gets really complicated. Right now we'll use the definition of trolling from the EE wikipedia:

Trolling is an action when a user ruins or disrupts a world.

Or, another definition from the "Troll" entry:

[...]will race around the level spawning blocks in lines to destroy a level

However there is a lot of ambiguity in that definition. Personally I'd call trolling:

A user who purposely introduces negative modifications to other users' edits, makes it difficult for users to create new content, or communicate.

Since "creating new content" applies to chat as well, it can be used for chat spammers. More on that in a bit.

---
We'd like to have a bit of consensus on how trolling is defined, so we'd have to make a global definition of it. What is your definition of trolling?

hummerz5 · 2015-09-15 23:23:57

Yeah, I sorta asked that question a while back, though I wasn't this exacting with my definitions
+1

SmittyW · 2015-09-15 23:32:58

We already have the ultimate source of troll detection. They are called humans. Is trolling really such a big deal that we need to bring bots into it?

robotkoer

Hexagon · 2015-09-15 23:34:44

SmittyW. wrote:

We have the ultimate source of troll detection. They are called humans. Is trolling really such a big deal that we need to bring bots into it?

Humans are error-prone, expensive, and can't be on 24/7. Bots are cheap, reliable, and can be on 24/7. Plus it's fun designing a NN.

Ben

Muftwin · 2015-09-15 23:57:12

SmittyW. wrote:

We already have the ultimate source of troll detection. They are called humans. Is trolling really such a big deal that we need to bring bots into it?

There never was a point to anything this guy does he just wants to do it so ignore it.

Different55 · 2015-09-16 00:48:15

Hexagon wrote:

SmittyW. wrote:
We have the ultimate source of troll detection. They are called humans. Is trolling really such a big deal that we need to bring bots into it?
Humans are error-prone

lol not as error prone as this will be if it ever gets off the ground. It would need to understand context and that's not something bots are great at, even when trained by humans.

Hexagon · 2015-09-20 14:38:15

I don't know if it's going to be that accurate, but it'll be cool to work on. The bot will "cheat" too by preventing high-risk players from editing at all, if they have trolled in the past.

Before we dive into the troll detection, what we'll do is start creating a NN or a classifier to detect whether or not a message is spam. From a small sample size, ~4% of all EE chat messages (excluding PM's) are spam. We'll see how accurate we can classify them. If that isn't able to get a good enough accuracy, or is too hard, then the troll detection would be much too hard.

TL;DR We're going to post chat messages from EE (removing usernames, world ids and email addresses) so that you can tell us whether a certain message is spam in the context of the chat log. Would anyone like to rate the messages whether or not some are spam? There's about 9.8 million chat messages and we need some help! The more messages you rate, the more accurate our model becomes at classifying spam messages, and helps EE become a better place.

SirJosh3917

Zumza · 2015-09-20 15:59:55

Ill volunteer for chat too.

SirJosh3917, Hexagon

Official Everybody Edits Forums

#1 2015-09-15 00:08:41

Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#2 2015-09-15 01:22:01, last edited by shadowda (2015-09-15 20:09:49)

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#3 2015-09-15 01:25:57, last edited by STD (2015-09-15 02:13:48)

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#4 2015-09-15 01:29:57

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#5 2015-09-15 01:56:45

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#6 2015-09-15 03:01:28

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#7 2015-09-15 03:15:42

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#8 2015-09-15 10:52:15

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#9 2015-09-15 12:20:48, last edited by Hexagon (2015-09-15 12:40:40)

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#10 2015-09-15 16:39:48

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#11 2015-09-15 16:55:11

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#12 2015-09-15 17:56:40

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#13 2015-09-15 18:09:37, last edited by skullz17 (2015-09-15 18:09:55)

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#14 2015-09-15 19:19:06

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#15 2015-09-15 19:40:44

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#16 2015-09-15 22:53:39, last edited by Hexagon (2015-09-15 22:58:28)

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#17 2015-09-15 23:23:57

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#18 2015-09-15 23:32:58, last edited by SmittyW (2015-09-15 23:34:07)

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#19 2015-09-15 23:34:44, last edited by Hexagon (2015-09-15 23:37:31)

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#20 2015-09-15 23:57:12

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#21 2015-09-16 00:48:15

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#22 2015-09-20 14:38:15

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

#23 2015-09-20 15:59:55

Re: Need help with Anti Vandal -- we have ~65 years of EE data in 6 months

Board footer