[Question] How to quantitatively measure EE experience

Hexagon · 2015-04-13 14:09:15

Is there a way (already discovered) to somewhat quantitatively measure someone's experience with EE (level solving ability?) For example, the ability to complete "hard" levels that would be challenging to others.

I know that there are a lot of mini games which have easy, medium and hard sections, but the variance appears to be high in my experience (some are super super easy, others are nearly impossible at the same level.) Having said that, it might be more appropriate to average many mini games with similar levels together, to create a more accurate result.

Has anyone tried doing this, or have an example that they could share?

Aside: something else to consider is allowing players to choose between different sections if they get stuck. For example, if someone finds the hook jumps difficult (or the one block hook jumps) they could do a few different sections with equal difficulty so that they could continue to progress. For normalization purposes, the easiest level should be able to be completed by 95-99% of all players, and the hardest by the top 1-5%.

Current Issues:
- when players play a level, other people can watch/spectate, and learn how to do it also (bias)
- players can chat to each other to tell how to do a level (bias)
- some may teleport/hack, which would make them appear much more experienced
- time of day plays an important factor, and whether or not you have played an ee level recently
- the entire level could be copied, and people could practise on a version of it to gain experience (to prevent this, force the player to use an alternate account created by the staff, and then link the results to the player. Only allow the accounts that are created by the staff specifically for that purpose to be logged in in that world, and immediately disable the account when they leave. Disable minimap to prevent screenshots, disable init grabbing, make world invisible in lobby. Have an algorithm that goes through newly created worlds and disables them if they appear to closely resemble world (add a secret block watermark to the level for each person so that you know who copied it). )

Anti copying techniques:
- only allow one player at a time in the level (and the bot, somewhere else)
- generate the level dynamically (with a bot) to prevent init scavengers
- place a special watermark in the blocks so that when someone takes a picture of it and shares it, it can be traced back to the owner
- make the user answer a captcha or a question that a computer would have a difficult time answering. Then they will have to perform a series of movements and jumps to "prove" that they are not a bot, answer another question, then the level will be drawn dynamically.
-- the first few mini games may not count, so that it can be proven that you didn't make a level saver join with chat enabled to answer the skill testing questions
- the "contest" will be open only for a day or two, and players may not discuss the level with anyone 48 hours after they have played it.
- verify that the movements of the player are correct to some extent (some people have high ping times) using EEPhysics or something else. If something looks amiss, they will be kicked.
- send a few events like kill and respawn to the player, and their client should respond with the correct events (this is easy to bypass but helps.)
- try to get as many people to do it at once
- once the player is finished, kick the user
- if the player leaves the session at any time, do not allow them back in
- dynamically erase the parts that the player has finished to prevent screenshots
- create two different versions for the two time zones
- send the data over https* to prevent packet sniffers
- to prevent users from trying to shift the colors, change the hue, or replace blocks, make certain paths only available depending on the color. So multiple paths could be available but you have to pick the one with the right color and be on the right colored block at the right time. This would have a water mark in it (the position of the colors of the blocks.)
- make the world extremely tall (200000x1000) and put the mini games at the top, to prevent people from screenshoting the minimap*
- block tor nodes*

* might not work or be unnecessary

Kyle97, some woman

Kyle97 · 2015-04-13 14:30:45

Interesting topic. Ive never really thought about it. You would have to group together a bunch of different minis and see what percent of players could complete each. You could then rank the minis based on percentage of players who completed it.

However, I think the order of the minis should be left up to the player, or give them the ability to skip the mini with points deducted, in order to get more complete results.

Another factor to consider is time. Should there be a time limit for each mini, so that the player must complete it before the limit is reached, or they get points deducted? Or should it be unlimited?

Finally, should the minis be grouped by type or by the type of skills required to complete it? For example, one mini could have an obvious route to solve it, and only requires quick reaction time and skill. While another mini could have a more complex or somewhat hidden route, or even many possible routes to complete it. This mini would require skill AND puzzle solving skills. Should these skills be rated individually?

Id like to see some sort of system for this, and would be happy to help develop it!

Mainx · 2015-04-13 14:35:50

Since my first day in EE, I desire about such system. Sadly I don't see how this could be incorporated.

The best mini-makers gather together and make a giant Skill-Test level and the system auto rates you in your profile, based on your performance?

skullz17 · 2015-04-13 14:38:48

To measure how experienced a player is, you need to measure how hard a level is. To measure how hard a level is, you need to measure how experienced a player is.

Xfrogman43, Bobithan

Mainx · 2015-04-13 14:47:48

skullz17 wrote:

To measure how experienced a player is, you need to measure how hard a level is. To measure how hard a level is, you need to measure how experienced a player is.

And we already know which players are very experienced in doing minis. You always try to look too smart with the answers, don't you?

skullz17 · 2015-04-13 14:49:46

Mainx wrote:

skullz17 wrote:
To measure how experienced a player is, you need to measure how hard a level is. To measure how hard a level is, you need to measure how experienced a player is.
And we already know which players are very experienced in doing minis. You always try to look too smart with the answers, don't you?

It doesn't matter if you "already know". You need to base it on something. That is the whole point of this topic. I'm saying that the sort of solution that is being suggested does not work.

If you take a sample of EE players, and get them to play some minigames, you might just have a sample of very similar experienced players. So the percentage thing wouldn't work so well. You would get very high percentages and very low percentages, because they would succeed on the same minis and fail on the same minis. To get a mixed sample with a variety of players with different levels of experience, you need to measure how much experience each player has. But now we've gone in a circle. We're back to the problem we had at the start.

That's why that solution doesn't work perfectly, unless you take a very large sample. Back to what you said: we already know which players are very experienced in doing minis. Rather than deciding for yourself who is the most experienced player, there could be some sort of vote. However, I think rather than using a vote for that, it would be better to use a vote for minis. Or even better, a rating. The average difficulty rating given to a mini could decide how hard it is. And then you use that to create a bunch of minis of different difficulties, and see how each person fares in each one, and use that to decide how experienced that player is.

Or, rather than see whether the player can complete it or not, have the player give the mini a rating. An average rating of a mini decides its difficulty. The rating the player gives in comparison to the average rating shows their level of experience.

Bobithan, N1KF

Mainx · 2015-04-13 15:08:42

tl;dr

EDIT-1: Readed briefly - Agree

Kyle97 · 2015-04-13 15:25:59

Why couldn't you just create a set of minis in a world, and open it for a set amount of time and track how many people can complete each mini. We want to represent the average population of EE, so allow anyone who joins to participate. Repeat for consistency.

Then look at the data. Ex. For mini x, only 30% of players who attempted this mini passed it. So, if you are able to complete this mini, you are roughly in the top 30% out of all players.

This would require many repeated tests and might have some flaws, but its a baseline that could then be used to develop more accurate and complete data.

skullz17 · 2015-04-13 16:05:25

Yeah, but how do you know that anyone who joins represent the average population? The best way to do it would probably be to put it in the news thing when you log in. That way, everyone who logs in will see it and has a chance to play the world.

Slushie · 2015-04-13 16:10:09

I think measuring how long it takes a player to complete a mini is the best way to quantify skill. Any player could beat any mini if he or she tried long enough, so the "x% of players beat this mini" statistic is useless.

Players could be timed for several different minis to calculate their overall skill.

Kyle97 · 2015-04-13 16:15:54

Well it would more or less represent the population of ee at that current time. With multiple trials, you could combine the results to a very large sample size and then analyze it. Hopefully with a large sample the errors will be minimized.

Another problem I see with this though is players joining and not completing the entire world, thus giving a false result...

BEE · 2015-04-13 23:21:39

EE experience =/= ability to do minigames.

Case in point: Me.

I've got plenty of EE experience, yet I can land a normal hook only 1/5 times and am terrible at minigames. I am often the "is this too hard for a nub" tester for levels.

You could measure minigame skill level, sure.

Also, I've personally love a "go to an easier section by portal" world, that sounds like fun, go make it

Muftwin · 2015-04-13 23:46:37

I think before we can figure this out we need to have stated what purpose this could possibly have. All the little kids that make rank levels dont consider anything they just build, and thats perfectly fine. My point is that you cant even consider whats at the top of these levels to even be remotely hard most of the time theyre tedious and lucky.

Mainx · 2015-04-14 06:32:36

There had to be an XP system /leveling up based/ since day one.

gkaby · 2015-04-14 06:38:07

i propose a SI unit of Woot (experience/player)

SI unit of Woots can be used to quantitatively measure the EE experience.

to find how many Woots of experience per player a person has:

count the number of Woots in all of their levels.
The total number is the Woots of that player

Kyle97 · 2015-04-14 06:45:26

gkaby wrote:

i propose a SI unit of Woot (experience/player)
SI unit of Woots can be used to quantitatively measure the EE experience.
to find how many Woots of experience per player a person has:
count the number of Woots in all of their levels.
The total number is the Woots of that player

Eh... Maybe
But this is more about skill in completing minigames, not creating worlds (but I like the idea of seeing who creates the bests worlds as well). Plus the winner would most likely be "woot for edit" worlds, so plays might be a better stat to use for that purpose.

MIHB_casts_confuseplayer · 2015-04-14 07:39:10

Why would you even want to do this?

Any player could beat any mini if he or she tried long enough, so the "x% of players beat this mini" statistic is useless.

If you're saying "could beat this mini given eternity", all minis could be beaten by somebody with the necessary physical dexterity. I'm assuming that this is a "theoretical" answer rather than a practical answer.

On a more practical level, there are plenty of minis that are functionally impossible for many, even most players.

Mainx · 2015-04-14 09:04:57

Yes wrote:

I think measuring how long it takes a player to complete a mini is the best way to quantify skill. Any player could beat any mini if he or she tried long enough, so the "x% of players beat this mini" statistic is useless.
Players could be timed for several different minis to calculate their overall skill.

When someone times me or even observes me I always perform the worst.

Agreed with MIHB.

Hexagon · 2015-04-14 10:33:06

Thanks for the interest so far!

Kyle97 wrote:

Another problem I see with this though is players joining and not completing the entire world, thus giving a false result...

I have a bot that captures all events generated in a level, so it should be easy to detect whether they stop half way through.

MIHB_casts_confuseplayer wrote:

Why would you even want to do this?
Any player could beat any mini if he or she tried long enough, so the "x% of players beat this mini" statistic is useless.
If you're saying "could beat this mini given eternity", all minis could be beaten by somebody with the necessary physical dexterity. I'm assuming that this is a "theoretical" answer rather than a practical answer.

Both points are interesting. I think that time plays a role, but the amount of events generated in a specific zone are important too (so if I'm aggressively pressing the space bar rather than pressing it at the right moment it might indicate that I don't have as much experience and trying to do it based on luck.)

Mainx wrote:

When someone times me or even observes me I always perform the worst.

I also get a little bit bad when someone is watching me play or timing me too. Perhaps the level description could just say "oh, report how well you did after with this survey" and say nothing about that you're being timed and watched by a bot.

-----

Another big problem is that you will see others complete the same mini game at the same time. I learn by watching others play a world and am able to do things much more quickly than if I were to do it by myself (for example, finding invisible portals.) This could inflate the "experience" of a user and deflate the ones that cannot watch others just because they are watching a more experienced one. Perhaps there could be multiple copies of the mini game, and only one person is allowed to go at once.

However, other people could post the solutions in the chat (as chat messages are kept around for a while after someone leaves.) Maybe get a few bots to clear out the say_old messages by just saying random stuff.

skullz17 · 2015-04-14 11:09:11

I think there's a /clearchat command. I'm not sure if it clears for everyone or just you.

Hexagon · 2015-04-14 11:16:49

skullz17 wrote:

I think there's a /clearchat command. I'm not sure if it clears for everyone or just you.

Didn't know about this /clearchat thing, seems promising. I'll give it a shot and see if it clears for other people.

EDIT: it looks like it only clears for the person who issued the command. Maybe there could be a bot that kicks people if they chat (or say something other than a command), have a guardian temporarily mute them, or have a bot purposely spam the chat (might not be ideal.)

Mainx · 2015-04-14 11:54:08

Add date of registration in profile is a good starting point and something realistic at the moment. The "Player registered in 2010, but haven't played since then" situation will occur rarely, because if they didn't liked the game and left, I doubt they'll give it a second chance 5 years after. Also - inactive accounts could be deleted.

I hope an EE mod will join the discussion.

Hexagon · 2015-04-14 11:55:32

Mainx wrote:

Add date of registration in profile is a good starting point and something realistic at the moment. The "Player registered in 2010, but haven't played since then" situation will occur rarely, because if they didn't liked the game and left, I doubt they'll give it a second chance 5 years after. Also - inactive accounts could be deleted.
I hope an EE admin will join the discussion.

This is a very good point that you brought up. Using the profile data further, you can see if they registered like a day ago (or even a few minutes ago) to prevent people from spamming, and maybe they have to buy a few things with energy first to prevent alt accounts from being used.

Bobithan · 2015-04-14 14:00:06

skullz brings up a really good point in that to measure player experience, you need to have a specific measurement in minigame difficulty. If we can figure out how to determine exactly how hard a minigame is, player experience will be an easy next step.

This could be achieved somewhat by standardization since minigames can be objectively harder than other minigames. For example, a jump 12 blocks in length is objectively harder than a jump 5 blocks in length.

With that in mind, is it possible to determine the objective difficulty of a minigame? Could a standardized unit be achieved? Could somebody look at a minigame and say without question, "That minigame is a 7.6 on the Minsen Scale."?

Things that could be looked at:

-Precision
The amount of physical room for error. Making your way through a one block gap is objectively more precise than making your way through a ten block gap.
-Speed
The amount of room for error in timing. Hitting a one block jump is a lot more difficult when launching off a boost than it is just running over the block.
-Length
The longer the minigame is, the more chances there are for failure.
-Complexity
The more buttons and things you need to keep in mind, the more difficult the minigame is. A minigame that uses W, A, S, D, and space is much more complex than one that just uses D and space. Similarly, a minigame with a pattern is easier to solve than one without a pattern.

--

Also, for measurement of player skill, time isn't what you want to look at. Some minigames can take considerably longer to complete in one go than others, so the best method would instead be attempts rather than time spent.

If a standardized minigame difficulty unit is achieved, you can test a player's ability easily by giving them x number of tries on minigames of specific difficulty.

Kyle97

Hexagon · 2015-04-14 15:38:04

Bobithan wrote:

Also, for measurement of player skill, time isn't what you want to look at. Some minigames can take considerably longer to complete in one go than others, so the best method would instead be attempts rather than time spent

I sort of agree with this. I think that if you take more time on a certain level, it might show that you are not able to quickly cognitively process how to complete the level. Since experienced users have more experience, it is possible that they may have already seen the patterns in the minigames and so would be able to do it faster. However, I'm not sure if those would be correlated. It's an interesting point though.

For example, it usually takes me about 30 seconds to do one hook jump (I'm terrible at those) but it would take me one try. I'm not really experienced so it would take me more time.

---

Maybe users could be rated on separate tasks so for example could be on precision (jumping from block to block), timing (something with keys opening/closing gates), accuracy (only jumping on certain colored blocks in a pattern [every green then red one or something]), future thinking (ability to visualize which path would be the most feasible by looking at risk and judging which one would be much too difficult), impossibility (looking at a certain level and judging whether or not it would be impossible for everyone [for example jumping six blocks with no boosters]), endurance (how long you can go without taking a break).

Official Everybody Edits Forums

#1 2015-04-13 14:09:15, last edited by Hexagon (2015-04-14 21:21:26)

[Question] How to quantitatively measure EE experience

#2 2015-04-13 14:30:45

Re: [Question] How to quantitatively measure EE experience

#3 2015-04-13 14:35:50

Re: [Question] How to quantitatively measure EE experience

#4 2015-04-13 14:38:48

Re: [Question] How to quantitatively measure EE experience

#5 2015-04-13 14:47:48

Re: [Question] How to quantitatively measure EE experience

#6 2015-04-13 14:49:46, last edited by skullz17 (2015-04-13 15:04:55)

Re: [Question] How to quantitatively measure EE experience

#7 2015-04-13 15:08:42, last edited by Mainx (2015-04-13 15:12:22)

Re: [Question] How to quantitatively measure EE experience

#8 2015-04-13 15:25:59

Re: [Question] How to quantitatively measure EE experience

#9 2015-04-13 16:05:25

Re: [Question] How to quantitatively measure EE experience

#10 2015-04-13 16:10:09

Re: [Question] How to quantitatively measure EE experience

#11 2015-04-13 16:15:54

Re: [Question] How to quantitatively measure EE experience

#12 2015-04-13 23:21:39, last edited by BEE (2015-04-14 00:18:50)

Re: [Question] How to quantitatively measure EE experience

#13 2015-04-13 23:46:37

Re: [Question] How to quantitatively measure EE experience

#14 2015-04-14 06:32:36

Re: [Question] How to quantitatively measure EE experience

#15 2015-04-14 06:38:07

Re: [Question] How to quantitatively measure EE experience

#16 2015-04-14 06:45:26

Re: [Question] How to quantitatively measure EE experience

#17 2015-04-14 07:39:10, last edited by MIHB_casts_confuseplayer (2015-04-14 07:43:59)

Re: [Question] How to quantitatively measure EE experience

#18 2015-04-14 09:04:57

Re: [Question] How to quantitatively measure EE experience

#19 2015-04-14 10:33:06

Re: [Question] How to quantitatively measure EE experience

#20 2015-04-14 11:09:11

Re: [Question] How to quantitatively measure EE experience

#21 2015-04-14 11:16:49, last edited by Hexagon (2015-04-14 11:23:08)

Re: [Question] How to quantitatively measure EE experience

#22 2015-04-14 11:54:08, last edited by Mainx (2015-04-14 11:54:32)

Re: [Question] How to quantitatively measure EE experience

#23 2015-04-14 11:55:32

Re: [Question] How to quantitatively measure EE experience

#24 2015-04-14 14:00:06, last edited by Bobithan (2015-04-14 14:12:35)

Re: [Question] How to quantitatively measure EE experience

#25 2015-04-14 15:38:04, last edited by Hexagon (2015-04-14 15:55:46)

Re: [Question] How to quantitatively measure EE experience

Board footer