[Question] How to quantitatively measure EE experience

Kyle97 · 2015-04-14 15:52:26

Hexagon wrote:

Bobithan wrote:
Also, for measurement of player skill, time isn't what you want to look at. Some minigames can take considerably longer to complete in one go than others, so the best method would instead be attempts rather than time spent
I sort of agree with this. I think that if you take more time on a certain level, it might show that you are not able to quickly cognitively process how to complete the level. Since experienced users have more experience, it is possible that they may have already seen the patterns in the minigames and so would be able to do it faster. However, I'm not sure if those would be correlated. It's an interesting point though.
For example, it usually takes me about 30 seconds to do one hook jump (I'm terrible at those) but it would take me one try. I'm not really experienced so it would take me more time.

Why not both? Would be interesting to see how much of an effect rushing a minigame has on your death count, and if the increased attempts actually causes a player who rushes to take longer to finish than a player taking their time.

Bobithan

Hexagon · 2015-04-14 16:22:00

Kyle97 wrote:

Another factor to consider is time. Should there be a time limit for each mini, so that the player must complete it before the limit is reached, or they get points deducted? Or should it be unlimited?

I think it should be basically unlimited (but if you're spending 5 minutes trying to solve one game, you're probably not going to finish it.)

I'm not sure if time should be used, but if it is then:

- points should be deducted such that:
-- if you take 30 seconds on a level, and don't finish (and go to a different version) you will be docked more points than if you took 50 seconds and did finish it. This raises a question that if I hate hook jumps, and as soon as I go to the version with hook jumps I give up, how many points should I be docked?
- points might follow the equation y = -sqrt(1/x)(z) + (1/2)(1/x)^(3/2) * z - 2x where z = the rate at which people get more points based on how quickly they finish the level (excluding people who skip it.) The graph is basically 1/x until it meets the origin, at which point it decreases linearly.

Muftwin · 2015-04-14 20:42:52

i like how a lot of this is about if you can do a hook jump or not. if you can do a hook jump at all its completely trivial to do them.

you still havent explained what purpose this has. I think establishing a purpose is important to actually figuring out how to go about this.

MIHB_casts_confuseplayer

Hexagon · 2015-04-14 21:15:57

Kyle97 wrote:

Interesting topic. Ive never really thought about it. You would have to group together a bunch of different minis and see what percent of players could complete each. You could then rank the minis based on percentage of players who completed it.

I just re-read your post, and this is an extremely good idea. This would make things *much* easier, as people could essentially submit (or we could find) levels, and based on the percent completion can rearrange them in terms of difficulty. Once the levels are arranged, the experience level of the users can be determined, based on previous data.

Fradeshan · 2015-04-14 21:20:26

Hexagon wrote:

- make the user answer a capasha

funny

Hexagon · 2015-04-14 21:21:45

Fradeshan wrote:

Hexagon wrote:
- make the user answer a capasha
funny

Oops! I meant captcha. Updated post.

Tamashiimizu · 2015-04-14 21:35:30

But there's also a problem related to the keyboard. I always use the arrow keys, but some combinations don't work properly, making some minis impossible for arrow keys. Even if I use other part of the keyboard, my dexterity is reduced.

Hexagon · 2015-04-14 21:38:01

Tamashiimizu wrote:

But there's also a problem related to the keyboard. I always use the arrow keys, but some combinations don't work properly, making some minis impossible for arrow keys. Even if I use other part of the keyboard, my dexterity is reduced.

Could you expand on that? I want to make sure that everyone is able to complete the mini games with the same resources.

Tamashiimizu · 2015-04-14 21:44:25

For example, I can't even jump while holding the (Left+Up) arrow keys in my keyboard, but if I use the ASDW control, I can. But since i'm not used to play using ASDW, i wouldn't play as well as I would usually .

skullz17 · 2015-04-14 22:05:10

But the idea is that you play in the test the same way that you would play normally. So if you can't do that mini in a test, you can't do it in a normal world either. It does affect your skill level.

Hexagon · 2015-04-14 22:27:46

Tamashiimizu wrote:

For example, I can't even jump while holding the (Left+Up) arrow keys in my keyboard, but if I use the ASDW control, I can. But since i'm not used to play using ASDW, i wouldn't play as well as I would usually .

You might have to look into your hardware and your keyboard to make sure everything is working properly. If other users report this issue, I'll try to see if any accommodations can be made.

skullz17 wrote:

But the idea is that you play in the test the same way that you would play normally. So if you can't do that mini in a test, you can't do it in a normal world either. It does affect your skill level.

I realize that some players will never be able to do certain mini games (or just the top 1%) and won't be able to do them on the test, however there has to be a method to categorize the difficultly so that during the test players aren't bombarded with really hard stuff and quit early.

MIHB_casts_confuseplayer · 2015-04-14 23:01:46

Hexagon wrote:

I realize that some players will never be able to do certain mini games (or just the top 1%) and won't be able to do them on the test, however there has to be a method to categorize the difficultly so that during the test players aren't bombarded with really hard stuff and quit early.

I still don't understand what the point of this would be, but categorizing minis by difficulty should be a relatively simple process if you're willing to use player trials to update the mini difficulty. Basically all you need to do is assign a "rough" difficulty level to a minigame and then allow players to choose minis to play. Aggregate player success on different minis can be compared and the true difficulty level discovered accordingly.

Hexagon · 2015-04-15 00:29:00

MIHB_casts_confuseplayer wrote:

I still don't understand what the point of this would be, but categorizing minis by difficulty should be a relatively simple process if you're willing to use player trials to update the mini difficulty. Basically all you need to do is assign a "rough" difficulty level to a minigame and then allow players to choose minis to play. Aggregate player success on different minis can be compared and the true difficulty level discovered accordingly.

Well, the main point of the application is to give users an experience index, in which they can use in order to filter or sort levels by difficulty, depending on how other similar users faired. Mini games's difficulty can be gauged based on player success, however not all mini games incorporate very very difficult and very very easy levels, which decrease the granularity of the data. Some may favour one feature over another (such as hook jumps) and so unfairly rate people that could have gotten past it would have quit.

Furthermore, world owners could filter users out based on experience index, because each user has their own experience index created.

MIHB_casts_confuseplayer · 2015-04-15 06:12:31

Hexagon wrote:

MIHB_casts_confuseplayer wrote:
I still don't understand what the point of this would be, but categorizing minis by difficulty should be a relatively simple process if you're willing to use player trials to update the mini difficulty. Basically all you need to do is assign a "rough" difficulty level to a minigame and then allow players to choose minis to play. Aggregate player success on different minis can be compared and the true difficulty level discovered accordingly.
Well, the main point of the application is to give users an experience index, in which they can use in order to filter or sort levels by difficulty, depending on how other similar users faired. Mini games's difficulty can be gauged based on player success, however not all mini games incorporate very very difficult and very very easy levels, which decrease the granularity of the data. Some may favour one feature over another (such as hook jumps) and so unfairly rate people that could have gotten past it would have quit.
Furthermore, world owners could filter users out based on experience index, because each user has their own experience index created.

This sounds like an entirely different problem--rating levels based on difficulty, rather than minis or players. It doesn't matter at all if a level has a mix of minigame difficulty--completion of the world provides a pretty straightforward 0/1 data point, and across enough users and levels you should be able to achieve a reasonable estimate of the difficulties of different levels. Thats fine if you want to give players the ability to easily find a level with a specific difficulty.

If you're more concerned with just sticking players into levels of an appropriate difficulty level, thats even easier--0/1 data points for world completion make it very easy to recommend suitable levels with the right setup,

Hexagon · 2015-04-15 12:30:32

MIHB_casts_confuseplayer wrote:

This sounds like an entirely different problem--rating levels based on difficulty, rather than minis or players

Oops, sorry, I meant to say mini games. I should probably make sure that my terminology is consistent throughout posts.

I do agree with what you said concerning rating a level's difficulty, if need be, using the data points for world completion.

Fe · 2015-04-15 14:07:44

It's going to be like the tierlist for Smash Bros. People are going to argue constantly about what the exact quantitative value for this mini is, but they'll probably agree that X is harder than Y but A is about as difficult as B.

Hexagon · 2015-04-15 14:24:33

Fe wrote:

but they'll probably agree that X is harder than Y but A is about as difficult as B.

That's okay because if they agree that X is harder than Y, then they can agree that Y is harder than Z and so on until the entire hierarchy of difficulty is established.

goeyfun · 2015-04-15 14:49:46

Wtf with the anti-copy techniques lol...

Hexagon · 2015-04-15 15:10:15

I'm not sure if someone already mentioned this but difficulty based on completion rate is a good idea however the sample size and distribution can change the outcome of the completion rate and attempts. Some mini games have titles like "easy minigames" so it is possible that players who want easy mini games are going to play it and therefore the harder games will have a lower completion rate, making them appear much harder.

goeyfun wrote:

Wtf with the anti-copy techniques lol...

The only way to normalize the differences in mini games would be to give a player an XP level, and then based on their XP level it would have to compare to other people of a similar XP level and completion rate to determine difficulty, which would involve standardized testing. The standardized test should not be copied (in order to prevent cheating) which is because why the anti-copy techniques are so elaborate.

Kyle97 · 2015-04-15 15:25:12

Idea:
Lets strip minis down to the bone and divide them into there basic parts. Then, each element of a minigame can be tested individually to see how a player performs on each task, showing their strengths and weaknesses in each basic element.

For example, lets use a somewhat complex mini that involves climbing a hook jump, then completing two long jumps that require significant momentum to clear. In this mini, falling at any point will result in a death (I'll add a picture later for clarification). Now, this somewhat complex mini can be divided into 2 parts: the hook jump, and the long jump, both of which are equally important.

Now lets backtrack a bit. Lets say a world was created that tested this player in each of these basic elements, rating their performance on each and determined which they were best and worst at. Using this example data, the example player showed to excel at hook jumps, completing then quickly and with few errors. However, the player lacked in the ability to complete long jumps, often mistimeing jumps and dieing many times, as well as taking considerably longer time to complete then the average player.

Using this knowledge, we can expect the player to MOST LIKELY easily complete the hook jump, but fail at the long jump element of mini. We can also expect the difficulty of the entire mini FOR THIS PLAYER to be about average, because it involves elements that he is both good and bad at.

Now, lets say another player is also attempting this mini. This new player just started playing EE and lacks in both hook jump and long jump skills. Because of this, the mini is likely to be very difficult for this new player.

Finally, a very experienced player joins the room, excelling at both required elements. The difficulty of the mini for this user is likely to be very easy.

As you can see, using this we can estimate how each player would perform on almost any mini depending on the elements that make up the mini, assuming he has completed the element skills test.

We can also determine average difficulty. By averaging every players scores in each element, it would give a rough estimate of which element is most difficult to the community as a whole. We would be able to rank each element in difficulty and estimate how challenging a complex mini with many parts would be to the average EE user.

This system does have its problems, however, because elements can vary in difficulty. For example, 1 hook is much easier to complete than 5 stacked on top of each other. This would have to be taken into consideration when a minis difficulty is determined, either for an individual player or the entire community.

Well, thats the end of my novel. Once again I'd love to help develop a system like this if someone is actually going to.

Hexagon · 2015-04-15 16:44:20

If you could help kyle, that'd be great! Unfortunately I can't really do much this week though.

MIHB_casts_confuseplayer · 2015-04-15 20:54:35

Hexagon wrote:

MIHB_casts_confuseplayer wrote:
This sounds like an entirely different problem--rating levels based on difficulty, rather than minis or players
Oops, sorry, I meant to say mini games. I should probably make sure that my terminology is consistent throughout posts.
I do agree with what you said concerning rating a level's difficulty, if need be, using the data points for world completion.

.
Dude, you still haven't given a convincing reason why individual minis need to be rated rather than levels as a whole. Very few levels give the player the ability to choose what minigames they play inside the level, so variation in the difficulty of a single minigame matters very little, all that matters is how many minis a player can overcome before they reach a point where they can go no further. Completion is the simplest measure of this, time spent in a level is another simple measure. Based on a given user's data points of time they've spent in a level which levels they've completed, and inferred experience measures such as how long ago they signed up and amount of energy they've spent, you can do a couple things. You can either find them a "best match" player, "second best match", and so forth. Using that you can create a network of player activity which can give you an idea where a player sits in the "skill" heirarchy and an idea where a level sits in the "difficulty" hierarchy. Or, alternatively, you can use a simple classification metric and have players and levels automatically assigned to different classes, using class averages on inferred experience measures to rank on both skill and difficulty.

BEE · 2015-04-15 22:31:27

I still am not sure why you wish to do this, but I have a suggestion.

In rock climbing, the scales are not necessarily set in stone, but rather are a bit fluid.

For example, the scale is supposed to be 5.0-5.12, but there are walls labeled 5.8+ or 5.11a. The grade usually comes from the first people to climb it that are experienced. They give it a grade that hey think it is, then other climbers come along and agree or disagree. When an indoor climbing place creates a new route, they often put up a small dry erase board for climbers to put a tally mark on their vote for the specific grade. Because you are measuring the intersection of skill and difficulty, as well as factoring in difficulty, the voting system works the best. After a few weeks of votes, they label the new route with the highest voted, or even label it as a + or -, meaning it is slightly more difficult/easy than a normal grade, but not as easy as the grade below.

It would take experienced players to set up a system like this in EE, but shift already has a similar system. One where an original owner states a grade, and other players (well, mostly just Nou, though I'm sure other people have input if need be) accept that grade or change it according to what they think.

Hexagon · 2015-04-15 23:34:13

BEE wrote:

I still am not sure why you wish to do this, but I have a suggestion.

The main reasons I want to do this is because:
- be able to assess a mini games difficulty accurately, so that people who join are not confused about why this would be easy for someone else but would be impossible for them
-- there have been a few suggestions to make EE more popular, and this involves recruiting a lot of newer players. By finding easy levels (they are easy to make) and making the levels increase in difficulty will decrease the "rage quits" because you gain confidence that you can successively complete harder and harder minis, rather than being shocked by this super hard one
- being able to assign XP levels to players.
-- these XP levels can be used to allow minis/levels to be sorted by difficulty
-- used for EE Analytics for many of the systems derived thereof
--- anti-vandal: allow/disallow people with a certain XP in your world (allows newer plays to join with other newer players to make them feel better)
--- there might be a correlation between XP and the probability of trolling a world, I'm not sure yet
--- recommendation system: pick levels that other people played in the past dependent on their XP

BEE wrote:

In rock climbing, the scales are not necessarily set in stone, but rather are a bit fluid..

That's an interesting idea. It's sort of like the idea about using success rate to determine difficulty (as people who succeed would find it easier than others who don't.)

However, From the data that I collected a few days ago, using a sample size of about 25 there are very very few people who rate themselves as moderately experienced in EE (most are around 8-10 and 1-2); I'm not sure if this would affect the scale or not, probably not.

MIHB_casts_confuseplayer wrote:

Dude, you still haven't given a convincing reason why individual minis need to be rated rather than levels as a whole.

Well, I sort of make a weak assumption that the average difficulty of all minis in a level (or all components in a level) would be equal to the difficultly of the level.

MIHB_casts_confuseplayer wrote:

Very few levels give the player the ability to choose what minigames they play inside the level, so variation in the difficulty of a single minigame matters very little, all that matters is how many minis a player can overcome before they reach a point where they can go no further. Completion is the simplest measure of this, time spent in a level is another simple measure.

I was thinking about using mini games to derive the predicted XP of someone, because mini games have many components, and so the data can be very fine-grained. However you did bring up a good point, and this limits the applicability of the XP of the player to the difficultly of the level, because if someone can solve a mini game, well, it just shows a purely macro view of the player, and doesn't take into account contextual considerations.

MIHB_casts_confuseplayer wrote:

Based on a given user's data points of time they've spent in a level which levels they've completed, and inferred experience measures such as how long ago they signed up and amount of energy they've spent, you can do a couple things. You can either find them a "best match" player, "second best match", and so forth. Using that you can create a network of player activity which can give you an idea where a player sits in the "skill" hierarchy and an idea where a level sits in the "difficulty" hierarchy.

This sounds interesting. Since it's still in the planning stages, this could be implemented.

MIHB_casts_confuseplayer wrote:

Or, alternatively, you can use a simple classification metric and have players and levels automatically assigned to different classes, using class averages on inferred experience measures to rank on both skill and difficulty.

This is more what I think the program could be used for, but the other solution would be totally workable too.

BEE · 2015-04-15 23:51:28

Hexagon wrote:

MIHB_casts_confuseplayer wrote:
Based on a given user's data points of time they've spent in a level which levels they've completed, and inferred experience measures such as how long ago they signed up and amount of energy they've spent, you can do a couple things. You can either find them a "best match" player, "second best match", and so forth. Using that you can create a network of player activity which can give you an idea where a player sits in the "skill" hierarchy and an idea where a level sits in the "difficulty" hierarchy.
This sounds interesting. Since it's still in the planning stages, this could be implemented.

Careful with including account age. While I'm a rare nub, I would throw any data that factored that in.

According to http://ee.yonom.org/, I have had this account for almost 4.5 years, spent 234,245 energy and yet I am still a nub.

Official Everybody Edits Forums

#26 2015-04-14 15:52:26, last edited by Kyle97 (2015-04-14 15:54:42)

Re: [Question] How to quantitatively measure EE experience

#27 2015-04-14 16:22:00

Re: [Question] How to quantitatively measure EE experience

#28 2015-04-14 20:42:52

Re: [Question] How to quantitatively measure EE experience

#29 2015-04-14 21:15:57, last edited by Hexagon (2015-04-14 21:23:37)

Re: [Question] How to quantitatively measure EE experience

#30 2015-04-14 21:20:26

Re: [Question] How to quantitatively measure EE experience

#31 2015-04-14 21:21:45

Re: [Question] How to quantitatively measure EE experience

#32 2015-04-14 21:35:30

Re: [Question] How to quantitatively measure EE experience

#33 2015-04-14 21:38:01

Re: [Question] How to quantitatively measure EE experience

#34 2015-04-14 21:44:25, last edited by Tamashiimizu (2015-04-14 21:46:10)

Re: [Question] How to quantitatively measure EE experience

#35 2015-04-14 22:05:10

Re: [Question] How to quantitatively measure EE experience

#36 2015-04-14 22:27:46, last edited by Hexagon (2015-04-14 22:30:38)

Re: [Question] How to quantitatively measure EE experience

#37 2015-04-14 23:01:46

Re: [Question] How to quantitatively measure EE experience

#38 2015-04-15 00:29:00

Re: [Question] How to quantitatively measure EE experience

#39 2015-04-15 06:12:31

Re: [Question] How to quantitatively measure EE experience

#40 2015-04-15 12:30:32

Re: [Question] How to quantitatively measure EE experience

#41 2015-04-15 14:07:44

Re: [Question] How to quantitatively measure EE experience

#42 2015-04-15 14:24:33

Re: [Question] How to quantitatively measure EE experience

#43 2015-04-15 14:49:46

Re: [Question] How to quantitatively measure EE experience

#44 2015-04-15 15:10:15, last edited by Hexagon (2015-04-15 15:10:36)

Re: [Question] How to quantitatively measure EE experience

#45 2015-04-15 15:25:12

Re: [Question] How to quantitatively measure EE experience

#46 2015-04-15 16:44:20

Re: [Question] How to quantitatively measure EE experience

#47 2015-04-15 20:54:35, last edited by MIHB_casts_confuseplayer (2015-04-15 20:57:27)

Re: [Question] How to quantitatively measure EE experience

#48 2015-04-15 22:31:27

Re: [Question] How to quantitatively measure EE experience

#49 2015-04-15 23:34:13

Re: [Question] How to quantitatively measure EE experience

#50 2015-04-15 23:51:28

Re: [Question] How to quantitatively measure EE experience

Board footer