Scoring

Quote from Kender

Mar 09, 2011 - 20:50:03

So, our challenge is to arrange these three variables in a formula in such a way that everyone is happy:
- p_solved - Maximum number of "points" on a site divided by the users actual "points" on the site. (some sites use challs as points, some have their own points system, others use rank, etc.)
- base_score - Manually adjustable per-site fixed number used for weighting in rare cases.
- avg_solved - The average of p_solved of all the WeChall users linked to the site.

Here's a starter:
2000 * pow(p_solved,2) * 1/pow(1-avg_solved,2)

This gives for example:

GeSHi`ed Plaintext code

1 2 3 4 5

site            first   last    total SPOJ            0.65    2722.09 2847989 HackThisSite    3.50    710.25  36401.12 Hax.tor.hu      3.28    326.26  8238.82 first: points for 1st solve, last: points for last solve, total: total points when site complete

Which is still a long way from perfect, but you catch my drift ;)

That is actually looking considerably better to me. Thank you for considering the points raised in this thread.

I didn't notice where that '2000' came from. Is that the base_score you mentioned?

Again, I understand your hesitance about using user-voted difficulty but I have to stress that I consider the 'avg_solved' a pretty bad way to measure difficulty. I think this is because site age (and probably 'time that it has been linked to WeChall') is a factor, maybe a large one. The older the site the longer people have had to find the site and the longer people have had to solve the challenges. Thus, if used as a measure of difficulty, 'difficulty' is going to be skewed downward for older sites. That is why Ma's and Electrica are (absurdly in my opinion) mid-range difficulty judged by average solved.

Haxtor, I think, is by the same formula most difficult not because it is the most difficult, though it has some tough problems, but because of its linear design. You get locked at a challenge until you solve it, which is quite different from most sites. And you don't necessarily get locked at a challenge that is harder than those unsolved ones above it. Sometimes the next one is easier than the one you just solved. My guess is that a change to Hax.tor's structure would radically change its percentage solved.

Using 'avg_solved' also skews difficulty upwards for large sites, since the simple fact that there are more challenges means it takes longer to get a high percentage solved, and chances are fewer people persist long enough to get the high solution. Neither of those really indicates difficulty though.

Anyway, making progress...

Quote from Kender

Mar 09, 2011 - 20:50:03

Quote from dloser

Perhaps more interesting with new challenges is what this decrease in points is supposed to mean. Did my skills decrease? Sure, I'm getting older, but come on!

Well, if you solve the same old basic starter challenges on another dozen sites you score will increase. Does that mean you skills increased?

My point exactly. The scoring has little to do with skill.

Quote from Kender

Mar 09, 2011 - 20:50:03

Making the scoring linear so that you get the same amount of points regardless of how far you are on a site is not an option for me.
That would mean that a person doing he easiest 5 challenges on 20 different sites will rank higher than someone who has finished both Electrica and +Ma's.

Please remember the context in which this was suggested. The scoring here is not meant to reflect skill, intelligence or whatever you want it to be. Also, the same could be said on the level of a site itself. A person who does a bunch of easy ones on MiB is scored higher than a person who does almost as many difficult ones. It's just that you look at a higher level because that's all you've got.

Quote from Kender

Mar 09, 2011 - 20:50:03

Using the num_challs anywhere is also not an option for me.
Sites are free to structure their sites, scoring and ranking however they want and only report progress percentage to WeChall. Any other data has always been optional.

And is this because you want sites to be as free as possible or because you originally only used the percentages? And are percentages really the only and most general way to report progress? If you ask me it is just a (possibly historic) choice. Mapping percentages to a "challenges+score" framework doesn't seem particularly hard (and so far doesn't seem necessary anywhere).

Quote from Kender

Mar 09, 2011 - 20:50:03

So, our challenge is to arrange these three variables in a formula in such a way that everyone is happy:
- p_solved - Maximum number of "points" on a site divided by the users actual "points" on the site. (some sites use challs as points, some have their own points system, others use rank, etc.)
- base_score - Manually adjustable per-site fixed number used for weighting in rare cases.
- avg_solved - The average of p_solved of all the WeChall users linked to the site.

Of which only the first is actual input. The others are an arbitrary constant and a (somewhat arbitrary) derived value.

Quote from Kender

Mar 09, 2011 - 20:50:03

Here's a starter:
2000 * pow(p_solved,2) * 1/pow(1-avg_solved,2)

What does the last factor do? Apart from the (granted, unlikely) possibility of a division by zero, it seems to do little for "hard" sites (low avg_solved) and give enormous amounts of points for "easy" sites (high avg_solved). I also don't see the relation with your table, so I'm guessing something went wrong somewhere (not necessarily on your end).

Look, I don't intend to be a pedantic prick (which I surely am), but all this scoring business appears to me as almost randomly trying to find formulas that feel ok. Just because you want it to be something that cannot really be expressed in terms of the inputs you have.

Quote from dloser

Mar 10, 2011 - 01:53:36

Look, I don't intend to be a pedantic prick (which I surely am), but all this scoring business appears to me as almost randomly trying to find formulas that feel ok. Just because you want it to be something that cannot really be expressed in terms of the inputs you have.

Yeah, isn't it fun?
This is real life, nothing is going to be exact or precise.
"all this scoring business" is just me trying to work with people who have a problem with the current system in combination with SPOJ.