How to fake out the new credit system

Message boards : Number crunching : How to fake out the new credit system

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
Bird-Dog
Avatar

Send message
Joined: 15 Dec 05
Posts: 42
Credit: 116,749
RAC: 0
Message 25110 - Posted: 27 Aug 2006, 10:22:14 UTC - in response to Message 25098.  
Last modified: 27 Aug 2006, 10:24:45 UTC

Are there any hosts that run a standard client and unmodified xml benchmarks that are getting less granted than claimed?

Yes, I have some WU's.
click here

Cheers
It will be interesting to keep an eye on how the results pan out for this box.
Any others?


All of mine apart from 1 or 2 are granted less than claimed.
Stock everything. (edit)
ID: 25110 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 25112 - Posted: 27 Aug 2006, 10:37:44 UTC - in response to Message 25110.  



All of mine apart from 1 or 2 are granted less than claimed.
Stock everything. (edit)


Does it make any difference if you increase the target run time?


ID: 25112 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
TestPilot

Send message
Joined: 13 Jun 06
Posts: 29
Credit: 0
RAC: 0
Message 25113 - Posted: 27 Aug 2006, 10:47:31 UTC - in response to Message 25109.  

All PowerMACs with the old IBM processors get less than 50%. For example here:

Wow! So that means in the terms of the new credit system that powermacs either take longer to do the same work as the "average" host or that the standard boinc client overestimates the benchmarks for macs.

Yes! Exactly! So new system grant fair amount of points to computers, irrelevantly of claimed amount of points and benchmark results!

Furthermore. At the top of the thread it was suggested that if big enough team over claim points - they will eventually get more points. That is true, but ironically that would not help cheating team. Simply because by over claiming they increasing amount of points not just for themselves, but also for there rivals/competitors! The only effects would be that the project as whole, Rosetta at home will claim slightly more boinc points compare to Einstein@Home / ClimatePrediction.Net...
ID: 25113 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bird-Dog
Avatar

Send message
Joined: 15 Dec 05
Posts: 42
Credit: 116,749
RAC: 0
Message 25115 - Posted: 27 Aug 2006, 10:51:37 UTC - in response to Message 25112.  
Last modified: 27 Aug 2006, 10:52:26 UTC



All of mine apart from 1 or 2 are granted less than claimed.
Stock everything. (edit)


Does it make any difference if you increase the target run time?



I`ll increase it to 2hrs for now when the one its doing is finished. Can`t put it up longer just yet because of other projects. Will increase it more later.
ID: 25115 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 25117 - Posted: 27 Aug 2006, 10:58:17 UTC

Those mac results really throw a mental spanner in the works.

Because we're now using a system that deals wih averages I would've expected the higher scoring (performing) hosts to take a minor drop while the lower scoring hosts take a rise. Intuitively, I would have expected a slight drop in standard AMD's as reported by mnb-fin and Bird-Dog. AMD's were afterall (reportedly) the top of the heap.

The problem with this line of reasoning is the PowerMacs, does it really take one approximately 50% longer to create/find/calculate a model as indicated by David Kim's machine as compared to the "average" machine.

Are the results reported by David Kim's machine indicative of all powermacs?
ID: 25117 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 25119 - Posted: 27 Aug 2006, 11:12:29 UTC - in response to Message 25113.  



Furthermore. At the top of the thread it was suggested that if big enough team over claim points - they will eventually get more points. That is true, but ironically that would not help cheating team. Simply because by over claiming they increasing amount of points not just for themselves, but also for there rivals/competitors! The only effects would be that the project as whole, Rosetta at home will claim slightly more boinc points compare to Einstein@Home / ClimatePrediction.Net...


True, but if a team or teams (or any group together) set a high cache, overclaim, and have relatively quick machines they should be able to manipulate the system. Whether it's worth it is another matter. The high cache is necessary to get wu's of the same type, and/or to make an influence in a wu batch as early as possible. They will raise everybody elses score initially but the more standard client, and slower hosts return results the average will drop. If they are returning results faster than the averge host then this will help too.

ID: 25119 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
TestPilot

Send message
Joined: 13 Jun 06
Posts: 29
Credit: 0
RAC: 0
Message 25121 - Posted: 27 Aug 2006, 11:26:22 UTC - in response to Message 25119.  


True, but if a team or teams (or any group together) set a high cache, overclaim, and have relatively quick machines they should be able to manipulate the system. Whether it's worth it is another matter. The high cache is necessary to get wu's of the same type, and/or to make an influence in a wu batch as early as possible. They will raise everybody elses score initially but the more standard client, and slower hosts return results the average will drop. If they are returning results faster than the averge host then this will help too.

Yes, but to effectively profit from this you need:
1) Overclaim points only for some type of specially selected WU(which is relatively easy to implement)
2) Get thouse WU you are overclaiming on average more often than non cheating computer gets.
The problem is 2). Hm... Yes, you can count only WU of that type you are overclaiming! Just return all other types with errors, and wait until server give you thouse WU you really want!

So, solution to the topic question found, am I right?

ID: 25121 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
FluffyChicken
Avatar

Send message
Joined: 1 Nov 05
Posts: 1260
Credit: 369,635
RAC: 0
Message 25127 - Posted: 27 Aug 2006, 12:29:32 UTC

Trog Dog, I don't understand how increasing the runtime will alter teh credits ?

Since 1x4 result should be the same as 4x1 results, else there is a major flaw in the system?

It may 'look' better in the stats listing since the average of your result over 24hrs should be more consistent when shouwn as a bulk credit, than seeing them scored individually. But the overall amount will (should hopefully!) be the same.
Team mauisun.org
ID: 25127 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
STE\/E

Send message
Joined: 17 Sep 05
Posts: 125
Credit: 4,103,208
RAC: 167
Message 25129 - Posted: 27 Aug 2006, 12:45:48 UTC

I've tried running the WU's at different Time Lengths, 1 hr & 2 hr & 4 hr & 8 hr, the Credits seem to work out the same per hour average no matter what length I run them ...
ID: 25129 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bird-Dog
Avatar

Send message
Joined: 15 Dec 05
Posts: 42
Credit: 116,749
RAC: 0
Message 25130 - Posted: 27 Aug 2006, 13:00:17 UTC - in response to Message 25112.  
Last modified: 27 Aug 2006, 13:01:10 UTC



All of mine apart from 1 or 2 are granted less than claimed.
Stock everything. (edit)

Does it make any difference if you increase the target run time?


Increasing the time has made no differance, same average /hr
ID: 25130 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 25131 - Posted: 27 Aug 2006, 13:07:24 UTC - in response to Message 25121.  


Yes, but to effectively profit from this you need:
1) Overclaim points only for some type of specially selected WU(which is relatively easy to implement)
2) Get thouse WU you are overclaiming on average more often than non cheating computer gets.
The problem is 2). Hm... Yes, you can count only WU of that type you are overclaiming! Just return all other types with errors, and wait until server give you thouse WU you really want!

So, solution to the topic question found, am I right?


That's why the high cache, presuming that wu are relaeased in batches. 10 day cache you would presume a higher cluster of the same wu's than .1 day cache.

ID: 25131 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 25132 - Posted: 27 Aug 2006, 13:17:49 UTC - in response to Message 25127.  
Last modified: 27 Aug 2006, 13:28:12 UTC

Trog Dog, I don't understand how increasing the runtime will alter teh credits ?

Since 1x4 result should be the same as 4x1 results, else there is a major flaw in the system?

It may 'look' better in the stats listing since the average of your result over 24hrs should be more consistent when shouwn as a bulk credit, than seeing them scored individually. But the overall amount will (should hopefully!) be the same.


Since the credit granted is based on the models/decoys found it's just a check of whether the incidence of models/decoys in a wu increases or decreases over time.

So if the rate is constant within a wu (ie if you find 1 in a 1 hour wu, then you will find 10 in a 10 hour wu) then you have more chance of influencing the score by returning smaller wu's, because you are returning them quicker you get more bites at the cherry.
ID: 25132 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ingleside

Send message
Joined: 25 Sep 05
Posts: 107
Credit: 1,514,472
RAC: 0
Message 25134 - Posted: 27 Aug 2006, 13:58:24 UTC

Well, let's say 10% is trying to cheat by claiming too much, how will this effect the average granted credit?

Overclaim - increase in average granted credit per model:
5x - 40%
4x - 30%
3x - 20%
2x - 10%
1.5x - 5%
1.1x - 1%

Since a constant 2x-overclaim only gives 10% increase, an ocassional high claim will not give any noticeable effect, except if it's one of the very 1st. returned for a specific wu-type. Therefore, someone trying to cheat must basically all the time overclaim... But, this makes it easier to detect server-side, so can example discard any claims that is markedly higher than average, and maybe zeroing a couple of the "worst offenders"...

The "weak" spot is at the beginning, so my recommendation is to delay crediting until N% or N models is in...

"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
ID: 25134 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 11 Feb 06
Posts: 316
Credit: 6,621,003
RAC: 0
Message 25137 - Posted: 27 Aug 2006, 14:40:45 UTC - in response to Message 25129.  

I've tried running the WU's at different Time Lengths, 1 hr & 2 hr & 4 hr & 8 hr, the Credits seem to work out the same per hour average no matter what length I run them ...


My team has a stats site that shows your credits over the past 24hours (rolling).

When the new credit system came on, it was pointed out that those returning results first, or close to first, can get what they claim (or close to it). So I set my run time to te minimum (1 hr). I was getting consistantly in the high 5k range. Then I tried setting it to 6 hour run time. My numbers increased to the low 7k range. So then I tried 8 hour run time, and they dropped to high 6k range.

FWIW, after each change, I waited the runtime + 24 hours to let the rolling 24hr window settle.

I am going back to 6 hours to see if it goes back up.

It's like target practice in the dark! There is no pattern or reason to the results.
Reno, NV
Team: SETI.USA
ID: 25137 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
tralala

Send message
Joined: 8 Apr 06
Posts: 376
Credit: 581,806
RAC: 0
Message 25138 - Posted: 27 Aug 2006, 14:44:55 UTC - in response to Message 25127.  

Trog Dog, I don't understand how increasing the runtime will alter teh credits ?

Since 1x4 result should be the same as 4x1 results, else there is a major flaw in the system?

<nitpicking>
Rosetta spends about 1 minute in initialisation which needs to be done only at startup, so you loose with 1 hour WU about a minute every hour (~1.66 %). ;-)
</nitpicking>
ID: 25138 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Biggles
Avatar

Send message
Joined: 22 Sep 05
Posts: 49
Credit: 102,114
RAC: 0
Message 25145 - Posted: 27 Aug 2006, 15:33:18 UTC

I understand how your exploit works feet1st, but there's a way to limit its effectiveness. BOINC allows for a maximum daily work unit quota, which can be reduced for every failed work unit. If that is enforced that then limits the amount of work units a person can try cheating on.

Not perfect by a long shot, but it does limit the amount of times it can be done. And of course if a machine gets to the top host spot and has failed every work unit, it will stand out.
ID: 25145 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
FluffyChicken
Avatar

Send message
Joined: 1 Nov 05
Posts: 1260
Credit: 369,635
RAC: 0
Message 25155 - Posted: 27 Aug 2006, 18:04:04 UTC - in response to Message 25138.  

Trog Dog, I don't understand how increasing the runtime will alter teh credits ?

Since 1x4 result should be the same as 4x1 results, else there is a major flaw in the system?

<nitpicking>
Rosetta spends about 1 minute in initialisation which needs to be done only at startup, so you loose with 1 hour WU about a minute every hour (~1.66 %). ;-)
</nitpicking>

Doesn't it initialise at the beginning of ever 'run' ? (Since it kind of restarts after each one (a run lenght is actual split up into descrite time lengths)

and 1.66% will not be noticable in the variation of decoy lengths anyway ;-)
Team mauisun.org
ID: 25155 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
tralala

Send message
Joined: 8 Apr 06
Posts: 376
Credit: 581,806
RAC: 0
Message 25156 - Posted: 27 Aug 2006, 18:06:41 UTC - in response to Message 25155.  

Trog Dog, I don't understand how increasing the runtime will alter teh credits ?

Since 1x4 result should be the same as 4x1 results, else there is a major flaw in the system?

<nitpicking>
Rosetta spends about 1 minute in initialisation which needs to be done only at startup, so you loose with 1 hour WU about a minute every hour (~1.66 %). ;-)
</nitpicking>

Doesn't it initialise at the beginning of ever 'run' ? (Since it kind of restarts after each one (a run lenght is actual split up into descrite time lengths)

and 1.66% will not be noticable in the variation of decoy lengths anyway ;-)


No as I said the initialisation is only once performed at startup. Of course those 1.66 % are not noticeable, I just mentioned it for the sake of completeness.
ID: 25156 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 11 Feb 06
Posts: 316
Credit: 6,621,003
RAC: 0
Message 25159 - Posted: 27 Aug 2006, 18:50:19 UTC

So.... Has anyone hit the first/early return lotto yet?


Reno, NV
Team: SETI.USA
ID: 25159 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Feet1st
Avatar

Send message
Joined: 30 Dec 05
Posts: 1755
Credit: 4,690,520
RAC: 0
Message 25161 - Posted: 27 Aug 2006, 18:56:32 UTC - in response to Message 25145.  
Last modified: 27 Aug 2006, 18:57:12 UTC

I understand how your exploit works feet1st, but there's a way to limit its effectiveness. BOINC allows for a maximum daily work unit quota, which can be reduced for every failed work unit. If that is enforced that then limits the amount of work units a person can try cheating on.

Not perfect by a long shot, but it does limit the amount of times it can be done. And of course if a machine gets to the top host spot and has failed every work unit, it will stand out.

Biggles
I didn't intentionally abort the WUs. BOINC did it for me for unknown reason. If I kept modifying my file after every benchmark is run by BOINC, I could crunch forward with the claim and not require any extra number of WUs... because, afterall, I'm NOT crunching any faster.

My point was the old system is EASY to manipulate. And in various ways, many people did that, and so with relation to actual TeraFLOPS, Rosetta's daily credits issued was inflated as compared to the actual work done. Therefore, it is only to be expected that the daily number would drop under the new system. Teams pulling off... and on, certainly sways the numbers too.

The initialization also occurs when Rosetta has been removed from memory. Either by ending BOINC, or by preempting to another project and not keeping in memory.

Trog Dog
Consider this, if a team reports thousands of models crunched... and happens to both be SENT, AND to RETURN these WUs first with overclaimed credits... if they succeed at ALL of that... then the credit per model granted to EVERYONE downstream from there is also inflated. In essence, for all the effort, they've benefitted everyone (creditwise) as much as themselves. It's a zero-sum game.
Add this signature to your EMail:
Running Microsoft's "System Idle Process" will never help cure cancer, AIDS nor Alzheimer's. But running Rosetta@home just might!
https://boinc.bakerlab.org/rosetta/
ID: 25161 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : Number crunching : How to fake out the new credit system



©2024 University of Washington
https://www.bakerlab.org