@ Dave Baker (Deadlines causes EDF)
Message boards : Number crunching : @ Dave Baker (Deadlines causes EDF)
| Author | Message |
|---|---|
 [B@H] Ray Send message Joined: 20 Sep 05 Posts: 118 Credit: 100,251 RAC: 0 |
Dave Any chance of the deadlines going to 2 weeks like most other projects? This LINK goes to the Shorter WU deadline thread about how easily some users were put into EDF with the one week deadlines. Ray  Pizza@Home Rays Place Rays place Forums |
|
David Baker Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 17 Sep 05 Posts: 705 Credit: 559,847 RAC: 0 |
Dave I guess we could split our work units up into two classes--the very large scale runs where it will take a couple of weeks to collect enough data, and the short ones we want to draw conclusions from within a day or two. the former could have a two week deadline, the latter, a one week deadline. would this help? |
|
Divide Overflow Send message Joined: 17 Sep 05 Posts: 82 Credit: 921,382 RAC: 0 |
Dave Why? You seem to be overly concerned with the behavior of your short term queue. You might crunch some Rosetta for a while on EDF, but eventually the long term debt for the other projects you have active will prevent you from getting more Rosetta work and your focus will be on your other projects. The scheduler was designed to work with these variables to keep you as close as possible to your resource allocation over the long term. Each project should keep the settings that best meet their own demands. BOINC should be flexible enough to roll with the punches. |
Scribe Send message Joined: 2 Nov 05 Posts: 284 Credit: 157,359 RAC: 0 |
Yeah, I seem to remember that when certain others 'seemed' to suggest some changes to suit 'their' perceived requirements, the wrath of god seemed to descend upon them and they were reviled for daring to suggest it and basically told to shut up. |
Fuzzy Hollynoodles Send message Joined: 7 Oct 05 Posts: 234 Credit: 15,020 RAC: 0 |
David, it's your project, and you decide which deadlines, you need. There're no reasons for changing anything just to satify people's wish for micromanaging their BOINC manager, as it will adjust itself after one's preferences and cache. My own with a cache on 0.1 works fine and has all the time. For me the deadlines could be less than a week, as I return within a day or two anyway. But if you need some data to draw conclusions from imidiately, just set the deadline to what you need and want, and send the WU's to us with the shortest returntime. I crunch Seti Beta Astropulse WU's at the moment, where the deadline is May 2007, which takes about 31 hours on my computer to complete, and I return them after about 2 weeks, so the deadline next year have no influence on my short term debt what so ever in the sense, that the Beta WU's are being "starved" at the expense of other WU's. So decide which deadlines you need, and then set them for the relevant WU's. [b]"I'm trying to maintain a shred of dignity in this world." - Me[/b] 
|
|
Jeff Gilchrist Send message Joined: 7 Oct 05 Posts: 33 Credit: 2,398,990 RAC: 0 |
TDF isn't the only problem. BOINC will let uses queue work for 10 days, some of us have machines not connected to the net very often. This is causing me some problems. I have some machines that are not frequently connected to the Internet so I have my cache set to queue 10 days of work and I sometimes can only connect once a week to upload. I am now not able to fill up my cache because I cannot complete the work in time for the deadline. Could you reconsider this and possibly select 2 weeks instead of 1 so that you still get your work back reasonably fast but you dont penalize some people? |
|
NJMHoffmann Send message Joined: 17 Dec 05 Posts: 45 Credit: 45,891 RAC: 0 |
I have some machines that are not frequently connected to the Internet so I have my cache set to queue 10 days of work and I sometimes can only connect once a week to upload. I would let such machines work only on projects with deadlines of 14 days or more. Set the ressource share for Rosetta higer on the other computers. If the Rosetta team needs the data back earlier, than seldom connected computers can't help. Has nothing to do with "penalize". Norbert |
|
nasher Send message Joined: 5 Nov 05 Posts: 98 Credit: 625,341 RAC: 647 |
yes it is NICE to have projects that have at least a 2 week interval. mind you that i also have adjusted my systems (at least 2 of them) for running Pirates (if ya know anything about that one .. its about 1 day turnaround max...) personaly i understand that some times it will be nessary for the project to shorten there work times and i will be crunching even if you put your deadlines of less than 4 days ... but i think you might loose some people if you do (just a sugestion). all in all its up to the project when they NEED the results back and alot of times we dont understand whats up. So here is my sugestion. If you need to have some jobs completed quicker please make a post or put it in your news that the current target deadlines are shortened.. once we complete the yyy target we shall return to 2 week deadlines or such.  
|
|
Jeff Gilchrist Send message Joined: 7 Oct 05 Posts: 33 Credit: 2,398,990 RAC: 0 |
I would let such machines work only on projects with deadlines of 14 days or more. Set the ressource share for Rosetta higer on the other computers. I only wish to donate my CPU time to Rosetta, I do not run any other BOINC projects and thus cannot modify my resource share. I'm not saying that all WUs need a 2 week deadline, but David Baker suggesting having a queue with some work that can be returned in 2+ weeks would be very helpful for those of us with slower machines or non-permanent Internet connections. I just wanted to point out that there are some people out there that can't handle very short deadlines. |
|
Honza Send message Joined: 18 Sep 05 Posts: 48 Credit: 173,517 RAC: 0 |
I have suggested this several times: why don't use smarter scheduler unlike primitive FIFO that runs since BOINC started? Each host has several usefull characteristics. Among others like % BOINC is running is "Average turnaround time". Send WUs were you need results soon to those machine with (i) fast CPU, (ii) high % of BOINC running, (iii) low Average turnaround time and you get results the same day. Data of all host characterictic is all there - why don't use it? Improved scheduler on server side - FIFO was good in 60's...we should do better know. |
Keck_Komputers Send message Joined: 17 Sep 05 Posts: 211 Credit: 4,246,150 RAC: 0 |
The BOINC client uses earliest deadline first not FIFO. It may appear FIFO when all the tasks for a given project have the same initial deadline. BOINC WIKI   BOINCing since 2002/12/8 |
|
TPR_Mojo Send message Joined: 20 Sep 05 Posts: 4 Credit: 684,947 RAC: 0 |
The deadlines should be set to suit the project scientists only. Rosetta are not doing this for us, we are providing resources for them. If the workload is unsuitable for a machine's circumstances, then that machine is unsuited to the project. I'm sorry if that sounds callous or harsh but it is unfortunately the truth. |
Ziran Send message Joined: 3 Jan 06 Posts: 2 Credit: 15,082 RAC: 0 |
Why do i get the feeling that i am reading a one year old thread over at LHC? Every project has its specific requirements. The easier those requirements are to live up to, the more potential participants there are. More participants = more work done = more science (hopefully). It is seldom in the interest of a project to make it harder for participants then necessary. There are always tradeoffs to balance conflicting interests. If the science permits itself to be divided in to two groups, one with a 1-week deadline and one with a 2-week deadline, it is beneficial. Why? Because it makes it possible for the project to use its limited resources in a more efficient way.
This would probably help out many participants that fore one reason or another prefers longer deadlines. Under the condition that the scheduler can determine, who prefer longer deadlines. |
|
Jeff Gilchrist Send message Joined: 7 Oct 05 Posts: 33 Credit: 2,398,990 RAC: 0 |
The deadlines should be set to suit the project scientists only. Rosetta are not doing this for us, we are providing resources for them. If the workload is unsuitable for a machine's circumstances, then that machine is unsuited to the project. I'm sorry if that sounds callous or harsh but it is unfortunately the truth. Yes, and if the scientists want resources from people, it is in their best interest to accomodate the users as best they can. If there is an easy solution for Rosetta to allow some longer deadline workunits then that will allow more people to participate and stop others from leaving. If there is no solution then so be it, but it's worth at least looking into. You will notice that nobody is "demanding" the WUs have longer deadlines, people are politely asking if it is possible. |
|
Honza Send message Joined: 18 Sep 05 Posts: 48 Credit: 173,517 RAC: 0 |
The BOINC client uses earliest deadline first not FIFO. It may appear FIFO when all the tasks for a given project have the same initial deadline.Well, I was talking about server scheduler, not BOINC client! [It is apparent that when WUs are downloaded, there is not much space to handle WUs other than deadline, EDF etc.] AFAIK, scheduler is a simple pool: you put some WUs in queue and when machine ask for more WUs, scheduler provides them - old ones first. It ignores machine characteristics that helps faster return of WUs - Average turnaround time on first place. Again, it should be more effective to provide WUs with shorter deadline to machines with low average turnaround time (more reliable), small WUs cache, high % uptime etc. If there machine characteristics were considered, 'important' WUs should have been returned within hours. Does this make sense? |
John McLeod VII Send message Joined: 17 Sep 05 Posts: 108 Credit: 195,137 RAC: 0 |
The BOINC client uses earliest deadline first not FIFO. It may appear FIFO when all the tasks for a given project have the same initial deadline.Well, I was talking about server scheduler, not BOINC client! [It is apparent that when WUs are downloaded, there is not much space to handle WUs other than deadline, EDF etc.] At one point I submitted some pseudo code to do something like this. It has gone noplace. (Currently, I do not have access to a linux box, so it is impossible to write and test the code).   BOINC WIKI |
|
Paul D. Buck Send message Joined: 17 Sep 05 Posts: 815 Credit: 1,812,737 RAC: 0 |
Again, it should be more effective to provide WUs with shorter deadline to machines with low average turnaround time (more reliable), small WUs cache, high % uptime etc. If there machine characteristics were considered, 'important' WUs should have been returned within hours. The design currently faces some limitations. Of of these is the complxity of the Feeder. At the moment the feeder is, by default compiled with 100 "slots". These slots are filled by a enumeration query which CAN pull data in various orders from the available work "pool". The scheduler currently, as you noted, just pulls one or more slot's contents for issue. Though some projects have compiled the feeder to have more slots there is a fundamental limitation based on memory segment size, for the shared memory segment, how many slots can be in the feeder. Other design constraints can further limit the actual number of available work units. The data in the Feeder is, right now, just a simple id number and a "tag" ... discovery of other information about a work unit/result requires a database access. One of the whole points to the feeder is to limit the database accesses as much as possible so that they do not restrict the speed of the schedulers. I won't go into my feeling about the feeder design efficiency as, like John, I cannot currently compile the BOINC back end systems and start tests. I have started to do that, but, as I seem to be the first to be interested in doing this with OS-X it has been slow going. Of course it also does not help that my "working day" is running at something less than an hour. Anyway Honza, your question makes sense, I just don't see an easy way to get to there from here. OF course, this probably means that I shall be accused of a negative attitude again ... |
FluffyChicken Send message Joined: 1 Nov 05 Posts: 1260 Credit: 369,635 RAC: 0 |
Paul/John, Put it request for help on the alteration/code/ideas up in the developer/boinc forum and/or list ? Team mauisun.org |
|
Honza Send message Joined: 18 Sep 05 Posts: 48 Credit: 173,517 RAC: 0 |
@ JM7 and Paul. I'm happy that some BOINC old-timers with in-depth insight into BOINC feel that idea a was trying to express is not a nonsense. How about simply sort there slots where 'priority' would be a key? Some slots may server as 'express line'. I see that scheduler was not designed as a smart one. May serve pretty well for simple SETI where all WUs are the same (size, type and priority/deadline) and where there are x100k WUs sent/received every day. I remember Carl (of CPDN) was trying to improve the scheduler so it at least consider about of memory of a host (and sent less demanding Slab or more demanding Sulphur). This should be usefull for life-science projects as well where WUs types are present. Same for BURP. Just another example to have 'smarter' scheduler... Perhaps BOINC 6.x ? |
|
Paul D. Buck Send message Joined: 17 Sep 05 Posts: 815 Credit: 1,812,737 RAC: 0 |
Honza, I was hoping to look at that, my trouble is that my daily work time is still running about an hour ... :( So, I am still trying to stand up a BOINC project on my local LAN so I can expiriment with questions like this. So far I have discovered several problems with configuration questions on OS-X as a server and compiler platform. I have to get the latest code and try out the fixes for the next step and to start to write down the progress. Once I can compile things I will take the 300 or so "cookbooks" and hopefully make some meaningful "How-To" Guides on the development side. But, there are two conflicting tensions I see in the design, one is the fundamental limitation on shared memory segment size and secondly the ability to abstract meaningful information for the schedulers to be "smarter". Besides size of memory, disk space, and CPU speed, you have the time questions etc. The documentation, not unexpectedly to me, is silent on many of the questions of configuration and size. And these in part are what I should like to address. But, again, I am fighting the problem of having to follow the developers and we do not pool the work, inevitable duplication wastes effort, and the things *I* add to the wiki in the development side do not get back into the official documentation, unfortunately. Anyway, dead horse ... sorry ... |
Message boards :
Number crunching :
@ Dave Baker (Deadlines causes EDF)
©2024 University of Washington
https://www.bakerlab.org