How to prevent tasks from getting stuck in the stash

Message boards : Number crunching : How to prevent tasks from getting stuck in the stash

To post messages, you must log in.

AuthorMessage
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 99662 - Posted: 13 Nov 2020, 1:18:07 UTC

I've noticed that my BOINC installation, for some reason, seems to prioritize downloading new tasks and running those instead of finishing the tasks in the stash. This happens even when the newly downloaded tasks are further away from the deadline and take a lot less time to complete. This may result in the old tasks not being run or started until it's too late. I've set my BOINC manager to store at least 0.1 days of tasks and store up to an additional 0 days of work.

Is this normal? And if it is, is there are a way to force BOINC to prioritise tasks that have either been in storage for some time or are closer to their deadlines without user supervision?
ID: 99662 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1684
Credit: 17,959,099
RAC: 23,317
Message 99667 - Posted: 13 Nov 2020, 10:12:56 UTC - in response to Message 99662.  
Last modified: 13 Nov 2020, 10:13:39 UTC

The default for BOINC is first in first out, however- the size of your cache & deadlines for Tasks and your Resource share setting, and the number of available cores/threads on the system (and to a lesser extent system up time/time BOINC is able to process work) will result in non-first in first out Task processing as the Manager tries to process work in accordance with your settings- which may all contradict each other (such as running lots of projects on a system with limited cores/threads and huge differences in Task deadlines (such as days v weeks v months with a large cache).
The less cores & threads you have, and the more projects you run, then the smaller your cache needs to be in order to avoid issues due to differing deadlines & Task runtimes between the different projects. Ideally, no cache is best; especially so if you are running more than a couple of projects and none of them are set as a backup project.

So setting your cache smaller should (eventually) allow things to run as first in first out again (mostly- depending on your Resource share settings & the deadlines of Tasks there may still be occasions when a recently downloaded Task (say with a 3 day deadline) will start up before an older one (say a 1 month deadline) in order to meet your Resource share settings & avoid missing the deadline for the shorter deadline Task).
Grant
Darwin NT
ID: 99667 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 99670 - Posted: 13 Nov 2020, 16:45:56 UTC - in response to Message 99667.  
Last modified: 13 Nov 2020, 16:50:58 UTC

That is not what I'm witnessing, for some reason. In my case, Rosetta@home has a resource share of 200 instead of the default. I have a couple of Rosetta@home tasks (those were the only tasks in my cache, mind you) with a deadline 3 days away waiting to be started. Instead of running these tasks after completing a couple of other ones, BOINC downloaded new tasks that had a deadline a week away and ran those instead.

Rosetta@home has a higher resource share, so those tasks should be prioritized.
The Rosetta@home tasks came first, so they should've been run first.
The Rosetta@home tasks had a shorter deadline and had a longer estimated runtime, which should also cause them to be prioritized.
My cache size is set to 0.1 days + 0 days, which should mean there shouldn't be many tasks being cached.

Despite all that, BOINC decided to continually download new tasks from a different project and ignore the Rosetta tasks in my cache.
ID: 99670 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 393
Credit: 12,114,842
RAC: 4,200
Message 99671 - Posted: 13 Nov 2020, 16:55:56 UTC - in response to Message 99670.  

That is not what I'm witnessing, for some reason. In my case, Rosetta@home has a resource share of 200 instead of the default. I have a couple of Rosetta@home tasks (those were the only tasks in my cache, mind you) with a deadline 3 days away waiting to be started. Instead of running these tasks after completing a couple of other ones, BOINC downloaded new tasks that had a deadline a week away and ran those instead.

Rosetta@home has a higher resource share, so those tasks should be prioritized.
The Rosetta@home tasks came first, so they should've been run first.
The Rosetta@home tasks had a shorter deadline and had a longer estimated runtime, which should also cause them to be prioritized.
My cache size is set to 0.1 days + 0 days, which should mean there shouldn't be many tasks being cached.

Despite all that, BOINC decided to continually download new tasks from a different project and ignore the Rosetta tasks in my cache.


Is it prioritising a newly attached project? If so it is Boinc trying to balance the resource share of Rosetta with a rac of nnn against the new project with a rac of zero.
ID: 99671 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1684
Credit: 17,959,099
RAC: 23,317
Message 99672 - Posted: 13 Nov 2020, 21:59:21 UTC - in response to Message 99670.  
Last modified: 13 Nov 2020, 22:14:16 UTC

Despite all that, BOINC decided to continually download new tasks from a different project and ignore the Rosetta tasks in my cache.
BOINC version you are running is 7.16.11. I remember there was an issue with work fetch & scheduling under certain conditions with some older versions, but not sure which one(s). Maybe they've reintroduced in to the latest version?


Edit-

More likely- changing web based preferences it takes a while for them to propagate between all projects. If you change them on one project, but the Manager is repeatedly contacting another project for work- it won't pick up those changes till they get propagated to that project, and that doesn't happen until it next contacts the project that had the preferences there changed. Then the next time it contacts the project that didn't have the preferences changed, the new preferences will be sent there.

Short version- after changing the web preferences at a particular project, click on Update on the BOINC Manager on your computer for those changes to take effect immediately, and start them being sent to any other projects you are connected to (when the Manager next contacts them).


As Bryan mentioned, have you added a new project, or was the project quite for a while & recently come back to life, and the Manager is now trying to meet your Resource share settings?
Have you missed any deadlines?
Grant
Darwin NT
ID: 99672 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2125
Credit: 41,249,734
RAC: 9,368
Message 99682 - Posted: 14 Nov 2020, 17:48:02 UTC - in response to Message 99672.  

Despite all that, BOINC decided to continually download new tasks from a different project and ignore the Rosetta tasks in my cache.
BOINC version you are running is 7.16.11. I remember there was an issue with work fetch & scheduling under certain conditions with some older versions, but not sure which one(s). Maybe they've reintroduced in to the latest version?

You got me worried for a minute - I'm running 7.16.11 too
I just checked - 7.16.11 is the most recent Boinc release version for any platform
<phew>
ID: 99682 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 99683 - Posted: 15 Nov 2020, 1:30:14 UTC - in response to Message 99662.  

I've noticed that my BOINC installation, for some reason, seems to prioritize downloading new tasks and running those instead of finishing the tasks in the stash.

The recent versions of BOINC periodically lose it for some unknown reason and download too much work. I have seen several days download, even though I keep the default 0.1 + 0.5 days buffer.
I first noticed it over a year ago on WCG, and posted on it then, without any useful response.

It just bit a few of us on LHC also. It knows no boundaries, though I have not seen it recently on Rosetta.
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5534&postid=43594#43594
ID: 99683 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1684
Credit: 17,959,099
RAC: 23,317
Message 99684 - Posted: 15 Nov 2020, 2:47:15 UTC - in response to Message 99683.  

I keep the default 0.1 + 0.5 days buffer.
Having it as 0.5 + 0.1 will tend to result in a more consistent cache.
Grant
Darwin NT
ID: 99684 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 99688 - Posted: 15 Nov 2020, 9:31:16 UTC - in response to Message 99684.  

Having it as 0.5 + 0.1 will tend to result in a more consistent cache.

It will keep it more constant, since you allow only 0.1 days additional. But I would like to minimize the load on the servers too.
I think that is one of the reasons they set the defaults as they do.

That has nothing to do with the excess downloads however, which ignore the settings anyway.
ID: 99688 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : How to prevent tasks from getting stuck in the stash



©2024 University of Washington
https://www.bakerlab.org