MJH / Admins:

I'm getting several task errors (Windows 8.1 x64, 334.67 drivers) that say:

MJH:

Have you noticed an increase in instability, with 334.67 drivers, when suspending/resuming tasks, or shutting down and restarting BOINC quickly? If so, is there any way to determine if the application is the problem, or if the driver is the problem?

Confirm same Problems here with 332.21 Driver

589x-SANTI_MAR422cap310-12-32-RND9315_0
Arbeitspaket 5177762

Name 369x-SANTI_MAR422cap310-8-32-RND5608_0
Arbeitspaket 5175511

Simulation unstable. Flag 9 value 375
# The simulation has become unstable. Terminating to avoid lock-up

This happened again, where suspending the task, then closing BOINC, resulted in the task error'ing:
http://www.gpugrid.net/result.php?resultid=7810949

Can an admin please help to resolve this issue, or will it go unanswered?
I'm willing to offer whatever it takes to help test to get it resolved.

MJH?

Same issue here too...

Snap! GTX660, Win7-64, Driver 311.06

Anyone at GPUGrid care to fix this, like we did the previous suspend/resume problems? I'm willing to help test.

Have we any more complete idea of the cause yet? I've recently upgraded to the WHQL version of the driver (334.89) for my GTX 670: no crashes yet, but then I don't routinely suspend tasks once they've started. What I have noticed is the reduced CPU demand, and a welcome reduction in the runtime of the SIMAP tasks running at the same time.

I note that stderr says

I was able to get a task to fail by:

- Run BOINC such that the GPU Task is processing
- Right-click tray, choose "Snooze GPU"
- Verify task now says "GPU suspended"
- Right-click tray, "Exit", with "Stop running tasks" checked, click OK
- Start BOINC

They don't fail all the time, but... if you try those exact steps over and over, eventually you might get a failure.

I'd like this thread to focus on failures that are a result of those steps above. I hope we can solve it, but we'll need help from MJH.

They don't fail all the time, but... if you try those exact steps over and over, eventually you might get a failure.

I run applications that I have setup as "exclusive applications" in BOINC. And sometimes I shut down BOINC.

These, even in combination, should be supported, by the projects.
And I hope to have this issue resolved eventually. :)

Thank you.

I'm not sure how exactly to help, but I'm definitely willing. Last time, we iterated app versions with debug text to solve it, right? We might have to do something similar here.

I don't think this is a driver issue. I'd been error free for a long time but in the last 5 days have been seeing errors in SANTI_MAR WUs only. Some of them occur whenever BOINC is exited (gracefully, by exit dialog) for any reason. No other WU types are affected. At first I though the exit error was only on 1GB cards but now I see on other users that it's happening on 660 Ti cards also. The SANTI_MAR WUs also seem to be particularly sensitive to other conditions too and are failing at too high a rate IMO.

I have almost every other day an error of a Santi WU on my 660. On the 770 and 780Ti no errors (yet). I agree with Beyond (nice new picture of dog) that it is not the drivers. Santi's seem to be "special".
Coincidentally I found a crunchers tasks list with a Titan and all Santi's failed on that system, but the recent Noeilia's finished okay.
____________
Greetings from TJ

Matt:

It has been a while -- Have you made any progress on this?

I'm still regularly failing tasks during suspend and resume operations, especially SANTI_MAR tasks. It's especially painful to see 2 tasks fail simultaneously, which happens to me, because I have 2 GPUs dedicated to GPUGrid computing. Then when the tasks fail, instantly 10-20 hours of work, dead, to "Computation Error". Frustrating.

We need a fix! Please help!



Posted: 25 Feb 2014 | 23:29:42 UTC

I've scheduled some time to sort this out in a week or so, when I'll also be putting out Maxwell support.

Matt

I just had another one fail. I had 19 hours invested into it, and needed to restart my machine. I had suspended the task, I had closed BOINC, I restarted the machine, I resumed the task, and poof, Computation Error.

19 hours, wasted. This is very very frustrating.

And another one today.

The file exists.
(0x50) - exit code 80 (0x50)

MJH?

I just had another one fail. I had 19 hours invested into it, and needed to restart my machine. I had suspended the task, I had closed BOINC, I restarted the machine, I resumed the task, and poof, Computation Error.

19 hours, wasted. This is very very frustrating.

MJH: Please please please help.

I just threw away another several hours of GPUGrid work, because I had to restart BOINC, and the 2 GPUGrid tasks died. :( This time, I didn't suspend the tasks, I just exited BOINC normally. Then, upon restart, both tasks died. Surely this is fixable?!?!?

What was the problem, and what was the fix? When do you think it will land on the Long queue?

I will try to monitor application version numbers more closely, as I usually get a variety of Long/Short tasks.

When do you plan on deploying the 8.20 app, to Long-queue? People are still getting the "File already exists" error, losing tons of work, daily. If you were still testing it, then why was it not contained to the Beta-queue? Since it's already on Short, I think it should already be on Long too.

Sick of losing work because of this . . .

Jacob,

820 for cuda6 is on long now.

Matt

Have you been able to find a way of preventing the server from allocating cuda55 or cuda42 to Maxwell (CC 5.0) cards yet?

No idea, although haven't looked deeply into it yet.

Matt

Jacob,

820 for cuda6 is on long now.

Matt

Had to chime in again to say THANK YOU for fixing this. BOINC Task Stability is obviously very important to me, and this bug had been plaguing me for weeks. The new 8.20 app seems to be suspending/exiting/resuming much better for me thus far.

Thank you!

This has not been fixed. I have all CUDA 55 WU and if the light goes out, the work units get lost.

It looks like I've started getting some errors on my machine as well over the last few days. It's not running overly hot, not sure what's going on. This is the output from the last one:

Stderr output
<core_client_version>7.2.33</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code -97 (0xffffff9f)
</message>
<stderr_txt>
# GPU [GeForce GTX 760] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 0 :
# Name : GeForce GTX 760
# ECC : Disabled
# Global mem : 2048MB
# Capability : 3.0
# PCI ID : 0000:01:00.0
# Device clock : 1084MHz
# Memory clock : 3404MHz
# Memory width : 256bit
# Driver version : r334_00 : 33489
# GPU 0 : 44C
# GPU 0 : 45C
# GPU 0 : 47C
# GPU 0 : 48C
# GPU 0 : 49C
# GPU 0 : 50C
# GPU 0 : 51C
# GPU 0 : 52C
# GPU 0 : 53C
# GPU 0 : 54C
# GPU 0 : 55C
# GPU 0 : 56C
# GPU 0 : 57C
# The simulation has become unstable. Terminating to avoid lock-up (1)
# Attempting restart (step 76000)
# GPU [GeForce GTX 760] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 0 :
# Name : GeForce GTX 760
# ECC : Disabled
# Global mem : 2048MB
# Capability : 3.0
# PCI ID : 0000:01:00.0
# Device clock : 1084MHz
# Memory clock : 3404MHz
# Memory width : 256bit
# Driver version : r334_00 : 33489
# GPU 0 : 56C
# GPU 0 : 57C
# The simulation has become unstable. Terminating to avoid lock-up (1)
# Attempting restart (step 174000)
# GPU [GeForce GTX 760] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 0 :
# Name : GeForce GTX 760
# ECC : Disabled
# Global mem : 2048MB
# Capability : 3.0
# PCI ID : 0000:01:00.0
# Device clock : 1084MHz
# Memory clock : 3404MHz
# Memory width : 256bit
# Driver version : r334_00 : 33489
# GPU 0 : 56C
# GPU 0 : 57C
# The simulation has become unstable. Terminating to avoid lock-up (1)
# Attempting restart (step 175000)
# GPU [GeForce GTX 760] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 0 :
# Name : GeForce GTX 760
# ECC : Disabled
# Global mem : 2048MB
# Capability : 3.0
# PCI ID : 0000:01:00.0
# Device clock : 1084MHz
# Memory clock : 3404MHz
# Memory width : 256bit
# Driver version : r334_00 : 33489
# The simulation has become unstable. Terminating to avoid lock-up (1)

</stderr_txt>
]]>

It looks like I've started getting some errors on my machine as well over the last few days. It's not running overly hot, not sure what's going on.

I have the same issue on two different GPUs with different drivers.

On GTX 275:

Perhaps a little help. Yesterday I needed to boot all my systems for the necessary Windows updates after running for 26 days.
First thing I do is set to accept no new work so the queue can empty. Eventually I needed to go to bed but still WU's running. I suspended all work in BOINC manager and did then a cold boot (install updates and then power off system). After starting the PC's I went to the BOINC manager again and resumed work. All worked fine without error.

I know this is not the option Jacob, the original poster wants, but at least in my case it did not result in loss of work.

Edit: I need to mention I am still using 331.82 graphics driver
____________
Greetings from TJ

Thank you!

Thank you too, for your help in diagnosing it.
On to the next problem!

Matt

I downclocked my card slightly (~50MHz), or more precisely reduced the overclock, and haven't gotten any more errors since. Not sure if that's causal or coincidental since I haven't bumped it back up yet to test.

Variable: Your issue(s) are different than the one posted in this thread (see post 1). If you continue to have problems, please create a new thread.

Thanks,
Jacob

And... another 8.15 task crashed just now, losing tons of work. Why are we still using 8.15?!?

Thank you!

Thank you too, for your help in diagnosing it.
On to the next problem!

Matt

I thought this problem was fixed -- why are we still receiving 8.15 tasks? I just had 2 more fail, losing several hours of work, presumably because they were 8.15 instead of 8.20. Upsetting.

Power went out yesterday, I lost work units. Power went out today, I lost work units. This needs to get fixed!!!!!

MJH:

Although the 8.41 app appears to have improved the situation, I am still occasionally getting what appears to be the same error. I think the scenario is suspending activity, then restarting BOINC. Can you see if there's some scenario/condition that still causes the task to fail?

Error summary:
Exit status 80 (0x50) Unknown error number
The file exists.
(0x50) - exit code 80 (0x50)

Last messaged logged in stderr.txt:
# BOINC suspending at user request (exit)

Task results and stderr.txt:

http://www.gpugrid.net/result.php?resultid=9339200

Name I188-NATHAN_RPS1_adapt4-1-5-RND2310_0
Workunit 6566597
Created 25 Apr 2014 | 22:04:16 UTC
Sent 26 Apr 2014 | 11:18:41 UTC
Received 27 Apr 2014 | 4:06:56 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 80 (0x50) Unknown error number
Computer ID 153764
Report deadline 1 May 2014 | 11:18:41 UTC
Run time 38,039.02
CPU time 6,213.84
Validate state Invalid
Credit 0.00
Application version Long runs (8-12 hours on fastest card) v8.41 (cuda60)
Stderr output

<core_client_version>7.3.15</core_client_version>
<![CDATA[
<message>
The file exists.
(0x50) - exit code 80 (0x50)
</message>
<stderr_txt>
# GPU [GeForce GTX 660 Ti] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 0 :
# Name : GeForce GTX 660 Ti
# ECC : Disabled
# Global mem : 3072MB
# Capability : 3.0
# PCI ID : 0000:09:00.0
# Device clock : 1124MHz
# Memory clock : 3004MHz
# Memory width : 192bit
# Driver version : DM337_50 : 33761
# GPU 0 : 68C
# GPU 1 : 69C
# GPU 2 : 77C
# GPU 0 : 69C
# GPU 1 : 70C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 70C
# GPU 1 : 69C
# GPU 2 : 78C
# GPU 1 : 70C
# GPU 1 : 71C
# GPU 2 : 79C
# GPU 0 : 71C
# GPU 1 : 72C
# GPU 0 : 72C
# GPU 0 : 73C
# GPU 2 : 80C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 70C
# GPU 1 : 65C
# GPU 2 : 74C
# GPU 0 : 71C
# GPU 1 : 68C
# GPU 2 : 75C
# GPU 0 : 72C
# GPU 1 : 70C
# GPU 1 : 72C
# GPU 2 : 76C
# GPU 2 : 77C
# GPU 1 : 73C
# GPU 2 : 79C
# GPU 1 : 74C
# GPU 1 : 75C
# GPU 1 : 76C
# GPU 1 : 77C
# GPU 1 : 78C
# GPU 1 : 79C
# GPU 1 : 80C
# GPU 0 : 74C
# GPU 0 : 75C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 60C
# GPU 1 : 58C
# GPU 2 : 55C
# GPU 0 : 65C
# GPU 1 : 63C
# GPU 2 : 70C
# GPU 0 : 68C
# GPU 1 : 67C
# GPU 2 : 74C
# GPU 0 : 70C
# GPU 1 : 69C
# GPU 2 : 75C
# GPU 0 : 71C
# GPU 1 : 70C
# GPU 2 : 76C
# GPU 0 : 72C
# GPU 1 : 73C
# GPU 0 : 73C
# GPU 2 : 77C
# GPU 2 : 78C
# GPU 2 : 79C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 59C
# GPU 1 : 58C
# GPU 2 : 55C
# GPU 0 : 61C
# GPU 1 : 63C
# GPU 0 : 63C
# GPU 1 : 67C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 61C
# GPU 1 : 64C
# GPU 2 : 55C
# GPU 0 : 63C
# GPU 1 : 68C
# GPU 0 : 64C
# GPU 1 : 70C
# GPU 0 : 65C
# GPU 1 : 71C
# GPU 2 : 56C
# GPU 0 : 66C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 63C
# GPU 1 : 66C
# GPU 2 : 62C
# GPU 0 : 65C
# GPU 1 : 70C
# GPU 0 : 66C
# GPU 1 : 74C
# GPU 0 : 67C
# GPU 1 : 78C
# GPU 0 : 69C
# GPU 0 : 71C
# GPU 0 : 73C
# GPU 2 : 71C
# GPU 2 : 73C
# GPU 2 : 74C
# GPU 2 : 75C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 61C
# GPU 1 : 63C
# GPU 2 : 57C
# GPU 0 : 63C
# GPU 1 : 67C
# GPU 0 : 64C
# GPU 1 : 70C
# GPU 0 : 66C
# GPU 1 : 71C
# GPU 1 : 73C
# GPU 0 : 67C
# GPU 1 : 74C
# GPU 1 : 75C
# GPU 0 : 71C
# GPU 0 : 72C
# GPU 2 : 69C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 59C
# GPU 1 : 59C
# GPU 2 : 56C
# GPU 0 : 65C
# GPU 1 : 64C
# GPU 2 : 64C
# GPU 0 : 68C
# GPU 1 : 67C
# GPU 2 : 69C
# GPU 0 : 69C
# GPU 1 : 69C
# GPU 2 : 71C
# GPU 0 : 71C
# GPU 2 : 72C
# GPU 0 : 73C
# GPU 1 : 72C
# GPU 2 : 73C
# GPU 1 : 73C
# GPU 2 : 74C
# GPU 2 : 75C
# GPU 2 : 76C
# BOINC suspending at user request (exit)
# GPU [GeForce GTX 460] Platform [Windows] Rev [3301M] VERSION [60]
# SWAN Device 1 :
# Name : GeForce GTX 460
# ECC : Disabled
# Global mem : 1024MB
# Capability : 2.1
# PCI ID : 0000:08:00.0
# Device clock : 1526MHz
# Memory clock : 1900MHz
# Memory width : 256bit
# Driver version : DM337_50 : 33761
# GPU 0 : 62C
# GPU 1 : 65C
# GPU 2 : 57C
# GPU 0 : 63C
# GPU 1 : 68C
# BOINC suspending at user request (exit)

</stderr_txt>
]]>

I have highlighted the problem in counting the cards gtx 680 a month now happens to me from . Every day becomes that the tasks of collapse in such a weird way-slow down your PC system in windows and also according to GPU-Z stops the card count. entire system is as if in slow motion ... only helps suspend computation on graphics card, abortions every task and the new has withdrawn. ., and after about cca 6-12 aborted about the tasks shall start another 3 working normally .. it's weird errors and concerns only nvidia cards 600, to 700 card counting goes perfectly.
I play with the problem for months.... and computing of other projects without problems.
It's not boiling cards or a weak PSU.. I'm not able to count on 680 of these normally GPUGRID, consider selling them or any other project..

MJH:

The v8.41 version of the application still has the occasional "The file exists. (0x50) - exit code 80 (0x50)" error, trashing loads of work :( Can you please invest some time to fix it?

http://www.gpugrid.net/result.php?resultid=10318262

MJH:

The v8.41 version of the application still has the occasional "The file exists. (0x50) - exit code 80 (0x50)" error, trashing loads of work :( Can you please invest some time to fix it?

http://www.gpugrid.net/result.php?resultid=10318262

Every time the lights go out I lose all the units that are being worked on. If I restart the system using the proper procedures, no problem. This has been going on for months and I am really getting sick of it. Bought UPS, now lets see.

Is your error:

The file exists.
(0x50) - exit code 80 (0x50)

This is *STILL* an issue. When can we finally get it fully fixed? :(

http://www.gpugrid.net/result.php?resultid=12800989
Outcome Computation error
Client state Compute error
Exit status 80 (0x50) Unknown error number
Run time 32,087.62

I'm on the road, but will be home tonight. I'll try to re-review, probably tomorrow. Thanks!

Jacob,

You are in luck. It's time for another round of GPUGRID development. Remind me, please, the circumstance under which this is occuring.

Matt

Tricky - the GPU ordering from the temperature query interface doesn't correspond to the CUDA ordering.

MJH:

I've reviewed the notes in the thread. The main posts that detail the problem are:
http://www.gpugrid.net/forum_thread.php?id=3621&nowrap=true#35348
http://www.gpugrid.net/forum_thread.php?id=3621&nowrap=true#37242

It is not easy to reproduce on demand. I suspect that your best bet is to investigate/walk the code, to find an area that could result in:
<message>
The file exists.
(0x50) - exit code 80 (0x50)
</message>

It seems to happen more frequently when the task is suspended before BOINC is shutdown, but suspending the task might not be a requirement of the bug.

Testing should involve suspending BOINC, and then shutting BOINC down, and then starting BOINC back up. Also, to test the "power outage" scenario, I think testing could involve right clicking boincmgr.exe in Task Manager, and clicking "End process tree".

I hope this helps. The focus should be on code areas that could result in that error message.

Regards,
Jacob

That exit circumstance is the failsafe exit that stops a WU getting stuck in an endless cycle of abort - resume, without making any progress. It should only trigger if the machine has been up for a few minutes (from which we infer that the WU crashed the machine).

Matt

That exit circumstance is the failsafe exit that stops a WU getting stuck in an endless cycle of abort - resume, without making any progress. It should only trigger if the machine has been up for a few minutes (from which we infer that the WU crashed the machine).

Matt

Perhaps you could give me even more clues on how to reproduce the error on demand? It seems that it is currently too stringent, causing otherwise-healthy tasks to fail when starting BOINC.

He said:

It should only trigger if the machine has been up for a few minutes

Matt,

Could you please give me more details about that exit algorithm? Maybe even pseudocode or something, please? Details, like "If it restarts x times without saving a checkpoint" or "If it restarts x times during a computer-uptime-session" or "If it restarts x times during the course of the task", etc.

... Just so I can easily reproduce the issue on demand, and thus help you test/solve it.

Matt,

Could you please give me more details about that exit algorithm? Maybe even pseudocode or something, please? Details, like "If it restarts x times without saving a checkpoint" or "If it restarts x times during a computer-uptime-session" or "If it restarts x times during the course of the task", etc.

... Just so I can easily reproduce the issue on demand, and thus help you test/solve it.

Jacob,

When the simulation starts computing, ACEMD puts a file called "canary" in the slot directory, which it then removes the first time it writes a restart file set.

When ACEMD is starting up it looks for the "canary" file - if it finds it that means the simulation aborted for some reason very soon after it started before making significant progress. In this case, if the system has been booted for less than 10 minutes we interpret this as meaning that the last instance of ACEMD crashed the machine and so abort the WU as bad.

Matt

Alright.... So, it looks like the slot directory does get the canary file when the tasks are started within the session. And, by utilizing the <checkpoint_debug> flag in cc_config.xml, I believe I see the file being removed whenever the task's first checkpoint of the session is performed.

So, I've tried closing BOINC (normally) about 2 seconds after startup, which leaves the canary files in my slot directories. But, upon starting BOINC, with those files in the directories, it does not fail the tasks.

How can I get these tasks to easily fail on-demand? Is there more to the logic that decides when to fail them?

EDIT: I just re-read your post... I see "if the system has been booted for less than 10 minutes".... hmm... Let me restart Windows, and perform the same test.

Hurray! I've been able to make all 3 of my tasks fail, essentially on-demand! All of them with error: "The file exists. (0x50) - exit code 80 (0x50)" ... This genuinely excites me!

Here's what I did:
- restarted my computer
- monitored Task Manager's Performance tab on the CPU selection, to make sure "Up time" was less than 10 minutes
- started BOINC
- saw the canary files
- exited BOINC
- confirmed the canary files were still present
- started BOINC again
- ...and watched the tasks fail.

Good thing I didn't mind failing them :)

Next thing I'll do (later today if I find time) will be to test whether it is "must see canary on task start within 10 minutes of up-time" or "must see canary on task start within 10 minutes of logged-in time"

Either way, though... This algorithm doesn't jive well. Are you able to make changes to it? Perhaps we could work together to develop a better algorithm that hopefully still accomplishes your goals, without killing tasks?

Let me know,
Thanks,
Jacob

Either way, though... This algorithm doesn't jive well. Are you able to make changes to it? Perhaps we could work together to develop a better algorithm that hopefully still accomplishes your goals, without killing tasks?

Jacob,

Can you explain exactly the circumstances under which you are getting a false activation of the trap? It sounds to me something like:

* You've stopped BOINC because you want the machine for something else. Some of the WUs have only just started running, and haven't reached their first checkpoint, so leave canary files.
* You turn off the machine
* Later,you turn it back on again and the WUs that had barely started are incorrectly assumed to have been defective and aborted.

Is this really a such common occurrence? The window of vulnerability for a WU is pretty narrow - the interval between starting and first checkpoint should only be a few minutes.

Anyway, you've hit on a reasonable improvement - to remove the canary if the tasks are responding to a suspend request from the client.

Matt

Matt,

I do all sorts of crazy fun stuff with my computer. Sometimes, I suspend BOINC, because I need the CPUs for something else. A lot of times, I actually close BOINC, because I want the CPUs and the memory, for my main game, iRacing. :) But I think the culprit scenario is likely a bit different. Here goes.

The "triggering" scenario goes something like this:
- I'm doing something that requires a restart. Maybe I'm installing new software. Go with that as the assumption. Let's say Windows Update required a restart, and I clicked OK to restart Windows. Canary files are not present, because tasks checkpointed before I clicked OK.
- I restart, log in, and immediately pause or exit BOINC (bolded for emphasis as the condition that doesn't jive well with the current canary implementation), because I want resources available. Maybe I realized I have to update additional software, that I know will require a restart, and I want to make this installation go quicker. Or maybe I HAVE A RACE RIGHT NOW (and so, close BOINC, to give me resources for iRacing). So, BOINC gets closed. Canary files are present, because tasks started before I closed. Right?
- So, later, I start BOINC. And then cry. Because all my GPUGrid work is lost. I have 3 GPUs, and all 3 tasks (which could have been up to 30 hours of work) are lost. I weep the tears of a thousand kernels, swept away in an erroneous exit condition. :)

Personally, I think the exit condition might not be needed at all. Have you seen a reason to require it? I assume you want to keep it.

If the tasks are responding to a suspend request from the client (ie: BOINC is closed normally, right? That's what you meant, right?), then... Yes, removing the canary file should solve the problem for my scenario above. It won't solve all the problems (as, I could kill BOINC in Task Manager, and then canary files would still be present, and also I think upgrading BOINC causes the tasks to be killed ungracefully), but it should solve the normal scenarios (normal shutdowns).

Can you implement it? I'd love to test it.

I've noticed that GPUGrid tasks fail with the "file exists" error when I'm restarting my PC immediately after a restart. I thought that I should wait for the workunits made their first checkpoint to avoid this error, but I didn't thought that it's a protective algorithm.
Two (or more) fast system restart is needed (for me) when the USB controllers on my motherboard became unusable in Windows XP after a Windows 7 session on that PC, and I have to physically switch off the power from the PC to fix it. Fast system restart(s) is also needed when updating different drivers / software in succession, or when fixing other hardware related problems (for example: I have a PCIe ethernet controller card in this motherboard. At some point the ethernet card has disappeared from device manager, so there was no network connectivity on this PC which is crucial. I had to restart the PC several times, and make changes in the BIOS to fix it)
So this problem can be solved by making this protective algorithm complete: it should delete the canary file during a graceful shutdown.
EDIT: an additional safety algorithm could be this: the workunit should abort itself when it's progressing very slowly (for example: if it couldn't finish in 5 days)

Any progress?

Matt,

Has there been any progress on improving the canary-file-detection? I almost got bit by it again, when I installed a round of Windows updates, logged into Windows (which launches BOINC), immediately exited BOINC, so I could install round 2 of updates.

Good thing I remembered about the canary issue, and remember to wait until it deleted the files to close BOINC. But, closing BOINC normally should have deleted the canary files.

Please fix this.

Jacob,

It's on the todo list. It'll get done early September, after vacaciones.

Matt[/u]

Sure hope this gets fixed. Updating my machines from 7.4.8 to 7.4.18, carefully shutting down 7.4.8 before installing the new client yielded 3 aborted GPUGrid WUs out of 7. This happens only with GPUGrid WUs, no other projects that I run (many) behave in this way.

Jacob,

It's on the todo list. It'll get done early September, after vacaciones.

Matt[/u]

Thank you. Are there minimum requirements for getting tasks on that beta app?