Advanced search

Message boards : News : Project restarted

Author Message
Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55495 - Posted: 9 Oct 2020 | 13:20:48 UTC
Last modified: 9 Oct 2020 | 13:24:31 UTC

Dears, thanks for your patience.

I updated the acemd3 apps.

Also, I verified that there were very few results by apps from the old CUDA version, so I won't be re-deploying them. In other words, apps now require CUDA 10 (Linux) and CUDA 10.1 (Windows). In terms of drivers versions:

CUDA 10.1 (10.1.105) >= 418.39 for Windows
CUDA 10.0 (10.0.130) >= 410.48 for Linux

https://docs.nvidia.com/deploy/cuda-compatibility/index.html

Ian&Steve C.
Avatar
Send message
Joined: 21 Feb 20
Posts: 117
Credit: 1,582,087,015
RAC: 9,893,052
Level
His
Scientific publications
wat
Message 55496 - Posted: 9 Oct 2020 | 13:37:47 UTC - in response to Message 55495.

do you have an ETA on when we can expect some CUDA 11.1 apps with Ampere support? I'm a little surprised you didnt recompile with CUDA 11.1 while you were already rebuilding the app for this most recent issue.
____________

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55498 - Posted: 9 Oct 2020 | 13:40:24 UTC - in response to Message 55496.

do you have an ETA on when we can expect some CUDA 11.1 apps with Ampere support? I'm a little surprised you didnt recompile with CUDA 11.1 while you were already rebuilding the app for this most recent issue.

Won't be long. Awaiting official releases from underlying libraries.

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55499 - Posted: 9 Oct 2020 | 13:41:18 UTC - in response to Message 55498.

Please post here if you manage to get WUs.

Richard Haselgrove
Send message
Joined: 11 Jul 09
Posts: 1030
Credit: 2,838,667,345
RAC: 3,857,128
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55500 - Posted: 9 Oct 2020 | 13:42:43 UTC
Last modified: 9 Oct 2020 | 13:46:56 UTC

First Windows task has downloaded and is running cleanly on GTX 1660 Super.

4lziA01_320_0-TONI_MDADex2sl-11-50-RND7994_1

Linux reports 'no tasks available'.

09/10/2020 14:44:40 | GPUGRID | No tasks are available for New version of ACEMD

Ian&Steve C.
Avatar
Send message
Joined: 21 Feb 20
Posts: 117
Credit: 1,582,087,015
RAC: 9,893,052
Level
His
Scientific publications
wat
Message 55501 - Posted: 9 Oct 2020 | 13:51:02 UTC

on Linux, my systems are only seeing "no tasks available" during the schedule request.
____________

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55502 - Posted: 9 Oct 2020 | 13:53:50 UTC - in response to Message 55501.

Oh no here we go again

curiously_indifferent
Send message
Joined: 20 Nov 17
Posts: 5
Credit: 584,460,232
RAC: 474,706
Level
Lys
Scientific publications
watwatwat
Message 55503 - Posted: 9 Oct 2020 | 13:54:45 UTC - in response to Message 55499.

My Win10 machine is being told 'No tasks available"

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55504 - Posted: 9 Oct 2020 | 13:56:15 UTC - in response to Message 55500.

First Windows task has downloaded and is running cleanly on GTX 1660 Super.

4lziA01_320_0-TONI_MDADex2sl-11-50-RND7994_1

Linux reports 'no tasks available'.

09/10/2020 14:44:40 | GPUGRID | No tasks are available for New version of ACEMD


This should provide an answer to those asking what's so difficult.

tullio
Send message
Joined: 8 May 18
Posts: 171
Credit: 79,415,920
RAC: 258,785
Level
Thr
Scientific publications
wat
Message 55505 - Posted: 9 Oct 2020 | 14:06:54 UTC

Got one task on the Windows 10 PC with GTX 1650, but the expected time seems to be too long.
Tullio
____________

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55506 - Posted: 9 Oct 2020 | 14:13:49 UTC - in response to Message 55505.

I think expected times estimates were broken by past failures. Size/time should be reasonable.

Richard Haselgrove
Send message
Joined: 11 Jul 09
Posts: 1030
Credit: 2,838,667,345
RAC: 3,857,128
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55507 - Posted: 9 Oct 2020 | 14:14:21 UTC - in response to Message 55505.

... the expected time seems to be too long.
Tullio

That's normal with a new application version, and it's the fault of the client - nothing for Toni to worry about.

It'll slowly get it right over time.

[CSF] Aleksey Belkov
Avatar
Send message
Joined: 26 Dec 13
Posts: 43
Credit: 845,921,512
RAC: 282,742
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55508 - Posted: 9 Oct 2020 | 14:16:19 UTC - in response to Message 55499.

Also can't get WUs.
Host: Windows 10 LTSC x64 / GTX 1060 6GB / Driver 451.67


| GPUGRID | Sending scheduler request: To fetch work.
| GPUGRID | Requesting new tasks for CPU and NVIDIA GPU
| GPUGRID | Scheduler request completed: got 0 new tasks
| GPUGRID | No tasks sent
| GPUGRID | No tasks are available for Long runs (8-12 hours on fastest card)
| GPUGRID | No tasks are available for New version of ACEMD

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55509 - Posted: 9 Oct 2020 | 14:20:18 UTC - in response to Message 55507.

Actually a couple of linux WUs went through. I hope it sorts out by itself.

Richard Haselgrove
Send message
Joined: 11 Jul 09
Posts: 1030
Credit: 2,838,667,345
RAC: 3,857,128
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55510 - Posted: 9 Oct 2020 | 14:24:00 UTC

It's probably just taking time to re-prepare all the tasks for the new app version, and we're all grabbing them faster than they become available.

My first Windows task has just returned and was validated.

spRocket
Send message
Joined: 27 Mar 20
Posts: 4
Credit: 32,928,494
RAC: 160,416
Level
Val
Scientific publications
wat
Message 55511 - Posted: 9 Oct 2020 | 14:30:32 UTC
Last modified: 9 Oct 2020 | 14:39:35 UTC

Getting "no work units" here on a GTX 960. I checked my driver version... 390.138.

*sigh* I hope Ubuntu 18.04LTS can cough up a newer driver on its own. Otherwise it's Einstein for now.

Edit: There is indeed a 450 driver for 18.04LTS in the automatic installer. Now I'm waiting for some long-running work units on WCG to get to their next checkpoint so I can reboot.

Richard Haselgrove
Send message
Joined: 11 Jul 09
Posts: 1030
Credit: 2,838,667,345
RAC: 3,857,128
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55512 - Posted: 9 Oct 2020 | 14:39:31 UTC

My Linux (Mint) has driver: 450.66, and was running Cuda 10 before the interruption. I'll keep watching and waiting for new tasks.

Profile Steve Dodd
Send message
Joined: 26 Dec 08
Posts: 17
Credit: 468,598,500
RAC: 2,261,083
Level
Gln
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55513 - Posted: 9 Oct 2020 | 14:59:11 UTC
Last modified: 9 Oct 2020 | 14:59:32 UTC

No WUs sent - 4 Windows 10 machines, varying driver versions all above 452.xx

Profile Steve Dodd
Send message
Joined: 26 Dec 08
Posts: 17
Credit: 468,598,500
RAC: 2,261,083
Level
Gln
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55515 - Posted: 9 Oct 2020 | 15:34:05 UTC - in response to Message 55513.

This isn't because we reported so many errored WUs when the project restarted is it? (No task available)

tullio
Send message
Joined: 8 May 18
Posts: 171
Credit: 79,415,920
RAC: 258,785
Level
Thr
Scientific publications
wat
Message 55516 - Posted: 9 Oct 2020 | 15:35:45 UTC

Firat task completed in about 6000 s.
Tullio
____________

DataC
Avatar
Send message
Joined: 15 Feb 10
Posts: 9
Credit: 9,174,431
RAC: 133,015
Level
Ser
Scientific publications
watwatwatwatwat
Message 55517 - Posted: 9 Oct 2020 | 15:45:56 UTC - in response to Message 55495.

Is there a reason why these new work units take insanely longer than before? Like from taking hours to days.

Richard Haselgrove
Send message
Joined: 11 Jul 09
Posts: 1030
Credit: 2,838,667,345
RAC: 3,857,128
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55518 - Posted: 9 Oct 2020 | 15:48:43 UTC - in response to Message 55517.

Is there a reason why these new work units take insanely longer than before? Like from taking hours to days.

They actually take much the same time as before. The figure you are seeing is an estimate and it's wrong for the first few tasks. It'll settle down.

Aurum
Avatar
Send message
Joined: 12 Jul 17
Posts: 285
Credit: 10,157,263,816
RAC: 5,791,452
Level
Trp
Scientific publications
watwatwat
Message 55519 - Posted: 9 Oct 2020 | 16:28:20 UTC - in response to Message 55517.

Is there a reason why these new work units take insanely longer than before? Like from taking hours to days.
Make your own estimate. How long did 10% complete take times ten.

Gunde
Send message
Joined: 6 Jan 15
Posts: 40
Credit: 5,765,127,206
RAC: 3,608,608
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 55520 - Posted: 9 Oct 2020 | 17:43:13 UTC

Got task for linux host now and running well now after 10 min.

A mention before new application would change the EST time in client and would not be accurate. On my host it would say 1 day 11 hours but it would go down when more host report completed task.
It will be adjusted after 1-2 days don't worry if this high.

Ian&Steve C.
Avatar
Send message
Joined: 21 Feb 20
Posts: 117
Credit: 1,582,087,015
RAC: 9,893,052
Level
His
Scientific publications
wat
Message 55521 - Posted: 9 Oct 2020 | 18:04:59 UTC - in response to Message 55520.

I concur. my linux hosts are back up and running.

Thanks Toni!
____________

tullio
Send message
Joined: 8 May 18
Posts: 171
Credit: 79,415,920
RAC: 258,785
Level
Thr
Scientific publications
wat
Message 55522 - Posted: 9 Oct 2020 | 18:06:51 UTC

Two more tasks on another PC with a GTX 1060 board. Estimated time is much lower.
Tullio
____________

spRocket
Send message
Joined: 27 Mar 20
Posts: 4
Credit: 32,928,494
RAC: 160,416
Level
Val
Scientific publications
wat
Message 55523 - Posted: 9 Oct 2020 | 18:07:03 UTC

Looks like things are back to normal for me, with the new driver installed.

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 568
Credit: 685,989,835
RAC: 1,684,873
Level
Lys
Scientific publications
watwatwatwatwat
Message 55525 - Posted: 9 Oct 2020 | 18:51:45 UTC - in response to Message 55521.

I concur. my linux hosts are back up and running.

Thanks Toni!

Thanks Toni.

[CSF] Aleksey Belkov
Avatar
Send message
Joined: 26 Dec 13
Posts: 43
Credit: 845,921,512
RAC: 282,742
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55526 - Posted: 9 Oct 2020 | 20:03:29 UTC

Apparently, the issue with getting WU has solved.
Thx Toni!

Profile ServicEnginIC
Avatar
Send message
Joined: 24 Sep 10
Posts: 236
Credit: 1,597,109,297
RAC: 2,268,011
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55527 - Posted: 9 Oct 2020 | 20:15:30 UTC

One more badge to your bag of Problems Solved.
Thank you, Toni

ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 17 Aug 08
Posts: 2695
Credit: 1,276,847,920
RAC: 407,123
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55529 - Posted: 9 Oct 2020 | 20:33:05 UTC

First 3 WUs returned successfully (Win 10, GTX1070, driver 445.87).

MrS
____________
Scanning for our furry friends since Jan 2002

Greg _BE
Send message
Joined: 30 Jun 14
Posts: 66
Credit: 70,368,572
RAC: 120,034
Level
Thr
Scientific publications
watwatwatwatwatwat
Message 55530 - Posted: 9 Oct 2020 | 23:31:41 UTC

Just picked up 4 tasks. GPU is busy with another project at the moment.

Stacie
Send message
Joined: 29 Mar 20
Posts: 11
Credit: 91,992,161
RAC: 190,221
Level
Thr
Scientific publications
wat
Message 55531 - Posted: 10 Oct 2020 | 0:20:52 UTC

Got 2 tasks. Back on the move!
____________

Greg _BE
Send message
Joined: 30 Jun 14
Posts: 66
Credit: 70,368,572
RAC: 120,034
Level
Thr
Scientific publications
watwatwatwatwatwat
Message 55534 - Posted: 10 Oct 2020 | 18:07:44 UTC

Name e7s38_e5s39p0f2-ADRIA_NTL9Bandit100ns-0-1-RND5048_0
Workunit 24115911


Finished ok on either my 1050TI or my 1080. Only the 1080 is dedicated to another project for 4hrs and then will start whichever project needs it next.

csbyseti
Send message
Joined: 4 Oct 09
Posts: 4
Credit: 727,196,630
RAC: 1,264,046
Level
Lys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55585 - Posted: 13 Oct 2020 | 5:34:05 UTC

After my restart on Monday i got some WU's and all seems to be normal.

But now i got no new WU's, Server Status looks OK with lots of WU's.

24938 GPUGRID 13.10.2020 07:01:04 Scheduler request completed: got 0 new tasks
24939 GPUGRID 13.10.2020 07:01:04 No tasks sent
24940 GPUGRID 13.10.2020 07:01:04 No tasks are available for New version of ACEMD
24941 GPUGRID 13.10.2020 07:01:04 No tasks are available for Anaconda Python 3 Environment

Driver on all Linux System is higher than 440.x

Tried Windows 10 System, same problem.

Erich56
Send message
Joined: 1 Jan 15
Posts: 712
Credit: 3,340,295,280
RAC: 621,453
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 55586 - Posted: 13 Oct 2020 | 5:56:33 UTC - in response to Message 55585.

After my restart on Monday i got some WU's and all seems to be normal.

But now i got no new WU's, Server Status looks OK with lots of WU's.

I have had the same problem since yesterday evening;
and from what one can read in other threads here in the forum, other crunchers (in fact: all of them?) also seem not to receive any tasks.

So let's hope that Toni can straighten this out ASAP.

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55587 - Posted: 13 Oct 2020 | 6:01:20 UTC - in response to Message 55586.
Last modified: 13 Oct 2020 | 6:03:09 UTC

That's true, thanks. Fixed.

roundup
Send message
Joined: 11 May 10
Posts: 32
Credit: 151,942,268
RAC: 495,171
Level
Ile
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55616 - Posted: 18 Oct 2020 | 8:00:24 UTC - in response to Message 55585.
Last modified: 18 Oct 2020 | 8:37:10 UTC

After my restart on Monday i got some WU's and all seems to be normal.

But now i got no new WU's, Server Status looks OK with lots of WU's.

24938 GPUGRID 13.10.2020 07:01:04 Scheduler request completed: got 0 new tasks
24939 GPUGRID 13.10.2020 07:01:04 No tasks sent
24940 GPUGRID 13.10.2020 07:01:04 No tasks are available for New version of ACEMD
24941 GPUGRID 13.10.2020 07:01:04 No tasks are available for Anaconda Python 3 Environment

Driver on all Linux System is higher than 440.x

Tried Windows 10 System, same problem.


Same problem here:
So 18 Okt 2020 09:55:47 | GPUGRID | Sending scheduler request: To fetch work.
So 18 Okt 2020 09:55:47 | GPUGRID | Requesting new tasks for NVIDIA GPU
So 18 Okt 2020 09:55:48 | GPUGRID | Scheduler request completed: got 0 new tasks
So 18 Okt 2020 09:55:48 | GPUGRID | No tasks sent
So 18 Okt 2020 09:55:48 | GPUGRID | No tasks are available for New version of ACEMD
So 18 Okt 2020 09:55:48 | GPUGRID | No tasks are available for Anaconda Python 3 Environment
So 18 Okt 2020 09:55:48 | GPUGRID | This computer has finished a daily quota of 1 tasks

GPU is a GTX 2070 Super which worked nicely before.
Latest Ubuntu, Nvidia Driver 450.80.02.

Any ideas?

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2210
Credit: 15,871,135,399
RAC: 930,957
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55617 - Posted: 18 Oct 2020 | 13:28:12 UTC - in response to Message 55616.

Same problem here:
So 18 Okt 2020 09:55:47 | GPUGRID | Sending scheduler request: To fetch work.
So 18 Okt 2020 09:55:47 | GPUGRID | Requesting new tasks for NVIDIA GPU
So 18 Okt 2020 09:55:48 | GPUGRID | Scheduler request completed: got 0 new tasks
So 18 Okt 2020 09:55:48 | GPUGRID | No tasks sent
So 18 Okt 2020 09:55:48 | GPUGRID | No tasks are available for New version of ACEMD
So 18 Okt 2020 09:55:48 | GPUGRID | No tasks are available for Anaconda Python 3 Environment
So 18 Okt 2020 09:55:48 | GPUGRID | This computer has finished a daily quota of 1 tasks

GPU is a GTX 2070 Super which worked nicely before.
Latest Ubuntu, Nvidia Driver 450.80.02.

Any ideas?
The message marked in red means that your host is "banned" for a day. This could be due to that too many tasks have failed on your given host successively. Reboot your host, if you haven't done it since the last driver update. If there's any overclock, revert back to original settings. Check all power connectors (unplug, check for burn marks, plug in again). When your host gets a task, check for GPU temperature.

Stacie
Send message
Joined: 29 Mar 20
Posts: 11
Credit: 91,992,161
RAC: 190,221
Level
Thr
Scientific publications
wat
Message 55618 - Posted: 20 Oct 2020 | 6:53:54 UTC - in response to Message 55495.
Last modified: 20 Oct 2020 | 6:54:21 UTC

Does anyone know what these messages on my Preferences page mean? Do they have anything to do with me not getting any work recently? What if anything should I do about them? Thanks-

Warning: Creating default object from empty value in /home/ps3grid/projects/PS3GRID/html/inc/prefs_util.inc on line 218

Warning: Creating default object from empty value in /home/ps3grid/projects/PS3GRID/html/project/project_specific_prefs.inc on line 240
____________

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2210
Credit: 15,871,135,399
RAC: 930,957
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55619 - Posted: 20 Oct 2020 | 11:44:10 UTC - in response to Message 55618.

Does anyone know what these messages on my Preferences page mean?
These are the results of the last BOINC server update at GPUGrid.
Do they have anything to do with me not getting any work recently?
No, they don't.
What if anything should I do about them?
Just ignore them.

Stacie
Send message
Joined: 29 Mar 20
Posts: 11
Credit: 91,992,161
RAC: 190,221
Level
Thr
Scientific publications
wat
Message 55620 - Posted: 20 Oct 2020 | 12:18:26 UTC - in response to Message 55619.

Okay, thank you!
____________

johndad5
Send message
Joined: 14 Nov 10
Posts: 1
Credit: 17,596,352
RAC: 18,438
Level
Pro
Scientific publications
watwatwatwatwatwat
Message 55622 - Posted: 24 Oct 2020 | 4:06:48 UTC

Hello, I am not getting any new tasks at all. I did abort 2 tasks because they stated there was no GPU. I can verify I have a GPU - GTX 960M that is up to date and working fine.

Any ideas oh what is going on?

Ian&Steve C.
Avatar
Send message
Joined: 21 Feb 20
Posts: 117
Credit: 1,582,087,015
RAC: 9,893,052
Level
His
Scientific publications
wat
Message 55623 - Posted: 24 Oct 2020 | 15:07:37 UTC - in response to Message 55622.

Hello, I am not getting any new tasks at all. I did abort 2 tasks because they stated there was no GPU. I can verify I have a GPU - GTX 960M that is up to date and working fine.

Any ideas oh what is going on?


sounds like you have a problem with the drivers not installed or not installed correctly.

it would be more helpful to asses the state of your system if it were not hidden. so i can't really give more help than that.
____________

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 974
Credit: 5,068,599
RAC: 16,648
Level
Ser
Scientific publications
watwatwatwat
Message 55624 - Posted: 24 Oct 2020 | 18:04:59 UTC - in response to Message 55623.
Last modified: 24 Oct 2020 | 18:07:16 UTC

If no GPU is detected, the host shouldn't get tasks.

Post to thread

Message boards : News : Project restarted