A New version of ACEMD 2.17 was introduced on September 2nd 2021.
This was announced by GDF at message #57257
Current and previous program versions can be consulted at Gpugrid apps page.

This new version presents a peculiar behavior on multi GPU hosts, not seen on previous app versions, and first mentioned by Ian&Steve C. at message #57261.

For a deeper study, I happened to have two tasks of this new app version 2.17 simultaneously running at this twin GTX 1650 GPU system.
Device #0 at this host is an ASUS ROG-STRIX-GTX1650-O4G-GAMING graphics card.
I took screenshot for Boinc Manager readings.
As seen at this image, Boinc Manager "thinks" that the first task is running at device #0, and the second task in device #1.
However, looking at Psensor monitoring utility, while device #0 is running at 100% activity, memory usage 77% and PCIe usage 61%, device #1 is really cold and inactive.
This lead me to think that both tasks were running in fact at the same device #0, being its resources splitted between these two tasks.
And it was confirmed when results came out: Both task #27078190 and task #27078191 are shown to have run at device #0.
Then I happened to catch one more only task for the same system, that was executed in exclusive.
Result for this task #27078213 can be seen here.
While tasks #27078190 and #27078191 that were run concurrently took 32451 and 32473 seconds respectively, task #27078213 that run in exclusive, took 15328 seconds. That is, well less than half the execution times of the tasks executed concurrently.

For additional confirmation.
At other of my hosts, this triple GTX 1650 GPU system, were simultaneously executed these three tasks.
Device #0 at this system is the same ASUS ROG-STRIX-GTX1650-O4G-GAMING graphics card than in previous mentioned twin GPU system.
While executing, Boinc Manager was showing that they were running at devices #0, #1 and #2.
But, as can be seen at this Psensor image, only device #0 is performing at full rate, while devices #1 and #2 are actually idle.
After finishing, execution times for these three tasks were 46512, 46677 and 46719 seconds.
If we take 15328 seconds as the run time for a single task executed in exclusive, above times are about 3x longer than this time. And the other two GPUs #1 and #2 have stayed unused in the meantime. For both Gpugrid and other GPU projects.

In massive multi GPU hosts, like may be this impressive Ian&Steve C. 8x GPU one, the potential performance drop is multiplied by N.
And maybe even Device #0 resources can result overflowed when attempting to simultaneously execute such a number of tasks...

BOINC has had two different ways of signalling to a science app which device number it should utilise.

OLD (deprecated) - pass the device number on the command line
NEW (current) - pass the device number in an XML file init_data.xml in the slot directory.

The situation is complicated here by the use of the wrapper app in the calling chain. Your screen shot (and every task I've looked at so far) is showing that the wrapper is calling the acemd3 application, and it looks like it's using the command line convention. That shouldn't be happening.

I've still failed to catch any of these new tasks. I need to maintain my Linux hosts soon, so I'll look for a newer driver while I'm at it. But if anyone catches a live task, and can examine init_data.xml, we might get some extra clues.

Edit - look for lines like

The fact that the BOINC “thinks” it’s running on the wrong device exposes two issues.

1. BOINC assigned it to the device it intended, but that was not properly communicated to the application. Maybe a disconnect through the wrapper process, or possibly it’s hard coded for device 0, as an artifact from dev testing.

2. This also shows that BOINC doesn’t do any verification of what device a process is ACTUALLY running on. It just trusts that the message was received. If it did verification it would show it running on the wrong device.

It goes further than simply checking psensor for device activity. If you run nvidia-smi, you can see that the acemd is actually assigned to and running on device0, leaving the idle device with no process running.

Since I only had one GPUGRID task running, and GPUGRID has a 100 resource share, with Einstein at 0 RS, when I rebooted BOINC, the GPUGRID process restarted on 0 and BOINC showed/assigned at 0 as well. This was likely a “when stars align” scenario. If I had more GPUGRID tasks, I would have undoubtedly continued to experience the issue.

Regarding the method of passing the arguments, it should depend on what server version is being used (Einstein with its old server version has no problem with this method). it does appear to be some kind of command line argument, but at which step of the process (pre or post wrapper) is referred to by the stderr file is not clear. The command is being sent as --boinc input --device 0. This is the exact same command structure as used with no issue on previous apps/tasks.
____________

Thank you both for sharing your knowledge once more.
I'm prioritizing my twin GPU host for requesting new tasks (although at the cost of penalizing my other hosts chances).
I'll consider your tips if getting some (currently scarce) Gpugrid task.

Currently this system is running two Genefer 21 Primegrid tasks, also very GPU power demanding.
Device #0 task properties
Device #1 task properties
I found their assignments as described by Richard Haselgrove in their init_data.xml files at directories /var/lib/boinc-client/slots/4 and /var/lib/boinc-client/slots/5 respectively.
At this situation, nvidia-smi command returns as follows:

Regarding the method of passing the arguments, it should depend on what server version is being used (Einstein with its old server version has no problem with this method). it does appear to be some kind of command line argument, but at which step of the process (pre or post wrapper) is referred to by the stderr file is not clear. The command is being sent as --boinc input --device 0. This is the exact same command structure as used with no issue on previous apps/tasks.

Today when returning home I've got a surprise waiting:
Two new version of ACEMD 2.17 long tasks are running concurrently at mentioned twin GPU host.
First of all, I've ceased requesting new tasks from all my hosts, to increase the chance for any newly generated tasks to be catched by other users.
Then, I've checked that the mentioned behavior at the beginning of this thread is reproducing with these two tasks also.

Task #32640189
Task #32640190

Boinc Manager reading

Device #0 task properties (as shown by Boinc Manager)
Device #1 task properties (as shown by Boinc Manager)

init_data.xml slots/5 (Device #1 task)

<gpu_type>NVIDIA</gpu_type>
<gpu_device_num>1</gpu_device_num>
<gpu_opencl_dev_index>1</gpu_opencl_dev_index>
<gpu_usage>1.000000</gpu_usage>
<ncpus>0.490000</ncpus>

<gpu_type>NVIDIA</gpu_type>
<gpu_device_num>0</gpu_device_num>
<gpu_opencl_dev_index>0</gpu_opencl_dev_index>
<gpu_usage>1.000000</gpu_usage>
<ncpus>0.490000</ncpus>

I can think of three things I'd really like to see:

1) In the slot directories: is there an init_data.xml file, and does the content match the <device_number> reported by BOINC for that slot?
2) What command line was used to launch the ACEMD3 process? I'd use process explorer to see that in Windows.
3) Does the ACEMD3 v2.17 binary have an API_VERSION text string embedded within it?

1) In the slot directories: is there an init_data.xml file, and does the content match the <device_number> reported by BOINC for that slot?

Thanks. I finally got my own task(s), so I can answer my own other questions.

2) I used htop to view the actual command lines. There are a huge number of acemd3 threads - at least 18 (??!!). but the command line is uniform:

- At previous v2.12 version, wrapper was running directly the acemd3 process. At current v2.17 version, a preliminary decompressing stage seems to be executed, then bin/acemd3 process is run.

there seems to be a difference in command used for the device assignment from the past (I missed this before)

old tasks/apps used this:

The command line for acemd3 is set by the wrapper.

I can't see a meaningful use for the word 'input' on that line. Keywords are set with '--' leading, values without. So the old syntax might have been

--boinc [filename]
--device [integer]

It feels odd, but it's possible - there is a file in the workunit bundle which is given the logical name 'input' at runtime.

In which case, why is it missing from the 217 app template, and what is the effect likely to be? (noting that the 217 apps are actually running with the input files supplied)

Just possibly, the sequence might be

Detected keyword --boinc
Expecting filename
No filename found
Abandon parsing of command line
Using default filename 'input'
Using default device '0'
Proceeding with task

just noting the obvious difference between what worked in the past vs what doesn't work now. it worked before, but not now. what's different? the lack of "input" in the command.
____________

And I'm trying to think like a computer, and trying to work out why the observed difference might cause the observed effect.

based on the assumption that the string "input" relates to a filename. rather could be a preprogrammed specific command, telling the app/wrapper to perform some function. without full knowledge of the code, it's just speculation at this point.
____________

API_VERSION does appear in the wrapper binary too. you can see this by converting "API_VERSION" to a hex string and searching the binary with hexedit.

which makes sense since I was under the impression that the whole point of the wrapper was to be a middle man between BOINC and the science app, allowing some greater flexibility in how the project packages and delivers their apps.

older 2.12 apps have wrapper API_VERSION_7.7.0, the new 2.17 app has wrapper API_VERSION_7.5.0
____________

Yes, I'd expect that. The whole point of the wrapper is to be 'boinc-aware': it will need to read init_data.xml, and export the readings (like device number) into an external format like the one that the new version of acemd3 can act on.

Or not, as the case may be.

Finally, this problem has been corrected by the deployment of New Version of ACEMD 2.18 on Sep 14 2021 | 11:44:39 UTC
The wrapper version packed with this new app is the same that already was with previous New Version of ACEMD 2.12: 7.7.26016
It seems that this has been the solution.

To Gpugrid Project Developers, once more: Well done!!!

Have you actually received the new app with some new work?
No luck here for the past few days.

Have you actually received the new app with some new work?

I have two ADRIA tasks running now on host 132158 - Linux Mint, driver v460.

htop shows that they have different command lines, ending in '--device 0' and '--device 1'.
nvidia-smi shows an acemd3 app running on GPU 0, and another running on GPU 1.

All is looking good so far.

The only strange thing is that one is running app version 101, and the other is running version 1121. Two identical cards, so we'll see who wins!

The only strange thing is that one is running app version 101, and the other is running version 1121. Two identical cards, so we'll see who wins!

I got two new 2.18 tasks, one each on two hosts. Both CUDA_101 though.

Here's a taster show, after 4 hours elapsed:

v1121 at 12.727%
(device 1, in 4x PCIe slot)

v101 at 11.368%
(device 0, in 16x PCIe slot, driving monitor)

I received 4 GPUGrid WU's on my dual GPU system - RTX3070 and RTX2070.....
And it was happily crunching 1 unit on each of the GPu's, until Boinc downloaded and ran a WCG unit. The GPUGrid unit then failed with this message....

<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
15:50:51 (128895): wrapper (7.7.26016): starting
15:50:51 (128895): wrapper (7.7.26016): starting
15:50:51 (128895): wrapper: running /bin/tar (xf conda-pack.tar.bz2)
15:52:07 (128895): /bin/tar exited; CPU time 75.576773
15:52:07 (128895): wrapper: running bin/acemd3 (--boinc --device 0)
19:27:16 (136305): wrapper (7.7.26016): starting
19:27:16 (136305): wrapper (7.7.26016): starting
19:27:16 (136305): wrapper: running bin/acemd3 (--boinc --device 1)
ERROR: /home/user/conda/conda-bld/acemd3_1618916459379/work/src/mdsim/context.cpp line 318: Cannot use a restart file on a different device!
19:27:20 (136305): bin/acemd3 exited; CPU time 3.452513
19:27:20 (136305): app exit status: 0x9e
19:27:20 (136305): called boinc_finish(195)

19:27:16 is exactly the timestamp that the WGC process started.

looks like it wont play happily with different projects. Has anyone else seen this?
I've suspended WCG for the moment.

Has anyone else seen this?

I received 4 GPUGrid WU's on my dual GPU system - RTX3070 and RTX2070.....
And it was happily crunching 1 unit on each of the GPu's, until Boinc downloaded and ran a WCG unit. The GPUGrid unit then failed with this message....

<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
15:50:51 (128895): wrapper (7.7.26016): starting
15:50:51 (128895): wrapper (7.7.26016): starting
15:50:51 (128895): wrapper: running /bin/tar (xf conda-pack.tar.bz2)
15:52:07 (128895): /bin/tar exited; CPU time 75.576773
15:52:07 (128895): wrapper: running bin/acemd3 (--boinc --device 0)
19:27:16 (136305): wrapper (7.7.26016): starting
19:27:16 (136305): wrapper (7.7.26016): starting
19:27:16 (136305): wrapper: running bin/acemd3 (--boinc --device 1)
ERROR: /home/user/conda/conda-bld/acemd3_1618916459379/work/src/mdsim/context.cpp line 318: Cannot use a restart file on a different device!
19:27:20 (136305): bin/acemd3 exited; CPU time 3.452513
19:27:20 (136305): app exit status: 0x9e
19:27:20 (136305): called boinc_finish(195)

19:27:16 is exactly the timestamp that the WGC process started.

looks like it wont play happily with different projects. Has anyone else seen this?
I've suspended WCG for the moment.

Thanks for the quick reply clarifying the problem.
But 2 years and no fix to what seems like quite a basic problem......

Wait a minute . . . . . I thought I read in this thread on the previous beta releases that the restarting on a different device issue was solved???

Wait a minute . . . . . I thought I read in this thread on the previous beta releases that the restarting on a different device issue was solved???

I'd certainly be interested to know if one is significantly faster than the other.

If they keep both apps active, then the BOINC mechanism for choosing the most efficient application should become active once 10 valid tasks are completed on both apps.

The 1121 app's APR should prevail.

Hmmmm . . . . not enough tasks to draw a concrete conclusion, but on my daily driver with three identical RTX 2080 cards, the CUDA101 app was 2000 seconds faster than the CUDA1121 app.

https://www.gpugrid.net/results.php?userid=516740&offset=0&show_names=0&state=3&appid=

Though might be attributed to restarting on a different device. But same type of card. All cards are hybrids and have temps well under control and boost the same.

Anybody seen any sign of your credits exported to 3rd party aggregation websites yet?
Finished work over a day ago and still no stats from GPUGrid.

Anybody seen any sign of your credits exported to 3rd party aggregation websites yet?

Anybody seen any sign of your credits exported to 3rd party aggregation websites yet?
Finished work over a day ago and still no stats from GPUGrid.

Thanks, Gianni.

fixed

By the way, and returning to the original topic of this thread to finish with some conclusion:
Current New Version of ACEMD 2.18 came back to the same wrapper v7.7.26016 previously used and working with old app v2.12
When I was trying to debug, it coincided that I had stopped one of my hosts while v2.12 was the active app version.
I recovered the wrapper packed with that app version, file wrapper_26198_x86_64-pc-linux-gnu
I renamed it the same than the failing wrapper included with app v2.17, wrapper_26014_x86_64-pc-linux-gnu
Then, I copied it to folder /var/lib/boinc-client/projects/www.gpugrid.net in replacement of existing wrapper file.
After that, I was able to finish v2.17 task 1_4-CRYPTICSCOUT_pocket_discovery_9a871de0_3995_4230_8b91_6874b82272c0-1-2-RND6341_2, which, at last, partially ran at device other than 0: device 2
I kept screenshot of this:



Based on this test, I'm pretty sure that the problem in New Version of ACEMD 2.17 app was due to wrapper version 7.5.26014 packed with it.

Finally, this problem was corrected by the deployment of New Version of ACEMD 2.18 on Sep 14 2021 | 11:44:39 UTC
Gpugrid apps page