Just starting the thread for discussion of this new application. ATM = AToM.

after a little snafu with the first batch (incorrect config files), the latest batch seems to run on my system. no idea for runtime yet or if it will finish successfully.

This is another Python-based application. the package ships with the python environment similar to how the PythonGPU Reinforcement Learning (RL) app does.

Test Bench:
Xeon E5-2697Av4 (16c/32t)
64GB DDR4-2400 RDIMM (ECC)
RTX 3060 12GB
Ubuntu 22.04.1


So far observed behavior:
-uses ~97% of the GPU core, ~45% GPU memory bus, ~0-1% PCIe bus, close to full power use.
-about 400-500MB VRAM used (low, like acemd3)
-does not like to be paused and resumed, or BOINC stopped and restarted. it causes the task to fail

unknown total runtime expectation since the one task I had failed when I restarted BOINC lol.
____________

about the restart failure. looks like it fails trying to create a directory that already exists.

another quality of life improvement should be adding a <weight> line to the main task in the job.xml file. right now with 2 tasks in the file, and no weights defined, I'm guessing it splits it 50/50 and it thinks the task is 50% done once the extraction phase is complete.
____________

task ran to completion in about an hour. but hit an error and threw it all away because the file size is too big.

Have over a dozen of quick-failing ATM tasks.

The wrapper does not have a correctly name tar file or something.

02:56:29 (1242346): wrapper: running /bin/tar (xf input.tar.bz2)
/bin/tar: This does not look like a tar archive
bzip2: (stdin) is not a bzip2 file.
/bin/tar: Child returned status 2
/bin/tar: Error is not recoverable: exiting now

looks like the small batch of tasks that went out today are better setup. ran for about an hour and completed successfully without the file size issue when complete.

great :)

still would like a little more background info on these tasks, what they are doing, and the goal of the research.
____________

This one https://www.gpugrid.net/workunit.php?wuid=27399736 is runnig for about 11 hours and it is stuck at 66,666% for at least 4 hours now. There is almost no load on the GPU. Just a few percent (3-5) once in a while, but constantly some load on the memory controller (10-30). Hope it will finish some day :)

still no official communication from the project about these tasks.

the recent batches have been very hit or miss and exhibit much different behavior than my initial post.

"TL2" tasks, ran for hours and hours with little to no GPU or CPU use. I aborted them and moved on.

"TL3" tasks yesterday, also had little to no GPU or CPU use, but did complete in about 30 mins.

"TL4" tasks today seem like a repeat of TL2. no GPU use, runs for hours with no progress.

also weights need to be defined in the jobs.xml file so the tasks don't jump to 75% after a few seconds and then sit there for hours doing nothing.
____________

Just been sent a TL4 from WU 27405970. I see you've aborted two previous tasks from the same WU, Ian, on two different machines. Did you get any CPU usage figures from previous runs? I think I'll start it up with the GTX 1660 plus one core, but I'll probably abort it myself if it doesn't show much response.

they spin up multiple processes like the Python tasks do. but i didnt catch them at the very beginning to see if they spike in use or anything like that.

once they get going, they basically sit idle as far as the GPU and CPU go. little to no use at all. i just killed them rather than letting them sit there for hours occupying my GPU.
____________

OK, I've set 3 CPUs for continuity from the current Python task, and I've put weights of 1-1-1-97 in the job file so I can see what's happening.

My normal remote monitoring console shows the current average CPU usage, and I've put nvidia-smi on a five second loop. If either of those drops to zero, I'll abort it.

Chocks away!

I see what you mean. Nearly half an hour in, CPU usage is showing around 25% of a single core, and GPU usage spiked once, to 41%, after about a quarter of an hour. It's one way of saving electricity, but I'd rather be doing something useful. Aborting.

1.13 ATM running fine for me.
Keep aborting them and I'll run them for you.

FWIW, the first task I received completed successfully.

http://www.gpugrid.net/workunit.php?wuid=27410175
____________
Reno, NV
Team: SETI.USA

I've also finished one:
https://www.gpugrid.net/workunit.php?wuid=27410166

We're both using Linux Mint. It seems to crash on Win 10 machines (computer #600532 is mine, too).

Over night, I had 4 of these tasks cancelled by server.
____________
Reno, NV
Team: SETI.USA

1.13 ATM running fine for me.
Keep aborting them and I'll run them for you.

keep in mind these are bata tasks, and the batch being sent NOW are not necessarily the same as the batch sent last week and wont be the same as whatever is sent sometime in the future, until they get all the bugs worked out. the tasks last week basically ran with no perceived use of the GPU or CPU, so what were they doing? who knows. no official word from the project about these tasks at all. I wasn't willing to let the GPU/CPU be occupied for hours on end with the task spinning it's wheels when they could be doing something more useful.

maybe this current batch has been tweaked from last week and thats why they are working OK, for those that have completed this latest batch, did they have any meaningful use of the GPU or CPU? it also seems this batch was released with a new Windows application (they were Linux only before) for testing.
____________

keep in mind these are bata tasks, and the batch being sent NOW are not necessarily the same as the batch sent last week and wont be the same as whatever is sent sometime in the future, until they get all the bugs worked out. the tasks last week basically ran with no perceived use of the GPU or CPU, so what were they doing? who knows. no official word from the project about these tasks at all. I wasn't willing to let the GPU/CPU be occupied for hours on end with the task spinning it's wheels when they could be doing something more useful.

maybe this current batch has been tweaked from last week and thats why they are working OK, for those that have completed this latest batch, did they have any meaningful use of the GPU or CPU? it also seems this batch was released with a new Windows application (they were Linux only before) for testing.

Well, most of us know that Abouh reads every word written on these threads and without much song and dance, makes changes. He is the Only Admin on all the projects who diligently attend. Maybe, quite possibly. No arguments with your tweaking statement.

keep in mind these are bata tasks, and the batch being sent NOW are not necessarily the same as the batch sent last week and wont be the same as whatever is sent sometime in the future, until they get all the bugs worked out. the tasks last week basically ran with no perceived use of the GPU or CPU, so what were they doing? who knows. no official word from the project about these tasks at all. I wasn't willing to let the GPU/CPU be occupied for hours on end with the task spinning it's wheels when they could be doing something more useful.

maybe this current batch has been tweaked from last week and thats why they are working OK, for those that have completed this latest batch, did they have any meaningful use of the GPU or CPU? it also seems this batch was released with a new Windows application (they were Linux only before) for testing.

_______________________

Well, most of us know that Abouh reads every word written on these threads and without much song and dance, makes changes. He is the Only Admin on all the projects who diligently attend. Maybe, quite possibly. No arguments with your tweaking statement.

https://gpugrid.net/result.php?resultid=33321222

ran for 10+hours, failed due to file size limit after an otherwise successful computation.

:(
____________

... failed due to file size limit
:(

... failed due to file size limit
:(

I am just trying to remember with which other application we've had the same problem some time ago - last year or 2 years ago ???

Yea, I got my first ATM checkpoint :-)
Now my list of ATM ULs are stuck.

Yea, I got my first ATM checkpoint :-)
Now my list of ATM ULs are stuck.

On past chance, I bet and lost.
Currently, I'm only processing ACEMD tasks, when available. I happened to catch one this morning.

GDF, Should I Abort these 12 completed ATM WUs that won't upload or is there a reasonable chance you'll fix it?

Well, I just achieved my 100 hours, which was my 1st priority. I will abort and reset (if necessary) the completed tasks I have. If/when the project gets its act together, I'll be back.
____________
Reno, NV
Team: SETI.USA

For me it's just this:

I just aborted the upload (not the workunit) and then it was reported as valid.

https://www.gpugrid.net/results.php?hostid=604029

I just aborted the upload (not the workunit) and then it was reported as valid.

https://www.gpugrid.net/results.php?hostid=604029

For me it's just this:

So 26 Feb 2023 11:57:00 CET | GPUGRID | Started upload of TL9_55-RAIMIS_TEST_ATM-0-1-RND1804_0_0
So 26 Feb 2023 11:57:02 CET | GPUGRID | Backing off 04:12:16 on upload of TL9_55-RAIMIS_TEST_ATM-0-1-RND1804_0_0
So 26 Feb 2023 11:57:19 CET | GPUGRID | Started upload of TL9_55-RAIMIS_TEST_ATM-0-1-RND1804_0_0
So 26 Feb 2023 11:57:22 CET | GPUGRID | Backing off 05:10:06 on upload of TL9_55-RAIMIS_TEST_ATM-0-1-RND1804_0_0

No message about the size, just about backing off.
Hooray!

I just aborted the upload (not the workunit) and then it was reported as valid.

I just aborted the upload (not the workunit) and then it was reported as valid.

https://www.gpugrid.net/results.php?hostid=604029

I just aborted the upload (not the workunit) and then it was reported as valid.

https://www.gpugrid.net/results.php?hostid=604029

Indeed, this worked out for me as well.

task ran to completion in about an hour. but hit an error and threw it all away because the file size is too big.

upload failure: <file_xfer_error>
<file_name>T11_4-RAIMIS_TEST_ATM-0-1-RND7054_2_0</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>

what a waste.

this now is a topic also on this thread:
https://www.gpugrid.net/forum_thread.php?id=5379
which has been opened by the developer Quico

How can I get ATM ?
Serverstatus tells me, there are more then hundred WUs ready to send at the moment.
Boinc Manager tells me:
Sa 25 Mär 2023 14:20:07 CET | GPUGRID | No tasks are available for ATM: Free energy calculations of protein-ligand binding

The PC is running with Ubuntu 20LTS, Geforce1070ti and driver 470.16

you need to enable beta/test applications in your project preferences
____________

Ah, Thanks. The "test application" setting I have missed.

Now I have to wait some hours for the download to be finished.

So far, I noticed on ATM tasks an abnormal progress notification.
Progress usually jumped from 0% to 0.199% in a short first step, and then directly to 100% in a second long step, staying so until task completion.
Along this second step, estimated time remaining was not shown ( "---" shown instead)

Example of wrong progress notification:
CDK2_29_26_5-QUICO_ATM_OFF_STEPS-2-5-RND5867_0




Today, I catched two ATM tasks showing a linear progression and accurate estimated time remaining.
At this moment, both of them are still in progress.

Examples of right progress notification:
Tyk2_jmc_23_jmc_27_2-QUICO_ATM_OFF12_STEPS-0-5-RND0292_2




Tyk2_jmc_23_ejm_55_5-QUICO_ATM_OFF12_STEPS-0-5-RND1896_3

There is still a mix of old, broken progress tasks along with fixed progress tasks in rotation.

Just depends on whether you get a new _0 or an older _x wingman task.

No, it's not the replication number.

The clue is in the task name: one with "STEPS-0-5" will show normal progress, one with any other "STEPS-n-5" will jump quickly to 100%.

The old, very long running, tasks processed 341 samples all in one go. The new shorter ones have been split into five shorter runs, processing 70 samples each (confusingly numbered 0 to 4 - I've never seen a 'steps-5-5').

Number zero - the first in the chain - processes samples 1 to 70, which is what the progress display expects. The second processes samples 71 to 140 - so it starts beyond the finishing point. And so on.

Nice explanation.
This makes full sense to that behavior.