Steve replied:

yes this make sense unfortunately. In the previous round of "inputs_v3**" it was calculating things incorrectly for any molecule containing Iodine. This is heaviest element in our dataset. The computational cost of this QM method scales with the size of the elements (it depends on the number of electrons). We are resending the incorrect calculations for Iodine containing molecules in this round of "v4" work units. Therefore the v4 set is a subset of the previous v3 WUs containing heavier elements, hence there are more OOM errors.

Any change in this situation?

I got my 12GB card back and my haphazard data collection seems to have it under a 9% error rate and with the very last grab showing 5.85%.

yes this make sense unfortunately. In the previous round of "inputs_v3**" it was calculating things incorrectly for any molecule containing Iodine. This is heaviest element in our dataset. The computational cost of this QM method scales with the size of the elements (it depends on the number of electrons). We are resending the incorrect calculations for Iodine containing molecules in this round of "v4" work units. Therefore the v4 set is a subset of the previous v3 WUs containing heavier elements, hence there are more OOM errors.

Download error causing the zip file to be corrupted because it is missing the end of file signature.

I was getting that on a Google Drive zip archive a couple of days ago. Switching browsers let me download the archive correctly so it would unpack.

to be expected with 8-10GB cards.

might get better context if you split the graphs up by card type. so you can see the relative error rate vs different VRAM sizes. I'm guessing most errors come from the 8GB cards.

between your systems and mine, looking at the error rates;

~23% of tasks need more than 8GB
~17% of tasks need more than 10GB
~4% of tasks need more than 12GB
<1% of tasks need more than 16GB

me personally, i wouldn't run these (as they are now) with less than 12GB VRAM.

Not sure why but...

Error rates seemed to start dropping after 5pm (23:00 Zulu) today. Overall error average since 2/11 across my 5 Nvid cards was 26.7% with it slowly creeping down over time. Early on a little bit of this was the result of lowering clocks to eliminate the occasional segfault (0x8b).

The average of the last two captures today across the 5 cards was 20.5%

For the last 6 hour period I just checked, my 10GB card average error rate dropped to 17.3% (15.92 & 18.7) and the 8GB card error rate was at 21.3%.

Skip