Advanced search

Message boards : Number crunching : PCIe 2.0 and GTX 970

Author Message
Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42762 - Posted: 6 Feb 2016 | 13:51:41 UTC

I just picked up a GTX 970 to add to this project and am contemplating where to place it in the computer. I have three PCIe slots I could place it in on my Asus Sabertooth Z97 board. The top two are PCIe 3.0 x16 and the bottom is PCIe 2.0 x16. The top slot has a GTX 980Ti in it and I was thinking about placing the new 970 in the bottom slot to give the cards a little more "breathing room" but I don't know how much of a difference placing it in a PCIe 2.0 slot makes.

Both cards are EVGA with the ACX 2.0 cooling which means they're dumping heat into the case, but I have a rather large case with really good airflow (Phanteks Enthoo Primo) so I don't know how much of a problem it will be with a second card. For reference, my 980Ti nearly always stays under 60C on GPUGrid with the default fan curve.

So it comes down to, is it better to place both cards in PCIe 3.0 slots and have them run warmer, or to skip a slot and have one card in a PCIe 2.0 slot?

Thanks for any advice.

Betting Slip
Send message
Joined: 5 Jan 09
Posts: 670
Credit: 2,498,095,550
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42764 - Posted: 6 Feb 2016 | 14:35:59 UTC - in response to Message 42762.

PCIe 2 slot will show no noticeable difference.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42765 - Posted: 6 Feb 2016 | 14:37:54 UTC - in response to Message 42764.

PCIe 2 slot will show no noticeable difference.


Okay. Thanks for the reply!

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2343
Credit: 16,201,255,749
RAC: 7,520
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42766 - Posted: 6 Feb 2016 | 20:36:56 UTC - in response to Message 42762.

I just picked up a GTX 970 to add to this project and am contemplating where to place it in the computer. I have three PCIe slots I could place it in on my Asus Sabertooth Z97 board. The top two are PCIe 3.0 x16 and the bottom is PCIe 2.0 x16.
The bottom slot is mechanically x16, but electrically it's only x4, so it's 8 times slower than the top slot.

The top slot has a GTX 980Ti in it and I was thinking about placing the new 970 in the bottom slot to give the cards a little more "breathing room" but I don't know how much of a difference placing it in a PCIe 2.0 slot makes.
It depends on the workunit type. For example the recent GERARD_A2AR workunits will be significantly slower in that slot, the others won't.

Both cards are EVGA with the ACX 2.0 cooling which means they're dumping heat into the case, but I have a rather large case with really good airflow (Phanteks Enthoo Primo) so I don't know how much of a problem it will be with a second card. For reference, my 980Ti nearly always stays under 60C on GPUGrid with the default fan curve.
For these cards a fan placed on the side plate blowing the hot air outwards is recommended.

So it comes down to, is it better to place both cards in PCIe 3.0 slots and have them run warmer, or to skip a slot and have one card in a PCIe 2.0 slot?
I would place it in the lowest slot to give breathing room for the 980Ti, regardless of the performance decrease it could bring. Moreover if you put the 2nd card in the 2nd PCIe3.0 slot, then both will run at only x8, reducing the PCIe bandwith for the GTX980Ti, which should be avoided.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42767 - Posted: 6 Feb 2016 | 20:56:45 UTC - in response to Message 42766.

Thanks, Retvari. I was kind of hoping you would see this as you always give good detailed replies.

I'll put it in the bottom slot and see how it performs.

Adding a fan to the side panel isn't an option on this case, unfortunately, but there are two bottom-mounted fans which direct cool air up toward the GPUs. Hopefully that won't send too much hot air up to the 980Ti from the 970. I'm doubtful that reversing the fans to create an exhaust on the bottom would provide much benefit.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42768 - Posted: 6 Feb 2016 | 21:26:22 UTC

I was just thinking, another option might be to swap some cards around between rigs. I have another machine with two 750Ti cards. I could mix them up so that one case has 980Ti + 750Ti and the other with 970 + 750Ti. I could then put the 970 in a PCIe 3.0 slot in the other machine and put the 750Ti in the second slot on both machines as they don't put out much heat. That way, all 4 cards would be in PCIe 3.0 slots and the heat management may not be as much of a concern.

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2343
Credit: 16,201,255,749
RAC: 7,520
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42769 - Posted: 6 Feb 2016 | 23:24:09 UTC - in response to Message 42768.
Last modified: 6 Feb 2016 | 23:33:59 UTC

I would put the GTX 750Ti to the PCIe 2.0 slot as well. Your hosts have OSes with WDDM, and the PCIe bandwidth limitation has lower impact on GPU performance than WDDM, so there's no point giving a mid-range card (like the GTX 750Ti) a PCIe3.0x8 slot and limit the airflow of a high-end card and limit its PCIe bandwidth to x8 at the same time.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42770 - Posted: 6 Feb 2016 | 23:32:39 UTC - in response to Message 42769.
Last modified: 6 Feb 2016 | 23:36:15 UTC

So the CPU's 16 PCIe lanes don't divide up evenly between the 3.0 and 2.0 slots? I like your idea if that is the case.

The 970 won't arrive for a few days yet. I'll try that configuration with the 980Ti and the 970 occupying the top slot in each machine with a 750Ti in the bottom slot of each.

Thanks for the feedback.

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2343
Credit: 16,201,255,749
RAC: 7,520
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42771 - Posted: 6 Feb 2016 | 23:37:43 UTC - in response to Message 42770.
Last modified: 6 Feb 2016 | 23:40:01 UTC

So the CPU's 16 PCIe lanes don't divide up evenly between the 3.0 and 2.0 slots? I like your idea if that is the case.
No, the PCIe3.0x16 comes directly from the CPU, and divided between the two PCIe3.0 slots if necessary (when there is a GPU in the 2nd PCIe3.0 slot), while the PCIe2.0 comes from the "South Bridge" (Z97) chip.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42772 - Posted: 6 Feb 2016 | 23:39:54 UTC - in response to Message 42771.

So the CPU's 16 PCIe lanes don't divide up evenly between the 3.0 and 2.0 slots? I like your idea if that is the case.
No, the PCIe3.0x16 comes directly from the CPU, and divided between the two PCIe3.0 slots if necessary (when there is a GPU in the 2nd PCIe3.0 slot), while the PCIe2.0 comes from the "South Bridge" (Z97) chip.


Okay. Thank you for clearing that up. Always learning here.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42775 - Posted: 9 Feb 2016 | 13:04:43 UTC - in response to Message 42770.

The 970 won't arrive for a few days yet. I'll try that configuration with the 980Ti and the 970 occupying the top slot in each machine with a 750Ti in the bottom slot of each.


In support of this setup I've been looking at my app_config and trying to figure out how to adjust it such that the top card (980Ti/970) will run two tasks at once and the bottom card (750Ti) only run one task. Is this possible? I'm not seeing any tags on the BOINC Client Config guide that would apply to this.

Jacob Klein
Send message
Joined: 11 Oct 08
Posts: 1127
Credit: 1,901,927,545
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42783 - Posted: 9 Feb 2016 | 23:01:17 UTC - in response to Message 42775.
Last modified: 9 Feb 2016 | 23:03:15 UTC

The 970 won't arrive for a few days yet. I'll try that configuration with the 980Ti and the 970 occupying the top slot in each machine with a 750Ti in the bottom slot of each.


In support of this setup I've been looking at my app_config and trying to figure out how to adjust it such that the top card (980Ti/970) will run two tasks at once and the bottom card (750Ti) only run one task. Is this possible? I'm not seeing any tags on the BOINC Client Config guide that would apply to this.


You can't do it, with an app_config.xml file. I was in a similar situation to you a while back, where I had 3 GPUs, but only 1 beefy GPU could support 2-task-per-GPU. I had to just run 1-task-per-GPU, across the 3 GPUs, until I upgraded the GPUs again.

You *MIGHT* be able to do something strange, like setup GPU Exclusions and an app_info.xml file such that the beefy GPU runs one app at 2-tasks-per-GPU, while the non-beefy GPU runs another app at 1-task-per-GPU... but I don't recommend it, for 2 reasons. First, GPUGrid barely has tasks for 1 app, and it's likely a GPU would starve if you limited to a certain app type. Second, work fetch isn't really designed to handle this scenario. The client doesn't ask the host for work from a specific app type. Instead, work fetch treats all the GPUs as "NVIDIA" when fetching work, or deciding if work is needed.

Sorry you can't really do what you want. Like I said, I was the same way... until I upgraded my GPUs.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42784 - Posted: 9 Feb 2016 | 23:06:22 UTC - in response to Message 42783.

Thanks, Jacob. I had some of those same thoughts and figured it wouldn't work. Thanks for confirming my suspicion.

I'll likely have to go back to 1 at a time across all GPUs. That, or move my 750Ti cards to another project. How does a 970 compare to 2 750Tis on GPUGrid?

Jacob Klein
Send message
Joined: 11 Oct 08
Posts: 1127
Credit: 1,901,927,545
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42786 - Posted: 9 Feb 2016 | 23:09:36 UTC - in response to Message 42784.
Last modified: 9 Feb 2016 | 23:10:10 UTC

How does a 970 compare to 2 750Tis on GPUGrid?

Here's a great post, that describes GPU Grid performance (column 1), along with performance-per-watt (useful for those conserving electricity):
https://www.gpugrid.net/forum_thread.php?id=1150&nowrap=true#41294

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2343
Credit: 16,201,255,749
RAC: 7,520
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42787 - Posted: 10 Feb 2016 | 3:37:05 UTC - in response to Message 42775.

In support of this setup I've been looking at my app_config and trying to figure out how to adjust it such that the top card (980Ti/970) will run two tasks at once and the bottom card (750Ti) only run one task. Is this possible? I'm not seeing any tags on the BOINC Client Config guide that would apply to this.
I can imagine only one way to achieve this: to install two separate BOINC managers in separate program & data folders, then set up the 1st to run two workunits on the GPUs and exclude the lesser GPU; then set up the 2nd to exclude the bigger GPU. I've never done it, but theoretically it could work.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42788 - Posted: 10 Feb 2016 | 3:41:12 UTC - in response to Message 42787.

In support of this setup I've been looking at my app_config and trying to figure out how to adjust it such that the top card (980Ti/970) will run two tasks at once and the bottom card (750Ti) only run one task. Is this possible? I'm not seeing any tags on the BOINC Client Config guide that would apply to this.
I can imagine only one way to achieve this: to install two separate BOINC managers in separate program & data folders, then set up the 1st to run two workunits on the GPUs and exclude the lesser GPU; then set up the 2nd to exclude the bigger GPU. I've never done it, but theoretically it could work.


That's another thought I had. Probably more than I really want to deal with. I'll set it up for 1x initially and see how it goes. Thanks, everyone.

Matt
Avatar
Send message
Joined: 11 Jan 13
Posts: 216
Credit: 846,538,252
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 43217 - Posted: 17 Apr 2016 | 13:33:54 UTC

Just an update on this to illustrate the difference between PCIe 2.0 and 3.0 on this project.

I swapped out my 2 750Tis for 2 960s. One of the 960s is in this host and is placed in the PCIe 2.0 slot with a 980Ti in the top PCIe 3.0 slot. A typical BestUmbrella Long Run task looks like this.

My other 960 is in this host and is the only GPU and so is placed in the top PCIe 3.0 slot. Here is a BestUmbrella task from this host.

The two cards are identical (same vendor, model, clock, etc.) and yet the card in the PCIe 2.0 slot is taking around 50% longer to complete a task as the card in the PCIe 3.0 slot.

At this point, I am considering moving the card in the 2.0 slot up into the other available 3.0 slot in that machine. This means both it and the 980Ti will then run in PCIe 3.0 x8, but I'm guessing that it will still be faster overall than 980Ti in PCIe 3.0 x16 with the 960 in PCIe 2.0. The other option is to move it to my machine which currently has a 970 in it and have those two share the PCIe 3.0 lanes.

Post to thread

Message boards : Number crunching : PCIe 2.0 and GTX 970

//