Todd Pisek
2013-10-02 17:34:18 UTC
Greetings,
I'm not new to Infiniband, but I am new to Windows.
I'm running the 3.2 OFW release on Windows 2008R2 using a Dell R720 with dual sandy bridge 4-core processors. The HCA is a Mellanox ConnectX3-vpi. The application is a video stream simulator running atop our file system which is a kernel file system filter driver. The experiment is accessing block storage using SRP.
As the number of parallel video streams is increased, the bandwidth flattens. It appears the issue is saturation of one cpu that is processing all the SRP completion dpc's. Looking at the OFED source, I believe the reason is that srp uses one sq/qp/rq per target. I believe this means that all response completion event dpc's will be queued to the cpu associated with the target's rq cq.
One solution I'd like to try is to put the requestor's cpu number in the MSB of the command tag and then use that information in __srp_process_session_recv_completion to obtain the cpu of the initiator and then construct a dpc targeted to the initiating cpu to finish the completion processing (call __srp_process_recv_completion)
I have 3 questions:
1) Is my analysis correct, that is, are all srp responses from the same target processed by the same cpu ?
2) Does my solution seem reasonable?
3) How do I build the OFW code?
Thanks.
--- Todd
----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.
I'm not new to Infiniband, but I am new to Windows.
I'm running the 3.2 OFW release on Windows 2008R2 using a Dell R720 with dual sandy bridge 4-core processors. The HCA is a Mellanox ConnectX3-vpi. The application is a video stream simulator running atop our file system which is a kernel file system filter driver. The experiment is accessing block storage using SRP.
As the number of parallel video streams is increased, the bandwidth flattens. It appears the issue is saturation of one cpu that is processing all the SRP completion dpc's. Looking at the OFED source, I believe the reason is that srp uses one sq/qp/rq per target. I believe this means that all response completion event dpc's will be queued to the cpu associated with the target's rq cq.
One solution I'd like to try is to put the requestor's cpu number in the MSB of the command tag and then use that information in __srp_process_session_recv_completion to obtain the cpu of the initiator and then construct a dpc targeted to the initiating cpu to finish the completion processing (call __srp_process_recv_completion)
I have 3 questions:
1) Is my analysis correct, that is, are all srp responses from the same target processed by the same cpu ?
2) Does my solution seem reasonable?
3) How do I build the OFW code?
Thanks.
--- Todd
----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.