unknown
1970-01-01 00:00:00 UTC
cd <top of OFED src tree>
#Following setup the build env
set _DDK=%systemDrive%\WInDDK\7600.16385.1
%_DDK%\bin\setenv.bat %_DDK% fre x64 WIN7 no_oacr
#following may not be required ?
Set PLATFORM_SDK_PATH=C:\PROGRA~1\MICROS~3\Windows\v6.1
Set ND_SDK_PATH=C:\PROGRA~1\MI1DC7C~1\NetDIrect
cd core\
build
cd ..\ulp\srp
build
obj/dll files in ..\..\bin\kernel\...
OFED 3.2 is the last OFED for Windows release; see Mellanox WinOF distribution - you may be able to build the SRP from their distribution?
Why the last, Microsoft has released NetworkDirect as their RDMA API; network vendors are moving towards supporting NetworkDirect which alleviates need for OFED for Windows.
Stan.
Thanks.
--- Todd
________________________________
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.
--_000_2DDE0F15AAFF91448E6F813C42EF4CBE7B695479ORSMSX105amrcor_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ofw-***@lists.openfabrics.org [mailto:ofw-***@lists.openfabrics.org]
<b>On Behalf Of </b>Todd Pisek<br>
<b>Sent:</b> Wednesday, October 02, 2013 10:34 AM<br>
<b>To:</b> ***@lists.openfabrics.org<br>
<b>Subject:</b> [ofw] Windows SRP perf question<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">Greetings,</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">I'm not new to Infiniband, but I am new to Windows.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">I'm running the 3.2 OFW release on Windows 2008R2 using a Dell R720 with dual sandy bridge 4-core processors. The HCA is a Mellanox ConnectX3-vpi. The application is a video stream simulator
running atop our file system which is a kernel file system filter driver. The experiment is accessing block storage using SRP.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">As the number of parallel video streams is increased, the bandwidth flattens. It appears the issue is saturation of one cpu that is processing all the SRP completion dpc's. Looking at the OFED
source, I believe the reason is that srp uses one sq/qp/rq per target. I believe this means that all response completion event dpc's will be queued to the cpu associated with the target's rq cq. </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">One solution I'd like to try is to put the requestor's cpu number in the MSB of the command tag and then use that information in __srp_process_session_recv_completion to obtain the cpu of
the initiator and then construct a dpc targeted to the initiating cpu to finish the completion processing (call __srp_process_recv_completion)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">I have 3 questions:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">1) Is my analysis correct, that is, are all srp responses from the same target processed by the same cpu ?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2) Does my solution seem reasonable?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3) How do I build the OFW code?<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> A couple of points along the way.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">SRP initiator has been without support for a couple of years.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Your analysis sounds reasonable for what little I know of SRP internals, although worth an experiment.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Download the Windows 7 WDK (Windows Driver Kit) 7600.16385.1 ; install WDK ( you should see C:\WDK\7600.16385.1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">
#Following setup the build env
set _DDK=%systemDrive%\WInDDK\7600.16385.1
%_DDK%\bin\setenv.bat %_DDK% fre x64 WIN7 no_oacr
#following may not be required ?
Set PLATFORM_SDK_PATH=C:\PROGRA~1\MICROS~3\Windows\v6.1
Set ND_SDK_PATH=C:\PROGRA~1\MI1DC7C~1\NetDIrect
cd core\
build
cd ..\ulp\srp
build
obj/dll files in ..\..\bin\kernel\...
OFED 3.2 is the last OFED for Windows release; see Mellanox WinOF distribution - you may be able to build the SRP from their distribution?
Why the last, Microsoft has released NetworkDirect as their RDMA API; network vendors are moving towards supporting NetworkDirect which alleviates need for OFED for Windows.
Stan.
Thanks.
--- Todd
________________________________
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.
--_000_2DDE0F15AAFF91448E6F813C42EF4CBE7B695479ORSMSX105amrcor_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ofw-***@lists.openfabrics.org [mailto:ofw-***@lists.openfabrics.org]
<b>On Behalf Of </b>Todd Pisek<br>
<b>Sent:</b> Wednesday, October 02, 2013 10:34 AM<br>
<b>To:</b> ***@lists.openfabrics.org<br>
<b>Subject:</b> [ofw] Windows SRP perf question<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">Greetings,</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">I'm not new to Infiniband, but I am new to Windows.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">I'm running the 3.2 OFW release on Windows 2008R2 using a Dell R720 with dual sandy bridge 4-core processors. The HCA is a Mellanox ConnectX3-vpi. The application is a video stream simulator
running atop our file system which is a kernel file system filter driver. The experiment is accessing block storage using SRP.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">As the number of parallel video streams is increased, the bandwidth flattens. It appears the issue is saturation of one cpu that is processing all the SRP completion dpc's. Looking at the OFED
source, I believe the reason is that srp uses one sq/qp/rq per target. I believe this means that all response completion event dpc's will be queued to the cpu associated with the target's rq cq. </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">One solution I'd like to try is to put the requestor's cpu number in the MSB of the command tag and then use that information in __srp_process_session_recv_completion to obtain the cpu of
the initiator and then construct a dpc targeted to the initiating cpu to finish the completion processing (call __srp_process_recv_completion)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">I have 3 questions:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">1) Is my analysis correct, that is, are all srp responses from the same target processed by the same cpu ?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2) Does my solution seem reasonable?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3) How do I build the OFW code?<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> A couple of points along the way.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">SRP initiator has been without support for a couple of years.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Your analysis sounds reasonable for what little I know of SRP internals, although worth an experiment.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Download the Windows 7 WDK (Windows Driver Kit) 7600.16385.1 ; install WDK ( you should see C:\WDK\7600.16385.1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">