[ofw] Windows IB APIs

Discussion:

William Deck

2014-01-29 16:52:54 UTC

Hello All,

I am sure this question has been asked before however when I searched the
archives I could not find anything.

I am working on a Windows program that must connect to both Windows and Linux
machines across Infiniband. While first doing research I was told to use
NetworkDirect on Windows and rdma_cm/verbs on Linux to connect the two
systems. On initial tests I could not get the two systems to connect. Then
when I asked around I was told to use ib verbs on both the Windows and Linux.
So what I really want to know is what is the best API to use on Windows to
connect to a Linux system.

Thank you.

--
Will

Hefty, Sean

2014-01-29 17:12:58 UTC

Permalink

NetworkDirect is the only supported (i.e. Microsoft mandated) API on Windows. Anything beyond that requires talking with your Windows HCA vendor.

-----Original Message-----
Sent: Wednesday, January 29, 2014 8:53 AM
Subject: [ofw] Windows IB APIs
Hello All,
I am sure this question has been asked before however when I searched the
archives I could not find anything.
I am working on a Windows program that must connect to both Windows and Linux
machines across Infiniband. While first doing research I was told to use
NetworkDirect on Windows and rdma_cm/verbs on Linux to connect the two
systems. On initial tests I could not get the two systems to connect.
Then
when I asked around I was told to use ib verbs on both the Windows and Linux.
So what I really want to know is what is the best API to use on Windows to
connect to a Linux system.
Thank you.
--
Will
_______________________________________________
ofw mailing list
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw

Deck, William

2014-01-29 17:30:31 UTC

Permalink

I understand that NetworkDirect is the Microsoft supported API. But does that API allow connections to Linux or does that depend on the HCA Vendor?

-----Original Message-----
From: Hefty, Sean [mailto:***@intel.com]
Sent: Wednesday, January 29, 2014 12:13 PM
To: Deck, William; ***@lists.openfabrics.org
Subject: RE: [ofw] Windows IB APIs

NetworkDirect is the only supported (i.e. Microsoft mandated) API on Windows. Anything beyond that requires talking with your Windows HCA vendor.

Hefty, Sean

2014-01-29 17:33:19 UTC

Permalink

Post by Deck, William
I understand that NetworkDirect is the Microsoft supported API. But does
that API allow connections to Linux or does that depend on the HCA Vendor?

AFAIK, it depends on the HCA vendor implementation of that interface. But, in theory, it should work with an application written to verbs/rdmacm on Linux.

Deck, William

2014-01-29 17:43:40 UTC

Permalink

Thank you for that answer.

I noticed there were some Mellanox emails on the list and this question is for them. Does the Mellanox implementation of NetworkDirect allow connections from rdmacm/ib verbs? Or is there a specific Mellanox Windows API to use to connect to Linux?

Thanks.

Fab Tillier

2014-01-29 22:33:00 UTC

Permalink

Hefty, Sean wrote on Wed, 29 Jan 2014 at 09:33:19

Post by Hefty, Sean

Post by Deck, William
I understand that NetworkDirect is the Microsoft supported API. But
does that API allow connections to Linux or does that depend on the
HCA Vendor?

AFAIK, it depends on the HCA vendor implementation of that interface.
But, in theory, it should work with an application written to
verbs/rdmacm on Linux.

I'll make a stronger statement here - if interop between Windows (via Network Direct) and Linux (via verbs/rdmacm) does not work, then there is a driver bug somewhere. You should be able to establish connections between a Windows Network Direct application and a Linux verbs/rdmacm application.

-Fab

Hefty, Sean

2014-01-29 22:41:55 UTC

Permalink

Post by Fab Tillier
I'll make a stronger statement here - if interop between Windows (via
Network Direct) and Linux (via verbs/rdmacm) does not work, then there is a
driver bug somewhere. You should be able to establish connections between
a Windows Network Direct application and a Linux verbs/rdmacm application.

If ND does not mandate a specific wire protocol, then interoperability is left to the implementation of the provider.

Tziporet Koren

2014-01-30 05:49:05 UTC

Permalink

Mellanox ND is interoperable with Linux RDMACM.
We also have a test that check this.

Dotan - can you share the test we use to ensure this?

I agree with Fab - if something is not working - there is a bug somewhere

Tziporet

-----Original Message-----
From: ofw-***@lists.openfabrics.org [mailto:ofw-***@lists.openfabrics.org] On Behalf Of Hefty, Sean
Sent: Thursday, January 30, 2014 12:42 AM
To: FabTillier; Deck, William; ***@lists.openfabrics.org
Subject: Re: [ofw] Windows IB APIs

Post by Fab Tillier
I'll make a stronger statement here - if interop between Windows (via
Network Direct) and Linux (via verbs/rdmacm) does not work, then there
is a driver bug somewhere. You should be able to establish
connections between a Windows Network Direct application and a Linux verbs/rdmacm application.

If ND does not mandate a specific wire protocol, then interoperability is left to the implementation of the provider.

Deck, William

2014-01-30 21:05:32 UTC

Permalink

Using a modified HPC 2012 NDv2 sample program and a Linux rdmacm test program I have successfully opened a connection and done some I/O. I will be working on getting RDMAs to work and I will be sure to mention if I see any issues. Thank you everyone for your help.

--
Will

-----Original Message-----
From: Tziporet Koren [mailto:***@mellanox.com]
Sent: Thursday, January 30, 2014 12:49 AM
To: Hefty, Sean; FabTillier; Deck, William; ***@lists.openfabrics.org; Dotan Barak
Subject: RE: [ofw] Windows IB APIs

Mellanox ND is interoperable with Linux RDMACM.
We also have a test that check this.

Dotan - can you share the test we use to ensure this?

I agree with Fab - if something is not working - there is a bug somewhere

Tziporet

-----Original Message-----
From: ofw-***@lists.openfabrics.org [mailto:ofw-***@lists.openfabrics.org] On Behalf Of Hefty, Sean
Sent: Thursday, January 30, 2014 12:42 AM
To: FabTillier; Deck, William; ***@lists.openfabrics.org
Subject: Re: [ofw] Windows IB APIs

Post by Fab Tillier
I'll make a stronger statement here - if interop between Windows (via
Network Direct) and Linux (via verbs/rdmacm) does not work, then there
is a driver bug somewhere. You should be able to establish
connections between a Windows Network Direct application and a Linux verbs/rdmacm application.

If ND does not mandate a specific wire protocol, then interoperability is left to the implementation of the provider.

Tziporet Koren

2014-02-02 09:54:29 UTC

Permalink

Thanks for the update.

BTW - which version of WinOF are you using?

Tziporet

-----Original Message-----
From: Deck, William [mailto:***@unisys.com]
Sent: Thursday, January 30, 2014 11:06 PM
To: Tziporet Koren; Hefty, Sean; FabTillier; ***@lists.openfabrics.org; Dotan Barak
Subject: RE: [ofw] Windows IB APIs

Using a modified HPC 2012 NDv2 sample program and a Linux rdmacm test program I have successfully opened a connection and done some I/O. I will be working on getting RDMAs to work and I will be sure to mention if I see any issues. Thank you everyone for your help.

--
Will

-----Original Message-----
From: Tziporet Koren [mailto:***@mellanox.com]
Sent: Thursday, January 30, 2014 12:49 AM
To: Hefty, Sean; FabTillier; Deck, William; ***@lists.openfabrics.org; Dotan Barak
Subject: RE: [ofw] Windows IB APIs

Mellanox ND is interoperable with Linux RDMACM.
We also have a test that check this.

Dotan - can you share the test we use to ensure this?

I agree with Fab - if something is not working - there is a bug somewhere

Tziporet

-----Original Message-----
From: ofw-***@lists.openfabrics.org [mailto:ofw-***@lists.openfabrics.org] On Behalf Of Hefty, Sean
Sent: Thursday, January 30, 2014 12:42 AM
To: FabTillier; Deck, William; ***@lists.openfabrics.org
Subject: Re: [ofw] Windows IB APIs

Post by Fab Tillier
I'll make a stronger statement here - if interop between Windows (via
Network Direct) and Linux (via verbs/rdmacm) does not work, then there
is a driver bug somewhere. You should be able to establish
connections between a Windows Network Direct application and a Linux verbs/rdmacm application.

If ND does not mandate a specific wire protocol, then interoperability is left to the implementation of the provider.

Deck, William

2014-02-03 16:06:16 UTC

Permalink

I am using 4.42 WinOF.

--
Will

-----Original Message-----
From: Tziporet Koren [mailto:***@mellanox.com]
Sent: Sunday, February 02, 2014 4:54 AM
To: Deck, William; Hefty, Sean; FabTillier; ***@lists.openfabrics.org; Dotan Barak
Subject: RE: [ofw] Windows IB APIs

Thanks for the update.

BTW - which version of WinOF are you using?

Tziporet

-----Original Message-----
From: Deck, William [mailto:***@unisys.com]
Sent: Thursday, January 30, 2014 11:06 PM
To: Tziporet Koren; Hefty, Sean; FabTillier; ***@lists.openfabrics.org; Dotan Barak
Subject: RE: [ofw] Windows IB APIs

Using a modified HPC 2012 NDv2 sample program and a Linux rdmacm test program I have successfully opened a connection and done some I/O. I will be working on getting RDMAs to work and I will be sure to mention if I see any issues. Thank you everyone for your help.

--
Will

-----Original Message-----
From: Tziporet Koren [mailto:***@mellanox.com]
Sent: Thursday, January 30, 2014 12:49 AM
To: Hefty, Sean; FabTillier; Deck, William; ***@lists.openfabrics.org; Dotan Barak
Subject: RE: [ofw] Windows IB APIs

Mellanox ND is interoperable with Linux RDMACM.
We also have a test that check this.

Dotan - can you share the test we use to ensure this?

I agree with Fab - if something is not working - there is a bug somewhere

Tziporet

-----Original Message-----
From: ofw-***@lists.openfabrics.org [mailto:ofw-***@lists.openfabrics.org] On Behalf Of Hefty, Sean
Sent: Thursday, January 30, 2014 12:42 AM
To: FabTillier; Deck, William; ***@lists.openfabrics.org
Subject: Re: [ofw] Windows IB APIs

Post by Fab Tillier
I'll make a stronger statement here - if interop between Windows (via
Network Direct) and Linux (via verbs/rdmacm) does not work, then there
is a driver bug somewhere. You should be able to establish
connections between a Windows Network Direct application and a Linux verbs/rdmacm application.

If ND does not mandate a specific wire protocol, then interoperability is left to the implementation of the provider.