[ofw] WinVerbs address translation required administrative privileges

[Hefty, Sean]

> Didn't we solve the device object open() problems in ibat a few years ago?
> Did we lose a fix?
>
> Thoughts?

Does anyone test OFED using a standard user account, or does everyone run as an administrator?

I may be able to look into this in more detail in a couple of weeks, but can't at the moment. I agree with Christoph that running as an administrator isn't viable, so this does need to be fixed.

- Sean

> > -----Ursprüngliche Nachricht-----
> > Von: Hefty, Sean [mailto:***@intel.com]
> > Gesendet: Freitag, 27. Januar 2012 19:18
> > An: Christoph Müller
> > Cc: ofw_list
> > Betreff: RE: Name resolution in WinVerbs/Multiple cards in one machine
>
>
> > > I fixed the network so far that this part seems to work again. I also
> > > looked into the implementation of TranslateAddress() in the meantime
> > > and have understood what it is doing. However, the problem with UAC
> > > remains. I need to run my program in elevated mode to work. Otherwise
> > > BindAddress() returns E_ACCESSDENIED. If I call TranslateAddress()
> > > directly, it does so, too. It seems that it cannot open the device object.
> On
> > the other hand, calling e. g.
> > > OpenDevice() directly works perfectly without elevation. It seems that
> > > in the former case, WinVerbs cannot open the device object for making
> > > the IOCTL calls for some reason. As the problem does not occur with
> > > all calls, I assume that I have a problem with the order of my API
> > > calls or so. Is there anything special to look for? I create a
> > > provider, a connect end point and then BindAddress(), which explodes.
> >
> > Btw - depending on the port number that you're trying to bind to, you may
> > need administrative privileges. It sounds like there may be a permission
> > issue with the 'ibat' (IB translation) file that is used to implement these
> calls.
> > Someone more familiar with ipoib would need to look into that however.
>
> can anyone help me with the problem described above? In the meantime, I tried
> several things (WinVerbs, libibverbs, librdma), but all comes back to the
> problem that I cannot bind/create an end point without elevation. I have also
> tried that on several W2k8R2 machines all having OFED 3.0. Sean mentioned (see
> above) a problem with the "ibat" file, but I do not really know what this is.
> I think I can also rule out the port number as I use 12345, which is well
> above 1024.
>
> To add some information: I use a standard HPC unattended installation as
> described in the OFED documentation, which I have deployed using clusrun. The
> Open Fabrics certificate is installed as trusted publisher on all nodes and
> the installation completed successfully. I have also tested an installation
> including SDK on a Windows 7 development machine, which behaves exactly the
> same wrt the elevation problem.
>
> Is there anything I can do about this? It is not really a viable solution for
> me running all IB-enabled programs as administrator.
>
> Best regards,
> Christoph
>
> PS: I also found that a problem with librdma creating only one connection
> successfully in rdma_get_request() mentioned in
> http://permalink.gmane.org/gmane.linux.drivers.rdma/6188 is still present in
> WinOFED v3.0 (GA) x64. Is that true?

[Hefty, Sean]

According to the initial bug report, the function below is what's failing. There's nothing obvious to me why it should fail without administrative permission. It sounds like the CreateFileW() call is failing.

TranslateAddress(const SOCKADDR* pAddress, WV_DEVICE_ADDRESS* pDeviceAddress)
{
HANDLE hIbat;
IOCTL_IBAT_IP_TO_PORT_IN addr;
IBAT_PORT_RECORD port;
DWORD bytes;
HRESULT hr;

hIbat = CreateFileW(IBAT_WIN32_NAME, GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (hIbat == INVALID_HANDLE_VALUE) {
return HRESULT_FROM_WIN32(GetLastError());
}

addr.Version = IBAT_IOCTL_VERSION;
if (pAddress->sa_family == AF_INET) {
addr.Address.IpVersion = 4;
RtlCopyMemory(addr.Address.Address + 12,
&((SOCKADDR_IN *)pAddress)->sin_addr, 4);
} else {
addr.Address.IpVersion = 6;
RtlCopyMemory(addr.Address.Address,
&((SOCKADDR_IN6 *)pAddress)->sin6_addr, 16);
}

if (DeviceIoControl(hIbat, IOCTL_IBAT_IP_TO_PORT,
&addr, sizeof addr, &port, sizeof port, &bytes, NULL)) {
hr = WV_SUCCESS;
pDeviceAddress->DeviceGuid = port.CaGuid;
pDeviceAddress->Pkey = port.PKey;
pDeviceAddress->PortNumber = port.PortNum;
} else {
hr = HRESULT_FROM_WIN32(GetLastError());
}

CloseHandle(hIbat);
return hr;
}

If the printip test is included in the OFED release, that could be run to see if the permission issue is specific to IBAT, rather than some interaction between winverbs and ibat. The following code in ipoib is where the ibat file gets created.

void
ipoib_ref_ibat()
{
UNICODE_STRING DeviceName;
UNICODE_STRING DeviceLinkUnicodeString;
NDIS_DEVICE_OBJECT_ATTRIBUTES DeviceObjectAttributes;
PDRIVER_DISPATCH DispatchTable[IRP_MJ_MAXIMUM_FUNCTION+1];

NDIS_STATUS Status = NDIS_STATUS_SUCCESS;

IPOIB_ENTER( IPOIB_DBG_IOCTL );

if( InterlockedIncrement( &g_ipoib.ibat_ref ) == 1 )
{

memset(DispatchTable, 0, (IRP_MJ_MAXIMUM_FUNCTION+1) * sizeof(PDRIVER_DISPATCH));

DispatchTable[IRP_MJ_CREATE] = __ipoib_create;
DispatchTable[IRP_MJ_CLEANUP] = __ipoib_cleanup;
DispatchTable[IRP_MJ_CLOSE] = __ipoib_close;
DispatchTable[IRP_MJ_DEVICE_CONTROL] = __ipoib_dispatch;
DispatchTable[IRP_MJ_INTERNAL_DEVICE_CONTROL] = __ipoib_dispatch;


NdisInitUnicodeString( &DeviceName, IBAT_DEV_NAME );
NdisInitUnicodeString( &DeviceLinkUnicodeString, IBAT_DOS_DEV_NAME );


memset(&DeviceObjectAttributes, 0, sizeof(NDIS_DEVICE_OBJECT_ATTRIBUTES));

DeviceObjectAttributes.Header.Type = NDIS_OBJECT_TYPE_DEFAULT; // type implicit from the context
DeviceObjectAttributes.Header.Revision = NDIS_DEVICE_OBJECT_ATTRIBUTES_REVISION_1;
DeviceObjectAttributes.Header.Size = sizeof(NDIS_DEVICE_OBJECT_ATTRIBUTES);
DeviceObjectAttributes.DeviceName = &DeviceName;
DeviceObjectAttributes.SymbolicName = &DeviceLinkUnicodeString;
DeviceObjectAttributes.MajorFunctions = &DispatchTable[0];
DeviceObjectAttributes.ExtensionSize = 0;
DeviceObjectAttributes.DefaultSDDLString = NULL;
DeviceObjectAttributes.DeviceClassGuid = 0;

Status = NdisRegisterDeviceEx(
g_IpoibMiniportDriverHandle,
&DeviceObjectAttributes,
&g_ipoib.h_ibat_dev,
&g_ipoib.h_ibat_dev_handle);



if( Status != NDIS_STATUS_SUCCESS )
{
IPOIB_PRINT( TRACE_LEVEL_ERROR, IPOIB_DBG_ERROR,
("NdisRegisterDeviceEx failed with status of %d\n", Status) );
}
}

IPOIB_EXIT( IPOIB_DBG_IOCTL );
}

Note that the default SDDL string is set to NULL. Is there a registry setting for ipoib (possibly inherited from NDIS or somewhere else) that gets used if the SDDL is NULL?

- Sean

[Christoph Müller]

Dear Sean,

just to be sure that I am using the same version. I have installed ofed_3-0_wlh_x64.msi, which has an SHA1 checksum of 58bf5dd31bdd6f15bed1bc2b4daf7aa9d85bcc5b. The README.txt is of [12-12-2011]. I must admit though that I did not install the package myself, but the MSI DB shows the correct version...

The sample code I use is from the SVN at svn://sofa.openfabrics.org/ofw/gen1/trunk. Is that the active code path?

As suggested, I have tried the printip tool. Interestingly, the one included with the OFED MSI package did not work:

.\PrintIP.exe lip 192.168.219.250
failed to open the kernel device 'ibat' hr=0x80070005
get_rdma_dev_IP_addrs() failed?


However, the one I built from the SVN source did:

.\PrintIP.exe lip 192.168.219.250
Found 1 IP active ports
0: ca guid 0x2c9030010d2a8 port guid 0x2c9030010d2a9 [192.168.219.250]


The one from the distribution does work, too, if I use an elevated shell. The output is the same as for the one I built myself.


I also tested the rdma_server sample, which does not work without elevation:

.\rdma_server.exe
rdma_server: start
rdma_create_ep 0
rdma_server: end -1

There also seems to be a problem with errno not being set as this is always 0 in case of an error. I can also observe this behaviour in my own program.


Additionally, I wanted to include the ACLs of the infamous "ibat file", but I could not find it. I see from the source that it is located at L"\\\\.\\ibat", but I do not know the current working directory and a recursive search on c:\ did not yield a result.

Best regards,
Christoph

> -----Ursprüngliche Nachricht-----
> Von: Hefty, Sean [mailto:***@intel.com]
> Gesendet: Mittwoch, 7. März 2012 23:08
> An: Tzachi Dar; Smith, Stan; Fab Tillier; Christoph Müller
> Cc: ***@lists.openfabrics.org
> Betreff: RE: [ofw] WinVerbs address translation required administrative
> privileges
>
> According to the initial bug report, the function below is what's failing.
> There's nothing obvious to me why it should fail without administrative
> permission. It sounds like the CreateFileW() call is failing.
>
> TranslateAddress(const SOCKADDR* pAddress, WV_DEVICE_ADDRESS*
> pDeviceAddress) {
> HANDLE hIbat;
> IOCTL_IBAT_IP_TO_PORT_IN addr;
> IBAT_PORT_RECORD port;
> DWORD bytes;
> HRESULT hr;
>
> hIbat = CreateFileW(IBAT_WIN32_NAME, GENERIC_READ |
> GENERIC_WRITE,
> FILE_SHARE_READ |
> FILE_SHARE_WRITE, NULL,
> OPEN_EXISTING,
> FILE_ATTRIBUTE_NORMAL, NULL);
> if (hIbat == INVALID_HANDLE_VALUE) {
> return HRESULT_FROM_WIN32(GetLastError());
> }
>
> addr.Version = IBAT_IOCTL_VERSION;
> if (pAddress->sa_family == AF_INET) {
> addr.Address.IpVersion = 4;
> RtlCopyMemory(addr.Address.Address + 12,
> &((SOCKADDR_IN *)pAddress)-
> >sin_addr, 4);
> } else {
> addr.Address.IpVersion = 6;
> RtlCopyMemory(addr.Address.Address,
> &((SOCKADDR_IN6 *)pAddress)-
> >sin6_addr, 16);
> }
>
> if (DeviceIoControl(hIbat, IOCTL_IBAT_IP_TO_PORT,
> &addr, sizeof addr, &port,
> sizeof port, &bytes, NULL)) {
> hr = WV_SUCCESS;
> pDeviceAddress->DeviceGuid = port.CaGuid;
> pDeviceAddress->Pkey = port.PKey;
> pDeviceAddress->PortNumber = port.PortNum;
> } else {
> hr = HRESULT_FROM_WIN32(GetLastError());
> }
>
> CloseHandle(hIbat);
> return hr;
> }
>
> If the printip test is included in the OFED release, that could be run to see if
> the permission issue is specific to IBAT, rather than some interaction
> between winverbs and ibat. The following code in ipoib is where the ibat file
> gets created.
>
> void
> ipoib_ref_ibat()
> {
> UNICODE_STRING DeviceName;
> UNICODE_STRING DeviceLinkUnicodeString;
> NDIS_DEVICE_OBJECT_ATTRIBUTES DeviceObjectAttributes;
> PDRIVER_DISPATCH DispatchTable[IRP_MJ_MAXIMUM_FUNCTION+1];
>
> NDIS_STATUS Status = NDIS_STATUS_SUCCESS;
>
> IPOIB_ENTER( IPOIB_DBG_IOCTL );
>
> if( InterlockedIncrement( &g_ipoib.ibat_ref ) == 1 )
> {
>
> memset(DispatchTable, 0,
> (IRP_MJ_MAXIMUM_FUNCTION+1) * sizeof(PDRIVER_DISPATCH));
>
> DispatchTable[IRP_MJ_CREATE]
> = __ipoib_create;
> DispatchTable[IRP_MJ_CLEANUP]
> = __ipoib_cleanup;
> DispatchTable[IRP_MJ_CLOSE]
> = __ipoib_close;
> DispatchTable[IRP_MJ_DEVICE_CONTROL]
> = __ipoib_dispatch;
> DispatchTable[IRP_MJ_INTERNAL_DEVICE_CONTROL]
> = __ipoib_dispatch;
>
>
> NdisInitUnicodeString( &DeviceName, IBAT_DEV_NAME );
> NdisInitUnicodeString( &DeviceLinkUnicodeString,
> IBAT_DOS_DEV_NAME );
>
>
> memset(&DeviceObjectAttributes, 0,
> sizeof(NDIS_DEVICE_OBJECT_ATTRIBUTES));
>
> DeviceObjectAttributes.Header.Type =
> NDIS_OBJECT_TYPE_DEFAULT; // type implicit from the context
> DeviceObjectAttributes.Header.Revision =
> NDIS_DEVICE_OBJECT_ATTRIBUTES_REVISION_1;
> DeviceObjectAttributes.Header.Size =
> sizeof(NDIS_DEVICE_OBJECT_ATTRIBUTES);
> DeviceObjectAttributes.DeviceName = &DeviceName;
> DeviceObjectAttributes.SymbolicName =
> &DeviceLinkUnicodeString;
> DeviceObjectAttributes.MajorFunctions = &DispatchTable[0];
> DeviceObjectAttributes.ExtensionSize = 0;
> DeviceObjectAttributes.DefaultSDDLString = NULL;
> DeviceObjectAttributes.DeviceClassGuid = 0;
>
> Status = NdisRegisterDeviceEx(
>
> g_IpoibMiniportDriverHandle,
>
> &DeviceObjectAttributes,
> &g_ipoib.h_ibat_dev,
>
> &g_ipoib.h_ibat_dev_handle);
>
>
>
> if( Status != NDIS_STATUS_SUCCESS )
> {
> IPOIB_PRINT( TRACE_LEVEL_ERROR,
> IPOIB_DBG_ERROR,
> ("NdisRegisterDeviceEx failed with status
> of %d\n", Status) );
> }
> }
>
> IPOIB_EXIT( IPOIB_DBG_IOCTL );
> }
>
> Note that the default SDDL string is set to NULL. Is there a registry setting for
> ipoib (possibly inherited from NDIS or somewhere else) that gets used if the
> SDDL is NULL?
>
> - Sean

[Christoph Müller]

Hi Sean,

as I had more urgent things to do, it took me some time doing your test. I now have upgraded my machines to 3.1 GA, which also took some time as it required a firmware upgrade, but the problem persists in exactly the same way. If I create an RDMA end point as user it fails, as administrator it works. I have also the problem that the RDMA functions do not return useful return codes (is this normal?), so everything is just a guess based on the driver code from SVN and playing around with it.

> -----Ursprüngliche Nachricht-----
> Von: Hefty, Sean [mailto:***@intel.com]
> Gesendet: Mittwoch, 7. März 2012 23:08
> An: Tzachi Dar; Smith, Stan; Fab Tillier; Christoph Müller
> Cc: ***@lists.openfabrics.org
> Betreff: RE: [ofw] WinVerbs address translation required administrative
> privileges
>
> According to the initial bug report, the function below is what's failing.
> There's nothing obvious to me why it should fail without administrative
> permission. It sounds like the CreateFileW() call is failing.
>
> TranslateAddress(const SOCKADDR* pAddress, WV_DEVICE_ADDRESS*

[...]

> If the printip test is included in the OFED release, that could be run to see if
> the permission issue is specific to IBAT, rather than some interaction
> between winverbs and ibat. The following code in ipoib is where the ibat file
> gets created.

printip is included and I have tested it.

[Hefty, Sean]

> as I had more urgent things to do, it took me some time doing your test. I now
> have upgraded my machines to 3.1 GA, which also took some time as it required a
> firmware upgrade, but the problem persists in exactly the same way. If I create
> an RDMA end point as user it fails, as administrator it works. I have also the
> problem that the RDMA functions do not return useful return codes (is this
> normal?), so everything is just a guess based on the driver code from SVN and
> playing around with it.

What is the return code that you're seeing?

> > Note that the default SDDL string is set to NULL. Is there a registry
> setting for
> > ipoib (possibly inherited from NDIS or somewhere else) that gets used if the
> > SDDL is NULL?
>
> Unfortunately, I do not have sufficient knowledge to understand what I should
> do with this information. If you have more detailed instruction on what to do,
> I have no problem trying it. I might also be able to able to attach a kernel
> debugger to the machine, but I would need some information on what I need to
> build and how to install my own build.

This comment was directed more at the other developers on the mailing list.

[Christoph Müller]

Hi Sean,

> -----Ursprüngliche Nachricht-----
> Von: Hefty, Sean [mailto:***@intel.com]
> Gesendet: Mittwoch, 18. Juli 2012 17:18
> An: Christoph Müller
> Cc: ***@lists.openfabrics.org
> Betreff: RE: [ofw] WinVerbs address translation required administrative
> privileges
>
> > as I had more urgent things to do, it took me some time doing your
> > test. I now have upgraded my machines to 3.1 GA, which also took some
> > time as it required a firmware upgrade, but the problem persists in
> > exactly the same way. If I create an RDMA end point as user it fails,
> > as administrator it works. I have also the problem that the RDMA
> > functions do not return useful return codes (is this normal?), so
> > everything is just a guess based on the driver code from SVN and playing
> around with it.
>
> What is the return code that you're seeing?

I get a -1 as espected for an error case, but errno is always 0. I have never seen any other value for errno than zero. Please not that I read errno immediately after the rdma_* call failed.

Best regards,
Christoph