64bit RPC traffic fails across ISA Sever 2006

[LEFT][CODE]https://blogs.technet.com/isablog/archive/2008/07/21/64-bit-rpc-traffic-fails-across-isa-sever-2006.aspx[/CODE]

[B][SIZE=3][FONT=Calibri]1. Introduction[/FONT][/SIZE][/B]
[FONT=Calibri][SIZE=3] [/SIZE][/FONT]
[SIZE=3][FONT=Calibri]This post describes an issue where two 64-bit Windows hosts are failing to communicate to each other using RPC . The hosts each operate in a network physically separated from each other by ISA Server 2006. Figure 1 illustrates the basic scenario.[/FONT][/SIZE]

[IMG]http://blogs.technet.com/photos/isablog/images/3092059/original.aspx[/IMG]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]Figure 1 – Sample network diagram.[/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri]All other traffic allowed between these hosts functioned normally; only RPC calls were failing. [/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri][B]2. Identifying the problem[/B][/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri]The traffic across the networks was working without problems for most of the protocols (ICMP, SMB, DNS, HTTP, etc). Only RPC Calls were failing and the actual RPC error was exposed by using the [I]repadmin /showreps [/I]command when run from one DC. We got the error message below
[/FONT][/SIZE][CODE][SIZE=3][FONT=Calibri][I]The replication generated an error (1727): The remote procedure call failed and did not execute.[/I][/FONT][/SIZE][/CODE]
[SIZE=3][FONT=Calibri]We used [URL="http://support.microsoft.com/kb/911799/en-us"][COLOR=#800080]KB911799[/COLOR][/URL] (method 1) approach to understand where was failing and if ISA Server 2006 was really causing that. The tests showed that the RPC communication was failing only for communications between 64 bit platforms (Windows Server 2008 64 with Windows Server 2008 64, Windows Vista 64 with Windows Server 2008 64bit). [/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri]Looking to the network monitor trace that was taken from the DC (10.30.30.10) it was possible to see the moment of the failure when the DC from one side is trying to bind to the RPC in the other side:[/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri][B]TCP 3 Way Handshake for RPC
[/B][/FONT][/SIZE][CODE]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.30.30.10 10.40.40.16 TCP TCP:Flags=......S., SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102417964, Ack=0, Win=8192 (scale factor 8) = 2097152[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.40.40.16 10.30.30.10 TCP TCP:Flags=...A..S., SrcPort=DCE endpoint resolution(135), DstPort=36457, PayloadLen=0, Seq=2217804385, Ack=102417965, Win=16384 (scale factor 0) = 16384[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.30.30.10 10.40.40.16 TCP TCP:Flags=...A...., SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102417965, Ack=2217804386, Win=257 (scale factor 8) = 65792[/FONT][/SIZE][/FONT]
[/CODE]
[SIZE=3][FONT=Calibri][B]RPC Bind Request for the End Point Mapper (End Point Mapper's UUID is E1AF8308-5D1F-11C9-91A4-08002B14A0FA):[/B][/FONT][/SIZE]
[CODE]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.30.30.10 10.40.40.16 MSRPC MSRPC:c/o Bind: UUID{E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT Call=0x1 Assoc Grp=0x0 Xmit=0x16D0 Recv=0x16D0[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri]- RPC: c/o Bind: UUID{E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT Call=0x1 Assoc Grp=0x0 Xmit=0x16D0 Recv=0x16D0 [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]- Bind: {E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]RpcVers: 5 (0x5)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]RpcVersMinor: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]PType: 0x0B - Bind[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PfcFlags: 3 (0x3)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PackedDrep: 0x10[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]FragLength: 160 (0xA0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]AuthLength: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]CallId: 1 (0x1)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]MaxXmitFrag: 5840 (0x16D0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]MaxRecvFrag: 5840 (0x16D0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=3][FONT=Calibri]AssocGroupId: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]- PContextElem: [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]NContextElem: 3 (0x3)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=3][FONT=Calibri]Reserved: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]Reserved2: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PContElem: 0x1[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]- PContElem: 0x1[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]PContId: 1 (0x1)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]NTransferSyn: 1 (0x1)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]Reserved: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ AbstractSyntax: {E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT[/FONT][/SIZE][/FONT]
[SIZE=3][FONT=Calibri][B][COLOR=red][FONT=Courier New] + TransferSyntaxes: {71710533-BEBA-4937-8319-B5DBEF9CCC36} NDR64[/FONT][/COLOR][/B][/FONT][/SIZE]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PContElem: 0x1[/FONT][/SIZE][/FONT]
[/CODE]
[SIZE=3][FONT=Calibri][B]RPC Bind Response:[/B][/FONT][/SIZE]
[CODE]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.40.40.16 10.30.30.10 MSRPC MSRPC:c/o Bind Ack: Call=0x1 Assoc Grp=0x87F3 [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri]- RPC: c/o Bind Ack: Call=0x1 Assoc Grp=0x87F3 Xmit=0x16D0 Recv=0x16D0 [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]- BindAck: [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]RpcVers: 5 (0x5)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]RpcVersMinor: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]PType: 0x0C - Bind Ack[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PfcFlags: 3 (0x3)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PackedDrep: 0x10[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]FragLength: 108 (0x6C)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]AuthLength: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]CallId: 1 (0x1)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]MaxXmitFrag: 5840 (0x16D0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]MaxRecvFrag: 5840 (0x16D0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]AssocGroupId: 34803 (0x87F3)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ SecAddr: 135[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ Pad2: 0x1[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]- PResultList: [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]NResults: 3 (0x3)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]Reserved: 0 (0x0)[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]Reserved2: 0 (0x0)[/FONT][/SIZE][/FONT]
[SIZE=3][FONT=Calibri][B][COLOR=red][FONT=Courier New] - PResults: Provider rejection, Reason=Proposed transfer syntaxes not supported[/FONT][/COLOR][/B][/FONT][/SIZE]
[SIZE=3][FONT=Calibri][B][COLOR=red][FONT=Courier New] Result: Provider rejection[/FONT][/COLOR][/B][/FONT][/SIZE]
[SIZE=3][FONT=Calibri][B][COLOR=red][FONT=Courier New] Reason: Proposed transfer syntaxes not supported[/FONT][/COLOR][/B][/FONT][/SIZE]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ TransferSyntax: {00000000-0000-0000-0000-000000000000} unknown[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PResults: Acceptance, Reason=n/a[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][SIZE=3][FONT=Calibri]+ PResults: Negotiate Ack, Security Context Multiplexing Supported[/FONT][/SIZE][/FONT]
[/CODE]
[SIZE=3][FONT=Calibri][B]The DC (10.30.30.10) sends an endpoint request for [/B][URL="http://msdn.microsoft.com/en-us/library/cc205571.aspx"][B]NTFRS Service[/B][/URL][B] UUID: f5cc59b4-4264-101a-8c59-08002b2f8426…[/B][/FONT][/SIZE]
[CODE]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.30.30.10 10.40.40.16 EPM EPM:Request: ept_map: NDR, FrsRpc {F5CC59B4-4264-101A-8C59-08002B2F8426} v1.1, RPC v5, 0.0.0.0:135 (0x87) [DCE endpoint resolution(135)][/FONT][/SIZE][/FONT]
[/CODE]
[SIZE=3][FONT=Calibri][B]..but the DC 10.40.40.16 closes the connection[/B][/FONT][/SIZE]
[CODE]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.40.40.16 10.30.30.10 TCP TCP:Flags=...A...F, SrcPort=DCE endpoint resolution(135), DstPort=36457, PayloadLen=0, Seq=2217804494, Ack=102418293, Win=65207 (scale factor 0) = 65207[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.30.30.10 10.40.40.16 TCP TCP:Flags=...A...., SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102418293, Ack=2217804495, Win=257 (scale factor 8) = 65792[/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=3][FONT=Calibri]10.30.30.10 10.40.40.16 TCP TCP:Flags=...A...F, SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102418293, Ack=2217804495, Win=257 (scale factor 8) = 65792[/FONT][/SIZE][/FONT]
[/CODE]
[SIZE=3][FONT=Calibri]The key element in the above trace is the [I]Provider Results[/I], that appears as [I][COLOR=red]Provider rejection, Reason=Proposed transfer syntaxes not supported[/COLOR].[/I] Briefly this means that the DC 10.40.40.16 appeared to reject the bind request from DC 10.30.30.10 because the proposed transfer syntax is no supported. But this was not actually the DC 10.40.40.16 that sent that, that was ISA Server, let's understand why.[/FONT][/SIZE]
[CODE][SIZE=3][FONT=Calibri][B]Note1:[/B] for a complete list of the meaning of each rejection reasons review Section 2 of the [URL="http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=24364"]ISO 8823 standard[/URL].[/FONT][/SIZE][/CODE]
[SIZE=3][FONT=Calibri][B]3. RPC Filter[/B][/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri]If you review the [URL="http://technet2.microsoft.com/WindowsServer/en/library/4dbc4c95-935b-4617-b4f8-20fc947c72881033.mspx?mfr=true"][COLOR=#800080]RPC Architecture[/COLOR][/URL] you will notice that there is a component that belongs to the rpcrt4.dll called [I]Marshalling Engine[/I]. This component is responsible for providing a common RPC interface between RPC clients and servers through NDR (Network Data Representation). There are two transfer syntaxes variations for the NDR:[/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[FONT=Symbol][SIZE=3][FONT=Calibri]·[FONT=Times New Roman] [/FONT][/FONT][/SIZE][/FONT][SIZE=3][FONT=Calibri][URL="http://msdn.microsoft.com/en-us/library/cc238901.aspx"][COLOR=#800080]NDR20[/COLOR][/URL] – Used in 32-bit Architecture.[/FONT][/SIZE]
[FONT=Symbol][SIZE=3][FONT=Calibri]·[FONT=Times New Roman] [/FONT][/FONT][/SIZE][/FONT][SIZE=3][FONT=Calibri][URL="http://msdn.microsoft.com/en-us/library/cc232141.aspx"][COLOR=#800080]NDR64[/COLOR][/URL] – Used in 64-bit Architecture.[/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri]When two 64bit Windows Operating System are communicating using RPC they will negotiate the marshalling engine to use. Since they both prefer NDR64, this is likely to be the format used. Natively, ISA Server 2006 RPC Filter doesn’t support NDR64; therefore the RPC Filter will reject any RPC communication which uses NDR64. [/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri][B]4. Resolution[/B][/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri]To fix the problem you should install [URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;943462"]ISA Server 2006 SP1[/URL].[/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri] [/FONT][/SIZE]
[SIZE=3][FONT=Calibri][B][U]Author[/U][/B][/FONT][/SIZE]
[SIZE=3][FONT=Calibri]Yuri Diogenes[/FONT][/SIZE][FONT=Times New Roman][/FONT]
[SIZE=3][FONT=Calibri]Security Support Engineer – Microsoft CSS Forefront (ISA/TMG) Team[/FONT][/SIZE][FONT=Times New Roman][/FONT]
[FONT=Times New Roman][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT]
[SIZE=3][FONT=Calibri][B][U]Technical Reviewers[/U][/B][/FONT][/SIZE]
[SIZE=3][FONT=Calibri]Jim Harrison[/FONT][/SIZE][FONT=Times New Roman][/FONT]
[SIZE=3][FONT=Calibri]Microsoft Forefront (ISA/TMG) Sustained Engineering Team[/FONT][/SIZE][FONT=Times New Roman][/FONT]
[FONT=Times New Roman][SIZE=3][FONT=Calibri] [/FONT][/SIZE][/FONT]
[SIZE=3][FONT=Calibri]Doron Juster[/FONT][/SIZE][FONT=Times New Roman][/FONT]
[SIZE=3][FONT=Calibri]Microsoft Forefront (ISA/TMG) Sustained Engineering Team[/FONT][/SIZE]


[/LEFT]