Storage Maintenance (StorM) Working Group                    Michael Ko
Internet Draft                                               Consultant
Intended status: Proposed Standard                  Alexander Nezhinsky
Expires: January 2014                                          Mellanox Engineering Task Force (IETF)                             M. Ko
Request for Comments: 7145
Obsoletes: 5046                                            July 9, 2013

                  iSCSI                                             A. Nezhinsky
Category: Standards Track                                       Mellanox
ISSN: 2070-1721                                               April 2014

      Internet Small Computer System Interface (iSCSI) Extensions
        for RDMA Specification
                       draft-ietf-storm-iser-15.txt

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with
   the provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time.  It is inappropriate to use Internet-Drafts as
   reference material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on January, 2014. Remote Direct Memory Access (RDMA) Specification

Abstract

   iSCSI

   Internet Small Computer System Interface (iSCSI) Extensions for
   Remote Direct Memory Access (RDMA) provides the RDMA data transfer
   capability to iSCSI by layering iSCSI on top of an RDMA-Capable
   Protocol.  An RDMA-Capable Protocol provides RDMA Read and Write
   services, which enable data to be transferred directly into SCSI I/O
   Buffers without intermediate data copies.  This document describes
   the extensions to the iSCSI protocol to support RDMA services as
   provided by an RDMA-Capable Protocol.

   This document obsoletes RFC 5046.

   Table

Status of This Memo

   This is an Internet Standards Track document.

   This document is a product of Contents

   1    Definitions and Acronyms ....................................6
   1.1  Definitions .................................................6
   1.2  Acronyms ...................................................12
   1.3  Conventions ................................................14
   2    Introduction ...............................................15
   2.1  Motivation .................................................15
   2.2  iSCSI/iSER Layering ........................................16
   2.3  Architectural Goals ........................................17
   2.4  Protocol Overview ..........................................17
   2.5  RDMA services and iSER .....................................19
   2.5.1  STag......................................................19
   2.5.2  Send......................................................20
   2.5.3  RDMA Write................................................21
   2.5.4  RDMA Read.................................................21
   2.6  SCSI Read Overview .........................................21
   2.7  SCSI Write Overview ........................................22
   3    Upper Layer Interface Requirements .........................23
   3.1  Operational Primitives offered by iSER .....................23
   3.1.1  Send_Control..............................................24
   3.1.2  Put_Data..................................................24
   3.1.3  Get_Data..................................................24
   3.1.4  Allocate_Connection_Resources.............................25
   3.1.5  Deallocate_Connection_Resources...........................25
   3.1.6  Enable_Datamover..........................................25
   3.1.7  Connection_Terminate......................................26
   3.1.8  Notice_Key_Values.........................................26
   3.1.9  Deallocate_Task_Resources.................................26
   3.2  Operational Primitives used by iSER ........................27
   3.2.1  Control_Notify............................................27
   3.2.2  Data_Completion_Notify....................................27
   3.2.3  Data_ACK_Notify...........................................28
   3.2.4  Connection_Terminate_Notify...............................28
   3.3  iSCSI Protocol Usage Requirements ..........................28
   4    Lower Layer Interface Requirements .........................30
   4.1  Interactions with the RCaP Layer ...........................30
   4.2  Interactions with Internet Engineering Task Force
   (IETF).  It represents the Transport Layer ......................31
   5    Connection Setup consensus of the IETF community.  It has
   received public review and Termination ...........................32
   5.1  iSCSI/iSER Connection Setup ................................32
   5.1.1  Initiator Behavior........................................33
   5.1.2  Target Behavior...........................................35
   5.1.3  iSER Hello Exchange.......................................36
   5.2  iSCSI/iSER Connection Termination ..........................39
   5.2.1  Normal Connection Termination at has been approved for publication by the Initiator............39
   5.2.2  Normal Connection Termination at
   Internet Engineering Steering Group (IESG).  Further information on
   Internet Standards is available in Section 2 of RFC 5741.

   Information about the Target...............40
   5.2.3  Termination without current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   http://www.rfc-editor.org/info/rfc7145.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1. Introduction ....................................................5
      1.1. Motivation .................................................5
      1.2. iSCSI/iSER Layering ........................................6
      1.3. Architectural Goals ........................................7
      1.4. Protocol Overview ..........................................7
      1.5. RDMA Services and iSER .....................................9
           1.5.1. STag ................................................9
           1.5.2. Send ...............................................10
           1.5.3. RDMA Write .........................................11
           1.5.4. RDMA Read ..........................................11
      1.6. SCSI Read Overview ........................................11
      1.7. SCSI Write Overview .......................................12
   2. Definitions and Acronyms .......................................12
      2.1. Definitions ...............................................12
      2.2. Acronyms ..................................................18
      2.3. Conventions ...............................................20
   3. Upper-Layer Interface Requirements .............................20
      3.1. Operational Primitives offered by iSER ....................21
           3.1.1. Send_Control .......................................21
           3.1.2. Put_Data ...........................................21
           3.1.3. Get_Data ...........................................22
           3.1.4. Allocate_Connection_Resources ......................22
           3.1.5. Deallocate_Connection_Resources ....................23
           3.1.6. Enable_Datamover ...................................23
           3.1.7. Connection_Terminate ...............................23
           3.1.8. Notice_Key_Values ..................................24
           3.1.9. Deallocate_Task_Resources ..........................24
      3.2. Operational Primitives Used by iSER .......................24
           3.2.1. Control_Notify .....................................25
           3.2.2. Data_Completion_Notify .............................25
           3.2.3. Data_ACK_Notify ....................................25
           3.2.4. Connection_Terminate_Notify ........................26
      3.3. iSCSI Protocol Usage Requirements .........................26
   4. Lower-Layer Interface Requirements .............................27
      4.1. Interactions with the RCaP Layer ..........................27
      4.2. Interactions with the Transport Layer .....................28
   5. Connection Setup and Termination ...............................28
      5.1. iSCSI/iSER Connection Setup ...............................28
           5.1.1. Initiator Behavior .................................30
           5.1.2. Target Behavior ....................................31
           5.1.3. iSER Hello Exchange ................................33
      5.2. iSCSI/iSER Connection Termination .........................36
           5.2.1. Normal Connection Termination at the Initiator .....36
           5.2.2. Normal Connection Termination at the Target ........36
           5.2.3. Termination without Logout Request/Response PDUs..........40
   6 PDUs ...37
   6. Login/Text Operational Keys ................................42
   6.1 ....................................38
      6.1. HeaderDigest and DataDigest ................................42
   6.2 ...............................38
      6.2. MaxRecvDataSegmentLength ...................................42
   6.3 ..................................38
      6.3. RDMAExtensions .............................................43
   6.4 ............................................39
      6.4. TargetRecvDataSegmentLength ................................44
   6.5 ...............................40
      6.5. InitiatorRecvDataSegmentLength .............................44
   6.6 ............................41
      6.6. OFMarker and IFMarker ......................................45
   6.7 .....................................41
      6.7. MaxOutstandingUnexpectedPDUs ...............................45
   6.8 ..............................41
      6.8. MaxAHSLength ...............................................46
   6.9 ..............................................42
      6.9. TaggedBufferForSolicitedDataOnly ...........................46
   6.10   iSERHelloRequired.........................................47
   7 ..........................43
      6.10. iSERHelloRequired ........................................43
   7. iSCSI PDU Considerations ...................................48
   7.1 .......................................44
      7.1. iSCSI Data-Type PDU ........................................48
   7.2 .......................................44
      7.2. iSCSI Control-Type PDU .....................................49
   7.3 ....................................45
      7.3. iSCSI PDUs .................................................49
   7.3.1 ................................................45
           7.3.1. SCSI Command..............................................49
   7.3.2 Command .......................................45
           7.3.2. SCSI Response.............................................51
   7.3.3 Response ......................................47
           7.3.3. Task Management Function Request/Response.................53
   7.3.4 Request/Response ..........49
           7.3.4. SCSI Data-out.............................................54
   7.3.5 Data-out ......................................50
           7.3.5. SCSI Data-in..............................................55
   7.3.6 Data-in .......................................51
           7.3.6. Ready To Transfer (R2T)...................................57
   7.3.7 (R2T) ............................53
           7.3.7. Asynchronous Message......................................59
   7.3.8 Message ...............................55
           7.3.8. Text Request & and Text Response..............................59
   7.3.9 Response .....................55
           7.3.9. Login Request & and Login Response............................60
   7.3.10 Response ...................55
           7.3.10. Logout Request & and Logout Response ........................60
   7.3.11 ................56
           7.3.11. SNACK Request ...........................................60
   7.3.12 .....................................56
           7.3.12. Reject ..................................................60
   7.3.13 ............................................56
           7.3.13. NOP-Out & and NOP-In ........................................61
   8 ................................57
   8. Flow Control and STag Management ...........................62
   8.1 ...............................57
      8.1. Flow Control for RDMA Send Messages ........................62
   8.1.1 .......................57
           8.1.1. Flow Control for Control-Type PDUs from the Initiator.....62
   8.1.2
                  Initiator ..........................................58
           8.1.2. Flow Control for Control-Type PDUs from the Target........65
   8.2
                  Target .............................................60
      8.2. Flow Control for RDMA Read Resources .......................66
   8.3 ......................61
      8.3. STag Management ............................................67
   8.3.1 ...........................................62
           8.3.1. Allocation of STags.......................................67
   8.3.2 STags ................................62
           8.3.2. Invalidation of STags.....................................67
   9 STags ..............................62
   9. iSER Control and Data Transfer .............................69
   9.1 .................................64
      9.1. iSER Header Format .........................................69
   9.2 ........................................64
      9.2. iSER Header Format for iSCSI Control-Type PDU ..............69
   9.3 .............65
      9.3. iSER Header Format for iSER Hello Message ..................72
   9.4 .................67
      9.4. iSER Header Format for iSER HelloReply Message .............73
   9.5 ............68
      9.5. SCSI Data Transfer Operations ..............................74
   9.5.1 .............................69
           9.5.1. SCSI Write Operation......................................74
   9.5.2 Operation ...............................69
           9.5.2. SCSI Read Operation.......................................75
   9.5.3 Operation ................................70
           9.5.3. Bidirectional Operation...................................76
   10 Operation ............................70
   10. iSER Error Handling and Recovery ...........................77
   10.1 ..............................71
      10.1. Error Handling............................................77
   10.1.1 Handling ...........................................71
           10.1.1. Errors in the Transport Layer ...........................77
   10.1.2 .....................71
           10.1.2. Errors in the RCaP Layer ................................78
   10.1.3 ..........................72
           10.1.3. Errors in the iSER Layer ................................78
   10.1.4 ..........................73
           10.1.4. Errors in the iSCSI Layer ...............................80
   10.2 .........................75
      10.2. Error Recovery............................................82
   10.2.1 Recovery ...........................................76
           10.2.1. PDU Recovery ............................................82
   10.2.2 ......................................77
           10.2.2. Connection Recovery .....................................83
   11 ...............................77
   11. Security Considerations ....................................84
   12 .......................................78
   12. IANA Considerations ........................................85
   13 ...........................................79
   13. References .................................................86
   13.1 ....................................................79
      13.1. Normative References......................................86
   13.2 References .....................................79
      13.2. Informative References....................................86
   14 References ...................................80
   Appendix A: A. Summary of Changes from RFC 5046 ...............88
   15 ......................81
   Appendix B: B. Message Format for iSER ........................90
   15.1 ...............................83
   B.1. iWARP Message Format for iSER Hello Message...............90
   15.2 Message ..................83
   B.2. iWARP Message Format for iSER HelloReply Message..........91
   15.3 Message .............84
   B.3. iSER Header Format for SCSI Read Command PDU..............92
   15.4 PDU .................85
   B.4. iSER Header Format for SCSI Write Command PDU.............93
   15.5 PDU ................86
   B.5. iSER Header Format for SCSI Response PDU..................94
   16 PDU .....................87
   Appendix C: C. Architectural discussion of iSER over InfiniBand95
   16.1 InfiniBand ......88
   C.1. Host side Side of iSCSI & and iSER connections Connections in Infiniband.......95
   16.2 InfiniBand ........88
   C.2. Storage side Side of iSCSI & and iSER mixed network environment....96
   16.3 Mixed Network Environment .....89
   C.3. Discovery processes Processes for an InfiniBand Host................96
   16.4 Host ...................89
   C.4. IBTA Connection specifications............................97
   17 Specifications ...............................90
   Appendix D. Acknowledgments ............................................98 .......................................90

Table of Figures

   Figure 1 1. Example of iSCSI/iSER Layering in Full Feature Phase ...16 .....6
   Figure 2 2. iSER Header Format .....................................69 ......................................64
   Figure 3 3. iSER Header Format for iSCSI Control-Type PDU ..........70 ...........65
   Figure 4 4. iSER Header Format for iSER Hello Message ..............72 ...............67
   Figure 5 5. iSER Header Format for iSER HelloReply Message .........73 ..........68
   Figure 6 6. SendSE Message containing Containing an iSER Hello Message ........90 .........83
   Figure 7 7. SendSE Message containing Containing an iSER HelloReply Message ...91 ....84
   Figure 8 8. iSER Header Format for SCSI Read Command PDU ...........92 ............85
   Figure 9 9. iSER Header Format for SCSI Write Command PDU ..........93 ...........86
   Figure 10 10. iSER Header Format for SCSI Response PDU ..............94 ...............87
   Figure 11 11. iSCSI and iSER on IB ..................................95 ...................................88
   Figure 12 12. Storage Controller with TCP, iWARP, and IB Connections 96

1  Definitions and Acronyms

1.1  Definitions

   Advertisement (Advertised, Advertise, Advertisements, Advertises) - .89

1.  Introduction

1.1.  Motivation

   The act of informing a remote iSER (iSCSI Extensions for RDMA)
       Layer that a local node's buffer iSCSI protocol ([iSCSI]) is available to it.  A Node
       makes a buffer available for incoming RDMA Read Request Message
       or incoming RDMA Write Message access by informing the remote
       iSER Layer mapping of the Tagged Buffer identifiers (STag, Base Offset, SCSI Architecture
   Model (see [SAM5] and buffer length).  Note that this Advertisement of Tagged
       Buffer information is the responsibility of [iSCSI-SAM]) over the iSER Layer on
       either end TCP protocol.  SCSI
   commands are carried by iSCSI requests, and is not defined SCSI responses and status
   are carried by the RDMA-Capable Protocol.  A
       typical method would be for the iSER Layer to embed the Tagged
       Buffer's STag, Base Offset, iSCSI responses.  Other iSCSI protocol exchanges and buffer length
   SCSI Data are also transported in iSCSI PDUs.

   Out-of-order TCP segments in a message
       destined for the remote iSER Layer.

   Base Offset - A value when added Traditional iSCSI model have to be
   stored and reassembled before the Buffer Offset forms the
       Tagged Offset.

   Completion (Completed, Complete, Completes) - Completion is defined
       as the process by which the RDMA-Capable Protocol iSCSI protocol layer informs within an end
   node can place the iSER Layer, that a particular RDMA Operation has performed
       all functions specified for data in the RDMA Operation.

   Connection - A connection iSCSI buffers.  This reassembly is a logical bidirectional communication
       channel between the initiator
   required because not every TCP segment is likely to contain an iSCSI
   header to enable its placement and the target, e.g., TCP itself does not have a built-
   in mechanism for signaling ULP (Upper Level Protocol) message
   boundaries to aid placement of out-of-order segments.  This TCP
       connection.  Communication between
   reassembly at high network speeds is quite counterproductive for the initiator
   following reasons: wasted memory bandwidth in data copying, need for
   reassembly memory, wasted CPU cycles in data copying, and the target
       occurs over one or more connections. general
   store-and-forward latency from an application perspective.

   The connections carry
       control messages, SCSI commands, parameters, and data within
       iSCSI generic term RDMA-Capable Protocol Data Units (iSCSI PDUs).

   Connection Handle - An information element (RCaP) is used to refer to
   protocol stacks that identifies provide the
       particular iSCSI connection and is unique for a given iSCSI
       Layer Remote Direct Memory Access (RDMA)
   functionality, such as iWARP and InfiniBand.

   With the underlying iSER Layer.  Every invocation availability of an
       Operational Primitive RDMA-Capable Controllers within a host
   system, it is qualified with appropriate for iSCSI to be able to exploit the Connection Handle.

   Data Sink - The peer receiving a direct
   data payload.  Note that placement function of the Data
       Sink can be required RDMA-Capable Controller like other
   applications.

   iSCSI Extensions for RDMA (iSER) is designed precisely to both send and receive RCaP (RDMA-Capable
       Protocol) Messages take
   advantage of generic RDMA technologies -- iSER's goal is to transfer a permit
   iSCSI to employ direct data payload.

   Data Source - The peer sending placement and RDMA capabilities using a data payload.  Note that
   generic RDMA-Capable Controller.  In summary, the Data
       Source can be required iSCSI/iSER protocol
   stack is designed to both send and receive RCaP Messages enable scaling to
       transfer high speeds by relying on a
   generic data payload.

   Datamover Interface (DI) - The interface between the iSCSI Layer placement process and
       the Datamover Layer as described in [DA].

   Datamover Layer - A layer RDMA technologies and products
   that is directly below the iSCSI Layer enable direct data placement of both in-order and
       above the underlying transport layers. out-of-order
   data.

   This layer exposes and
       uses document describes iSER as a set of transport independent Operational Primitives protocol extension to iSCSI, both
   for
       the communication between the iSCSI Layer convenience of description and itself.  The
       Datamover layer, operating also because it is true in conjunction with a very
   strict protocol sense.  However, it is to be noted that iSER is in
   reality extending the transport
       layers, moves connectivity of the control iSCSI protocol defined in
   [iSCSI], and data information on the iSCSI
       connection.  In name "iSER" reflects this specification, the iSER Layer is reality.

   When the
       Datamover layer.

   Datamover Protocol - A Datamover iSCSI protocol is as defined in [iSCSI] (i.e., without the wire-protocol that iSER
   enhancements) is defined to realize intended in the Datamover layer functionality.  In
       this specification, rest of the iSER protocol document, the term
   "Traditional iSCSI" is used to make the Datamover protocol.

   Inbound RDMA Read Queue Depth (IRD) - The maximum number intention clear.

   This document obsoletes RFC 5046.  See Appendix A for the list of incoming
       outstanding
   changes from RFC 5046.

1.2.  iSCSI/iSER Layering

   iSCSI Extensions for RDMA Read Requests that (iSER) is layered between the RDMA-Capable Controller
       can handle on a particular RCaP Stream at iSCSI layer
   and the Data Source.  For
       some RDMA-Capable RCaP layer.

         +--------------------------------------------------------+
         |                        SCSI                            |
         +--------------------------------------------------------+
         |                        iSCSI                           |
   DI -> +--------------------------------------------------------+
         |                         iSER                           |
         +-------+--------------------------+---------------------+
         | RDMAP |                          |                     |
         +-------+      InfiniBand          |                     |
         |  DDP  |       Reliable           |       Other         |
         +-------+       Connected          |        RDMA         |
         |  MPA  |       Transport          |       Capable       |
         +-------+        Service           |       Protocol layers,      |
         |  TCP  |                          |                     |
         +-------+--------------------------+---------------------+
         |  IP   | InfiniBand Network Layer | Other Network Layer |
         +-------+--------------------------+---------------------+

    Figure 1: Example of iSCSI/iSER Layering in Full Feature Phase
   Figure 1 shows an example of the relationship between SCSI, iSCSI,
   iSER, and the term "IRD" may be known
       by a different name. RCaP layers.  For example, for InfiniBand, TCP, the
       equivalent for IRD RCaP is iWARP.
   For InfiniBand, the Responder Resources.

   I/O Buffer - A buffer that RCaP is used in a SCSI Read or Write operation
       so SCSI data may be sent from or received into the Reliable Connected Transport Service.
   Note that buffer.

   iSCSI - The the iSCSI protocol layer as defined described here supports the RDMA
   Extensions as used in [iSCSI] is a mapping of iSER.

1.3.  Architectural Goals

   This section summarizes the
       SCSI Architecture Model of SAM-5 over TCP.

   iSCSI control-type PDU - Any iSCSI PDU architectural goals that is not guided the
   design of iSER.

   1.  Provide an iSCSI data-
       type PDU and also not a SCSI Data-out PDU carrying solicited RDMA data is defined as an transfer model for iSCSI control-type PDU.  Specifically, it
       is to be noted that enables direct
       in-order or out-of-order data placement of SCSI Data-out PDUs for unsolicited data are
       defined as iSCSI control-type PDUs.

   iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI
       PDU that causes into pre-
       allocated SCSI buffers while maintaining in-order data transfer via RDMA operations at the iSER
       layer, transparent delivery.

   2.  Do not require any major changes to the remote iSCSI Layer, to take place
       between SCSI Architecture Model
       [SAM5] and SCSI command set standards.

   3.  Utilize the peer existing iSCSI nodes on infrastructure (sometimes referred to
       as "iSCSI ecosystem") including but not limited to MIB,
       bootstrapping, negotiation, naming and discovery, and security.

   4.  Enable a full feature phase iSCSI
       connection.  An iSCSI data-type PDU, when requested for
       transmission by the sender iSCSI Layer, results session to operate in the
       associated Traditional iSCSI data
       transfer without the participation of the remote
       iSCSI Layer, i.e. the PDU itself mode if iSER is not delivered as-is to supported by either the
       remote iSCSI Layer.  The following iSCSI PDUs constitute initiator or
       the set
       of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU. target.  (Do not require iSCSI Layer - A layer in the protocol stack implementation within Full Feature Phase
       interoperability between an end node that implements the operating in Traditional
       iSCSI protocol mode and interfaces with
       the iSER Layer via the Datamover Interface.

   iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI Layer at the an end node operating in iSER-assisted mode.)

   5.  Allow initiator and the iSCSI Layer at the target divide their
       communications into messages.  The term "iSCSI protocol data
       unit" (iSCSI PDU) is used for these messages.

   iSCSI/iSER Connection - An iSER-assisted iSCSI connection.  An implementations to utilize generic
       RDMA-Capable Controllers such as RNICs or to implement iSCSI
       connection that is and
       iSER in software.  (Do not iSER-assisted always maps onto a TCP
       connection at require iSCSI- or iSER-specific
       assists in the transport level.  But an iSER-assisted iSCSI
       connection may not have an underlying TCP connection.  For some RCaP implementation (e.g., iWARP), an iSER-assisted or RDMA-Capable Controller.)

   6.  Implement a lightweight Datamover protocol for iSCSI
       connection has an underlying TCP connection.  For other RCaP
       implementation (e.g., InfiniBand), there is no underlying TCP
       connection.  (In with minimal
       state maintenance.

1.4.  Protocol Overview

   Consistent with the specific example of InfiniBand [IB], an
       iSER-assisted iSCSI connection is directly mapped onto architectural goals stated in Section 1.3, the
   iSER protocol does not require changes in the
       InfiniBand RC channel.)

   iSCSI/iSER Session - An iSER-assisted iSCSI session.  All
       connections ecosystem or any
   related SCSI specifications.  The iSER protocol defines the mapping
   of an iSCSI PDUs to RCaP Messages in such a way that it is entirely
   feasible to realize iSCSI/iSER session implementations that are iSCSI/iSER connections. based on
   generic RDMA-Capable Controllers.  The iSER - iSCSI Extensions for RDMA, the protocol defined in this
       document.

   iSER-assisted - A term generally used layer requires
   minimal state maintenance to describe assist a connection during the operation of iSCSI when
   Full Feature Phase, besides being oblivious to the notion of an iSCSI
   session.  The crucial protocol aspects of iSER functionality may be summarized as
   follows:

   1.  iSER-assisted mode is also enabled below negotiated during the iSCSI Layer for the specific iSCSI/iSER connection login in question.

   iSER-IRD - This variable represents the maximum number
       leading connection for each session, and an entire iSCSI session
       can only operate in one mode (i.e., a connection in a session
       cannot operate in iSER-assisted mode if a different connection of incoming
       outstanding RDMA Read Requests that
       the iSER Layer at same session is already in Full Feature Phase in the
       initiator grants
       Traditional iSCSI mode).

   2.  Once in iSER-assisted mode, all iSCSI interactions on a particular RCaP Stream.

   iSER-ORD - This variable represents the maximum number of
       outstanding RDMA Read Requests that the iSER Layer can initiate
       on a particular
       connection use RCaP Stream.  This variable Messages.

   3.  A Send Message is maintained only used for carrying an iSCSI control-type PDU
       preceded by the an iSER Layer at header.  See Section 7.2 for more details on
       iSCSI control-type PDUs.

   4.  RDMA Write, RDMA Read Request, and RDMA Read Response Messages
       are used for carrying control and all data information associated
       with the target. iSCSI data-type PDUs (i.e., SCSI Data-In PDUs and R2T
       PDUs).  iSER Layer - does not use SCSI Data-Out PDUs for solicited data,
       and SCSI Data-Out PDUs for unsolicited data are not treated as
       iSCSI data-type PDUs by iSER because RDMA is not used.  See
       Section 7.1 for more details on iSCSI data-type PDUs.

   5.  The layer that implements target drives all data transfer (with the exception of iSCSI Extensions
       unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA
       (iSER) protocol.
       Read Requests and RDMA Writes, respectively.

   6.  RCaP is responsible for ensuring data integrity.  (For example,
       iWARP - A suite of wire protocols comprising includes a CRC-enhanced framing layer called MPA on top of [RDMAP], [DDP],
       TCP; and
       [MPA] when layered above [TCP].  [RDMAP] for InfiniBand, the CRCs are included in the Reliable
       Connection mode).  For this reason, iSCSI header and [DDP] may be
       layered above SCTP or other transport protocols.

   Local Mapping - A task state record maintained data digests
       are negotiated to "None" for iSCSI/iSER sessions.

   7.  The iSCSI error recovery hierarchy defined in [iSCSI] is fully
       supported by iSER.  (However, see Section 7.3.11 on the handling
       of SNACK Request PDUs.)

   8.  iSER Layer requires no changes to iSCSI security and text mode
       negotiation mechanisms.

   Note that Traditional iSCSI implementations may have to be adapted to
   employ iSER.  It is expected that associates the Initiator Task Tag adaptation when required is
   likely to be centered around the Local STag(s).
       The specifics upper-layer interface requirements
   of iSER (Section 3).

1.5.  RDMA Services and iSER

   iSER is designed to work with software and/or hardware protocol
   stacks providing the record structure are implementation
       dependent.

   Local Peer - protocol services defined in RCaP documents such
   as [RDMAP], [IB], etc.  The implementation of following subsections describe the RDMA-Capable Protocol key
   protocol elements of RCaP services on which iSER relies.

1.5.1.  STag

   An STag is the
       local end identifier of the connection.  Used to refer an I/O Buffer unique to an RDMA-Capable
   Controller that the local entity
       when describing protocol exchanges or other interactions between
       two Nodes.

   Node - A computing device attached iSER layer Advertises to one or more links of a
       network.  A Node the remote iSCSI/iSER
   node in this context does not refer order to complete a specific
       application or protocol instantiation running on SCSI I/O.

   In iSER, Advertisement is the computer.
       A Node may consist act of one or more RDMA-Capable Controllers
       installed in a host computer.

   Operational Primitive - An Operational Primitive is an abstract
       functional interface procedure informing the target by the
   initiator that requests another layer to
       perform a specific action on an I/O Buffer is available at the requestor's behalf initiator for RDMA
   Read or notifies RDMA Write access by the other layer of some event. target.  The Datamover Interface between
       an iSCSI Layer initiator Advertises
   the I/O Buffer by including the STag and a Datamover layer within an iSCSI end node
       uses a set of Operational Primitives to define the functional
       interface between Base Offset in the two layers.  Note that not every
       invocation
   header of an Operational Primitive may elicit a response
       from the requested layer.  A full discussion of iSER Message containing the Operational
       Primitive types and request-response semantics available SCSI Command PDU to
       iSCSI and iSER can be found in [DA].

   Outbound RDMA Read Queue Depth (ORD) - the
   target.  The maximum number of
       outstanding RDMA Read Requests that buffer length is as specified in the RDMA-Capable Controller
       can initiate on a particular RCaP Stream SCSI Command PDU.

   The iSER layer at the Data Sink.  For
       some RDMA-Capable Protocol layer, initiator Advertises the term "ORD" may be known by
       a different name.  For example, for InfiniBand, STag and the equivalent Base
   Offset for ORD is the Initiator Depth.

   Phase Collapse - Refers I/O Buffer of each SCSI I/O to the optimization in iSCSI where iSER layer at the SCSI
       status is transferred along with
   target in the final SCSI Data-in PDU from iSER header of a target.  See section 4.2 in [iSCSI].

   RCaP Send Message - One or more packets of containing the network layer comprising a
       single RDMA operation or a part SCSI
   Command PDU, unless the I/O can be completely satisfied by
   unsolicited data alone.  The SendSE Message should be used if
   supported by the RCaP layer (e.g., iWARP).

   The iSER layer at the target provides the STag for the I/O Buffer
   that is the Data Sink of an RDMA Read Operation of (Section 1.5.4) to
   the
       RDMA-Capable Protocol.  For iWARP, an RCaP Message layer on the initiator node -- i.e., this is known as
       an RDMAP Message.

   RCaP Stream - A single bidirectional association between completely
   transparent to the peer
       RDMA-Capable Protocol layers on two Nodes over iSER layer at the initiator.

   The iSER layer at the initiator SHOULD invalidate the Advertised STag
   upon a single
       transport-level stream.   For iWARP, an normal completion of the associated task.  The Send with
   Invalidate Message, if supported by the RCaP Stream layer (e.g., iWARP), can
   be used for automatic invalidation when it is known as
       an RDMAP Stream, and used to carry the association is created following SCSI
   Response PDU.  There are two exceptions to this automatic
   invalidation -- bidirectional commands and abnormal completion of a
       successful Login Phase during which iSER support is negotiated.

   RDMA-Capable Protocol (RCaP) -
   command.  The protocol or protocol suite iSER layer at the initiator SHOULD explicitly
   invalidate the STag in these two cases.  That iSER layer MUST check
   that
       provides STag invalidation has occurred whenever receipt of a reliable RDMA transport functionality, e.g., iWARP,
       InfiniBand, etc.

   RDMA-Capable Controller - A network I/O adapter or embedded
       controller Send with RDMA functionality.  For example, for iWARP,
       this could be
   Invalidate message is the expected means of causing an RNIC, STag to be
   invalidated, and for InfiniBand, this could be it MUST perform the STag invalidation if the STag
   has not already been invalidated (e.g., because a HCA
       (Host Channel Adapter) or TCA (Target Channel Adapter).

   RDMA-enabled Network Interface Controller (RNIC) - A network I/O
       adapter or embedded controller with iWARP functionality.

   RDMA Operation - A sequence Send Message was
   used instead of RCaP Messages, including control
       Messages, to transfer data from a Data Source to a Data Sink.
       The following RDMA Operations are defined - RDMA Write
       Operation, RDMA Read Operation, and Send Operation.

   RDMA Protocol (RDMAP) - A wire protocol that supports RDMA
       Operations to transfer ULP (Upper Level Protocol) data between a
       Local Peer and with Invalidate).

   If the Remote Peer Advertised STag is not invalidated as described recommended in [RDMAP].

   RDMA Read Operation - An RDMA Operation used by the Data Sink
   foregoing paragraph (e.g., in order to
       transfer cache the contents of a Data Source buffer from STag for future
   reuse), the Remote
       Peer I/O Buffer remains exposed to a Data Sink buffer at the Local Peer.  An RDMA Read
       operation consists of a single RDMA Read Request Message and a
       single RDMA Read Response Message.

   RDMA Read Request - An RCaP Message used network for access by
   the Data Sink to request
       the Data Source to transfer the contents RCaP.  Such an I/O Buffer is capable of a buffer.  The RDMA
       Read Request Message describes both the Data Source and the Data
       Sink buffers.

   RDMA Read Response - An RCaP Message used being read or written by
   the Data Source to
       transfer RCaP outside the contents scope of a buffer to the Data Sink, in response
       to an RDMA Read Request. iSCSI operation for which it was
   originally established; this fact has both robustness and security
   considerations.  The RDMA Read Response Message only
       describes robustness considerations are that the Data Sink buffer.

   RDMA Write Operation - An RDMA Operation used by system
   containing the Data Source iSER initiator may react poorly to
       transfer the contents an unexpected
   modification of a Data Source buffer from its memory.  For the Local
       Peer security considerations, see
   Section 11.

1.5.2.  Send

   Send is the RDMA Operation that is not addressed to a Data Sink an Advertised
   buffer at and uses Untagged buffers as the Remote Peer. message is received.

   The RDMA Write
       Message only describes the Data Sink buffer.

   Remote Direct Memory Access (RDMA) - A method of accessing memory on
       a remote system in which the local system specifies iSER layer at the remote
       location of initiator uses the data Send Operation to be transferred.  Employing an RDMA-
       Capable Controller in the remote system allows the access transmit
   any iSCSI control-type PDU to take
       place without interrupting the processing of the CPU(s) on the
       system.

   Remote Mapping - A task state record maintained by target.  As an example, the
   initiator uses Send Operations to transfer iSER Layer
       that associates the Initiator Task Tag Messages containing
   SCSI Command PDUs to the Advertised STag(s)
       and the Base Offset(s).  The specifics of the record structure
       are implementation dependent.

   Remote Peer - The implementation of iSER layer at the RDMA-Capable Protocol on target.

   An iSER layer at the
       opposite end of target uses the connection.  Used Send Operation to refer transmit any
   iSCSI control-type PDU to the remote
       entity when describing protocol exchanges or other interactions
       between two Nodes.

   SCSI Layer - This layer builds/receives SCSI CDBs (Command
       Descriptor Blocks) and sends/receives them with the remaining
       command execute [SAM5] parameters to/from initiator.  As an example, the iSCSI Layer. target
   uses Send - An RDMA Operation that transfers the content of a buffer from
       the Local Peer Operations to an untagged buffer transfer iSER Messages containing SCSI
   Response PDUs to the iSER layer at the Remote Peer.

   SendInvSE Message - A Send with Solicited Event initiator.

   For interoperability, iSER implementations SHOULD accept and Invalidate
       Message.
   correctly process SendSE Message - A Send with Solicited Event Message.

   Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for
       an R2T PDU.  The semantics for both types of sequence numbers SendInvSE messages.  However, SendSE and
   SendInvSE messages are as defined in [iSCSI].

   Session, iSCSI Session - The group of Connections that link an
       initiator SCSI port with a target SCSI port form an iSCSI
       session (equivalent to a SCSI I-T nexus).  Connections can be
       added regarded as optimizations or
   enhancements to the basic Send Message, and removed from a session even while their support may vary by
   RCaP protocol and specific implementation.  In general, these
   messages SHOULD NOT be used, unless the I-T nexus is
       intact.  Across RCaP requires support for
   them in all connections within a session, an initiator
       sees one and implementations.  If these messages are used, the same target.

   Steering Tag (STag) - An identifier
   implementation SHOULD be capable of reverting to use of Send in order
   to work with a Tagged Buffer on receiver that does not support these messages.
   Attempted use of these messages with a Node
       (Local or Remote) as defined peer that does not support
   them may result in [RDMAP] and [DDP].  For other
       RDMA-Capable Protocols, a fatal error that closes the Steering Tag may be known by
       different names but will be herein referred to as STags. RCaP connection.
   For example, these messages SHOULD NOT be used with the InfiniBand
   RCaP because InfiniBand does not require support for Infiniband, a Remote STag is known as an R-Key, them in all
   cases.  New iSER implementations SHOULD use Send (and not SendSE or
   SendInvSE) unless there are compelling reasons for doing otherwise.
   Similarly, iSER implementations SHOULD NOT rely on events triggered
   by SendSE and
       a Local STag is known SendInvSE, as an L-Key, and both will these messages may not be considered
       STags.

   Tagged Buffer - A buffer used.

1.5.3.  RDMA Write

   RDMA Write is the RDMA Operation that is explicitly Advertised used to the iSER
       Layer place data into an
   Advertised buffer at the remote node through Data Sink.  The Data Source addresses the exchange of
   Message using an STag, Base
       Offset, STag and length. a Tagged Offset - that are valid on the Data
   Sink.

   The offset within a Tagged Buffer.

   Traditional iSCSI - Refers to iSER layer at the iSCSI protocol as defined in
       [iSCSI] (i.e. without target uses the iSER enhancements).

   Untagged RDMA Write Operation to
   transfer the contents of a local I/O Buffer - A buffer that is not explicitly to an Advertised I/O
   Buffer at the initiator.  The iSER layer at the target uses the RDMA
   Write to transfer the whole data or part of the data required to
   complete a SCSI Read command.

   The iSER Layer layer at the remode node.

1.2  Acronyms

   Acronym        Definition

   --------------------------------------------------------------

   AHS            Additional Header Segment

   BHS            Basic Header Segment

   CO             Connection Only

   CRC            Cyclic Redundancy Check

   DDP            Direct Data Placement Protocol

   DI             Datamover Interface

   HCA            Host Channel Adapter

   IANA           Internet Assigned Numbers Authority

   IB             Infiniband

   IETF           Internet Engineering Task Force

   I/O            Input - Output

   IO             Initialize Only

   IP             Internet Protocol

   IPoIB          IP over Infiniband

   IPsec          Internet Protocol Security

   iSER           iSCSI Extensions for initiator does not employ RDMA Writes.

1.5.4.  RDMA Read

   RDMA Read is the RDMA

   ITT            Initiator Task Tag
   LO             Leading Only

   MPA            Marker PDU Aligned Framing for TCP

   NOP            No Operation

   NSG            Next Stage (during that is used to retrieve data from an
   Advertised buffer at the iSCSI Login Phase)

   PDU            Protocol Data Unit

   R2T            Ready To Transfer

   R2TSN          Ready To Transfer Sequence Number

   RCaP           RDMA-Capable Protocol Source.  The sender of the RDMA           Remote Direct Memory Access

   RDMAP          Remote Direct Memory Access Protocol

   RFC Read
   Request For Comments

   RNIC           RDMA-enabled Network Interface Controller

   SAM5           SCSI Architecture Model - 5

   SCSI           Small Computer Systems Interface

   SNACK          Selective Negative Acknowledgment - also

                  Sequence Number Acknowledgement for data addresses the Message using an STag           Steering Tag

   SW             Session Wide

   TCA            Target Channel Adapter

   TCP            Transmission Control Protocol

   TMF            Task Management Function

   TTT            Target Transfer Tag

   ULP            Upper Level Protocol

1.3  Conventions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document a Tagged Offset that
   are to be interpreted as described valid on the Data Source in [RFC2119].

2  Introduction

2.1  Motivation

   The iSCSI protocol ([iSCSI]) is addition to providing a mapping of the SCSI Architecture
   Model (see [SAM5] valid local
   STag and [iSCSI-SAM]) over Tagged Offset that identify the TCP protocol.  SCSI
   commands are carried by iSCSI requests and SCSI responses and status
   are carried by iSCSI responses.  Other iSCSI protocol exchanges and
   SCSI Data are also transported in iSCSI PDUs.

   Out-of-order TCP segments in Sink.

   The iSER layer at the Traditional iSCSI model have target uses the RDMA Read Operation to be
   stored and reassembled before transfer
   the iSCSI protocol layer within contents of an end
   node can place Advertised I/O Buffer at the data in initiator to a local
   I/O Buffer at the iSCSI buffers.  This reassembly is
   required because not every TCP segment is likely target.  The iSER layer at the target uses the RDMA
   Read to contain an iSCSI
   header fetch whole or part of the data required to enable its placement and TCP itself complete a SCSI
   Write Command.

   The iSER layer at the initiator does not have a built-
   in mechanism for signaling ULP message boundaries to aid placement
   of out-of-order segments.  This TCP reassembly employ RDMA Reads.

1.6.  SCSI Read Overview

   The iSER layer at high network
   speeds is quite counter-productive for the following reasons: wasted
   memory bandwidth in data copying, need for reassembly memory, wasted
   CPU cycles in data copying, and initiator receives the general store-and-forward
   latency SCSI Command PDU from an application perspective.

   The generic term RDMA-Capable Protocol (RCaP) is used to refer to
   protocol stacks that provide
   the Remote Direct Memory Access (RDMA)
   functionality, such as iWARP and InfiniBand.

   With iSCSI layer.  The iSER layer at the availability of RDMA-Capable Controllers within a host
   system, it is appropriate initiator generates an STag
   for iSCSI to be able to exploit the direct
   data placement function I/O Buffer of the RDMA-Capable Controller like other
   applications.

   iSCSI Extensions for RDMA (iSER) is designed precisely to take
   advantage of generic RDMA technologies - iSER's goal is to permit
   iSCSI to employ direct data placement SCSI Read and RDMA capabilities using a
   generic RDMA-Capable Controller.  In summary, iSCSI/iSER protocol
   stack is designed to enable scaling to high speeds Advertises the buffer by relying on a
   generic data placement process and RDMA technologies
   including the STag and products,
   which enable direct data placement the Base Offset as part of both in-order and out-of-order
   data.

   This document describes the iSER as a protocol extension to iSCSI, both header for convenience of description and also because it is true in a very
   strict protocol sense.  However, it
   the PDU.  The iSER Message is transferred to the target using a Send
   Message.  The SendSE Message should be noted that used if supported by the RCaP
   layer (e.g., iWARP).

   The iSER is in
   reality extending layer at the connectivity of target uses one or more RDMA Writes to transfer
   the iSCSI protocol defined in
   [iSCSI], and data required to complete the name SCSI Read.

   The iSER reflects this reality.

   When the iSCSI protocol as defined in [iSCSI] (i.e. without layer at the iSER
   enhancements) is intended in target uses a Send Message to transfer the rest of SCSI
   Response PDU back to the document, iSER layer at the term
   "Traditional iSCSI" is used to make initiator.  The iSER layer
   at the intention clear.

   This document obsoletes RFC 5046.  See Section 14 for initiator invalidates the list of
   changes from RFC 5046.

2.2  iSCSI/iSER Layering

   iSCSI Extensions for RDMA (iSER) is layered between STag and notifies the iSCSI layer
   and of
   the RCaP layer.

          +--------------------------------------------------------+

          |                        SCSI                            |

          +--------------------------------------------------------+

          |                        iSCSI                           |

    DI -> +--------------------------------------------------------+

          |                         iSER                           |

          +-------+--------------------------+---------------------+

          | RDMAP |                          |                     |

          +-------+      Infiniband          |                     |

          |  DDP  |       Reliable           |       Other         |

          +-------+       Connected          |        RDMA         |

          |  MPA  |       Transport          |       Capable       |

          +-------+        Service           |       Protocol      |

          |  TCP  |                          |                     |

          +-------+--------------------------+---------------------+

          |  IP   | Infiniband Network Layer | Other Network Layer |

          +-------+--------------------------+---------------------+

       Figure 1 Example of iSCSI/iSER Layering in Full Feature Phase
   Figure 1 shows an example availability of the relationship between SCSI, iSCSI,
   iSER, and SCSI Response PDU.  The Send with Invalidate
   Message, if supported by the different RCaP layers.  For TCP, layer (e.g., iWARP), can be used
   for automatic invalidation of the RCaP is iWARP.
   For Infiniband, STag.

1.7.  SCSI Write Overview

   The iSER layer at the RCaP is initiator receives the Reliable Connected Transport
   Service.  Note that SCSI Command PDU from
   the iSCSI layer as described here supports the
   RDMA Extensions as used in iSER.

2.3  Architectural Goals

   This section summarizes layer.  If solicited data transfer is involved, the architectural goals that guided iSER
   layer at the
   design of iSER.

   1. Provide initiator generates an RDMA data transfer model STag for iSCSI that enables direct
     in order or out of order data placement the I/O Buffer of the
   SCSI data into pre-
     allocated SCSI buffers while maintaining in order data delivery.

   2. Not require any major changes to SCSI Architecture Model [SAM5] Write and SCSI command set standards.

   3. Utilize existing iSCSI infrastructure (sometimes referred to as
     "iSCSI ecosystem") Advertises the buffer by including but not limited to MIB,
     bootstrapping, negotiation, naming & discovery, and security.

   4. Enable a session to operate in the Traditional iSCSI data transfer
     mode if STag and the
   Base Offset as part of the iSER header for the PDU.  The iSER Message
   is not transferred to the target using a Send Message.  The SendSE
   Message should be used if supported by either the RCaP layer (e.g., iWARP).

   The iSER layer at the initiator may optionally send one or more non-
   immediate unsolicited data PDUs to the target (not require iSCSI full feature phase interoperability
     between an end node operating in Traditional iSCSI mode, and an
     end node operating in iSER-assisted mode).

   5. Allow initiator and target implementations to utilize generic
     RDMA-Capable Controllers such as RNICs, or implement iSCSI and
     iSER in software (not require iSCSI or using Send Messages.

   If solicited data transfer is involved, the iSER specific assists in layer at the RCaP implementation target
   uses one or RDMA-Capable Controller).

   6. Implement a light weight Datamover protocol for iSCSI with minimal
     state maintenance.

2.4  Protocol Overview

   Consistent with the architectural goals stated in section 2.3, more RDMA Reads to transfer the
   iSER protocol does not require changes in data required to complete
   the iSCSI ecosystem or any
   related SCSI specifications. Write.

   The iSER protocol defines layer at the mapping of
   iSCSI PDUs to RCaP Messages in such target uses a way that it is entirely
   feasible Send Message to realize iSCSI/iSER implementations that are based on
   generic RDMA-Capable Controllers.  The transfer the SCSI
   Response PDU back to the iSER protocol layer requires
   minimal state maintenance to assist an iSCSI full feature phase
   connection, besides being oblivious to at the notion of an iSCSI
   session. initiator.  The crucial protocol aspects of iSER may be summarized
   thus:

   1. iSER-assisted mode is negotiated during layer
   at the iSCSI login in initiator invalidates the
      leading connection for each session, STag and an entire notifies the iSCSI session
      can only operate in one mode (i.e. a connection in a session
      cannot operate in iSER-assisted mode if a different connection layer of
   the same session is already in full feature phase in availability of the
      Traditional iSCSI mode).

   2. Once in iSER-assisted mode, all iSCSI interactions on that
      connection use RCaP Messages.

   3. A SCSI Response PDU.  The Send Message is with Invalidate
   Message, if supported by the RCaP layer (e.g., iWARP), can be used
   for carrying an iSCSI control-type PDU
      preceded by an automatic invalidation of the STag.

2.  Definitions and Acronyms

2.1.  Definitions

   Advertisement (Advertised, Advertise, Advertisements, Advertises) --
      The act of informing a remote iSER header.  See section 7.2 (iSCSI Extensions for more details on
      iSCSI control-type PDUs.

   4. RDMA Write, RDMA)
      layer that a local node's buffer is available to it.  A node makes
      a buffer available for incoming RDMA Read Request, and Request Message or
      incoming RDMA Read Response Messages
      are used for carrying control Write Message access by informing the remote iSER
      layer of the Tagged Buffer identifiers (STag, Base Offset, and all data
      buffer length).  Note that this Advertisement of Tagged Buffer
      information associated
      with is the responsibility of the iSCSI data-type PDUs (i.e., SCSI Data-In PDUs and R2T
      PDUs). iSER does layer on either end
      and is not use SCSI Data-Out PDUs defined by the RDMA-Capable Protocol.  A typical method
      would be for solicited data, the iSER layer to embed the Tagged Buffer's STag,
      Base Offset, and SCSI Data-Out PDUs buffer length in a message destined for unsolicited data are not treated as
      iSCSI data-type PDUs by the
      remote iSER because RDMA layer.

   Base Offset - A value when added to the Buffer Offset forms the
      Tagged Offset.

   Completion (Completed, Complete, Completes) - Completion is not used.  See
      section 7.1 for more details on iSCSI data-type PDUs.

   5. Target drives all data transfer (with defined
      as the exception of iSCSI
      unsolicited data) for SCSI writes and SCSI reads, process by issuing which the RDMA-Capable Protocol layer informs
      the iSER layer that a particular RDMA
      Read Requests and Operation has performed all
      functions specified for the RDMA Writes respectively.

   6. RCaP Operation.

   Connection - A connection is responsible for ensuring data integrity.  (For example,
      iWARP includes a CRC-enhanced framing layer called MPA on top of
      TCP; logical bidirectional communication
      channel between the initiator and for Infiniband, the CRCs are included in target, e.g., a TCP
      connection.  Communication between the Reliable initiator and the target
      occurs over one or more connections.  The connections carry
      control messages, SCSI commands, parameters, and data within iSCSI
      Protocol Data Units (iSCSI PDUs).

   Connection mode).  For this reason, Handle - An information element that identifies the
      particular iSCSI header connection and data digests
      are negotiated to "None" is unique for iSCSI/iSER sessions.

   7. The a given iSCSI error recovery hierarchy defined in [iSCSI] is fully
      supported by iSER.  (However, see section 7.3.11 on layer
      and the handling
      of SNACK Request PDUs.)

   8. underlying iSER requires no changes to iSCSI security and text mode
      negotiation mechanisms. layer.  Every invocation of an Operational
      Primitive is qualified with the Connection Handle.

   Data Sink - The peer receiving a data payload.  Note that Traditional iSCSI implementations may have to the Data
      Sink can be adapted required to employ iSER.  It is expected both send and receive RCaP (RDMA-Capable
      Protocol) Messages to transfer a data payload.

   Data Source - The peer sending a data payload.  Note that the adaptation when Data
      Source can be required is
   likely to be centered around both send and receive RCaP Messages to
      transfer a data payload.

   Datamover Interface (DI) - The interface between the upper iSCSI layer interface requirements
   of iSER (section 3).

2.5  RDMA services and iSER

   iSER is designed to work with software and/or hardware protocol
   stacks providing
      the protocol services defined in RCaP documents
   such Datamover Layer as [RDMAP], [IB], etc.  The following subsections describe the
   key protocol elements of RCaP services described in [DA].

   Datamover Layer - A layer that iSER relies on.

2.5.1  STag

   An STag is directly below the identifier of an I/O Buffer unique to an RDMA-Capable
   Controller that the iSER Layer Advertises to iSCSI layer and
      above the remote iSCSI/iSER
   node in order to complete underlying transport layers.  This layer exposes and
      uses a SCSI I/O.

   In iSER, Advertisement is the act set of informing the target by the
   initiator that an I/O Buffer is available at the initiator transport-independent Operational Primitives for RDMA
   Read or RDMA Write access by the target.
      communication between the iSCSI layer and itself.  The initiator Advertises Datamover
      layer, operating in conjunction with the I/O Buffer by including transport layers, moves
      the STag control and data information on the Base Offset in iSCSI connection.  In this
      specification, the
   header of an iSER Message containing layer is the SCSI Command PDU Datamover layer.

   Datamover Protocol - A Datamover protocol is the wire protocol that
      is defined to realize the
   target.  The buffer length Datamover-layer functionality.  In this
      specification, the iSER protocol is as specified in the SCSI Command PDU. Datamover protocol.

   Inbound RDMA Read Queue Depth (IRD) - The iSER Layer at maximum number of incoming
      outstanding RDMA Read Requests that the initiator Advertises RDMA-Capable Controller
      can handle on a particular RCaP Stream at the STag and Data Source.  For
      some RDMA-Capable Protocol layers, the Base
   Offset term "IRD" may be known by
      a different name.  For example, for InfiniBand, the I/O Buffer of each SCSI I/O equivalent to
      IRD is the iSER Layer at the
   target Responder Resources.

   I/O Buffer - A buffer that is used in the iSER header of a Send Message containing the SCSI
   Command PDU, unless the I/O can be completely satisfied by
   unsolicited Read or Write operation
      so SCSI data alone.  The SendSE Message should may be used if
   supported by the RCaP layer (e.g., iWARP).

   The iSER Layer at the target provides the STag for the I/O Buffer sent from or received into that buffer.

   iSCSI - The iSCSI protocol as defined in [iSCSI] is a mapping of the Data Sink
      SCSI Architecture Model of SAM-5 over TCP.

   iSCSI control-type PDU - Any iSCSI PDU that is not an RDMA Read Operation (section 2.5.4) iSCSI data-
      type PDU and also not a SCSI Data-Out PDU carrying solicited data
      is defined as an iSCSI control-type PDU.  Specifically, it is to
   the RCaP layer on the initiator node
      be noted that SCSI Data-Out PDUs for unsolicited data are defined
      as iSCSI control-type PDUs.

   iSCSI data-type PDU - i.e. this An iSCSI data-type PDU is completely
   transparent to the iSER Layer defined as an iSCSI
      PDU that causes data transfer via RDMA operations at the initiator.

   The iSER layer at
      layer, transparent to the initiator SHOULD invalidate remote iSCSI layer, to take place
      between the Advertised
   STag upon peer iSCSI nodes on a normal completion of Full Feature Phase iSCSI
      connection.  An iSCSI data-type PDU, when requested for
      transmission by the sender iSCSI layer, results in the associated task.  The Send with
   Invalidate Message, if supported by
      data transfer without the RCaP layer (e.g., iWARP),
   can be used for automatic invalidation when it participation of the remote iSCSI layer,
      i.e., the PDU itself is used not delivered as-is to carry the remote iSCSI
      layer.  The following iSCSI PDUs constitute the set of iSCSI data-
      type PDUs -- SCSI Response Data-In PDU and R2T PDU.  There are two exceptions to this automatic
   invalidation

   iSCSI Layer - bidirectional commands, A layer in the protocol stack implementation within an
      end node that implements the iSCSI protocol and abnormal completion of a
   command.  The interfaces with
      the iSER Layer layer via the Datamover Interface.

   iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI layer at the
      initiator SHOULD explicitly
   invalidate and the STag in these two cases.  That iSER iSCSI layer MUST check at the target divide their
      communications into messages.  The term "iSCSI Protocol Data Unit"
      (iSCSI PDU) is used for these messages.

   iSCSI/iSER Connection - An iSER-assisted iSCSI connection.  An iSCSI
      connection that STag invalidation has occurred whenever receipt of a Send with
   Invalidate message is not iSER assisted always maps onto a TCP
      connection at the expected means of causing transport level.  But an STag to be
   invalidated, and MUST perform the STag invalidation if the STag has iSER-assisted iSCSI
      connection may not already been invalidated have an underlying TCP connection.  For some
      RCaP implementations (e.g., because a Send message was used
   instead of Send with Invalidate).

   If the Advertised STag iWARP), an iSER-assisted iSCSI
      connection has an underlying TCP connection.  For other RCaP
      implementations (e.g., InfiniBand), there is not invalidated as recommended in no underlying TCP
      connection.  (In the
   foregoing paragraph (e.g., in order to cache specific example of InfiniBand [IB], an iSER-
      assisted iSCSI connection is directly mapped onto the STag InfiniBand
      Reliable Connection-based (RC) channel.)

   iSCSI/iSER Session - An iSER-assisted iSCSI session.  All connections
      of an iSCSI/iSER session are iSCSI/iSER connections.

   iSER - iSCSI Extensions for future
   reuse), RDMA, the I/O Buffer remains exposed protocol defined in this
      document.

   iSER-assisted - A term generally used to describe the network for access by
   the RCaP.  Such an I/O Buffer is capable operation of being read or written by
   the RCaP outside
      iSCSI when the scope of iSER functionality is also enabled below the iSCSI operation
      layer for which it was
   originally established, which has both robustness and security
   considerations.  The robustness considerations are that the system
   containing specific iSCSI/iSER connection in question.

   iSER-IRD - This variable represents the iSER initiator may react poorly to an unexpected
   modification maximum number of its memory.  For the security considerations, see
   Section 11.

2.5.2  Send

   Send is the incoming
      outstanding RDMA Operation Read Requests that is not addressed to an Advertised
   buffer, and uses Untagged buffers as the message is received.

   The iSER Layer layer at the
      initiator uses the Send Operation to transmit
   any iSCSI control-type PDU to grants on a particular RCaP Stream.

   iSER-ORD - This variable represents the target.  As an example, maximum number of outstanding
      RDMA Read Requests that the
   initiator uses Send Operations to transfer iSER Messages containing
   SCSI Command PDUs to layer can initiate on a
      particular RCaP Stream.  This variable is maintained only by the
      iSER Layer layer at the target.

   An

   iSER Layer - The layer at the target uses that implements the Send Operation to transmit any iSCSI control-type PDU to the initiator.  As an example, the target
   uses Send Operations to transfer iSER Messages containing SCSI
   Response PDUs to the iSER Layer at the initiator.

   For interoperability, iSER implementations SHOULD accept and
   correctly process SendSE Extensions for RDMA
      (iSER) protocol.

   iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and SendInvSE messages.  However, SendSE
      [MPA] when layered above [TCP].  [RDMAP] and SendInvSE messages are to [DDP] may be regarded as optimizations layered
      above SCTP or
   enhancements other transport protocols.

   Local Mapping - A task state record maintained by the iSER layer that
      associates the Initiator Task Tag to the basic Send message, and their support may vary
   by RCaP protocol and specific implementation.  In general, these
   messages SHOULD NOT be used, unless Local STag(s).  The
      specifics of the RCaP requires support for
   them in all implementations.  If these messages record structure are used, the implementation SHOULD be capable dependent.

   Local Peer - The implementation of reverting to use the RDMA-Capable Protocol on the
      local end of Send in
   order the connection.  Used to work with a receiver that does not support these message.
   Attempted use refer to the local entity
      when describing protocol exchanges or other interactions between
      two nodes.

   Node - A computing device attached to one or more links of these messages with a peer that network.
      A node in this context does not support
   them may result in refer to a fatal error that closes the RCaP connection.
   For example, these messages SHOULD NOT be used with the InfiniBand
   RCaP because InfiniBand does not require support for them in all
   cases.  New iSER implementations SHOULD use Send (and not SendSE specific application or
   SendInvSE) unless there are compelling reasons for doing otherwise.
   Similarly, iSER implementations SHOULD NOT rely
      protocol instantiation running on events triggered
   by SendSE and SendInvSE, as these messages the computer.  A node may not be used.

2.5.3  RDMA Write

   RDMA Write
      consist of one or more RDMA-Capable Controllers installed in a
      host computer.

   Operational Primitive - An Operational Primitive is the RDMA Operation an abstract
      functional interface procedure that is used requests another layer to place data into an
   Advertised buffer at
      perform a specific action on the Data Sink.  The Data Source addresses requestor's behalf or notifies
      the
   Message using other layer of some event.  The Datamover Interface between an STag
      iSCSI layer and a Tagged Offset that are valid on the Data
   Sink.

   The iSER Layer at the target Datamover layer within an iSCSI end node uses the RDMA Write Operation to
   transfer the contents of a local I/O Buffer
      set of Operational Primitives to an Advertised I/O
   Buffer at define the initiator.  The iSER Layer at functional interface
      between the target uses two layers.  Note that not every invocation of an
      Operational Primitive may elicit a response from the RDMA
   Write to transfer whole or part requested
      layer.  A full discussion of the data required Operational Primitive types and
      request-response semantics available to complete a
   SCSI Read command.

   The iSCSI and iSER Layer at the initiator does not employ RDMA Writes.

2.5.4 can be
      found in [DA].

   Outbound RDMA Read

   RDMA Read is the RDMA Operation that is used to retrieve data from
   an Advertised buffer at the Data Source. Queue Depth (ORD) - The sender maximum number of the
      outstanding RDMA Read Request addresses the Message using an STag and a Tagged Offset Requests that are valid on the Data Source in addition to providing RDMA-Capable Controller
      can initiate on a valid
   local STag and Tagged Offset that identify particular RCaP Stream at the Data Sink.

   The iSER Layer at the target uses the RDMA Read Operation to
   transfer the contents of an Advertised I/O Buffer at  For
      some RDMA-Capable Protocol layer, the initiator
   to term "ORD" may be known by a local I/O Buffer at the target.  The iSER Layer at the target
   uses
      different name.  For example, for InfiniBand, the RDMA Read equivalent to fetch whole or part of
      ORD is the data required Initiator Depth.

   Phase Collapse - Refers to
   complete a SCSI Write Command.

   The iSER Layer at the initiator does not employ RDMA Reads.

2.6  SCSI Read Overview

   The iSER Layer at optimization in iSCSI where the initiator receives SCSI
      status is transferred along with the final SCSI Command Data-In PDU from
   the iSCSI Layer.  The iSER Layer at the initiator generates an STag
   for the I/O Buffer a
      target.  See Section 4.2 in [iSCSI].

   RCaP Message - One or more packets of the SCSI Read and Advertises the buffer by
   including the STag and the Base Offset as network layer that
      constitute a single RDMA operation or a part of an RDMA Read
      Operation of the iSER header
   for the PDU.  The iSER RDMA-Capable Protocol.  For iWARP, an RCaP
      Message is transferred to the target using a
   Send Message.  The SendSE Message should be used if supported by known as an RDMAP Message.

   RCaP Stream - A single bidirectional association between the peer
      RDMA-Capable Protocol layers on two nodes over a single transport-
      level stream.  For iWARP, an RCaP layer (e.g., iWARP).

   The iSER Layer at Stream is known as an RDMAP
      Stream, and the target uses one association is created following a successful
      Login Phase during which iSER support is negotiated.

   RDMA-Capable Protocol (RCaP) - The protocol or more protocol suite that
      provides a reliable RDMA Writes transport functionality, e.g., iWARP,
      InfiniBand, etc.

   RDMA-Capable Controller - A network I/O adapter or embedded
      controller with RDMA functionality.  For example, for iWARP, this
      could be an RNIC, and for InfiniBand, this could be a HCA (Host
      Channel Adapter) or TCA (Target Channel Adapter).

   RDMA-enabled Network Interface Controller (RNIC) - A network I/O
      adapter or embedded controller with iWARP functionality.

   RDMA Operation - A sequence of RCaP Messages, including control
      messages, to transfer the data required from a Data Source to complete the SCSI Read.

   The iSER Layer at the target uses a Data Sink.  The
      following RDMA Operations are defined -- RDMA Write Operation,
      RDMA Read Operation, and Send Message Operation.

   RDMA Protocol (RDMAP) - A wire protocol that supports RDMA Operations
      to transfer ULP data between a Local Peer and the
   SCSI Response PDU back Remote Peer as
      described in [RDMAP].

   RDMA Read Operation - An RDMA Operation used by the Data Sink to
      transfer the iSER Layer at contents of a Data Source buffer from the initiator.  The iSER
   Layer Remote Peer
      to a Data Sink buffer at the initiator invalidates the STag and notifies the iSCSI
   Layer of the availability Local Peer.  An RDMA Read operation
      consists of the SCSI a single RDMA Read Request Message and a single RDMA
      Read Response PDU.  The Send with
   Invalidate Message, if supported by the Message.

   RDMA Read Request - An RCaP layer (e.g., iWARP),
   can be Message used for automatic invalidation of the STag.

2.7  SCSI Write Overview

   The iSER Layer at the initiator receives by the SCSI Command PDU from Data Sink to request
      the iSCSI Layer.  If solicited data Data Source to transfer is involved, the iSER
   Layer at the initiator generates an STag for the I/O Buffer contents of a buffer.  The RDMA
      Read Request Message describes both the
   SCSI Write Data Source and Advertises the buffer Data
      Sink buffers.

   RDMA Read Response - An RCaP Message used by including the STag and Data Source to
      transfer the
   Base Offset as part contents of the iSER header for the PDU.  The iSER
   Message is transferred a buffer to the target using a Send Message. Data Sink, in response to
      an RDMA Read Request.  The
   SendSE RDMA Read Response Message should be only
      describes the Data Sink buffer.

   RDMA Write Operation - An RDMA Operation used if supported by the RCaP layer (e.g.,
   iWARP).

   The iSER Layer at the initiator may optionally send one or more non-
   immediate unsolicited data PDUs Data Source to the target using Send Messages.

   If solicited data
      transfer is involved, the iSER Layer at the target
   uses one or more RDMA Reads to transfer contents of a Data Source buffer from the data required Local Peer
      to
   complete a Data Sink buffer at the SCSI Write. Remote Peer.  The iSER Layer at RDMA Write Message
      only describes the target uses Data Sink buffer.

   Remote Direct Memory Access (RDMA) - A method of accessing memory on
      a Send Message remote system in which the local system specifies the remote
      location of the data to transfer be transferred.  Employing an RDMA-
      Capable Controller in the
   SCSI Response PDU back remote system allows the access to take
      place without interrupting the iSER Layer at processing of the CPU(s) on the
      system.

   Remote Mapping - A task state record maintained by the initiator.  The iSER
   Layer at layer
      that associates the initiator invalidates Initiator Task Tag to the STag Advertised STag(s)
      and notifies the iSCSI
   Layer Base Offset(s).  The specifics of the availability record structure are
      implementation dependent.

   Remote Peer - The implementation of the RDMA-Capable Protocol on the
      opposite end of the connection.  Used to refer to the remote
      entity when describing protocol exchanges or other interactions
      between two nodes.

   SCSI Response PDU.  The Layer - This layer builds/receives SCSI CDBs (Command Descriptor
      Blocks) and sends/receives them with the remaining command execute
      [SAM5] parameters to/from the iSCSI layer.

   Send - An RDMA Operation that transfers the content of a buffer from
      the Local Peer to an untagged buffer at the Remote Peer.

   SendInvSE Message - A Send with Solicited Event and Invalidate Message, if supported by the RCaP layer (e.g., iWARP),
   can be used
      Message.

   SendSE Message - A Send with Solicited Event Message.

   Sequence Number (SN) - DataSN for automatic invalidation a SCSI Data-In PDU and R2TSN for an
      R2T PDU.  The semantics for both types of the STag.

3  Upper Layer sequence numbers are as
      defined in [iSCSI].

   Session, iSCSI Session - The group of connections that link an
      initiator SCSI port with a target SCSI port form an iSCSI session
      (equivalent to a SCSI Initiator-Target (I-T) nexus).  Connections
      can be added to and removed from a session even while the I-T
      nexus is intact.  Across all connections within a session, an
      initiator sees one and the same target.

   Steering Tag (STag) - An identifier of a Tagged Buffer on a node
      (Local or Remote) as defined in [RDMAP] and [DDP].  For other
      RDMA-Capable Protocols, the Steering Tag may be known by different
      names but will be referred to herein as STags.  For example, for
      InfiniBand, a Remote STag is known as an R-Key, and a Local STag
      is known as an L-Key, and both will be considered STags.

   Tagged Buffer - A buffer that is explicitly Advertised to the iSER
      layer at the remote node through the exchange of an STag, Base
      Offset, and length.

   Tagged Offset - The offset within a Tagged Buffer.

   Traditional iSCSI - Refers to the iSCSI protocol as defined in
      [iSCSI] (i.e., without the iSER enhancements).

   Untagged Buffer - A buffer that is not explicitly Advertised to the
      iSER layer at the remode node.

2.2.  Acronyms

   Acronym        Definition

   --------------------------------------------------------------

   AHS            Additional Header Segment

   BHS            Basic Header Segment

   CO             Connection Only

   CRC            Cyclic Redundancy Check

   DDP            Direct Data Placement Protocol

   DI             Datamover Interface

   HCA            Host Channel Adapter

   IANA           Internet Assigned Numbers Authority
   IB             InfiniBand

   IETF           Internet Engineering Task Force

   I/O            Input - Output

   IO             Initialize Only

   IP             Internet Protocol

   IPoIB          IP over InfiniBand

   IPsec          Internet Protocol Security

   iSER           iSCSI Extensions for RDMA

   ITT            Initiator Task Tag

   LO             Leading Only

   MPA            Marker PDU Aligned Framing for TCP

   NOP            No Operation

   NSG            Next Stage (during the iSCSI Login Phase)

   PDU            Protocol Data Unit

   R2T            Ready To Transfer

   R2TSN          Ready To Transfer Sequence Number

   RCaP           RDMA-Capable Protocol

   RDMA           Remote Direct Memory Access

   RDMAP          Remote Direct Memory Access Protocol

   RFC            Request For Comments

   RNIC           RDMA-enabled Network Interface Controller

   SAM5           SCSI Architecture Model - 5

   SCSI           Small Computer System Interface
   SNACK          Selective Negative Acknowledgment - also

                  Sequence Number Acknowledgement for data

   STag           Steering Tag

   SW             Session Wide

   TCA            Target Channel Adapter

   TCP            Transmission Control Protocol

   TMF            Task Management Function

   TTT            Target Transfer Tag

   ULP            Upper Level Protocol

2.3.  Conventions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

3.  Upper-Layer Interface Requirements

   This section discusses the upper layer upper-layer interface requirements in the
   form of an abstract model of the required interactions between the
   iSCSI Layer layer and the iSER Layer. layer.  The abstract model used here is
   derived from the architectural model described in [DA].  [DA] also
   provides a functional overview of the interactions between the iSCSI
   Layer
   layer and the datamover Datamover layer as intended by the Datamover
   Architecture.

   The interface requirements are specified by Operational Primitives.
   An Operational Primitive is an abstract functional interface
   procedure between the iSCSI Layer layer and the iSER Layer layer that requests
   one layer to perform a specific action on behalf of the other layer
   or notifies the other layer of some event.  Whenever an Operational
   Primitive in invoked, the Connection_Handle qualifier is used to
   identify a particular iSCSI connection.  For some Operational
   Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI data
   buffer associated with the requested or completed operation.

   The abstract model and the Operational Primitives defined in this
   section facilitate the description of the iSER protocol.  In the rest
   of the iSER specification, the compliance statements related to the
   use of these Operational Primitives are only for the purpose of the
   required interactions between the iSCSI Layer layer and the iSER
   Layer. layer.
   Note that the compliance statements related to the Operational
   Primitives in the rest of this specification only mandate functional
   equivalence on implementations, but do not put any requirements on
   the implementation specifics of the interface between the iSCSI Layer layer
   and the iSER Layer. layer.

   Each Operational Primitive is invoked with a set of qualifiers which
   specify the information context for performing the specific action
   being requested of the Operational Primitive.  While the qualifiers
   are required, the method of realizing the qualifiers (e.g., by
   passing synchronously with invocation, or by retrieving from task
   context, or by retrieving from shared memory, etc.) is implementation
   dependent.

3.1

3.1.  Operational Primitives offered by iSER

   The iSER protocol layer MUST support the following Operational
   Primitives to be used by the iSCSI protocol layer.

3.1.1

3.1.1.  Send_Control

      Input qualifiers:  Connection_Handle, BHS and AHS (if any) of the
      iSCSI PDU, PDU-specific qualifiers

      Return results:  Not specified

   This is used by the iSCSI Layers layers at the initiator and the target to
   request the outbound transfer of an iSCSI control-type PDU (see
   section
   Section 7.2).  Qualifiers that only apply for a particular control-
   type PDU are known as PDU-specific qualifiers, e.g.,
   ImmediateDataSize for a SCSI Write command.  For details on PDU-
   specific qualifiers, see section Section 7.3.  The iSCSI Layer layer can only
   invoke the Send_Control Operational Primitive when the connection is
   in iSER-assisted mode.

3.1.2

3.1.2.  Put_Data

      Input qualifiers:  Connection_Handle, content of a SCSI Data-in Data-In
      PDU header, Data_Descriptor, Notify_Enable

      Return results:  Not specified

   This is used by the iSCSI Layer layer at the target to request the outbound
   transfer of data for a SCSI Data-in Data-In PDU from the buffer identified by
   the Data_Descriptor qualifier.  The iSCSI Layer layer can only invoke the
   Put_Data Operational Primitive when the connection is in iSER-assisted iSER-
   assisted mode.

   The Notify_Enable qualifier is used to indicate to the iSER Layer layer
   whether or not it should generate an eventual local completion
   notification to the iSCSI Layer. layer.  See section Section 3.2.2 on
   Data_Completion_Notify for details.

3.1.3

3.1.3.  Get_Data

      Input qualifiers:  Connection_Handle, content of an R2T PDU,
      Data_Descriptor, Notify_Enable

      Return results:  Not specified

   This is used by the iSCSI Layer layer at the target to request the inbound
   transfer of solicited data requested by an R2T PDU into the buffer
   identified by the Data_Descriptor qualifier.  The iSCSI Layer layer can
   only invoke the Get_Data Operational Primitive when the connection is
   in iSER-assisted mode.

   The Notify_Enable qualifier is used to indicate to the iSER Layer layer
   whether or not it should generate the eventual local completion
   notification to the iSCSI Layer. layer.  See section Section 3.2.2 on
   Data_Completion_Notify for details.

3.1.4

3.1.4.  Allocate_Connection_Resources

      Input qualifiers:  Connection_Handle, Resource_Descriptor
      (optional)

      Return results:  Status

   This is used by the iSCSI Layers layers at the initiator and the target to
   request the allocation of all connection resources necessary to
   support RCaP for an operational iSCSI/iSER connection.  The iSCSI
   Layer
   layer may optionally specify the implementation-specific resource
   requirements for the iSCSI connection using the Resource_Descriptor
   qualifier.

   A return result of Status=success means the invocation succeeded, and
   a return result of Status=failure means that the invocation failed.
   If the invocation is for a Connection_Handle for which an earlier
   invocation succeeded, the request will be ignored by the iSER Layer layer
   and the result of Status=success will be returned.  Only one
   Allocate_Connection_Resources Operational Primitive invocation can be
   outstanding for a given Connection_Handle at any time.

3.1.5

3.1.5.  Deallocate_Connection_Resources

      Input qualifiers:  Connection_Handle

      Return results:  Not specified

   This is used by the iSCSI Layers layers at the initiator and the target to
   request the deallocation of all connection resources that were
   allocated earlier as a result of a successful invocation of the
   Allocate_Connection_Resources Operational Primitive.

3.1.6

3.1.6.  Enable_Datamover

      Input qualifiers:  Connection_Handle,
      Transport_Connection_Descriptor, Final Login_Response_PDU
      (optional)

      Return results:  Not specified

   This is used by the iSCSI Layers layers at the initiator and the target to
   request that iSER-assisted mode be used for the connection.  The
   Transport_Connection_Descriptor qualifier is used to identify the
   specific connection associated with the Connection_Handle.  The iSCSI
   layer can only invoke the Enable_Datamover Operational Primitive when
   there was a corresponding prior resource allocation.

   The Final_Login_Response_PDU input qualifier is applicable only for a target,
   target and contains the final Login Response PDU that concludes the
   iSCSI Login Phase.

3.1.7

3.1.7.  Connection_Terminate

      Input qualifiers:  Connection_Handle

      Return results:  Not specified

   This is used by the iSCSI Layers layers at the initiator and the target to
   request that a specified iSCSI/iSER connection be terminated and all
   associated connection and task resources be freed.  When this
   Operational Primitive invocation returns to the iSCSI layer, the
   iSCSI layer may assume full ownership of all iSCSI-level resources,
   e.g.
   e.g., I/O Buffers, associated with the connection.

3.1.8

3.1.8.  Notice_Key_Values

      Input qualifiers:  Connection_Handle, number of keys, list of
       Key-Value Key-
      Value pairs

      Return results:  Not specified

   This is used by the iSCSI Layers layers at the initiator and the target to
   request the iSER Layer layer to take note of the specified Key-Value pairs
   which
   that were negotiated by the iSCSI peers for the connection.

3.1.9

3.1.9.  Deallocate_Task_Resources

      Input qualifiers:  Connection_Handle, ITT

      Return results:  Not specified

   This is used by the iSCSI Layers layers at the initiator and the target to
   request the deallocation of all RCaP-specific resources allocated by
   the iSER Layer layer for the task identified by the ITT qualifier.  The
   iSER Layer layer may require a certain number of RCaP-specific resources
   associated with the ITT for each new iSCSI task.  In the normal
   course of execution, these task-level resources in the iSER Layer layer are
   assumed to be transparently allocated on each task initiation and
   deallocated on the conclusion of each task as appropriate.  In
   exception scenarios where the task does not conclude with a SCSI
   Response PDU, the iSER Layer layer needs to be notified of the individual
   task terminations to aid its task-level resource management.  This
   Operational Primitive is used for this purpose, purpose and is not needed when
   a SCSI Response PDU normally concludes a task.  Note that RCaP-
   specific task resources are deallocated by the iSER Layer layer when a SCSI
   Response PDU normally concludes a task, even if the SCSI Status status was
   not success.

3.2

3.2.  Operational Primitives used Used by iSER

   The iSER layer MUST use the following Operational Primitives offered
   by the iSCSI protocol layer when the connection is in iSER-assisted
   mode.

3.2.1

3.2.1.  Control_Notify

      Input qualifiers:  Connection_Handle, an iSCSI control-type PDU

      Return results:  Not specified

   This is used by the iSER Layers layers at the initiator and the target to
   notify the iSCSI Layer layer of the availability of an inbound iSCSI
   control-type PDU.  A PDU is described as "available" to the iSCSI
   Layer
   layer when the iSER Layer layer notifies the iSCSI Layer layer of the reception
   of that inbound PDU, along with an implementation-specific indication
   as to where the received PDU is.

3.2.2

3.2.2.  Data_Completion_Notify

      Input qualifiers:  Connection_Handle, ITT, SN

      Return results:  Not specified

   This is used by the iSER Layer layer to notify the iSCSI Layer layer of the
   completion of the outbound data transfer that was requested by the
   iSCSI
   Layer layer only if the invocation of the Put_Data Operational
   Primitive (see section Section 3.1.2) was qualified with Notify_Enable set.
   SN refers to the DataSN associated with the SCSI Data-In PDU.

   This is used by the iSER Layer layer to notify the iSCSI Layer layer of the
   completion of the inbound data transfer that was requested by the
   iSCSI
   Layer layer only if the invocation of the Get_Data Operational
   Primitive (see section Section 3.1.3) was qualified with Notify_Enable set.
   SN refers to the R2TSN associated with the R2T PDU.

3.2.3

3.2.3.  Data_ACK_Notify

      Input qualifier:  Connection_Handle, ITT, DataSN

      Return results:  Not specified

   This is used by the iSER Layer layer at the target to notify the iSCSI
   Layer
   layer of the arrival of the data acknowledgement (as defined in
   [iSCSI]) requested earlier by the iSCSI Layer layer for the outbound data
   transfer via an invocation of the Put_Data Operational Primitive
   where the A-bit in the SCSI Data-in Data-In PDU is set to 1. one.  See section Section
   7.3.5.  DataSN refers to the expected DataSN of the next SCSI Data-
   in Data-In
   PDU which that immediately follows the SCSI Data-in Data-In PDU with the A-bit set
   to which this notification corresponds, with semantics as defined in
   [iSCSI].

3.2.4

3.2.4.  Connection_Terminate_Notify

      Input qualifiers:  Connection_Handle

      Return results:  Not specified

   This is used by the iSER Layers layers at the initiator and the target to
   notify the iSCSI Layer layer of the unsolicited termination or failure of
   an iSCSI/iSER connection.  The iSER Layer layer MUST deallocate the
   connection and task resources associated with the terminated
   connection before the invocation of this Operational Primitive.  Note
   that the Connection_Terminate_Notify Operational Primitive is not
   invoked when the termination of the connection was earlier requested
   by the local iSCSI Layer.

3.3 layer.

3.3.  iSCSI Protocol Usage Requirements

   To operate in an iSER-assisted mode, the iSCSI Layers layers at both the
   initiator and the target MUST negotiate the RDMAExtensions key (see
   section
   Section 6.3) to "Yes" on the leading connection.  If the
   RDMAExtensions key is not negotiated to "Yes", then iSER-assisted
   mode MUST NOT be used.  If the RDMAExtensons key is negotiated to
   "Yes"
   "Yes", but the invocation of the Allocate_Connection_Resources
   Operational Primitive to the iSER layer fails, the iSCSI layer MUST
   fail the iSCSI Login process or terminate the connection as
   appropriate.  See section Section 10.1.3.1 for details.

   If the RDMAExtensions key is negotiated to "Yes", the iSCSI Layer layer
   MUST satisfy the following protocol usage requirements from the iSER
   protocol:

   1.  The iSCSI Layer layer at the initiator MUST set ExpDataSN to 0 zero in
       Task Management Function Requests for Task Allegiance
       Reassignment for read/bidirectional commands, so as to cause the
       target to send all unacknowledged read data.

   2.  The iSCSI Layer layer at the target MUST always return the SCSI status
       in a separate SCSI Response PDU for read commands, i.e., there
       MUST NOT be a "phase collapse" in concluding a SCSI Read Command.

   3.  The iSCSI Layers layers at both the initiator and the target MUST
       support the keys as defined in section Section 6 on Login/Text
       Operational Keys.  If used as specified, these keys MUST NOT be
       answered with NotUnderstood NotUnderstood, and the semantics as defined MUST be
       followed for each iSER-assisted connection.

   4.  The iSCSI Layer layer at the initiator MUST NOT issue SNACKs for PDUs.

4  Lower Layer

4.  Lower-Layer Interface Requirements

4.1

4.1.  Interactions with the RCaP Layer

   The iSER protocol layer is layered on top of an RCaP layer (see
   Figure 1) and the following are the key features that are assumed to
   be supported by any RCaP layer:

   *  The RCaP layer supports all basic RDMA operations, including the
      RDMA Write Operation, RDMA Read Operation, and Send Operation.

   *  The RCaP layer provides reliable, in-order message delivery and
      direct data placement.

   *  When the iSER Layer layer initiates an RDMA Read Operation following an
      RDMA Write Operation on one RCaP Stream, the RDMA Read Response
      Message processing on the remote node will be started only after
      the preceding RDMA Write Message payload is placed in the memory
      of the remote node.

   *  The RCaP layer encapsulates a single iSER Message into a single
      RCaP Message on the Data Source side.  The RCaP layer decapsulates
      the iSER Message before delivering it to the iSER
      Layer layer on the
      Data Sink side.

   *  For a an RCaP layer that supports the Send with Invalidate Message
      (e.g., iWARP), when the iSER Layer layer provides the STag to be
      remotely invalidated to the RCaP layer for a Send with Invalidate
      Message, the RCaP layer uses this STag as the STag to be
      invalidated in the Send with Invalidate Message.

   *  The RCaP layer uses the STag and Tagged Offset provided by the
      iSER Layer layer for the RDMA Write and RDMA Read Request Messages.

   *  When the RCaP layer delivers the content of an RDMA Send Message
      to the iSER Layer, layer, the RCaP layer provides the length of the RDMA
      Send message. Message.  This ensures that the iSER Layer layer does not have to
      carry a length field in the iSER header.

   *  When the RCaP layer delivers the Send Message to the iSER Layer, layer,
      it notifies the iSER Layer layer with the mechanism provided on that
      interface.

   *  For a an RCaP layer that supports the Send with Invalidate Message
      (e.g., iWARP), when the RCaP layer delivers a Send with Invalidate
      Message to the iSER Layer, layer, it passes the value of the STag that
      was invalidated.

   *  The RCaP layer propagates all status and error indications to the
      iSER Layer. layer.

   *  For a transport layer that operates in byte stream mode such as
      TCP, the RCaP implementation supports the enabling of the RDMA
      mode after Connection connection establishment and the exchange of Login
      parameters in byte stream mode.  For a transport layer that
      provides message delivery capability such as [IB], the RCaP
      implementation supports the direct use of the messaging capability
      by the iSCSI Layer directly layer for the Login phase Phase after connection
      establishment and before enabling iSER-assisted mode.  (In the
      specific example of InfiniBand [IB], the iSCSI Layer layer uses IB
      messages to transfer iSCSI PDUs for the Login phase Phase after
      connection establishment and before enabling iSER-assisted mode.)

   *  Whenever the iSER Layer layer terminates the RCaP Stream, the RCaP layer
      terminates the associated Connection.

4.2 connection.

4.2.  Interactions with the Transport Layer

   After the iSER connection is established, the RCaP layer and the
   underlying transport layer are responsible for maintaining the
   Connection
   connection and reporting to the iSER Layer layer any Connection connection failures.

5

5.  Connection Setup and Termination

5.1

5.1.  iSCSI/iSER Connection Setup

   During connection setup, the iSCSI Layer layer at the initiator is
   responsible for establishing a connection with the target.  After the
   connection is established, the iSCSI Layers layers at the initiator and the
   target enter the Login Phase using the same rules as outlined in
   [iSCSI].  The connection transitions into the iSCSI full feature
   phase Full Feature
   Phase in iSER-assisted mode following a successful login negotiation
   between the initiator and the target in which iSER-assisted mode is
   negotiated and the connection resources necessary to support RCaP
   have been allocated at both the initiator and the target.  The same
   connection MUST be used for both the iSCSI Login phase Phase and the
   subsequent iSER-assisted full feature phase. Full Feature Phase.

   For a transport layer that operates in byte stream mode such as TCP,
   the RCaP implementation supports the enabling of the RDMA mode after
   Connection
   connection establishment and the exchange of Login parameters in byte
   stream mode.  For a transport layer that provides message delivery
   capability such as [IB], the RCaP implementation supports the use of
   the messaging capability by the iSCSI Layer layer directly for the Login phase
   Phase after connection establishment before enabling iSER-
   assisted iSER-assisted
   mode.

   iSER-assisted mode MUST NOT be enabled unless it is negotiated on the
   leading connection during the LoginOperationalNegotiation Stage stage of
   the iSCSI Login Phase.  iSER-assisted mode is negotiated using the
   RDMAExtensions=<boolean-value> key.  Both the initiator and the
   target MUST exchange the RDMAExtensions key with the value set to
   "Yes" to enable iSER-assisted mode.  If both the initiator and the
   target fail to negotiate the RDMAExtensions key set to "Yes", then
   the connection MUST continue with the login semantics as defined in
   [iSCSI].  If the RDMAExtensions key is not negotiated to Yes, then
   for some RCaP implementation (such as [IB]), the existing connection
   may need to be torn down and a new connection may need to be
   established in TCP capable TCP-capable mode.  (For InfiniBand InfiniBand, this will require
   an [IPoIB] type connection.)
   a connection like [IPoIB].)

   iSER-assisted mode is defined for a Normal session only only, and the
   RDMAExtensions key MUST NOT be negotiated for a Discovery session.
   Discovery sessions are always conducted using the transport layer as
   described in [iSCSI].

   An iSER enabled iSER-enabled node is not required to initiate the RDMAExtensions
   key exchange if its preference is for the Traditional iSCSI mode.
   The RDMAExtensions key, if offered, MUST be sent in the first
   available Login Response or Login Request PDU in the
   LoginOperationalNegotiation stage.  This is due to the fact that the
   value of some login Login parameters might depend on whether iSER-assisted or not iSER-
   assisted mode is enabled or not. enabled.

   iSER-assisted mode is a session-wide attribute.  If both the
   initiator and the target negotiated RDMAExtensions="Yes" on the
   leading connection of a session, then all subsequent connections of
   the same session MUST enable iSER-assisted mode without having to
   exchange RDMAExtensions key keys during the iSCSI Login Phase.
   Conversely, if both the initiator and the target failed to negotiate
   RDMAExtensions to "Yes" on the leading connection of a session, then
   the RDMAExtensions key MUST NOT be negotiated further on any
   additional subsequent connection of the session.

   When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest
   and the DataDigest keys MUST be negotiated to "None" on all
   iSCSI/iSER connections participating in that iSCSI session.  This is
   because, for an iSCSI/iSER connection, RCaP is responsible for
   providing error detection that is at least as good as a 32-bit CRC
   for all iSER Messages.  Furthermore, all SCSI Read data are sent
   using RDMA Write Messages instead of the SCSI Data-in Data-In PDUs, and all
   solicited SCSI write Write data are sent using RDMA Read Response Messages
   instead of the SCSI Data-out Data-Out PDUs.  HeaderDigest and DataDigest
   which that
   apply to iSCSI PDUs would not be appropriate for RDMA Read and RDMA
   Write operations used with iSER.

5.1.1

5.1.1.  Initiator Behavior

   If the outcome of the iSCSI negotiation is to enable iSER-assisted
   mode, then on the initiator side, prior to sending the Login Request
   with the T (Transit) bit set to 1 one and the NSG (Next Stage) field
   set to FullFeaturePhase, the iSCSI Layer layer SHOULD request the iSER Layer
   layer to allocate the connection resources necessary to support RCaP
   by invoking the Allocate_Connection_Resources Operational Primitive.
   The connection resources required are defined by the implementation
   and are outside the scope of this specification.  The iSCSI Layer layer may
   invoke the Notice_Key_Values Operational Primitive before invoking
   the Allocate_Connection_Resources Operational Primitive to request
   the iSER Layer layer to take note of the negotiated values of the iSCSI
   keys for the Connection. connection.  The specific keys to be passed in as input
   qualifiers are implementation dependent.  These may include, but are
   not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. MaxOutstandingR2T and ErrorRecoveryLevel.

   Among the connection resources allocated at the initiator is the
   Inbound RDMA Read Queue Depth (IRD).  As described in section Section 9.5.1,
   R2Ts are transformed by the target into RDMA Read operations.  IRD
   limits the maximum number of simultaneously incoming outstanding RDMA
   Read Requests per an RCaP Stream from the target to the initiator.
   The required value of IRD is outside the scope of the iSER
   specification.  The iSER Layer layer at the initiator MUST set IRD to 1 or
   higher if R2Ts are to be used in the connection.  However, the iSER Layer
   layer at the initiator MAY set IRD to 0 zero based on implementation
   configuration which
   configuration; setting IRD to zero indicates that no R2Ts will be
   used on that connection.  Initially, the iSER-IRD value at the
   initiator SHOULD be set to the IRD value at the initiator and MUST
   NOT be more than the IRD value.

   On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be
   set to 0 zero since the iSER Layer layer at the initiator does not issue RDMA
   Read Requests to the target.

   Failure to allocate the requested connection resources locally
   results in a login failure failure, and its handling is described in section Section
   10.1.3.1.

   The iSER Layer layer MUST return a success status to the iSCSI Layer layer in
   response to the Allocate_Connection_Resources Operational Primitive.

   After the target returns the Login Response with the T bit set to 1 one
   and the NSG field set to FullFeaturePhase, and a status class Status-Class of 0 0x00
   (Success), the iSCSI Layer layer MUST invoke the Enable_Datamover
   Operational Primitive with the following qualifiers.  (See section Section
   10.1.4.6 for the case when the status class Status-Class is not Success.): Success.)

      a. Connection_Handle that identifies the iSCSI connection.

      b. Transport_Connection_Descriptor which that identifies the specific
         transport connection associated with the Connection_Handle.

   The iSER Layer layer MUST send the iSER Hello Message as the first iSER
   Message only if iSERHelloRequired is negotiated to "Yes".  See
   Section 5.1.3 on iSER Hello Exchange.

   If the iSCSI Layer layer on the initiator side allocates the connection
   resources to support RCaP only after it receives the final Login
   Response PDU from the target, then it may not be able to handle the
   number of unexpected iSCSI control-type PDUs (as declared by the
   MaxOutstandingUnexpectedPDUs key from the initiator) that can be sent
   by the target before the buffer resources are allocated at the
   initiator side.  In this case case, the iSERHelloRequired key SHOULD be
   negotiated to "Yes" so that the initiator can allocate the connection
   resources before sending the iSER Hello Message.  See
   section Section 5.1.3
   for more details.

5.1.2

5.1.2.  Target Behavior

   If the outcome of the iSCSI negotiation is to enable iSER-assisted
   mode, then on the target side, prior to sending the Login Response
   with the T (Transit) bit set to 1 one and the NSG (Next Stage) field
   set to FullFeaturePhase, the iSCSI Layer layer MUST request the iSER Layer layer
   to allocate the resources necessary to support RCaP by invoking the
   Allocate_Connection_Resources Operational Primitive.  The connection
   resources required are defined by implementation and are outside the
   scope of this specification.  Optionally, the iSCSI Layer layer may invoke
   the Notice_Key_Values Operational Primitive before invoking the
   Allocate_Connection_Resources Operational Primitive to request the
   iSER Layer layer to take note of the negotiated values of the iSCSI keys
   for the Connection. connection.  The specific keys to be passed in as input
   qualifiers are implementation dependent.  These may include, but not
   limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. MaxOutstandingR2T and ErrorRecoveryLevel.

   Premature allocation of RCaP connection resources can expose an iSER
   target to a resource exhaustion attack on those resources via
   multiple iSER connections that progress only to the point at which
   the implementation allocates the RCaP connection resources.  The
   countermeasure for this attack is initiator authentication; the iSCSI Layer
   layer MUST NOT request the iSER Layer layer to allocate the connection
   resources necessary to support RCaP until the iSCSI layer is
   sufficiently far along in the iSCSI Login Phase that it is reasonably
   certain that the peer side is not an attacker.  In particular, if the
   Login Phase includes a SecurityNegotiation stage, the iSCSI Layer layer
   MUST defer the connection resource allocation (i.e. (i.e., invoking the
   Allocate_Connection_Resources Operational Primitive) to the
   LoginOperationalNegotiation stage ([iSCSI]) so that the resource
   allocation occurs after the authentication phase is completed.

   Among the connection resources allocated at the target is the
   Outbound RDMA Read Queue Depth (ORD).  As described in section Section 9.5.1,
   R2Ts are transformed by the target into RDMA Read operations.  The
   ORD limits the maximum number of simultaneously outstanding RDMA Read
   Requests per RCaP Stream from the target to the initiator.
   Initially, the iSER-ORD value at the target SHOULD be set to the ORD
   value at the target.

   On the other hand, the IRD at the target MAY be set to 0 zero since the
   iSER Layer layer at the target does not expect RDMA Read Requests to be
   issued by the initiator.

   Failure to allocate the requested connection resources locally
   results in a login failure failure, and its handling is described in section Section
   10.1.3.1.

   If the iSER Layer layer at the target is successful in allocating the
   connection resources necessary to support RCaP, the following events
   MUST occur in the specified sequence:

   1. The iSER Layer layer MUST return a success status to the iSCSI Layer layer in
      response to the Allocate_Connection_Resources Operational
      Primitive.

   2. The iSCSI Layer layer MUST invoke the Enable_Datamover Operational
      Primitive with the following qualifiers:

      a. Connection_Handle that identifies the iSCSI connection.

      b. Transport_Connection_Descriptor which that identifies the specific
         transport connection associated with the Connection_Handle.

      c. The final transport layer (e.g. transport-layer (e.g., TCP) message containing the
         Login Response with the T bit set to 1 one and the NSG field set
         to FullFeaturePhase FullFeaturePhase.

   3. The iSER Layer layer MUST send the final Login Response PDU in the
      native transport mode to conclude the iSCSI Login Phase.  If the
      underlying transport is TCP, then the iSER Layer layer MUST send the
      final Login Response PDU in byte stream mode.

   4. After receiving the iSER Hello Message from the initiator, the
      iSER Layer layer MUST respond with the iSER HelloReply Message to be
      sent as the first iSER Message if iSERHelloRequired is negotiated
      to "Yes".  If the iSER layer receives an iSER Hello Message when
      iSERHelloRequired is negotiated to "No", then this MUST be treated
      as an iSER protocol error.  See section Section 5.1.3 on iSER Hello
      Exchange for more details.

   Note: In the above sequence, the operations as described in bullets items 3
   and 4 MUST be performed atomically for iWARP connections.  Failure to
   do this may result in race conditions.

5.1.3

5.1.3.  iSER Hello Exchange

   If iSERHelloRequired is negotiated to "Yes", the first iSER Message
   sent by the iSER Layer layer at the initiator to the target MUST be the
   iSER Hello Message.  The iSER Hello Message is used by the iSER
   Layer layer
   at the initiator to declare iSER parameters to the target.  See section
   Section 9.3 on iSER Header Format for iSER Hello Message.
   Conversely, if iSERHelloRequired is negotiated to "No", then the iSER Layer
   layer at the initiator MUST NOT send an iSER Hello Message.

   In response to the iSER Hello Message, the iSER Layer layer at the target
   MUST return the iSER HelloReply Message as the first iSER Message
   sent by the target if iSERHelloRequired is negotiated to "Yes".  The
   iSER HelloReply Message is used by the iSER Layer layer at the target to
   declare iSER parameters to the initiator.  See section Section 9.4 on iSER
   Header Format for iSER HelloReply Message.  If the iSER layer
   receives an iSER Hello Message when iSERHelloRequired is negotiated
   to "No", then this MUST be treated as an iSER protocol error.  See
   section
   Section 10.1.3.4 on iSER Protocol Errors on for more details details.

   In the iSER Hello Message, the iSER Layer layer at the initiator declares
   the iSER-IRD value to the target.

   Upon receiving the iSER Hello Message, the iSER Layer layer at the target
   MUST set the iSER-ORD value to the minimum of the iSER-ORD value at
   the target and the iSER-IRD value declared by the initiator.  The  In
   order to free up the unused resources, the iSER Layer layer at the target
   MAY adjust (lower) its ORD value to match the iSER-ORD value if the
   iSER-ORD value is smaller than the ORD value at the target in order to free up the unused resources. target.

   In the iSER HelloReply Message, the iSER Layer layer at the target declares
   the iSER-ORD value to the initiator.

   Upon receiving the iSER HelloReply Message, the iSER Layer layer at the
   initiator MAY adjust (lower) its IRD value to match the iSER-ORD
   value in order to free up the unused resources, if the iSER-ORD value
   declared by the target is smaller than the iSER-IRD value declared by
   the initiator.

   It is an iSER level iSER-level negotiation failure if the iSER parameters
   declared in the iSER Hello Message by the initiator are unacceptable
   to the target.  This includes the following:

   *  The initiator-declared iSER-IRD value is greater than 0 0, and the
      target-declared iSER-ORD value is 0.

   *  The initiator-supported and the target-supported iSER protocol
      versions do not overlap.

   See section Section 10.1.3.2 on the handling of the error situation.

   An initiator that conforms to [RFC5046] allocates connection
   resources before seding sending the Login Request with the T (Transit) bit
   set to 1 one and the NSG (Next Stage) field set to FullFeaturePhase.
   (For brevity, this is referred to as "early" connection allocation.)
   The current iSER specification relaxes this requirement to allow an
   initiator to allocate connection resources after it receives the
   final Login Response PDU from the target.  (For brevity, this is
   referred to as "late" connection allocation.)  An initiator that
   employs "late" connection allocation may encounter problems (e.g.,
   RCaP connection closure) with a target that sends unexpected iSCSI
   PDUs immediately upon transitioning to Full Feature Phase, as allowed
   by the negotiated value of the MaxOustandingUnexpectedPDUs MaxOutstandingUnexpectedPDUs key.  The
   only way to prevent this situation in full generality is to use iSER
   Hello Messages, as they enable the initiator to allocate its
   connection resources before sending its iSER Hello Message.  The
   iSERHelloRequired key is used by the initiator to determine if it is
   dealing with a target that supports the iSER Hello exchanges.
   Fortunately, known iSER target implementations do not take full
   advantage of the number of allowed unexpected PDUs immediately upon
   transitioning into full feature phase, Full Feature Phase, thus enabling an initiator
   workaround that involves a smaller quantity of connection resources
   prior to full-feature phase, Full Feature Phase, as explained further below.

   In the following summary summary, where "late" connection allocation is
   practised,
   practiced, an initiator that follows [RFC5046] is referred to as an
   "old" initiator; otherwise otherwise, it is referred to as a "new" initiator.
   Similarly, a target that does not support the iSERHelloRequired key
   (and responds with "NotUnderstood" when negotiating the
   iSERHelloRequired key) is referred to as an "old" target; otherwise otherwise,
   it is referred to as a "new" target.  Note that an "old" target can
   still support the iSER Hello exchanges exchanges, but this fact is not known by
   the initiator.  A "new" target can also respond with "No" when
   negotiating the iSERHelloRequired key.  In this case case, its behavior
   with respect to "late" connection allocation is similar to an "old"
   target.

   A "new" initiator will work fine with a "new" target.

   For an "old" initiator and an "old" target, the failure by the
   initiator to handle the number of unexpected iSCSI control-type PDUs
   that are sent by the target before the buffer resources are allocated
   at the initiator can result in the failure of the iSER session caused
   by closure of the underlying RCaP connection.  For the "old" target,
   there is a known implementation that sends one unexpected iSCSI
   control-type PDU after sending the final Login Response and then
   waits awhile before sending the next one.  This tends to alleviate
   somewhat the buffer allocation problem at the initiator.

   For a "new" initiator and an "old" target, the failure by the
   initiator to handle the number of unexpected iSCSI control-type PDUs
   that are sent by the target before the buffer resources are allocated
   at the initiator can result in the failure of the iSER session caused
   by closure of the underlying RCaP connection.  A "new" initiator MAY
   choose to terminate the connection; otherwise otherwise, it SHOULD do one of
   the following:

   1. Allocate the connection resources before sending the final Login
      Request PDU.

   2. Allocate one or more buffers for receiving unexpected control-
      type control-type
      PDUs from the target before sending the final Login Request PDU.
      This reduces the possibility of the unexpected control-type PDUs
      causing the RCaP connection to close before the connection
      resources have been allocated.

   For an "old" initiator and a "new" target, if the iSERHelloRequired
   key is not negotiated, a "new" target MUST still respond with the
   iSER HelloReply Message when it receives the iSER Hello Message.  If
   the iSERHelloRequired key is negotiated to "No" or "NotUnderstood", a
   "new" target MAY choose to terminate the connection; otherwise otherwise, it
   SHOULD delay sending any unexpected control-type PDUs until one of
   the following events has occurred:

   1. A PDU is received from the initiator after it sends the final
      Login Response PDU.

   2. A system configurable system-configurable timeout period, say period (say, one second, second) has
      expired.

5.2

5.2.  iSCSI/iSER Connection Termination

5.2.1

5.2.1.  Normal Connection Termination at the Initiator

   The iSCSI Layer layer at the initiator terminates an iSCSI/iSER connection
   normally by invoking the Send_Control Operational Primitive qualified
   with the Logout Request PDU.  The iSER Layer layer at the initiator MUST
   use a Send Message to send the Logout Request PDU to the target.  The
   SendSE Message should be used if supported by the RCaP layer (e.g.,
   iWARP).  After the iSER Layer layer at the initiator receives the Send
   Message containing the Logout Response PDU from the target, it MUST
   notify the iSCSI Layer layer by invoking the Control_Notify Operational
   Primitive qualified with the Logout Response PDU.

   After the iSCSI logout process is complete, the iSCSI layer at the
   target is responsible for closing the iSCSI/iSER connection as
   described in Section 5.2.2.  After the RCaP layer at the initiator
   reports that the Connection connection has been closed, the iSER Layer layer at the
   initiator MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local Mappings (if
   any) before notifying the iSCSI Layer layer by invoking the
   Connection_Terminate_Notify Operational Primitive.

5.2.2

5.2.2.  Normal Connection Termination at the Target

   Upon receiving the Send Message containing the Logout Request PDU,
   the iSER Layer layer at the target MUST notify the iSCSI Layer layer at the
   target by invoking the Control_Notify Operational Primitive qualified
   with the Logout Request PDU.  The iSCSI Layer layer completes the logout
   process by invoking the Send_Control Operational Primitive qualified
   with the Logout Response PDU.  The iSER Layer layer at the target MUST use
   a Send Message to send the Logout Response PDU to the initiator.  The
   SendSE Message should be used if supported by the RCaP layer (e.g.,
   iWARP).  After the iSCSI logout process is complete, the iSCSI Layer layer
   at the target MUST request the iSER Layer layer at the target to terminate
   the RCaP Stream by invoking the Connection_Terminate Operational
   Primitive.

   As part of the termination process, the RCaP layer MUST close the
   Connection.
   connection.  When the RCaP layer notifies the iSER Layer layer after the
   RCaP Stream and the associated Connection connection are terminated, the iSER
   Layer
   layer MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local and Remote
   Mappings (if any).

5.2.3

5.2.3.  Termination without Logout Request/Response PDUs

5.2.3.1

5.2.3.1.  Connection Termination Initiated by the iSCSI Layer layer

   The Connection_Terminate Operational Primitive MAY be invoked by the
   iSCSI Layer layer to request the iSER Layer layer to terminate the RCaP Stream
   without having previously exchanged the Logout Request and Logout
   Response PDUs between the two iSCSI/iSER nodes.  As part of the
   termination process, the RCaP layer will close the Connection. connection.  When
   the RCaP layer notifies the iSER Layer layer after the RCaP Stream and the
   associated Connection connection are terminated, the iSER Layer layer MUST perform the
   following actions.

   If the Connection_Terminate Operational Primitive is invoked by the
   iSCSI Layer layer at the target, then the iSER Layer layer at the target MUST
   deallocate all connection and task resources (if any) associated with
   the connection, and invalidate the Local and Remote Mappings (if
   any).

   If the Connection_Terminate Operational Primitive is invoked by the
   iSCSI Layer layer at the initiator, then the iSER Layer layer at the initiator
   MUST deallocate all connection and task resources (if any) associated
   with the connection, and invalidate the Local Mappings (if any).

5.2.3.2

5.2.3.2.  Connection Termination Notification to the iSCSI Layer layer

   If the iSCSI/iSER connection is terminated without the invocation of
   Connection_Terminate from the iSCSI Layer, layer, the iSER Layer layer MUST notify
   the iSCSI Layer layer that the iSCSI/iSER connection has been terminated by
   invoking the Connection_Terminate_Notify Operational Primitive.

   Prior to invoking Connection_Terminate_Notify, the iSER Layer layer at the
   target MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local and Remote
   Mappings (if any).

   Prior to invoking Connection_Terminate_Notify, the iSER Layer layer at the
   initiator MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local Mappings (if
   any).

   If the remote iSCSI/iSER node initiated the closing of the
   Connection connection
   (e.g., by sending a TCP FIN or TCP RST), the iSER Layer layer MUST notify
   the iSCSI Layer layer after the RCaP layer reports that the
   Connection connection is
   closed by invoking the Connection_Terminate_Notify Operational
   Primitive.

   Another example of a Connection connection termination without a preceding
   logout is when the iSCSI Layer layer at the initiator does an implicit
   logout (connection reinstatement).

6

6.  Login/Text Operational Keys

   Certain iSCSI login/text operational keys have restricted usage in
   iSER, and additional keys are used to support the iSER protocol
   functionality.  All other keys defined in [iSCSI] and not discussed
   in this section may be used on iSCSI/iSER connections with the same
   semantics.

6.1

6.1.  HeaderDigest and DataDigest

   Irrelevant when: RDMAExtensions=Yes

   Negotiations resulting in RDMAExtensions=Yes for a session implies imply
   HeaderDigest=None and DataDigest=None for all connections in that
   session and overrides both override the settings, whether default and an explicit setting.

6.2 or configured.

6.2.  MaxRecvDataSegmentLength

   For an iSCSI connection belonging to a session in which
   RDMAExtensions=Yes was negotiated on the leading connection of the
   session, MaxRecvDataSegmentLength need not be declared in the Login
   Phase, and MUST be ignored if it is declared.  Instead  Instead,
   InitiatorRecvDataSegmentLength (as described in section Section 6.5) and
   TargetRecvDataSegmentLength (as described in section Section 6.4) keys are
   negotiated.  The values of the local and remote
   MaxRecvDataSegmentLength are derived from the
   InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys.

   In the full feature phase, Full Feature Phase, the initiator MUST consider the value of
   its local MaxRecvDataSegmentLength (that it would have declared to
   the target) as having the value of InitiatorRecvDataSegmentLength,
   and the value of the remote MaxRecvDataSegmentLength (that would have
   been declared by the target) as having the value of
   TargetRecvDataSegmentLength.  Similarly, the target MUST consider the
   value of its local MaxRecvDataSegmentLength (that it would have
   declared to the initiator) as having the value of
   TargetRecvDataSegmentLength, and the value of the remote
   MaxRecvDataSegmentLength (that would have been declared by the
   initiator) as having the value of InitiatorRecvDataSegmentLength.

   Note that RFC 3720 requires that when a target receives a NOP-Out
   request with a valid Initiator Task Tag, it responds with a NOP-In
   with the same Initiator Task Tag that was provided in the NOP-Out
   request.  Furthermore, it returns the first MaxRecvDataSegmentLength
   bytes of the initiator provided initiator-provided Ping Data.  Since there is no
   MaxRecvDataSegmentLength common to the initiator and the target in
   iSER, the length of the data sent with the NOP-Out request MUST NOT
   exceed InitiatorMaxRecvDataSegmentLength.

   The MaxRecvDataSegmentLength key is applicable only for iSCSI
   control-type PDUs.

6.3

6.3.  RDMAExtensions

   Use: LO (leading only)

   Senders: Initiator and Target

   Scope: SW (session-wide)

   RDMAExtensions=<boolean-value>

   Irrelevant when: SessionType=Discovery

   Default is No

   Result function is AND

   This key is used by the initiator and the target to negotiate the
   support for iSER-assisted mode.  To enable the use of iSER-assisted
   mode, both the initiator and the target MUST exchange
   RDMAExtensions=Yes.  iSER-assisted mode MUST NOT be used if either
   the initiator or the target offers RDMAExtensions=No.

   An iSER-enabled node is not required to initiate the RDMAExtensions
   key exchange if it prefers to operate in the Traditional iSCSI mode.
   However, if the RDMAExtensions key is to be negotiated, an initiator
   MUST offer the key in the first Login Request PDU in the
   LoginOperationalNegotiation stage of the leading connection, and a
   target MUST offer the key in the first Login Response PDU with which
   it is allowed to do so (i.e., the first Login Response PDU issued
   after the first Login Request PDU with the C bit set to 0) zero) in the
   LoginOperationalNegotiation stage of the leading connection.  In
   response to the offered key=value pair of RDMAExtensions=yes, an
   initiator MUST respond in the next Login Request PDU with which it is
   allowed to do so, and a target MUST respond in the next Login
   Response PDU with which it is allowed to do so.

   Negotiating the RDMAExtensions key first enables a node to negotiate
   the optimal value for other keys.  Certain iSCSI keys such as
   MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T,
   ImmediateData, etc., may be negotiated differently depending on
   whether the connection is in Traditional iSCSI mode or iSER-assisted
   mode.

6.4

6.4.  TargetRecvDataSegmentLength

   Use: IO (Initialize only)

   Senders: Initiator and Target

   Scope: CO (connection-only)

   Irrelevant when: RDMAExtensions=No

   TargetRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>

   Default is 8192 bytes

   Result function is minimum

   This key is relevant only for the iSCSI connection of an iSCSI
   session if RDMAExtensions=Yes was negotiated on the leading
   connection of the session.  It is used by the initiator and the
   target to negotiate the maximum size of the data segment that an
   initiator may send to the target in an iSCSI control-type PDU in the
   full feature phase.
   Full Feature Phase.  For SCSI Command PDUs and SCSI Data-out Data-Out PDUs
   containing non-immediate unsolicited data to be sent by the
   initiator, the initiator MUST send all non-Final PDUs with a data
   segment size of exactly TargetRecvDataSegmentLength whenever the PDUs
   constitute a data sequence whose size is larger than
   TargetRecvDataSegmentLength.

6.5

6.5.  InitiatorRecvDataSegmentLength

   Use: IO (Initialize only)

   Senders: Initiator and Target

   Scope: CO (connection-only)

   Irrelevant when: RDMAExtensions=No

   InitiatorRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>

   Default is 8192 bytes

   Result function is minimum

   This key is relevant only for the iSCSI connection of an iSCSI
   session if RDMAExtensions=Yes was negotiated on the leading
   connection of the session.  It is used by the initiator and the
   target to negotiate the maximum size of the data segment that a
   target may send to the initiator in an iSCSI control-type PDU in the
   full feature phase.

6.6
   Full Feature Phase.

6.6.  OFMarker and IFMarker

   Irrelevant when: RDMAExtensions=Yes

   Negotiations resulting in RDMAExtensions=Yes for a session implies imply
   OFMarker=No and IFMarker=No for all connections in that session and
   overrides both
   override the settings, whether default and an explicit setting.

6.7 or configured.

6.7.  MaxOutstandingUnexpectedPDUs

   Use: LO (leading only), Declarative

   Senders: Initiator and Target

   Scope: SW (session-wide)

   Irrelevant when: RDMAExtensions=No

   MaxOutstandingUnexpectedPDUs=<numerical-value-from-2-to-(2**32-1)

   MaxOutstandingUnexpectedPDUs=
      <numerical-value-from-2-to-(2**32-1) | 0>

   Default is 0
   This key is used by the initiator and the target to declare the
   maximum number of outstanding "unexpected" iSCSI control-type PDUs
   that it can receive in the full feature phase. Full Feature Phase.  It is intended to
   allow the receiving side to determine the amount of buffer resources
   needed beyond the normal flow control mechanism available in iSCSI.
   An initiator or target should select a value such that it would not
   impose an unnecessary constraint on the iSCSI Layer layer under normal
   circumstances.  The value of 0 is defined to indicate that the
   declarer has no limit on the maximum number of outstanding
   "unexpected" iSCSI control-type PDUs that it can receive.  See
   sections
   Sections 8.1.1 and 8.1.2 for the usage of this key.  Note that iSER
   Hello and HelloReply Messages are not iSCSI control-type PDUs and are
   not affected by this key.

   For interoperability with implementations based on [RFC5046], this
   key SHOULD be negotiated because the default value of 0 in [RFC5046]
   is problematic for most implementations as it does not impose a bound
   on resources consumable by unexpected PDUs.

6.8

6.8.  MaxAHSLength

   Use: LO (leading only), Declarative

   Senders: Initiator and Target

   Scope: SW (session-wide)

   Irrelevant when: RDMAExtensions=No

   MaxAHSLength=<numerical-value-from-2-to-(2**32-1) | 0>

   Default is 256

   This key is used by the intiator initiator and target to declare the maximum
   size of AHS in an iSCSI control-type PDU that it can receive in the
   full feature phase.
   Full Feature Phase.  It is intended to allow the receiving side to
   determine the amount of resources needed for receive buffering.  An
   initiator or target should select a value such that it would not
   impose an unnecessary constraint on the iSCSI Layer layer under normal
   circumstances.  The value of 0 is defined to indicate that the
   declarer has no limit on the maximum size of AHS in iSCSI control-
   type PDUs that it can receive.

   For interoperability with implementations based on [RFC5046], an
   initiator or target MAY terminate the connection if it anticipates
   MaxAHSLength to be greater than 256 and the key is not understood by
   its peer.

6.9

6.9.  TaggedBufferForSolicitedDataOnly

   Use: LO (leading only), Declarative

   Senders: Initiator

   Scope: SW (session-wide)

   RDMAExtensions=<boolean-value>

   Irrelevant when: RDMAExtensions=No

   Default is No

   This key is used by the intiator initiator to declare to the target the usage
   of the Write Base Offset in the iSER header of an iSCSI control-type
   PDU.  When set to No, the Base Offset is associated with an I/O
   buffer that contains all the write data, including both unsolicited
   and solicited data.  When set to Yes, the Base Offset is associated
   with an I/O buffer that only contains solicited data.

6.10

6.10.  iSERHelloRequired

   Use: LO (leading only), Declarative

   Senders: Initiator

   Scope: SW (session-wide)

   RDMAExtensions=<boolean-value>

   Irrelevant when: RDMAExtensions=No

   Default is No

   This key is relevant only for the iSCSI connection of an iSCSI
   session if RDMAExtensions=Yes was negotiated on the leading
   connection of the session.  It is used by the intiator initiator to declare to
   the target if whether the iSER Hello Exchange is required.  When set to
   Yes, the iSER layers MUST perform the iSER Hello Exchange as
   described in Section 5.1.3.  When set to No, the iSER layers MUST NOT
   perform the iSER Hello Exchange.

7

7.  iSCSI PDU Considerations

   When a connection is in the iSER-assisted mode, two types of message
   transfers are allowed between the iSCSI Layer at layer (at the initiator initiator) and
   the iSCSI Layer at layer (at the target. target).  These are known as the iSCSI data-
   type PDUs and the iSCSI control-type PDUs PDUs, and these terms are
   described in the following sections.

7.1

7.1.  iSCSI Data-Type PDU

   An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
   transfer, transparent to the remote iSCSI layer, to take place
   between the peer iSCSI nodes in the full feature phase Full Feature Phase of an
   iSCSI/iSER connection.  An iSCSI data-type PDU, when requested for
   transmission by the iSCSI Layer layer in the sending node, results in the
   data being transferred
   data's transfer without the participation of the iSCSI Layers layers at the
   sending and the receiving nodes.  This is due to the fact that the
   PDU itself is not delivered as-is to the iSCSI Layer layer in the receiving
   node.  Instead, the data transfer operations are transformed into the
   appropriate RDMA operations operations, which are handled by the RDMA-Capable
   Controller.  The set of iSCSI data-type PDUs consists of SCSI Data-in Data-In
   PDUs and R2T PDUs.

   If the invocation of the Operational Primitive by the iSCSI Layer layer to
   request the iSER Layer layer to process an iSCSI data-type PDU is qualified
   with Notify_Enable set, then upon completing the RDMA operation, the
   iSER Layer layer at the target MUST notify the iSCSI Layer layer at the target by
   invoking the Data_Completion_Notify Operational Primitive qualified
   with the ITT and SN.  There is no data completion notification at the
   initiator since the RDMA operations are completely handled by the
   RDMA-Capable Controller at the initiator and the iSER Layer layer at the
   initiator is not involved with the data transfer associated with
   iSCSI data-type PDUs.

   If the invocation of the Operational Primitive by the iSCSI Layer layer to
   request the iSER Layer layer to process an iSCSI data-type PDU is qualified
   with Notify_Enable cleared, then upon completing the RDMA operation,
   the iSER Layer layer at the target MUST NOT notify the iSCSI
   Layer layer at the
   target and MUST NOT invoke the Data_Completion_Notify Operational
   Primitive.

   If an operation associated with an iSCSI data-type PDU fails for any
   reason, the contents of the Data Sink buffers associated with the
   operation are considered indeterminate.

7.2

7.2.  iSCSI Control-Type PDU

   Any iSCSI PDU that is not an iSCSI data-type PDU and also not a SCSI
   Data-out
   Data-Out PDU carrying solicited data is defined as an iSCSI control-
   type PDU.  The iSCSI Layer layer invokes the Send_Control Operational
   Primitive to request the iSER Layer layer to process an iSCSI control-type
   PDU.  iSCSI control-type PDUs are transferred using Send Messages of
   RCaP.  Specifically, it is to be noted that SCSI Data-Out PDUs
   carrying unsolicited data are defined as iSCSI control-type PDUs.
   See section Section 7.3.4 on the treatment of SCSI Data-out Data-Out PDUs.

   When the iSER Layer layer receives an iSCSI control-type PDU, it MUST
   notify the iSCSI Layer layer by invoking the Control_Notify Operational
   Primitive qualified with the iSCSI control-type PDU.

7.3

7.3.  iSCSI PDUs

   This section describes the handling of each of the iSCSI PDU types by
   the iSER Layer. layer.  The iSCSI Layer layer requests the iSER Layer layer to process
   the iSCSI PDU by invoking the appropriate Operational Primitive.  A
   Connection_Handle MUST qualify each of these invocations.  In
   addition, the BHS and the optional AHS of the iSCSI PDU as defined in
   [iSCSI] MUST qualify each of the invocations.  The qualifying
   Connection_Handle, the BHS BHS, and the AHS are not explicitly listed in
   the subsequent sections.

7.3.1

7.3.1.  SCSI Command

      Type:  control-type PDU

      PDU-specific qualifiers (for SCSI Write or bidirectional command):
      ImmediateDataSize, UnsolicitedDataSize, DataDescriptorOut

      PDU-specific qualifiers (for SCSI Read or bidirectional command):
      DataDescriptorIn

   The iSER Layer layer at the initiator MUST send the SCSI command in a Send
   Message to the target.  The SendSE Message should be used if
   supported by the RCaP layer (e.g., iWARP).

   For a SCSI Write or bidirectional command, the iSCSI Layer layer at the
   initiator MUST invoke the Send_Control Operational Primitive as
   follows:

   *  If there is immediate data to be transferred for the SCSI write or
      bidirectional command, the qualifier ImmediateDataSize MUST be
      used to define the number of bytes of immediate unsolicited data
      to be sent with the write or bidirectional command, and the
      qualifier DataDescriptorOut MUST be used to define the initiator's
      I/O Buffer containing the SCSI Write data.

   *  If there is unsolicited data to be transferred for the SCSI Write
      or bidirectional command, the qualifier UnsolicitedDataSize MUST
      be used to define the number of bytes of immediate and non-
      immediate unsolicited data for the command.  The iSCSI Layer layer will
      issue one or more SCSI Data-out Data-Out PDUs for the non-immediate
      unsolicited data.  See Section 7.3.4 on SCSI Data-out. Data-Out.

   *  If there is solicited data to be transferred for the SCSI Write or
      bidirectional command, as indicated by when the Expected Data
      Transfer Length in the SCSI Command PDU exceeding exceeds the value of
      UnsolicitedDataSize, the iSER Layer layer at the initiator MUST do the
      following:

      a. It MUST allocate a Write STag for the I/O Buffer defined by the
         qualifier DataDescriptorOut.  DataDescriptorOut describes the
         I/O buffer starting with the immediate unsolicited data (if
         any), followed by the non-immediate unsolicited data (if any)
         and solicited data.  When TaggedBufferForSolicitedDataOnly is
         negotiated to No, the Base Offset is associated with this I/O
         Buffer.  When TaggedBufferForSolicitedDataOnly is negotiated to
         Yes, the Base Offset is associated with an I/O Buffer that
         contains only solicited data.

      b. It MUST establish a Local Mapping that associates the Initiator
         Task Tag (ITT) to the Write STag.

      c. It MUST Advertise the Write STag and the Base Offset to the
         target by sending them in the iSER header of the iSER Message
         (the payload of the Send Message of RCaP) containing the SCSI
         Write or bidirectional command PDU.  The SendSE Message should
         be used if supported by the RCaP layer (e.g., iWARP).  See section
         Section 9.2 on iSER Header Format for iSCSI Control-Type PDU.

   For a SCSI Read or bidirectional command, the iSCSI Layer layer at the
   initiator MUST invoke the Send_Control Operational Primitive
   qualified with DataDescriptorIn DataDescriptorIn, which defines the initiator's I/O
   Buffer for receiving the SCSI Read data.  The iSER Layer layer at the
   initiator MUST do the following:

      a. It MUST allocate a Read STag for the I/O Buffer and note the
         Base Offset for this I/O Buffer.

      b. It MUST establish a Local Mapping that associates the Initiator
         Task Tag (ITT) to the Read STag.

      c. It MUST Advertise the Read STag and the Base Offset to the
         target by sending them in the iSER header of the iSER Message
         (the payload of the Send Message of RCaP) containing the SCSI
         Read or bidirectional command PDU.  The SendSE Message should
         be used if supported by the RCaP layer (e.g., iWARP).  See section
         Section 9.2 on iSER Header Format for iSCSI Control-Type PDU.

   If the amount of unsolicited data to be transferred in a SCSI Command
   exceeds TargetRecvDataSegmentLength, then the iSCSI Layer layer at the
   initiator MUST segment the data into multiple iSCSI control-type
   PDUs, with the data segment length in all PDUs generated except PDUs (except the
   last one one) having exactly the size TargetRecvDataSegmentLength.  The
   data segment length of the last iSCSI control-type PDU carrying the
   unsolicited data can be up to TargetRecvDataSegmentLength.

   When the iSER Layer layer at the target receives the SCSI Command, it MUST
   establish a Remote Mapping that associates the ITT to the Base
   Offset(s) and the Advertised STag(s) in the iSER header.  The Write
   STag is used by the iSER Layer layer at the target in handling the data
   transfer associated with the R2T PDU(s) as described in section Section
   7.3.6.  The Read STag is used in handling the SCSI Data-in Data-In PDU(s)
   from the iSCSI Layer layer at the target as described in section Section 7.3.5.

7.3.2

7.3.2.  SCSI Response

      Type:  control-type PDU

      PDU-specific qualifiers:  DataDescriptorStatus

   The iSCSI Layer layer at the target MUST invoke the Send_Control
   Operational Primitive qualified with DataDescriptorStatus DataDescriptorStatus, which
   defines the buffer containing the sense and response information.
   The iSCSI Layer layer at the target MUST always return the SCSI status for
   a SCSI command in a separate SCSI Response PDU.  "Phase collapse" for
   transferring SCSI status in a SCSI Data-in Data-In PDU MUST NOT be used. The
   iSER Layer layer at the target sends the SCSI Response PDU according to the
   following rules:

   *  If no STags were Advertised by the initiator in the iSER Message
      containing the SCSI command PDU, then the iSER Layer layer at the target
      MUST send a Send Message containing the SCSI Response PDU.  The
      SendSE Message should be used if supported by the RCaP layer
      (e.g., iWARP).

   *  If the initiator Advertised a Read STag in the iSER Message
      containing the SCSI Command PDU, then the iSER Layer layer at the target
      MUST send a Send Message containing the SCSI Response PDU.  The
      header of the Send Message MUST carry the Read STag to be
      invalidated at the initiator.  The Send with Invalidate Message,
      if supported by the RCaP layer (e.g., iWARP), can be used for the
      automatic invalidation of the STag.

   *  If the initiator Advertised only the Write STag in the iSER
      Message containing the SCSI command PDU, then the iSER Layer layer at
      the target MUST send a Send Message containing the SCSI Response
      PDU.  The header of the Send Message MUST carry the Write STag to
      be invalidated at the initiator.  The Send with Invalidate
      Message, if supported by the RCaP layer (e.g., iWARP), can be used
      for the automatic invalidation of the STag.

   When the iSCSI Layer layer at the target invokes the Send_Control
   Operational Primitive to send the SCSI Response PDU, the iSER Layer layer
   at the target MUST invalidate the Remote Mapping before transferring
   the SCSI Response PDU to the initiator.

   Upon receiving a Send Message containing the SCSI Response PDU from
   the target, the iSER layer at the initiator MUST invalidate the
   STag(s) specified in the header.  (If a Send with Invalidate Message
   is supported by the RCaP layer (e.g., iWARP) and is used to carry the
   SCSI Response PDU, the RCaP layer at the initiator will invalidate
   the STag.  The iSER Layer layer at the initiator MUST ensure that the
   correct STag is invalidated.  If both the Read and the Write STags
   were Advertised earlier by the initiator, then the iSER
   Layer layer at the
   initiator MUST explicitly invalidate the Write STag upon receiving
   the Send with Invalidate Message because the header of the Send with
   Invalidate Message can only carry one STag (in this
   case case, the Read
   STag) to be invalidated.)

   The iSER Layer layer at the initiator MUST ensure the invalidation of the
   STag(s) used in a command before notifying the iSCSI Layer layer at the
   initiator by invoking the Control_Notify Operational Primitive
   qualified with the SCSI Response.  This precludes the possibility of
   using the STag(s) after the completion of the command thereby
   causing command; such use would
   cause data corruption.

   When the iSER Layer layer at the initiator receives a Send Message
   containing the SCSI Response PDU, it SHOULD invalidate the Local
   Mapping.  The iSER Layer layer MUST ensure that all local STag(s)
   associated with the ITT are invalidated before notifying the iSCSI
   Layer
   layer of the SCSI Response PDU by invoking the Control_Notify
   Operational Primitive qualified with the SCSI Response PDU.

7.3.3

7.3.3.  Task Management Function Request/Response

      Type:  control-type PDU

      PDU-specific qualifiers (for TMF Request):  DataDescriptorOut,
      DataDescriptorIn

   The iSER Layer layer MUST use a Send Message to send the Task Management
   Function Request/Response PDU.  The SendSE Message should be used if
   supported by the RCaP layer (e.g., iWARP).

   For the Task Management Function Request with the TASK REASSIGN
   function, the iSER Layer layer at the initiator MUST do the following:

   *  It MUST use the ITT as specified in the Referenced Task Tag from
      the Task Management Function Request PDU to locate the existing
      STags (if any) in the Local Mappings.

   *  It MUST invalidate the existing STags (if any) and the Local
      Mappings.

   *  It MUST allocate a Read STag for the I/O Buffer and note the Base
      Offset associated with the I/O Buffer as defined by the qualifier
      DataDescriptorIn if the Send_Control Operational Primitive
      invocation is qualified with DataDescriptorIn.

   *  It MUST allocate a Write STag for the I/O Buffer and note the Base OIffset
      Offset associated with the I/O Buffer as defined by the qualifier
      DataDescriptorOut if the Send_Control Operational Primitive
      invocation is qualified with DataDescriptorOut.

   *  If STags are allocated, it MUST establish new Local Mapping(s)
      that associate the ITT to the allocated STag(s).

   *  It MUST Advertise the STags and the Base Offsets, if allocated, to
      the target in the iSER header of the Send Message carrying the
      iSCSI PDU, as described in section Section 9.2.  The SendSE Message should
      be used if supported by the RCaP layer (e.g., iWARP).

   For the Task Management Function Request with the TASK REASSIGN
   function for a SCSI Read or bidirectional command, the iSCSI Layer layer at
   the initiator MUST set ExpDataSN to 0 zero since the data transfer and
   acknowledgements happen transparently to the iSCSI Layer layer at the
   initiator.  This provides the flexibility to the iSCSI Layer layer at the
   target to request transmission of only the unacknowledged data as
   specified in [iSCSI].

   When the iSER Layer layer at the target receives the Task Management
   Function Request with the TASK REASSIGN function, it MUST do the
   following:

   *  It MUST use the ITT as specified in the Referenced Task Tag from
      the Task Management Function Request PDU to locate the Local and
      Remote Mappings (if any).

   *  It MUST invalidate the local STaqs STags (if any) associated with the
      ITT.

   *  It MUST replace the Base Offset(s) and the Advertised STag(s) in
      the Remote Mapping with the Base Offset(s) and the Advertised
      STag(s) in the iSER header.  The Write STag is used in the
      handling of the R2T PDU(s) from the iSCSI Layer layer at the target as
      described in section Section 7.3.6.  The Read STag is used in the handling
      of the SCSI Data-in Data-In PDU(s) from the iSCSI Layer layer at the target as
      described in section Section 7.3.5.

7.3.4

7.3.4.  SCSI Data-out Data-Out

      Type:  control-type PDU

      PDU-specific qualifiers:  DataDescriptorOut

   The iSCSI Layer layer at the initiator MUST invoke the Send_Control
   Operational Primitive qualified with DataDescriptorOut DataDescriptorOut, which defines
   the initiator's I/O Buffer containing unsolicited SCSI Write data.

   If the amount of unsolicited data to be transferred as SCSI Data-out Data-Out
   exceeds TargetRecvDataSegmentLength, then the iSCSI Layer layer at the
   initiator MUST segment the data into multiple iSCSI control-type
   PDUs, with where the DataSegmentLength having has the value of
   TargetRecvDataSegmentLength in all PDUs generated PDUs except the last
   one.  The DataSegmentLength of the last iSCSI control-type PDU
   carrying the unsolicited data can be up to
   TargetRecvDataSegmentLength.  The iSCSI Layer layer at the target MUST
   perform the reassembly function for the unsolicited data.

   For unsolicited data, the iSER Layer layer at the initiator MUST use a Send
   Message to send the SCSI Data-out Data-Out PDU.  If the F bit is set to 1, the
   SendSE Message shoud should be used if supported by the RCaP layer (e.g.,
   iWARP).

   Note that for solicited data, the SCSI Data-out Data-Out PDUs are not used
   since R2T PDUs are not delivered to the iSCSI layer at the initiator; instead
   instead, R2T PDUs are transformed by the iSER layer at the target
   into RDMA Read operations.  (See section Section 7.3.6.)

7.3.5

7.3.5.  SCSI Data-in Data-In

      Type:  data-type PDU

      PDU-specific qualifiers:  DataDescriptorIn

   When the iSCSI Layer layer at the target is ready to return the SCSI Read
   data to the initiator, it MUST invoke the Put_Data Operational
   Primitive qualified with DataDescriptorIn DataDescriptorIn, which defines the SCSI
   Data-in
   Data-In buffer.  See section Section 7.1 on the general requirement on the
   handling of iSCSI data-type PDUs.  SCSI Data-in Data-In PDU(s) are used in
   SCSI Read data transfer as described in section Section 9.5.2.

   The iSER Layer layer at the target MUST do the following for each
   invocation of the Put_Data Operational Primitive:

   1. It MUST use the ITT in the SCSI Data-in Data-In PDU to locate the remote
      Read STag and the Base Offset in the Remote Mapping.  The Remote
      Mapping was established earlier by the iSER Layer layer at the target
      when the SCSI Read Command was received from the initiator.

   2. It MUST generate and send an RDMA Write Message containing the
      read data to the initiator.

      a. It MUST use the remote Read STag as the Data Sink STag of the
         RDMA Write Message.

      b. It MUST add the Buffer Offset from the SCSI Data-in Data-In PDU to the
         Base Offset from the Remote Mapping as the Data Sink Tagged
         Offset of the RDMA Write Message.

      c. It MUST use DataSegmentLength from the SCSI Data-in Data-In PDU to
         determine the amount of data to be sent in the RDMA Write
         Message.

   3. It MUST associate the DataSN and ITT from the SCSI Data-in Data-In PDU
      with the RDMA Write operation.  If the Put_Data Operational
      Primitive invocation was qualified with Notify_Enable set, then
      when the iSER Layer layer at the target receives a completion from the
      RCaP layer for the RDMA Write Message, the iSER Layer layer at the
      target MUST notify the iSCSI Layer layer by invoking the
      Data_Completion_Notify Operational Primitive qualified with the
      DataSN and ITT.  Conversely, if the Put_Data Operational Primitive
      invocation was qualified with Notify_Enable cleared, then the iSER Layer
      layer at the target MUST NOT notify the iSCSI
       Layer layer on completion
      and MUST NOT invoke the Data_Completion_Notify Operational
      Primitive.

   When the A-bit is set to 1 one in the SCSI Data-in Data-In PDU, the iSER Layer layer
   at the target MUST notify the iSCSI Layer layer at the target when the data
   transfer is complete at the initiator.  To perform this additional
   function, the iSER Layer layer at the target can take advantage of the
   operational ErrorRecoveryLevel if previously disclosed by the iSCSI Layer
   layer via an earlier invocation of the Notice_Key_Values Operational
   Primitive.  There are two approaches that can be taken:

   1. If the iSER Layer layer at the target knows that the operational
      ErrorRecoveryLevel is 2, or if the iSER Layer layer at the target does
      not know the operational ErrorRecoveryLevel, then the iSER Layer layer
      at the target MUST issue a zero-length RDMA Read Request Message
      following the RDMA Write Message.  When the iSER Layer layer at the
      target receives a completion for the RDMA Read Request Message
      from the RCaP layer, implying that the RDMA-Capable Controller at
      the initiator has completed processing the RDMA Write Message due
      to the completion ordering semantics of RCaP, the iSER Layer layer at
      the target MUST notify the iSCSI Layer layer at the target by invoking
      the Data_Ack_Notify Data_ACK_Notify Operational Primitive qualified with ITT and
      DataSN (see section Section 3.2.3).

   2. If the iSER Layer layer at the target knows that the operational
      ErrorRecoveryLevel is 1, then the iSER Layer layer at the target MUST do
      one of the following:

      a. It MUST notify the iSCSI Layer layer at the target by invoking the
           Data_Ack_Notify
         Data_ACK_Notify Operational Primitive qualified with ITT and
         DataSN (see section Section 3.2.3) when it receives the local
         completion from the RCaP layer for the RDMA Write Message.
         This is allowed since digest errors do not occur in iSER (see section
         Section 10.1.4.2) and a CRC error will cause the connection to
         be terminated and the task to be terminated anyway.  The local
         RDMA Write completion from the RCaP layer guarantees that the
         RCaP layer will not access the I/O Buffer again to transfer the
         data associated with that RDMA Write operation.

      b. Alternatively, it MUST use the same procedure for handling the
         data transfer completion at the initiator as for
         ErrorRecoveryLevel 2.

   It should be noted that the iSCSI Layer layer at the target cannot set the
   A-bit to 1 if the ErrorRecoveryLevel=0.

   SCSI status MUST always be returned in a separate SCSI Response PDU.
   The S bit in the SCSI Data-in Data-In PDU MUST always be set to 0. zero.  There
   MUST NOT be a "phase collapse" in the SCSI Data-in Data-In PDU.

   Since the RDMA Write Message only transfers the data portion of the
   SCSI Data-in Data-In PDU but not the control information in the header, such
   as ExpCmdSN, if timely updates of such information is are crucial, the
   iSCSI Layer layer at the initiator MAY issue NOP-Out PDUs to request the
   iSCSI Layer layer at the target to respond with the information using NOP-
   In
   NOP-In PDUs.

7.3.6

7.3.6.  Ready To Transfer (R2T)

      Type:  data-type PDU

      PDU-specific qualifiers:  DataDescriptorOut

   In order to send an R2T PDU, the iSCSI Layer layer at the target MUST
   invoke the Get_Data Operational Primitive qualified with
   DataDescriptorOut
   DataDescriptorOut, which defines the I/O Buffer for receiving the
   SCSI Write data from the initiator.  See section Section 7.1 on the general
   requirements on the handling of iSCSI data-type PDUs.

   The iSER Layer layer at the target MUST do the following for each
   invocation of the Get_Data Operational Primitive:

   1. It MUST ensure a valid local STag for the I/O Buffer and a valid
      Local Mapping.  This may involve allocating a valid local STag and
      establishing a Local Mapping.

   2. It MUST use the ITT in the R2T to locate the remote Write STag and
      the Base Offset in the Remote Mapping.  The Remote Mapping was
      established earlier by the iSER Layer layer at the target when the iSER
      Message containing the Advertised Write STag, the Base
       Offset Offset, and
      the SCSI Command PDU for a SCSI Write or bidirectional command was
      received from the initiator.

   3. If the iSER-ORD value at the target is set to 0, zero, the iSER Layer layer
      at the target MUST terminate the connection and free up the
      resources associated with the connection (as described in Section
      5.2.3) if it received the R2T PDU from the iSCSI Layer layer at the
      target.  Upon termination of the connection, the iSER Layer layer at the
      target MUST notify the iSCSI Layer layer at the target by invoking the
       Connection Terminate Notify
      Connection_Terminate_Notify Operational Primitive.

   4. If the iSER-ORD value at the target is set to greater than 0, the
      iSER Layer layer at the target MUST transform the R2T PDU into an RDMA
      Read Request Message.  While transforming the R2T PDU, the iSER Layer
      layer at the target MUST ensure that the number of outstanding
      RDMA Read Request Messages does not exceed the iSER-ORD value.  To
      transform the R2T PDU, the iSER Layer layer at the target:

      a. MUST derive the local STag and local Tagged Offset from the
         DataDescriptorOut that qualified the Get_Data invocation.

      b. MUST use the local STag as the Data Sink STag of the RDMA Read
         Request Message.

      c. MUST use the local Tagged Offset as the Data Sink Tagged Offset
         of the RDMA Read Request Message.

      d. MUST use the Desired Data Transfer Length from the R2T PDU as
         the RDMA Read Message Size of the RDMA Read Request Message.

      e. MUST use the remote Write STag as the Data Source STag of the
         RDMA Read Request Message.

      f. MUST add the Buffer Offset from the R2T PDU to the Base Offset
         from the Remote Mapping as the Data Source Tagged Offset of the
         RDMA Read Request Message.

   5. It MUST associate the R2TSN and ITT from the R2T PDU with the RDMA
      Read operation.  If the Get_Data Operational Primitive invocation
      was qualified with Notify_Enable set, then when the iSER Layer layer at
      the target receives a completion from the RCaP layer for the RDMA
      Read operation, the iSER Layer layer at the target MUST notify the iSCSI Layer
      layer by invoking the Data_Completion_Notify Operational Primitive
      qualified with the R2TSN and ITT.  Conversely, if the Get_Data
      Operational Primitive invocation was qualified with Notify_Enable
      cleared, then the iSER Layer layer at the target MUST NOT notify the
      iSCSI
       Layer layer on completion and MUST NOT invoke the
      Data_Completion_Notify Operational Primitive.

   When the RCaP layer at the initiator receives a valid RDMA Read
   Request Message, it will return an RDMA Read Response Message
   containing the solicited write data to the target.  When the RCaP
   layer at the target receives the RDMA Read Response Message from the
   initiator, it will place the solicited data in the I/O Buffer
   referenced by the Data Sink STag in the RDMA Read Response Message.

   Since the RDMA Read Request Message from the target does not transfer
   the control information in the R2T PDU such as ExpCmdSN, if timely
   updates of such information is are crucial, the iSCSI Layer layer at the
   initiator MAY issue NOP-Out PDUs to request the iSCSI Layer layer at the
   target to respond with the information using NOP-In PDUs.

   Similarly, since the RDMA Read Response Message from the initiator
   only transfers the data but not the control information normally
   found in the SCSI Data-out Data-Out PDU, such as ExpStatSN, if timely updates
   of such information is are crucial, the iSCSI Layer layer at the target MAY
   issue NOP-In PDUs to request the iSCSI Layer layer at the initiator to
   respond with the information using NOP-Out PDUs.

7.3.7

7.3.7.  Asynchronous Message

      Type:  control-type PDU

      PDU-specific qualifiers:  DataDescriptorSense

   The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
   qualified with DataDescriptorSense DataDescriptorSense, which defines the buffer
   containing the sense and iSCSI event information.  The iSER Layer layer
   MUST use a Send Message to send the Asynchronous Message PDU.  The
   SendSE Message should be used if supported by the RCaP layer (e.g.,
   iWARP).

7.3.8

7.3.8.  Text Request & and Text Response

      Type:  control-type PDU

      PDU-specific qualifiers:  DataDescriptorTextOut (for Text
      Request), DataDescriptorIn (for Text Response)

   The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
   qualified with DataDescriptorTextOut (or DataDescriptorIn) DataDescriptorIn), which
   defines the Text Request (or Text Response) buffer.  The iSER Layer layer
   MUST use Send Messages to send the Text Request (or Text Response
   PDUs).  The SendSE Message should be used if supported by the RCaP
   layer (e.g., iWARP).

7.3.9

7.3.9.  Login Request & and Login Response

   During the login negotiation, the iSCSI Layer layer interacts with the
   transport layer directly directly, and the iSER Layer layer is not involved.  See
   section
   Section 5.1 on iSCSI/iSER Connection Setup.  If the underlying
   transport is TCP, the Login Request PDUs and the Login Response PDUs
   are exchanged when the connection between the initiator and the
   target is still in the byte stream mode.

   The iSCSI Layer layer MUST NOT send a Login Request (or a Login Response)
   PDU during the full feature phase. Full Feature Phase.  A Login Request (or a Login
   Response) PDU, if used, MUST be treated as an iSCSI protocol error.
   The iSER Layer layer MAY reject such a PDU from the iSCSI Layer layer with an
   appropriate error code.  If a Login Request PDU is received by the
   iSCSI Layer layer at the target, it MUST respond with a Reject PDU with a
   reason code of "protocol error".

7.3.10

7.3.10.  Logout Request & and Logout Response

      Type:  control-type PDU

      PDU-specific qualifiers:  None

   The iSER Layer layer MUST use a Send Message to send the Logout Request or
   Logout Response PDU.  The SendSE Message should be used if supported
   by the RCaP layer (e.g., iWARP).  Section  Sections 5.2.1 and 5.2.2 describe
   the handling of the Logout Request and the Logout Response at the
   initiator and the target and the interactions between the initiator
   and the target to terminate a connection.

7.3.11

7.3.11.  SNACK Request

   Since HeaderDigest and DataDigest must be negotiated to "None", there
   are no digest errors when the connection is in iSER-assisted mode.  Also
   Also, since RCaP delivers all messages in the order they were sent,
   there are no sequence errors when the connection is in iSER-
   assisted iSER-assisted
   mode.  Therefore  Therefore, the iSCSI Layer layer MUST NOT send SNACK Request PDUs.
   A SNACK Request PDU, if used, MUST be treated as an iSCSI protocol
   error.  The iSER Layer layer MAY reject such a PDU from the iSCSI Layer layer
   with an appropriate error code.  If a SNACK Request PDU is received
   by the iSCSI Layer layer at the target, it MUST respond with a Reject PDU
   with a reason code of "protocol error".

7.3.12

7.3.12.  Reject

      Type:  control-type PDU

      PDU-specific qualifiers:  DataDescriptorReject

   The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
   qualified with DataDescriptorReject DataDescriptorReject, which defines the Reject buffer.
   The iSER Layer layer MUST use a Send Message to send the Reject PDU.  The
   SendSE Message should be used if supported by the RCaP layer (e.g.,
   iWARP).

7.3.13

7.3.13.  NOP-Out & and NOP-In

      Type:  control-type PDU

      PDU-specific qualifiers:  DataDescriptorNOPOut (for NOP-Out),
      DataDescriptorNOPIn (for NOP-In)

   The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
   qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn) DataDescriptorNOPIn), which
   defines the Ping (or Return Ping) data buffer.  The iSER Layer layer MUST
   use Send Messages to send the NOP-Out (or NOP-In) PDU.  The SendSE
   Message should be used if supported by the RCaP layer (e.g., iWARP).

8

8.  Flow Control and STag Management

8.1

8.1.  Flow Control for RDMA Send Messages

   Send Messages in RCaP are used by the iSER Layer layer to transfer iSCSI
   control-type PDUs.  Each Send Message in RCaP consumes an Untagged
   Buffer at the Data Sink.  However, neither the RCaP layer nor the
   iSER Layer layer provides an explicit flow control mechanism for the Send
   Messages.  Therefore, the iSER Layer layer SHOULD provision enough Untagged
   buffers for handling incoming Send Messages to prevent buffer
   exhaustion at the RCaP layer.  If buffer exhaustion occurs, it may
   result in the termination of the connection.

   An implementation may choose to satisfy the buffer requirement by
   using a common buffer pool shared across multiple connections, with
   usage limits on a per connection per-connection basis and usage limits on the buffer
   pool itself.  In such an implementation, exceeding the buffer usage
   limit for a connection or the buffer pool itself may trigger
   interventions from the iSER Layer layer to replenish the buffer pool and/or
   to isolate the connection causing the problem.

   iSER also provides the MaxOutstandingUnexpectedPDUs key to be used by
   the initiator and the target to declare the maximum number of
   outstanding "unexpected" control-type PDUs that it can receive.  It
   is intended to allow the receiving side to determine the amount of
   buffer resources needed beyond the normal flow control mechanism
   available in iSCSI.

   The buffer resources required at both the initiator and the target as
   a result of control-type PDUs sent by the initiator is are described in section
   Section 8.1.1.  The buffer resources required at both the initiator
   and target as a result of control-type PDUs sent by the target is are
   described in section Section 8.1.2.

8.1.1

8.1.1.  Flow Control for Control-Type PDUs from the Initiator

   The control-type PDUs that can be sent by an initiator to a target
   can be grouped into the following categories:

   1. Regulated:  Control-type PDUs in this category are regulated by
      the iSCSI CmdSN window mechanism mechanism, and the immediate flag is not
      set.

   2. Unregulated but Expected:  Control-type PDUs in this category are
      not regulated by the iSCSI CmdSN window mechanism but are expected
      by the target.

   3. Unregulated and Unexpected:  Control-type PDUs in this category
      are not regulated by the iSCSI CmdSN window mechanism and are
      "unexpected" by the target.

8.1.1.1

8.1.1.1.  Control-Type PDUs from the Initiator in the Regulated Category

   Control-type PDUs that can be sent by the initiator in this category
   are regulated by the iSCSI CmdSN window mechanism mechanism, and the immediate
   flag is not set.

   The queuing capacity required of the iSCSI layer at the target is
   described in section Section 4.2.2.1 of [iSCSI].  For each of the control-
   type PDUs that can be sent by the initiator in this category, the
   initiator MUST provision for the buffer resources required for the
   corresponding control-type PDU sent as a response from the target.
   The following is a list of the PDUs that can be sent by the initiator
   and the PDUs that are sent by the target in response:

      a. When an initiator sends a SCSI Command PDU, it expects a SCSI
         Response PDU from the target.

      b. When the initiator sends a Task Management Function Request
         PDU, it expects a Task Management Function Response PDU from
         the target.

      c. When the initiator sends a Text Request PDU, it expects a Text
         Response PDU from the target.

      d. When the initiator sends a Logout Request PDU, it expects a
         Logout Response PDU from the target.

      e. When the initiator sends a NOP-Out PDU as a ping request with
         ITT != 0xffffffff and TTT = 0xffffffff, it expects a NOP-In PDU
         from the target with the same ITT and TTT as in the ping
         request.

   The response from the target for any of the PDUs enumerated here may
   alternatively be in the form of a Reject PDU sent instead before the task is
   active, as described in section Section 7.3 of [iSCSI].

8.1.1.2

8.1.1.2.  Control-Type PDUs from the Initiator in the Unregulated but
          Expected Category

   For the control-type PDUs in the Unregulated but Expected category,
   the amount of buffering resources required at the target can be
   predetermined.  The following is a list of the PDUs in this category:

      a. SCSI Data-out Data-Out PDUs are used by the initiator to send
         unsolicited data.  The amount of buffer resources required by
         the target can be determined using FirstBurstLength.  Note that
         SCSI Data-out Data-Out PDUs are not used for solicited data since the
         R2T PDU PDU, which is used for solicitation solicitation, is transformed into
         RDMA Read operations by the iSER layer at the target.  See section
         Section 7.3.4.

      b. A NOP-Out PDU with TTT != 0xffffffff is sent as a ping response
         by the initiator to the NOP-In PDU sent as a ping request by
         the target.

8.1.1.3

8.1.1.3.  Control-Type PDUs from the Initiator in the Unregulated and
          Unexpected Category

   PDUs in the Unregulated and Unexpected category are PDUs with the
   immediate flag set.  The number of PDUs that are in this category which and
   can be sent by an initiator is controlled by the value of
   MaxOutstandingUnexpectedPDUs declared by the target.  (See section Section
   6.7.)  After a PDU in this category is sent by the initiator, it is
   outstanding until it is retired.  At any time, the number of
   outstanding unexpected PDUs MUST NOT exceed the value of
   MaxOutstandingUnexpectedPDUs declared by the target.

   The target uses the value of MaxOutstandingUnexpectedPDUs that it
   declared to determine the amount of buffer resources required for
   control-type PDUs in this category that can be sent by an initiator.
   For the initiator, for each of the control-type PDUs that can be sent
   in this category, the initiator MUST provision for the buffer
   resources if required for the corresponding control-type PDU that can
   be sent as a response from the target.

   An outstanding PDU in this category is retired as follows.  If the
   CmdSN of the PDU sent by the initiator in this category is x, the PDU
   is outstanding until the initiator sends a non-immediate control-type
   PDU on the same connection with CmdSN = y (where y is at least x) and
   the target responds with a control-type PDU on any connection where
   ExpCmdSN is at least y+1.

   When the number of outstanding unexpected control-type PDUs equals
   MaxOutstandingUnexpectedPDUs, the iSCSI Layer layer at the initiator MUST
   NOT generate any unexpected PDUs PDUs, which otherwise it would have
   generated, even if it the unexpected PDU is intended for immediate
   delivery.

8.1.2

8.1.2.  Flow Control for Control-Type PDUs from the Target

   Control-type PDUs that can be sent by a target and are expected by
   the initiator are listed in the Regulated category.  (See section Section
   8.1.1.1.)

   For the control-type PDUs that can be sent by a target and are
   unexpected by the initiator, the number is controlled by
   MaxOutstandingUnexpectedPDUs declared by the initiator.  (See
   section Section
   6.7.)  After a PDU in this category is sent by a target, it is
   outstanding until it is retired.  At any time, the number of
   outstanding unexpected PDUs MUST NOT exceed the value of
   MaxOutstandingUnexpectedPDUs declared by the initiator.  The
   initiator uses the value of MaxOutstandingUnexpectedPDUs that it
   declared to determine the amount of buffer resources required for
   control-type PDUs in this category that can be sent by a target.  The
   following is a list of the PDUs in this category and the conditions
   for retiring the outstanding PDU:

      a. For an Asynchronous Message PDU with StatSN = x, the PDU is
         outstanding until the initiator sends a control-type PDU with
         ExpStatSN set to at least x+1.

      b. For a Reject PDU with StatSN = x x, which is sent after a task is
         active, the PDU is outstanding until the initiator sends a
         control-type PDU with ExpStatSN set to at least x+1.

      c. For a NOP-In PDU with ITT = 0xffffffff and StatSN = x, the PDU
         is outstanding until the initiator responds with a control-type
         PDU on the same connection where ExpStatSN is at least x+1.
         But if the NOP-In PDU is sent as a ping request with
         TTT != 0xffffffff, the PDU can also be retired when the
         initiator sends a NOP-Out PDU with the same ITT and TTT as in
         the ping request.  Note that when a target sends a NOP-In PDU
         as a ping request, it must provision a buffer for the NOP-Out
         PDU sent as a ping response from the initiator.

   When the number of outstanding unexpected control-type PDUs equals
   MaxOutstandingUnexpectedPDUs, the iSCSI Layer layer at the target MUST NOT
   generate any unexpected PDUs PDUs, which otherwise it would have
   generated, even if its intent is to indicate an iSCSI error condition
   (e.g., Asynchronous Message, Reject).  Task timeouts timeouts, as in the initiator
   initiator's waiting for a command completion or other connection and session level exceptions
   session-level exceptions, will ensure that correct operational
   behavior will result in these cases despite not generating the PDU.
   This rule overrides any other requirements elsewhere which that require
   that a Reject PDU MUST be sent.

   (Implementation note:  SCSI task timeout and recovery can be a
   lengthy process and hence SHOULD be avoided by proper provisioning of
   resources.)

   (Implementation note:  To ensure that the initiator has a means to
   inform the target that outstanding PDUs have been retired, the target
   should reserve the last unexpected control-type PDU allowable by the
   value of MaxOutstandingUnexpectedPDUs declared by the initiator for
   sending a NOP-In ping request with TTT != 0xffffffff to allow the
   initiator to return the NOP-Out ping response with the current
   ExpStatSN.)

8.2

8.2.  Flow Control for RDMA Read Resources

   If iSERHelloRequired is negotiated to "Yes", then the total number of
   RDMA Read operations that can be active simultaneously on an
   iSCSI/iSER connection depends on the amount of resources allocated as
   declared in the iSER Hello exchange described in section Section 5.1.3.
   Exceeding the number of RDMA Read operations allowed on a connection
   will result in the connection being terminated by the RCaP layer.
   The iSER Layer layer at the target maintains the iSER-ORD to keep track of
   the maximum number of RDMA Read Requests that can be issued by the
   iSER Layer layer on a particular RCaP Stream.

   During connection setup (see section Section 5.1), iSER-IRD is known at the
   initiator and iSER-ORD is known at the target after the iSER Layers layers
   at the initiator and the target have respectively allocated the
   connection resources necessary to support RCaP, as directed by the
   Allocate_Connection_Resources Operational Primitive from the iSCSI
   Layer
   layer before the end of the iSCSI Login Phase.  In the full feature
   phase, Full Feature
   Phase, if iSERHelloRequired is ngtiated negotiated to "Yes", then the first
   message sent by the initiator is the iSER Hello Message (see section
   9.3) Section
   9.3), which contains the value of iSER-IRD.  In response to the iSER
   Hello Message, the target sends the iSER HelloReply Message (see
   section 9.4)
   Section 9.4), which contains the value of iSER-ORD.  The iSER Layer layer
   at both the initiator and the target MAY adjust (lower) the resources
   associated with iSER-IRD and iSER-ORD respectively iSER-ORD, respectively, to match the
   iSER-ORD value declared in the HelloReply Message.  The iSER Layer layer at
   the target MUST flow control the flow of the RDMA Read Request Messages to so
   that it does not exceed the iSER-ORD value at the target.

   If iSERHelloRequired is negotiated to "No", then the maximum number
   of RDMA Read operations that can be active is negotiated via other
   means outside the scope of this document.  For example, in
   InfiniBand, iSER connection setup uses InfiniBand CM MADs, Connection Manager
   (CM) Management Datagrams (MADs), with additional iSER information
   exchanged in the private data.

8.3

8.3.  STag Management

   An STag is an identifier of a Tagged Buffer used in an RDMA
   operation.  The allocation and the subsequent invalidation of the
   STags are specified in this document if  If the STags are exposed on the wire by being Advertised
   in the iSER header or declared in the header of an RCaP Message.

8.3.1 Message, then
   the allocation and the subsequent invalidation of the STags are as
   specified in this document.

8.3.1.  Allocation of STags

   When the iSCSI Layer layer at the initiator invokes the Send_Control
   Operational Primitive to request the iSER Layer layer at the initiator to
   process a SCSI Command, zero, one, or two STags may be allocated by
   the iSER Layer. layer.  See section Section 7.3.1 for details.  The number of STags
   allocated depends on whether the command is unidirectional or
   bidirectional and whether or not solicited write data transfer is involved
   or not.
   involved.

   When the iSCSI Layer layer at the initiator invokes the Send_Control
   Operational Primitive to request the iSER Layer layer at the initiator to
   process a Task Management Function Request with the TASK REASSIGN
   function, besides allocating zero, one, or two STags, the iSER Layer layer
   MUST invalidate the existing STags (if any) associated with the ITT.
   See section Section 7.3.3 for details.

   The iSER Layer layer at the target allocates a local Data Sink STag when
   the iSCSI Layer layer at the target invokes the Get_Data Operational
   Primitive to request the iSER Layer layer to process an R2T PDU.  See
   section
   Section 7.3.6 for details.

8.3.2

8.3.2.  Invalidation of STags

   The invalidation of the STags at the initiator at the completion of a
   unidirectional or bidirectional command when the associated SCSI
   Response PDU is sent by the target is described in section Section 7.3.2.

   When a unidirectional or bidirectional command concludes without the
   associated SCSI Response PDU being sent by the target, the iSCSI
   Layer
   layer at the initiator MUST request the iSER Layer layer at the initiator
   to invalidate the STags by invoking the Deallocate_Task_Resources
   Operational Primitive qualified with ITT.  In response, the iSER
   Layer
   layer at the initiator MUST locate the STags (if any) in the Local
   Mapping.  The iSER Layer layer at the initiator MUST invalidate the STags
   (if any) and the Local Mapping.

   For an RDMA Read operation used to realize a SCSI Write data
   transfer, the iSER Layer layer at the target SHOULD invalidate the Data
   Sink STag at the conclusion of the RDMA Read operation referencing
   the Data Sink STag (to permit the immediate reuse of buffer
   resources).

   For an RDMA Write operation used to realize a SCSI Read data
   transfer, the Data Source STag at the target is not declared to the
   initiator and is not exposed on the wire.  Invalidation of the STag
   is thus not specified.

   When a unidirectional or bidirectional command concludes without the
   associated SCSI Response PDU being sent by the target, the iSCSI
   Layer
   layer at the target MUST request the iSER Layer layer at the target to
   invalidate the STags by invoking the Deallocate_Task_Resources
   Operational Primitive qualified with ITT.  In response, the iSER
   Layer
   layer at the target MUST locate the local STags (if any) in the Local
   Mapping.  The iSER Layer layer at the target MUST invalidate the local
   STags (if any) and the Local Mapping.

9

9.  iSER Control and Data Transfer

   For iSCSI data-type PDUs (see section Section 7.1), the iSER Layer layer uses RDMA
   Read and RDMA Write operations to transfer the solicited data.  For
   iSCSI control-type PDUs (see section Section 7.2), the iSER Layer layer uses Send
   Messages of RCaP.

9.1

9.1.  iSER Header Format

   An iSER header MUST be present in every Send Message of RCaP.  The
   iSER header is located in the first 28 bytes of the message payload
   of the Send Message of RCaP, as shown in Figure 2.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Opcode|                  Opcode Specific Fields               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Opcode Specific Fields (32 bits)           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                    Opcode Specific Fields (64 bits)           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Opcode Specific Fields (32 bits)           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                    Opcode Specific Fields (64 bits)           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                     Figure 2 2: iSER Header Format

   Opcode - Operation Code: 4 bits

         The Opcode field identifies the type of iSER Messages:

              0001b = iSCSI control-type PDU

              0010b = iSER Hello Message

              0011b = iSER HelloReply Message

              All other opcodes Opcodes are reserved.

9.2 unassigned.

9.2.  iSER Header Format for iSCSI Control-Type PDU

   The iSER Layer layer uses Send Messages of RCaP to transfer iSCSI control-
   type PDUs (see section Section 7.2).  The message payload of each of the Send
   Messages of RCaP used for transferring an iSER Message contains an
   iSER Header followed by an iSCSI control-type PDU.

   The iSER header in a Send Message of RCaP carrying an iSCSI control-
   type PDU MUST have the format as described in Figure 3.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       |W|R|                                                   |
   | 0001b |S|S|                  Reserved                         |
   |       |V|V|                                                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Write STag                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                         Write Base Offset                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             Read STag                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                         Read Base Offset                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

       Figure 3 3: iSER Header Format for iSCSI Control-Type PDU

   WSV - Write STag Valid flag: 1 bit

      This flag indicates the validity of the Write STag field and the
      Write Base Offset field of the iSER Header.  If set to one, the
      Write STag field and the Write Base Offset field in this iSER
      Header are valid.  If set to zero, the Write STag field and the
      Write Base Offset field in this iSER Header MUST be ignored at the
      receiver.  The Write STag Valid flag is set to one when there is
      solicited data to be transferred for a SCSI Write or bidirectional
      command, or when there are non-immediate unsolicited and solicited
      data to be transferred for the referenced task specified in a Task
      Management Function Request with the TASK REASSIGN function.

   RSV - Read STag Valid flag: 1 bit

      This flag indicates the validity of the Read STag field and the
      Read Base Offset field of the iSER Header.  If set to one, the
      Read STag field and the Read Base Offset field in this iSER Header is
      are valid.  If set to zero, the Read STag field and the Read Base
      Offset field in this iSER Header MUST be ignored at the receiver.
      The Read STag Valid flag is set to one for a SCSI Read or
      bidirectional command, or a Task Management Function Request with
      the TASK REASSIGN function.

   Write STag - Write Steering Tag: 32 bits

      This field contains the Write STag when the Write STag Valid flag
      is set to one.  For a SCSI Write or bidirectional command, the
      Write STag is used to Advertise the initiator's I/O Buffer
      containing the solicited data.  For a Task Management Function
      Request with the TASK REASSIGN function, the Write STag is used to
      Advertise the initiator's I/O Buffer containing the non-
        immediate non-immediate
      unsolicited data and solicited data.  This Write STag is used as
      the Data Source STag in the resultant RDMA Read operation(s).
      When the Write STag Valid flag is set to zero, this field MUST be
      set to zero and ignored on receive.

   Write Base Offset: 64 bits

      This field contains the Base Offset associated with the I/O Buffer
      for the SCSI Write command when the Write STag Valid flag is set
      to one.  When the Write STag Valid flag is set to zero, this field
      MUST be set to zero and ignored on receive.

   Read STag - Read Steering Tag: 32 bits

      This field contains the Read STag when the Read STag Valid flag is
      set to one.  The Read STag is used to Advertise the initiator's
      Read I/O Buffer of a SCSI Read or bidirectional command, or a Task
      Management Function Request with the TASK REASSIGN function.  This
      Read STag is used as the Data Sink STag in the resultant RDMA
      Write operation(s).  When the Read STag Valid flag is zero, this
      field MUST be set to zero and ignored on receive.

   Read Base Offset: 64 bits

      This field contains the Base Offset associated with the I/O Buffer
      for the SCSI Read command when the Read STag Valid flag is set to
      one.  When the Read STag Valid flag is set to zero, this field
      MUST be set to zero and ignored on receive.

   Reserved:

      Reserved fields MUST be set to zero on transmit and MUST be
      ignored on receive.

9.3

9.3.  iSER Header Format for iSER Hello Message

   An iSER Hello Message MUST only contain the iSER header header, which MUST
   have the format as described in Figure 4.  If iSERHelloRequired is
   negotiated to "Yes", then iSER Hello Message is the first iSER
   Message sent on the RCaP Stream from the iSER Layer layer at the initiator
   to the iSER Layer layer at the target.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       |       |       |       |                               |
      | 0010b | Rsvd  | MaxVer| MinVer|           iSER-IRD            |
      |       |       |       |       |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                           Reserved                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      |                           Reserved                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                           Reserved                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      |                           Reserved                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

            Figure 4 4: iSER Header Format for iSER Hello Message

   MaxVer - Maximum Version: 4 bits

      This field specifies the maximum version of the iSER protocol
      supported.  It MUST be set to 10 to indicate the version of the
      specification described in this document.

   MinVer - Minimum Version: 4 bits

      This field specifies the minimum version of the iSER protocol
      supported.  It MUST be set to 10 to indicate the version of the
      specification described in this document.

   iSER-IRD: 16 bits

      This field contains the value of the iSER-IRD at the initiator.

   Reserved (Rsvd):

      Reserved fields MUST be set to zero on transmit, transmit and MUST be
      ignored on receive.

9.4

9.4.  iSER Header Format for iSER HelloReply Message

   An iSER HelloReply Message MUST only contain the iSER header header, which
   MUST have the format as described in Figure 5.  If iSERHelloRequired
   is negotiated to "Yes", then the iSER HelloReply Message is the first
   iSER Message sent on the RCaP Stream from the iSER Layer layer at the
   target to the iSER Layer layer at the initiator.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       |     |R|       |       |                               |
   | 0011b |Rsvd |E| MaxVer| CurVer|           iSER-ORD            |
   |       |     |J|       |       |                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           Reserved                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           Reserved                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           Reserved                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           Reserved                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

       Figure 5 5: iSER Header Format for iSER HelloReply Message

   REJ - Reject flag: 1 bit

      This flag indicates whether the target is rejecting this
      connection.  If set to one, the target is rejecting the
      connection.

   MaxVer - Maximum Version: 4 bits

      This field specifies the maximum version of the iSER protocol
      supported.  It MUST be set to 10 to indicate the version of the
      specification described in this document.

   CurVer - Current Version: 4 bits

      This field specifies the current version of the iSER protocol
      supported.  It MUST be set to 10 to indicate the version of the
      specification described in this document.

   iSER-ORD: 16 bits

      This field contains the value of the iSER-ORD at the target.

   Reserved (Rsvd):

      Reserved fields MUST be set to zero on transmit, transmit and MUST be
      ignored on receive.

9.5

9.5.  SCSI Data Transfer Operations

   The iSER Layer layer at the initiator and the iSER Layer layer at the target
   handle each SCSI Write, SCSI Read, and bidirectional operation as
   described below.

9.5.1

9.5.1.  SCSI Write Operation

   The iSCSI Layer layer at the initiator MUST invoke the Send_Control
   Operational Primitive to request the iSER Layer layer at the initiator to
   send the SCSI Write Command.  The iSER Layer layer at the initiator MUST
   request the RCaP layer to transmit a Send Message with the message
   payload consisting of the iSER header followed by the SCSI Command
   PDU and immediate data (if any).  The SendSE Message should be used
   if supported by the RCaP layer (e.g., iWARP).  If there is solicited
   data, the iSER Layer layer MUST Advertise the Write STag and the Base
   Offset in the iSER header of the Send Message, as described in
   section
   Section 9.2.  Upon receiving the Send Message, the iSER Layer layer at the
   target MUST notify the iSCSI Layer layer at the target by invoking the
   Control_Notify Operational Primitive qualified with the SCSI Command
   PDU.  See section Section 7.3.1 for details on the handling of the SCSI Write
   Command.

   For the non-immediate unsolicited data, the iSCSI Layer layer at the
   initiator MUST invoke a Send_Control Operational Primitive qualified
   with the SCSI Data-out Data-Out PDU.  Upon receiving each Send Message
   containing the non-immediate unsolicited data, the iSER Layer layer at the
   target MUST notify the iSCSI Layer layer at the target by invoking the
   Control_Notify Operational Primitive qualified with the SCSI Data-
   out Data-Out
   PDU.  See section Section 7.3.4 for details on the handling of the SCSI
   Data-out Data-
   Out PDU.

   For the solicited data, when the iSCSI Layer layer at the target has an I/O
   Buffer available, it MUST invoke the Get_Data Operational Primitive
   qualified with the R2T PDU.  See section Section 7.3.6 for details on the
   handling of the R2T PDU.

   When the data transfer associated with this SCSI Write operation is
   complete, the iSCSI Layer layer at the target MUST invoke the Send_Control
   Operational Primitive when it is ready to send the SCSI Response PDU.
   Upon receiving a Send Message containing the SCSI Response PDU, the
   iSER Layer layer at the initiator MUST notify the iSCSI Layer layer at the
   initiator by invoking the Control_Notify Operational Primitive
   qualified with the SCSI Response PDU.  See section Section 7.3.2 for details
   on the handling of the SCSI Response PDU.

9.5.2

9.5.2.  SCSI Read Operation

   The iSCSI Layer layer at the initiator MUST invoke the Send_Control
   Operational Primitive to request the iSER Layer layer at the initiator to
   send the SCSI Read Command.  The iSER Layer layer at the initiator MUST
   request the RCaP layer to transmit a Send Message with the message
   payload consisting of the iSER header followed by the SCSI Command
   PDU.  The SendSE Message should be used if supported by the RCaP
   layer (e.g., iWARP).  The iSER Layer layer at the initiator MUST Advertise
   the Read STag and the Base Offset in the iSER header of the Send
   Message, as described in section Section 9.2.  Upon receiving the Send
   Message, the iSER Layer layer at the target MUST notify the iSCSI Layer layer at
   the target by invoking the Control_Notify Operational Primitive
   qualified with the SCSI Command PDU.  See section Section 7.3.1 for details
   on the handling of the SCSI Read Command.

   When the requested SCSI data is available in the I/O Buffer, the
   iSCSI Layer layer at the target MUST invoke the Put_Data Operational
   Primitive qualified with the SCSI Data-in Data-In PDU.  See section Section 7.3.5 for
   details on the handling of the SCSI Data-in Data-In PDU.

   When the data transfer associated with this SCSI Read operation is
   complete, the iSCSI Layer layer at the target MUST invoke the Send_Control
   Operational Primitive when it is ready to send the SCSI Response PDU.
   The SendInvSE Message should be used if supported by the RCaP layer
   (e.g., iWARP).  Upon receiving the Send Message containing the SCSI
   Response PDU, the iSER Layer layer at the initiator MUST notify the iSCSI Layer
   layer at the initiator by invoking the Control_Notify Operational
   Primitive qualified with the SCSI Response PDU.  See
   section Section 7.3.2
   for details on the handling of the SCSI Response PDU.

9.5.3

9.5.3.  Bidirectional Operation

   The initiator and the target handle the SCSI Write and the SCSI Read
   portions of this bidirectional operation the same as described in
   Section
   Sections 9.5.1 and Section 9.5.2 9.5.2, respectively.

10

10.  iSER Error Handling and Recovery

   RCaP provides the iSER Layer layer with reliable in-order delivery.
   Therefore, the error management needs of an iSER-assisted connection
   are somewhat different than those of a Traditional iSCSI connection.

10.1

10.1.  Error Handling

   iSER error handling is described in the following sections,
   classified loosely based on the sources of errors:

   1. Those originating at the transport layer (e.g., TCP).

   2. Those originating at the RCaP layer.

   3. Those originating at the iSER Layer. layer.

   4. Those originating at the iSCSI Layer.

10.1.1 layer.

10.1.1.  Errors in the Transport Layer

   If the transport layer is TCP, then TCP packets with detected errors
   are silently dropped by the TCP layer and result in retransmission at
   the TCP layer.  This has no impact on the iSER Layer. layer.  However,
   connection loss (e.g., link failure) and unexpected termination
   (e.g., TCP graceful or abnormal close without the iSCSI Logout
   exchanges) at the transport layer will cause the iSCSI/iSER
   connection to be terminated as well.

10.1.1.1

10.1.1.1.  Failure in the Transport Layer Before RCaP Mode is Enabled

   If the Connection connection is lost or terminated before the iSCSI Layer layer
   invokes the Allocate_Connection_Resources Operational Primitive, the
   login process is terminated and no further action is required.

   If the Connection connection is lost or terminated after the iSCSI Layer layer has
   invoked the Allocate_Connection_Resources Operational Primitive, then
   the iSCSI Layer layer MUST request the iSER Layer layer to deallocate all
   connection resources by invoking the Deallocate_Connection_Resources
   Operational Primitive.

10.1.1.2

10.1.1.2.  Failure in the Transport Layer After RCaP Mode is Enabled

   If the Connection connection is lost or terminated after the iSCSI Layer layer has
   invoked the Enable_Datamover Operational Primitive, the iSER Layer layer
   MUST notify the iSCSI Layer layer of the connection loss by invoking the
   Connection_Terminate_Notify Operational Primitive.  Prior to invoking
   the Connection_Terminate_Notify Operational Primitive, the iSER layer
   MUST perform the actions described in Section 5.2.3.2.

10.1.2

10.1.2.  Errors in the RCaP Layer

   The RCaP layer does not have error recovery operations built in.  If
   errors are detected at the RCaP layer, the RCaP layer will terminate
   the RCaP Stream and the associated Connection.

10.1.2.1 connection.

10.1.2.1.  Errors Detected in the Local RCaP Layer

   If an error is encountered at the local RCaP layer, the RCaP layer
   MAY send a Send Message to the Remote Peer to report the error if
   possible.  (For iWARP, see [RDMAP] for the list of errors where a
   Terminate Message is sent.)  The RCaP layer is responsible for
   terminating the Connection. connection.  After the RCaP layer notifies the iSER
   Layer
   layer that the Connection connection is terminated, the iSER Layer layer MUST notify
   the iSCSI Layer layer by invoking the Connection_Terminate_Notify
   Operational Primitive.  Prior to invoking the Connection Terminate
   Notify
   Connection_Terminate_Notify Operational Primitive, the iSER layer
   MUST perform the actions described in Section 5.2.3.2.

10.1.2.2

10.1.2.2.  Errors Detected in the RCaP Layer at the Remote Peer

   If an error is encountered at the RCaP layer at the Remote Peer, the
   RCaP layer at the Remote Peer may send a Send Message to report the
   error if possible.  If it is unable to send a Send Message, the
   Connection
   connection is terminated.  This is treated the same as a failure in
   the transport layer after RDMA is enabled enabled, as described in section Section
   10.1.1.2.

   If an error is encountered at the RCaP layer at the Remote Peer and
   it is able to send a Send Message, the RCaP layer at the Remote Peer
   is responsible for terminating the connection.  After the local RCaP
   layer notifies the iSER Layer layer that the Connection connection is terminated, the
   iSER Layer layer MUST notify the iSCSI Layer layer by invoking the Connection
   Terminate Notify
   Connection_Terminate_Notify Operational Primitive.  Prior to invoking
   the Connection_Terminate_Notify Operational Primitive, the iSER layer
   MUST perform the actions described in Section 5.2.3.2.

10.1.3

10.1.3.  Errors in the iSER Layer

   The error handling due to errors at the iSER Layer layer is described in
   the following sections.

10.1.3.1

10.1.3.1.  Insufficient Connection Resources to Support RCaP at
           Connection Setup

   After the iSCSI Layer layer at the initiator invokes the
   Allocate_Connection_Resources Operational Primitive during the iSCSI
   login negotiation phase, if the iSER Layer layer at the initiator fails to
   allocate the connection resources necessary to support RCaP, it MUST
   return a status of failure to the iSCSI Layer layer at the initiator.  The
   iSCSI Layer layer at the initiator MUST terminate the Connection connection as
   described in Section 5.2.3.1.

   After the iSCSI Layer layer at the target invokes the
   Allocate_Connection_Resources Operational Primitive during the iSCSI
   login negotiation phase, if the iSER Layer layer at the target fails to
   allocate the connection resources necessary to support RCaP, it MUST
   return a status of failure to the iSCSI Layer layer at the target.  The
   iSCSI Layer layer at the target MUST send a Login Response with a status
   class Status-
   Class of 3 0x03 (Target Error), and a status code Status-Code of "0302" 0x02 (Out of
   Resources).  The iSCSI Layers layers at the initiator and the target MUST
   terminate the Connection connection as described in Section 5.2.3.1.

10.1.3.2

10.1.3.2.  iSER Negotiation Failures

   If iSERHelloRequired is negotiated to "Yes" and the RCaP or iSER
   related parameters declared by the initiator in the iSER Hello
   Message is are unacceptable to the iSER Layer layer at the target, the iSER
   Layer
   layer at the target MUST set the Reject (REJ) flag, as described in
   section
   Section 9.4, in the iSER HelloReply Message.  The following are the
   cases when the iSER Layer layer MUST set the REJ flag to 1 in the
   HelloReply Message:

   *  The initiator-declared iSER-IRD value is greater than 0 0, and the
      target-declared iSER-ORD value is 0.

   *  The initiator-supported and the target-supported iSER protocol
      versions do not overlap.

   After requesting the RCaP layer to send the iSER HelloReply Message,
   the handling of the error situation is the same as that for iSER
   format errors as described in section Section 10.1.3.3.

10.1.3.3.

10.1.3.3  iSER Format Errors

   The following types of errors in an iSER header are considered format
   errors:

   *  Illegal contents of any iSER header field

   *  Inconsistent field contents in an iSER header

   *  Length error for an iSER Hello or HelloReply Message (see section Sections
      9.3 and 9.4)

   When a format error is detected, the following events MUST occur in
   the specified sequence:

   1. The iSER Layer layer MUST request the RCaP layer to terminate the RCaP
      Stream.  The RCaP layer MUST terminate the associated
       Connection. connection.

   2.  The iSER Layer layer MUST notify the iSCSI Layer layer of the connection
      termination by invoking the Connection_Terminate_Notify
      Operational Primitive.  Prior to invoking the
      Connection_Terminate_Notify Operational Primitive, the iSER layer
      MUST perform the actions described in Section 5.2.3.2.

10.1.3.4

10.1.3.4.  iSER Protocol Errors

   If iSERHelloRequired is negotiated to "Yes", then the first iSER
   Message sent by the iSER Layer layer at the initiator MUST be the iSER
   Hello Message (see section Section 9.3).  In this case the first iSER Message
   sent by the iSER Layer layer at the target MUST be the iSER HelloReply
   Message (see section Section 9.4).  Failure to send the iSER Hello or
   HelloReply Message, as indicated by the wrong Opcode in the iSER
   header, is a protocol error.  Conversely, if the iSER Hello Message
   is sent by the iSER Layer layer at the initiator when iSERHelloRequired is
   negotiated to "No", the iSER Layer layer at the target MAY treat this as a
   protocol error or respond with an iSER HelloReply Message.  The
   handling of iSER protocol errors is the same as that for iSER format
   errors as described in section Section 10.1.3.3.

   If the sending side of an iSER-enabled connection acts in a manner
   not permitted by the negotiated or declared login/text operational
   key values as described in section Section 6, this is a protocol error and
   the receiving side MAY handle this the same as for iSER format errors
   as described in section Section 10.1.3.3.

10.1.4

10.1.4.  Errors in the iSCSI Layer

   The error handling due to errors at the iSCSI Layer layer is described in
   the following sections.  For error recovery, see section Section 10.2.

10.1.4.1

10.1.4.1.  iSCSI Format Errors

   When an iSCSI format error is detected, the iSCSI Layer layer MUST request
   the iSER Layer layer to terminate the RCaP Stream by invoking the
   Connection_Terminate Operational Primitive.  For more details on the
   connection termination, see Section 5.2.3.1.

10.1.4.2

10.1.4.2.  iSCSI Digest Errors

   In the iSER-assisted mode, the iSCSI Layer layer will not see any digest
   error because both the HeaderDigest and the DataDigest keys are
   negotiated to "None".

10.1.4.3

10.1.4.3.  iSCSI Sequence Errors

   For Traditional iSCSI, sequence errors are caused by dropped PDUs due
   to header or data digest errors.  Since digests are not used in
   iSER-assisted iSER-
   assisted mode and the RCaP layer will deliver all messages in the
   order they were sent, sequence errors will not occur in iSER-
   assisted iSER-assisted
   mode.

10.1.4.4

10.1.4.4.  iSCSI Protocol Error

   When the iSCSI Layer layer handles certain protocol errors by dropping the
   connection, the error handling is the same as that for iSCSI format
   errors as described in section Section 10.1.4.1.

   When the iSCSI Layer layer uses the iSCSI Reject PDU and response codes to
   handle certain other protocol errors, no special handling at the iSER Layer
   layer is required.

10.1.4.5

10.1.4.5.  SCSI Timeouts and Session Errors

   This is handled at the iSCSI Layer layer, and no special handling at the
   iSER Layer layer is required.

10.1.4.6

10.1.4.6.  iSCSI Negotiation Failures

   For negotiation failures that happen during the Login Phase at the
   initiator after the iSCSI Layer layer has invoked the
   Allocate_Connection_Resources Operational Primitive and before the
   Enable_Datamover Operational Primitive has been invoked, the iSCSI
   Layer
   layer MUST request the iSER Layer layer to deallocate all connection
   resources by invoking the Deallocate_Connection_Resources Operational
   Primitive.  The iSCSI Layer layer at the initiator MUST terminate the Connection.
   connection.

   For negotiation failures during the Login Phase at the target, the
   iSCSI Layer layer can use a Login Response with a status class Status-Class other than 0
   (success) to terminate the Login Phase.  If the iSCSI Layer layer has
   invoked the Allocate_Connection_Resources Operational Primitive and
   before
   has not yet invoked the Enable_Datamover Operational Primitive has been invoked, Primitive, the
   iSCSI Layer layer at the target MUST request the iSER Layer layer at the target
   to deallocate all connection resources by invoking the
   Deallocate_Connection_Resources Operational Primitive.  The iSCSI
   Layer
   layer at both the initiator and the target MUST terminate the
   Connection.
   connection.

   During the iSCSI Login Phase, if the iSCSI Layer layer at the initiator
   receives a Login Response from the target with a status class Status-Class other
   than 0 (Success) after the iSCSI Layer layer at the initiator has invoked
   the Allocate_Connection_Resources Operational Primitive, the iSCSI
   Layer
   layer MUST request the iSER Layer layer to deallocate all connection
   resources by invoking the Deallocate_Connection_Resources Operational
   Primitive.  The iSCSI Layer layer MUST terminate the
   Connection connection in this
   case.

   For negotiation failures during the full feature phase, Full Feature Phase, the error
   handling is left to the iSCSI Layer layer and no special handling at the
   iSER Layer layer is required.

10.2

10.2.  Error Recovery

   Error recovery requirements of iSCSI/iSER are the same as that of
   Traditional iSCSI.  All three ErrorRecoveryLevels as defined in
   [iSCSI] are supported in iSCSI/iSER.

   *  For ErrorRecoveryLevel 0, session recovery is handled by iSCSI and
      no special handling by the iSER Layer layer is required.

   *  For ErrorRecoveryLevel 1, see section Section 10.2.1 on PDU Recovery.

   *  For ErrorRecoveryLevel 2, see section Section 10.2.2 on Connection
      Recovery.

   The iSCSI Layer layer may invoke the Notice_Key_Values Operational
   Primitive during connection setup to request the iSER Layer layer to take
   note of the value of the operational ErrorRecoveryLevel, as described
   in sections Sections 5.1.1 and 5.1.2.

10.2.1

10.2.1.  PDU Recovery

   As described in sections Sections 10.1.4.2 and 10.1.4.3, digest and sequence
   errors will not occur in the iSER-assisted mode.  If the RCaP layer
   detects an error, it will close the iSCSI/iSER connection, as
   described in section Section 10.1.2.  Therefore, PDU recovery is not useful
   in the iSER-assisted mode.

   The iSCSI Layer layer at the initiator SHOULD disable iSCSI timeout-driven
   PDU retransmissions.

10.2.2

10.2.2.  Connection Recovery

   The iSCSI Layer layer at the initiator MAY reassign connection allegiance
   for non-immediate commands which that are still in progress and are
   associated with the failed connection by using a Task Management
   Function Request with the TASK REASSIGN function.  See section Section 7.3.3
   for more details.

   When the iSCSI Layer layer at the initiator does a task reassignment for a
   SCSI Write command, it MUST qualify the Send_Control Operational
   Primitive invocation with DataDescriptorOut DataDescriptorOut, which defines the I/O
   Buffer for both the non-immediate unsolicited data and the solicited
   data.  This allows the iSCSI Layer layer at the target to use recovery R2Ts
   to request for data originally sent as unsolicited and solicited from the
   initiator.

   When the iSCSI Layer layer at the target accepts a reassignment request for
   a SCSI Read command, it MUST request the iSER Layer layer to process SCSI Data-in
   Data-In for all unacknowledged data by invoking the Put_Data
   Operational Primitive.  See section Section 7.3.5 on the handling of SCSI
   Data-in.
   Data-In.

   When the iSCSI Layer layer at the target accepts a reassignment request for
   a SCSI Write command, it MUST request the iSER Layer layer to process a
   recovery R2T for any non-immediate unsolicited data and any solicited
   data sequences that have not been received by invoking the Get_Data
   Operational Primitive.  See section Section 7.3.6 on the handling of Ready To
   Transfer (R2T).

   The iSCSI Layer layer at the target MUST NOT issue recovery R2Ts on an
   iSCSI/iSER connection for a task for which the connection allegiance
   was never reassigned.  The iSER Layer layer at the target MAY reject such a
   recovery R2T received via the Get_Data Operational Primitive
   invocation from the iSCSI Layer layer at the target, with an appropriate
   error code.

   The iSER Layer layer at the target will process the requests invoked by the
   Put_Data and Get_Data Operational Primitives for a reassigned task in
   the same way as for the original commands.

11

11.  Security Considerations

   When iSER is layered on top of an RCaP layer and provides the RDMA
   extensions to the iSCSI protocol, the security considerations of iSER
   are the same as that of the underlying RCaP layer.  For iWARP, this
   is described in [RDMAP] and [RDDPSEC], plus the updates to both of
   those RFCs that are contained in [IPSEC-IPS].

   Since iSER-assisted iSCSI protocol is still functionally iSCSI from a
   security considerations perspective, all of the iSCSI security
   requirements as described in [iSCSI] applies. apply.  If iSER is layered on
   top of a non-IP based non-IP-based RCaP layer, all the security protocol
   mechanisms applicable to that RCaP layer is are also applicable to an
   iSCSI/iSER connection.  If iSER is layered on top of a non-IP
   protocol, the IPsec mechanism as specified in [iSCSI] MUST be
   implemented at any point where the iSER protocol enters the IP
   network (e.g., via gateways), and the non-IP protocol SHOULD
   implement (optional to use) a packet by packet packet-by-packet security protocol
   equal in strength to the IPsec mechanism specified by [iSCSI].

   In order to protect target RCaP connection resources from possible
   resource exhaustion attacks, allocation of such resources for a new
   connection MUST be delayed until it is reasonably certain that the
   new connection is not part of a resource exhaustion attack (e.g.,
   until after the SecurityNegotiation stage of Login), Login); see section Section
   5.1.2.

   A valid STag exposes I/O Buffer resources to the network for access
   via the RCaP.  The security measures for the RCAP and iSER described
   in the above paragraphs can be used to protect data in an I/O buffer
   from undesired disclosure or modification, and these measures are of
   heightened importance for implementations that retain (e.g., cache)
   STags for use in multiple tasks (e.g., iSCSI I/O operations) because
   the resources are exposed to the network for a longer period of time.

   A complementary means of controlling I/O Buffer resource exposure is
   invalidation of the STag after completion of the associated task,
   which is RECOMMENDED as
   specified in Section 2.5.1. 1.5.1.  The use of Send with Invalidate messages
   (which cause remote STag invalidation) is OPTIONAL, therefore the
   iSER layer MUST NOT rely on use of a Send with Invalidate by its
   Remote Peer to cause local STag invalidation.  If an STag is expected
   to be invalid after completion of a task, the iSER layer MUST check
   the STag and invalidate it if it is still valid.

12

12.  IANA Considerations

   IANA is requested to add has added the following entries to the "iSCSI Login/Text Keys" registry of "iSCSI Parameters":
   registry:

      MaxAHSLength, [RFCXXXX] RFC 7145

      TaggedBufferForSolicitedDataOnly, [RFCXXXX]

   iSERHelloRequired, [RFCXXXX] RFC Editor: Please replace XXXX in all instances of [RFCXXXX] above
   with the 7145

      iSERHelloRequired, RFC number of this document and remove this note. 7145

   IANA is requested to update has updated the following entries in the "iSCSI Login/Text Keys"
   registry of "iSCSI Parameters" to reference the RFC
   number of this draft when it is published as an RFC.

      InitiatorRecvDataSegmentLength

      MaxOutstandingUnexpectedPDUs

      RDMAExtensions

      TargetRecvDataSegmentLength

   IANA is has also requested to change changed the RFC5046 reference to RFC 5046 for the iSCSI "iSCSI
   Login/Text Keys Keys" registry to the RFC number of refer to this document. RFC.

   IANA is requested to update has updated the registrations of the iSER Opcodes 1-
   3 1-3 in the iSER Opcodes
   "iSER Opcodes" registry to reference this RFC.  IANA has also changed
   the reference to RFC number of 5046 for the "iSER Opcodes" registry to refer to
   this
   draft when it is published as an RFC.

13

13.  References

13.1

13.1.  Normative References

   [RFC5046] M. Ko et al., "iSCSI Externsions   Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah,
               H., and P. Thaler, "Internet Small Computer System
               Interface (iSCSI) Extensions for Remote Direct Memory
       Access",
               Access (RDMA)", RFC 5046, October 2007 2007.

   [iSCSI] Chadalapaka et al., "iSCSI     Chadalapaka, M., Satran, J., Meth, K., and D. Black,
               "Internet Small Computer System Interface (iSCSI)
               Protocol (Consolidated)", draft-
       ietf-storm-iscsi-cons-08.txt (work in progress), January 2013 RFC 7143, April 2014.

   [RDMAP] R. Recio et al., "An RDMA     Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
               Garcia, "A Remote Direct Memory Access Protocol
               Specification", RFC 5040, October 2007 2007.

   [DDP] H. Shah et al.,       Shah, H., Pinkerton, J., Recio, R., and P. Culley,
               "Direct Data Placement over Reliable Transports", RFC
               5041, October 2007 2007.

   [MPA] P. Culley et al.,       Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
               Carrier, "Marker PDU Aligned Framing for TCP
               Specification", RFC 5044, October 2007 2007.

   [RDDPSEC]   Pinkerton, J. Pinkerton et al., "DDP/RDMAP and E. Deleganes, "Direct Data Placement
               Protocol (DDP) / Remote Direct Memory Access Protocol
               (RDMAP) Security", RFC 5042, October 2007 2007.

   [TCP]       Postel, J., "Transmission Control Protocol", STD 7, RFC
               793, September 1981 1981.

   [RFC2119]   Bradner, S., "Key Words words for use in RFCs to Indicate
               Requirement Levels", BCP 14, RFC 2119, March 1997

   [IPS-IPSEC] 1997.

   [IPSEC-IPS] Black, D. Black et al., and P. Koning, "Securing Block Storage
               Protocols over IP: RFC 3723 Requirements Update for IPsec
               v3", draft-ietf-
       storm-ipsec-ips-update-03 (work in progress), July 2013

13.2 RFC 7146, April 2014.

13.2.  Informative References

   [SAM5] T10/2104D rev r04, SCSI      INCITS Technical Committee T10, "SCSI Architecture Model
               - 5 (SAM-5), (SAM-5)", T10/BSR INCITS 515 rev 04, Committee Draft.

   [iSCSI-SAM] Knight, F. Knight et al., and M. Chadalapaka, "Internet Small Computer Systems
               System Interface (iSCSI) SCSI Architecture Features Update", draft-
       ietf-storm-iscsi-sam-04.txt (work in progress), August 2011 RFC 7144,
               April 2014.

   [DA] M. Chadalapaka et al., "Datamover        Chadalapaka, M., Hufferd, J., Satran, J., and H. Shah,
               "DA: Datamover Architecture for iSCSI", the Internet Small
               Computer System Interface (iSCSI)", RFC 5047, October 2007
               2007.

   [IB]        InfiniBand Architecture Specification Volume 1 Release
               1.2, October 2004

   [IPoIB] H.K. Chu et al,     Chu, J. and V. Kashyap, "Transmission of IP over InfiniBand",
               InfiniBand (IPoIB)", RFC 4391, March 2006

14 April 2006.

Appendix A: A.  Summary of Changes from RFC 5046

   All changes are backward compatible with RFC 5046 except for item #8 #8,
   which reflects all known implementations of iSER, each of which has
   implemented this change, despite its absence in RFC 5046.  As a
   result, a hypothetical implementation based on RFC 5046 will not
   interoperate with an implementation based on this version of the
   specification.

   1.  Removed the requirement that a connection be opened in "normal"
       TCP mode and transitioned to zero-copy mode.  This allows the spec
       specification to conform to existing implementation implementations for both Infiniband
       InfiniBand and iWARP.  Changes were made in sections 2, Sections 1, 3.1.6,
       4.2, 5.1, 5.1.1, 5.1.2, 5.1.3, 10.1.3.4, and 11.

   2.  Added a clause in section Section 6.2 to clarify that
       MaxRecvDataSegmentLength must be ignored if it is declared in the
       Login Phase.

   3.  Added a clause in section Section 6.2 to clarify that the initiator must
       not send more than InitiatorMaxRecvDataSegmentLength worth of
       data when a NOP-Out request is sent with a valid Initiator Task
       Tag.  Since InitiatorMaxRecvDataSegmentLength can be smaller than
       TargetMaxRecvDataSegmentLength, returning the original data in
       the NOP-Out request in this situation can overflow the receive
       buffer unless the length of the data sent with the NOP-Out
       request is less than InitiatorMaxRecvDataSegmentLength.

   4.  Added a SHOULD negotiate recommendation for
       MaxOutstandingUnexpectedPDUs in section Section 6.7.

   5.  Added MaxAHSLength key in section Section 6.8 to set a limit on the AHS
       Length.  This is useful when posting receive buffers in knowing
       what the maximum possible message length is in a PDU which that
       contains AHS.

   6.  Added TaggedBufferForSolicitedDataOnly key in section Section 6.9 to
       indicate how the memory region will be used.  An initiator can
       treat the memory regions intended for unsolicited and solicited
       data differently, differently and can use different registration modes.  In
       contrast, RFC 5046 treats the memory occupied by the data as a
       contiguous (or virtually contiguous, by means of scatter-gather
       mechanisms) and homogenous region.  Adding a new key will allow
       different memory models to be accommodated.  Changes were also
       made in section Section 7.3.1.

   7.  Added iSERHelloRequired key in section Section 6.10 to allow an initiator
       to allocate connection resources after the login process by
       requiring the use of the iSER Hello messages before sending iSCSI
       PDUs.  The default is "No" since iSER Hello messages have not
       been implemented and are not in use.  Changes were made in sections
       Sections 5.1.1, 5.1.2, 5.1.3, 8.2, 9.3, 9.4, 10.1.3.2 10.1.3.2, and
       10.1.3.4.

   8.  Added two 64-bit fields in iSER header in section Section 9.2 for the
       Read Base Offset and the Write Base Offset to accommodate a non-zero non-
       zero Base Offset.  This allows one implementation such as the OFED
       Open Fabrics Enterprise Distribution (OFED) stack to be used in
       both the Infiniband InfiniBand and the iWARP environment.

       Changes were made in the definition definitions of Base Offset,
       Advertisement, and Tagged Buffer.  Changes were also made in sections 2.4.1, 2.5,
     2.6,
       Sections 1.5.1, 1.6, 1.7, 7.3.1, 7.3.3, 7.3.5, 7.3.6, 9.1, 9.3,
       9.4, 9.5.1, and 9.5.2.  This change is not backward compatible
       with RFC 5046, but is it was part of all known implementations of
       iSER at the time this document was developed.

   9.  Remove iWARP specific iWARP-specific behavior.  Changes were made in the
     definition section on
       definitions of RDMA Operation and Send Message Type.

       Clarifications were added in section 2.4.2 Section 1.5.2 on the use of SendSE
       and SendInvSE.  These clarifications reflect a removal of the
       requirements in RFC 5046 for the use of these messages, as
       implementations have not followed RFC 5046 in this area.  Changes
       affecting Send with Invalidate were made in sections 2.4.1, 2.5,
     2.6, Sections 1.5.1, 1.6,
       1.7, 4.1, and 7.3.2.  Changes affecting Terminate were made in
     sections
       Sections 10.1.2.1 and 10.1.2.2.  Changes were made in section 15 Appendix B
       to remove iWARP headers.

   10. Removed denial of service denial-of-service descriptions for the initiator in
     section
       Section 5.1.1 since it is they are applicable for the target only.

   11. Clarified in section 2.4.1 Section 1.5.1 that STag invalidation is the
       initiator's responsibility for security reasons, and the
       initiator cannot rely on the target using an Invalidate version
       of Send.  Added text in section Section 11 on Stag invalidation.

15

Appendix B: B.  Message Format for iSER

   This section is for information only and is NOT part of the standard.

15.1

B.1.  iWARP Message Format for iSER Hello Message

   The following figure depicts an iSER Hello Message encapsulated in an
   iWARP SendSE Message.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         MPA Header            |  DDP Control  | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Reserved                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       (Send) Queue Number                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 (Send) Message Sequence Number                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      (Send) Message Offset                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | 0010b | Zeros | 0001b | 0001b |           iSER-IRD            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           All Zeros                           |

   |                           MPA CRC                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Figure 6 6: SendSE Message containing Containing an iSER Hello Message

15.2

B.2.  iWARP Message Format for iSER HelloReply Message

   The following figure depicts an iSER HelloReply Message encapsulated
   in an iWARP SendSE Message.  The Reject (REJ) flag is set to 0. zero.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         MPA Header            |  DDP Control  | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Reserved                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       (Send) Queue Number                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 (Send) Message Sequence Number                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      (Send) Message Offset                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | 0011b |Zeros|0| 0001b | 0001b |           iSER-ORD            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           MPA CRC                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Figure 7 7: SendSE Message containing Containing an iSER HelloReply Message

15.3

B.3.  iSER Header Format for SCSI Read Command PDU

   The following figure depicts a SCSI Read Command PDU embedded in an
   iSER Message.  For this particular example, in the iSER header, the
   Write STag Valid flag is set to zero, the Read STag Valid flag is set
   to one, the Write STag field is set to all zeros, the Write Base
   Offset field is set to all zeros, the Read STag field contains a
   valid Read STag, and the Read Base Offset field contains a valid Base
   Offset for the Read Tagged Buffer.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | 0001b |0|1|                  All zeros                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         All Zeros                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                         All Zeros                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Read STag                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                       Read Base Offset                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       SCSI Read Command PDU                   |
   //                                                             //
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        Figure 8 8: iSER Header Format for SCSI Read Command PDU

15.4

B.4.  iSER Header Format for SCSI Write Command PDU

   The following figure depicts a SCSI Write Command PDU embedded in an
   iSER Message.  For this particular example, in the iSER header, the
   Write STag Valid flag is set to one, the Read STag Valid flag is set
   to zero, the Write STag field contains a valid Write STag, the Write
   Base Offset field contains a valid Base Offset for the Write Tagged
   Buffer, the Read STag field is set to all zeros since it is not used,
   and the Read Base Offset field is set to all zeros.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | 0001b |1|0|                  All zeros                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Write STag                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                      Write Base Offset                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         All Zeros                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                         All Zeros                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       SCSI Write Command PDU                  |
   //                                                             //
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

       Figure 9 9: iSER Header Format for SCSI Write Command PDU

15.5

B.5.  iSER Header Format for SCSI Response PDU

   The following figure depicts a SCSI Response PDU embedded in an iSER
   Message:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | 0001b |0|0|                  All Zeros                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                           All Zeros                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       SCSI Response PDU                       |
   //                                                             //
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

           Figure 10 10: iSER Header Format for SCSI Response PDU

16

Appendix C: C.  Architectural discussion Discussion of iSER over InfiniBand

   This section explains how an InfiniBand network (with Gateways) would
   be structured.  It is informational only and is intended to provide
   insight on how iSER is used in an InfiniBand environment.

16.1

C.1.  Host side Side of iSCSI & and iSER connections Connections in Infiniband InfiniBand

   Figure 11 defines the topologies in which iSCSI and iSER will be able
   to operate on an InfiniBand Network.

   +---------+ +---------+ +---------+ +---------+ +--- -----+
   |  Host   | |  Host   | |   Host  | |   Host  | |   Host  |
   |         | |         | |         | |         | |         |
   +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+
   |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA|
   +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+
     |----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB
   IB|        IB |        IB |        IB |        IB |    SubNet2 SWTCH
   +-v-----------v-----------v-----------v-----------v---------+
   |                  InfiniBand Switch for Subnet1            |
   +---+-----+--------+-----+--------+-----+------------v------+
       | TCA |        | TCA |        | TCA |            |
       +-----+        +-----+        +-----+            | IB
      /  IB   \      /  IB   \      /       \     +--+--v--+--+
     |  iSER   |    |  iSER   |    |  IPoIB  |    |  | TCA |  |
     | Gateway |    | Gateway |    | Gateway |    |  +-----+  |
     |   to    |    |   to    |    |   to    |    | Storage   |
     |  iSCSI  |    |  iSER   |    |   IP    |    | Controller|
     |   TCP   |    |  iWARP  |    |Ethernet |    +-----+-----+
     +---v-----|    +---v-----|    +----v----+
         | EN           | EN            | EN
         +--------------+---------------+----> to IP based storage
           Ethernet links that carry iSCSI or iWARP

                Figure 11 11: iSCSI and iSER on IB

   In Figure 11, the Host systems are connected via the InfiniBand Host
   Channel Adapters (HCAs) to the InfiniBand links.  With the use of IB
   switch(es), the InfiniBand links connect the HCA to InfiniBand Target
   Channel Adapters (TCAs) located in gateways or Storage Controllers.
   An iSER-capable IB-IP Gateway converts the iSER Messages encapsulated
   in IB protocols to either standard iSCSI, or iSER Messages for iWARP.
   An [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol to IP
   protocol, and in the iSCSI case, permits iSCSI to be operated on an
   IB Network between the Hosts and the [IPoIB] Gateway.

16.2

C.2.  Storage side Side of iSCSI & and iSER mixed network environment Mixed Network Environment

   Figure 12 shows a storage controller that has three different portal
   groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP
   or iSCSI (TPG-2), and one supporting iSER/IB (TPG-1).  Here, "TPG"
   stands for "Target Portal Group".

                  |                |                |
                  |                |                |
            +--+--v--+----------+--v--+----------+--v--+--+
            |  | IB  |          |iWARP|          | EN  |  |
            |  |     |          | TCP |          | NIC |  |
            |  |(TCA)|          | RNIC|          |     |  |
            |  +-----|          +-----+          +-----+  |
            |   TPG-1            TPG-2            TPG-4   |
            |  9.1.3.3          9.1.2.4          9.1.2.6  |
            |                                             |
            |                  Storage Controller         |
            |                                             |
            +---------------------------------------------+

   Figure 12 12: Storage Controller with TCP, iWARP, and IB Connections

   The normal iSCSI portal group advertising processes (via SLP, iSNS, the Service
   Location Protocol (SLP), Internet Storage Name Service (iSNS), or
   SendTargets) are available to a Storage Controller.

16.3

C.3.  Discovery processes Processes for an InfiniBand Host

   An InfiniBand Host system can gather portal group IP address addresses from
   SLP, iSNS, or the SendTargets discovery processes by using TCP/IP via
   [IPoIB].  After obtaining one or more remote portal IP addresses, the
   Initiator uses the standard IP mechanisms to resolve the IP address
   to a local outgoing interface and the destination hardware address
   (Ethernet MAC or IB GID InfiniBand Global Identifier (GID) of the target or
   a gateway leading to the target).  If the resolved interface is an
   [IPoIB] network interface, then the target portal can be reached
   through an InfiniBand fabric.  In this case case, the Initiator can
   establish an iSCSI/TCP or iSCSI/iSER session with the Target over
   that InfiniBand interface, using the Hardware Address hardware address (InfiniBand
   GID) obtained through the standard Address Resolution Protocol (ARP)
   processes.

   If more than one IP address are is obtained through the discovery
   process, the Initiator should select a Target IP address that is on
   the same IP subnet as the Initiator Initiator, if one exists.  This will avoid
   a potential overhead of going through a gateway when a direct path
   exists.

   In addition addition, a user can configure manual static IP route entries if a
   particular path to the target is preferred.

16.4

C.4.  IBTA Connection specifications Specifications

   It is outside the scope of this document, but it is expected that the
   InfiniBand Trade Association (IBTA) has or will define:

   *  The iSER ServiceID

   *  A Means means for permitting a Host to establish a connection with a
      peer InfiniBand end-node, and that peer indicating when that
      end-node end-
      node supports iSER, so the Host would be able to fall back to
      iSCSI/TCP over [IPoIB].

   *  A Means means for permitting the Host to establish connections with IB
      iSER connections on storage controllers or IB iSER connected iSER-connected
      Gateways in preference to [IPoIB] connected IPoIB-connected Gateways/Bridges or
      connections to Target Storage Controllers that also accept iSCSI
      via [IPoIB].

   *  A Means means for combining the IB ServiceID for iSER and the IP port
      number such that the IB Host can use normal IB connection
      processes, yet ensure that the iSER target peer can actually
      connect to the required IP port number.

17

Appendix D.  Acknowledgments

   The authors acknowledge the following individuals for identifying
   implementation issues and/or suggesting resolutions to the issues
   clarified in this document: Robert Russell, Arne Redlich, David
   Black, Mallikarjun Chadalapaka, Tom Talpey, Felix Marti, Robert
   Sharp, Caitlin Bestler, Hemal Shah, Spencer Dawkins, Pete Resnick,
   Ted Lemon, Pete McCann, and Steve Kent.  Credit also goes to the
   authors of the original iSER Specification [RFC5046], including
   Michael Ko, Mallikarjun Chadalapaka, John Hufferd, Uri Elzur, Hemal
   Shah, and Patricia Thaler.  This document benefited from all of their
   contributions.

Author's Address

Authors' Addresses

   Michael Ko
   Email:

   EMail: mkosjc@gmail.com

   Alexander Nezhinsky
   Mellanox Technologies
   13 Zarchin St.
   Raanana 43662, 43662
   Israel

   Phone: +972-74-712-9000
   Email:
   EMail: alexandern@mellanox.com, nezhinsky@gmail.com

Copyright Notice

   Copyright (c) 2012 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document.  Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.