<?xml version='1.0' encoding='utf-8'?>
<rfc xmlns:xi="http://www.w3.org/2001/XInclude" version="3" ipr="trust200902" updates="" obsoletes="" category="std" docName="draft-ietf-ipsecme-multi-sa-performance-09" number="9611" consensus="true" submissionType="IETF" tocInclude="true" tocDepth="4" symRefs="true" sortRefs="true" prepTime="2024-07-19T10:02:33" indexInclude="true" scripts="Common,Latin">
  <link href="https://datatracker.ietf.org/doc/draft-ietf-ipsecme-multi-sa-performance-09" rel="prev"/>
  <link href="https://dx.doi.org/10.17487/rfc9611" rel="alternate"/>
  <link href="urn:issn:2070-1721" rel="alternate"/>
  <front>
    <title abbrev="IKEv2 Support for Per-Resource Child SAs">Internet Key Exchange Protocol Version 2 (IKEv2) Support for Per‑Resource Child Security Associations (SAs)</title>
    <seriesInfo name="RFC" value="9611" stream="IETF"/>
    <author fullname="Antony Antony" initials="A." surname="Antony">
      <organization abbrev="secunet" showOnFrontPage="true">secunet Security Networks AG</organization>
      <address>
        <email>antony.antony@secunet.com</email>
      </address>
    </author>
    <author initials="T." surname="Brunner" fullname="Tobias Brunner">
      <organization abbrev="codelabs" showOnFrontPage="true">codelabs GmbH</organization>
      <address>
        <email>tobias@codelabs.ch</email>
      </address>
    </author>
    <author fullname="Steffen Klassert" initials="S." surname="Klassert">
      <organization abbrev="secunet" showOnFrontPage="true">secunet Security Networks AG</organization>
      <address>
        <email>steffen.klassert@secunet.com</email>
      </address>
    </author>
    <author initials="P." surname="Wouters" fullname="Paul Wouters">
      <organization showOnFrontPage="true">Aiven</organization>
      <address>
        <email>paul.wouters@aiven.io</email>
      </address>
    </author>
    <date month="07" year="2024"/>
    <area>SEC</area>
    <workgroup>ipsecme</workgroup>
    <keyword>IKEv2</keyword>
    <keyword>IPsec</keyword>
    <abstract pn="section-abstract">
      <t indent="0" pn="section-abstract-1">
   In order to increase the bandwidth of IPsec traffic between peers,
   this document defines one Notify Message Status Types and one Notify
   Message Error Types payload for the Internet Key Exchange Protocol
   Version 2 (IKEv2) to support the negotiation of multiple Child
   Security Associations (SAs) with the same Traffic Selectors used on
   different resources, such as CPUs.
      </t>
      <t indent="0" pn="section-abstract-2">
       The SA_RESOURCE_INFO notification is used to convey information that the
       negotiated Child SA and subsequent new Child SAs with the same Traffic Selectors
       are a logical group of Child SAs where most or all of the Child SAs are
       bound to a specific resource, such as a specific CPU. The TS_MAX_QUEUE
       notify conveys that the peer is unwilling to create more additional Child
       SAs for this particular negotiated Traffic Selector combination.
      </t>
      <t indent="0" pn="section-abstract-3">
       Using multiple Child SAs with the same Traffic Selectors has the benefit
       that each resource holding the Child SA has its own Sequence Number Counter,
       ensuring that CPUs don't have to synchronize their cryptographic state or disable
       their packet replay protection.
      </t>
    </abstract>
    <boilerplate>
      <section anchor="status-of-memo" numbered="false" removeInRFC="false" toc="exclude" pn="section-boilerplate.1">
        <name slugifiedName="name-status-of-this-memo">Status of This Memo</name>
        <t indent="0" pn="section-boilerplate.1-1">
            This is an Internet Standards Track document.
        </t>
        <t indent="0" pn="section-boilerplate.1-2">
            This document is a product of the Internet Engineering Task Force
            (IETF).  It represents the consensus of the IETF community.  It has
            received public review and has been approved for publication by
            the Internet Engineering Steering Group (IESG).  Further
            information on Internet Standards is available in Section 2 of 
            RFC 7841.
        </t>
        <t indent="0" pn="section-boilerplate.1-3">
            Information about the current status of this document, any
            errata, and how to provide feedback on it may be obtained at
            <eref target="https://www.rfc-editor.org/info/rfc9611" brackets="none"/>.
        </t>
      </section>
      <section anchor="copyright" numbered="false" removeInRFC="false" toc="exclude" pn="section-boilerplate.2">
        <name slugifiedName="name-copyright-notice">Copyright Notice</name>
        <t indent="0" pn="section-boilerplate.2-1">
            Copyright (c) 2024 IETF Trust and the persons identified as the
            document authors. All rights reserved.
        </t>
        <t indent="0" pn="section-boilerplate.2-2">
            This document is subject to BCP 78 and the IETF Trust's Legal
            Provisions Relating to IETF Documents
            (<eref target="https://trustee.ietf.org/license-info" brackets="none"/>) in effect on the date of
            publication of this document. Please review these documents
            carefully, as they describe your rights and restrictions with
            respect to this document. Code Components extracted from this
            document must include Revised BSD License text as described in
            Section 4.e of the Trust Legal Provisions and are provided without
            warranty as described in the Revised BSD License.
        </t>
      </section>
    </boilerplate>
    <toc>
      <section anchor="toc" numbered="false" removeInRFC="false" toc="exclude" pn="section-toc.1">
        <name slugifiedName="name-table-of-contents">Table of Contents</name>
        <ul bare="true" empty="true" indent="2" spacing="compact" pn="section-toc.1-1">
          <li pn="section-toc.1-1.1">
            <t indent="0" keepWithNext="true" pn="section-toc.1-1.1.1"><xref derivedContent="1" format="counter" sectionFormat="of" target="section-1"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-introduction">Introduction</xref></t>
            <ul bare="true" empty="true" indent="2" spacing="compact" pn="section-toc.1-1.1.2">
              <li pn="section-toc.1-1.1.2.1">
                <t indent="0" keepWithNext="true" pn="section-toc.1-1.1.2.1.1"><xref derivedContent="1.1" format="counter" sectionFormat="of" target="section-1.1"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-requirements-language">Requirements Language</xref></t>
              </li>
              <li pn="section-toc.1-1.1.2.2">
                <t indent="0" keepWithNext="true" pn="section-toc.1-1.1.2.2.1"><xref derivedContent="1.2" format="counter" sectionFormat="of" target="section-1.2"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-terminology">Terminology</xref></t>
              </li>
            </ul>
          </li>
          <li pn="section-toc.1-1.2">
            <t indent="0" pn="section-toc.1-1.2.1"><xref derivedContent="2" format="counter" sectionFormat="of" target="section-2"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-performance-bottlenecks">Performance Bottlenecks</xref></t>
          </li>
          <li pn="section-toc.1-1.3">
            <t indent="0" pn="section-toc.1-1.3.1"><xref derivedContent="3" format="counter" sectionFormat="of" target="section-3"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-negotiation-of-resource-spe">Negotiation of Resource-Specific Child SAs</xref></t>
          </li>
          <li pn="section-toc.1-1.4">
            <t indent="0" pn="section-toc.1-1.4.1"><xref derivedContent="4" format="counter" sectionFormat="of" target="section-4"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-implementation-consideratio">Implementation Considerations</xref></t>
          </li>
          <li pn="section-toc.1-1.5">
            <t indent="0" pn="section-toc.1-1.5.1"><xref derivedContent="5" format="counter" sectionFormat="of" target="section-5"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-payload-format">Payload Format</xref></t>
            <ul bare="true" empty="true" indent="2" spacing="compact" pn="section-toc.1-1.5.2">
              <li pn="section-toc.1-1.5.2.1">
                <t indent="0" pn="section-toc.1-1.5.2.1.1"><xref derivedContent="5.1" format="counter" sectionFormat="of" target="section-5.1"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-sa_resource_info-notify-mes">SA_RESOURCE_INFO Notify Message Status Type Payload</xref></t>
              </li>
              <li pn="section-toc.1-1.5.2.2">
                <t indent="0" pn="section-toc.1-1.5.2.2.1"><xref derivedContent="5.2" format="counter" sectionFormat="of" target="section-5.2"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-ts_max_queue-notify-message">TS_MAX_QUEUE Notify Message Error Type Payload</xref></t>
              </li>
            </ul>
          </li>
          <li pn="section-toc.1-1.6">
            <t indent="0" pn="section-toc.1-1.6.1"><xref derivedContent="6" format="counter" sectionFormat="of" target="section-6"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-operational-considerations">Operational Considerations</xref></t>
          </li>
          <li pn="section-toc.1-1.7">
            <t indent="0" pn="section-toc.1-1.7.1"><xref derivedContent="7" format="counter" sectionFormat="of" target="section-7"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-security-considerations">Security Considerations</xref></t>
          </li>
          <li pn="section-toc.1-1.8">
            <t indent="0" pn="section-toc.1-1.8.1"><xref derivedContent="8" format="counter" sectionFormat="of" target="section-8"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-iana-considerations">IANA Considerations</xref></t>
          </li>
          <li pn="section-toc.1-1.9">
            <t indent="0" pn="section-toc.1-1.9.1"><xref derivedContent="9" format="counter" sectionFormat="of" target="section-9"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-references">References</xref></t>
            <ul bare="true" empty="true" indent="2" spacing="compact" pn="section-toc.1-1.9.2">
              <li pn="section-toc.1-1.9.2.1">
                <t indent="0" pn="section-toc.1-1.9.2.1.1"><xref derivedContent="9.1" format="counter" sectionFormat="of" target="section-9.1"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-normative-references">Normative References</xref></t>
              </li>
              <li pn="section-toc.1-1.9.2.2">
                <t indent="0" pn="section-toc.1-1.9.2.2.1"><xref derivedContent="9.2" format="counter" sectionFormat="of" target="section-9.2"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-informative-references">Informative References</xref></t>
              </li>
            </ul>
          </li>
          <li pn="section-toc.1-1.10">
            <t indent="0" pn="section-toc.1-1.10.1"><xref derivedContent="" format="none" sectionFormat="of" target="section-appendix.a"/><xref derivedContent="" format="title" sectionFormat="of" target="name-acknowledgements">Acknowledgements</xref></t>
          </li>
          <li pn="section-toc.1-1.11">
            <t indent="0" pn="section-toc.1-1.11.1"><xref derivedContent="" format="none" sectionFormat="of" target="section-appendix.b"/><xref derivedContent="" format="title" sectionFormat="of" target="name-authors-addresses">Authors' Addresses</xref></t>
          </li>
        </ul>
      </section>
    </toc>
  </front>
  <middle>
    <section numbered="true" toc="include" removeInRFC="false" pn="section-1">
      <name slugifiedName="name-introduction">Introduction</name>
      <t indent="0" pn="section-1-1">
       Most IPsec implementations are currently limited to using one
       hardware queue or a single CPU resource for a Child SA. 

Running
       packet stream encryption in parallel can be done, but there is a bottleneck of
       different parts of the hardware locking or waiting to get their
       sequence number assigned for the packet being encrypted. The
       result is that a machine with many such resources is limited to
       using only one of these resources per Child SA. This severely
       limits the throughput that can be attained. For example, at the
       time of writing, an unencrypted link of 10 Gbps or more is commonly
       reduced to 2-5 Gbps when IPsec is used to encrypt the link using
       AES-GCM. By using the implementation specified in this document,
       aggregate throughput increased from 5Gbps using 1 CPU to 40-60
       Gbps using 25-30 CPUs.
      </t>
      <t indent="0" pn="section-1-2">
       While this could be (partially) mitigated by setting up multiple
       narrowed Child SAs (for example, using Populate From Packet (PFP)
       as specified in IPsec architecture <xref target="RFC4301" format="default" sectionFormat="of" derivedContent="RFC4301"/>), this
       IPsec feature would cause too many Child SAs (one per network flow)
       or too few Child SAs (one network flow used on multiple CPUs). PFP is
       also not widely implemented.
      </t>
      <t indent="0" pn="section-1-3">
       To make better use of multiple network queues and CPUs, it can
       be beneficial to negotiate and install multiple Child
       SAs with identical Traffic Selectors. IKEv2 <xref target="RFC7296" format="default" sectionFormat="of" derivedContent="RFC7296"/>
       already allows installing multiple Child SAs with identical Traffic
       Selectors, but it offers no method to indicate that the additional Child
       SA is being requested for performance increase reasons and is restricted
       to some resource (queue or CPU).
      </t>
      <t indent="0" pn="section-1-4">
       When an IKEv2 peer is receiving more additional Child SAs for a single
       set of Traffic Selectors than it is willing to create, it can return an
       error notify of TS_MAX_QUEUE.

      </t>
      <section numbered="true" toc="include" removeInRFC="false" pn="section-1.1">
        <name slugifiedName="name-requirements-language">Requirements Language</name>
        <t indent="0" pn="section-1.1-1">
    The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>",
    "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>",
    "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>",
    "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
    "<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to be
    interpreted as described in BCP 14 <xref target="RFC2119" format="default" sectionFormat="of" derivedContent="RFC2119"/> <xref target="RFC8174" format="default" sectionFormat="of" derivedContent="RFC8174"/> when, and only when, they appear in all capitals, as
    shown here.
        </t>
      </section>
      <section numbered="true" toc="include" removeInRFC="false" pn="section-1.2">
        <name slugifiedName="name-terminology">Terminology</name>
        <t indent="0" pn="section-1.2-1">This document uses the following terms defined in IKEv2 <xref target="RFC7296" format="default" sectionFormat="of" derivedContent="RFC7296"/>:
          Notification Data, Traffic Selector (TS), Traffic Selector initiator (TSi), Traffic Selector responder (TSr), Child SA,
          Configuration Payload (CP), IKE SA, CREATE_CHILD_SA, and NO_ADDITIONAL_SAS.
        </t>
        <t indent="0" pn="section-1.2-2">This document also uses the following terms defined in <xref target="RFC4301" format="default" sectionFormat="of" derivedContent="RFC4301"/>:
          Security Policy Database (SPD), SA. 
        </t>
      </section>
    </section>
    <section anchor="performance" numbered="true" toc="include" removeInRFC="false" pn="section-2">
      <name slugifiedName="name-performance-bottlenecks">Performance Bottlenecks</name>
      <t indent="0" pn="section-2-1">
      There are several pragmatic reasons why most implementations must restrict a
      Child Security Association (SA) to a single specific hardware resource. A
      primary limitation arises from the challenges associated with sharing
      cryptographic states, counters, and sequence numbers among multiple CPUs. When
      these CPUs attempt to simultaneously utilize shared states, it becomes
      impractical to do so without incurring a significant performance penalty. It is
      necessary to negotiate and establish multiple Child SAs
      with identical Traffic Selector initiator (TSi) and Traffic Selector responder
      (TSr) on a per-resource basis.
      </t>
    </section>
    <section anchor="neg_cpu" numbered="true" toc="include" removeInRFC="false" pn="section-3">
      <name slugifiedName="name-negotiation-of-resource-spe">Negotiation of Resource-Specific Child SAs</name>
      <t indent="0" pn="section-3-1">
       An initial IKEv2 exchange is used to set up an IKE SA and the
       initial Child SA. If multiple Child SAs with the same Traffic
       Selectors that are bound to a single resource are desired, the
       initiator will add the SA_RESOURCE_INFO notify payload to the
       Exchange negotiating the Child SA (e.g., IKE_AUTH or CREATE_CHILD_SA).
       If this initial Child SA will be tied to a specific resource, it
       <bcp14>MAY</bcp14> indicate this by including an identifier in the Notification
       Data. 

A responder that is willing to have multiple Child SAs
       for the same Traffic Selectors will respond by also adding the
       SA_RESOURCE_INFO notify payload in which it <bcp14>MAY</bcp14> add a non-zero
       Notification Data.
      </t>
      <t indent="0" pn="section-3-2">
       Additional resource-specific Child SAs are negotiated as regular Child
       SAs using the CREATE_CHILD_SA exchange and are similarly identified by an
       accompanying SA_RESOURCE_INFO notification.</t>
      <t indent="0" pn="section-3-3">
       Upon installation, each resource-specific Child SA is
       associated with an additional local selector, such as the CPU.
       These resource-specific Child SAs <bcp14>MUST</bcp14> be negotiated with identical
       Child SA properties that were negotiated for the initial Child
       SA. This includes cryptographic algorithms, Traffic Selectors,
       Mode (e.g., transport mode), compression usage, etc. However, each
       Child SA does have its own keying material that is individually derived
       according to the regular IKEv2 process. The SA_RESOURCE_INFO
       notify payload <bcp14>MAY</bcp14> be empty or <bcp14>MAY</bcp14> contain some identifying data.
       This identifying data <bcp14>SHOULD</bcp14> be a unique identifier within all
       the Child SAs with the same TS payloads, and the peer <bcp14>MUST</bcp14> only
       use it for debugging purposes.
      </t>
      <t indent="0" pn="section-3-4">
       Additional Child SAs can be started on demand or can be started
       all at once. Peers may also delete specific per-resource Child
       SAs if they deem the associated resource to be idle.
      </t>
      <t indent="0" pn="section-3-5">
       During the CREATE_CHILD_SA rekey for the Child SA, the
       SA_RESOURCE_INFO notification <bcp14>MAY</bcp14> be included, but regardless of
       whether or not it is included, the rekeyed Child SA should be bound
       to the same resource(s) as the Child SA that is being rekeyed.
      </t>
    </section>
    <section anchor="impl_consider" numbered="true" toc="include" removeInRFC="false" pn="section-4">
      <name slugifiedName="name-implementation-consideratio">Implementation Considerations</name>
      <t indent="0" pn="section-4-1">
       There are various considerations that an implementation can
       use to determine the best procedure to install multiple Child SAs.
      </t>
      <t indent="0" pn="section-4-2">
       A simple procedure could be to install one additional Child SA
       on each CPU. An implementation can ensure that one Child SA can be
       used by all CPUs, so that while negotiating a new per-CPU Child SA,
       which typically takes 1 RTT delay, the CPU with no CPU-specific
       Child SA can still encrypt its packets using the Child SA that is
       available for all CPUs. Alternatively, if an implementation finds
       it needs to encrypt a packet but the current CPU does not have
       the resources to encrypt this packet, it can relay that packet
       to a specific CPU that does have the capability to encrypt the
       packet, although this will come with a performance penalty.
      </t>
      <t indent="0" pn="section-4-3">
       Performing per-CPU Child SA negotiations can result in both peers
       initiating additional Child SAs simultaneously. This is especially likely
       if per-CPU Child SAs are triggered by individual SADB_ACQUIRE messages
       <xref target="RFC2367" format="default" sectionFormat="of" derivedContent="RFC2367"/>. Responders should install the
       additional Child SA on a CPU with the least amount of additional
       Child SAs for this TSi/TSr pair.
      </t>
      <t indent="0" pn="section-4-4">
       When the number of queue or CPU resources are different between the
       peers, the peer with the least amount of resources may decide to
       not install a second outbound Child SA for the same resource, as
       it will never use it to send traffic. However, it must install
       all inbound Child SAs because it has committed to receiving traffic
       on these negotiated Child SAs.
      </t>
      <t indent="0" pn="section-4-5">
       If per-CPU packet trigger (e.g., SADB_ACQUIRE) messages are implemented
       (see <xref target="Operations" format="default" sectionFormat="of" derivedContent="Section 6"/>),
       the Traffic Selector (TSi) entry containing the information of the
       trigger packet should be included in the TS set similarly to
       regular Child SAs as specified in IKEv2 <xref target="RFC7296" format="default" section="2.9" sectionFormat="comma" derivedLink="https://rfc-editor.org/rfc/rfc7296#section-2.9" derivedContent="RFC7296"/>.
       Based on the trigger TSi entry, an implementation can select the most
       optimal target CPU to install the additional Child SA on. For example,
       if the trigger packet was for a TCP destination to port 25 (SMTP), it
       might be able to install the Child SA on the CPU that is also running
       the mail server process. Trigger packet Traffic Selectors are
       documented in IKEv2 <xref target="RFC7296" format="default" section="2.9" sectionFormat="comma" derivedLink="https://rfc-editor.org/rfc/rfc7296#section-2.9" derivedContent="RFC7296"/>.
      </t>
      <t indent="0" pn="section-4-6">
       As per IKEv2, rekeying a Child SA <bcp14>SHOULD</bcp14> use the same (or
       wider) Traffic Selectors to ensure that the new Child SA covers
       everything that the rekeyed Child SA covers. 
       This includes
       Traffic Selectors negotiated via Configuration Payloads 
       such as INTERNAL_IP4_ADDRESS, which may use the original wide TS
       set or use the narrowed TS set.
      </t>
    </section>
    <section anchor="payload_formats" numbered="true" toc="include" removeInRFC="false" pn="section-5">
      <name slugifiedName="name-payload-format">Payload Format</name>
      <t indent="0" pn="section-5-1">
	The Notify Payload format is defined in IKEv2 <xref target="RFC7296" format="default" section="3.10" sectionFormat="comma" derivedLink="https://rfc-editor.org/rfc/rfc7296#section-3.10" derivedContent="RFC7296"/>, and is copied here for convenience.
      </t>
      <t indent="0" pn="section-5-2">
      All multi-octet fields representing integers are laid out in big
      endian order (also known as "most significant byte first", or
      "network byte order").
      </t>
      <section anchor="payload_info_cpu" numbered="true" toc="include" removeInRFC="false" pn="section-5.1">
        <name slugifiedName="name-sa_resource_info-notify-mes">SA_RESOURCE_INFO Notify Message Status Type Payload</name>
        <artwork align="left" name="" type="" alt="" pn="section-5.1-1">
                    1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-------------------------------+-------------------------------+
| Next Payload  |C|  RESERVED   |         Payload Length        |
+---------------+---------------+-------------------------------+
|  Protocol ID  |   SPI Size    |      Notify Message Type      |
+---------------+---------------+-------------------------------+
|                                                               |
~               Resource Identifier (optional)                  ~
|                                                               |
+-------------------------------+-------------------------------+
            </artwork>
        <dl indent="3" newline="false" spacing="normal" pn="section-5.1-2">
          <dt pn="section-5.1-2.1">(C)ritical flag -</dt>
          <dd pn="section-5.1-2.2">
            <bcp14>MUST</bcp14> be 0.</dd>
          <dt pn="section-5.1-2.3">Protocol ID (1 octet) -</dt>
          <dd pn="section-5.1-2.4">
            <bcp14>MUST</bcp14> be 0. <bcp14>MUST</bcp14> be ignored if not 0.</dd>
          <dt pn="section-5.1-2.5">SPI Size (1 octet) -</dt>
          <dd pn="section-5.1-2.6">
            <bcp14>MUST</bcp14> be 0. <bcp14>MUST</bcp14> be ignored if not 0.</dd>
          <dt pn="section-5.1-2.7">Notify Status Message Type value (2 octets) -</dt>
          <dd pn="section-5.1-2.8">set to 16444.</dd>
          <dt pn="section-5.1-2.9">Resource Identifier (optional) -</dt>
          <dd pn="section-5.1-2.10">This opaque data may be set to convey the local identity of the resource.</dd>
        </dl>
      </section>
      <section anchor="payload_max_q" numbered="true" toc="include" removeInRFC="false" pn="section-5.2">
        <name slugifiedName="name-ts_max_queue-notify-message">TS_MAX_QUEUE Notify Message Error Type Payload</name>
        <artwork align="left" name="" type="" alt="" pn="section-5.2-1">
                    1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------------------------------+
| Next Payload  |C|  RESERVED   |         Payload Length        |
+---------------+---------------+-------------------------------+
|  Protocol ID  |   SPI Size    |      Notify Message Type      |
+---------------+---------------+-------------------------------+
            </artwork>
        <dl indent="3" newline="false" spacing="normal" pn="section-5.2-2">
          <dt pn="section-5.2-2.1">(C)ritical flag -</dt>
          <dd pn="section-5.2-2.2">
            <bcp14>MUST</bcp14> be 0.</dd>
          <dt pn="section-5.2-2.3">Protocol ID (1 octet) -</dt>
          <dd pn="section-5.2-2.4">
            <bcp14>MUST</bcp14> be 0. <bcp14>MUST</bcp14> be ignored if not 0.</dd>
          <dt pn="section-5.2-2.5">SPI Size (1 octet) -</dt>
          <dd pn="section-5.2-2.6">
            <bcp14>MUST</bcp14> be 0. <bcp14>MUST</bcp14> be ignored if not 0.</dd>
          <dt pn="section-5.2-2.7">Notify Message Error Type (2 octets) -</dt>
          <dd pn="section-5.2-2.8">set to 48.</dd>
        </dl>
        <t indent="0" pn="section-5.2-3">There is no data associated with this Notify type.</t>
      </section>
    </section>
    <section anchor="Operations" numbered="true" toc="include" removeInRFC="false" pn="section-6">
      <name slugifiedName="name-operational-considerations">Operational Considerations</name>
      <t indent="0" pn="section-6-1">
      Implementations supporting per-CPU SAs <bcp14>SHOULD</bcp14> extend their local
      SPD selector, and the mechanism of on-demand negotiation that is
      triggered by traffic to include a CPU (or queue) identifier in
      their packet trigger (e.g., SADB_ACQUIRE) message from the SPD to
      the IKE daemon. An implementation that does not support
      receiving per-CPU packet trigger messages <bcp14>MAY</bcp14> initiate all its Child
      SAs immediately upon receiving the (only) packet trigger message it
      will receive from the IPsec stack. Such an implementation also needs
      to be careful when receiving a Delete Notify request for a per-CPU
      Child SA, as it has no method to detect when it should bring up such
      a per-CPU Child SA again later. Also, bringing the deleted per-CPU
      Child SA up again immediately after receiving the Delete Notify
      might cause an infinite loop between the peers. Another issue with
      not bringing up all its per-CPU Child SAs is that if the peer acts
      similarly, the two peers might end up with only the first Child
      SA without ever activating any per-CPU Child SAs. It is therefore
      <bcp14>RECOMMENDED</bcp14> to implement per-CPU packet trigger messages.
      </t>
      <t indent="0" pn="section-6-2">
      Peers <bcp14>SHOULD</bcp14> be flexible with the maximum number of Child SAs they
      allow for a given TSi/TSr combination in order to account for corner cases. For
      example, during Child SA rekeying, there might be a large number
      of additional Child SAs created before the old Child SAs are torn
      down. Similarly, when using on-demand Child SAs, both ends could
      trigger multiple Child SA requests as the initial packet causing
      the Child SA negotiation might have been transported to the peer
      via the first Child SA, where its reply packet might also trigger an
      on-demand Child SA negotiation to start. As additional Child SAs
      consume little additional resources, allowing at the very least double
      the number of available CPUs is <bcp14>RECOMMENDED</bcp14>. An implementation <bcp14>MAY</bcp14> allow
      unlimited additional Child SAs and only limit this number based on its
      generic resource protection strategies that are used to require COOKIES
      or refuse new IKE or Child SA negotiations. Although having a very large
      number (e.g., hundreds or thousands) of SAs may slow down per-packet SAD lookup.
      </t>
      <t indent="0" pn="section-6-3">
     Implementations might support dynamically moving a per-CPU Child
     SA from one CPU to another CPU. If this method is supported,
     implementations must be careful to move both the inbound and outbound
     SAs. If the IPsec endpoint is a gateway, it can move the inbound SA
     and outbound SA independently of each other. It is likely that
     for a gateway, IPsec traffic would be asymmetric.  If the IPsec
     endpoint is the same host responsible for generating the traffic,
     the inbound and outbound SAs <bcp14>SHOULD</bcp14> remain as a pair on the same CPU.
     If a host previously skipped installing an outbound SA because it
     would be an unused duplicate outbound SA, it will have to create
     and add the previously skipped outbound SA to the SAD with the new
     CPU ID. The inbound SA may not have a CPU ID in the SAD.  Adding the
     outbound SA to the SAD requires access to the key material, whereas
     updating the CPU selector on an existing outbound SAs might not require access
     to key material.  To support this, the IKE
     software might have to hold on to the key material longer than it
     normally would, as it might actively attempt to destroy key material
     from memory that the IKE daemon no longer needs access to.
      </t>
      <t indent="0" pn="section-6-4">
   An implementation that does not accept any further resource-specific
   Child SAs <bcp14>MUST NOT</bcp14> return the NO_ADDITIONAL_SAS error because it
   could be misinterpreted by the peer to mean that no other Child SA with
   a different TSi and/or TSr is allowed either. Instead, it <bcp14>MUST</bcp14> return TS_MAX_QUEUE.
      </t>
    </section>
    <section anchor="Security" numbered="true" toc="include" removeInRFC="false" pn="section-7">
      <name slugifiedName="name-security-considerations">Security Considerations</name>
      <t indent="0" pn="section-7-1">
      Similar to how an implementation should limit the number of
      half-open SAs to limit the impact of a denial-of-service attack,
      it is <bcp14>RECOMMENDED</bcp14> that an implementation limits the maximum number of additional
      Child SAs allowed per unique TSi/TSr.
      </t>
      <t indent="0" pn="section-7-2">
      Using multiple resource-specific child SAs makes sense for
      high-volume IPsec connections on IPsec gateway machines where the
      administrator has a trust relationship with the peer's
      administrator and abuse is unlikely and easily escalated to resolve.
      </t>
      <t indent="0" pn="section-7-3">
      This trust relationship is usually not present for the deployments of remote
      access VPNs, and allowing per-CPU Child SAs
      is <bcp14>NOT RECOMMENDED</bcp14> in these scenarios. Therefore, it is also <bcp14>NOT RECOMMENDED</bcp14> to allow per-CPU Child SAs by default.
      </t>
      <t indent="0" pn="section-7-4">
      The SA_RESOURCE_INFO notify contains an optional data payload that
      can be used by the peer to identify the Child SA belonging to a
      specific resource.  Notification data <bcp14>SHOULD NOT</bcp14> be an identifier that
      can be used to gain information about the hardware. For example,
      using the CPU number itself as the identifier might give an attacker
      knowledge of which packets are handled by which CPU ID, and it might
      optimize a brute-force attack against the system.
      </t>
    </section>
    <section anchor="IANA" numbered="true" toc="include" removeInRFC="false" pn="section-8">
      <name slugifiedName="name-iana-considerations">IANA Considerations</name>
      <t indent="0" pn="section-8-1">
        IANA has registered one new value in the "IKEv2 Notify Message Status Types" registry.
      </t>
      <table anchor="iana_requests_i" align="center" pn="table-1">
        <thead>
          <tr>
            <th align="left" colspan="1" rowspan="1">Value</th>
            <th align="left" colspan="1" rowspan="1">Notify Message Status Type</th>
            <th align="left" colspan="1" rowspan="1">Reference</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td align="left" colspan="1" rowspan="1">16444</td>
            <td align="left" colspan="1" rowspan="1">SA_RESOURCE_INFO</td>
            <td align="left" colspan="1" rowspan="1">RFC 9611</td>
          </tr>
        </tbody>
      </table>
      <t indent="0" pn="section-8-3">
        IANA has registered one new value in the "IKEv2 Notify Message Error Types" registry.
      </t>
      <table anchor="iana_requests_e" align="center" pn="table-2">
        <thead>
          <tr>
            <th align="left" colspan="1" rowspan="1">Value</th>
            <th align="left" colspan="1" rowspan="1">Notify Message Error Type</th>
            <th align="left" colspan="1" rowspan="1">Reference</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td align="left" colspan="1" rowspan="1">48</td>
            <td align="left" colspan="1" rowspan="1">TS_MAX_QUEUE</td>
            <td align="left" colspan="1" rowspan="1">RFC 9611</td>
          </tr>
        </tbody>
      </table>
    </section>
  </middle>
  <back>
    <references pn="section-9">
      <name slugifiedName="name-references">References</name>
      <references pn="section-9.1">
        <name slugifiedName="name-normative-references">Normative References</name>
        <reference anchor="RFC2119" target="https://www.rfc-editor.org/info/rfc2119" quoteTitle="true" derivedAnchor="RFC2119">
          <front>
            <title>Key words for use in RFCs to Indicate Requirement Levels</title>
            <author fullname="S. Bradner" initials="S." surname="Bradner"/>
            <date month="March" year="1997"/>
            <abstract>
              <t indent="0">In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.</t>
            </abstract>
          </front>
          <seriesInfo name="BCP" value="14"/>
          <seriesInfo name="RFC" value="2119"/>
          <seriesInfo name="DOI" value="10.17487/RFC2119"/>
        </reference>
        <reference anchor="RFC7296" target="https://www.rfc-editor.org/info/rfc7296" quoteTitle="true" derivedAnchor="RFC7296">
          <front>
            <title>Internet Key Exchange Protocol Version 2 (IKEv2)</title>
            <author fullname="C. Kaufman" initials="C." surname="Kaufman"/>
            <author fullname="P. Hoffman" initials="P." surname="Hoffman"/>
            <author fullname="Y. Nir" initials="Y." surname="Nir"/>
            <author fullname="P. Eronen" initials="P." surname="Eronen"/>
            <author fullname="T. Kivinen" initials="T." surname="Kivinen"/>
            <date month="October" year="2014"/>
            <abstract>
              <t indent="0">This document describes version 2 of the Internet Key Exchange (IKE) protocol. IKE is a component of IPsec used for performing mutual authentication and establishing and maintaining Security Associations (SAs). This document obsoletes RFC 5996, and includes all of the errata for it. It advances IKEv2 to be an Internet Standard.</t>
            </abstract>
          </front>
          <seriesInfo name="STD" value="79"/>
          <seriesInfo name="RFC" value="7296"/>
          <seriesInfo name="DOI" value="10.17487/RFC7296"/>
        </reference>
        <reference anchor="RFC8174" target="https://www.rfc-editor.org/info/rfc8174" quoteTitle="true" derivedAnchor="RFC8174">
          <front>
            <title>Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words</title>
            <author fullname="B. Leiba" initials="B." surname="Leiba"/>
            <date month="May" year="2017"/>
            <abstract>
              <t indent="0">RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.</t>
            </abstract>
          </front>
          <seriesInfo name="BCP" value="14"/>
          <seriesInfo name="RFC" value="8174"/>
          <seriesInfo name="DOI" value="10.17487/RFC8174"/>
        </reference>
      </references>
      <references pn="section-9.2">
        <name slugifiedName="name-informative-references">Informative References</name>
        <reference anchor="RFC2367" target="https://www.rfc-editor.org/info/rfc2367" quoteTitle="true" derivedAnchor="RFC2367">
          <front>
            <title>PF_KEY Key Management API, Version 2</title>
            <author fullname="D. McDonald" initials="D." surname="McDonald"/>
            <author fullname="C. Metz" initials="C." surname="Metz"/>
            <author fullname="B. Phan" initials="B." surname="Phan"/>
            <date month="July" year="1998"/>
            <abstract>
              <t indent="0">A generic key management API that can be used not only for IP Security but also for other network security services is presented in this document. This memo provides information for the Internet community. It does not specify an Internet standard of any kind.</t>
            </abstract>
          </front>
          <seriesInfo name="RFC" value="2367"/>
          <seriesInfo name="DOI" value="10.17487/RFC2367"/>
        </reference>
        <reference anchor="RFC4301" target="https://www.rfc-editor.org/info/rfc4301" quoteTitle="true" derivedAnchor="RFC4301">
          <front>
            <title>Security Architecture for the Internet Protocol</title>
            <author fullname="S. Kent" initials="S." surname="Kent"/>
            <author fullname="K. Seo" initials="K." surname="Seo"/>
            <date month="December" year="2005"/>
            <abstract>
              <t indent="0">This document describes an updated version of the "Security Architecture for IP", which is designed to provide security services for traffic at the IP layer. This document obsoletes RFC 2401 (November 1998). [STANDARDS-TRACK]</t>
            </abstract>
          </front>
          <seriesInfo name="RFC" value="4301"/>
          <seriesInfo name="DOI" value="10.17487/RFC4301"/>
        </reference>
      </references>
    </references>
    <section anchor="acks" numbered="false" toc="include" removeInRFC="false" pn="section-appendix.a">
      <name slugifiedName="name-acknowledgements">Acknowledgements</name>
      <t indent="0" pn="section-appendix.a-1">The following people provided reviews and valuable feedback:
      <contact fullname="Roman Danyliw"/>, <contact fullname="Warren Kumari"/>,
      <contact fullname="Tero Kivinen"/>, <contact fullname="Murray Kucherawy"/>,
      <contact fullname="John Scudder"/>, <contact fullname="Valery Smyslov"/>,
      <contact fullname="Gunter van de Velde"/>, and <contact fullname="Éric Vyncke"/>.
      </t>
    </section>
    <section anchor="authors-addresses" numbered="false" removeInRFC="false" toc="include" pn="section-appendix.b">
      <name slugifiedName="name-authors-addresses">Authors' Addresses</name>
      <author fullname="Antony Antony" initials="A." surname="Antony">
        <organization abbrev="secunet" showOnFrontPage="true">secunet Security Networks AG</organization>
        <address>
          <email>antony.antony@secunet.com</email>
        </address>
      </author>
      <author initials="T." surname="Brunner" fullname="Tobias Brunner">
        <organization abbrev="codelabs" showOnFrontPage="true">codelabs GmbH</organization>
        <address>
          <email>tobias@codelabs.ch</email>
        </address>
      </author>
      <author fullname="Steffen Klassert" initials="S." surname="Klassert">
        <organization abbrev="secunet" showOnFrontPage="true">secunet Security Networks AG</organization>
        <address>
          <email>steffen.klassert@secunet.com</email>
        </address>
      </author>
      <author initials="P." surname="Wouters" fullname="Paul Wouters">
        <organization showOnFrontPage="true">Aiven</organization>
        <address>
          <email>paul.wouters@aiven.io</email>
        </address>
      </author>
    </section>
  </back>
</rfc>
