Guidelines for Adding
Congestion Notification to Protocols that Encapsulate IPBTB54/77, Adastral ParkMartlesham HeathIpswichIP5 3REUK+44 1473 645196bob.briscoe@bt.comhttp://bobbriscoe.net/Huawei5340 Legacy Drive, Suite 175PlanoTexas75024USAjohn.kaippallimalil@huawei.comBroadcom Corporation5025 Keane DriveCarmichaelCA95608USApthaler@broadcom.com
Transport
Transport Area Working GroupCongestion Control and ManagementCongestion NotificationInformation SecurityTunnellingEncapsulation & DecapsulationProtocolECNLayeringThe purpose of this document is to guide the design of congestion
notification in any lower layer or tunnelling protocol that encapsulates
IP. The aim is for explicit congestion signals to propagate consistently
from lower layer protocols into IP. Then the IP internetwork layer can
act as a portability layer to carry congestion notification from
non-IP-aware congested nodes up to the transport layer (L4). Following
these guidelines should assure interworking between new lower layer
congestion notification mechanisms, whether specified by the IETF or
other standards bodies.The benefits of Explicit Congestion Notification (ECN) described
below can only be fully realised if support for ECN is added to the
relevant subnetwork technology, as well as to IP. When a lower layer
buffer drops a packet obviously it does not just drop at that layer; the
packet disappears from all layers. In contrast, when a lower layer marks
a packet with ECN, the marking needs to be explicitly propagated up the
layers. The same is true if a buffer marks the outer header of a packet
that encapsulates inner tunnelled headers. Forwarding ECN is not as
straightforward as other headers because it has to be assumed ECN may be
only partially deployed. If an egress at any layer is not ECN-aware, or
if the ultimate receiver or sender is not ECN-aware, congestion needs to
be indicated by dropping a packet, not marking it.The purpose of this document is to guide the addition of congestion
notification to any subnet technology or tunnelling protocol, so that
lower layer equipment can signal congestion explicitly and it will
propagate consistently into encapsulated (higher layer) headers,
otherwise the signals will not reach their ultimate destination.ECN is defined in the IP header (v4 & v6) to allow a resource to notify the onset of queue
build-up without having to drop packets, by explicitly marking a
proportion of packets with the congestion experienced (CE)
codepoint.Given a suitable marking scheme, ECN removes nearly all congestion
loss and it cuts delays for two main reasons: It avoids the delay when recovering from congestion losses, which
particularly benefits small flows or real-time flows, making their
delivery time predictably short ;As ECN is used more widely by end-systems, it will gradually
remove the need to configure a degree of delay into buffers before
they start to notify congestion (the cause of bufferbloat). This is
because drop involves a trade-off between sending a timely signal
and trying to avoid impairment, whereas ECN is solely a signal not
an impairment, so there is no harm triggering it earlier.Some lower layer technologies (e.g. MPLS, Ethernet) are used to form
subnetworks with IP-aware nodes only at the edges. These networks are
often sized so that it is rare for interior queues to overflow. However,
this has often be more due to the inability of the original TCP protocol
to saturate the links. For many years, fixes such as window scaling
proved hard to deploy. But now that modern
operating systems are finally capable of saturating interior links, even
the buffers of well-provisioned interior switches will need to signal
episodes of queuing.Propagation of ECN is defined for MPLS , and
is being defined for TRILL , but
it remains to be defined for a number of other subnetwork
technologies.Similarly, ECN propagation is yet to be defined for many tunnelling
protocols. defines how ECN should be propagated
for IP-in-IP and IPsec
tunnels. However, as Section 9.3 of RFC3168 pointed out, ECN support
will need to be defined for other tunnelling protocols, e.g. L2TP , GRE , , PPTP and GTP , , .Incremental deployment is the most tricky aspect when adding support
for ECN. The original ECN protocol in IP was
carefully designed so that a congested buffer would not mark a packet
(rather than drop it) unless both source and destination hosts were
ECN-capable. Otherwise its congestion markings would never be detected
and congestion would just deteriorate further. However, to support
congestion marking below the IP layer, it is not sufficient to only
check that the two end-points support ECN; correct operation also
depends on the decapsulator at each subnet egress faithfully propagating
congestion notifications to the higher layer. Otherwise, a legacy
decapsulator might silently fail to propagate any ECN signals from the
outer to the forwarded header. Then the lost signals would never be
detected and again congestion would deteriorate further. The guidelines
given later require protocol designers to carefully consider incremental
deployment, and suggest various safe approaches for different
circumstances.Of course, the IETF does not have standards authority over every link
layer protocol. So this document gives guidelines for designing
propagation of congestion notification across the interface between IP
and protocols that may encapsulate IP (i.e. that can be layered beneath
IP). Each lower layer technology will exhibit different issues and
compromises, so the IETF or the relevant standards body must be free to
define the specifics of each lower layer congestion notification scheme.
Nonetheless, if the guidelines are followed, congestion notification
should interwork between different technologies, using IP in its role as
a 'portability layer'.Therefore, the capitalised term 'SHOULD' or 'SHOULD NOT' are often
used in preference to 'MUST' or 'MUST NOT', because it is difficult to
know the compromises that will be necessary in each protocol design. If
a particular protocol design chooses to contradict a 'SHOULD (NOT)'
given in the advice below, it MUST include a sound justification.It has not been possible to give common guidelines for all lower
layer technologies, because they do not all fit a common pattern.
Instead they have been divided into a few distinct modes of operation:
feed-forward-and-upward; feed-upward-and-forward; feed-backward; and
null mode. These modes are described in ,
then in the following sections separate guidelines are given for each
mode.This document updates the advice to subnetwork designers about ECN in
Section 13 of .This document only concerns wire protocol processing of explicit
notification of congestion and makes no changes or recommendations
concerning algorithms for congestion marking or for congestion
response (algorithm issues should be independent of the layer the
algorithm operates in).The question of congestion notification signals with different
semantics to those of ECN in IP is touched on in a couple of specific
cases (e.g. QCN ) and with schemes with
multiple severity levels such as PCN ).
However, no attempt is made to give guidelines about schemes with
different semantics that are yet to be invented.The semantics of congestion signals can be relative to the traffic
class. Therefore correct propagation of congestion signals could
depend on correct propagation of any traffic class field between the
layers. In this document, correct propagation of traffic class
information is assumed, while what 'correct' means and how it is
achieved is covered elsewhere (e.g. ) and is
outside the scope of the present document.Note that these guidelines do not require the subnet wire protocol
to be changed to accommodate congestion notification. Another way to
add congestion notification without consuming header space in the
subnet protocol might be to use a parallel control plane protocol.This document focuses on the congestion notification interface
between IP and lower layer protocols that can encapsulate IP, where
the term 'IP' includes v4 or v6, unicast, multicast or anycast.
However, it is likely that the guidelines will also be useful when a
lower layer protocol or tunnel encapsulates itself (e.g. Ethernet MAC
in MAC ) or when it encapsulates other
protocols. In the feed-backward mode, propagation of congestion
signals for multicast and anycast packets is out-of-scope (because it
would be so complicated that it is hoped no-one would attempt such an
abomination).The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 .Further terminology used within this document:Information that is
delivered as a unit among peer entities of a layered network
consisting of protocol control information (typically a header) and
possibly user data (payload) of that layer. The scope of this
document includes layer 2 and layer 3 networks, where the PDU is
respectively termed a frame or a packet (or a cell in ATM). PDU is a
general term for any of these. This definition also includes a
payload with a shim header lying somewhere between layer 2 &
3.The end-to-end transmission control
function, conventionally considered at layer-4 in the OSI reference
model. Given the audience for this document will often use the word
transport to mean low level bit carriage, whenever the term is used
it will be qualified, e.g. 'L4 transport'.The link or tunnel endpoint function
that adds an outer header to a PDU (also termed the 'link ingress',
the 'subnet ingress', the 'ingress tunnel endpoint' or just the
'ingress' where the context is clear).The link or tunnel endpoint function
that removes an outer header from a PDU (also termed the 'link
egress', the 'subnet egress', the 'egress tunnel endpoint' or just
the 'egress' where the context is clear).The header of an arriving PDU before
encapsulation.The header added to encapsulate a
PDU.The header encapsulated by the outer
header.The header forwarded by the
decapsulator.Congestion Experienced ECN-Capable Transport Not ECN-Capable Transport A PDU that is part of a feedback loop within
which all the nodes that need to propagate explicit congestion
notifications back to the Load Regulator are ECN-capable. An IP
packet with a non-zero ECN field implies that the endpoints are
ECN-capable, so this would be an ECN-PDU. However, ECN-PDU is
intended to be a general term for a PDU at any layer, not just
IP.A PDU that is part of a feedback-loop
within which some nodes necessary to propagate explicit congestion
notifications back to the load regulator are not ECN-capable.For each flow of PDUs, the transport
function that is capable of controlling the data rate. Typically
located at the data source, but in-path nodes can regulate load in
some congestion control arrangements (e.g. admission control or
policing nodes). Note the term "a function capable of controlling
the load" deliberately includes a transport that doesn't actually
control the load but ideally it ought to (e.g. a sending application
without congestion control that uses UDP).The location of the function on
the path that initialised the values of all congestion notification
fields in a sequence of packets, before any are set to the
congestion experienced (CE) codepoint if they experience congestion
further downstream. Typically the original data source at
layer-4.This section sets down the different modes by which congestion
information is passed between the lower layer and the higher one. It
acts as a reference framework for the following sections, which give
normative guidelines for designers of explicit congestion notification
protocols, taking each mode in turn:Nodes feed forward congestion
notification towards the egress within the lower layer then up and
along the layers towards the end-to-end destination at the transport
layer. The following local optimisation is possible:A lower layer switch feeds-up
congestion notification directly into the ECN field in the
higher layer (e.g. IP) header, irrespective of whether the node
is at the egress of a subnet.Nodes feed back congestion signals
towards the ingress of the lower layer and (optionally) attempt to
control congestion within their own layer.Nodes cannot experience congestion at the lower
layer except at ingress nodes (which are IP-aware or equivalently
higher-layer-aware).Like IP and MPLS, many subnet technologies are based on
self-contained protocol data units (PDUs) or frames sent unreliably.
They provide no feedback channel at the subnetwork layer, instead
relying on higher layers (e.g. TCP) to feed back loss signals.In these cases, ECN may best be supported by standardising explicit
notification of congestion into the lower layer protocol that carries
the data forwards. It will then also be necessary to define how the
egress of the lower layer subnet propagates this explicit signal into
the forwarded upper layer (IP) header. It can then continue forwards
until it finally reaches the destination transport (at L4). Then
typically the destination will feed this congestion notification back
to the source transport using an end-to-end protocol (e.g. TCP). This
is the arrangement that has already been used to add ECN to IP-in-IP
tunnels , IP-in-MPLS and MPLS-in-MPLS .This mode is illustrated in . Along the middle of the
figure, layers 2, 3 & 4 of the protocol stack are shown, and one
packet is shown along the bottom as it progresses across the network
from source to destination, crossing two subnets connected by a
router, and crossing two switches on the path across each subnet.
Congestion at the output of the first switch (shown as *) leads to a
congestion marking in the L2 header (shown as C in the illustration of
the packet). The chevrons show the progress of the resulting
congestion indication. It is propagated from link to link across the
subnet in the L2 header, then when the router removes the marked L2
header, it propagates the marking up into the L3 (IP) header. The
router forwards the marked L3 header into subnet 2, and when it adds a
new L2 header it copies the L3 marking into the L2 header as well, as
shown by the 'C's in both layers (assuming the technology of subnet 2
also supports explicit congestion marking).Note that there is no implication that each 'C' marking is encoded
the same; a different encoding might be used for the 'C' marking in
each protocol.Finally, for completeness, we show the L3 marking arriving at the
destination, where the host transport protocol (e.g. TCP) feeds it
back to the source in the L4 acknowledgement (the 'C' at L4 in the
packet at the top of the diagram).Of course, modern networks are rarely as simple as this text-book
example, often involving multiple nested layers. For example, a 3GPP
mobile network may have two IP-in-IP (GTP) tunnels in series and an
MPLS backhaul between the base station and the first router.
Nonetheless, the example illustrates the general idea of feeding
congestion notification forward then upward whenever a header is
removed at the egress of a subnet.Note that the FECN (forward ECN) bit in Frame Relay and the
explicit forward congestion indication (EFCI ) bit in ATM user data cells follow a
feed-forward pattern. However, in ATM, this is only as part of a
feed-forward-and-backward pattern at the lower layer, not
feed-forward-and-up out of the lower layer—the intention was
never to interface to IP ECN at the subnet egress. To our knowledge,
Frame Relay FECN is solely used to detect where more capacity should
be provisioned .Ethernet is particularly difficult to extend incrementally to
support explicit congestion notification. One way to support ECN in
such cases has been to use so called 'layer-3 switches'. These are
Ethernet switches that bury into the Ethernet payload to find an IP
header and manipulate or act on certain IP fields (specifically
Diffserv & ECN). For instance, in Data Center TCP , layer-3 switches are configured to mark the ECN
field of the IP header within the Ethernet payload when their output
buffer becomes congested. With respect to switching, a layer-3 switch
acts solely on the addresses in the Ethernet header; it doesn't use IP
addresses, and it doesn't decrement the TTL field in the IP
header.By comparing with , it can be seen that
subnet E (perhaps a subnet of layer-3 Ethernet switches) works in
feed-up-and-forward mode by notifying congestion directly into L3 at
the point of congestion, even though the congested switch does not
otherwise act at L3. In this example, the technology in subnet F (e.g.
MPLS) does support ECN natively, so when the router adds the layer-2
header it copies the ECN marking from L3 to L2 as well.In some layer 2 technologies, explicit congestion notification has
been defined for use internally within the subnet with its own
feedback and load regulation, but typically the interface with IP for
ECN has not been defined.For instance, for the available bit-rate (ABR) service in ATM, the
relative rate mechanism was one of the more popular mechanisms for
managing traffic, tending to supersede earlier designs. In this
approach ATM switches send special resource management (RM) cells in
both the forward and backward directions to control the ingress rate
of user data into a virtual circuit. If a switch buffer is approaching
congestion or congested it sends an RM cell back towards the ingress
with respectively the No Increase (NI) or Congestion Indication (CI)
bit set in its message type field . The
ingress then holds or decreases its sending bit-rate accordingly.ATM's feed-backward approach doesn't fit well when layered beneath
IP's feed-forward approach—unless the initial data source is the
same node as the ATM ingress. shows the feed-backward approach
being used in subnet H. If the final switch on the path is congested
(*), it doesn't feed-forward any congestion indications on packet (U).
Instead it sends a control cell (V) back to the router at the ATM
ingress.However, the backward feedback doesn't reach the original data
source directly because IP doesn't support backward feedback (and
subnet G is independent of subnet H). Instead, the router in the
middle throttles down its sending rate but the original data sources
don't reduce their rates. The resulting rate mismatch causes the
middle router's buffer at layer 3 to back up until it becomes
congested, which it signals forwards on later data packets at layer 3
(e.g. packet W). Note that the forward signal from the middle router
is not triggered directly by the backward signal. Rather, it is
triggered by congestion resulting from the middle router's mismatched
rate response to the backward signal.In response to this later forward signalling, end-to-end feedback
at layer-4 finally completes the tortuous path of congestion
indications back to the origin data source, as before.Often link and physical layer resources are 'non-blocking' by
design. In these cases congestion notification may be implemented but
it does not need to be deployed at the lower layer; ECN in IP would be
sufficient.A degenerate example is a point-to-point Ethernet link. Excess
loading of the link merely causes the queue from the higher layer to
back up, while the lower layer remains immune to congestion. Even a
whole meshed subnetwork can be made immune to interior congestion by
limiting ingress capacity and careful sizing of links, particularly if
multi-path routing is used to ensure even worst-case patterns of load
cannot congest any link.Feed-forward-and-up is the mode already used for signalling ECN up
the layers through MPLS into IP and through
IP-in-IP tunnels . These RFCs take a consistent
approach and the following guidelines are designed to ensure this
consistency continues as ECN support is added to other protocols that
encapsulate IP. The guidelines are also designed to ensure compliance
with the more general best current practice for the design of alternate
ECN schemes given in .The rest of this section is structured as follows: addresses
the most straightforward cases, where can
be applied directly to add ECN to tunnels that are effectively the
same as IP-in-IP tunnels.The subsequent sections give guidelines for adding ECN to a
subnet technology that uses feed-forward-and-up mode like IP, but it
is not so similar to IP that rules can be
applied directly. Specifically:Sections , and
respectively address how to add ECN support to the wire protocol
and to the encapsulators and decapsulators at the ingress and
egress of the subnet. deals with the special,
but common, case of sequences of tunnels or subnets that all use
the same technology deals with the question
of reframing when IP packets do not map 1:1 into lower layer
frames.A common pattern for many tunnelling protocols is to encapsulate an
inner IP header with shim header(s) then an outer IP header. In many
cases the shim header(s) always have to be tightly coupled to the
outer IP header because they are not sufficient as outer headers in
their own right. In such cases the shim header(s) and the outer IP
header are always added (or removed) in the same operation. Therefore,
in all such tightly coupled IP-in-IP tunnelling protocols, the rules
in for propagating the ECN field between the
two IP headers SHOULD be applied directly.Examples of tightly coupled IP-in-IP tunnelling protocols where
can be applied directly are: L2TP GRE , PPTP GTP , , VXLAN .This section is intended to guide the redesign of any lower layer
protocol that encapsulate IP to add native ECN support at the lower
layer. It reflects the approaches used in and
in . Therefore IP-in-IP tunnels or IP-in-MPLS
or MPLS-in-MPLS encapsulations that already comply with or will already satisfy
this guidance.A lower layer (or subnet) congestion notification system:SHOULD NOT apply explicit congestion notifications to PDUs that
are destined for legacy layer-4 transport implementations that
will not understand ECN, andSHOULD NOT apply explicit
congestion notifications to PDUs if the egress of the subnet might
not propagate congestion notifications onward into the higher
layer.We use the term ECN-PDUs for a PDU
on a feedback loop that will propagate congestion notification
properly because it meets both the above criteria. And a
Not-ECN-PDU is a PDU on a feedback loop that does not meet both
criteria, and will therefore not propagate congestion notification
properly. A corollary of the above is that a lower layer
congestion notification protocol:SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs.Note that there is no need for all interior nodes within a subnet
to be able to mark congestion explicitly. A mix of ECN and drop
signals from different nodes is fine. However, if any
interior nodes might generate ECN markings, guideline above says that all
relevant egress node(s) SHOULD be able to propagate those markings up
to the higher layer.In IP, if the ECN field in each PDU is cleared to the Not-ECT (not
ECN-capable transport) codepoint, it indicates that the L4 transport
will not understand congestion markings. A congested buffer must not
mark these Not-ECT PDUs, and therefore drops them instead.The mechanism a lower layer uses to distinguish the ECN-capability
of PDUs need not mimic that of IP. All the above guidelines say is
that the lower layer system, as a whole, should achieve the same
outcome. For instance, ECN-capable feedback loops might use PDUs that
are identified by a particular set of labels or tags. Alternatively,
logical link protocols that use flow state might determine whether a
PDU can be congestion marked by checking for ECN-support in the flow
state. Other protocols might depend on out-of-band control
signals.The per-domain checking of ECN support in MPLS is a good example of a way to avoid sending
congestion markings to transports that will not understand them,
without using any header space in the subnet protocol.In MPLS, header space is extremely limited, therefore RFC5129 does
not provide a field in the MPLS header to indicate whether the PDU is
an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are
allowed to set explicit congestion indications without checking
whether the PDU is destined for a transport that will understand them.
Nonetheless, this is made safe by requiring that the network operator
upgrades all decapsulating edges of a whole domain at once, as soon as
even one switch within the domain is configured to mark rather than
drop during congestion. Therefore, any edge node that might
decapsulate a packet will be capable of checking whether the higher
layer transport is ECN-capable. When decapsulating a CE-marked packet,
if the decapsulator discovers that the higher layer (inner header)
indicates the transport is not ECN-capable, it drops the
packet—effectively on behalf of the earlier congested node (see
Decapsulation Guideline in ).It was only appropriate to define such an incremental deployment
strategy because MPLS is targeted solely at professional operators,
who can be expected to ensure that a whole subnetwork is consistently
configured. This strategy might not be appropriate for other link
technologies targeted at zero-configuration deployment or deployment
by the general public (e.g. Ethernet). For such 'plug-and-play'
environments it will be necessary to invent a failsafe approach that
ensures congestion markings will never fall into black holes, no
matter how inconsistently a system is put together. Alternatively,
congestion notification relying on correct system configuration could
be confined to flavours of Ethernet intended only for professional
network operators, such as IEEE 802.1ah Provider Backbone Bridges
(PBB).QCN provides another example of how
to indicate to lower layer devices that the end-points will not
understand ECN. An operator can define certain 802.1p classes of
service to indicate non-QCN frames and an ingress bridge is required
to map arriving not-QCN-capable IP packets to one of these non-QCN
802.1p classes.This section is intended to guide the redesign of any node that
encapsulates IP with a lower layer header when adding native ECN
support to the lower layer protocol. It reflects the approaches used
in and in . Therefore
IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that
already comply with or will already satisfy this guidance.Egress Capability Check: A subnet ingress needs to be sure that
the corresponding egress of a subnet will propagate any congestion
notification added to the outer header across the subnet. This is
necessary in addition to checking that an incoming PDU indicates
an ECN-capable (L4) transport. Examples of how this guarantee
might be provided include:by configuration (e.g. if any label switches in a domain
support ECN marking, requires all
egress nodes to have been configured to propagate ECN)by the ingress explicitly checking that the egress
propagates ECN (e.g. TRILL uses IS-IS to check path
capabilities before using critical options )by inherent design of the protocol (e.g. by encoding ECN
marking on the outer header in such a way that a legacy egress
that does not understand ECN will consider the PDU corrupt and
discard it, thus at least propagating a form of congestion
signal).Egress Fails Capability Check: If the ingress cannot guarantee
that the egress will propagate congestion notification, the
ingress SHOULD disable ECN when it forwards the PDU at the lower
layer. An example of how the ingress might disable ECN at the
lower layer would be by setting the outer header of the PDU to
identify it as a Not-ECN-PDU, assuming the subnet technology
supports such a concept.Standard Congestion Monitoring
Baseline: Once the ingress to a subnet has established that the
egress will correctly propagate ECN, on encapsulation it SHOULD
encode the same level of congestion in outer headers as is
arriving in incoming headers. For example it might copy any
incoming congestion notification into the outer header of the
lower layer protocol.This ensures that all
outer headers reflect congestion accumulated along the whole
upstream path since the Load Regulator, not just since the ingress
of the subnet. A node that is not the Load Regulator SHOULD NOT
re-initialise the level of CE markings in the outer to zero.
This guideline is intended to ensure that
any bulk congestion monitoring of outer headers (e.g. by a network
management node monitoring ECN in passing frames) is most
meaningful. For instance, if an operator measures CE in 0.4% of
passing outer headers, this information is only useful if the
operator knows where the proportion of CE markings was last
initialised to 0% (the Congestion Baseline). Such monitoring
information will not be useful if some subnet ingress nodes reset
all outer CE markings while others copy incoming CE markings into
the outer.Most information can be
extracted if the Congestion Baseline is standardised at the node
that is regulating the load (the Load Regulator—typically
the data source). Then the operator can measure both congestion
since the Load Regulator, and congestion since the subnet ingress.
The latter might be measurable by subtracting the level of CE
markings on inner headers from that on outer headers (see Appendix
C of ).This section is intended to guide the redesign of any node that
decapsulates IP from within a lower layer header when adding native
ECN support to the lower layer protocol. It reflects the approaches
used in and in .
Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS
encapsulations that already comply with or
will already satisfy this guidance.A subnet egress SHOULD NOT simply copy congestion notification from
outer headers to the forwarded header. It SHOULD calculate the
outgoing congestion notification field from the inner and outer
headers using the following guidelines. If there is any conflict,
rules earlier in the list take precedence over rules later in the
list:If the arriving inner
header is a Not-ECN-PDU it implies the L4 transport will not
understand explicit congestion markings. Then:If the outer header carries an explicit congestion marking,
the packet SHOULD be dropped—the only indication of
congestion that the L4 transport will understand.If the outer is an ECN-PDU that carries no indication of
congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but
still as a Not-ECN-PDU.If the outer header does not support explicit congestion
notification (a Not-ECN-PDU), but the inner header does (an
ECN-PDU), the inner header SHOULD be forwarded unchanged.In some lower layer protocols congestion may be signalled as a
numerical level, such as in the control frames of quantised
congestion notification . If such a
multi-bit encoding encapsulates an ECN-capable IP data packet, a
function will be needed to convert the quantised congestion level
into the frequency of congestion markings in outgoing IP
packets.Congestion indications may be encoded by a severity level. For
instance increasing levels of congestion might be encoded by
numerically increasing indications, e.g. pre-congestion
notification (PCN) can be encoded in each PDU at three severity
levels in IP or MPLS .If the arriving inner header is an ECN-PDU, where
the inner and outer headers carry indications of congestion of
different severity, the more severe indication SHOULD be forwarded
in preference to the less severe.The inner and outer headers might carry a combination of
congestion notification fields that should not be possible given
any currently used protocol transitions. For instance, if
Encapsulation Guideline in had been followed, it should
not be possible to have a less severe indication of congestion in
the outer than in the inner. It MAY be appropriate to log
unexpected combinations of headers and possibly raise an alarm.
If a safe outgoing codepoint can be
defined for such a PDU, the PDU SHOULD be forwarded rather than
dropped. Some implementers discard PDUs with currently unused
combinations of headers just in case they represent an attack.
However, an approach using alarms and policy-mediated drop is
preferable to hard-coded drop, so that operators can keep track of
possible attacks but currently unused combinations are not
precluded from future use through new standards actions.In some deployments, particularly in 3GPP networks, an IP packet
may traverse two or more IP-in-IP tunnels in sequence that all use
identical technology (e.g. GTP).In such cases, it would be sufficient for every encapsulation and
decapsulation in the chain to comply with RFC 6040. Alternatively, as
an optimisation, a node that decapsulates a packet and immediately
re-encapsulates it for the next tunnel MAY copy the incoming outer ECN
field directly to the outgoing outer and the incoming inner ECN field
directly to the outgoing inner. Then the overall behavior across the
sequence of tunnel segments would still be consistent with RFC
6040.Appendix C of RFC6040 describes how a tunnel egress can monitor how
much congestion has been introduced within a tunnel. A network
operator might want to monitor how much congestion had been introduced
within a whole sequence of tunnels. Using the technique in Appendix C
of RFC6040 at the final egress, the operator could monitor the whole
sequence of tunnels, but only if the above optimisation were used
consistently along the sequence of tunnels, in order to make it appear
as a single tunnel. Therefore, tunnel endpoint implementations SHOULD
allow the operator to configure whether this optimisation is
enabled.When ECN support is added to a subnet technology, consideration
SHOULD be given to a similar optimisation between subnets in sequence
if they all use the same technology.The guidance in this section is worded in terms of framing
boundaries, but it applies equally whether the protocol data units are
frames, cells or packets.Where framing boundaries are different between two layers,
congestion indications SHOULD be propagated on the basis that a
congestion indication on a PDU applies to all the octets in the PDU.
On average, an encapsulator or decapsulator SHOULD approximately
preserve the number of marked octets arriving and leaving (counting
the size of inner headers, but not added encapsulating headers).The next departing frame SHOULD be immediately marked even if only
enough incoming marked octets have arrived for part of the departing
frame. This ensures that any outstanding congestion marked octets are
propagated immediately, rather than held back waiting for a frame no
bigger than the outstanding marked octets—which might involve a
long wait.For instance, an algorithm for marking departing frames could
maintain a counter representing the balance of arriving marked octets
minus departing marked octets. It adds the size of every marked frame
that arrives and if the counter is positive it marks the next frame to
depart and subtracts its size from the counter. This will often leave
a negative remainder in the counter, which is deliberate.The guidance in this section is applicable when IP packets:are encapsulated in Ethernet headers;are forwarded by the eNode-B (base station) of a 3GPP radio
access network, which is required to apply ECN marking during
congestion .This guidance also generalises to encapsulation by other subnet
technologies with no native support for explicit congestion notification
at the lower layer, but with support for finding and processing an IP
header. It is unlikely to be applicable or necessary for IP-in-IP
encapsulation, where feed-forward-and-up mode based on would be more appropriate.Marking the IP header while switching at layer-2 (by using a layer-3
switch) or while forwarding in a radio access network seems to represent
a layering violation. However, it can be considered as a benign
optimisation if the guidelines below are followed. Feed-up-and-forward
is certainly not a general alternative to implementing feed-forward
congestion notification in the lower layer, because:IPv4 and IPv6 are not the only layer-3 protocols that might be
encapsulated by lower layer protocolsLink-layer encryption might be in use, making the layer-2 payload
inaccessibleMany Ethernet switches do not have 'layer-3 switch' capabilities
so they cannot read or modify an IP payloadIt might be costly to find an IP header (v4 or v6) when it may be
encapsulated by more than one lower layer header, e.g. Ethernet MAC
in MAC .Nonetheless, configuring lower layer equipment to look for an ECN
field in an encapsulated IP header is a useful optimisation. If the
implementation follows the guidelines below, this optimisation does not
have to be confined to a controlled environment such as within a data
centre; it could usefully be applied on any network—even if the
operator is not sure whether the above issues will never apply:If a native lower-layer congestion notification mechanism exists
for a subnet technology, it is safe to mix feed-up-and-forward with
feed-forward-and-up on other switches in the same subnet. However,
it will generally be more efficient to use the native mechanism.The depth of the search for an IP header SHOULD be limited. If an
IP header is not found soon enough, or an unrecognised or unreadable
header is encountered, the switch SHOULD resort to an alternative
means of signalling congestion (e.g. drop, or the native lower layer
mechanism if available).It is sufficient to use the first IP header found in the stack;
the egress of the relevant tunnel can propagate congestion
notification upwards to any more deeply encapsulated IP headers
later.It can be seen from that
congestion notification in a subnet using feed-backward mode has
generally not been designed to be directly coupled with IP layer
congestion notification. The subnet attempts to minimise congestion
internally, and if the incoming load at the ingress exceeds the capacity
somewhere through the subnet, the layer 3 buffer into the ingress backs
up. Thus, a feed-backward mode subnet is in some sense similar to a null
mode subnet, in that there is no need for any direct interaction between
the subnet and higher layer congestion notification. Therefore no
detailed protocol design guidelines are appropriate. Nonetheless, a more
general guideline is appropriate: A subnetwork technology intended to eventually interface to IP
SHOULD NOT be designed using only the feed-backward mode, which is
certainly best for a stand-alone subnet, but would need to be
modified to work efficiently as part of the wider Internet, because
IP uses feed-forward-and-up mode.The feed-backward approach at least works beneath IP, where the term
'works' is used only in a narrow functional sense because feed-backward
can result in very inefficient and sluggish congestion
control—except if it is confined to the subnet directly connected
to the original data source, when it is faster than feed-forward. It
would be valid to design a protocol that could work in feed-backward
mode for paths that only cross one subnet, and in feed-forward-and-up
mode for paths that cross subnets.In the early days of TCP/IP, a similar feed-backward approach was
tried for explicit congestion signalling, using source-quench (SQ) ICMP
control packets. However, SQ fell out of favour and is now formally
deprecated . The main problem was that it is
hard for a data source to tell the difference between a spoofed SQ
message and a quench request from a genuine buffer on the path. It is
also hard for a lower layer buffer to address an SQ message to the
original source port number, which may be buried within many layers of
headers, and possibly encrypted.Quantised congestion notification (QCN—also known as backward
congestion notification or BCN) uses a
feed-backward mode structurally similar to ATM's relative rate
mechanism. However, QCN confines its applicability to scenarios such as
some data centres where all endpoints are directly attached by the same
Ethernet technology. If a QCN subnet were later connected into a wider
IP-based internetwork (e.g. when attempting to interconnect multiple
data centres) it would suffer the inefficiency shown .This memo includes no request to IANA.If a lower layer wire protocol is redesigned to include explicit
congestion signalling in-band in the protocol header, care SHOULD be
take to ensure that the field used is specified as mutable during
transit. Otherwise interior nodes signalling congestion would invalidate
any authentication protocol applied to the lower layer header—by
altering a header field that had been assumed as immutable.The redesign of protocols that encapsulate IP in order to propagate
congestion signals between layers raises potential signal integrity
concerns. Experimental or proposed approaches exist for assuring the
end-to-end integrity of in-band congestion signals, e.g.:Congestion exposure (ConEx ) for networks to audit that their
congestion signals are not being suppressed by other networks or by
receivers, and for networks to police that senders are responding
sufficiently to the signals, irrespective of the transport protocol
used .The ECN nonce for a TCP sender to detect
whether a network or the receiver is suppressing congestion
signals.A test with the same goals as the ECN nonce, but without the need
for the receiver to co-operate with the protocol .Given these end-to-end approaches are already being specified,
it would make little sense to attempt to design hop-by-hop congestion
signal integrity into a new lower layer protocol, because end-to-end
integrity inherently achieves hop-by-hop integrity.Following the guidance in the document enables ECN support to be
extended to numerous protocols that encapsulate IP (v4 & v6) in a
consistent way, so that IP continues to fulfil its role as an end-to-end
interoperability layer. This includes:A wide range of tunnelling protocols with various forms of shim
header between two IP headers;A wide range of subnet technologies, particularly those that work
in the same 'feed-forward-and-up' mode that is used to support ECN
in IP and MPLS.Guidelines have been defined for supporting propagation of ECN
between Ethernet and IP on so-called Layer-3 Ethernet switches, using a
'feed-up-an-forward' mode. This approach could enable other subnet
technologies to pass ECN signals into the IP layer, even if they do not
support ECN natively.Finally, attempting to add ECN to a subnet technology in
feed-backward mode is deprecated except in special cases, due to its
likely sluggish response to congestion.Thanks to Gorry Fairhurst for extensive reviews. Thanks also to the
following reviewers: Ingemar Johansson and Piers O'Hanlon and Michael
Welzl, who pointed out that lower layer congestion notification signals
may have different semantics to those in IP.Bob Briscoe was part-funded by the European Community under its
Seventh Framework Programme through the Trilogy project (ICT-216372) for
initial drafts and through the Reducing Internet Transport Latency
(RITE) project (ICT-317700) subsequently. The views expressed here are
solely those of the authors.Comments and questions are encouraged and very welcome. They can be
addressed to the IETF Transport Area working group mailing list
<tsvwg@ietf.org>, and/or to the authors.RBridges: Further TRILL Header ExtensionsThe TRILL base protocol standard specifies minimal hooks to
safely support TRILL Header extensions. Initial extensions have
been specified in RFC [ExtendHeader]. This document specifies the
format for further such extensions and specifies some further
specific extensions.VXLAN: A Framework for Overlaying Virtualized Layer 2
Networks over Layer 3 NetworksThis document describes Virtual eXtensible Local Area Network
(VXLAN), which is used to address the need for overlay networks
within virtualized data centers accommodating multiple tenants.
The scheme and the related protocols can be used in cloud service
provider and enterprise data center networks.Congestion Exposure (ConEx) Concepts and Abstract
MechanismThis document describes an abstract mechanism by which senders
inform the network about the congestion encountered by packets
earlier in the same flow. Today, network elements at any layer may
signal congestion to the receiver by dropping packets or by ECN
markings, and the receiver passes this information back to the
sender in transport-layer feedback. The mechanism described here
enables the sender to also relay this congestion information back
into the network in-band at the IP layer, such that the total
amount of congestion from all elements on the path is revealed to
all IP elements along the path, where it could, for example, be
used to provide input to traffic management. This mechanism is
called congestion exposure or ConEx. The companion document "ConEx
Concepts and Use Cases" provides the entry-point to the set of
ConEx documentation.A TCP Test to Allow Senders to Identify Receiver
Non-ComplianceThe TCP protocol relies on receivers sending accurate and
timely feedback to the sender. Currently the sender has no means
to verify that a receiver is correctly sending this feedback
according to the protocol. A receiver that is non-compliant has
the potential to disrupt a sender's resource allocation,
increasing its transmission rate on that connection which in turn
could adversely affect the network itself. This document presents
a two stage test process that can be used to identify whether a
receiver is non-compliant. The tests enshrine the principle that
one shouldn't attribute to malice that which may be accidental.
The first stage test causes minimum impact to the receiver but
raises a suspicion of non-compliance. The second stage test can
then be used to verify that the receiver is non-compliant. This
specification does not modify the core TCP protocol - the tests
can either be implemented as a test suite or as a stand-alone test
through a simple modification to the sender implementation.IEEE Standard for Local and Metropolitan Area
Networks—Virtual Bridged Local Area Networks—Amendment
6: Provider Backbone BridgesIEEE(Access Controlled link within page)IEEE Standard for Local and Metropolitan Area
Networks—Virtual Bridged Local Area Networks - Amendment 13:
Congestion NotificationCisco(Access Controlled link within page)Traffic Control and Congestion Control in B-ISDNITU-TData Center TCP (DCTCP)Frame Relay: Technology and PracticeGPRS Tunnelling Protocol (GTP) across the Gn and Gp
interface3GPPGeneral Packet Radio System (GPRS) Tunnelling Protocol User
Plane (GTPv1-U)3GPPEvolved Universal Terrestrial Radio Access (E-UTRA) and
Evolved Universal Terrestrial Radio Access Network (E-UTRAN);
Overall description; Stage 23GPPEvolved General Packet Radio Service (GPRS) Tunnelling
Protocol for Control plane (GTPv2-C)3GPPUnderstanding the Available Bit Rate (ABR) Service Category
for ATM VCsCisco[GF] Concern that certain guidelines warrant a MUST (NOT) rather
than a SHOULD (NOT). Given the guidelines say that if any SHOULD
(NOT)s are not followed, a strong justification will be needed, they
have been left as SHOULD (NOT) pending further list discussion. In
particular:If inner is a Not-ECN-PDU and Outer is CE (or highest
severity congestion level), MUST (not SHOULD) drop?Consider whether an IETF Standard Track doc will be needed to
Update the IP-in-IP protocols listed in —at least those
that the IETRe-arranged the introduction to describe the purpose of the
document first before introducing ECN in more depth. And
clarified the introduction throughout.Added applicability to 3GPP TS 36.300.Scope section:Added dependence on correct propagation of traffic class
informationFor the feed-backward mode, deemed multicast and anycast
out of scopeEnsured all guidelines referring to subnet technologies also
refer to tunnels and vice versa by adding applicability
sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and
5.Added Security Considerations on ensuring congestion signal
fields are classed as immutable and on using end-to-end
congestion signal integrity technologies rather than
hop-by-hop.Added authors: JK & PTAdded Section 4.1 "IP-in-IP Tunnels with Tightly Coupled Shim
Headers"Section 4.5 "Sequences of Similar Tunnels or Subnets"roadmap at the start of Section 4, given the subsections
have become quite fragmented.Section 9 "Conclusions"Clarified why transports are starting to be able to saturate
interior linksUnder Section 1.1, addressed the question of alternative
signal semantics and included multicast & anycast.Under Section 3.1, included a 3GPP example.Section 4.2. "Wire Protocol Design":Altered guideline 2. to make it clear that it only
applies to the immediate subnet egress, not later onesAdded a reminder that it is only necessary to check that
ECN propagates at the egress, not whether interior nodes
mark ECNAdded example of how QCN uses 802.1p to indicate support
for QCN.Added references to Appendix C of RFC6040, about monitoring
the amount of congestion signals introduced within a tunnelAppendix A: Added more issues to be addressed, including plan
to produce a standards track update to IP-in-IP tunnel
protocols.Updated acks and referencesIntended status: BCP (was Informational) & updates 3819
added.Briefer Introduction: Introductory para justifying benefits
of ECN. Moved all but a brief enumeration of modes of operation
to their own new section (from both Intro & Scope).
Introduced incr. deployment as most tricky part.Tightened & added to terminology sectionStructured with Modes of Operation, then Guidelines section
for each mode.Tightened up guideline text to remove vagueness / passive
voice / ambiguity and highlight main guidelines as numbered
items.Added Outstanding Document Issues AppendixUpdated references