<?xml version='1.0' encoding='utf-8'?>
<rfc xmlns:xi="http://www.w3.org/2001/XInclude" version="3" category="std" docName="draft-ietf-nfsv4-layrec-04" number="9737" ipr="trust200902" obsoletes="" updates="" sortRefs="true" submissionType="IETF" symRefs="true" tocDepth="3" tocInclude="true" consensus="true" xml:lang="en" prepTime="2025-02-26T15:32:45" indexInclude="true" scripts="Common,Latin">
  <link href="https://datatracker.ietf.org/doc/draft-ietf-nfsv4-layrec-04" rel="prev"/>
  <link href="https://dx.doi.org/10.17487/rfc9737" rel="alternate"/>
  <link href="urn:issn:2070-1721" rel="alternate"/>
  <front>
    <title abbrev="Reporting Errors via LAYOUTRETURN">Reporting Errors in NFSv4.2 via LAYOUTRETURN</title>
    <seriesInfo name="RFC" value="9737" stream="IETF"/>
    <author fullname="Thomas Haynes" initials="T." surname="Haynes">
      <organization abbrev="Hammerspace" showOnFrontPage="true">Hammerspace</organization>
      <address>
        <email>loghyr@gmail.com</email>
      </address>
    </author>
    <author fullname="Trond Myklebust" initials="T." surname="Myklebust">
      <organization abbrev="Hammerspace" showOnFrontPage="true">Hammerspace</organization>
      <address>
        <email>trondmy@hammerspace.com</email>
      </address>
    </author>
    <date month="02" year="2025"/>
    <area>WIT</area>
    <workgroup>nfsv4</workgroup>
    <keyword>NFSv4</keyword>
    <abstract pn="section-abstract">
      <t indent="0" pn="section-abstract-1">
      The Parallel Network File System (pNFS) allows
      for a file's metadata and data to be on different
      servers (i.e., the metadata server (MDS) and the data server (DS)). When the MDS is restarted, the client
      can still modify the data file component.
      During the
      recovery phase of startup, the MDS and the
      DSs work together to recover state. If the client
      has not encountered errors with the data files, then the state can be
      recovered and the resilvering of the data files can be avoided. With any
      errors, there is no means by which the client can report errors to the
      MDS. As such, the MDS has to
      assume that a file needs resilvering. This document presents an
      extension to RFC 8435 to allow the client to update the metadata
      via LAYOUTRETURN and avoid the resilvering.
      </t>
    </abstract>
    <boilerplate>
      <section anchor="status-of-memo" numbered="false" removeInRFC="false" toc="exclude" pn="section-boilerplate.1">
        <name slugifiedName="name-status-of-this-memo">Status of This Memo</name>
        <t indent="0" pn="section-boilerplate.1-1">
            This is an Internet Standards Track document.
        </t>
        <t indent="0" pn="section-boilerplate.1-2">
            This document is a product of the Internet Engineering Task Force
            (IETF).  It represents the consensus of the IETF community.  It has
            received public review and has been approved for publication by
            the Internet Engineering Steering Group (IESG).  Further
            information on Internet Standards is available in Section 2 of 
            RFC 7841.
        </t>
        <t indent="0" pn="section-boilerplate.1-3">
            Information about the current status of this document, any
            errata, and how to provide feedback on it may be obtained at
            <eref target="https://www.rfc-editor.org/info/rfc9737" brackets="none"/>.
        </t>
      </section>
      <section anchor="copyright" numbered="false" removeInRFC="false" toc="exclude" pn="section-boilerplate.2">
        <name slugifiedName="name-copyright-notice">Copyright Notice</name>
        <t indent="0" pn="section-boilerplate.2-1">
            Copyright (c) 2025 IETF Trust and the persons identified as the
            document authors. All rights reserved.
        </t>
        <t indent="0" pn="section-boilerplate.2-2">
            This document is subject to BCP 78 and the IETF Trust's Legal
            Provisions Relating to IETF Documents
            (<eref target="https://trustee.ietf.org/license-info" brackets="none"/>) in effect on the date of
            publication of this document. Please review these documents
            carefully, as they describe your rights and restrictions with
            respect to this document. Code Components extracted from this
            document must include Revised BSD License text as described in
            Section 4.e of the Trust Legal Provisions and are provided without
            warranty as described in the Revised BSD License.
        </t>
      </section>
    </boilerplate>
    <toc>
      <section anchor="toc" numbered="false" removeInRFC="false" toc="exclude" pn="section-toc.1">
        <name slugifiedName="name-table-of-contents">Table of Contents</name>
        <ul bare="true" empty="true" indent="2" spacing="compact" pn="section-toc.1-1">
          <li pn="section-toc.1-1.1">
            <t indent="0" keepWithNext="true" pn="section-toc.1-1.1.1"><xref derivedContent="1" format="counter" sectionFormat="of" target="section-1"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-introduction">Introduction</xref></t>
            <ul bare="true" empty="true" indent="2" spacing="compact" pn="section-toc.1-1.1.2">
              <li pn="section-toc.1-1.1.2.1">
                <t indent="0" keepWithNext="true" pn="section-toc.1-1.1.2.1.1"><xref derivedContent="1.1" format="counter" sectionFormat="of" target="section-1.1"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-definitions">Definitions</xref></t>
              </li>
              <li pn="section-toc.1-1.1.2.2">
                <t indent="0" keepWithNext="true" pn="section-toc.1-1.1.2.2.1"><xref derivedContent="1.2" format="counter" sectionFormat="of" target="section-1.2"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-requirements-language">Requirements Language</xref></t>
              </li>
            </ul>
          </li>
          <li pn="section-toc.1-1.2">
            <t indent="0" pn="section-toc.1-1.2.1"><xref derivedContent="2" format="counter" sectionFormat="of" target="section-2"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-layout-state-recovery">Layout State Recovery</xref></t>
            <ul bare="true" empty="true" indent="2" spacing="compact" pn="section-toc.1-1.2.2">
              <li pn="section-toc.1-1.2.2.1">
                <t indent="0" pn="section-toc.1-1.2.2.1.1"><xref derivedContent="2.1" format="counter" sectionFormat="of" target="section-2.1"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-when-to-resilver">When to Resilver</xref></t>
              </li>
              <li pn="section-toc.1-1.2.2.2">
                <t indent="0" pn="section-toc.1-1.2.2.2.1"><xref derivedContent="2.2" format="counter" sectionFormat="of" target="section-2.2"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-version-mismatch-considerat">Version Mismatch Considerations</xref></t>
              </li>
            </ul>
          </li>
          <li pn="section-toc.1-1.3">
            <t indent="0" pn="section-toc.1-1.3.1"><xref derivedContent="3" format="counter" sectionFormat="of" target="section-3"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-security-considerations">Security Considerations</xref></t>
          </li>
          <li pn="section-toc.1-1.4">
            <t indent="0" pn="section-toc.1-1.4.1"><xref derivedContent="4" format="counter" sectionFormat="of" target="section-4"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-iana-considerations">IANA Considerations</xref></t>
          </li>
          <li pn="section-toc.1-1.5">
            <t indent="0" pn="section-toc.1-1.5.1"><xref derivedContent="5" format="counter" sectionFormat="of" target="section-5"/>.  <xref derivedContent="" format="title" sectionFormat="of" target="name-normative-references">Normative References</xref></t>
          </li>
          <li pn="section-toc.1-1.6">
            <t indent="0" pn="section-toc.1-1.6.1"><xref derivedContent="" format="none" sectionFormat="of" target="section-appendix.a"/><xref derivedContent="" format="title" sectionFormat="of" target="name-acknowledgments">Acknowledgments</xref></t>
          </li>
          <li pn="section-toc.1-1.7">
            <t indent="0" pn="section-toc.1-1.7.1"><xref derivedContent="" format="none" sectionFormat="of" target="section-appendix.b"/><xref derivedContent="" format="title" sectionFormat="of" target="name-authors-addresses">Authors' Addresses</xref></t>
          </li>
        </ul>
      </section>
    </toc>
  </front>
  <middle>
    <section anchor="sec_intro" numbered="true" toc="include" removeInRFC="false" pn="section-1">
      <name slugifiedName="name-introduction">Introduction</name>
      <t indent="0" pn="section-1-1">
    In the Network File System version 4 (NFSv4) with a Parallel NFS
    (pNFS) flexible file layout <xref target="RFC8435" format="default" sectionFormat="of" derivedContent="RFC8435"/> server, during recovery after a restart,
    there is no mechanism for the client
    to inform the metadata server (MDS) about an error that occurred during a
    WRITE operation (see <xref section="18.32" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-18.32" derivedContent="RFC8881"/>) to the data servers (DSs) in the period of
    the outage.
      </t>
      <t indent="0" pn="section-1-2">
    Using the process detailed in <xref target="RFC8178" format="default" sectionFormat="of" derivedContent="RFC8178"/>, the revisions in this document become an
    extension of NFSv4.2 <xref target="RFC7862" format="default" sectionFormat="of" derivedContent="RFC7862"/>. They are built on top of the External Data
    Representation (XDR) <xref target="RFC4506" format="default" sectionFormat="of" derivedContent="RFC4506"/> generated from <xref target="RFC7863" format="default" sectionFormat="of" derivedContent="RFC7863"/>.
      </t>
      <section anchor="sec_defs" numbered="true" toc="include" removeInRFC="false" pn="section-1.1">
        <name slugifiedName="name-definitions">Definitions</name>
        <t indent="0" pn="section-1.1-1">
      See <xref section="1.1" target="RFC8435" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8435#section-1.1" derivedContent="RFC8435"/> for a set of definitions.
        </t>
      </section>
      <section numbered="true" toc="include" removeInRFC="false" pn="section-1.2">
        <name slugifiedName="name-requirements-language">Requirements Language</name>
        <t indent="0" pn="section-1.2-1">
    The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
    "<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to be interpreted as
    described in BCP 14 <xref target="RFC2119" format="default" sectionFormat="of" derivedContent="RFC2119"/> <xref target="RFC8174" format="default" sectionFormat="of" derivedContent="RFC8174"/> 
    when, and only when, they appear in all capitals, as shown here.
        </t>
      </section>
    </section>
    <section anchor="layout_state_recovery" numbered="true" toc="include" removeInRFC="false" pn="section-2">
      <name slugifiedName="name-layout-state-recovery">Layout State Recovery</name>
      <t indent="0" pn="section-2-1">
    When an MDS restarts, clients are provided a grace recovery period where
    they are allowed to recover any state that
    they had established. With open files, the client can send an OPEN operation (see
    <xref section="18.16" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-18.16" derivedContent="RFC8881"/>)
    with a claim type of CLAIM_PREVIOUS (see <xref section="9.11" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-9.11" derivedContent="RFC8881"/>). The client
    uses the RECLAIM_COMPLETE operation (see <xref section="18.51" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-18.51" derivedContent="RFC8881"/>) 
    to notify the MDS that it is done reclaiming state.
      </t>
      <t indent="0" pn="section-2-2">
    The NFSv4 flexible file layout type allows for the client to mirror files
    (see <xref section="8" target="RFC8435" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8435#section-8" derivedContent="RFC8435"/>).
    With client-side mirroring, it is important for the client to inform
    the MDS of any I/O errors encountered with one of the mirrors.
    This is the only way for the MDS to determine if one or more
    of the mirrors are corrupt and then repair the mirrors via resilvering
    (see <xref section="1.1" target="RFC8435" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8435#section-1.1" derivedContent="RFC8435"/>).
    The client can use LAYOUTRETURN (see
    <xref section="18.44" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-18.44" derivedContent="RFC8881"/>)
    and the ff_ioerr4 structure (see <xref section="9.1.1" target="RFC8435" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8435#section-9.1.1" derivedContent="RFC8435"/>) to inform
    the MDS of I/O errors.
      </t>
      <t indent="0" pn="section-2-3">
    A problem arises when the MDS restarts and the client has
    errors it needs to report but cannot do so. <xref section="12.7.4" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-12.7.4" derivedContent="RFC8881"/> requires
    that the client <bcp14>MUST</bcp14> stop using layouts. While the
    intent there is that the client <bcp14>MUST</bcp14> stop doing I/O
    to the storage devices, it is also true that the layout stateids
    are no longer valid. The LAYOUTRETURN needs
    a layout stateid to proceed, and the client cannot get a layout
    during grace recovery (see <xref section="12.7.4" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-12.7.4" derivedContent="RFC8881"/>) to
    recover layout state. As such, clients have no choice but to not recover
    files with I/O errors. In turn, the MDS <bcp14>MUST</bcp14>
    assume that the mirrors are inconsistent and pick one for resilvering.
    It is a <bcp14>MUST</bcp14> because even if the MDS can
    determine that the client did modify data during the outage, it <bcp14>MUST NOT</bcp14>
    assume those modifications were consistent.
      </t>
      <t indent="0" pn="section-2-4">
    To fix this issue, the MDS <bcp14>MUST</bcp14> accept
    the anonymous stateid of all zeros (see <xref section="8.2.3" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-8.2.3" derivedContent="RFC8881"/>) for the lrf_stateid in LAYOUTRETURN (see <xref section="18.44.1" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-18.44.1" derivedContent="RFC8881"/>).
    The client can use this anonymous stateid to
    inform the MDS of errors
    encountered. The MDS can then
    accurately resilver the file by picking the mirror(s) that does not
    have any associated errors.
      </t>
      <t indent="0" pn="section-2-5">
    During the grace period, if the client sends an lrf_stateid
    in the LAYOUTRETURN with any value other than the
    anonymous stateid of all zeros, then the MDS
    <bcp14>MUST</bcp14> respond with an error of
    NFS4ERR_GRACE (see <xref section="15.1.9.2" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-15.1.9.2" derivedContent="RFC8881"/>).
    After the grace period, if the client sends an lrf_stateid
    in the LAYOUTRETURN with a value of the anonymous stateid of all zeros, then the MDS
    <bcp14>MUST</bcp14> respond with an error of
    NFS4ERR_NO_GRACE (see <xref section="15.1.9.3" target="RFC8881" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8881#section-15.1.9.3" derivedContent="RFC8881"/>).
      </t>
      <t indent="0" pn="section-2-6">
    
    Also, when the MDS builds the reply to the LAYOUTRETURN
    with an lrf_stateid with the value of the anonymous stateid of all zeros,
    it <bcp14>MUST NOT</bcp14> bump the seqid of the lorr_stateid.
      </t>
      <t indent="0" pn="section-2-7">
    If the MDS detects that the layout being returned in
    the LAYOUTRETURN does not match the current mirror instances found
    for the file, then it <bcp14>MUST</bcp14> ignore the LAYOUTRETURN and resilver the
    file in question.
      </t>
      <t indent="0" pn="section-2-8">
    The MDS <bcp14>MUST</bcp14> resilver any files
    that are neither explicitly recovered with a CLAIM_PREVIOUS nor
    have a reported error via a LAYOUTRETURN.
    The client has most likely restarted and lost any state.
      </t>
      <section anchor="sec_when_to_resilver" numbered="true" toc="include" removeInRFC="false" pn="section-2.1">
        <name slugifiedName="name-when-to-resilver">When to Resilver</name>
        <t indent="0" pn="section-2.1-1">
      A write intent occurs when a client opens a file and gets
      a LAYOUTIOMODE4_RW from the MDS. The MDS
      <bcp14>MUST</bcp14> track outstanding write intents, and when it
      restarts, it <bcp14>MUST</bcp14> track recovery of those
      write intents.
      The method that the MDS uses to track write intents is
      implementation specific, i.e., outside the scope of this document.
        </t>
        <t indent="0" pn="section-2.1-2">
      The decision to resilver a file depends on how the client recovers the
      file before the grace period ends. If the client reclaims the file
      and reports no errors, the MDS <bcp14>MUST NOT</bcp14>
      resilver the file. If the client reports an error on the file,
      then the file <bcp14>MUST</bcp14> be resilvered. If the client
      does not reclaim or report an error before the grace period ends,
      then under the old behavior, the MDS <bcp14>MUST</bcp14>
      resilver the file.
        </t>
        <t indent="0" pn="section-2.1-3">
      The resilvering process is broadly to:
        </t>
        <ol indent="adaptive" spacing="normal" start="1" type="1" pn="section-2.1-4">
      <li pn="section-2.1-4.1" derivedCounter="1.">
        fence the file (see <xref section="2.2" target="RFC8435" format="default" sectionFormat="of" derivedLink="https://rfc-editor.org/rfc/rfc8435#section-2.2" derivedContent="RFC8435"/>),
      </li>
          <li pn="section-2.1-4.2" derivedCounter="2.">
        record the need to resilver,
      </li>
          <li pn="section-2.1-4.3" derivedCounter="3.">
        release the write intent, and
      </li>
          <li pn="section-2.1-4.4" derivedCounter="4.">
        once there are no write intents on the file, start the resilvering process.
      </li>
        </ol>
        <t indent="0" pn="section-2.1-5">
      The MDS <bcp14>MUST NOT</bcp14> resilver a file if there
      are clients with outstanding write intents, i.e., multiple clients
      might have the file open with write intents.  As the MDS <bcp14>MUST</bcp14>
      track write intents, it <bcp14>MUST</bcp14> also track the need to
      resilver, i.e., if the MDS restarts during the grace
      period, it <bcp14>MUST</bcp14> restart the file recovery if it
      replays the write intent, or else it <bcp14>MUST</bcp14> start
      the resilvering if it replays the resilvering intent.
        </t>
        <t indent="0" pn="section-2.1-6">
      Whether the MDS prevents all I/O to
      the file until the resilvering is done, forces all I/O to go through
      the MDS, or allows a proxy server to update the new data
      file as it is being resilvered is all an implementation choice. The
      constraint is that the MDS is responsible for the
      reconstruction of the data file and for the consistency of the
      mirrors.
        </t>
        <t indent="0" pn="section-2.1-7">
      If the MDS does allow the client access to the
      file during the resilvering, then the client <bcp14>MUST</bcp14> have
      the same layout (set of mirror instances) after the MDS
      as before. One way that such a resilvering can occur is for a proxy
      server to be inserted into the layout. That server will be copying
      a good mirror instance to a new instance. As it gets I/O via the
      layout, it will be responsible for updating the copy it is performing.
      This requirement is that the proxy server <bcp14>MUST</bcp14>
      stay in the layout until the grace period is finished.
        </t>
      </section>
      <section anchor="sec_vers_mismatch" numbered="true" toc="include" removeInRFC="false" pn="section-2.2">
        <name slugifiedName="name-version-mismatch-considerat">Version Mismatch Considerations</name>
        <t indent="0" pn="section-2.2-1">
      The MDS has no expectations for the client to use this
      new functionality. Therefore, if the client does not use it, the
      MDS will function normally.
        </t>
        <t indent="0" pn="section-2.2-2">
      If the client does use the new functionality and the MDS does
      not support it, then the MDS <bcp14>MUST</bcp14> reply with
      a NFS4ERR_BAD_STATEID to the LAYOUTRETURN. If the client detects
      a NFS4ERR_BAD_STATEID error in this scenario, it should fall back to
      the old behavior of not reporting errors.
        </t>
      </section>
    </section>
    <section anchor="sec_security" numbered="true" toc="include" removeInRFC="false" pn="section-3">
      <name slugifiedName="name-security-considerations">Security Considerations</name>
      <t indent="0" pn="section-3-1">
    There are no new security considerations beyond those in
    <xref target="RFC7862" format="default" sectionFormat="of" derivedContent="RFC7862"/>.
      </t>
    </section>
    <section anchor="sec_iana" numbered="true" toc="include" removeInRFC="false" pn="section-4">
      <name slugifiedName="name-iana-considerations">IANA Considerations</name>
      <t indent="0" pn="section-4-1">
 This document has no IANA actions.
      </t>
    </section>
  </middle>
  <back>
    <references pn="section-5">
      <name slugifiedName="name-normative-references">Normative References</name>
      <reference anchor="RFC2119" target="https://www.rfc-editor.org/info/rfc2119" quoteTitle="true" derivedAnchor="RFC2119">
        <front>
          <title>Key words for use in RFCs to Indicate Requirement Levels</title>
          <author fullname="S. Bradner" initials="S." surname="Bradner"/>
          <date month="March" year="1997"/>
          <abstract>
            <t indent="0">In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.</t>
          </abstract>
        </front>
        <seriesInfo name="BCP" value="14"/>
        <seriesInfo name="RFC" value="2119"/>
        <seriesInfo name="DOI" value="10.17487/RFC2119"/>
      </reference>
      <reference anchor="RFC4506" target="https://www.rfc-editor.org/info/rfc4506" quoteTitle="true" derivedAnchor="RFC4506">
        <front>
          <title>XDR: External Data Representation Standard</title>
          <author fullname="M. Eisler" initials="M." role="editor" surname="Eisler"/>
          <date month="May" year="2006"/>
          <abstract>
            <t indent="0">This document describes the External Data Representation Standard (XDR) protocol as it is currently deployed and accepted. This document obsoletes RFC 1832. [STANDARDS-TRACK]</t>
          </abstract>
        </front>
        <seriesInfo name="STD" value="67"/>
        <seriesInfo name="RFC" value="4506"/>
        <seriesInfo name="DOI" value="10.17487/RFC4506"/>
      </reference>
      <reference anchor="RFC7862" target="https://www.rfc-editor.org/info/rfc7862" quoteTitle="true" derivedAnchor="RFC7862">
        <front>
          <title>Network File System (NFS) Version 4 Minor Version 2 Protocol</title>
          <author fullname="T. Haynes" initials="T." surname="Haynes"/>
          <date month="November" year="2016"/>
          <abstract>
            <t indent="0">This document describes NFS version 4 minor version 2; it describes the protocol extensions made from NFS version 4 minor version 1. Major extensions introduced in NFS version 4 minor version 2 include the following: Server-Side Copy, Application Input/Output (I/O) Advise, Space Reservations, Sparse Files, Application Data Blocks, and Labeled NFS.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="7862"/>
        <seriesInfo name="DOI" value="10.17487/RFC7862"/>
      </reference>
      <reference anchor="RFC7863" target="https://www.rfc-editor.org/info/rfc7863" quoteTitle="true" derivedAnchor="RFC7863">
        <front>
          <title>Network File System (NFS) Version 4 Minor Version 2 External Data Representation Standard (XDR) Description</title>
          <author fullname="T. Haynes" initials="T." surname="Haynes"/>
          <date month="November" year="2016"/>
          <abstract>
            <t indent="0">This document provides the External Data Representation (XDR) description for NFS version 4 minor version 2.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="7863"/>
        <seriesInfo name="DOI" value="10.17487/RFC7863"/>
      </reference>
      <reference anchor="RFC8174" target="https://www.rfc-editor.org/info/rfc8174" quoteTitle="true" derivedAnchor="RFC8174">
        <front>
          <title>Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words</title>
          <author fullname="B. Leiba" initials="B." surname="Leiba"/>
          <date month="May" year="2017"/>
          <abstract>
            <t indent="0">RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.</t>
          </abstract>
        </front>
        <seriesInfo name="BCP" value="14"/>
        <seriesInfo name="RFC" value="8174"/>
        <seriesInfo name="DOI" value="10.17487/RFC8174"/>
      </reference>
      <reference anchor="RFC8178" target="https://www.rfc-editor.org/info/rfc8178" quoteTitle="true" derivedAnchor="RFC8178">
        <front>
          <title>Rules for NFSv4 Extensions and Minor Versions</title>
          <author fullname="D. Noveck" initials="D." surname="Noveck"/>
          <date month="July" year="2017"/>
          <abstract>
            <t indent="0">This document describes the rules relating to the extension of the NFSv4 family of protocols. It covers the creation of minor versions, the addition of optional features to existing minor versions, and the correction of flaws in features already published as Proposed Standards. The rules relating to the construction of minor versions and the interaction of minor version implementations that appear in this document supersede the minor versioning rules in RFC 5661 and other RFCs defining minor versions.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="8178"/>
        <seriesInfo name="DOI" value="10.17487/RFC8178"/>
      </reference>
      <reference anchor="RFC8435" target="https://www.rfc-editor.org/info/rfc8435" quoteTitle="true" derivedAnchor="RFC8435">
        <front>
          <title>Parallel NFS (pNFS) Flexible File Layout</title>
          <author fullname="B. Halevy" initials="B." surname="Halevy"/>
          <author fullname="T. Haynes" initials="T." surname="Haynes"/>
          <date month="August" year="2018"/>
          <abstract>
            <t indent="0">Parallel NFS (pNFS) allows a separation between the metadata (onto a metadata server) and data (onto a storage device) for a file. The flexible file layout type is defined in this document as an extension to pNFS that allows the use of storage devices that require only a limited degree of interaction with the metadata server and use already-existing protocols. Client-side mirroring is also added to provide replication of files.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="8435"/>
        <seriesInfo name="DOI" value="10.17487/RFC8435"/>
      </reference>
      <reference anchor="RFC8881" target="https://www.rfc-editor.org/info/rfc8881" quoteTitle="true" derivedAnchor="RFC8881">
        <front>
          <title>Network File System (NFS) Version 4 Minor Version 1 Protocol</title>
          <author fullname="D. Noveck" initials="D." role="editor" surname="Noveck"/>
          <author fullname="C. Lever" initials="C." surname="Lever"/>
          <date month="August" year="2020"/>
          <abstract>
            <t indent="0">This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 7530) and protocol extensions made subsequently. The later minor version has no dependencies on NFS version 4 minor version 0, and is considered a separate protocol.</t>
            <t indent="0">This document obsoletes RFC 5661. It substantially revises the treatment of features relating to multi-server namespace, superseding the description of those features appearing in RFC 5661.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="8881"/>
        <seriesInfo name="DOI" value="10.17487/RFC8881"/>
      </reference>
    </references>
    <section numbered="false" toc="include" removeInRFC="false" pn="section-appendix.a">
      <name slugifiedName="name-acknowledgments">Acknowledgments</name>
      <t indent="0" pn="section-appendix.a-1"><contact fullname="Tigran Mkrtchyan"/>, <contact fullname="Jeff       Layton"/>, and <contact fullname="Rick Macklem"/> provided reviews of
      the document.</t>
    </section>
    <section anchor="authors-addresses" numbered="false" removeInRFC="false" toc="include" pn="section-appendix.b">
      <name slugifiedName="name-authors-addresses">Authors' Addresses</name>
      <author fullname="Thomas Haynes" initials="T." surname="Haynes">
        <organization abbrev="Hammerspace" showOnFrontPage="true">Hammerspace</organization>
        <address>
          <email>loghyr@gmail.com</email>
        </address>
      </author>
      <author fullname="Trond Myklebust" initials="T." surname="Myklebust">
        <organization abbrev="Hammerspace" showOnFrontPage="true">Hammerspace</organization>
        <address>
          <email>trondmy@hammerspace.com</email>
        </address>
      </author>
    </section>
  </back>
</rfc>
