Internet-Draft Bhuvaneswaran Vengainathan Network Working Group Anton Basil Intended Status: Informational Veryx Technologies Expires: September 2014 Vishwas Manral Ionos Corp March 24, 2014 Benchmarking Methodology for OpenFlow SDN Controller Performance draft-bhuvan-bmwg-of-controller-benchmarking-00 Abstract This document discusses and defines a number of tests that may be used to measure the performance characteristics of OpenFlow (OF) SDN controller. In addition to defining the tests, this document also describes specific formats for reporting the results of the tests. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress. This Internet-Draft will expire on September 20, 2014. Copyright Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. Bhuvan Expires September 20, 2014 [Page 1] Internet Draft OF Controller Benchmarking Methodology March 2014 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.1. Test Setup for Standalone Controller . . . . . . . . . . . 4 4.1.1. Proactive Mode . . . . . . . . . . . . . . . . . . . . . 4 4.1.2. Reactive Mode . . . . . . . . . . . . . . . . . . . . . 5 4.2. Test Setup for Controller Teaming . . . . . . . . . . . . 5 4.2.1. Proactive Mode . . . . . . . . . . . . . . . . . . . . . 5 4.2.2. Reactive Mode . . . . . . . . . . . . . . . . . . . . . . 6 5. Test Considerations . . . . . . . . . . . . . . . . . . . . . 6 5.1. Virtual Switches/Applications . . . . . . . . . . . . . . . 6 5.2. Test Traffic Requirements . . . . . . . . . . . . . . . . . 6 5.3. Southbound Connection Setup Requirements . . . . . . . . . 7 5.3.1. OpenFlow Channel Setup . . . . . . . . . . . . . . . . . 7 5.3.2. Test Recommendations . . . . . . . . . . . . . . . . . . 8 6. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 8 6.1 Performance . . . . . . . . . . . . . . . . . . . . . . . 8 6.1.1 Flow setup rate . . . . . . . . . . . . . . . . . . . 8 6.1.1.1 Flow setup rate under constant network load . . . 8 6.1.1.2 Flow setup rate under incremental network load . . 10 6.1.2 Flow setup delay . . . . . . . . . . . . . . . . . . . 12 6.1.2.1 Flow setup delay under constant network load . . . 12 6.1.2.2 Flow setup delay under incremental network load . . 13 6.1.2.3 Flow setup delay under stress network load . . . . 14 6.1.3 End-End flow setup duration . . . . . . . . . . . . . 16 6.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . 18 6.2.1 OpenFlow connections capacity . . . . . . . . . . . . 17 6.2.2 Switch scalable limit . . . . . . . . . . . . . . . . 19 6.2.3 Flow scalable limit . . . . . . . . . . . . . . . . . 20 6.3 Reliability . . . . . . . . . . . . . . . . . . . . . . . 21 6.3.1 Errored OpenFlow connections handling . . . . . . . . 21 6.3.2 Denial of service handling . . . . . . . . . . . . . . 23 6.3.3 Controller failover time . . . . . . . . . . . . . . . 24 6.3.4 Data path re-convergence time . . . . . . . . . . . . 24 Bhuvan Expires September 20, 2014 [Page 2] Internet Draft OF Controller Benchmarking Methodology March 2014 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 26 7.1. Normative References . . . . . . . . . . . . . . . . . . 26 7.2. Informative References . . . . . . . . . . . . . . . . 26 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 9. Security Considerations . . . . . . . . . . . . . . . . . . . 26 10. Appendix A - Test setup and test applicability . . . . . . . 26 11. Appendix B - OpenFlow protocol overview . . . . . . . . . . 27 11.1. Protocol Overview . . . . . . . . . . . . . . . . . . . 27 11.2. Messages Overview . . . . . . . . . . . . . . . . . . . 27 11.3. Connection Overview . . . . . . . . . . . . . . . . . . 28 12. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 28 1. Introduction This document provides methodologies for benchmarking OpenFlow SDN controller performance. This includes benchmarking controller for flow performance, scalability and reliability. In addition to defining tests, this document also describes specific formats for reporting test results. The results of these tests will provide the vendor and user quantitative data with which to evaluate the controller performance. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 2. Terminology Reactive Flow Setup: Controller can add, update or delete flow entries in flow tables of a switch in response to packets received from the switch Proactive Flow Setup: Controller can add, update or delete flow entries in flow tables of a switch based on trigger from controller's management plane. REST: Representational state transfer OFR: OpenFlow response messages Bhuvan Expires September 20, 2014 [Page 3] Internet Draft OF Controller Benchmarking Methodology March 2014 3. Scope SDN defines logically centralized control plane to enable network programmability and agility. The logically centralized control plane can be a composition of single SDN controller (standalone) or a cluster of SDN controllers (controller teaming) enabling network programmability through one or more southbound interfaces. OpenFlow is one among the protocol widely used for southbound communication. This document defines methodologies for benchmarking OpenFlow based SDN controller performance independent of composition. 4. Test Setup Test configurations defined in this document will be confined to standalone controller and cluster of controllers supporting proactive and/or reactive mode of flow setup. Not all tests described in this document are to be performed for all test setups. These tests are performed by simulating real network scenarios in a lab environment. Appendix A lists the test applicability for each test setup. All of the tests that are relevant to an SDN controller SHOULD be performed. 4.1. Test Setup for Standalone Controller 4.1.1. Proactive Mode -------------------- | Orchestration | -------------------- | | (Northbound interface) ----------------------- | SDN Controller | | (DUT) | ----------------------- | (Southbound interface) | --------------------------- | | | ---------- ---------- ---------- | SDN | | SDN |..| SDN | | Switch 1 | | Switch 2 | | Switch n | ---------- ---------- ---------- Figure 1 Bhuvan Expires September 20, 2014 [Page 4] Internet Draft OF Controller Benchmarking Methodology March 2014 4.1.2. Reactive Mode ----------------------- | SDN Controller | | (DUT) | ----------------------- | (Southbound interface) | --------------------------- | | | ---------- ---------- ---------- | SDN | | SDN |..| SDN | | Switch 1 | | Switch 2 | | Switch n | ---------- ---------- ---------- Figure 2 4.2. Test Setup for Controller Teaming In controller clustering, the connectivity and the communication between controllers are outside the scope of this document. 4.2.1. Proactive Mode -------------------- | Orchestration | -------------------- | | | (Northbound interface) --------------------------------------------------------- | ------------------ ------------------ | | | SDN Controller 1 | <--E/W--> | SDN Controller n | | | ------------------ ------------------ | --------------------------------------------------------- | (Southbound interface) | --------------------------- | | | ---------- ---------- ---------- | SDN | | SDN |..| SDN | | Switch 1 | | Switch 2 | | Switch n | ---------- ---------- ---------- Figure 3 Bhuvan Expires September 20, 2014 [Page 5] Internet Draft OF Controller Benchmarking Methodology March 2014 4.2.2. Reactive Mode --------------------------------------------------------- | ------------------ ------------------ | | | SDN Controller 1 | <--E/W--> | SDN Controller n | | | ------------------ ------------------ | --------------------------------------------------------- | (Southbound interface) | --------------------------- | | | ---------- ---------- ---------- | SDN | | SDN |..| SDN | | Switch 1 | | Switch 2 | | Switch n | ---------- ---------- ---------- Figure 4 5. Test Considerations 5.1. Virtual Switches/Applications SDN controller performance testing involves connections and flows setup from multiple switches, hosts and/or applications. These entities may not necessarily be a real physical hardware and can be emulated in a single hardware platform. The test report MUST indicate the number of switches and hosts used in the test. 5.2. Test Traffic Requirements While the key function of an SDN controller is to program the data plane, the SDN controller may use Layer 2, Layer 3 or Layer 4 fields for programming the data plane. The author understands that it will take a considerable period of time to perform tests for various traffic types and packet sizes. We believe that the results are worth the effort. For the purposes of benchmarking SDN controller performance, the document references traffic type IP of size 128 bytes Bhuvan Expires September 20, 2014 [Page 6] Internet Draft OF Controller Benchmarking Methodology March 2014 5.3. Southbound Connection Setup Requirements The methodologies defined in this document are independent of OpenFlow version and channel setup ways and connection type (e.g., TCP/TLS or SSL).However this document describes a number of channel setup ways, and recommends some of the channel setup ways and connection type over which the tests MUST be carried out and measure the performance. 5.3.1. OpenFlow Channel Setup This section describes three ways of OpenFlow channel setup. 5.3.1.1. Single Connection By default, a single OpenFlow channel is established between each OpenFlow switch and controller when a single controller (standalone) manages a network. 5.3.1.2. Multiple Connections Multiple OpenFlow channels are established from each OpenFlow switch to multiple controllers when multiple controllers are clustered and manage a network In such case, each OpenFlow switch can employ any one of the below modes. Active-Passive: In this mode, one controller takes a MASTER role and other controllers in the cluster take SLAVE role. The selection process is out of this document scope. The OpenFlow channel between the switch and master controller is in active state and others are in standby state. Any asynchronous communications from the switch have to be sent on the active channel. Active-Active: In this mode, all controllers take role as EQUAL. The OpenFlow channel between the switch and all controllers are in active state. Communication between OpenFlow switch and controller cluster can be sent on any active channels. 5.3.1.3. Auxiliary Connection Additional connections apart from the primary connection (as described in section 5.3.1.1. and 5.3.1.2.) can be established between a switch to a controller in order to improve the switch processing performance. Bhuvan Expires September 20, 2014 [Page 7] Internet Draft OF Controller Benchmarking Methodology March 2014 5.3.2. Test Recommendations For standalone controller, the performance tests MUST be carried on 'single connection' using 'TCP'. Tests over other connections like 'auxiliary connections' are optional, depending on a specific support of the product. For controller teaming, the performance tests MUST be carried on 'multiple connections - active-passive mode' using 'TCP'. Tests over other connections like 'multiple connections - active-active', 'auxiliary connections' are optional, depending on a specific support of the product. 6. Benchmarking Tests 6.1 Performance 6.1.1 Flow setup rate Maximum number of flows setup by a controller, expressed in flows per second. This metric is measured in two modes, using constant network load and incremental network load. 6.1.1.1 Flow setup rate under constant network load 6.1.1.1.1 Objective To measure the total number of OpenFlow flow responses (Packet- Out/Flow-Mod) received from the controller under constant network load (fixed number of switches and flows) 6.1.1.1.2 Setup Parameters The following parameters MUST be defined: Network setup parameters: Number of Switches - Defines the number of OpenFlow switch connections established with controller. Number of Flows - Defines unique number of flows setup on each OpenFlow connections. Bhuvan Expires September 20, 2014 [Page 8] Internet Draft OF Controller Benchmarking Methodology March 2014 OpenFlow setup parameters: OpenFlow Version - Defines OpenFlow protocol version used for OpenFlow connection setup. Channel Type - Defines OpenFlow connection setup type used for OpenFlow connection setup, for example TCP or TLS. Test setup parameters: Test Iterations - Defines the number of times the test need to be repeated. Test Duration - Defines the duration of test iteration, expressed in seconds. 6.1.1.1.3 Procedure Reactive Mode Flow Setup: Establish OpenFlow connection from all the specified number of switches. Load the controller with Packet-In messages send from all switches for a configured number of flows and for a specified test duration. Single Packet-In message can have one or more flow headers in its payload. Proactive Mode Flow Setup: Establish OpenFlow connection from all the specified number of switches. Load the controller with configured number of flows through the northbound interface of controller for a specified test duration. The test SHOULD be repeated a number of times for each of the above procedure and record the results. It is recommended to give sufficient time for the controller to flush all the pending response between test iteration. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. Bhuvan Expires September 20, 2014 [Page 9] Internet Draft OF Controller Benchmarking Methodology March 2014 6.1.1.1.4 Measurement Total (OFRi) Flow setup rate (FSRi) = --------------- Test Duration SUM (FSR1, FSR2,..FSRn) Average flow setup rate = ------------------------ Test Iterations Maximum flow setup rate = Max (FSR1,FSR2,..FSRn) Minimum flow setup rate = Min (FSR1,FSR2,..FSRn) Where, FSR1 is the average OpenFlow responses received in 1st iteration and FSRn is the average OpenFlow responses received in nth iteration. Measurement MUST take into account all the Packet-Outs embedded in a single OpenFlow response message (Packet-Out/Flow-Mod). 6.1.1.1.5 Reporting Format The results of the flow setup rate under constant network load test for each of the test procedure SHOULD be plotted as a graph. If this is done then the X axis MUST be the test iteration number The Y axis MUST be the OpenFlow responses per second. The Y axis value range should be between the minimum and maximum OpenFlow setup rate. Plot the flow setup rate results obtained on each test iteration in the graph. The maximum, average and minimum flow setup rate should be reported at the bottom of the graph. The test report MUST indicate whether learning was enabled or disabled. Also report the total number of frames transmitted, number of valid responses and error responses received for each iteration in a table. 6.1.1.2 Flow setup rate under incremental network load 6.1.1.2.1 Objective To measure the total OpenFlow flow responses (Packet-Out/Flow-Mod) received from the controller for incremental network load (increasing number of switches and flows). 6.1.1.2.2 Setup Parameters Use the same setup parameters as defined in section 6.1.1.1.2 Bhuvan Expires September 20, 2014 [Page 10] Internet Draft OF Controller Benchmarking Methodology March 2014 6.1.1.2.3 Procedure Reactive Mode Flow Setup: Establish OpenFlow connection from 10% of the specified number of switches. Load controller with Packet-In messages send from 10% of switches for a configured number of flows and for a specified sample interval. Single Packet-In message can have one or more flow headers in its payload. The test SHOULD be repeated with 20% of the specified number of switches in the next iteration and continue the test until the configured number of switches is included in the test. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. 6.1.1.2.4 Measurement Total (OFRi) Flow setup rate (FSRi) = --------------- Test Duration Maximum flow setup rate = Max (FSR1,FSR2,..FSR10) Minimum flow setup rate = Min (FSR1,FSR2,..FSR10) Where, FSR1 is the average OpenFlow responses received in 1st iteration and FSR10 is the average OpenFlow responses received in 10th iteration. Measurement MUST take into account all the Packet-Outs embedded in a single OpenFlow response message (Packet-Out/Flow-Mod). 6.1.1.2.5 Reporting Format The results of the flow setup rate under incremental network load test SHOULD be plotted as a graph. If this is done then the X axis MUST be the number of switches per test iteration The Y axis MUST be the OpenFlow responses per second. The Y axis value range should be between the minimum and maximum OpenFlow setup rate. Plot the flow setup rate results obtained on each test iteration in the graph. The maximum and minimum flow setup rate should be reported at the bottom of the graph. The test report MUST indicate whether learning was enabled or disabled. Bhuvan Expires September 20, 2014 [Page 11] Internet Draft OF Controller Benchmarking Methodology March 2014 Also report the total number of frames transmitted, number of valid responses and error responses received for each iteration in a table. 6.1.2 Flow setup delay Time taken by the controller to setup a flow, expressed in milliseconds. This metric is measured in three modes, using constant network load, incremental network load and stress network load. 6.1.2.1 Flow setup delay under constant network load 6.1.2.1.1 Objective To measure the time taken by controller to setup a flow under low network load. 6.1.2.1.2 Setup Parameters Use the same setup parameters as defined in section 6.1.1.1.2 6.1.2.1.3 Procedure Reactive Mode Flow Setup: Establish OpenFlow connection from all the specified number of switches. Send Packet-In messages from each of the switches one at a time and wait for the response before soliciting the next flow request. Single Packet-In message MUST have one flow header in its payload. This process is repeated as many times as possible within the configured sample time interval. The test SHOULD be repeated a number of times and record the values. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. 6.1.2.1.4 Measurement Test Duration Flow setup delay (FSDi) = --------------- Total (OFRi) SUM (FSD1, FSD2,..FSDn) Average flow setup delay = ------------------------- Test Iterations Bhuvan Expires September 20, 2014 [Page 12] Internet Draft OF Controller Benchmarking Methodology March 2014 Maximum flow setup delay = Max (FSD1,FSD2,..FSDn) Minimum flow setup delay = Min (FSD1,FSD2,..FSDn) Where, FSD1 is the average time taken to setup a flow in first iteration and FSDn is the average time taken to setup a flow in nth iteration. 6.1.2.1.5 Reporting Format The results of the flow setup delay under constant network load test SHOULD be plotted as a graph. If this is done then the X axis MUST be the test iteration number The Y axis MUST be the Flow setup delay in milliseconds. The Y axis value range should be between the minimum and maximum OpenFlow setup delay. Plot the flow setup delay results obtained on each test iteration in the graph. The maximum, average and minimum flow setup delay should also be reported at the bottom of the graph. The test report MUST indicate whether learning was enabled or disabled. Also report the total number of frames transmitted, number of valid responses and error responses received for each iteration in a table. 6.1.2.2 Flow setup delay under incremental network load 6.1.2.2.1 Objective To measure the amount of time taken by controller to setup a flow under gradual increase of network load. 6.1.2.2.2 Setup Parameters Use the same setup parameters as defined in section 6.1.1.1.2 6.1.2.2.3 Procedure Reactive Mode Flow Setup: Establish OpenFlow connection from 10% of the specified number of switches. Send Packet-In messages from each of the switches one at a time and wait for the response before soliciting the next flow request. Single Packet-In message MUST have one flow header in its payload. This process is repeated as many times as possible within the configured sample time interval. Bhuvan Expires September 20, 2014 [Page 13] Internet Draft OF Controller Benchmarking Methodology March 2014 The test SHOULD be repeated with 20% of the specified number of switches in the next iteration and continue the test until the configured number of switches are included in the test. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. 6.1.2.2.4 Measurement Test Duration Flow setup delay (FSDi) = --------------- Total (OFRi) Maximum flow setup delay = Max (FSD1,FSD2,..FSD10) Minimum flow setup delay = Min (FSD1,FSD2,..FSD10) Where, FSD1 is the average time taken to setup a flow in first iteration and FSD10 is the average time taken to setup a flow in 10th iteration. 6.1.2.2.5 Reporting Format The results of the flow setup delay under incremental network load test SHOULD be plotted as a graph. If this is done then the X axis MUST be the number of switches per test iteration. The Y axis MUST be the Flow setup delay in milliseconds. The Y axis value range should be between the minimum and maximum OpenFlow setup delay. Plot the flow setup delay results obtained on each test iteration in the graph. The maximum and minimum flow setup delay should also be reported at the bottom of the graph. The test report MUST indicate whether learning was enabled or disabled. Also report the total number of frames transmitted, number of valid responses and error responses received for each iteration in a table. 6.1.2.3 Flow setup delay under stress network load 6.1.2.3.1 Objective To measure the amount of time taken by controller to setup a flow under high network load. 6.1.2.3.2 Setup Parameters Use the same setup parameters as defined in section 6.1.1.1.2 Bhuvan Expires September 20, 2014 [Page 14] Internet Draft OF Controller Benchmarking Methodology March 2014 6.1.2.3.3 Procedure Reactive Mode Flow Setup: Establish OpenFlow connection from all the specified number of switches. Reserve one switch as a test switch and send Packet-In messages from other switches for a configured number of flows for a specified sample interval. The Packet-In message sent from other switches can have one or more flow headers in its payload. From the test switch, send Packet-In message one at a time and wait for the response before soliciting the next flow request. At regular frequency, insert time stamp to the Packet-In message and measure the round trip time it takes for the flow to get added in the switch. This process is repeated as many times as possible within the configured sample time interval. The test SHOULD be repeated a number of times and record the values. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. 6.1.2.3.4 Measurement Test Duration Flow setup delay (FSDi) = ------------------------- Total (OFRi on test switch) SUM (FSD1, FSD2,..FSDn) Average flow setup delay = ------------------------ Test Iterations Maximum flow setup delay = Max (FSD1,FSD2,..FSDn) Minimum flow setup delay = Min (FSD1,FSD2,..FSDn) Where, FSD1 is the average time taken to setup a flow in first iteration and FSDn is the average time taken to setup a flow in nth iteration. Bhuvan Expires September 20, 2014 [Page 15] Internet Draft OF Controller Benchmarking Methodology March 2014 6.1.2.3.5 Reporting Format The results of the flow setup delay under constant network load test SHOULD be plotted as a graph. If this is done then the X axis MUST be the test iteration number The Y axis MUST be the Flow setup delay in milliseconds. The Y axis value range should be between the minimum and maximum OpenFlow setup delay. Plot the flow setup delay results obtained on each test iteration in the graph. The maximum, average and minimum flow setup delay should also be reported at the bottom of the graph. The test report MUST indicate whether learning was enabled or disabled. Also report the total number of frames transmitted, number of valid responses and error responses received for each iteration in a table. 6.1.3 End-End flow setup duration 6.1.3.1 Objective To determine the time taken by the controller to setup a flow between a source to a destination, expressed in milliseconds. 6.1.3.2 Setup Parameters The following parameters MUST be defined: Network setup parameters: Number of switches - Total number of switches inter-connected in a topology. OpenFlow setup parameters: OpenFlow Version - Defines OpenFlow protocol version used for OpenFlow connection setup. Channel Type - Defines OpenFlow connection setup type used for OpenFlow connection setup, for example TCP or TLS. Bhuvan Expires September 20, 2014 [Page 16] Internet Draft OF Controller Benchmarking Methodology March 2014 6.1.3.4 Procedure Inter connect the specified number of switches in linear or tree fashion. Establish OpenFlow connection from all switches. To ensure that the controller completes the network discovery process, send traffic stream from a source connected on one end of the network to the destination connected on the other end of network and verify that the traffic reaches the destination properly. Prior sending the traffic, send gratuitous ARP to the controller for the destination. On completion of above verification, send a traffic stream to a different destination with the timestamp inserted on each frame. On completion of the test, gracefully close all the OpenFlow connection established with the controller. The test SHOULD be repeated by varying the number of inter- connected switches. 6.1.3.5 Measurement Flow setup delay = Flow transmit time at the source - Flow reception time at the destination. 6.1.3.6 Reporting Format The test results MUST be reported in a table format with a row for number of switches and a column for Flow setup delay. 6.2 Scalability 6.2.1 OpenFlow connections capacity 6.2.1.1 Objective Maximum number of concurrent OpenFlow connections supported by a controller. 6.2.1.2 Setup Parameters The following parameters MUST be defined: Network setup parameters: Connection Setup Rate - Maximum number of TCP connection requests accept per second. Bhuvan Expires September 20, 2014 [Page 17] Internet Draft OF Controller Benchmarking Methodology March 2014 OpenFlow setup parameters: OpenFlow Version - Defines OpenFlow protocol version used for OpenFlow connection setup. Channel Type - Defines OpenFlow connection setup type used for OpenFlow connection setup, for example TCP or TLS. Test setup parameters: Test Iterations - Defines the number of times the test need to be repeated. 6.2.1.3 Procedure Establish OpenFlow connection with the controller at the specified connection setup rate until the controller drops the OpenFlow connection request. To validate each connection, send a Packet-In message after the OpenFlow connection has been established. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. The test SHOULD be repeated a number of times. Each iteration, record total number of responses received from the controller and the test duration, in milliseconds. 6.2.1.4 Measurement Maximum Concurrent OpenFlow Connections = Max (OFR1, OFR2, .. OFRn) Minimum OpenFlow Connection Establishment Time, in milliseconds Test duration of Min (OFR1, OFR2, .. OFRn) = ------------------------------------------ Min (OFR1, OFR2, .. OFRn) 6.2.1.5 Reporting Format The results of the OpenFlow connection capacity test SHOULD be plotted as a graph. If this is done then the X axis MUST be the test iteration number The Y axis MUST be the OpenFlow connection capacity. Plot the OpenFlow responses obtained on each test iteration in the graph. The Maximum Concurrent OpenFlow Connections and Minimum OpenFlow Connection Establishment Time should be reported at the bottom of the graph. Bhuvan Expires September 20, 2014 [Page 18] Internet Draft OF Controller Benchmarking Methodology March 2014 6.2.2 Switch scalable limit 6.2.2.1 Objective To determine the number of switches a controller can optimally manage. 6.2.2.2 Setup Parameters The following parameters MUST be defined: Network setup parameters: Acceptable Flow Setup Delay - Acceptable flow response time, expressed in milliseconds. Connection Setup Rate - Maximum number of TCP connection requests accept per second. Test setup parameters: Test Iterations - Defines the number of times the test need to be repeated. OpenFlow setup parameters: OpenFlow Version - Defines OpenFlow protocol version used for OpenFlow connection setup. Channel Type - Defines OpenFlow connection setup type used for OpenFlow connection setup, for example TCP or TLS. 6.2.2.3 Procedure Establish OpenFlow connection with the controller at the specified connection setup rate. Reserve one switch as a test switch and send Packet-In messages from other switches at a constant load. The Packet-In message sent from other switches can have one or more flow headers in its payload. From the test switch, send Packet-In message one at a time and wait for the response before soliciting the next flow request. At regular frequency, insert time stamp to the Packet-In message and measure the round trip time it takes for the flow to get added in the switch. Compare the measured flow round trip time with the configured acceptable flow setup delay value at each step. Bhuvan Expires September 20, 2014 [Page 19] Internet Draft OF Controller Benchmarking Methodology March 2014 The above step should be continued until the measured response time is lower or equal to the configured response time. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. The test SHOULD be repeated with varying number of network loads. 6.2.2.4 Measurement Switch Scalable Limit = Number of switch connections active at the step when the measured value starts exceeding the acceptable value 6.2.2.5 Reporting Format The test report MUST note the switch scalable limit factor and the offered network load in terms of flows for every iteration. The results SHOULD be reported in the format of a table with a row for offered network load and columns for number of switch connections attempted and number of active switch connections (Switch Scalable Limit). 6.2.3 Flow scalable limit 6.2.3.1 Objective To determine the controller OpenFlow table capacity 6.2.3.2 Setup Parameters The following parameters MUST be defined: Network setup parameters: Number of Flows - Defines unique number of flows send on the established OpenFlow connection at the beginning of the test. OpenFlow setup parameters: OpenFlow Version - Defines OpenFlow protocol version used for OpenFlow connection setup. Channel Type - Defines OpenFlow connection setup type used for OpenFlow connection setup, for example TCP or TLS. Bhuvan Expires September 20, 2014 [Page 20] Internet Draft OF Controller Benchmarking Methodology March 2014 6.2.3.3 Procedure Establish OpenFlow connection from a switch with the controller. Send Packet-In messages from the established connection for a constant number of flows and measure the Flow-Mod messages received from the controller. Make sure that the controller has learnt all flows destination prior the test. One way to achieve this is by sending Packet-In messages with gratuitous ARP frame for all flows. The Packet-In message sent on each connection can have one or more flow headers in its payload. Repeat the iteration by varying the number of flows until the maximum flow request, at which no flow response loss occurs, is found. Since the controller may employ aging time mechanism for the flow, the controller flow aging time SHOULD be set to a maximum possible value On the completion of the test, gracefully close the OpenFlow session established with controller. 6.2.3.4 Measurement Flow Scalable Limit: Maximum flow request, expressed in packets per second, at which no flow response loss is detected. 6.2.3.5 Reporting Format The test report MUST note the offered load and the responses received on each iteration. The results SHOULD be reported in the format of a table with a row for the offered load and a column for the number of responses received. 6.3 Reliability 6.3.1 Errored OpenFlow connections handling 6.3.1.1 Objective To characterize the behavior of the controller when presented with a combination of both legal and Illegal OpenFlow messages. Bhuvan Expires September 20, 2014 [Page 21] Internet Draft OF Controller Benchmarking Methodology March 2014 6.3.1.2 Setup Parameters The following parameters MUST be defined: Network setup parameters: Number of Switches - Defines the number of OpenFlow switch connections established with controller. Number of Flows - Defines unique number of flows setup on each OpenFlow connections. OpenFlow setup parameters: OpenFlow Version - Defines OpenFlow protocol version used for OpenFlow connection setup. Channel Type - Defines OpenFlow connection setup type used for OpenFlow connection setup, for example TCP or TLS. Test setup parameters: Test Iterations - Defines the number of times the test need to be repeated. Test Duration - Defines the duration of test iteration, expressed in seconds. 6.3.1.3 Procedure Establish OpenFlow connection from all the specified number of switches. Load controller with Packet-In messages send from all switches for a configured number of flows and for a specified test duration. On each connection, after every 5% of flows transmission, send an invalid Packet-In message (message with wrong header field value). Single Packet-In message can have one or more flow headers in its payload. The test SHOULD be repeated a number of times by varying the offered network load (number of switches) and record the results. On the completion of the test, gracefully close all the OpenFlow sessions established with controller. 6.3.1.4 Measurement Average response rate: Total number of OpenFlow responses received divided by the test duration, expressed in responses per second. Bhuvan Expires September 20, 2014 [Page 22] Internet Draft OF Controller Benchmarking Methodology March 2014 6.3.1.5 Reporting Format The test report MUST note the number of switches, average offered load (correct and incorrect messages) and the average responses received. The results SHOULD be reported in the format of a table with a row for the number of switches and columns for the average offered load and average responses received. 6.3.2 Denial of service handling 6.3.2.1 Objective To determine the effect of a denial of service attack on a controller OpenFlow connection establishment rates. The denial of service handling test MUST be run after obtaining baseline measurements from section 6.3. 6.3.2.2 Setup Parameters Use the same setup parameters defined in section 6.2.1.2 In addition, the following setup parameters MUST be defined: OpenFlow connection attack rate - Rate at which the TCP connections are established without sending any OpenFlow messages. 6.3.2.3 Procedure Use the same procedure as defined in section 6.2.1.3. In addition, establish TCP connections at the specified OpenFlow attack rate without performing any Hello exchanges. 6.3.2.4 Measurement Perform the same measurements as defined in section 6.2.1.4. In addition, track the number of TCP connections established without performing any Hello exchanges. 6.3.2.5 Reporting Format The test SHOULD use the same format as defined in section 6.2.1.5. In addition, the test SHOULD report the number of OpenFlow connections attempted without performing any Hello exchanges at the bottom of the report. Bhuvan Expires September 20, 2014 [Page 23] Internet Draft OF Controller Benchmarking Methodology March 2014 6.3.3 Controller failover time 6.3.3.1 Objective To determine the time taken to switch from one controller to another when the controllers are teamed and the master controller fails. 6.3.3.2 Setup Parameters Use the same setup parameters defined in section 6.1.3.2. 6.3.3.3 Procedure Establish OpenFlow connection with MASTER and SLAVE controllers. Duplicate Packet-In message and send on both connections one at a time. Insert timestamp to each Packet-In message sent on both connections. During the test, bring down the master controller. The test SHOULD be continued until first flow response is received from the new master. The test SHOULD be repeated a number of times by varying the number of switches. 6.3.3.4 Measurement Failover Time = (Time at which last response received from the old master) - (Time at which the first response received from the new master) Packet Losses = Total number of unique requests sent - Total number of unique responses received. 6.3.3.5 Reporting Format The test MUST note the number of flow requests sent on each connection and the number of responses received. The results SHOULD be reported in the format of a table with a row for the number of switches and columns for number of requests sent, responses received, packet losses and failover time. 6.3.4 Data path re-convergence time 6.3.4.1 Objective To determine the time taken to re-route a flow by the controller when there is a failure in the existing flow path. Bhuvan Expires September 20, 2014 [Page 24] Internet Draft OF Controller Benchmarking Methodology March 2014 6.3.4.2 Setup Parameters Use the same setup parameters defined in section 6.1.3.2 6.3.4.3 Procedure Inter connect the specified number of switches in tree fashion. Establish OpenFlow connection from all switches. To ensure that the controller completes the network discovery process, send traffic stream from a source connected on one end of the network to the destination connected on the other end of network and verify that the traffic reaches the destination properly. Prior sending the traffic, send gratuitous ARP to the controller for learning the destination. In addition, ensure that the topology has a redundant path from a source to destination. Send a traffic stream for different destination with sequence number inserted in each of the frame. During the course of the test, bring down the link of the traffic path. On completion of the test, gracefully close all the OpenFlow connection established with the controller. Repeat the test with varying number of inter-connected switches. 6.3.4.4 Measurement Failover time: Delta between the receive timestamp of the errored sequence packet (first packet received after re-convergence) and the last valid packet, expressed in milliseconds. Packet Loss: Difference between the sequence number of errored sequence packet and the last valid packet. 6.3.4.5 Reporting Format The test results MUST be reported in a table format with a row for number of switches and a column for failover time and packet losses. Bhuvan Expires September 20, 2014 [Page 25] Internet Draft OF Controller Benchmarking Methodology March 2014 7. References 7.1. Normative References [1] OpenFlow Switch Specification, Version 1.4.0 (Wire Protocol 0x05), October 14, 2013 7.2. Informative References 8. IANA Considerations This document does not have any IANA requests. 9. Security Considerations The primary goal of this document is to provide methodologies in benchmarking OpenFlow controller's performance with respect to southbound communication. There may arise some performance and security issues while interacting through northbound interfaces, assessment of such issues are outside the scope of this document. 10. Appendix A - Test setup and test applicability +-------------------------------------------------------------------+ |Benchmarking Tests |Standalone Controller|Controller Teaming | | |---------------------|---------------------| | |Proactive |Reactive |Proactive |Reactive | +-------------------------------------------------------------------+ |1. Flow Setup Rate | | | | | | - Constant Loadi |Applicable|Applicable|Applicable|Applicable| | - Incremental Load | NA |Applicable| NA |Applicable| | | | | | | |2. Flow Setup Delay | | | | | | - Constant Load | NA |Applicable| NA |Applicable| | - Incremental Load | NA |Applicable| NA |Applicable| | - Stress Load | NA |Applicable| NA |Applicable| | | | | | |3. OpenFlow Connections| | | | | | Capacity | NA |Applicable| NA |Applicable| | | | | | | |4. Switch Scalable | | | | | | Limit | NA |Applicable| NA |Applicable| | | | | | | |5. Flow Scalable Limit | NA |Applicable| NA |Applicable| | | | | | | |6. Errored OpenFlow | | | | | | Connections Handling| NA |Applicable| NA |Applicable| | | | | | | Bhuvan Expires September 20, 2014 [Page 26] Internet Draft OF Controller Benchmarking Methodology March 2014 |7. Denial of Service | | | | | | Handling | NA |Applicable| NA |Applicable| | | | | | | |8. End-End Flow Setup | | | | | | Duration | NA |Applicable| NA |Applicable| | | | | | | |9. Controller Failover | | | | | | Time | NA |Applicable| NA |Applicable| | | | | | | |10. Datapath | | | | | | Re-convergence Time| NA |Applicable| NA |Applicable| +-------------------------------------------------------------------+ 11. Appendix B - OpenFlow protocol overview 11.1. Protocol Overview OpenFlow is an open standard protocol defined by Open Networking Foundation (ONF), used for programming the forwarding plane of network switches or routers via a centralized controller. 11.2. Messages Overview OpenFlow protocol supports three messages types namely controller- to-switch, asynchronous and symmetric. Controller-to-switch messages are initiated by the controller and used to directly manage or inspect the state of the switch. These messages allow controllers to query/configure the switch (Features, Configuration messages), collect information from switch (Read- State message), send packets on specified port of switch (Packet- out message), and modify switch forwarding plane and state (Modify- State, Role-Request messages etc.). Asynchronous messages are generated by the switch without a controller soliciting them. These messages allow switches to update controllers to denote an arrival of new flow (Packet-in), switch state change (Flow-Removed, Port-status) and error (Error). Symmetric messages are generated in either direction without solicitation. These messages allow switches and controllers to set up connection (Hello), verify for liveness (Echo) and offer additional functionalities (Experimenter). Bhuvan Expires September 20, 2014 [Page 27] Internet Draft OF Controller Benchmarking Methodology March 2014 11.3. Connection Overview OpenFlow channel is used to exchange OpenFlow message between an OpenFlow switch and an OpenFlow controller. The OpenFlow channel connection can be setup using plain TCP or TLS. By default, a switch establishes single connection with SDN controller. A switch may establish multiple parallel connections to single controller (auxiliary connection) or multiple controllers to handle controller failures and load balancing. 12. Authors' Addresses Bhuvaneswaran Vengainathan Veryx Technologies Inc. 1 International Plaza, Suite 550 Philadelphia PA 19113 Email: bhuvaneswaran.vengainathan@veryxtech.com Anton Basil Veryx Technologies Inc. 1 International Plaza, Suite 550 Philadelphia PA 19113 Email: anton.basil@veryxtech.com Vishwas Manral Ionos Corp, 4100 Moorpark Ave, San Jose, CA Email: vishwas@ionosnetworks.com Bhuvan Expires September 20, 2014 [Page 28]