From afab54226927711c719e8012c43333273f581ac7 Mon Sep 17 00:00:00 2001

From: Numan Siddique <numans@ovn.org>

Date: Sat, 1 Feb 2020 16:23:43 +0530

Subject: [PATCH] ovn-northd: Address scale issues with DNAT flows.

When the commit [1] added Distributed NAT support in OVN, it didn't address

the requirement of making East/West NAT traffic distributed. The E/W NAT

traffic was still centralized. Later a couple of patches [2], addressed this

requirement. But the approach taken in [2] resulted in a lot of logical flows

as number of dnat_and_snat entries increase, as reported in @Reported-at.

This patch

  - reverts the approch taken in [2].

  - removing the flows which does the NAT direct (REGBIT_NAT_REDIRECT) to

    the gateway chassis.

  - and to solve the E/W centralized NAT it does the following:

     * Since for each NAT entry we know the MAC binding to be used for the

       external_ip - either the external_mac if set or the MAC of the

       distributed gateway router port, this patch adds the flows in the

       S_ROUTER_IN_ARP_RESOLVE stage to set the eth.dst to the MAC if the

       IP destination is external_ip.

     * The existing flows in the S_ROUTER_OUT_EGR_LOOP are now added by additional

       match -  is_chassis_resident('P') - where 'P' is logical_port of the NAT entry

       if set, otherwise it is the chassis resident port of distributed router port.

       With this additional match, the packet will be loopbacked to apply the unSNAT/DNAT

       rules on the relevant chassis.

Suppose if a logical port 'P' with IP 'A' has a dnat_and_snat entry with external_mac/logical_port

set, and if the packet's IP destination is one of the DNAT IP - then the packet will be sent out

of the local chassis, since eth.dst is resolved in the S_ROUTER_IN_ARP_RESOLVE stage.

If the external_mac/logical_port is not in NAT entry, then the packet will be redirected to

the gateway chassis.

With this patch, for the logical resource reported in @Reported-at, the number of logical

flows come down to around 45k from 650k.

[1] - ceacd9d49316("ovn: distributed NAT flows")

[2] - 551e3d989557("OVN: fix DVR Floating IP support")

      8244c6b6bd88("OVN: do not distribute traffic for local FIP")

Reported-at: https://mail.openvswitch.org/pipermail/ovs-discuss/2020-January/049714.html

Reported-by: Daniel Alvarez Sanchez <dalvarez@redhat.com>

Signed-off-by: Numan Siddique <numans@ovn.org>

Acked-by: Dumitru Ceara <dceara@redhat.com>

Tested-By: Daniel Alvarez Sanchez <dalvarez@redhat.com>

Acked-By: Daniel Alvarez Sanchez <dalvarez@redhat.com>

(cherry-picked from upstream commit 2dc7869436de32205f60128172196b3a207ab265)

Conflicts:

	ovn/northd/ovn-northd.c

Change-Id: I7684c7f5114ba7c800293e843d5d4b856dedbb96

---

 ovn/northd/ovn-northd.8.xml | 191 ++++++++------------------

 ovn/northd/ovn-northd.c     | 263 +++++-------------------------------

 tests/ovn-northd.at         |   8 +-

 3 files changed, 98 insertions(+), 364 deletions(-)

diff --git a/ovn/northd/ovn-northd.8.xml b/ovn/northd/ovn-northd.8.xml

index d94d9aef9..a42a67c19 100644

--- a/ovn/northd/ovn-northd.8.xml

+++ b/ovn/northd/ovn-northd.8.xml

@@ -1487,6 +1487,24 @@ next;

+      

+        

+          For each NAT entry of a distributed logical router  (with

+          distributed gateway router port) of type snat,

+          a priorirty-120 flow with the match inport == P

+          && ip4.src == A advances the packet to

+          the next pipeline, where P is the distributed logical

+          router port and A is the external_ip set

+          in the NAT entry. If A is an IPv6 address, then

+          ip6.src is used for the match.

+        

+

+        

+          The above flow is required to handle the routing of the East/west NAT

+          traffic.

+        

+      

+

           L3 admission control: A priority-100 flow drops packets that match

@@ -1977,21 +1995,6 @@ icmp6 {

           redirect-chassis.

-        

-          For each configuration in the OVN Northbound database, that asks

-          to change the source IP address of a packet from A to

-          B, a priority-50 flow matches

-          ip && ip4.dst == B or

-          ip && ip6.dst == B

-          with an action

-          REGBIT_NAT_REDIRECT = 1; next;.  This flow is for

-          east/west traffic to a NAT destination IPv4/IPv6 address.  By

-          setting the REGBIT_NAT_REDIRECT flag, in the

-          ingress table Gateway Redirect this will trigger a

-          redirect to the instance of the gateway port on the

-          redirect-chassis.

-        

-

           A priority-0 logical flow with match 1 has actions

           next;.

@@ -2147,20 +2150,6 @@ icmp6 {

           redirect-chassis.

-        

-          For each configuration in the OVN Northbound database, that asks

-          to change the destination IP address of a packet from A to

-          B, a priority-50 flow matches ip &&

-          ip4.dst == B or ip &&

-          ip6.dst == B with an action

-          REGBIT_NAT_REDIRECT = 1; next;.  This flow is for

-          east/west traffic to a NAT destination IPv4/IPv6 address.  By

-          setting the REGBIT_NAT_REDIRECT flag, in the

-          ingress table Gateway Redirect this will trigger a

-          redirect to the instance of the gateway port on the

-          redirect-chassis.

-        

-

           A priority-0 logical flow with match 1 has actions

           next;.

@@ -2285,54 +2274,6 @@ output;

-      

-        

-          For distributed logical routers where one of the logical router

-          ports specifies a redirect-chassis, a priority-400

-          logical flow for each ip source/destination couple that matches the

-          dnat_and_snat NAT rules configured. These flows will

-          allow to properly forward traffic to the external connections if

-          available and avoid sending it through the tunnel.

-          Assuming the two following NAT rules have been configured:

-        

-

-        

-external_ip{0,1} = EIP{0,1};

-external_mac{0,1} = MAC{0,1};

-logical_ip{0,1} = LIP{0,1};

-        

-

-        

-            the following action will be applied:

-        

-

-        

-eth.dst = MAC0;

-eth.src = MAC1;

-reg0 = ip4.dst; /* xxreg0 = ip6.dst; in the IPv6 case */

-reg1 = EIP1; /* xxreg1 in the IPv6 case */

-outport = redirect-chassis-port;

-REGBIT_DISTRIBUTED_NAT = 1; next;.

-        

-

-        

-            Morover a priority-400 logical flow is configured for each

-            dnat_and_snat NAT rule configured in order to

-            not send traffic for local FIP through the overlay tunnels

-            but manage it in the local hypervisor

-        

-      

-

-      

-        

-          For distributed logical routers where one of the logical router

-          ports specifies a redirect-chassis, a priority-300

-          logical flow with match REGBIT_NAT_REDIRECT == 1 has

-          actions ip.ttl--; next;.  The outport

-          will be set later in the Gateway Redirect table.

-        

-      

-

           IPv4 routing table.  For each route to IPv4 network N with

@@ -2427,23 +2368,6 @@ next;

-      

-        

-          For distributed logical routers where one of the logical router

-          ports specifies a redirect-chassis, a priority-400

-          logical flow with match REGBIT_DISTRIBUTED_NAT == 1

-          has action next;

-        

-        

-          For distributed logical routers where one of the logical router

-          ports specifies a redirect-chassis, a priority-200

-          logical flow with match REGBIT_NAT_REDIRECT == 1 has

-          actions eth.dst = E; next;, where

-          E is the ethernet address of the router's distributed

-          gateway port.

-        

-      

-

           Static MAC bindings.  MAC bindings can be known statically based on

@@ -2518,6 +2442,35 @@ next;

+      

+        

+          Static MAC bindings from NAT entries.  MAC bindings can also be known

+          for the entries in the NAT table. Below flows are

+          programmed for distributed logical routers i.e with a distributed

+          router port.

+        

+

+        

+          For each row in the NAT table with IPv4 address

+          A in the 

+          table="NAT" db="OVN_Northbound"/> column of

+          <ref table="NAT" db="OVN_Northbound"/> table, a priority-100

+          flow with the match outport === P &&

+          reg0 == A has actions eth.dst = E;

+          next;, where P is the distributed logical router

+          port, E is the Ethernet address if set in the

+          <ref column="external_mac" table="NAT" db="OVN_Northbound"/> column

+          of <ref table="NAT" db="OVN_Northbound"/> table for of type

+          dnat_and_snat, otherwise the Ethernet address of the

+          distributed logical router port.

+        

+

+        

+          For IPv6 NAT entries, same flows are added, but using the register

+          xxreg0 for the match.

+        

+      

+

           Dynamic MAC bindings.  These flows resolve MAC-to-IP bindings

@@ -2640,20 +2593,6 @@ icmp4 {

-      

-        A priority-300 logical flow with match

-        REGBIT_DISTRIBUTED_NAT == 1 has action

-        next;

-      

-      

-        A priority-200 logical flow with match

-        REGBIT_NAT_REDIRECT == 1 has actions

-        outport = CR; next;, where CR

-        is the chassisredirect port representing the instance

-        of the logical router distributed gateway port on the

-        redirect-chassis.

-      

-

         A priority-150 logical flow with match

         outport == GW &&

@@ -2945,19 +2884,6 @@ nd_ns {

       ports specifies a redirect-chassis.

-    

-      Earlier in the ingress pipeline, some east-west traffic was

-      redirected to the chassisredirect port, based on

-      flows in the UNSNAT and DNAT ingress

-      tables setting the REGBIT_NAT_REDIRECT flag, which

-      then triggered a match to a flow in the

-      Gateway Redirect ingress table.  The intention was

-      not to actually send traffic out the distributed gateway port

-      instance on the redirect-chassis.  This traffic was

-      sent to the distributed gateway port instance in order for DNAT

-      and/or SNAT processing to be applied.

-    

-

       While UNDNAT and SNAT processing have already occurred by this

       point, this traffic needs to be forced through egress loopback on

@@ -2973,23 +2899,20 @@ nd_ns {

-        

-          For each dnat_and_snat NAT rule couple in the

-          OVN Northbound database on a distributed router,

-          a priority-200 logical with match

-          ip4.dst == external_ip0 &&

-          ip4.src == external_ip1, has action

-          next;

-        

-

           For each NAT rule in the OVN Northbound database on a

           distributed router, a priority-100 logical flow with match

           ip4.dst == E &&

-          outport == GW, where E is the

-          external IP address specified in the NAT rule, and GW

-          is the logical router distributed gateway port, with the

-          following actions:

+          outport == GW &&

+          is_chassis_resident(P), where E is the

+          external IP address specified in the NAT rule, GW

+          is the logical router distributed gateway port. For dnat_and_snat

+          NAT rule, P is the logical port specified in the NAT rule.

+          If 

+          table="NAT" db="OVN_Northbound"/> column of

+          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then

+          P is the chassisredirect port of

+          GW with the following actions:

diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c

index 58213742d..e2df94f4b 100644

--- a/ovn/northd/ovn-northd.c

+++ b/ovn/northd/ovn-northd.c

@@ -205,17 +205,16 @@ enum ovn_stage {

 #define REGBIT_ND_RA_OPTS_RESULT "reg0[5]"

 /* Register definitions for switches and routers. */

-#define REGBIT_NAT_REDIRECT     "reg9[0]"

+

 /* Indicate that this packet has been recirculated using egress

  * loopback.  This allows certain checks to be bypassed, such as a

  * logical router dropping packets with source IP address equals

  * one of the logical router's own IP addresses. */

-#define REGBIT_EGRESS_LOOPBACK  "reg9[1]"

-#define REGBIT_DISTRIBUTED_NAT  "reg9[2]"

+#define REGBIT_EGRESS_LOOPBACK  "reg9[0]"

 /* Register to store the result of check_pkt_larger action. */

-#define REGBIT_PKT_LARGER        "reg9[3]"

-#define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[4]"

-#define REGBIT_SKIP_LOOKUP_NEIGHBOR "reg9[5]"

+#define REGBIT_PKT_LARGER        "reg9[1]"

+#define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"

+#define REGBIT_SKIP_LOOKUP_NEIGHBOR "reg9[3]"

 #define FLAGBIT_NOT_VXLAN "flags[1] == 0"

@@ -6599,128 +6598,6 @@ build_routing_policy_flow(struct hmap *lflows, struct ovn_datapath *od,

     ds_destroy(&actions);

 }

-static void

-add_distributed_nat_routes(struct hmap *lflows, const struct ovn_port *op)

-{

-    struct ds actions = DS_EMPTY_INITIALIZER;

-    struct ds match = DS_EMPTY_INITIALIZER;

-

-    if (!op->od->l3dgw_port) {

-        return;

-    }

-

-    if (!op->peer || !op->peer->od->nbs) {

-        return;

-    }

-

-    for (size_t i = 0; i < op->od->nbr->n_nat; i++) {

-        const struct nbrec_nat *nat = op->od->nbr->nat[i];

-        bool found = false;

-        struct eth_addr mac;

-

-        if (strcmp(nat->type, "dnat_and_snat") ||

-                !nat->external_mac ||

-                !eth_addr_from_string(nat->external_mac, &mac) ||

-                !nat->external_ip || !nat->logical_port) {

-            continue;

-        }

-

-        const struct ovn_datapath *peer_dp = op->peer->od;

-        for (size_t j = 0; j < peer_dp->nbs->n_ports; j++) {

-            if (!strcmp(peer_dp->nbs->ports[j]->name, nat->logical_port)) {

-                found = true;

-                break;

-            }

-        }

-        if (!found) {

-            continue;

-        }

-

-        /* Determine if we need to create IPv4 or IPv6 flows */

-        ovs_be32 ip;

-        struct in6_addr ipv6;

-        int family = AF_INET;

-        if (!ip_parse(nat->external_ip, &ip) || !ip) {

-            family = AF_INET6;

-            if (!ipv6_parse(nat->external_ip, &ipv6)) {

-                static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);

-                VLOG_WARN_RL(&rl, "bad ip address %s in nat configuration "

-                             "for router %s", nat->external_ip, op->key);

-                /* We'll create IPv6 flows anyway, but the address

-                 * is probably bogus ... */

-            }

-        }

-

-        ds_put_format(&match, "inport == %s && "

-                      "ip%s.src == %s && ip%s.dst == %s",

-                       op->json_key,

-                       family == AF_INET ? "4" : "6",

-                       nat->logical_ip,

-                       family == AF_INET ? "4" : "6",

-                       nat->external_ip);

-        ds_put_format(&actions, "outport = %s; eth.dst = %s; "

-                      REGBIT_DISTRIBUTED_NAT" = 1; "

-                      REGBIT_NAT_REDIRECT" = 0; next;",

-                      op->od->l3dgw_port->json_key,

-                      nat->external_mac);

-        ovn_lflow_add(lflows, op->od, S_ROUTER_IN_IP_ROUTING, 400,

-                      ds_cstr(&match), ds_cstr(&actions));

-        ds_clear(&match);

-        ds_clear(&actions);

-

-        for (size_t j = 0; j < op->od->nbr->n_nat; j++) {

-            const struct nbrec_nat *nat2 = op->od->nbr->nat[j];

-            struct eth_addr mac2;

-

-            if (nat == nat2 || strcmp(nat2->type, "dnat_and_snat") ||

-                    !nat2->external_mac ||

-                    !eth_addr_from_string(nat2->external_mac, &mac2) ||

-                    !nat2->external_ip) {

-                continue;

-            }

-

-            family = AF_INET;

-            if (!ip_parse(nat2->external_ip, &ip) || !ip) {

-                family = AF_INET6;

-                if (!ipv6_parse(nat2->external_ip, &ipv6)) {

-                    static struct vlog_rate_limit rl =

-                        VLOG_RATE_LIMIT_INIT(5, 1);

-                    VLOG_WARN_RL(&rl, "bad ip address %s in nat configuration "

-                                 "for router %s", nat2->external_ip, op->key);

-                    /* We'll create IPv6 flows anyway, but the address

-                     * is probably bogus ... */

-                }

-            }

-

-            ds_put_format(&match, "inport == %s && "

-                          "ip%s.src == %s && ip%s.dst == %s",

-                          op->json_key,

-                          family == AF_INET ? "4" : "6",

-                          nat->logical_ip,

-                          family == AF_INET ? "4" : "6",

-                          nat2->external_ip);

-            ds_put_format(&actions, "outport = %s; "

-                          "eth.src = %s; eth.dst = %s; "

-                          "%sreg0 = ip%s.dst; %sreg1 = %s; "

-                          REGBIT_DISTRIBUTED_NAT" = 1; "

-                          REGBIT_NAT_REDIRECT" = 0; next;",

-                          op->od->l3dgw_port->json_key,

-                          op->od->l3dgw_port->lrp_networks.ea_s,

-                          nat2->external_mac,

-                          family == AF_INET ? "" : "xx",

-                          family == AF_INET ? "4" : "6",

-                          family == AF_INET ? "" : "xx",

-                          nat->external_ip);

-            ovn_lflow_add(lflows, op->od, S_ROUTER_IN_IP_ROUTING, 400,

-                          ds_cstr(&match), ds_cstr(&actions));

-            ds_clear(&match);

-            ds_clear(&actions);

-        }

-    }

-    ds_destroy(&match);

-    ds_destroy(&actions);

-}

-

 static void

 add_route(struct hmap *lflows, const struct ovn_port *op,

           const char *lrp_addr_s, const char *network_s, int plen,

@@ -8144,17 +8021,6 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,

                     ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 100,

                                   ds_cstr(&match), ds_cstr(&actions));

-

-                    /* Traffic received on other router ports must be

-                     * redirected to the central instance of the l3dgw_port

-                     * for NAT processing. */

-                    ds_clear(&match);

-                    ds_put_format(&match, "ip && ip%s.dst == %s",

-                                  is_v6 ? "6" : "4",

-                                  nat->external_ip);

-                    ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 50,

-                                  ds_cstr(&match),

-                                  REGBIT_NAT_REDIRECT" = 1; next;");

                 }

             }

@@ -8220,18 +8086,33 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,

                     ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 100,

                                   ds_cstr(&match), ds_cstr(&actions));

+                }

+            }

-                    /* Traffic received on other router ports must be

-                     * redirected to the central instance of the l3dgw_port

-                     * for NAT processing. */

+            /* ARP resolve for NAT IPs. */

+            if (od->l3dgw_port) {

+                if (!strcmp(nat->type, "snat")) {

                     ds_clear(&match);

-                    ds_put_format(&match, "ip && ip%s.dst == %s",

-                                  is_v6 ? "6" : "4",

-                                  nat->external_ip);

-                    ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 50,

-                                  ds_cstr(&match),

-                                  REGBIT_NAT_REDIRECT" = 1; next;");

+                    ds_put_format(

+                        &match, "inport == %s && %s == %s",

+                        od->l3dgw_port->json_key,

+                        is_v6 ? "ip6.src" : "ip4.src", nat->external_ip);

+                    ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_INPUT, 120,

+                                  ds_cstr(&match), "next;");

                 }

+

+                ds_clear(&match);

+                ds_put_format(

+                    &match, "outport == %s && %s == %s",

+                    od->l3dgw_port->json_key,

+                    is_v6 ? "xxreg0" : "reg0", nat->external_ip);

+                ds_clear(&actions);

+                ds_put_format(

+                    &actions, "eth.dst = %s; next;",

+                    distributed ? nat->external_mac :

+                    od->l3dgw_port->lrp_networks.ea_s);

+                ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_RESOLVE, 100,

+                                ds_cstr(&match), ds_cstr(&actions));

             }

             /* Egress UNDNAT table: It is for already established connections'

@@ -8378,49 +8259,19 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,

              * ingress pipeline with inport = outport. */

             if (od->l3dgw_port) {

                 /* Distributed router. */

-                if (!strcmp(nat->type, "dnat_and_snat") &&

-                        nat->external_mac && nat->external_ip &&

-                        eth_addr_from_string(nat->external_mac, &mac)) {

-                    for (int j = 0; j < od->nbr->n_nat; j++) {

-                        const struct nbrec_nat *nat2 = od->nbr->nat[j];

-

-                        if (nat2 == nat ||

-                            strcmp(nat2->type, "dnat_and_snat") ||

-                            !nat2->external_mac || !nat2->external_ip) {

-                            continue;

-                        }

-

-                        ds_clear(&match);

-                        ds_put_format(&match, "is_chassis_resident(\"%s\") && "

-                                      "ip%s.src == %s && ip%s.dst == %s",

-                                      nat->logical_port,

-                                      is_v6 ? "6" : "4", nat2->external_ip,

-                                      is_v6 ? "6" : "4", nat->external_ip);

-                        ds_clear(&actions);

-                        ds_put_format(&actions,

-                                      "inport = outport; outport = \"\"; "

-                                      "flags = 0; flags.loopback = 1; "

-                                      REGBIT_EGRESS_LOOPBACK" = 1; "

-                                      "next(pipeline=ingress, table=0); ");

-                        ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 300,

-                                      ds_cstr(&match),  ds_cstr(&actions));

-

-                        ds_clear(&match);

-                        ds_put_format(&match,

-                                      "ip%s.src == %s && ip%s.dst == %s",

-                                      is_v6 ? "6" : "4", nat2->external_ip,

-                                      is_v6 ? "6" : "4", nat->external_ip);

-                        ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 200,

-                                      ds_cstr(&match), "next;");

-                        ds_clear(&match);

-                    }

-                }

-

                 ds_clear(&match);

                 ds_put_format(&match, "ip%s.dst == %s && outport == %s",

                               is_v6 ? "6" : "4",

                               nat->external_ip,

                               od->l3dgw_port->json_key);

+                if (!distributed) {

+                    ds_put_format(&match, " && is_chassis_resident(%s)",

+                                  od->l3redirect_port->json_key);

+                } else {

+                    ds_put_format(&match, " && is_chassis_resident(\"%s\")",

+                                  nat->logical_port);

+                }

+

                 ds_clear(&actions);

                 ds_put_format(&actions,

                               "clone { ct_clear; "

@@ -8491,40 +8342,6 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,

             * we can do it here, saving a future re-circulation. */

             ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 50,

                           "ip", "flags.loopback = 1; ct_dnat;");

-        } else {

-            ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_RESOLVE, 400,

-                          REGBIT_DISTRIBUTED_NAT" == 1", "next;");

-

-            /* For NAT on a distributed router, add flows to Ingress

-             * IP Routing table, Ingress ARP Resolution table, and

-             * Ingress Gateway Redirect Table that are not specific to a

-             * NAT rule. */

-

-            /* The highest priority IN_IP_ROUTING rule matches packets

-             * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages),

-             * with action "ip.ttl--; next;".  The IN_GW_REDIRECT table

-             * will take care of setting the outport. */

-            ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING, 300,

-                          REGBIT_NAT_REDIRECT" == 1", "ip.ttl--; next;");

-

-            /* The highest priority IN_ARP_RESOLVE rule matches packets

-             * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages),

-             * then sets eth.dst to the distributed gateway port's

-             * ethernet address. */

-            ds_clear(&actions);

-            ds_put_format(&actions, "eth.dst = %s; next;",

-                          od->l3dgw_port->lrp_networks.ea_s);

-            ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_RESOLVE, 200,

-                          REGBIT_NAT_REDIRECT" == 1", ds_cstr(&actions));

-

-            /* The highest priority IN_GW_REDIRECT rule redirects packets

-             * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages) to

-             * the central instance of the l3dgw_port for NAT processing. */

-            ds_clear(&actions);

-            ds_put_format(&actions, "outport = %s; next;",

-                          od->l3redirect_port->json_key);

-            ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 200,

-                          REGBIT_NAT_REDIRECT" == 1", ds_cstr(&actions));

         }

         /* Load balancing and packet defrag are only valid on

@@ -8720,9 +8537,6 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,

             continue;

         }

-        /* create logical flows for DVR floating IPs */

-        add_distributed_nat_routes(lflows, op);

-

         for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {

             add_route(lflows, op, op->lrp_networks.ipv4_addrs[i].addr_s,

                       op->lrp_networks.ipv4_addrs[i].network_s,

@@ -9252,9 +9066,6 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,

             continue;

         }

         if (od->l3dgw_port && od->l3redirect_port) {

-            ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 300,

-                          REGBIT_DISTRIBUTED_NAT" == 1", "next;");

-

             /* For traffic with outport == l3dgw_port, if the

              * packet did not match any higher priority redirect

              * rule, then the traffic is redirected to the central

diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at

index da566f900..3e4120ac5 100644

--- a/tests/ovn-northd.at

+++ b/tests/ovn-northd.at

@@ -990,7 +990,7 @@ echo "CR-LRP UUID is: " $uuid

 # IPV4

 ovn-nbctl lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11

-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

+OVS_WAIT_UNTIL([test 2 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

 wc -l`])

 AT_CHECK([ovn-sbctl dump-flows R1 | grep ct_snat | wc -l], [0], [2

@@ -1008,7 +1008,7 @@ AT_CHECK([ovn-sbctl dump-flows R1 | grep ip4.src=| wc -l], [0], [0

 ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1

 ovn-nbctl --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11

-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

+OVS_WAIT_UNTIL([test 2 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

 wc -l`])

 AT_CHECK([ovn-sbctl dump-flows R1 | grep ct_snat | wc -l], [0], [0

@@ -1027,7 +1027,7 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1

 # IPV6

 ovn-nbctl lr-nat-add R1 dnat_and_snat fd01::1 fd11::2

-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

+OVS_WAIT_UNTIL([test 2 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

 wc -l`])

 AT_CHECK([ovn-sbctl dump-flows R1 | grep ct_snat | wc -l], [0], [2

@@ -1045,7 +1045,7 @@ AT_CHECK([ovn-sbctl dump-flows R1 | grep ip6.src=| wc -l], [0], [0

 ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1

 ovn-nbctl --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2

-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

+OVS_WAIT_UNTIL([test 2 = `ovn-sbctl dump-flows R1 | grep lr_in_unsnat | \

 wc -l`])

 AT_CHECK([ovn-sbctl dump-flows R1 | grep ct_snat | wc -l], [0], [0

-- 

2.24.1