File Based Dynamic Routing Configuration by envoyproxy

Envoy Dynamic Configuration

In the previous scenarios we've defined the static configuration. However this has made it difficult to reload the configuration when changes are required. To solve this, the static configuration can be defined as Dynamic Configuration. With Dynamic Configuration, when changes are made, Envoy will automatically reload the changes and apply them to the configuration and traffic routing.

Envoy supports different parts of the configuration as dynamic. The APIs available are:

EDS: The Endpoint Discovery Service (EDS) API provides a way Envoy can discover members of an upstream cluster. This allows you to dynamically add and remove servers handling the traffic.
CDS: The Cluster Discovery Service (CDS) API layers on a mechanism by which Envoy can discover upstream clusters used during routing.
RDS: The Route Discovery Service (RDS) API layers on a mechanism by which Envoy can discover the entire route configuration for an HTTP connection manager filter at runtime. This would enable concepts such as dynamically changing traffic shifting and blue/green releases.
LDS: The Listener Discovery Service (LDS) layers on a mechanism by which Envoy can discover entire listeners at runtime.
SDS: The Secret Discovery Service (SDS) layers on a mechanism by which Envoy can discover cryptographic secrets (certificate plus private key, TLS session ticket keys) for its listeners, as well as the configuration of peer certificate validation logic (trusted root certs, revocations, etc).

The value for configuration can come from the filesystem, REST-JSON or gRPC endpoints.

More information can be found in the Envoy documentation overview

In the next steps, we'll change our configuration to use Endpoint Discovery Service (EDS) allowing nodes to be dynamically added based with data coming from the filesystem.

Step 2 - Cluster ID

An initial outline of the Envoy configuration required is available at envoy.yaml

The first changes required is to add a Node. This allows the Envoy node to be identified, potentially allowing for unique configurations to be applied.

Task

Prepend the following snippet to the top of the envoy.yaml file.

node:
  id: id_1
  cluster: test

The API also has support for additional metadata, such as locality for providing region and zone-based information.

Step 3 - EDS Configuration

The EDS configuration is defined to allow the upstream clusters to be controlled dynamically.

Within the static configuration, this was defined as:

  clusters:
  - name: targetCluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts: [
      { socket_address: { address: 172.18.0.3, port_value: 80 }},
      { socket_address: { address: 172.18.0.4, port_value: 80 }}
    ]

Convert to EDS

To convert this to EDS based a eds_cluster_config is required and changing the type to EDS.

Add the following cluster to the end of the Envoy configuration.

  clusters:
  - name: targetCluster
    connect_timeout: 0.25s
    lb_policy: ROUND_ROBIN
    type: EDS
    eds_cluster_config:
      service_name: localservices
      eds_config:
        path: '/etc/envoy/eds.conf'

The values for the upstream servers, such as 172.18.0.3 and 172.18.0.4, will come from the file eds.conf.

Step 4 - EDS Configuration

The contents of eds.conf is a JSON definition of the same information defined within our static configuration.

Task

Create eds.conf file with the following content:

{
  "version_info": "0",
  "resources": [{
    "@type": "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment",
    "cluster_name": "localservices",
    "endpoints": [{
      "lb_endpoints": [{
        "endpoint": {
          "address": {
            "socket_address": {
              "address": "172.18.0.3",
              "port_value": 80
            }
          }
        }
      }]
    }]
  }]
}

This defines a single endpoint at 172.18.0.3.

Step 5 - Start Envoy

With the Envoy configuration and EDS conf defined, it's now possible to start the proxy.

Task

Launch the container with the following command:

docker run --name=proxy-eds-filebased -d \
    -p 9901:9901 \
    -p 80:10000 \
    -v /root/:/etc/envoy \
    envoyproxy/envoy:latest

Start two HTTP servers to handle the incoming requests.

docker run -d katacoda/docker-http-server; docker run -d katacoda/docker-http-server;

Based on the current EDS configuration, Envoy will send all the traffic to a single node. Try it with curl localhost

In the next step we'll update the configuration and add an additional node.

Step 6 - Apply Changes

With the configuration based on EDS, when the services need to be scaled up a new endpoint can be added to the eds.conf. Envoy will then automatically include the changes.

Task

Replace the configuration with the following to add a new endpoint to the cluster.

{
  "version_info": "0",
  "resources": [{
    "@type": "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment",
    "cluster_name": "localservices",
    "endpoints": [{
      "lb_endpoints": [{
        "endpoint": {
          "address": {
            "socket_address": {
              "address": "172.18.0.3",
              "port_value": 80
            }
          }
        }
      },
        {
        "endpoint": {
          "address": {
            "socket_address": {
              "address": "172.18.0.4",
              "port_value": 80
            }
          }
        }
      }]
    }]
  }]
}

Based on how Docker handles file inode tracking, sometimes the filesystem change isn't triggered and detected. Force the change with the command mv eds.conf tmp; mv tmp eds.conf

Envoy should automatically reload the configuration and add the new node into the load balancing rotation curl localhost

Step 7 - CDS Configuration

With EDS in place, it's possible to move to scale up the upstream clusters. If we wanted to be able to dynamically add new domains and clusters, the Cluster Discovery Service (CDS) API needs to be implemented. In the following steps, we are configuring the Cluster Discovery Service (CDS) and The Listener Discovery Service (LDS).

You need to create a file to put the configuration for the clusters: cds.conf.

{
  "version_info": "0",
  "resources": [{
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "targetCluster",
            "connect_timeout": "0.25s",
            "lb_policy": "ROUND_ROBIN",
            "type": "EDS",
            "eds_cluster_config": {
                "service_name": "localservices",
                "eds_config": {
                    "path": "/etc/envoy/eds.conf"
                }
            }
  }]
}

And also, you need to create a file to put the configuration for the listeners: lds.conf.

{
    "version_info": "0",
    "resources": [{
            "@type": "type.googleapis.com/envoy.api.v2.Listener",
            "name": "listener_0",
            "address": {
                "socket_address": {
                    "address": "0.0.0.0",
                    "port_value": 10000
                }
            },
            "filter_chains": [
                {
                    "filters": [
                        {
                            "name": "envoy.http_connection_manager",
                            "config": {
                                "stat_prefix": "ingress_http",
                                "codec_type": "AUTO",
                                "route_config": {
                                    "name": "local_route",
                                    "virtual_hosts": [
                                        {
                                            "name": "local_service",
                                            "domains": [
                                                "*"
                                            ],
                                            "routes": [
                                                {
                                                    "match": {
                                                        "prefix": "/"
                                                    },
                                                    "route": {
                                                        "cluster": "targetCluster"
                                                    }
                                                }
                                            ]
                                        }
                                    ]
                                },
                                "http_filters": [
                                    {
                                        "name": "envoy.router"
                                    }
                                ]
                            }
                        }
                    ]
                }
            ]
    }]
}

The content of files cds.conf and lds.conf is a JSON definition of with the same information defined within our static configuration.

With the externalized the configuration of clusters and listeners, you need to modify your Envoy's configuration to make reference to these files. This can be accomplish changing all the static_resources for dynamic_resources.

Open the Envoy configuration file envoy1.yaml, and add the following configuration:

dynamic_resources:
  cds_config:
    path: "/etc/envoy/cds.conf"
  lds_config:
    path: "/etc/envoy/lds.conf"

After that, launch the container with the following command:

docker run --name=proxy-eds-cds-lds-filebased -d \
    -p 9902:9901 \
    -p 81:10000 \
    -v /root/:/etc/envoy \
    -v /root/envoy1.yaml:/etc/envoy/envoy.yaml \
    envoyproxy/envoy:latest

Note: to avoid port conflicts, we exposed the ports with offset 1.

Execute the following command: curl localhost:81

Step 8 - CDS Apply Changes

With the configuration based on CDS, LDS and EDS, we can dynamically add a new cluster.

Open the file cds.conf.

Add new cluster

We'll call this new cluster newTargetCluster. Replace the configuration with the following to add a new cluster.

{
  "version_info": "0",
  "resources": [{
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "targetCluster",
            "connect_timeout": "0.25s",
            "lb_policy": "ROUND_ROBIN",
            "type": "EDS",
            "eds_cluster_config": {
                "service_name": "localservices",
                "eds_config": {
                    "path": "/etc/envoy/eds.conf"
                }
            }
  },
  {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "newTargetCluster",
            "connect_timeout": "0.25s",
            "lb_policy": "ROUND_ROBIN",
            "type": "EDS",
            "eds_cluster_config": {
                "service_name": "localservices",
                "eds_config": {
                    "path": "/etc/envoy/eds1.conf"
                }
            }
  }]
}

You also need to create the eds_cluster_config file for this new cluster. Create the file eds1.conf with this content:

{
  "version_info": "0",
  "resources": [{
    "@type": "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment",
    "cluster_name": "localservices",
    "endpoints": [{
      "lb_endpoints": [{
        "endpoint": {
          "address": {
            "socket_address": {
              "address": "172.18.0.6",
              "port_value": 80
            }
          }
        }
      },
        {
        "endpoint": {
          "address": {
            "socket_address": {
              "address": "172.18.0.7",
              "port_value": 80
            }
          }
        }
      }]
    }]
  }]
}

And you can use this new cluster, in the listener that you previously configured. Open the file lds.conf. Replace the target cluster with the name of the new cluster newTargetCluster.

  "route": {
      "cluster": "newTargetCluster"
  }

The configuration of lds.conf should look like:

...
            "filter_chains": [
                {
                    "filters": [
                        {
                            "name": "envoy.http_connection_manager",
                            "config": {
                                "stat_prefix": "ingress_http",
                                "codec_type": "AUTO",
                                "route_config": {
                                    "name": "local_route",
                                    "virtual_hosts": [
                                        {
                                            "name": "local_service",
                                            "domains": [
                                                "*"
                                            ],
                                            "routes": [
                                                {
                                                    "match": {
                                                        "prefix": "/"
                                                    },
                                                    "route": {
                                                       "cluster": "newTargetCluster"
                                                    }
                                                }
                                            ]
                                        }
                                    ]
                                },
                                "http_filters": [
                                    {
                                        "name": "envoy.router"
                                    }
                                ]
                            }
                        }
                    ]
                }
            ]
...

Start two HTTP servers to handle the incoming requests for the new cluster docker run -d katacoda/docker-http-server; docker run -d katacoda/docker-http-server;

Based on how Docker handles file inode tracking, sometimes the filesystem change isn't triggered and detected. Force the change with the command: mv cds.conf tmp; mv tmp cds.conf; mv lds.conf tmp; mv tmp lds.conf

Envoy should automatically reload the configuration and add the new cluster. You can try running the following command: curl localhost:81.

You can notice with the response of each request, that the ID of the nodes changes, corresponding to the nodes of newTargetCluster.

cd <directory>	Change directory
ls	List directory
echo 'contents' > <file>	Write contents to a file
cat <file>	Output contents of file

i	In Command Mode	Change into Insert Mode, you can now insert and edit text in the file
esc	In Insert Mode	Change into Command Mode, you can now execute commands
:wq	In Command Mode	Save and Exit

Welcome!

File Based Dynamic Routing Configuration

Congratulations!

You've completed the scenario!

Scenario Rating

Steps