Install MetalLB

MetalLB runs in two parts: a cluster-wide controller, and a per-machine BGP speaker. Since Minikube is a Kubernetes cluster with a single VM, we'll end up with the controller and one BGP speaker.

Install MetalLB by applying the manifest:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/master/manifests/metallb.yaml

This manifest creates a bunch of resources. Most of them are related to access control, so that MetalLB can read and write the Kubernetes objects it needs to do its job.

Ignore those bits for now, the two pieces of interest are the "controller" deployment, and the "speaker" daemonset.

Wait for these to start by monitoring kubectl get pods -n metallb-system.

Eventually, you should see two running pods, in addition to the BGP router from the previous step (again, the pod name suffixes will be different on your cluster).

Refresh the test-bgp-router-ui tab from earlier, and... It's the same! MetalLB is still not connected, and our routers still know nothing about cluster services.

That's because the MetalLB installation manifest doesn't come with a configuration, so both the controller and BGP speaker are sitting idle, waiting to be told what they should do. Let's fix that!

Configure MetalLB

We have a sample MetalLB configuration in manifests/tutorial-1.yaml. Let's take a look at it before applying it:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - my-asn: 64500
      peer-asn: 64500
      peer-address: 10.96.0.100
    - my-asn: 64500
      peer-asn: 64500
      peer-address: 10.96.0.101
    address-pools:
    - name: my-ip-space
      protocol: bgp
      addresses:
      - 198.51.100.0/24

MetalLB's configuration is a standard Kubernetes ConfigMap, config under the metallb-system namespace. It contains two pieces of information: who MetalLB should talk to, and what IP addresses it's allowed to hand out.

In this configuration, we're setting up a BGP peering with 10.96.0.100, 10.96.0.101, 10.96.0.102, which are the addresses of the test-bgp-router-bird, and test-bgp-router-quagga services respectively. And we're giving MetalLB 256 IP addresses to use, from 198.51.100.0 to 198.51.100.255. The final section gives MetalLB some BGP attributes that it should use when announcing IP addresses to our router.

Apply this configuration now:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/master/manifests/tutorial-1.yaml

The configuration should take effect within a few seconds. Refresh the test-bgp-router-ui browser page again. If all went well, you should see happier routers.

Success! The MetalLB BGP speaker connected to our routers. You can verify this by looking at the logs for the BGP speaker.

POD=$(kubectl -n metallb-system get pods | grep speaker | awk '{print $1}')
kubectl -n metallb-system logs pods/$POD | grep established

However, as the BGP routers pointed out, MetalLB is connected, but isn't telling them about any services yet. That's because all the services we've defined so far are internal to the cluster. Let's change that!

Create a load-balanced service

manifests/tutorial-2.yaml contains a trivial service: an nginx pod, and a load-balancer service pointing at nginx. Deploy it to the cluster now:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/master/manifests/tutorial-2.yaml

Again, wait for nginx to start by monitoring kubectl get pods, until you see a running nginx pod.

Once it's running, take a look at the nginx service with kubectl get service nginx:

We have an external IP! Because the service is of type LoadBalancer, MetalLB took 198.51.100.0 from the address pool we configured, and assigned it to the nginx service. You can see this even more clearly by looking at the event history for the service, with

kubectl describe service nginx :

  Type    Reason          Age   From                Message
  ----    ------          ----  ----                -------
  Normal  IPAllocated     24m   metallb-controller  Assigned IP "198.51.100.0"

Refresh your test-bgp-router-ui page, and see what our routers thinks.

Success! MetalLB told our routers that 198.51.100.0 exists on our Cluster, and that the routers should forward any traffic for that IP to us.

Edit MetalLB's configuration

In the previous step, MetalLB assigned the address 198.51.100.0, the first address in the pool we gave it. That IP address is perfectly valid, but some old and buggy wifi routers mistakenly think it isn't, because it ends in .0.

As it turns out, one of our customers called and complained of this exact problem. Fortunately, MetalLB has a configuration option to address this. Take a look at the configuration in manifests/tutorial-3.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - my-asn: 64500
      peer-asn: 64500
      peer-address: 10.96.0.100
    - my-asn: 64500
      peer-asn: 64500
      peer-address: 10.96.0.101
    address-pools:
    - name: my-ip-space
      protocol: bgp
      avoid-buggy-ips: true
      addresses:
      - 198.51.100.0/24

There's just one change compared to our previous configuration: in the address pool configuration, we added avoid-buggy-ips: true. This tells MetalLB that IP addresses ending in .0 or .255 should not be assigned.

Sounds easy enough, let's apply that configuration:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/master/manifests/tutorial-3.yaml

Refresh the test-bgp-router-ui page and... Hmm, strange, our routers are still being told to use 198.51.100.0, even though we just told MetalLB that this address should not be used. What happened?

To answer that, let's inspect the running configuration in Kubernetes, by running :

kubectl describe configmap -n metallb-system config

At the bottom of the output, you should see an event log that looks like this:

Events:
  Type     Reason         Age   From                 Message
  ----     ------         ----  ----                 -------
  Warning  InvalidConfig  29s   metallb-controller   configuration rejected: new config not compatible with assigned IPs: service "default/nginx" cannot own "198.51.100.0" under new config
  Warning  InvalidConfig  29s   metallb-speaker      configuration rejected: new config not compatible with assigned IPs: service "default/nginx" cannot own "198.51.100.0" under new config

Oops! Both the controller and the BGP speaker rejected our new configuration, because it would break an already existing service. This illustrates an important policy that MetalLB tries to follow: applying new configurations should not break existing services.

(You might ask why we were able to apply an invalid configuration at all. Good question! This is a missing feature of MetalLB. In future, MetalLB will validate new configurations when they are submitted by kubectl, and make Kubernetes refuse unsafe configurations. But for now, it will merely complain after the fact, and ignore the new configuration.)

At this point, MetalLB is still running on the previous configuration, the one that allows nginx to use the IP it currently has. If this were a production cluster with Prometheus monitoring, we would be getting an alert now, warning us that the configmap written to the cluster is not compatible with the cluster's running state.

Okay, so how do we fix this? We need to explicitly change the configuration of the nginx service to be compatible with the new configuration. To do this, run kubectl edit service nginx

and in the spec section add: loadBalancerIP: 198.51.100.1.

Save the change, and run kubectl describe service nginx again. You should see an IPAllocated event showing that MetalLB changed the service's assigned address as instructed.

Now, the new configuration that we tried to apply is valid, because nothing is using the .0 address any more. Let's reapply it, so that MetalLB reloads again:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/master/manifests/tutorial-4.yaml

You may have noticed that we applied tutorial-4.yaml, not the tutorial-3.yaml from before. This is another rough edge in the current version of MetalLB: when we submitted the configuration in tutorial-3.yaml, MetalLB looked at it and rejected it, but will not look at it again to see if it has become valid. To make MetalLB examine the configuration again, we need to make some cosmetic change to the config, so that Kubernetes notifies MetalLB that there is a new configuration to load. tutorial-4.yaml just adds a no-op comment to the configuration to make Kubernetes signal MetalLB.

This piece of clunkiness will also go away when MetalLB learns to validate new configurations before accepting the submission from kubectl.

This time, MetalLB accepts the new configuration, and everything is happy once again. And, refreshing test-bgp-router-ui, we see that the routers did indeed see the change from .0 to .1.

One final bit of clunkiness: right now, you need to inspect metallb's logs to see that a new configuration was successfully loaded. Once MetalLB only allows valid configurations to be submitted, this clunkiness will also go away.