By default, recommendation v1 and v2 are being randomly load-balanced as that is the default behavior in Kubernetes/OpenShift

If you look for all recommendations pods that contains the label app=recommendation, you will find v1 and v2.

Try: oc get pods -l app=recommendation -n tutorial

To check the random load-balance, send several requests to the microservices on Terminal 2 to see their responses while true; do curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

Now check the files /istiofiles/destination-rule-recommendation.yml and /istiofiles/virtual-service-recommendation-503.yml.

Note that the VirtualService provides httpFault that will abort the request 50% of the time with a httpStatus=503 for the subset: app-recommendation.

Let's apply these files: istioctl create -f ~/projects/istio-tutorial/istiofiles/destination-rule-recommendation.yml -n tutorial; \ istioctl create -f ~/projects/istio-tutorial/istiofiles/virtual-service-recommendation-503.yml -n tutorial

To check the new behavior, make sure that the following command is running on Terminal 2: while true; do time curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

NOTE: It might take a couple of seconds until you see the new behavior

Clean up

Don't forget to remove the virtualservice and destinationrule executing ~/projects/istio-tutorial/scripts/clean.sh

To check if you have random load-balance without 503's, try the microservice on Terminal 2: while true; do curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

The most insidious of possible distributed computing faults is not a "down" service but a service that is responding slowly, potentially causing a cascading failure in your network of services.

Check the file /istiofiles/virtual-service-recommendation-delay.yml. and the file /istiofiles/destination-rule-recommendation.yml.

The VirtualService provides httpFault that will delay the request 50% of the time with a fixedDelay=7s.

Let's apply these files:

istioctl create -f ~/projects/istio-tutorial/istiofiles/virtual-service-recommendation-delay.yml -n tutorial; \ istioctl create -f ~/projects/istio-tutorial/istiofiles/destination-rule-recommendation.yml -n tutorial

To check the new behavior, send several requests to the microservices on Terminal 2 to see their responses while true; do time curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

You will notice many requets to the customer endpoint now have a delay of seven seconds.

NOTE: It might take a couple of seconds until you see the new behavior

Clean up

Don't forget to remove the virtualservice and destinationrule executing ~/projects/istio-tutorial/scripts/clean.sh

To check if you have random load-balance without delays, try the microservice on Terminal 2: while true; do time curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

Instead of failing immediately, retry the Service N more times

We will make pod recommendation-v2 fail 100% of the time. Get one of the pod names from your system and replace on the following command accordingly:

oc exec -it $(oc get pods|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash

You will be inside the application container of your pod recommendation-v2-2036617847-spdrb. Now execute the following command inside the recommenation-v2 pod:

curl localhost:8080/misbehave

Now exit from the recommendation-v2 pod:

exit

This is a special endpoint that will make our application return only 503s.

Now, if you hit the customer endpoint several times, you should see some 503’s

while true; do curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

Now, check the file istiofiles/virtual-service-recommendation-v2_retry.yml.

Note that this VirtualService provides simpleRetry that perform 3 attemps on recommendation, using a timeout of 2 seconds per try.

Let's apply this VirtualService:

istioctl create -f ~/projects/istio-tutorial/istiofiles/virtual-service-recommendation-v2_retry.yml -n tutorial

and after a few seconds, things will settle down and you will see it work every time.

To check this behavior, send several requests to the microservices on Terminal 2 to see their responses:

while true; do time curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

NOTE: You will see it work every time because Istio will retry the recommendation service and it will land on v1 only.

You can see the active Virtual Services via: istioctl get virtualservices -n tutorial

Clean up

Now, delete the retry rule and see the old behavior on Terminal 2, where v2 throws 503s: istioctl delete virtualservice recommendation -n tutorial

Now, make the pod v2 behave well again. Get one of the pod names from your system and replace on the following command accordingly:

oc exec -it $(oc get pods|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash

Now execute the following command inside the recommenation-v2 pod:

curl localhost:8080/behave

Now exit from the recommendation-v2 pod:

exit

The application is back to random load-balancing between v1 and v2. Check it at Terminal 2: while true; do curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

Wait only N seconds before giving up and failing.

First, introduce some wait time in recommendations v2 by uncommenting the line 48 that call the timeout method. This method, will cause a wait time of 3 seconds. Update /recommendation-v2/src/main/java/com/redhat/developer/demos/recommendation/RecommendationVerticle.java making it a slow perfomer.

       @Override
       public void start() throws Exception {
           Router router = Router.router(vertx);
           router.get("/").handler(this::timeout);
           router.get("/").handler(this::logging);
           router.get("/").handler(this::getRecommendations);
//         router.get("/").handler(this::getNow);
           router.get("/misbehave").handler(this::misbehave);
           router.get("/behave").handler(this::behave);

           HealthCheckHandler hc = HealthCheckHandler.create(vertx);
           hc.register("dummy-health-check", future -> future.complete(Status.OK()));
           router.get("/health").handler(hc);

           vertx.createHttpServer().requestHandler(router::accept).listen(LISTEN_ON);
       }

Note: The file is saved automatically.

Rebuild and redeploy the recommendation microservices.

Go to the recommendation folder cd ~/projects/istio-tutorial/recommendation-v2/

Compile the project with the modifications that you did.

mvn package

Execute docker build -t example/recommendation:v2 .

You can check the image that was create by typing docker images | grep recommendation

Now let's delete the previous v2 pod to force the creation of a new pod using the new image.

oc delete pod -l app=recommendation,version=v2 -n tutorial

To watch the creation of the pods, execute oc get pods -w

Once that the recommendation pods READY column are 2/2, you can hit CTRL+C.

Check Terminal 2 and make sure that you can see v2 responding in 3 seconds: while true; do time curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

Timeout rule

Check the file /istiofiles/virtual-service-recommendation-timeout.yml.

Note that this VirtualService provides a timeout of 1 second.

Let's apply this rule: istioctl create -f ~/projects/istio-tutorial/istiofiles/virtual-service-recommendation-timeout.yml -n tutorial

You should see it return v1 OR 504 upstream request timeout after waiting about 1 second, although v2 takes 3 seconds to complete.

To check this behavior, send several requests to the microservices on Terminal 2 to see their responses while true; do time curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done

Clean up

To remove the Timeout behavior, simply delete this VirtualService by executing istioctl delete -f ~/projects/istio-tutorial/istiofiles/virtual-service-recommendation-timeout.yml -n tutorial

To check if you have random load-balance with v2 replying in 3 seconds, try the microservice on Terminal 2: while true; do time curl http://customer-tutorial.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com; sleep .5; done