Welcome!
Portworx Troubleshooting lab
Portworx Volumes Lab. Explore existing volumes, create and update new volumes.
In this lab, we will explore the learn how to show and inspect volumes using pxctl. We will then create new volumes, attach them to a host and copy data in to the volume.
Congratulations!
You've completed the scenario!
Scenario Rating
Click on "Complete and continue" to proceed to the next lecture.
Your environment is currently being packaged as a Docker container and the download will begin shortly. To run the image locally, once Docker has been installed, use the commands
cat scrapbook_portworx_portworx-troubleshooting_container.tar | docker loaddocker run -it /portworx_portworx-troubleshooting:Oops!! Sorry, it looks like this scenario doesn't currently support downloads. We'll fix that shortly.
Steps
Portworx Troubleshooting lab
Lab Info
In this lab we will troubleshoot and fix issues with Stateful applications.
Please wait for the environment to load.
Here are some handy links that should help with the troubleshooting:
Please wait for the Portworx pods to be Ready.
watch kubectl -n kube-system get pods -l name=portworx
To get back to the terminal:
clear
Inspect PVC in Alpha Namespace
We have created a deployment in the alpha namespace. However, the pods have not been created yet.
Inspect them and find out why.
Why is the Deployment pending?
Why are the PVC's in a pending state?
Check the PVC status by running:
kubectl -n alpha describe pvc px-alpha-pvc
Does it use a valid Storage Class?
Fix the issues
Let us fix that. To do this, delete and recreate the PVC used by mysql with the correct storage class name.
StorageClass Name: px-alpha-sc
provisioner = portworx volume
io_priority = high
repl = 1
Note: Please wait for the pod of the mysql deployment to be Ready before proceeding to the next question.
Delete the existing PVC in the alpha namespace and recreate it with the correct storage class name.
kubectl -n alpha delete pvc px-alpha-pvc ; kubectl apply -f /root/alpha-pvc.yaml
Inspect Application in Beta Namespace
We have created a deployment in the beta namespace. However, the pods are again in a pending state.
Inspect and find the issue.
Why is the Deployment pending?
Why are the PVC's in a pending state now?
Check the PVC status by running:
kubectl -n beta describe pvc px-beta-pvc
What error do you see?
Fix the issues
Let us fix that. To do this, re-create the storage class with the auth secret.
name: px-beta-sc
provisioner: portworx volume
io_priority: high
repl: 3
io profile: db
openstorage.io/auth-secret-name: px-k8s-user
openstorage.io/auth-secret-namespace: portworx
Note: Please wait for the pod of the mysql deployment to be Ready before proceeding to the next question.
Delete the existing StorageClass px-beta-sc and recreate it with the correct parameters:
kubectl delete sc px-beta-sc ; kubectl apply -f /root/beta-sc.yaml
Inspect Application in Gamma Namespace
We have created a deployment in the gamma namespace. However, the pods again have not been scheduled.
Inspect the PVC.
Why is the PVC pending?
Why are the PVC's in a pending state now?
Check the PVC status by running:
kubectl -n gamma describe pvc px-gamma-pvc
What error do you see?
Update the Storage Class
The StorageClass px-gamma-sc does not have enough replicas to use a db io_profile. Let us fix that. To do this, delete and recreate the px-gamma-sc storage class with the correct parameters:
StorageClass Name: px-gamma-sc
provisioner = portworx volume
io_priority = high
io_profile = db
repl = 3
Once, you have recreated the StorageClass, click on Continue.
Delete the existing StorageClass px-gamma-sc and recreate it with the correct parameters:
kubectl delete sc px-gamma-sc ; kubectl apply -f /root/gamma-sc.yaml
Recreate the PVC if needed.
Why is Deployment still failing?
The PVC is still in a pending state. Inspect the PVC again and check the events to find out why.
Note: It may take a couple of minutes for the new events to be displayed.
Why are the PVC's in a pending state now?
Check the PVC status by running:
kubectl -n gamma describe pvc px-gamma-pvc
What error do you see?
Troubleshoot
Looks like one of the PX Nodes is Down! Identify the reason and fix it.
Hint: Check the cluster status and find out which node is down. SSH to that node and check the Portworx Service Status. Use journalctl -u portworx*.
Check the PX Cluster status from one of the nodes:
ssh node01 pxctl status
If you see that the node03 is Down, SSH to node03 and check the status of PX:
ssh node03
ssh node03 systemctl status portworx
Check for errors using journalctl:
journalctl -u portworx*
Note: We introduced an incorrect ETCD endpoint in the file /etc/pwx/config.json. You can see this by checking the port number for the etcd and compare it against the Portworx Deployment.
The OCI Monitor container should automatically detect and correct this. However, this is not happening as the docker service is down. As a result the Portworx service is broken on node03.
Start the docker service and check the journalctl logs again to see if the service is starting properly.
If that does not work, delete the portworx pod (of the portworx DaemonSet) running on node03. Wait for it to be re-created and in a Ready state.
Once this is done, all nodes should be online.
Validate
Now that all 3 PX nodes are online, check the status of the PVC and Pods in the gamma namespace. The PVC and Pods should have been created.
kubectl -n gamma get pvc,pods
