Chaos Engineering

1. Login Chaos Mesh Dashboard
2. Start from a simple chaos experiment
3. Use workflow to orchestrate multiple chaos experiments

In out experiments, we use Chaos Mesh to inject chaos into the database clusters. Chaos Mesh is a cloud-native Chaos Engineering platform that orchestrates chaos on Kubernetes environments. In k8s cluster setup, we have already installed Chaos Mesh. Note that here we suppose that you have already installed Chaos Mesh. In this section, we will introduce how to use Chaos Mesh to inject chaos into the TiDB cluster.

Login Chaos Mesh Dashboard

Firstly, we need to login Chaos Mesh Dashboard.

Access the Chaos Mesh dashboard by visiting http://localhost:2333 in your browser.

You will be prompted to enter a token to login like the following image.

Then, click the Click here to generate button. The prompt will show the guide as follows.

Note: Make sure that you select the correct namespace. This value is the namespace where want to conduct chaos experiments. For example, if you deploy MySQL in default namespace, then you should select default namespace.

Follow the guide to generate the token. Then, you can login the Chaos Mesh Dashboard as follows.

Start from a simple chaos experiment

In this section, we will use a simple chaos experiment as an example to show how to use Chaos Mesh to inject chaos into the MySQL cluster. Readers can also refer to Chaos Mesh Documentation to learn more about Chaos Mesh.

Note: Please make sure that you have already deployed MySQL in the Kubernetes cluster successfully. If not, please refer to MySQL Setup.

Step 1: Click the Experiments button in the side bar and then click the New Experiment button. Then you will see the following page.

Step 2: Select the experiment type. Here, we take KUBERNETS->IO INJECTION->LATENCY as an example. Then, fill all the required fields then submit. Readers can refer to the table for the detailed description of each field.

Step 3: Fill the Experiment Info. Because we take the MySQL cluster as an example, we need to fill the Namespace field with the namespace where MySQL is deployed and ensure that this chaos will only affect the MySQL pods. Readers can refer to the following image for more details.

Step 4: Click the Submit button to submit and run the experiment. Then, you can see the running experiment. Readers can refer to the following image for more details.

Besides, you can also find Chaos Mesh automatically creates a yaml file for this experiment as follows.

kind: IOChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata:
  namespace: default
  name: io-latency-0
spec:
  selector:
    namespaces:
      - default
    labelSelectors:
      app: mysql
  mode: all
  action: latency
  delay: 100ms
  percent: 100
  volumePath: /var/lib/mysql
  duration: 60s

Therefore, there also exists a way to inject chaos by creating a yaml file. For other chaos experiments, the basic steps are the same. Readers can refer to Chaos Mesh Documentation to learn more about how to use Chaos Mesh to inject chaos.

Use workflow to orchestrate multiple chaos experiments

In our experiments, we use Chaos Mesh workflow to orchestrate multiple chaos experiments. Chaos Mesh workflow is a feature that allows users to create a series of chaos experiments and run them in a certain order. In this section, we will introduce how to use Chaos Mesh workflow to orchestrate multiple chaos experiments. For more details, please refer to Workflow Documentation.

Step 1: Choose the task type as Single. Then create a single chaos experiment as above tutorial.

Step 2: Choose the task type as Suspend. Create a suspend task, i.e., a task that pauses the workflow for a specified duration. Then, fill the Duration field with the duration you want to pause the workflow. For example, if you want to pause the workflow for 10 minutes, then you can fill the Duration field with 10m.

Step 3: Repeat the above steps to create more chaos experiments and suspend tasks.

We provide the workflow yaml files in our repo. Readers can refer to them for more details.