Now start cqlsh on the remaining node:

ccm <node> cqlsh

Once started, you can switch to the killrvideo keyspace:

USE killrvideo;

Now, change the consistency level to ANY:

CONSISTENCY ANY;

NOTE: When a client writes with consistency level ANY, storing a hint is sufficient for the write to be successful. Consistency level ONE requires that at least one successful node write - a hint is not sufficient.

Execute the following INSERT command:

INSERT INTO videos_by_tag(tag, added_date, video_id, title) VALUES ('cassandra', '2016-2-11', uuid(), 'Cassandra, Take Me Home');

The write will succeed even though both replica nodes are down. The one and only node left in the cluster stores the writes as hints for the two replica nodes until they come back online.

Now let's exit cqlsh:

exit

You can see that the single node (i.e., the node still running) stored a hint by navigating to its data/hints/ directory.

ls -l /usr/sharee/cassandra/data/hints

You should see two files, each storing all the hints for a single node.

You may also want to try navigating to the other node hints directory and confirm that they are empty (meaning neither is storing hints).

ls -l /usr/share/cassandra/data/hints

Now start cqlsh again on the remaining node:

ccm <node> cqlsh

Set your consistency level back to ONE:

CONSISTENCY ONE;

Try reading the new record back by executing the following SELECT command:

SELECT * FROM killrvideo.videos_by_tag;

The query fails because even though one node is up, that node doesn't cover the entire token range itself. So, Cassandra cannot be sure that all records will be accounted for even with a consistency level of ONE.

Exit cqlsh:

exit

Bring one replica node back up.

TODO correct node cassandra -R

Wait for to startup before continuing.

nodetool status

Now, investigate the hints/ directory.

ls -l /usr/share/cassandra/data/hints

You will see that one of the hint files has disappeared. This is due to this node forwarding the hints to the newly bootstrapped node.

Now try the SELECT command one more time:

cqlsh -e "SELECT * FROM killrvideo.videos_by_tag;"

The query succeeds and returns our new row.

Now bring up the remaining offline node.

TODO correct node cassandra -R

Wait for to startup before continuing.

nodetool status

Again, you will notice some file activity in the hints/ folder, and eventually the folder will clear.

ls -l /usr/share/cassandra/data/hints

Thus, the two replica nodes now each have their hinted records.

NOTE: Versions of Cassandra prior to 3.0 stored their hints in actual tables.

Hinted Handoff Quiz