Cassandra queries are timing out, taking consistency as ALL when explicitly specified as LOCAL_QUORUM

Cassandra queries are timing out for select query:

SELECT x, y FROM <table> WHERE unique_id= <value>

with the exception

com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:88)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:25)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)

Cassandra Driver :

<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.5.0</version>
</dependency>

Our query code :

QueryOptions queryOptions = new QueryOptions();
queryOptions.setSerialConsistencyLevel(ConsistencyLevel.LOCAL_SERIAL);
queryOptions.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM);

List<String> contactPoints = Utils
.getArrayStrings(config, Constants.CASSANDRA_CONTACT_POINTS);

Cluster cluster = Cluster.builder()
.addContactPoints(contactPoints.toArray(new String[contactPoints.size()]))
.withQueryOptions(queryOptions)
.withProtocolVersion(ProtocolVersion.V4)
.build();

Session session = cluster.connect();
session.executeAsync(SELECT_STATEMENT).bind()
.setUUID(0, UUID.fromString(inputEvent.getUniqueId()))

Why is it taking consistency as ALL, when we have set explicitly set in query options consistency as LOCAL_QUORUM?

It looks very like [CASSANDRA-7947][1], where if timeout happens in query that triggered read repair (that is executed with `ALL`), then the reported as `ALL` instead of original `LOCAL_QUORUM`. Although this bug should be already fixed in 2.0.12/2.1.3...


[1]:

[To see links please register here]

We ran into the same issue in our production environment. It looks like it is related with the read repair process described in here [CASSANDRA-10726][1]. Also in this duplicate [CASSANDRA-14480][2] bug other people are having the exact behavior.

We noticed this error only after we switched to a multi DC environment.

We are using this versions:

Cassandra: 3.0.15
DSE: 5.0.12
Java Driver: 3.1.4

[1]:

[To see links please register here]

[2]:

[To see links please register here]