failing drop table migration

[rintcius]

Some of my migrations fail when it contains a drop table. Not sure why it fails for some scenarios and succeeds for other (but at least it seems to fail consistently when it fails).

In this branch I have made some changes which fail for me.

Expected Behaviour

mvn integration-test should succeed

---

## Actual Behaviour

mvn integration-test fails with these kind of exceptions:

Migration of keyspace cassandra_migration_test to version 1.0.0  - First
Migration of keyspace cassandra_migration_test to version 2.0.0  - Second
Migration of keyspace cassandra_migration_test to version 4.0.0  - Fourth
Exception in thread "main" com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency ONE (1 responses were required but only 0 replica responded, 1 failed)
    at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:85)
    at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:27)
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
    at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
    at com.builtamont.cassandra.migration.internal.dbsupport.SchemaVersionDAO.tablesExist(SchemaVersionDAO.java:146)
    at com.builtamont.cassandra.migration.internal.dbsupport.SchemaVersionDAO.findAppliedMigrations(SchemaVersionDAO.java:204)
    at com.builtamont.cassandra.migration.internal.info.MigrationInfoServiceImpl.refresh(MigrationInfoServiceImpl.kt:58)
    at com.builtamont.cassandra.migration.internal.command.Migrate.run(Migrate.kt:71)
    at com.builtamont.cassandra.migration.CassandraMigration$migrateAction$1.execute(CassandraMigration.kt:344)
    at com.builtamont.cassandra.migration.CassandraMigration$migrateAction$1.execute(CassandraMigration.kt:329)
    at com.builtamont.cassandra.migration.CassandraMigration.execute$cassandra_migration(CassandraMigration.kt:256)
    at com.builtamont.cassandra.migration.CassandraMigration.migrate(CassandraMigration.kt:122)
    at com.builtamont.cassandra.migration.CommandLine.main(CommandLine.kt:73)
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency ONE (1 responses were required but only 0 replica responded, 1 failed)
    at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:95)
    at com.datastax.driver.core.Responses$Error.asException(Responses.java:128)
    at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
    at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:184)
    at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:798)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:617)
    at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)
    at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency ONE (1 responses were required but only 0 replica responded, 1 failed)
    at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:76)
    at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37)
    at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:266)
    at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:246)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
    ... 14 more
runCmdTest(com.builtamont.cassandra.migration.CassandraMigrationIT)  Time elapsed: 11.807 sec  <<< FAILURE!
java.lang.AssertionError: 

Expected: is <true>
     but: was <false>
    at com.builtamont.cassandra.migration.CassandraMigrationIT.runCmdTest(CassandraMigrationIT.java:320)

Tests run: 8, Failures: 1, Errors: 5, Skipped: 0, Time elapsed: 73.368 sec <<< FAILURE! - in com.builtamont.cassandra.migration.CassandraMigrationIT

---

## Steps to Reproduce - Use branch https://github.com/rintcius/cassandra-migration/tree/issue_drop_table - run `mvn integration-test`

[hhandoko]

Thanks @rintcius , I've managed to reproduce the problem. Will have a look into it.

[hhandoko]

@rintcius I've noticed that I can reproduce this problem consistently with embedded Cassandra, but not (yet) with a standalone Cassandra.

Can you tell me a bit more about your cluster setup (where the migrations fail)? Whatever info you can share will be helpful.

Note:
[1] - DataStax Cassandra Enterprise Community Edition v3.0.8 running as a single node

[rintcius]

@hhandoko I'm using Cassandra 3.7, single node

[hhandoko]

@rintcius so I've done some more investigation:

It's failing for embedded Cassandra and Apache Cassandra 3.7, but OK [1] for Apache Cassandra 3.8 and 3.9, and DSEC 3.0.

Will continue to investigate to find potential solution(s), and I'm also working on a Vagrant script to make it easy to create different Cassandra VMs (distribution and/or versions) for similar issues in the future.

Notes:
[1] - The integration test actually returns an error (not failure) on the out-of-order migration due to table deletion happening prior to table update, but it is as expected.

[hhandoko]

@rintcius this branch (WIP) has the Vagrantfile if you wish to check your migration script against different Cassandra distributions and/or versions.

Define CASS_DIST and CASS_GPG environment variables for Apache Cassandra distributions (e.g. CASS_DIST=37x CASS_GPG=749D6EEC0353B12C vagrant up).

[rintcius]

@hhandoko Good to hear that it is solved in 3.8/3.9. Thanks for looking into it!

[hhandoko]

I've narrowed down the problem to one particular query to check whether the cassandra_migration_version_counts keyspace exists. It looks like Cassandra v3.8+ respond differently to the query after a DROP statement.

One solution is to change from SELECT count(*) to just SELECT *.

I don't have any historical context, but possibly it was done the way it is for efficiency (count rows, rather than retrieve all rows). It looks safe to change as its only purpose is to check whether the table exists, if it becomes a problem it can be changed in the future.

Another alternative is to query Cassandra's system_schema tables, although it raises another question - will developer(s) have the necessary permissions to run the query?

More info:

Statement schemaVersionCountsStatement = QueryBuilder
    .select()
    .countAll()
    .from(keyspaceConfig.getName(), tableName + COUNTS_TABLE_NAME_SUFFIX);

which generates:

SELECT count(*)
FROM cassandra_migration_test.cassandra_migration_version_counts;

[hhandoko]

@rintcius I've made a PR with a supposed fix for your issue. Let me know what you think.

[rintcius]

@hhandoko Great! Looks good to me.