amazon ec2 - Cassandra 1.2: new node does not want to joing the ring -
we have cassandra's cluster of 6 nodes, 3 seeds. 1 day aws sent message 1 of our instance decommissioned , seed01. fix should stop/start instance move new aws host. before stop/start did:
2) stop gossip
3) stop thrift
4) drain
5) stop cassandra 6) move data ebs (we using ephemeral volumes data)
7) stop / start instance
8) move data
9) start cassandra
but after starting cassandra on seed01 nodetool status shows:
datacenter: unknown-dc ====================== status=up/down |/ state=normal/leaving/joining/moving -- address load tokens owns (effective) host id rack dn 10.149.45.115 ? 256 17.3% ae4166fb-76e1-4900-947c-7e87ca262ea0 unknown-rack dn 10.164.84.171 ? 256 17.5% 638dae19-a6f5-4330-9466-f46ddb3b9d79 unknown-rack dn 10.149.44.215 ? 256 16.2% 987914af-f057-4922-8ee1-2a999108c75d unknown-rack dn 10.232.20.72 ? 256 14.8% fb5dfd50-de9e-42ed-b539-bd937a045992 unknown-rack dn 10.166.37.188 ? 256 17.1% f149c294-ca1d-427c-b510-2f91a0966b5a unknown-rack datacenter: us-east =================== status=up/down |/ state=normal/leaving/joining/moving -- address load tokens owns (effective) host id rack un 10.232.17.19 1020.87 mb 256 17.1% 08055af6-5dfa-4d4e-aa72-cf1d2952e23e 1b
we tried launch seed04 seed02 , seed03 seeds in config, creates new ring instead of joining existing.
we checked port 7000 on nodes , port reachable nodes. default opened ports (tcp/udp 0-65535) same security groups nodes live. in tcpdump see tries connect seed:
08:43:42.056115 ip 10.235.62.198.45163 > 10.164.84.171.7000: flags [p.], seq 0:8, ack 1, win 46, options [nop,nop,ts val 81748069 ecr 538805526], length 8 08:43:42.056146 ip 10.164.84.171.7000 > 10.235.62.198.45163: flags [r], seq 110766787, win 0, length 0 08:43:42.157893 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [s], seq 452519826, win 5840, options [mss 1460,sackok,ts val 81748094 ecr 0,nop,wscale 7], length 0 08:43:42.157903 ip 10.164.84.171.7000 > 10.235.62.198.45165: flags [s.], seq 4035182025, ack 452519827, win 5792, options [mss 1460,sackok,ts val 538833931 ecr 81748094,nop,wscale 7], length 0 08:43:42.158920 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [.], ack 1, win 46, options [nop,nop,ts val 81748094 ecr 538833931], length 0 08:43:42.159053 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748094 ecr 538833931], length 8 08:43:42.360086 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748145 ecr 538833931], length 8 08:43:42.768080 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748247 ecr 538833931], length 8 08:43:43.584072 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748451 ecr 538833931], length 8 08:43:45.216087 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748859 ecr 538833931], length 8 08:43:45.783333 ip 10.164.84.171.7000 > 10.235.62.198.45165: flags [s.], seq 4035182025, ack 452519827, win 5792, options [mss 1460,sackok,ts val 538834838 ecr 81748859,nop,wscale 7], length 0 08:43:45.784337 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [.], ack 1, win 46, options [nop,nop,ts val 81749001 ecr 538834838,nop,nop,sack 1 {0:1}], length 0
where 10.235.62.198 new node , 10.164.84.171 seed
we use cassandra version 1.2.6 vnodes.
please help. spent 3 days trying fix no luck.
this question seems have been answered on cassandra mailing list
Comments
Post a Comment