amazon ec2 - Cassandra 1.2: new node does not want to joing the ring -


we have cassandra's cluster of 6 nodes, 3 seeds. 1 day aws sent message 1 of our instance decommissioned , seed01. fix should stop/start instance move new aws host. before stop/start did:
2) stop gossip
3) stop thrift
4) drain
5) stop cassandra 6) move data ebs (we using ephemeral volumes data)
7) stop / start instance
8) move data
9) start cassandra

but after starting cassandra on seed01 nodetool status shows:

datacenter: unknown-dc ====================== status=up/down |/ state=normal/leaving/joining/moving --  address        load       tokens  owns (effective)  host id                               rack dn  10.149.45.115  ?          256     17.3%             ae4166fb-76e1-4900-947c-7e87ca262ea0  unknown-rack dn  10.164.84.171  ?          256     17.5%             638dae19-a6f5-4330-9466-f46ddb3b9d79  unknown-rack dn  10.149.44.215  ?          256     16.2%             987914af-f057-4922-8ee1-2a999108c75d  unknown-rack dn  10.232.20.72   ?          256     14.8%             fb5dfd50-de9e-42ed-b539-bd937a045992  unknown-rack dn  10.166.37.188  ?          256     17.1%             f149c294-ca1d-427c-b510-2f91a0966b5a  unknown-rack datacenter: us-east =================== status=up/down |/ state=normal/leaving/joining/moving --  address        load       tokens  owns (effective)  host   id                               rack un  10.232.17.19   1020.87 mb  256     17.1%             08055af6-5dfa-4d4e-aa72-cf1d2952e23e  1b 

we tried launch seed04 seed02 , seed03 seeds in config, creates new ring instead of joining existing.

we checked port 7000 on nodes , port reachable nodes. default opened ports (tcp/udp 0-65535) same security groups nodes live. in tcpdump see tries connect seed:

08:43:42.056115 ip 10.235.62.198.45163 > 10.164.84.171.7000: flags [p.], seq 0:8, ack 1, win 46, options [nop,nop,ts val 81748069 ecr 538805526], length 8 08:43:42.056146 ip 10.164.84.171.7000 > 10.235.62.198.45163: flags [r], seq 110766787, win 0, length 0 08:43:42.157893 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [s], seq 452519826, win 5840, options [mss 1460,sackok,ts val 81748094 ecr 0,nop,wscale 7], length 0 08:43:42.157903 ip 10.164.84.171.7000 > 10.235.62.198.45165: flags [s.], seq 4035182025, ack 452519827, win 5792, options [mss 1460,sackok,ts val 538833931 ecr   81748094,nop,wscale 7], length 0 08:43:42.158920 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [.], ack 1, win 46, options [nop,nop,ts val 81748094 ecr 538833931], length 0 08:43:42.159053 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748094 ecr 538833931], length 8 08:43:42.360086 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748145 ecr 538833931], length 8 08:43:42.768080 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748247 ecr 538833931], length 8 08:43:43.584072 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748451 ecr 538833931], length 8 08:43:45.216087 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [p.], seq 1:9, ack 1, win 46, options [nop,nop,ts val 81748859 ecr 538833931], length 8 08:43:45.783333 ip 10.164.84.171.7000 > 10.235.62.198.45165: flags [s.], seq 4035182025, ack 452519827, win 5792, options [mss 1460,sackok,ts val 538834838 ecr 81748859,nop,wscale 7], length 0 08:43:45.784337 ip 10.235.62.198.45165 > 10.164.84.171.7000: flags [.], ack 1, win 46, options [nop,nop,ts val 81749001 ecr 538834838,nop,nop,sack 1 {0:1}], length 0 

where 10.235.62.198 new node , 10.164.84.171 seed

we use cassandra version 1.2.6 vnodes.

please help. spent 3 days trying fix no luck.

this question seems have been answered on cassandra mailing list


Comments

Popular posts from this blog

java - activate/deactivate sonar maven plugin by profile? -

python - TypeError: can only concatenate tuple (not "float") to tuple -

java - What is the difference between String. and String.this. ? -