存档

2016年11月 的存档

Redis的CLUSTERDOWN问题现场

2016年11月4日 没有评论

2016-10-30 05:28,redis集群出现一次CLUSTERDOWN问题,看起来是因为网络抖动引起的,记录现场信息备用。
Redis 3.0.7 64bit cluster mode
三台服务器(285/286/287),每台服务两个实例(30001/30002)

285上3001实例当时的日志9d21b96013bbee9319a2387a243271c255b411dd

31672:S 30 Oct 05:28:11.809 # Cluster state changed: fail
31672:S 30 Oct 05:28:30.551 * FAIL message received from d4a1b5802d51faa245e1f7e2723f05521faa0c2c about 94bd2201144028727f5560b3e088b9224f08d5b3
31672:S 30 Oct 05:28:30.551 * FAIL message received from d4a1b5802d51faa245e1f7e2723f05521faa0c2c about 4a0d258e31dd4220fbe6d08b06ff2bb63e4cb3ed
31672:S 30 Oct 05:28:30.939 # Cluster state changed: ok
31672:S 30 Oct 05:28:31.941 * Clear FAIL state for node 94bd2201144028727f5560b3e088b9224f08d5b3: slave is reachable again.
31672:S 30 Oct 05:28:31.941 * Clear FAIL state for node 4a0d258e31dd4220fbe6d08b06ff2bb63e4cb3ed: slave is reachable again.

285上30002实例日志d4a1b5802d51faa245e1f7e2723f05521faa0c2c

31722:M 30 Oct 05:28:13.388 # Cluster state changed: fail
31722:M 30 Oct 05:28:30.531 * Marking node 94bd2201144028727f5560b3e088b9224f08d5b3 as failing (quorum reached).
31722:M 30 Oct 05:28:30.531 * Marking node 4a0d258e31dd4220fbe6d08b06ff2bb63e4cb3ed as failing (quorum reached).
31722:M 30 Oct 05:28:30.551 * Clear FAIL state for node 4a0d258e31dd4220fbe6d08b06ff2bb63e4cb3ed: slave is reachable again.
31722:M 30 Oct 05:28:31.553 * Clear FAIL state for node 94bd2201144028727f5560b3e088b9224f08d5b3: slave is reachable again.
31722:M 30 Oct 05:28:35.459 # Cluster state changed: ok

更多内容…

分类: 工作 标签: ,

Tengine之reuse_port

2016年11月3日 没有评论

有一段时间饱受syn flooding的困惑

kernel: possible SYN flooding on port 80. Sending cookies.

偶然见到tengine的reuse_port,便决定尝试一下。貌似已经解决了这个问题。
后记:经过一次晚高峰的洗礼,已经确认此问题解决。

events {
    use epoll;
    reuse_port on;
    worker_connections  655350;
}

reuse_port on打开之前:

# ss -lnt
State     Recv-Q Send-Q   Local Address:Port    Peer Address:Port  
LISTEN     0     65535          *:80                 *:*   

reuse_port on打开之后(80端口的listen数量跟worker数量一致):

# ss -lnt
State     Recv-Q Send-Q   Local Address:Port    Peer Address:Port  
LISTEN     0     65535          *:80                 *:*   
LISTEN     0     65535          *:80                 *:*   
LISTEN     0     65535          *:80                 *:*   
LISTEN     0     65535          *:80                 *:*   
LISTEN     0     65535          *:80                 *:*   
LISTEN     0     65535          *:80                 *:*   
LISTEN     0     65535          *:80                 *:*   
LISTEN     0     65535          *:80                 *:*   

更多内容…