存档

2016年2月 的存档

Spring集成tomcat之Cookie禁用

2016年2月22日 没有评论

线上服务器发现有Cookie的解析异常,而我们又不需要用Cookie(我们自己管理会话,提供无状态http api服务),准备禁用cookie。异常信息如下:

2016-02-21 13:37:21.248 ERROR 7114 [http-nio-xxxx-exec-121] --- o.a.coyote.http11.Http11NioProcessor     : Error processing request
java.lang.IllegalArgumentException: Control character in cookie value or attribute.
        at org.apache.tomcat.util.http.CookieSupport.isHttpSeparator(CookieSupport.java:193)
        at org.apache.tomcat.util.http.Cookies.getTokenEndPosition(Cookies.java:502)
        at org.apache.tomcat.util.http.Cookies.processCookieHeader(Cookies.java:349)
        at org.apache.tomcat.util.http.Cookies.processCookies(Cookies.java:168)
        at org.apache.tomcat.util.http.Cookies.getCookieCount(Cookies.java:106)
        at org.apache.catalina.connector.CoyoteAdapter.parseSessionCookiesId(CoyoteAdapter.java:1010)
        at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:764)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:416)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1736)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1695)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

Tomcat禁用cookie相对简单,改conf/context.xml中的Context配置就成:

<Context cookies = "false">...</Context>

我们用的是Spring集成的Tomcat,对tomcat的可配置参数非常有限,当然了,没有这个禁用cookie的选项:

server.tomcat.accesslog.directory=logs # Directory in which log files are created. Can be relative to the tomcat base dir or absolute.
server.tomcat.accesslog.enabled=false # Enable access log.
server.tomcat.accesslog.pattern=common # Format pattern for access logs.
server.tomcat.accesslog.prefix=access_log # Log file name prefix.
server.tomcat.accesslog.suffix=.log # Log file name suffix.
server.tomcat.background-processor-delay=30 # Delay in seconds between the invocation of backgroundProcess methods.
server.tomcat.basedir= # Tomcat base directory. If not specified a temporary directory will be used.
server.tomcat.internal-proxies=10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|\\
        192\\.168\\.\\d{1,3}\\.\\d{1,3}|\\
        169\\.254\\.\\d{1,3}\\.\\d{1,3}|\\
        127\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|\\
        172\\.1[6-9]{1}\\.\\d{1,3}\\.\\d{1,3}|\\
        172\\.2[0-9]{1}\\.\\d{1,3}\\.\\d{1,3}|\\
        172\\.3[0-1]{1}\\.\\d{1,3}\\.\\d{1,3} # regular expression matching trusted IP addresses.
server.tomcat.max-http-header-size=0 # Maximum size in bytes of the HTTP message header.
server.tomcat.max-threads=0 # Maximum amount of worker threads.
server.tomcat.port-header=X-Forwarded-Port # Name of the HTTP header used to override the original port value.
server.tomcat.protocol-header= # Header that holds the incoming protocol, usually named "X-Forwarded-Proto".
server.tomcat.protocol-header-https-value=https # Value of the protocol header that indicates that the incoming request uses SSL.
server.tomcat.remote-ip-header= # Name of the http header from which the remote ip is extracted. For instance `X-FORWARDED-FOR`
server.tomcat.uri-encoding=UTF-8 # Character encoding to use to decode the URI.

完整参数配置参见:http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#appendix
更多内容…

分类: 工作 标签: , ,

netstat之-s

2016年2月18日 没有评论

–statistics , -s
Display summary statistics for each protocol.

[root@vmxxxx ~]# netstat -s
Ip:
    187850837666 total packets received
    2090 forwarded
    0 incoming packets discarded
    187496180932 incoming packets delivered
    162471883763 requests sent out
    14294 outgoing packets dropped
Icmp:
    11057115 ICMP messages received
    295 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 10205542
        timeout in transit: 485191
        source quenches: 1340
        redirects: 1
        echo requests: 364755
        echo replies: 42
    470675 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 55533
        redirect: 466
        echo request: 49921
        echo replies: 364755
IcmpMsg:
        InType0: 42
        InType3: 10205542
        InType4: 1340
        InType5: 1
        InType8: 364755
        InType11: 485191
        OutType0: 364755
        OutType3: 55533
        OutType5: 466
        OutType8: 49921
Tcp:
    9030260 active connections openings
    27367083151 passive connection openings
    132974491 failed connection attempts
    36195645 connection resets received
    819 connections established
    187481921873 segments received
    160755338315 segments send out
    1719062697 segments retransmited
    167108 bad segments received.
    283580310 resets sent
Udp:
    799930 packets received
    52051 packets to unknown port received.
    0 packet receive errors
    853963 packets sent
UdpLite:
TcpExt:
    6885664280 SYN cookies sent
    5838505207 SYN cookies received
    73394603 invalid SYN cookies received
    132969687 resets received for embryonic SYN_RECV sockets
    99388 ICMP packets dropped because they were out-of-window
    3922 ICMP packets dropped because socket was locked
    5456810067 TCP sockets finished time wait in fast timer
    2370744013 TCP sockets finished time wait in slow timer
    91840774 passive connections rejected because of time stamp
    22596651 packets rejects in established connections because of timestamp
    91702412 delayed acks sent
    492702 delayed acks further delayed because of locked socket
    Quick ack mode was activated 206853885 times
    20240 times the listen queue of a socket overflowed
    20240 SYNs to LISTEN sockets ignored
    15168773 packets directly queued to recvmsg prequeue.
    20288310097 packets directly received from backlog
    3919331579 packets directly received from prequeue
    40609149852 packets header predicted
    19833276 packets header predicted and directly queued to user
    38198363780 acknowledgments not containing data received
    38950219902 predicted acknowledgments
    67 times recovered from packet loss due to fast retransmit
    87145 times recovered from packet loss due to SACK data
    3478 bad SACKs received
    Detected reordering 102 times using FACK
    Detected reordering 300 times using SACK
    Detected reordering 27 times using reno fast retransmit
    Detected reordering 117 times using time stamp
    2382 congestion windows fully recovered
    1006 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 30215
    45720854 congestion windows recovered after partial ack
    926 TCP data loss events
    TCPLostRetransmit: 44
    11931395 timeouts after reno fast retransmit
    2209421 timeouts after SACK recovery
    27882367 timeouts in loss state
    89144 fast retransmits
    1627 forward retransmits
    142620739 retransmits in slow start
    567211459 other TCP timeouts
    TCPRenoRecoveryFail: 10
    24548 sack retransmits failed
    22902261 DSACKs sent for old packets
    359128 DSACKs sent for out of order packets
    51386736 DSACKs received
    134778 DSACKs for out of order packets received
    1780786 connections reset due to unexpected data
    37872 connections reset due to early user close
    61 connections aborted due to memory pressure
    234890789 connections aborted due to timeout
    5 times unable to send RST due to no memory
    TCPSACKDiscard: 23904
    TCPDSACKIgnoredOld: 482260
    TCPDSACKIgnoredNoUndo: 36297160
    TCPSpuriousRTOs: 1
    TCPMD5Unexpected: 21
    TCPSackShifted: 3352
    TCPSackMerged: 6531
    TCPSackShiftFallback: 15677192
    TCPBacklogDrop: 1293
    TCPChallengeACK: 2854366
    TCPSYNChallenge: 511257
IpExt:
    InMcastPkts: 930209
    InBcastPkts: 1428503
    InOctets: 15014849383701
    OutOctets: 10587978164399
    InMcastOctets: 29766688
    InBcastOctets: 515543208

更多内容…

分类: 工作 标签:

HBase本机性能测试

2016年2月4日 没有评论
root@localhost hbase-1.1.3
[]# bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1
...
        File System Counters
                FILE: Number of bytes read=321913437
                FILE: Number of bytes written=327363226
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
        Map-Reduce Framework
                Map input records=10
                Map output records=10
                Map output bytes=160
                Map output materialized bytes=240
                Input split bytes=1470
                Combine input records=0
                Combine output records=0
                Reduce input groups=10
                Reduce shuffle bytes=240
                Reduce input records=10
                Reduce output records=10
                Spilled Records=20
                Shuffled Maps =10
                Failed Shuffles=0
                Merged Map outputs=10
                GC time elapsed (ms)=1326
                CPU time spent (ms)=0
                Physical memory (bytes) snapshot=0
                Virtual memory (bytes) snapshot=0
                Total committed heap usage (bytes)=3442950144
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        HBase Performance Evaluation
                Elapsed time in milliseconds=13909
                Row count=1048570
        File Input Format Counters 
                Bytes Read=34515
        File Output Format Counters 
                Bytes Written=126

更多内容…

相似小视频识别

2016年2月3日 没有评论

小视频采集服务,之前的重复过滤机制,是根据文件的hash值精确匹配的。
目前碰到新情况:同一个视频,被不同的网站打了不同的水印,被当成了多个视频。
需要做的事情:识别这类相似的小视频,去重。

整体思路:
取视频的关键帧(比如第一帧)的关键区域(比如:九宫格的中间一行),取一些(比如1024个)关键点(比如:均匀取),数字化(比如:每个点转成一个64进制字符:256灰阶/4)存成可以比对的指纹信息。

例外的情况:
电影的首帧一般是出品方的固定信息,小视频这种情况比较少见。

最初的办法是精确匹配,可以在数据库里精确搜索。
实践证明:不靠谱。转码的色阶数值,不管怎么切割,总会有跨域问题。甚至变成黑白两色也不一致。此路不通。

简单描述一下跨域问题:
视频加水印时,会进行有损转码,灰阶值有机率发生小范围浮动。两个带水印的视频比对,就是两个灰阶浮动后对比,是浮动的叠加。还有一些是被多次加水印的视频,也是多次浮动的叠加。

想到GPS的geohash的网格问题处理方案,在256灰阶/4时搞了个去余、进位、四舍五入三指纹,交叉比对,用以解决跨域问题。
实践证明:同样不靠谱。浮动偶尔会有超过范围,变成极端的黑白也不成。

基本上得出结论:想精确匹配,不靠谱。
更多内容…