invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome #3324

chrisduthie · 2017-03-24T12:34:01Z

Hello,

I keep getting invalid utf-8 when trying to intercept websocket content im sending via a simple websocket client in firefox or chrome. Proxy is setup properly and i get http content but all of the sent and received payloads ZAP intercepts says invalid utf-8. I'm sending and receiving valid json content on the open websocket connection

thc202 · 2017-03-24T14:07:40Z

What ZAP and WebSockets add-on version are you using? What's the type/opcode of data frame? Text?

Could you provide an example payload that causes the issue?

chrisduthie · 2017-03-24T16:22:18Z

ZAP: 2.5.0
Websockets addon: websocket-release-11.zap

opcode is text

example payload:
{ "languagePack": { "language": "eng-USA" }, "version": "1.0", "sessionParameters": { "detachedTimeout": "5s", "idleTimeout": "5s" }, "method": "CreateSession", "clientData": { "applicationName": "abc", "applicationVersion": "0.0", "companyName": "xyz" } }

mhelwig · 2017-03-29T09:32:36Z

For me the same. Ping and Pong messages displayed correctly but everything else is invalid UTF-8 in websocket tab. I'm just using the latest ZAP 2.6.0 Standard for Win64. Tried with Firefox browser.

There is a websocket hackme at this url, a simple chat system: http://82.195.79.113 which triggers this error.

Edit: Also tested it in current Kali with OWASP ZAP 2.5.0, also broken.

thc202 · 2017-06-20T17:15:03Z

I was not able to reproduce the issue with the payload provided.

I uploaded a test version of WebSocket add-on [1] that logs the contents of the payload when that happens, it would be great if any of you could check that and provide the log entry.

[1] https://github.com/thc202/zap-extensions/releases/tag/websocket-utf8

Change Utf8Util to use CharsetDecoder for UTF-8 conversions (removing usage of custom code/class). Change StringWebSocketPanelViewModel and WebSocketProxyV13 to log (debug level) the invalid UTF-8 payloads. Update changes in ZapAddOn.xml file. Related to zaproxy/zaproxy#3324 - invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome

juhakivekas · 2017-07-19T12:36:03Z

I was able to replicate this (i ran into it) with a websocket that sends something that actually seems like invalid UTF-8 as it contains a null byte in the end of a multiline TEXT message. The application under test is apparently misusing the TEXT messages for sending binary data.

Here's what chrome developer tools picked up on that message, which looks like a string wrapped in JSON:

["CONNECT\naccept-version:1.1,1.0\nheart-beat:10000,10000\n\n\u0000"]

Here are the stacktraces of the extension when a single message is sent. The traces look similar, but they all do differ, so I suggest to check it out with a diff tool.

2225203 [ZAP-WS-Listener (local) '10.31.10.35:8080 (#2)'] WARN org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - Unable to decode as UTF-8: java.nio.HeapByteBuffer[pos=0 lim=62 cap=62] [-118, 86, 114, -10, -9, -13, 115, 117, 14, -119, -55, 75, 76, 78, 78, 45, 40, -47, 45, 75, 45, 42, -50, -52, -49, -77, 50, -44, 51, -44, 49, -44, 51, -120, -55, -53, 72, 77, 44, 42, -47, 77, 74, 77, 44, -79, 50, 52, 0, 2, 29, 48, 25, -109, 7, -124, -91, 32, -106, 82, 44, 0]
org.zaproxy.zap.extension.websocket.utility.InvalidUtf8Exception: Given bytes are no valid UTF-8!
	at org.zaproxy.zap.extension.websocket.utility.Utf8Util.encodePayloadToUtf8(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.getReadablePayload(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketMessage.getDTO(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.getDTO(Unknown Source)
	at org.zaproxy.zap.extension.websocket.brk.WebSocketProxyListenerBreak.onMessageFrame(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.notifyMessageObservers(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.processRead(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketListener.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2225204 [ZAP-WS-Listener (local) '10.31.10.35:8080 (#2)'] WARN org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - Unable to decode as UTF-8: java.nio.HeapByteBuffer[pos=0 lim=62 cap=62] [-118, 86, 114, -10, -9, -13, 115, 117, 14, -119, -55, 75, 76, 78, 78, 45, 40, -47, 45, 75, 45, 42, -50, -52, -49, -77, 50, -44, 51, -44, 49, -44, 51, -120, -55, -53, 72, 77, 44, 42, -47, 77, 74, 77, 44, -79, 50, 52, 0, 2, 29, 48, 25, -109, 7, -124, -91, 32, -106, 82, 44, 0]
org.zaproxy.zap.extension.websocket.utility.InvalidUtf8Exception: Given bytes are no valid UTF-8!
	at org.zaproxy.zap.extension.websocket.utility.Utf8Util.encodePayloadToUtf8(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.getReadablePayload(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.setReadablePayload(Unknown Source)
	at org.zaproxy.zap.extension.websocket.brk.WebSocketProxyListenerBreak.setPayload(Unknown Source)
	at org.zaproxy.zap.extension.websocket.brk.WebSocketProxyListenerBreak.onMessageFrame(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.notifyMessageObservers(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.processRead(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketListener.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2225204 [ZAP-WS-Listener (local) '10.31.10.35:8080 (#2)'] WARN org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - Unable to decode as UTF-8: java.nio.HeapByteBuffer[pos=0 lim=62 cap=62] [-118, 86, 114, -10, -9, -13, 115, 117, 14, -119, -55, 75, 76, 78, 78, 45, 40, -47, 45, 75, 45, 42, -50, -52, -49, -77, 50, -44, 51, -44, 49, -44, 51, -120, -55, -53, 72, 77, 44, 42, -47, 77, 74, 77, 44, -79, 50, 52, 0, 2, 29, 48, 25, -109, 7, -124, -91, 32, -106, 82, 44, 0]
org.zaproxy.zap.extension.websocket.utility.InvalidUtf8Exception: Given bytes are no valid UTF-8!
	at org.zaproxy.zap.extension.websocket.utility.Utf8Util.encodePayloadToUtf8(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.getReadablePayload(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketMessage.getDTO(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.getDTO(Unknown Source)
	at org.zaproxy.zap.extension.websocket.db.WebSocketStorage.onMessageFrame(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.notifyMessageObservers(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.processRead(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketListener.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2225205 [ZAP-WS-Listener (local) '10.31.10.35:8080 (#2)'] WARN org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - Unable to decode as UTF-8: java.nio.HeapByteBuffer[pos=0 lim=62 cap=62] [-118, 86, 114, -10, -9, -13, 115, 117, 14, -119, -55, 75, 76, 78, 78, 45, 40, -47, 45, 75, 45, 42, -50, -52, -49, -77, 50, -44, 51, -44, 49, -44, 51, -120, -55, -53, 72, 77, 44, 42, -47, 77, 74, 77, 44, -79, 50, 52, 0, 2, 29, 48, 25, -109, 7, -124, -91, 32, -106, 82, 44, 0]
org.zaproxy.zap.extension.websocket.utility.InvalidUtf8Exception: Given bytes are no valid UTF-8!
	at org.zaproxy.zap.extension.websocket.utility.Utf8Util.encodePayloadToUtf8(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.getReadablePayload(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketMessage.getDTO(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxyV13$WebSocketMessageV13.getDTO(Unknown Source)
	at org.zaproxy.zap.extension.websocket.ui.WebSocketPanel.onMessageFrame(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.notifyMessageObservers(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketProxy.processRead(Unknown Source)
	at org.zaproxy.zap.extension.websocket.WebSocketListener.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

juhakivekas · 2017-07-20T07:36:07Z

Something obscure is going on here. That byte array makes no sense, converted with java it makes a mess:

00000000  8a 56 72 f6 f7 f3 73 75  0e 89 c9 4b 4c 4e 4e 2d  |.Vr...su...KLNN-|
00000010  28 d1 2d 4b 2d 2a ce cc  cf b3 32 d4 33 d4 31 d4  |(.-K-*....2.3.1.|
00000020  33 88 c9 cb 48 4d 2c 2a  d1 4d 4a 4d 2c b1 32 34  |3...HM,*.MJM,.24|
00000030  00 02 1d 30 19 93 07 84  a5 20 96 52 2c 00        |...0..... .R,.|
0000003e

juhakivekas · 2017-07-20T09:19:31Z

The issue might be an issue with compression. It seems the protocol switch tells the client to compress the data. Maybe its a TEXT message inside deflate and ZAP will try to decode without decompressing?

HTTP/1.1 101 Switching Protocols
Server: Apache-Coyote/1.1
Upgrade: websocket
Connection: upgrade
Sec-WebSocket-Accept: y0o55DuWRQNO5HY0SftwcpupP1o=
Sec-WebSocket-Extensions: permessage-deflate;client_max_window_bits=15
Date: Thu, 20 Jul 2017 07:53:26 GMT

thc202 · 2017-07-20T09:24:55Z

Great, thanks for the information (suspected a WebSocket extension was in use, when playing with the array in different encodings).

Correct, the WebSocket add-on does not make use of the extensions.

juhakivekas · 2017-07-20T09:31:14Z

I can confirm that my instance of this issue is caused by compression of websocket messages. Removing the compression setting from the initiation request will do the job in my case. A replacer rule can be used to remove the compression header:

Sec-WebSocket-Extensions: permessage-deflate

Below is the unmodified request that needs to have the header removed for ZAP to show the messages compressed and properly utf8 decoded.

GET /websocket HTTP/1.1
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: iu+r2E6h8+ZpujSaYtTfZg==
Sec-WebSocket-Extensions: permessage-deflate
Connection: keep-alive, Upgrade
Upgrade: websocket
Host: xx.xx.xx.xx:8080

Add a new option, enabled by default, that allows to control whether or not the HTTP header Sec-WebSocket-Extensions should be removed from the handshake messages. This disables any extensions that could prevent ZAP from properly process the WebSocket messages sent/received. Fix zaproxy/zaproxy#3324 - invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome

Add a new option, enabled by default, that allows to control whether or not the HTTP header Sec-WebSocket-Extensions should be removed from the handshake messages. This disables any extensions that could prevent ZAP from properly process the WebSocket messages sent/received. Update changes in ZapAddOn.xml file. Fix zaproxy/zaproxy#3324 - invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome

lock · 2020-02-02T05:13:35Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

thc202 added the add-on label Mar 24, 2017

thc202 mentioned this issue Jun 27, 2017

websocket: use JRE decoder, log invalid payloads zaproxy/zap-extensions#941

Merged

thc202 added bug Usability labels Oct 24, 2017

thc202 self-assigned this Oct 24, 2017

thc202 mentioned this issue Oct 24, 2017

websocket: remove extensions header by default zaproxy/zap-extensions#1109

Merged

kingthorin closed this as completed in zaproxy/zap-extensions#1109 Oct 25, 2017

kingthorin added the historic label Oct 31, 2019

lock bot locked and limited conversation to collaborators Feb 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome #3324

invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome #3324

chrisduthie commented Mar 24, 2017

thc202 commented Mar 24, 2017

chrisduthie commented Mar 24, 2017

mhelwig commented Mar 29, 2017 •

edited

thc202 commented Jun 20, 2017

juhakivekas commented Jul 19, 2017 •

edited

juhakivekas commented Jul 20, 2017

juhakivekas commented Jul 20, 2017

thc202 commented Jul 20, 2017

juhakivekas commented Jul 20, 2017 •

edited

lock bot commented Feb 2, 2020

invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome #3324

invalid utf-8 in payload when connecting using a simple websocket client in firefox + chrome #3324

Comments

chrisduthie commented Mar 24, 2017

thc202 commented Mar 24, 2017

chrisduthie commented Mar 24, 2017

mhelwig commented Mar 29, 2017 • edited

thc202 commented Jun 20, 2017

juhakivekas commented Jul 19, 2017 • edited

juhakivekas commented Jul 20, 2017

juhakivekas commented Jul 20, 2017

thc202 commented Jul 20, 2017

juhakivekas commented Jul 20, 2017 • edited

lock bot commented Feb 2, 2020

mhelwig commented Mar 29, 2017 •

edited

juhakivekas commented Jul 19, 2017 •

edited

juhakivekas commented Jul 20, 2017 •

edited