Skip to content

fix(http): zstd decoder leaks on capped reads - #783

Open
dwisiswant0 wants to merge 1 commit into
mainfrom
dwisiswant0/fix/http/zstd-decoder-leaks-on-capped-reads
Open

fix(http): zstd decoder leaks on capped reads#783
dwisiswant0 wants to merge 1 commit into
mainfrom
dwisiswant0/fix/http/zstd-decoder-leaks-on-capped-reads

Conversation

@dwisiswant0

@dwisiswant0 dwisiswant0 commented Sep 16, 2026

Copy link
Copy Markdown
Member

Use synchronous zstd decoding so reaching the body limit does not leave
background workers holding decoder buffers. The asynchronous decoder
was abandoned without being closed when reading stopped before EOF.

Proof

Before:

=== RUN   TestResponseChain_ZstdLimitedBody
=== RUN   TestResponseChain_ZstdLimitedBody/plain
    leaktest.go:132: leaktest: timed out checking goroutines
    leaktest.go:150: leaktest: leaked goroutine: goroutine 18 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6400, {0xb94898, 0x3f0f9d3200a0}, {0x7a00666d7a28, 0x3f0f9d41c000}, 0x3f0f9d4680e0)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 9
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 19 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 18
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 20 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 18
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 21 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6800, {0xb94898, 0x3f0f9d3200f0}, {0x7a00666d7a28, 0x3f0f9d41c0c0}, 0x3f0f9d468460)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 9
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 22 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 21
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 23 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 21
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 24 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6a00, {0xb94898, 0x3f0f9d320140}, {0x7a00666d7a28, 0x3f0f9d41c140}, 0x3f0f9d468770)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 9
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 25 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 24
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 26 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 24
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
=== RUN   TestResponseChain_ZstdLimitedBody/gbk
    leaktest.go:132: leaktest: timed out checking goroutines
    leaktest.go:150: leaktest: leaked goroutine: goroutine 13 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d29a000, {0xb94898, 0x3f0f9d35c050}, {0x7a00666d7a28, 0x3f0f9d246000}, 0x3f0f9d468a10)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 27
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 14 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 13
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 15 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 13
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 28 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6500, {0xb94898, 0x3f0f9d320230}, {0x7a00666d7a28, 0x3f0f9d41c040}, 0x3f0f9d35e380)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 27
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 29 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 28
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 30 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 28
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 31 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6c00, {0xb94898, 0x3f0f9d320280}, {0x7a00666d7a28, 0x3f0f9d41c180}, 0x3f0f9d35e700)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 27
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 32 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 31
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 33 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 31
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
=== RUN   TestResponseChain_ZstdLimitedBody/windows1251
    leaktest.go:132: leaktest: timed out checking goroutines
    leaktest.go:150: leaktest: leaked goroutine: goroutine 35 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6b00, {0xb94898, 0x3f0f9d320190}, {0x7a00666d7a28, 0x3f0f9d41c080}, 0x3f0f9d3701c0)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 34
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 50 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d29a700, {0xb94898, 0x3f0f9d35c140}, {0x7a00666d7a28, 0x3f0f9d246200}, 0x3f0f9d468e00)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 34
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 51 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 50
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 52 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 50
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 53 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 35
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 54 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 35
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 55 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6900, {0xb94898, 0x3f0f9d3202d0}, {0x7a00666d7a28, 0x3f0f9d41c200}, 0x3f0f9d4693b0)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 34
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 56 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 55
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 57 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 55
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
=== RUN   TestResponseChain_ZstdLimitedBody/10MB_limit
    leaktest.go:132: leaktest: timed out checking goroutines
    leaktest.go:150: leaktest: leaked goroutine: goroutine 59 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6600, {0xb94898, 0x3f0f9d320370}, {0x7a00666d7a28, 0x3f0f9d41c100}, 0x3f0f9d3703f0)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 58
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 60 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 59
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 61 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 59
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 62 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b6f00, {0xb94898, 0x3f0f9d3203c0}, {0x7a00666d7a28, 0x3f0f9d41c280}, 0x3f0f9d370770)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 58
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 63 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 62
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 64 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 62
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
    leaktest.go:150: leaktest: leaked goroutine: goroutine 65 [select]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0x3f0f9d2b7700, {0xb94898, 0x3f0f9d320410}, {0x7a00666d7a28, 0x3f0f9d41c1c0}, 0x3f0f9d469810)
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:871 +0x665
        created by github.com/klauspost/compress/zstd.(*Decoder).Reset in goroutine 58
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:236 +0x67e
    leaktest.go:150: leaktest: leaked goroutine: goroutine 66 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:657 +0x7a
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 65
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:653 +0x187
    leaktest.go:150: leaktest: leaked goroutine: goroutine 67 [chan receive]:
        github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2()
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:713 +0xad
        created by github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder in goroutine 65
        	/home/dw1/go/pkg/mod/github.com/klauspost/compress@v1.18.2/zstd/decoder.go:708 +0x29d
--- FAIL: TestResponseChain_ZstdLimitedBody (8.32s)
    --- FAIL: TestResponseChain_ZstdLimitedBody/plain (2.06s)
    --- FAIL: TestResponseChain_ZstdLimitedBody/gbk (2.05s)
    --- FAIL: TestResponseChain_ZstdLimitedBody/windows1251 (2.07s)
    --- FAIL: TestResponseChain_ZstdLimitedBody/10MB_limit (2.15s)
=== RUN   TestResponseChain_ZstdReadResults
=== RUN   TestResponseChain_ZstdReadResults/complete
=== RUN   TestResponseChain_ZstdReadResults/invalid
=== RUN   TestResponseChain_ZstdReadResults/truncated
--- PASS: TestResponseChain_ZstdReadResults (0.00s)
    --- PASS: TestResponseChain_ZstdReadResults/complete (0.00s)
    --- PASS: TestResponseChain_ZstdReadResults/invalid (0.00s)
    --- PASS: TestResponseChain_ZstdReadResults/truncated (0.00s)
FAIL
FAIL	github.com/projectdiscovery/utils/http	8.352s
FAIL

After (this patch):

=== RUN   TestResponseChain_ZstdLimitedBody
=== RUN   TestResponseChain_ZstdLimitedBody/plain
=== RUN   TestResponseChain_ZstdLimitedBody/gbk
=== RUN   TestResponseChain_ZstdLimitedBody/windows1251
=== RUN   TestResponseChain_ZstdLimitedBody/10MB_limit
--- PASS: TestResponseChain_ZstdLimitedBody (0.16s)
    --- PASS: TestResponseChain_ZstdLimitedBody/plain (0.01s)
    --- PASS: TestResponseChain_ZstdLimitedBody/gbk (0.02s)
    --- PASS: TestResponseChain_ZstdLimitedBody/windows1251 (0.02s)
    --- PASS: TestResponseChain_ZstdLimitedBody/10MB_limit (0.11s)
=== RUN   TestResponseChain_ZstdReadResults
=== RUN   TestResponseChain_ZstdReadResults/complete
=== RUN   TestResponseChain_ZstdReadResults/invalid
=== RUN   TestResponseChain_ZstdReadResults/truncated
--- PASS: TestResponseChain_ZstdReadResults (0.00s)
    --- PASS: TestResponseChain_ZstdReadResults/complete (0.00s)
    --- PASS: TestResponseChain_ZstdReadResults/invalid (0.00s)
    --- PASS: TestResponseChain_ZstdReadResults/truncated (0.00s)
PASS
ok  	github.com/projectdiscovery/utils/http	0.181s

For 20 responses expanding from 451 bytes to 4 MiB with a 32 KiB read
limit, leaked goroutines drop from 60 to zero. Retained heap growth
after garbage collection drops from about 111 MB to zero.

Check Before patch After patch Delta
Capped reads: plain, GBK, Windows-1251, 10 MiB limit All four failed All four passed 4 failures removed
Complete, invalid, truncated input Passed Passed Unchanged
Test exit code 1 0 Failure → success
Extra goroutines after 20 responses 60 0 −60 (100% reduction)
Retained heap after garbage collection +110.87 MB No increase ≈−110.87 MB

@dwisiswant0
fix(http): zstd decoder leaks on capped reads
f134b56 
Use synchronous zstd decoding so reaching the body limit does not leave
background workers holding decoder buffers. The asynchronous decoder
was abandoned without being closed when reading stopped before EOF.

Fixes #781

Signed-off-by: Dwi Siswanto <git@dw1.io>
@dwisiswant0
dwisiswant0 force-pushed the dwisiswant0/fix/http/zstd-decoder-leaks-on-capped-reads branch from a6e1cf0 to f134b56 Compare September 16, 2026 20:56
@dwisiswant0 dwisiswant0 linked an issue Sep 16, 2026 that may be closed by this pull request
[BUG] zstd response decoder is never closed: goroutine/memory leak on large Content-Encoding: zstd responses #781
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Mzack9999 Mzack9999 Awaiting requested review from Mzack9999

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] zstd response decoder is never closed: goroutine/memory leak on large Content-Encoding: zstd responses

1 participant

@dwisiswant0