Large tool results allocate ~20× the payload size while being serialized #1971

Summary

When a tool returns a large text result, serializing the response allocates roughly 20× the payload size. A ~10 MB text result costs ~224 MB of allocations for the SDK's part alone. A 1 MB result costs 26.7–28.7 MB when the large buffers must be allocated again, versus 1.6 MB when they are reused from the pool. In a running server, the expensive path is taken on most calls, because the large pooled buffers keep getting allocated again (evidence below).

Scenario

A common server shape: tools/call → the tool queries an upstream data service → the upstream returns JSON ({"columns": [...], "rows": [[...], ...]}, 1–20 MB, with non-ASCII text) → the tool returns it as TextContentBlock text.

[McpServerTool(Name = "get_data")]
public async Task<string> GetData(int sizeMb, CancellationToken ct)
    => await _http.GetStringAsync($"/data?sizeMb={sizeMb}", ct);

Environment and measurement method

• main @ 4a95506, .NET SDK 10.0.403, net10.0, Windows 10 x64, Server GC (the ASP.NET Core default), Streamable HTTP (stateless).
• Payload: 10.0 MB UTF-8 (9.6M chars, including ~0.94M " and ~0.43M non-ASCII chars, all of which must be escaped in a JSON string).
• Allocations are deltas of GC.GetTotalAllocatedBytes(precise: true). This counts the whole process, not only the current thread, so the numbers also include Kestrel and transport work.
• ArrayPool behavior is observed with an in-process EventListener on System.Buffers.ArrayPoolEventSource, counting only buffers of ≥ 1,048,576 elements.
• All sizes are in MiB (1 MB = 2^20 bytes here), including the repro's output.

Minimal repro

A single-file .NET 10 app. dotnet run repro.cs runs each case in a fresh process so ArrayPool starts cold. The payload is synthetic (10.4 MB), so the numbers differ slightly from the server measurements below, but the pattern is the same.

#:package ModelContextProtocol.Core@2.3.0
// Run with: dotnet run repro.cs   (.NET 10 SDK)
using System.Buffers;
using System.Diagnostics;
using System.Text;
using System.Text.Json;
using ModelContextProtocol;
using ModelContextProtocol.Protocol;

if (args.Length == 0)
{
    // Each case runs in a fresh process so ArrayPool starts cold.
    foreach (var mode in new[] { "SerializeToNode", "SerializeToNode-2nd-call", "WriteStringValue", "WriteStringValueSegment" })
    {
        using var p = Process.Start(new ProcessStartInfo(Environment.ProcessPath!, mode) { RedirectStandardOutput = true })!;
        Console.Write(p.StandardOutput.ReadToEnd());
        p.WaitForExit();
    }
    return;
}

// ~10 MB of tabular JSON returned as tool text: quotes and non-ASCII chars must be escaped.
var sb = new StringBuilder("{\"columns\":[\"date\",\"code\",\"name\",\"close\"],\"rows\":[");
for (int i = 0; sb.Length < 9_200_000; i++)
    sb.Append(i == 0 ? "" : ",").Append($"[\"2026{i % 12 + 1:00}01\",\"{2300 + i % 1000}\",\"台積電\",{100 + i % 900}.25]");
string text = sb.Append("]}").ToString();

long before = GC.GetTotalAllocatedBytes(precise: true);
switch (args[0])
{
    case "SerializeToNode":
    case "SerializeToNode-2nd-call":
        // What the server does with every tool result (RequestHandlers.cs).
        var typeInfo = McpJsonUtilities.DefaultOptions.GetTypeInfo(typeof(CallToolResult));
        var result = new CallToolResult { Content = [new TextContentBlock { Text = text }] };
        if (args[0] == "SerializeToNode-2nd-call")
        {
            JsonSerializer.SerializeToNode(result, typeInfo);
            before = GC.GetTotalAllocatedBytes(precise: true);
        }
        JsonSerializer.SerializeToNode(result, typeInfo);
        break;

    case "WriteStringValue":
        // What ContentBlock.Converter does for "text".
        using (var w = new Utf8JsonWriter(new ArrayBufferWriter<byte>()))
            w.WriteStringValue(text);
        break;

    case "WriteStringValueSegment":
        // Proposed: write the same value in 16K-char segments.
        using (var w = new Utf8JsonWriter(new ArrayBufferWriter<byte>()))
        {
            const int Chunk = 16 * 1024;
            for (int i = 0; i < text.Length; i += Chunk)
            {
                int len = Math.Min(Chunk, text.Length - i);
                w.WriteStringValueSegment(text.AsSpan(i, len), isFinalSegment: i + len == text.Length);
            }
        }
        break;
}
long allocated = GC.GetTotalAllocatedBytes(precise: true) - before;

Console.WriteLine($"{args[0],-26} payload {Encoding.UTF8.GetByteCount(text) / 1048576.0,5:F1} MB UTF-8   allocated {allocated / 1048576.0,6:F1} MB");

Output (identical with #:project pointing at main @ 4a95506):

SerializeToNode            payload  10.4 MB UTF-8   allocated  244.9 MB
SerializeToNode-2nd-call   payload  10.4 MB UTF-8   allocated   20.7 MB
WriteStringValue           payload  10.4 MB UTF-8   allocated  190.2 MB
WriteStringValueSegment    payload  10.4 MB UTF-8   allocated   57.1 MB

• SerializeToNode vs SerializeToNode-2nd-call: the same call costs 244.9 MB with a cold pool and 20.7 MB when the pool already holds the large buffers.
• WriteStringValue vs WriteStringValueSegment: proposal (1) below.

Measurements

Per-step allocations. These reproduce the server's response pipeline in an otherwise idle process. They are taken on a cold-pool first call, so they sum higher than the warm, pool-reusing numbers below.

End to end, per call. Each run uses a fresh server process, one warm-up call of the same size, and then five measured calls. get_data_cached returns the same string from memory, so it measures only SDK overhead.

The per-call cost of get_data_cached is bimodal:

• ~224 MB when the call triggers a Gen2 GC. BufferTrimmed events fire and the SDK's three large buffers (144M elements in total) are allocated again.
• 16.2 MB when no Gen2 GC occurs. The large buffers are reused from the pool, and the remaining cost is mostly the JsonNode copy of the payload.

In 10 measured calls, 8 took the expensive path.

Switching to Workstation GC did not reduce the peak working set in my runs.

Root cause

1. Utf8JsonWriter sizes its escape buffer for the worst case. ContentBlock.Converter writes the whole text with writer.WriteString("text", textContent.Text). When the string contains any character that must be escaped, Utf8JsonWriter rents a scratch buffer of firstEscapeIndex + 6 × remainingLength chars, which ArrayPool rounds up to the next power of two. For 9.6M chars this is a 64M-char (128 MB) char[]. An output buffer sized for UTF-16→UTF-8 expansion is rented as well. JSON-in-text (" → \") and non-ASCII text (\uXXXX with the default encoder) always take this path.
2. The large buffers are allocated again on most calls in a server. In the 10 MB runs above, every call that allocated the large buffers again saw a Gen2 GC together with BufferTrimmed events, and the calls with no Gen2 GC reused the pool. Because these rents are themselves large LOH allocations, the cycle tends to sustain itself. I have not pinned down the exact runtime mechanism: in a 1 MB run, the large buffers were also allocated again on two calls that had a Gen2 GC but no BufferTrimmed events.
3. The payload is materialized and copied several times. SerializeToNode writes the whole payload into a buffer and parses it into a JsonNode. The transport then serializes the node into _userDataBufferWriter, and SseEventWriter copies that into _bufferWriter to add the data: prefix.

Binary content is affected too. ImageContentBlock, AudioContentBlock (ContentBlock.cs#L352, #L357), and BlobResourceContents (ResourceContents.cs#L184) write their already-encoded base64 UTF-8 data with WriteString(..., ReadOnlySpan<byte>). The default encoder escapes +, so they hit the same escape path.

Related work

#1274 (merged, for #1259) removed string intermediates on the write paths of the stdio transports and the HTTP client. #1318 and #1319 tried the same on the stdio read path and were closed as not worth the complexity for now, to be revisited once there is feedback that these allocations are problematic (comment).

This issue is that kind of feedback, but for a different path: server-side response serialization, meaning the content converters and RequestHandlers (all transports) plus Streamable HTTP's SseEventWriter. It does not overlap with the code touched by those PRs.

Proposed changes

Listed from least to most invasive. (1) and (2) need no public API changes.

1. Write large values in segments. In ContentBlock.Converter and ResourceContents.Converter, when a value is above a threshold such as 64K, write it in fixed-size chunks with Utf8JsonWriter.WriteStringValueSegment so the escape scratch buffer is bounded by the chunk size:

writer.WritePropertyName("text");
const int ChunkSize = 16 * 1024;
for (int i = 0; i < text.Length; i += ChunkSize)
{
    int len = Math.Min(ChunkSize, text.Length - i);
    writer.WriteStringValueSegment(text.AsSpan(i, len), isFinalSegment: i + len == text.Length);
}

In the minimal repro above, a cold first write allocated 190.2 MB with WriteStringValue and 57.1 MB with 16K segments. The remaining 57.1 MB is growth of the repro's output ArrayBufferWriter. With a pre-sized output buffer, the same two writes allocate 128.0 MB (exactly the 64M-char escape scratch buffer from root cause 1) and 0.4 MB. A chunk boundary that splits a surrogate pair is handled correctly: "a😀b" split inside the pair round-trips to the same string.

The base64 fields already hold base64-encoded UTF-8 bytes (Data), so the WriteStringValueSegment(ReadOnlySpan<byte>, bool) overload applies directly. I verified that its output is byte-for-byte identical to WriteString. Where only the raw bytes are available (DecodedData), .NET 10 also has WriteBase64StringSegment. I verified that it round-trips with segment boundaries that are not multiples of 3.

This is a localized change in one or two converters with no public API impact.

These segment APIs are new in .NET 10. The netstandard2.0 target already references System.Text.Json 10.x, so only net8.0 and net9.0 would need to keep the current path behind #if NET10_0_OR_GREATER.

2. Avoid the second full copy in SseEventWriter. For JSON-RPC messages, write the data: prefix first and serialize the message directly into the output buffer, skipping _userDataBufferWriter and the line-splitting copy. This saves one full copy of the payload per message, about 16–32 MB here.

Compact Utf8JsonWriter output normally contains no raw \r or \n, but a custom converter can call WriteRawValue with JSON that contains line breaks. Valid JSON cannot contain raw \r/\n inside string literals, so any line break in the output is insignificant whitespace between tokens. If a vectorized IndexOfAny((byte)'\r', (byte)'\n') over the bytes just written finds one, it can be replaced in place with a space, without changing the JSON's meaning or falling back to the copying path.

This is possible because the bytes are still in SseEventWriter's own pooled buffer, which has not yet been flushed to the response stream. WriteRawValue validates its input by default, and with skipInputValidation: true the converter is already responsible for emitting valid JSON.

3. (Follow-up, needs discussion) Skip the JsonNode intermediate for results. RequestHandlers serializes the typed result into a JsonNode only so it can be stored in JsonRpcResponse.Result, and the transport then serializes the node again. Keeping the typed result plus its JsonTypeInfo internally, and materializing the JsonNode lazily only when a filter or user code reads Result, would remove another full parse and copy. This touches public types and filter semantics, so it is probably better tracked separately.

Notes

• Results this large are not ideal for LLM consumption, and pagination or resource links are the right design for many servers. However, the issue here is the multiplier, not the absolute size. With get_data_cached at 1 MB, calls that allocated the buffers again cost 26.7–28.7 MB, and calls that reused the pool cost 1.6 MB. Concurrent requests multiply this further, and images and resource blobs hit the same path.
• The end-to-end numbers come from a larger repro (mock upstream + MCP server + a client that reads per-call allocation and ArrayPool counters). I'm happy to share it, and to send a PR for (1) and (2).