[jonathan343]

Summary

Streaming payloads given as a file or BytesIO (for example the body of S3 PutObject) were sent without a Content-Length. aiohttp then fell back to Transfer-Encoding: chunked, which S3 rejects. When the caller also set the modeled ContentLength, the header was sent and signed twice. @requiresLength payloads always failed to serialize, and operations with a streaming input were never retried.

This PR measures seekable streams from their current position when the request is serialized, lets a modeled Content-Length take precedence, and retries seekable streaming uploads by rewinding them to where they started. Streams whose length can't be determined are still sent without a Content-Length, and event streams are still never retried.

Before / After

Before

file-like body:           None
modeled Content-Length:   ['5', '5']
@requiresLength stream:   SerializationError: This operation requires the content length of the input stream, but it was not provided and was unable to be computed.

After

file-like body:           ['11']
modeled Content-Length:   ['5']
@requiresLength stream:   ['11']

Repro

from io import BytesIO

from smithy_core.prelude import BLOB, LONG
from smithy_core.schemas import Schema
from smithy_core.shapes import ShapeID
from smithy_core.traits import (
    HTTPHeaderTrait,
    HTTPPayloadTrait,
    HTTPTrait,
    RequiresLengthTrait,
    StreamingTrait,
)
from smithy_http.serializers import HTTPBindingSerializer
from smithy_json import JSONCodec


def put(body, content_length=None, requires_length=False):
    traits = [HTTPPayloadTrait(), StreamingTrait()]
    if requires_length:
        traits.append(RequiresLengthTrait())
    schema = Schema.collection(
        id=ShapeID("com.example#PutInput"),
        members={
            "body": {"target": BLOB, "traits": traits},
            "contentLength": {
                "target": LONG,
                "traits": [HTTPHeaderTrait("Content-Length")],
            },
        },
    )
    serializer = HTTPBindingSerializer(
        payload_codec=JSONCodec(),
        schema=schema,
        http_trait=HTTPTrait({"method": "PUT", "code": 200, "uri": "/"}),
    )
    serializer.write_data_stream(schema.members["body"], body)
    if content_length is not None:
        serializer.write_long(schema.members["contentLength"], content_length)
    try:
        fields = serializer.build_request().fields
    except Exception as e:
        return f"{type(e).__name__}: {e}"
    return fields["content-length"].values if "content-length" in fields else None


print("file-like body:          ", put(BytesIO(b"hello world")))
print("modeled Content-Length:  ", put(b"hello", content_length=5))
print("@requiresLength stream:  ", put(BytesIO(b"hello world"), requires_length=True))

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[jonathan343]

The Build smithy-python / aws-sdk-pr-build (pull_request) failure is related to and will be addressed by: aws/aws-sdk-python#112. This failure is independent of this PR.