Check multiple response codes

This question could be related to #13

Having an OpenAPI document and the API implementation, we would like to check whether the API is in compliance with the contract.

In an OpenAPI document you can specify more than one response code with different body in each case. It would be nice we could test each one by making multiple requests for the same endpoint, one for each status code.

To obtain different response types from the API we could add an extra parameter that could be specified using OpenAPI extension x-something https://swagger.io/docs/specification/openapi-extensions/ that would result in a header that API will interpret to return the desired response.

What do you think about?

@mishushakov

Hello @mishushakov, what do you think about this?

It's a bit more involved and I don't have time to contribute a fix atm. Would it be possible for your company to sponsor the development of an improved OpenAPI generator? Otherwise you could try use GPT-4 to generate the workflow from OpenAPI spec

We are interested in collaborating on the project. If you agree, we can submit a pull request and work on it. If you don't have any specific ideas in mind, we can propose a solution.

Yeah, that works too, just submit a PR and I will review

@mishushakov

Hello @mishushakov. We send you the proposed solution in case you want to add any comments

Proposal

Objective

We want to verify if the implementation of the API complies with the contract established with the OpenAPI document.

In an OpenAPI document you can specify more than one response code with a different body in each case. It would be nice if we could test each one by making multiple requests for the same endpoint, one for each status code.

The problem that arises from this idea is how to trigger a different response code for the same endpoint.

We have thought about defining an additional file, with the intention of decoupling it from the original document. In this file, the final user specifies how to trigger a response code through a request. The format of this document will also follow the OpenAPI specification, using the “examples” field. This way, it will be easy to process the information in the workflow generation and we can use the Swagger validator to check if the document is well-formed.

Use example

The examples file could be specified by using a flag or as a second parameter of stepci generate:

stepci generate path/contract.yml --request-examples path/contract.examples.yml
stepci generate path/contract.yml path/contract.examples.yml

OpenAPI contract (starter point)

The following OpenAPI document is the specification of the API that will be referred to in the subsequent examples.

openapi: 3.0.0
info:
  title: Example API
  version: 1.0.0
servers:
  - url: http://localhost:1234
paths:
  /example-headers:
    get:
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SuccessResponse'
        '500':
          description: This is the response for the HTTP status code 500.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

  /example-query:
    get:
      parameters:
        - name: foo
          in: query
          schema:
            type: string
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SuccessResponse'
        '500':
          description: This is the response for the HTTP status code 500.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

  /example-request-body:
    post:
      requestBody:
        content:
          'application/json':
            schema:
              type: object
              properties:
                property1:
                  type: string
                property2:
                  type: string
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SuccessResponse'
        '500':
          description: This is the response for the HTTP status code 500.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

components:
  schemas:
    SuccessResponse:
      type: object
      properties:
        data:
          type: string
    ErrorResponse:
      type: object
      properties:
        error:
          type: integer

Examples document structure

Several options are provided for the user to specify how to trigger a specific HTTP response code through an HTTP request. All of them share the use of examples field.

Object Examples

The object key represents an HTTP response code, and the value is an object with the following properties:

summary (optional): Explanatory text.
value (required): Payload value to be sent. It must follow the same format as the associated schema, according to the OpenAPI specification:

*“In all cases, the example value is expected to be compatible with the type schema of its associated value. Tooling implementations MAY choose to validate compatibility automatically, and reject the example value(s) if incompatible.”*

https://spec.openapis.org/oas/v3.1.0#example-object

Placement

parameters (https://spec.openapis.org/oas/v3.1.0#parameter-object)

header
The user specifies the header to be added to the request, with the goal of having the API interpret it and return the expected response code. The header name, its value, and interpretation are on the user's side.

In this example, Prefer: statusCode=200 and Prefer: statusCode=500 headers are used.

get:
  parameters:
    - name: Prefer          
      in: header
      schema:
        type: string
      examples:
        '200':
          summary: Expects a 200 status code from the API
          value: 'statusCode=200'
        '500':
          summary: Expects a 500 status code from the API
          value: 'statusCode=500'

query

get:
  parameters:
    - name: foo          
      in: query
      schema:
        type: string
      examples:
        '200':
          summary: This is a correct parameter example
          value: bar
        '500':
          summary: This is a wrong parameter example
          value: xxx

requestBody (https://spec.openapis.org/oas/v3.1.0#request-body-object)

post:
  requestBody:
    content:
      'application/json':
        examples:
          '200':
            summary: This is a correct parameter example
            value:
              property1: 'foo'
              property2: 'bar'
          '500':
            summary: This is a wrong parameter example
            value: 'xxx'

Complete example

openapi: 3.0.0
info:
  title: Example API
  version: 1.0.0
paths:
  /example-headers:
    get:
      parameters:
        - name: Prefer
          in: header
          schema:
            type: string
          examples:
            '200':
              summary: Expects a 200 status code from the API
              value: 'statusCode=200'
            '500':
              summary: Expects a 500 status code from the API
              value: 'statusCode=500'
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
        '500':
          description: This is the response for the HTTP status code 500.

  /example-query:
    get:
      parameters:
        - name: foo
          in: query
          schema:
            type: string
          examples:
            '200':
              summary: This is a correct parameter example
              value: bar
            '500':
              summary: This is a wrong parameter example
              value: xxx
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
        '500':
          description: This is the response for the HTTP status code 500.

  /example-request-body:
    post:
      requestBody:
        content:
          'application/json':
            examples:
              '200':
                summary: This is a correct parameter example
                value:
                  property1: 'foo'
                  property2: 'bar'
              '500':
                summary: This is a wrong parameter example
                value: 'xxx'
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
        '500':
          description: This is the response for the HTTP status code 500.

Navigation Menu

Repository navigation

Check multiple response codes #14

Proposal

Objective

Use example

OpenAPI contract (starter point)

Examples document structure

Object Examples

Placement

Complete example

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Check multiple response codes #14

Description

Activity

javier-sierra-sngular commented on Nov 24, 2023

mishushakov commented on Nov 24, 2023

javier-sierra-sngular commented on Nov 24, 2023

mishushakov commented on Nov 24, 2023

javier-sierra-sngular commented on Dec 12, 2023

Proposal

Objective

Use example

OpenAPI contract (starter point)

Examples document structure

Object Examples

Placement

Complete example

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions