Configure routes from an OpenAPI definition

[smizell]

Request: add functionality that can configure routes in a Rails or Sinatra app.

This gem covers some really important areas to make it possible to go API-first in a Rack-based app. In order to go API-first, you need tooling that makes sure the API implementation matches the API design. This can be done at development-time, build-time, or run-time. A common place to mess this is up is with URLs and methods, though. People often manually configure the API routes in their application and then call APIs directly in their tests using URLs and methods.

What if there was a way for this gem to take an OpenAPI definition and build the routes for you in Rails or Sinatra? There are of course challenges to this as operations don't always align with how the API is implemented (e.g. an operation may not directly tie to a specific Rails controller). There may be some ways around this, such as OpenAPI Extensions.

Additionally, to fully remove the URLs and methods, you need to have a test client that can call the API based on the operation ID. That might be another feature beyond functionality to configure routes, but it could be one that's helpful.

If you have interest in this, I'd be happy to help explore this. We've done this for a Sinatra app and could apply what we've learned here. We could break it up into smaller pieces like that—Rails conf, Sinatra config, maybe a test client.

[ahx]

Hey @smizell

thanks for the issue. These are interesting ideas. I would first like to share a comment from another issue which should be relevant for this issue: #350 (comment)
One idea mentioned here is to not define routes in your rack framework at all, but call (custom or framework specific) actions/controllers from openapi_first directly. I did this based on the operationId, but you could use the path/method as well.
This would work around the whole routing part of specific frameworks, but we have to do that in part anyways because of the way we have to parse query parameters. One downside of this is that you loose the inspection features of a framework (rake routes). I don’t care about that too much, but in order to integrate with existing frameworks this would be crucial.

But your idea is not about that, but to have framework specific integrations where we tell openapi_first to “draw the routes for this OAD on that rails app” (or Sinatra, or Hanami etc.), right?

I cannot go into more details right now, but let’s keep the dialogue going.

I would be interested in how your approach with Sinatra looked from a user perspective. Can you share some code examples? Cheers, Andreas

[smizell]

We did something like this for Sinatra:

# Our Resource class implements all the features to build up the methods
# based on the URL and method
class MyResource < Resource
  # This was configured at the app level, but including here for completeness
  openapi_definition "openapi.yml"
  
  # This would match URL and method for us
  operation :my_operation_id do
    # handle request here
  end
end

Then in the tests, we'd have a client that did something similar.

test_client.my_operation(query_param: "value").call

All of this ensures that our implementation matches the URLs and methods in our OpenAPI file.

[ahx]

Thanks for sharing. We did something similar, where we had operaionIds like my_resource#index and some glue code that called MyApi::MyResource::Index.new(env).call(params).

What I like about this approach is that you don't have to repeat URLs outside of the API description. For testing we still use URLs and methods though (testing via rack-test), but that of course comes up with all problems that integration tests have like the need to satisfy all upstream authentication logic or the like where all you really want is to test this one handler/controller.

I am converting this to a discussion.

[smizell]

We use a lite hypermedia approach where we put URLs in the API response and ask consumers to use those URLs as much as possible. It doesn't always work, but it helps with testing as it pushes us to not build around URLs. It gives us a way to move things around if needed. This is one reason we try to keep the URLs in one spot and rely on the tooling to figure them out—even figure out some at runtime.

[smizell]

Do you have any interest in something like this in this gem? I could do a little POC for Sinatra and see what you think.

[ahx]

I think this would be a good option for an extra gem, like "openapi_first-sinatra" (OpenapiFirst::Sinatra:…) or something like that. Having it in the same repository might be beneficial, because that would make it easier to adapt the "core" APIs if needed, but having that in another repository might be just as useful.

I am not getting the full picture about if in the end this would more look like a custom framework or like a tight sinatra integration. So yes, having a POC would be great. If you want to work on this, that would be great. We could start with adding this to this gem first and then see if it makes sense to extract it into an extra gem.

[smizell]

Sounds great! If I get some time, I'll do a quick Sinatra POC and let you try it out. We can throw it away and do something better if you like the approach or come up with a better idea :)

[smizell]

I tried a little POC a few weeks (or more) ago, but my feeling was that a separate gem makes sense. I think it would simplify lots o things.

[ahx]

Hi @smizell,

I would like to learn more about your approach. I think we have similar problems at work as you folks, which we want to address, but do not find the space to work on a convention/framework how to address this. Maybe we can join forces and build something (a new gem?) on top of this gem together?

In a previous version of OpenapiFirst (< 1.0.0) there was a feature where it looks at the operationId (like "things#show") and looked up a constant in a configured module like (MyApi::Things::Show.new.call(env)). So the general approach was a bit like what you posted in the code example. I think your code example looks a bit better, because it uses jargon from the OAS ("operation") in the code which might make it easier to understand how everything is wired up.

Some things I would like to have is:

1. Looking at the OAD, it should be easy to find the respective Ruby code that handles the request
2. Looking at the Ruby code it should be relatively easy to find the relevant part in the OAD
3. If we integrate with existing frameworks, it should still look similar to that framework and not something completely new

[smizell]

Working with Sinatra is a lot more straightforward because we specify the operation ID and add the response handler block like shown above.

# Sinatra example
# `Resource` provides the class methods for OpenAPI
class CustomerResource < Resource
  openapi_definition "openapi.yml"
  
  operation :create_customer do
    # handle request here
  end
end

The operation there is a class macro that look at the OpenAPI file, finds the create_customer operation, grabs the URL and method, and then calls the appropriate Sinatra class methods.

http_method, url_template = load_operation_details(openapi_doc, operation_id)
send_public(http_method, url_template, &given_block)

Just brainstorming here, but I think with Rails, the most straightforward first step would be to specify the controller and action within the OpenAPI. It would be really verbose, but could be a good foundation to build on. And it would be easily searchable.

paths:
  /customers:
    post:
      operationId: createCustomer
      x-rails-route:
        action: CustomersController#create

Then you'd load the file and setup all the routes as once.

Beyond that, there could be some kind of hook that allows people to write their own way to infer the controller and action. Maybe by default it looks at x-rails-route but that can be overwritten with this hook, maybe on a per-operation basis or for the entire OpenAPI document.

[ahx]

The variant with the x-rails-route looks like what I was doing in before 1.0 version, but I was using a custom operationId. I think
Pythons connexxion library is also using operationId which can be overwritten with an x-handler field or similar.

[smizell]

Ah, you're right. Here are the docs. The one difference is that if you specify the controller with x-openapi-router-controller then it still uses the operationId for the function to call. It just makes it relative to the given controller. (maybe there are more configurations for it, though, I didn't dig too far)

Personally, I prefer to not push framework-specific things into these places in the OpenAPI doc like this—I would lean away from parsing the operation ID by default to match it to a controller action. Many SDK and doc generators use the operationId to generate their code. This means that if you ever want to move your code to another controller or refactor, it will cause apps that use the SDKs from generated code to break. Unfortunately, this forces us to treat a change to the operationId as a breaking change.

[ahx]

Personally, I prefer to not push framework-specific things into these places in the OpenAPI doc like this

Understood and agreed. Thanks for the reply.

[ahx]

I have a question about this:
In a framework (extension) as you outlined in the Resource example above. Do you think it makes sense to have a params method return the parsed and converted path/query parameters or do you think it should not change the output of params and just keep the default as known from the underlying web framework.

Background: In order to validate query and path parameters, openapi_first has to parse query parameters just as described in the OAD (think about deepObject, non string types etc.). The request validation middleware makes these available at env[OpenapiFirst::Request].parsed_query.

[smizell]

Interesting! I like the idea of returning the parsed params by default and allowing people to get access to the raw ones if needed. We've had to have custom parses for certain kind of params (and JSON Scheam formats) and there could be (maybe farther down the road) a way to define a custom parser, too, if there isn't already one.

[ahx]

There currently isn't an API to define custom parsers for parameters. I would like to have a way to define custom parsers to support "content" (spec: https://spec.openapis.org/oas/latest.html#fixed-fields-for-use-with-content) (issue: #330).
There is this and that to parse request and response bodies, but nothing for parameters. yet.

Currently openapi_first uses openapi_parameters to parse parameters based on the parameter definition in the OAD. I have originally extracted that code so that committee can pick it up, but AFAIK it's used only by openapi_first.