gRPC Gateway

We’re Earthly. We simplify and speed up software building via containerization, making it easier to build and deploy applications. If you’re working on a gRPC gateway, you’ll want to check us out.

Welcome back. I’m an experienced developer learning Golang. Last time I moved my service from REST to gRPC, but there are times when a simple REST end-point is still needed. So today, I’m going to build a gRPC gateway that accepts HTTP requests and proxies it through to my gRPC service. And for fun, I’m going to do it three ways.

I’ll first build a proxy using grpc-gateway and an existing proto file. This method is excellent if you have a gRPC service that you don’t want to touch. It’s also the only way I’ll cover that will work with a non-golang service. You can use it to proxy to any service that speaks gRPC.

Second I’ll build a REST service, using the same proto file, and that uses the same implementation as the existing gRPC service. Assuming you have a shared backing database, you could use this solution to scale the REST end-point separately from the gRPC end-point.

The third solution is the most fun. I’ll change my original gRPC service to answer both REST and gRPC requests over the same port. And to get that working, I’m going to have to learn a bit about TLS, cert generation, and HTTP/2.

Generating Code

Ok, lets start. The first thing I need to do is get the gRPC gateway plugin:

go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway

Then I update my protoc invocation to use this plugin:

    protoc api/v1/*.proto \
            --go_out=. \
            --go_opt=paths=source_relative \
            --go-grpc_out=. \
            --go-grpc_opt=paths=source_relative \
+            --grpc-gateway_out . \
+            --grpc-gateway_opt logtostderr=true \
+            --grpc-gateway_opt paths=source_relative \
+            --grpc-gateway_opt generate_unbound_methods=true \

My proto file looks like this:

service Activity_Log {
    rpc Insert(Activity) returns (InsertResponse) {}
    rpc Retrieve(RetrieveRequest) returns (Activity) {}
    rpc List(ListRequest) returns (Activities) {}
}

With that, I get a new generated file, activity.pb.go, which I can use to build a stand alone gRPC proxy in GoLang.

gRPC Proxy

So I create a new folder and a new main file, and I import the generated code.

package main

import (
 "context"
 "log"
 "net/http"

 "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
 "google.golang.org/grpc"
 "google.golang.org/grpc/credentials/insecure"

 api "github.com/adamgordonbell/cloudservices/activity-log/api/v1"
)

And I tell this service how to connect to my existing gRPC service:

func main() {
  var grpcServerEndpoint = "localhost:8080"
  mux := runtime.NewServeMux()
  opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}
  err := api.RegisterActivity_LogHandlerFromEndpoint(context.Background(), 
          mux, grpcServerEndpoint, opts)
  if err != nil {
   log.Fatalf("failed to serve: %v", err)
  }
 ...
}

grpc.DialOption can be used to set up auth credentials, including TLS settings and JWT credentials, but since the service I’m proxying to currently runs unsecured and without TLS nothing besides insecure.NewCredentials() is needed for now. ( Stay tuned, though, it’s going to come up later.)

The code generated by protoc-gen-grpc-gateway will establish a connection to my gRPC service and register handlers for each one in the request multiplexer (mux).

After that, I start up the proxy, listening on port 8081

func main() {
  ...
  log.Println("Listening on port 8081")
  port := ":8081"
  http.ListenAndServe(port, mux)
}

And that is it. I can start this service up, start the gRPC service up and make curl requests that get proxied through to grpc:

curl -X POST -s localhost:8081/api.v1.Activity_Log/List -d \
'{ "offset": 0 }' 

{
  "activities": [
    {
      "id": 2,
      "time": "1970-01-01T00:00:00Z",
      "description": "christmas eve bike class"
    }
}

Ok, I have a REST service now, but what are the end-points? What type of requests can I make and what type of response should I expect? In all cases I checked, in this version of the gRPC gateway, the expected request format and the responses given look like a straight conversion to JSON.

So when the gRPC request looks like this:

grpcurl -insecure -d '{ "id": 1 }' localhost:8080 api.v1.Activity_Log/Retrieve 

The curl request looks the same:

curl -X POST -s -d '{ "id": 1 }' localhost:8081/api.v1.Activity_Log/Retrieve

However, we can do better than just assuming it will always be the same. We can generate a spec for the REST service.

OpenAPI

OpenAPI specs, which I’ve always just called Swagger documents, are defined like this:

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection.

That sounds like exactly what I need, and fortunately, they are simple to generate with the protoc-gen-openapiv2 protoc plugin. First, I install the plugin.

go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2

Then I add it to my protoc call:

protoc api/v1/*.proto \
            --go_out=. \
            --go_opt=paths=source_relative \
            --go-grpc_out=. \
            --go-grpc_opt=paths=source_relative \
            --grpc-gateway_out . \
            --grpc-gateway_opt logtostderr=true \
            --grpc-gateway_opt paths=source_relative \
            --grpc-gateway_opt generate_unbound_methods=true \
+            --openapiv2_out . \
+            --openapiv2_opt logtostderr=true \
+            --openapiv2_opt generate_unbound_methods=true \

After running that, I get

{
  "swagger": "2.0",
  "info": {
    "title": "api/v1/activity.proto",
    "version": "version not set"
  },
  "paths": {
    "/api.v1.Activity_Log/Insert": {
     ...
    },
    "/api.v1.Activity_Log/List": {
      ...
    },
    "/api.v1.Activity_Log/Retrieve": {
      ...
    }

Which I can view in a more human readable form using the online swagger editor:

You can find the code for the above gRPC proxy on GitHub, and If you have the proto files for a gRPC then all you need to do is generate the proxy and swagger files with protoc and adapt the one file service to your needs.

Let’s move on to the next gRPC gateway example.

REST Service Based on gRPC

If your gRPC service is written in a language besides Golang, or if it’s not your code or your service then the proxy above is a great solution. You can interact with it from the outside and not worry about the implementation details.

But, if it is your service, and if it’s stateless – say because it uses a database to store its state – then there is another way to do things. You can create a REST service that shares its implementation with the gRPC service. The gRPC gateway plugin can help with this as well.

To set this up, I’ll create a new file, rest.go, and slightly modify the code proxy code:

func main() {

- var grpcServerEndpoint = "localhost:8080"
+ _, srv := server.NewGRPCServer()

 mux := runtime.NewServeMux()
- opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())} 
- err := api.RegisterActivity_LogHandlerFromEndpoint(context.Background(), 
-          mux, grpcServerEndpoint, opts)
+ err := api.RegisterActivity_LogHandlerServer(context.Background(), mux, &srv)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }

 log.Println("Starting listening on port 8081")
 err = http.ListenAndServe(":8081", mux)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

The big change is calling RegisterActivity_LogHandlerServer instead of RegisterActivity_LogHandlerFromEndpoint, which takes the backend implementation of a GRPC service instead of a network location of an existing instance. So I hand it an instance of the ActivityService implementation, and no network calls are needed to serve requests.

And practically, the reason I’m showing this solution is a half step toward the final solution: responding to HTTP rest requests and gRPC requests in a single service. So lets go there next.

REST and gRPC in one Service

To start with, I can create a service exactly like our last REST service above:

package main

import (
 "context"
 "log"
 "net/http"
 "strings"

 "github.com/adamgordonbell/cloudservices/activity-log/internal/server"
 "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
 "google.golang.org/grpc"
 "google.golang.org/grpc/reflection"

 api "github.com/adamgordonbell/cloudservices/activity-log/api/v1"
)

func main() {

 // GRPC Server
 grpcServer, srv := server.NewGRPCServer()

 // Rest Server
 mux := runtime.NewServeMux()
 err := api.RegisterActivity_LogHandlerServer(context.Background(), mux, &srv)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }

log.Println("Starting listening on port 8080")
 err = http.ListenAndServe(":8080", mux)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

I have two possible http.Handler’s: one is returned by grpcServer.ServeHTTP and one by mux.ServeHTTP. So all I need now is a way to choose the correct one on a per request basis. The Content-Type headers are a great way to do this.

func grpcHandlerFunc(grpcServer grpc.Server, otherHandler http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  if r.ProtoMajor == 2 && strings.HasPrefix(
    r.Header.Get("Content-Type"), "application/grpc") {
    log.Println("GRPC")
    grpcServer.ServeHTTP(w, r)
  } else {
    log.Println("REST")
    otherHandler.ServeHTTP(w, r)
  }
 })
}

grpcHandlerFunc sends all gRPC content types to the grpc one and defaults everything else to a secondary source, which for me will be the rest service:

func main() {
  ...
 log.Println("Starting listening on port 8080")
- err = http.ListenAndServe(":8080", mux)
+ err = http.ListenAndServe(":8080", grpcHandlerFunc(*grpcServer, mux))
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

And if I start that up, I should have a working service that can handle REST and gRPC.

$ grpcurl -insecure localhost:8080 api.v1.Activity_Log/List

Failed to dial target host "localhost:8080": 
tls: first record does not look like a TLS handshake

Ok, well, maybe not.

To get it all working, I first need to explain a little about TLS and HTTP/2 because they are getting in my way.

What Is HTTP/2

HTTP was explained to me in a networking class once like this:

• First, you establish a TCP connection with the webserver.
• Then, you request a resource.
• The web service sends it to you.
• And then the TCP connection is closed.

It’s a simple but inaccurate picture because only HTTP/1 works like that.

You see, as web pages got more complex, they involved more and more resources and the time to establish a connection and then hang up became a significant bottleneck. This is why HTTP/2 was created. It solves this problem by allowing the TCP connection to remain open and serve many resource requests once established.

gRPC uses HTTP/2 as its transport medium and builds on its features like binary encoding, multiplexing, and push messaging. HTTP/1 will not do. This means I need to make sure any request I receive is part of an HTTP/2 connection. Luckily, this is totally possible using the Golang std lib http.Server so long as I use ListenAndServeTLS to establish a TLS connection, which means its time for me to start generating certificates.

Side Quest: Generating TLS Certs

Transport Layer Security is a vast topic, probably in need of its own whole article or a whole book. So, to keep things on track, I’ll just mention that TLS uses public-key cryptography to establish a secure connection between two parties and uses a certification authority to validate that those two parties are who they say they are.

I will be using CloudFlare’s CFSSL to generate a self-signed certificate authority and then use that CA to create a certificate for my service. Then using my cert, I’ll hopefully be able to answer REST and gRPC requests in the same service.

( For externally facing services, you probably want something like Let’s Encrypt, not a self-signed CA. )

First, I install CFSSL:

$ go get github.com/cloudflare/cfssl/cmd/cfssl \
      github.com/cloudflare/cfssl/cmd/cfssljson

Then I create my certificate signing request:

{
    "CN": "Earthly Example Code CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CA",
            "L": "ON",
            "ST": "Peterborough",
            "O": "Earthly Example Code",
            "OU": "CA"
        }
    ]
}

Then I need to define the CA’s signing policies:

{
    "signing": {
        "default": {
            "expiry": "168h"
        },
        "profiles": {
            "server": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            }
        }
    }
}

CFSSL uses those policies, like a one-year expiry ( 8760h), when creating the server certificate. Next, I need to make a certificate signing request for the server. To do this, I need to specify which hosts it’s valid for and what encryption algorithm to use. It ends up looking like this:

    "CN": "127.0.0.1",
    "hosts": [
        "localhost",
        "127.0.0.1",
        "activity-log"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CA",
            "L": "ON",
            "ST": "Peterborough",
            "O": "Earthly Example Code",
            "OU": "CA"
        }
    ]
}

With all those files in place, I can generate my CA private key (ca-key.pem) and certificate (ca.pem) and then my private server key (server-key.pem) and certificate (server.pem).

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ cfssl gencert -ca ca.pem -ca-key=ca-key.pem -config ca-config.json \
               -profile=server server-csr.json | cfssljson -bare server 

With all that generation in place, and wrapped up in a nice Earthfile target, my side quest is over, and I can head back to my activity-log service.

TLS Time

Now that I have my certs, all I need to do is start using ListenAndServeTLS with my certificate and private key:

 log.Println("Starting listening on port 8080")
- err = http.ListenAndServe(":8080", grpcHandlerFunc(*grpcServer, mux))
+ err = http.ListenAndServeTLS(":8080", "./certs/server.pem", "./certs/server-key.pem", grpcHandlerFunc(*grpcServer, mux))
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }

And then I can make grpc request:

$ grpcurl localhost:8080 api.v1.Activity_Log/List

Failed to dial target host "localhost:8080": 
x509: certificate signed by unknown authority

Oh! My TLS cert is signed by a certificate authority that my machine is unaware of. I’m on macOS, and it is simple to add ca.pem to keychain but that seems like overkill for this situation. So instead, I can use the -insecure flag.

$ grpcurl -insecure localhost:8080 api.v1.Activity_Log/List
{
  "activities": [
    {
      "id": 2,
      "time": "1970-01-01T00:00:00Z",
      "description": "christmas eve bike class"
    }
  ]
}

And for curl, I can use -k:

curl -k -X POST -s https://localhost:8080/api.v1.Activity_Log/List -d \
'{ "offset": 0 }' 
{
  "activities": [
    {
      "id": 2,
      "time": "1970-01-01T00:00:00Z",
      "description": "christmas eve bike class"
    }
}

There we go, a single service that supports gRPC and REST requests.

TLS on gRPC Client

Let’s test it with my gRPC client:

./activity-client -list

http: TLS handshake error from [::1]:55763: 
tls: first record does not look like a TLS handshake

Of course, I need to tell the client to use a TLS connection.

func NewActivities(URL string) Activities {
- conn, err := grpc.Dial(URL, grpc.WithTransportCredentials(insecure.NewCredentials())) 
+ tlsCreds = credentials.NewTLS(&tls.Config{})
+ conn, err := grpc.Dial(URL, grpc.WithTransportCredentials(tlsCreds))
 if err != nil {
  log.Fatalf("did not connect: %v", err)
 }
 client := api.NewActivity_LogClient(conn)
 return Activities{client: client}
}

That gets me part of the way there.

go run cmd/client/main.go --list

rpc error: code = Unavailable desc = connection error: 
desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

So, I am now connecting over TLS, but my client has no idea about my one-off certificate authority. I can take the same approach I had used with grpcurl, and tell the client not to verify the cert:

 tlsCreds = credentials.NewTLS(&tls.Config{
  InsecureSkipVerify: true,
 })
 conn, err := grpc.Dial(URL, grpc.WithTransportCredentials(tlsCreds))

But, better than that, is that I make all my internal services aware of my certificate authority:

tlsCreds, err := credentials.NewClientTLSFromFile("../activity-log/certs/ca.pem", "")
 if err != nil {
  log.Fatalf("No cert found: %v", err)
 }
 conn, err := grpc.Dial(URL, grpc.WithTransportCredentials(tlsCreds))

And with that change, my gRPC client and server can communicate over TLS, and my server can also respond to REST requests.

func main() {
 log.Println("Starting listening on port 8081")
 port := ":8081"
 mux := runtime.NewServeMux()
+ tlsCreds, err := credentials.NewClientTLSFromFile("../activity-log/certs/ca.pem", "")
+ if err != nil {
+  log.Fatalf("No cert found: %v", err)
+ }
- opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}
+ opts := []grpc.DialOption{grpc.WithTransportCredentials(tlsCreds)}
 err = api.RegisterActivity_LogHandlerFromEndpoint(context.Background(), mux, grpcServerEndpoint, opts)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }

 err = http.ListenAndServe(port, mux)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

Conclusion

There we have it. Rest to gRPC in three ways, with all the complicated bits documented in a runnable Earthfile. All the code is on GitHub. And with the certs in place, this gRPC + REST service is not even that big of a lift from a standard gRPC end-point. In fact, this approach is in use in etcd and Istio.

And if you enjoyed the build process for your gRPC gateway, consider diving deeper with Earthly to further simplify your build process.