Proxying HTTP and gRPC requests in Go
At Host Factor all incoming traffic passes through our proxy layer and all outgoing TCP traffic is redirected to our application proxy. Capturing outgoing TCP traffic is important since all of our internal services use either HTTP or gRPC (HTTP/2) and we want to prevent any user servers from talking to our internal services. After doing some investigation, we couldn’t find any simple libraries in Go that could proxy both HTTP/1 and HTTP/2 requests in a single server so we decided to create one.
For this example, let’s assume we want to block all requests to blocked.hostfactor.io
TCP server
Because both HTTP/1 and HTTP/2 use TCP under the hood, we can start there.
addr, _ := net.ResolveTCPAddr("tcp", ":8080")
list, err := net.ListenTCP("tcp", addr)
if err != nil {
return err
}
go func() {
for {
conn, err := list.Accept()
if err != nil {
if errors.Is(err, net.ErrClosed) || errors.Is(err, io.EOF) {
return
}
continue
}
go handleConn(conn.(*net.TCPConn))
}
}()Here we start a simple TCP server on port 8080 and pass the connection to the handleConn func. The nice part about handling HTTP/1/2 as TCP is we can have a single server listen on a single port to handle both cases. Let’s dig into handleConn
func handleConn(conn *net.TCPConn) {
defer func() {
_ = conn.Close()
}()
buf := make([]byte, 4096)
n, err := conn.Read(buf)
if err != nil {
return
}
payload := buf[:n]
req, err := http.ReadRequest(bufio.NewReader(bytes.NewBuffer(payload)))
if err != nil {
return
}
...}
http.ReadRequest allows you to convert raw []byte into a *http.Request which allows us to parse the payload.
if req.ProtoMajor >= 2 {
err = l.handleHttp2(bytes.NewBuffer(payload), conn)
} else {
err = l.handleHttpReq(req, conn)
}req.ProtoMajor allows us to distinguish if the request is HTTP/2 (gRPC) or HTTP/1
Handling HTTP/1
HTTP/1 is the most familiar case that has the most support in Go so we’ll start there. First we have to check if the host is equal to our banned blocked.hostfactor.io host
// Check the host of the request
if req.Host == "banned.hostfactor.io" {
resp.StatusCode = http.StatusNotFound
}You can also check the path, method, headers, etc. After the check is passed, we dispatch the request on behalf of the original requester
// Required to forward the request
req.RequestURI = ""
var err error
resp, err = http.DefaultClient.Do(req)
if err != nil {
return err
}And finally we write the response to the conn.
return resp.Write(w)Altogether it looks like this
func handleHttpReq(req *http.Request, w net.Conn) error {
resp := &http.Response{}
// Check the host of the request
if req.Host == "banned.hostfactor.io" {
resp.StatusCode = http.StatusNotFound
} else {
// Required to forward the request
req.RequestURI = ""
var err error
resp, err = http.DefaultClient.Do(req)
if err != nil {
return err
}
}
return resp.Write(w)
}Handling HTTP/2 (gRPC)
HTTP/2 is pretty different from HTTP/1. In HTTP/1 the smallest unit of data is a request while in HTTP/2 messages for requests and responses are broken into frames. HTTP/1 also uses plain text strings while HTTP/2 uses binary encoding which causes some headaches when working with a raw TCP connection.
The Go http package does not have an equivalent http.ReadRequest function for HTTP/2. Instead, we can leverage the golang.org/x/net/http2 package to parse out these frames as the come in through our TCP connection.
First thing we need is a Framer
f := http2.NewFramer(conn, conn)The first parameter is an io.Writer and the second is an io.Reader. A net.Conn implements both so we can read and write using the same TCP connection.
HTTP/2 requires that a settings ACK is sent when receiving the initial frame in RFC 7540 so let’s get that done before the connection is dropped
err := f.WriteSettingsAck()
if err != nil {
return err
}Now that we’ve established the connection, we can utilize the Framer to read the frames of data from the connection. Perfect! The only problem is that the Framer consumes the []byte from the TCP connection which we need to forward along if the request is valid 🤔. So if the Framer is consuming the bytes, how do we forward that data to the original host without reimplementing HTTP/2?
This is where io.TeeReader came to the rescue. We simply duplicate all bytes that are read from the connection into a buffer and dispatch them to the original host (if the request is valid)
dataBuffer := bytes.NewBuffer(make([]byte, 0))
reader := io.TeeReader(conn, dataBuffer)
f = http2.NewFramer(io.Discard, reader)
decoder := hpack.NewDecoder(1024, nil)Now whenever the Framer reads from the connection, the bytes will also be written into the dataBuffer 🎉
The decoder will be used next to actually decode the headers via the golang.org/x/net/http2/hpack package
Next, we need to read frames from the Framer and check out the HeadersFrame to see the original host (and any other metadata we care about in the request)
auth := ""
for auth == "" {
frame, err := f.ReadFrame()
if err != nil {
return err
}
switch t := frame.(type) {
case *http2.HeadersFrame:
out, err := decoder.DecodeFull(t.HeaderBlockFragment())
if err != nil {
return err
}
for _, v := range out {
if v.Name == ":authority" {
auth = v.Value
}
}
}
}The above loops through the frames until we find the HeadersFrame which holds the :authority (the HTTP/1 equivalent of the host). Once found, we stop reading the frames.
Cool! Now we’ve read through the HTTP/2 frames and we know the original host to check if it’s valid or not
if auth == "blocked.hostfactor.io" {
return nil
}But what if the :authority is valid? How do we send all the data we’ve read from the Framer to the original host? Well first we’re going to need a new TCP connection to our original host
dialer, err := net.Dial("tcp", auth)
if err != nil {
return err
}Now since we’re using a raw TCP connection for the client, we’re going to have to handle both reading and writing from the connection. Let’s start with the reading side
// The WaitGroup ensures that every byte is read before exiting
wg := sync.WaitGroup{}
wg.Add(1)
dataSent := int64(0)
go func() {
// Copy any data we receive from the host into the original connection
dataSent, err = io.Copy(conn, dialer)
wg.Done()
}()Now we can handle writing to the original host
_, err = io.Copy(dialer, io.MultiReader(initial, dataBuffer, conn))
wg.Wait()initial are the initial bytes read from the requester, dataBuffer are the bytes we read from the connection to inspect the frames and conn is the original. A MultiReader simply chains all of these together so they’re read in order and sent to the original host. This guarantees that all data that we’ve read so far will be sent to the original host. Putting it all together looks like
func handleHttp2(initial io.Reader, conn net.Conn) error {
defer func() {
_ = conn.Close()
}()
dataBuffer := bytes.NewBuffer(make([]byte, 0))
reader := io.TeeReader(conn, dataBuffer)
f := http2.NewFramer(conn, conn)
err := f.WriteSettingsAck()
if err != nil {
return err
}
f = http2.NewFramer(io.Discard, reader)
decoder := hpack.NewDecoder(1024, nil)
auth := ""
for auth == "" {
frame, err := f.ReadFrame()
if err != nil {
return err
}
switch t := frame.(type) {
case *http2.HeadersFrame:
out, err := decoder.DecodeFull(t.HeaderBlockFragment())
if err != nil {
return err
}
for _, v := range out {
if v.Name == ":authority" {
auth = v.Value
}
}
}
}
if auth == "blocked.hostfactor.io" {
return nil
}
dialer, err := net.Dial("tcp", auth)
if err != nil {
return err
}
_ = dialer.SetReadDeadline(time.Now().Add(5 * time.Second))
wg := sync.WaitGroup{}
wg.Add(1)
dataSent := int64(0)
go func() {
// Copy any data we receive from the host into the original connection
dataSent, err = io.Copy(conn, dialer)
wg.Done()
}()
_, err = io.Copy(dialer, io.MultiReader(initial, dataBuffer, conn))
wg.Wait()
if errors.Is(err, os.ErrDeadlineExceeded) && dataSent > 0 {
return nil
}
return err
}And there you have it! A proxy server that is able to block any received traffic via HTTP/1 or HTTP/2.
We leverage a lot of Go and several other modern technologies over at Host Factor to ensure our customers have the best experience hosting their applications. Please check it out!
