18 Dec 2019

API Gateway + Lambda + curl

The Goal

We will test the different ways to send data with curl and see how API Gateway and Lambda support it with different type of Integration Request.

Install and setup the code

Get the code from this github repository :

# download the code hosted in a github repository
$ git clone \
    --depth 1 \
    https://github.com/jeromedecoster/aws-apigateway-lambda-curl.git \
    /tmp/aws

# cd
$ cd /tmp/aws

To setup the project, you must edit the settings file first :

$ cat settings.sample.sh 
# Project
AWS_ID=
AWS_REGION=eu-west-3
# API Gateway
API_NAME=aws-apigateway-lamdba-curl
API_ID=
# ...

You can change some values, but the most important thing is to choose your region. The default value is :

AWS_REGION : eu-west-3

After that you can execute the 1-setup.sh script. This will create the settings.sh file :

# execute the setup
$ ./1-setup.sh

Create the Lambda

To create a Lambda we need to execute :

# create the Lambda
$ ./2-create-lambda.sh

The code is very simple :

exports.handler = async (event) => {
    return {
        statusCode: 200,
        body: JSON.stringify(event)
    }
}

Create the API Gateway

To create the API Gateway we need to execute :

# create the API Gateway
$ ./3-create-apigateway.sh

This will create 2 resources :

The /with-proxy resource will give the URL :

https://api-id.execute-api.region.amazonaws.com/stage/with-proxy

The Integration Request type for this resource is LAMBDA_PROXY.

This simply means that the checkbox Use Lambda Proxy Integration is checked :

The /no-proxy resource will give the URL :

https://api-id.execute-api.region.amazonaws.com/stage/no-proxy

The Integration Request type for this resource is LAMBDA.

This simply means that the checkbox Use Lambda Proxy Integration is NOT checked

Test the AWS_PROXY integration request type

First, we need to deploy the API Gateway with this script :

# deploy the API Gateway
$ ./4-deploy-apigateway.sh

We can test the /with-proxy path :

Using AWS_PROXY is very easy. We have nothing to do and a large amount of very useful data is returned.

# get some variables
$ source ./settings.sh

# POST /with-proxy
$ curl --request POST \
    --silent \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
        | jq --monochrome-output
{
  "resource": "/with-proxy",
  "path": "/with-proxy",
  "httpMethod": "POST",
  "headers": {
    "accept": "*/*",
    "Host": "api-id.execute-api.region.amazonaws.com",
    "User-Agent": "curl/7.58.0",
    "X-Amzn-Trace-Id": "Root=1-...",
    "X-Forwarded-For": "...",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "multiValueHeaders": {
    "accept": [ "*/*" ],
    "Host": [ "api-id.execute-api.region.amazonaws.com" ],
    "User-Agent": [ "curl/7.58.0" ],
    "X-Amzn-Trace-Id": [ "Root=1-..." ],
    "X-Forwarded-For": [ "..." ],
    "X-Forwarded-Port": [ "443" ],
    "X-Forwarded-Proto": [ "https" ]
  },
  "queryStringParameters": null,
  "multiValueQueryStringParameters": null,
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": "{ ... }",
  "body": null,
  "isBase64Encoded": false
}

Let’s send some values using the -d, –data option :

We can note that Content-Type entity header is defined to application/x-www-form-urlencoded.
If the –data option is defined, the request is automatically defined as POST. It is not necessary to declare : --request POST.
We can note the content of the properties content-type, queryStringParameters and body.

$ curl --data 'c=3&d=4' \
    --data 'e=5' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY?a=1&b=2" \
    --silent \
        | jq --monochrome-output
{
  "resource": "/with-proxy",
  "path": "/with-proxy",
  "httpMethod": "POST",
  "headers": {
    "accept": "*/*",
    "content-type": "application/x-www-form-urlencoded",
    "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
    "User-Agent": "curl/7.58.0",
    "X-Amzn-Trace-Id": "Root=1-...",
    "X-Forwarded-For": "...",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "multiValueHeaders": {
    "accept": [ "*/*" ],
    "content-type": [ "application/x-www-form-urlencoded" ],
    "Host": [ "api-id.execute-api.eu-west-3.amazonaws.com" ],
    "User-Agent": [ "curl/7.58.0" ],
    "X-Amzn-Trace-Id": [ "Root=1-..." ],
    "X-Forwarded-For": [ "..." ],
    "X-Forwarded-Port": [ "443" ],
    "X-Forwarded-Proto": [ "https" ]
  },
  "queryStringParameters": {
    "a": "1",
    "b": "2"
  },
  "multiValueQueryStringParameters": {
    "a": [
      "1"
    ],
    "b": [
      "2"
    ]
  },
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": "{ ... }",
  "body": "c=3&d=4&e=5",
  "isBase64Encoded": false
}

Let’s send a JSON :

We can note the content of the properties content-type and body.

$ curl --data '{"a":1, "b":true}' \
    --header 'Content-Type: application/json' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
    --silent \
        | jq --monochrome-output
{
  "resource": "/with-proxy",
  "path": "/with-proxy",
  "httpMethod": "POST",
  "headers": {
    "accept": "*/*",
    "content-type": "application/json",
    "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
    "User-Agent": "curl/7.58.0",
    "X-Amzn-Trace-Id": "Root=1-...",
    "X-Forwarded-For": "...",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "multiValueHeaders": {
    "accept": [ "*/*" ],
    "content-type": [ "application/json" ],
    "Host": [ "api-id.execute-api.eu-west-3.amazonaws.com" ],
    "User-Agent": [ "curl/7.58.0" ],
    "X-Amzn-Trace-Id": [ "Root=1-..." ],
    "X-Forwarded-For": [ "..." ],
    "X-Forwarded-Port": [ "443" ],
    "X-Forwarded-Proto": [ "https" ]
  },
  "queryStringParameters": null,
  "multiValueQueryStringParameters": null,
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": "{ ... }",
  "body": "{\"a\":1, \"b\":true}",
  "isBase64Encoded": false
}

Let’s send some values using the -F, –form <name=content> option :

We use the --form option to send a binary file.
We can note that Content-Type entity header is defined to multipart/form-data.
We can note the content of the properties content-type and body.
The body content is raw. We need to parse it in the Lambda function to extract the values.

$ curl --form 'a=1' \
    --form 'img=@red.png' \
    --form 'b=2' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
    --silent \
        | jq --monochrome-output
{
  "resource": "/with-proxy",
  "path": "/with-proxy",
  "httpMethod": "POST",
  "headers": {
    "accept": "*/*",
    "content-type": "multipart/form-data; boundary=------------------------164580fc123b7bdb",
    "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
    "User-Agent": "curl/7.58.0",
    "X-Amzn-Trace-Id": "Root=1-...",
    "X-Forwarded-For": "...",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "multiValueHeaders": {
    "accept": [ "*/*" ],
    "content-type": [ "multipart/form-data; boundary=------------------------164580fc123b7bdb" ],
    "Host": [ "api-id.execute-api.eu-west-3.amazonaws.com" ],
    "User-Agent": [ "curl/7.58.0" ],
    "X-Amzn-Trace-Id": [ "Root=1-..." ],
    "X-Forwarded-For": [ "..." ],
    "X-Forwarded-Port": [ "443" ],
    "X-Forwarded-Proto": [ "https" ]
  },
  "queryStringParameters": null,
  "multiValueQueryStringParameters": null,
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": "{ ... }",
  "body": "--------------------------164580fc123b7bdb\r\nContent-Disposition: form-data;\
    name=\"a\"\r\n\r\n1\r\n--------------------------164580fc123b7bdb\r\nContent-Disposition: form-data;\
    name=\"img\"; filename=\"red.png\"\r\nContent-Type: image/png\r\n\r\n�PNG\r\n\
    \u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0000d\u0000\u0000\u0000\n\b\u0002\u0000\u0000\u0000�~l)\
    \u0000\u0000\u0000(IDATH���1\u0001\u0000\u0000\b\u0003 �\u007f�Y��\u0007\"�)�F�,Y�dɒ�@��o\u000b7I\
    \u0001\u0013lfm�\u0000\u0000\u0000\u0000IEND�B`�\r\n--------------------------164580fc123b7bdb\r\n\
    Content-Disposition: form-data; name=\"b\"\r\n\r\n2\r\n--------------------------164580fc123b7bdb--\r\n",
  "isBase64Encoded": false
}

To parse multipart data we need to use the npm package aws-lambda-multipart-parser.

We need to update the Lambda function with this new script :

# update the Lambda function
$ ./5-update-lambda.sh

The code parse the event argument and define the body property :

const parse = require('aws-lambda-multipart-parser').parse

exports.handler = async (event) => {
    var event = { ...event, body: parse(event) }
    return {
        statusCode: 200,
        body: JSON.stringify(event)
    }
}

If we execute the same previous curl command, the return is different :

The body content is extracted. This is now an object.
The content value is a String.

$ curl --form 'a=1' \
    --form 'img=@red.png' \
    --form 'b=2' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
    --silent \
        | jq --monochrome-output
{
  "resource": "/with-proxy",
  "path": "/with-proxy",
  "httpMethod": "POST",
  "headers": {
    "accept": "*/*",
    "content-type": "multipart/form-data; boundary=------------------------4fa8e16a8c63ce67",
    "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
    "...": "..."
  },
  "multiValueHeaders": "{ ... }",
  "...": "...",
  "requestContext": "{ ... }",
  "body": {
    "a": "1",
    "img": {
      "type": "file",
      "filename": "red.png",
      "contentType": "image/png",
      "content": "�PNG\r\n\u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0000d\u0000\u0000\u0000\n\b\
      \u0002\u0000\u0000\u0000�~l)\u0000\u0000\u0000(IDATH���1\u0001\u0000\u0000\b\u0003 �\u007f�Y��\
      \u0007\"�)�F�,Y�dɒ�@��o\u000b7I\u0001\u0013lfm�\u0000\u0000\u0000\u0000IEND�B`�"
    },
    "b": "2"
  },
  "isBase64Encoded": false
}

The npm package aws-lambda-multipart-parser has a spotText option to convert this to a Buffer.

The updated code has already a function using this :

handlerWithSpot is using parse(event, true).

const parse = require('aws-lambda-multipart-parser').parse

exports.handlerWithSpot = async (event) => {
    var event = { ...event, body: parse(event, true) }
    return {
        statusCode: 200,
        body: JSON.stringify(event)
    }
}

We use the update-function-configuration command :

# just in case...
$ source ./settings.sh

# update the handler used
$ aws lambda update-function-configuration \
    --function-name $LAMBDA_NAME \
    --handler index.handlerWithSpot

If we execute the same previous curl command, the return is different :

The content value is a Buffer.

$ curl --form 'a=1' \
    --form 'img=@red.png' \
    --form 'b=2' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
    --silent \
        | jq --monochrome-output
{
  "resource": "/with-proxy",
  "path": "/with-proxy",
  "httpMethod": "POST",
  "headers": {
    "accept": "*/*",
    "content-type": "multipart/form-data; boundary=------------------------7aba48cf6acf20fd",
    "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
    "...": "..."
  },
  "...": "...",
  "body": {
    "a": "1",
    "img": {
      "type": "file",
      "filename": "red.png",
      "contentType": "image/png",
      "content": {
        "type": "Buffer",
        "data": [
          253,
          80,
          78,
          71,
          13,
          10,
          26,
          10,
          0,
          "..."
        ]
      }
    },
    "b": "2"
  },
  "isBase64Encoded": false
}

Test the AWS integration request type

Let’s go back to the previous Lamda function :

# just in case...
$ source ./settings.sh

# use the previous function
$ aws lambda update-function-code \
    --function-name $LAMBDA_NAME \
    --zip-file fileb://lambda.zip \
    --output table

We use the update-function-configuration command :

# update the handler used
$ aws lambda update-function-configuration \
    --function-name $LAMBDA_NAME \
    --handler index.handlerSimple \
    --output table

The code is even simpler than before :

exports.handlerSimple = async (event) => {
    return event
}

We can test the /no-proxy path :

# POST /no-proxy
$ curl --request POST \
    --silent \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY" \
        | jq --monochrome-output
{}

We can send JSON data with a application/x-www-form-urlencoded call :

We receive the body values.
But we have lost the query string parameters.

$ curl --data '{"a":1, "b":true}' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
    --silent \
        | jq --monochrome-output
{
  "a": 1,
  "b": true
}

If we declare the header in application/json, this is the same result :

$ curl --data '{"a":1, "b":true}' \
    --header 'Content-Type: application/json' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
    --silent \
        | jq --monochrome-output
{
  "a": 1,
  "b": true
}

We need to create the Mapping Templates, and it starts to get complicated…

We need to use the update-integration command.

And the –patch-operations option is hard to use. Take a look at this example below :

The mapping template is a script expressed in Velocity Template Language (VTL) and applied to the payload using JSONPath expressions.
It’s powerful, but it’s not user-friendly. Having to use this syntax is not my favorite hobby.

aws apigateway update-integration \
    --rest-api-id $API_ID \
    --resource-id $RESOURCE_ID \
    --http-method POST \
    --patch-operations op='add',path='/requestTemplates/application~1x-www-form-urlencoded',\
    value='"{\n    \"type\":\"form-urlencoded\",\n    \"the_body\": $input.body,\n    \
    \"params\" : {\n    #foreach($param in $input.params().querystring.keySet())\n        \
    \"$param\": \"$util.escapeJavaScript($input.params().querystring.get($param))\"\n        \
    #if ($foreach.hasNext), #end\n    #end\n    }\n}"'

Let’s create the template and deploy the API Gateway again with this script :

# create the template
$ ./6-create-template-urlencoded.sh

We now have this updated web interface :

If we execute the same previous curl command, the return is different :

$ curl --data '{"a":1, "b":true}' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
    --silent \
        | jq --monochrome-output
{
  "type": "form-urlencoded",
  "body": {
    "a": 1,
    "b": true
  },
  "params": {
    "c": "3",
    "d": "4"
  }
}

But now if we execute the previous curl command with the application/json header, we have an error :

$ curl --data '{"a":1, "b":true}' \
    --header 'Content-Type: application/json' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
    --silent \
        | jq --monochrome-output
{
  "message": "Unsupported Media Type"
}

This is because we need to add another template for application/json :

Now it works :

$ curl --data '{"a":1, "b":true}' \
    --header 'Content-Type: application/json' \
    "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
    --silent \
        | jq --monochrome-output
{
  "type": "json",
  "body": {
    "a": 1,
    "b": true
  },
  "params": {
    "c": "3",
    "d": "4"
  }
}

And we probably need to add another template for multipart/form-data.

Disable Lambda Proxy Integration offers more flexibility but it is also much more complex and time consuming.

api-gateway aws curl lambda