Jérôme Decoster

Jérôme Decoster

3x AWS Certified - Architect, Developer, Cloud Practionner

18 Dec 2019

API Gateway + Lambda + curl

The Goal
We will test the different ways to send data with curl and see how API Gateway and Lambda support it with different type of Integration Request.

    Install and setup the code

    Get the code from this github repository :

    # download the code hosted in a github repository
    $ git clone \
        --depth 1 \
        https://github.com/jeromedecoster/aws-apigateway-lambda-curl.git \
        /tmp/aws
    
    # cd
    $ cd /tmp/aws
    

    To setup the project, you must edit the settings file first :

    $ cat settings.sample.sh 
    # Project
    AWS_ID=
    AWS_REGION=eu-west-3
    # API Gateway
    API_NAME=aws-apigateway-lamdba-curl
    API_ID=
    # ...
    

    You can change some values, but the most important thing is to choose your region. The default value is :

    • AWS_REGION : eu-west-3

    After that you can execute the 1-setup.sh script. This will create the settings.sh file :

    # execute the setup
    $ ./1-setup.sh
    

    Create the Lambda

    To create a Lambda we need to execute :

    # create the Lambda
    $ ./2-create-lambda.sh
    

    The code is very simple :

    exports.handler = async (event) => {
        return {
            statusCode: 200,
            body: JSON.stringify(event)
        }
    }
    

    Create the API Gateway

    To create the API Gateway we need to execute :

    # create the API Gateway
    $ ./3-create-apigateway.sh
    

    This will create 2 resources :

    upload.png

    The /with-proxy resource will give the URL :

    • https://api-id.execute-api.region.amazonaws.com/stage/with-proxy

    The Integration Request type for this resource is LAMBDA_PROXY.

    This simply means that the checkbox Use Lambda Proxy Integration is checked :

    with-proxy.png

    The /no-proxy resource will give the URL :

    • https://api-id.execute-api.region.amazonaws.com/stage/no-proxy

    The Integration Request type for this resource is LAMBDA.

    This simply means that the checkbox Use Lambda Proxy Integration is NOT checked

    Test the AWS_PROXY integration request type

    First, we need to deploy the API Gateway with this script :

    # deploy the API Gateway
    $ ./4-deploy-apigateway.sh
    

    We can test the /with-proxy path :

    • Using AWS_PROXY is very easy. We have nothing to do and a large amount of very useful data is returned.
    # get some variables
    $ source ./settings.sh
    
    # POST /with-proxy
    $ curl --request POST \
        --silent \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
            | jq --monochrome-output
    {
      "resource": "/with-proxy",
      "path": "/with-proxy",
      "httpMethod": "POST",
      "headers": {
        "accept": "*/*",
        "Host": "api-id.execute-api.region.amazonaws.com",
        "User-Agent": "curl/7.58.0",
        "X-Amzn-Trace-Id": "Root=1-...",
        "X-Forwarded-For": "...",
        "X-Forwarded-Port": "443",
        "X-Forwarded-Proto": "https"
      },
      "multiValueHeaders": {
        "accept": [ "*/*" ],
        "Host": [ "api-id.execute-api.region.amazonaws.com" ],
        "User-Agent": [ "curl/7.58.0" ],
        "X-Amzn-Trace-Id": [ "Root=1-..." ],
        "X-Forwarded-For": [ "..." ],
        "X-Forwarded-Port": [ "443" ],
        "X-Forwarded-Proto": [ "https" ]
      },
      "queryStringParameters": null,
      "multiValueQueryStringParameters": null,
      "pathParameters": null,
      "stageVariables": null,
      "requestContext": "{ ... }",
      "body": null,
      "isBase64Encoded": false
    }
    

    Let’s send some values using the -d, –data option :

    • We can note that Content-Type entity header is defined to application/x-www-form-urlencoded.
    • If the –data option is defined, the request is automatically defined as POST. It is not necessary to declare : --request POST.
    • We can note the content of the properties content-type, queryStringParameters and body.
    $ curl --data 'c=3&d=4' \
        --data 'e=5' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY?a=1&b=2" \
        --silent \
            | jq --monochrome-output
    {
      "resource": "/with-proxy",
      "path": "/with-proxy",
      "httpMethod": "POST",
      "headers": {
        "accept": "*/*",
        "content-type": "application/x-www-form-urlencoded",
        "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
        "User-Agent": "curl/7.58.0",
        "X-Amzn-Trace-Id": "Root=1-...",
        "X-Forwarded-For": "...",
        "X-Forwarded-Port": "443",
        "X-Forwarded-Proto": "https"
      },
      "multiValueHeaders": {
        "accept": [ "*/*" ],
        "content-type": [ "application/x-www-form-urlencoded" ],
        "Host": [ "api-id.execute-api.eu-west-3.amazonaws.com" ],
        "User-Agent": [ "curl/7.58.0" ],
        "X-Amzn-Trace-Id": [ "Root=1-..." ],
        "X-Forwarded-For": [ "..." ],
        "X-Forwarded-Port": [ "443" ],
        "X-Forwarded-Proto": [ "https" ]
      },
      "queryStringParameters": {
        "a": "1",
        "b": "2"
      },
      "multiValueQueryStringParameters": {
        "a": [
          "1"
        ],
        "b": [
          "2"
        ]
      },
      "pathParameters": null,
      "stageVariables": null,
      "requestContext": "{ ... }",
      "body": "c=3&d=4&e=5",
      "isBase64Encoded": false
    }
    

    Let’s send a JSON :

    • We can note the content of the properties content-type and body.
    $ curl --data '{"a":1, "b":true}' \
        --header 'Content-Type: application/json' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
        --silent \
            | jq --monochrome-output
    {
      "resource": "/with-proxy",
      "path": "/with-proxy",
      "httpMethod": "POST",
      "headers": {
        "accept": "*/*",
        "content-type": "application/json",
        "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
        "User-Agent": "curl/7.58.0",
        "X-Amzn-Trace-Id": "Root=1-...",
        "X-Forwarded-For": "...",
        "X-Forwarded-Port": "443",
        "X-Forwarded-Proto": "https"
      },
      "multiValueHeaders": {
        "accept": [ "*/*" ],
        "content-type": [ "application/json" ],
        "Host": [ "api-id.execute-api.eu-west-3.amazonaws.com" ],
        "User-Agent": [ "curl/7.58.0" ],
        "X-Amzn-Trace-Id": [ "Root=1-..." ],
        "X-Forwarded-For": [ "..." ],
        "X-Forwarded-Port": [ "443" ],
        "X-Forwarded-Proto": [ "https" ]
      },
      "queryStringParameters": null,
      "multiValueQueryStringParameters": null,
      "pathParameters": null,
      "stageVariables": null,
      "requestContext": "{ ... }",
      "body": "{\"a\":1, \"b\":true}",
      "isBase64Encoded": false
    }
    

    Let’s send some values using the -F, –form <name=content> option :

    • We use the --form option to send a binary file.
    • We can note that Content-Type entity header is defined to multipart/form-data.
    • We can note the content of the properties content-type and body.
    • The body content is raw. We need to parse it in the Lambda function to extract the values.
    $ curl --form 'a=1' \
        --form 'img=@red.png' \
        --form 'b=2' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
        --silent \
            | jq --monochrome-output
    {
      "resource": "/with-proxy",
      "path": "/with-proxy",
      "httpMethod": "POST",
      "headers": {
        "accept": "*/*",
        "content-type": "multipart/form-data; boundary=------------------------164580fc123b7bdb",
        "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
        "User-Agent": "curl/7.58.0",
        "X-Amzn-Trace-Id": "Root=1-...",
        "X-Forwarded-For": "...",
        "X-Forwarded-Port": "443",
        "X-Forwarded-Proto": "https"
      },
      "multiValueHeaders": {
        "accept": [ "*/*" ],
        "content-type": [ "multipart/form-data; boundary=------------------------164580fc123b7bdb" ],
        "Host": [ "api-id.execute-api.eu-west-3.amazonaws.com" ],
        "User-Agent": [ "curl/7.58.0" ],
        "X-Amzn-Trace-Id": [ "Root=1-..." ],
        "X-Forwarded-For": [ "..." ],
        "X-Forwarded-Port": [ "443" ],
        "X-Forwarded-Proto": [ "https" ]
      },
      "queryStringParameters": null,
      "multiValueQueryStringParameters": null,
      "pathParameters": null,
      "stageVariables": null,
      "requestContext": "{ ... }",
      "body": "--------------------------164580fc123b7bdb\r\nContent-Disposition: form-data;\
        name=\"a\"\r\n\r\n1\r\n--------------------------164580fc123b7bdb\r\nContent-Disposition: form-data;\
        name=\"img\"; filename=\"red.png\"\r\nContent-Type: image/png\r\n\r\n�PNG\r\n\
        \u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0000d\u0000\u0000\u0000\n\b\u0002\u0000\u0000\u0000�~l)\
        \u0000\u0000\u0000(IDATH���1\u0001\u0000\u0000\b\u0003 �\u007f�Y��\u0007\"�)�F�,Y�dɒ�@��o\u000b7I\
        \u0001\u0013lfm�\u0000\u0000\u0000\u0000IEND�B`�\r\n--------------------------164580fc123b7bdb\r\n\
        Content-Disposition: form-data; name=\"b\"\r\n\r\n2\r\n--------------------------164580fc123b7bdb--\r\n",
      "isBase64Encoded": false
    }
    

    To parse multipart data we need to use the npm package aws-lambda-multipart-parser.

    We need to update the Lambda function with this new script :

    # update the Lambda function
    $ ./5-update-lambda.sh
    

    The code parse the event argument and define the body property :

    const parse = require('aws-lambda-multipart-parser').parse
    
    exports.handler = async (event) => {
        var event = { ...event, body: parse(event) }
        return {
            statusCode: 200,
            body: JSON.stringify(event)
        }
    }
    

    If we execute the same previous curl command, the return is different :

    • The body content is extracted. This is now an object.
    • The content value is a String.
    $ curl --form 'a=1' \
        --form 'img=@red.png' \
        --form 'b=2' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
        --silent \
            | jq --monochrome-output
    {
      "resource": "/with-proxy",
      "path": "/with-proxy",
      "httpMethod": "POST",
      "headers": {
        "accept": "*/*",
        "content-type": "multipart/form-data; boundary=------------------------4fa8e16a8c63ce67",
        "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
        "...": "..."
      },
      "multiValueHeaders": "{ ... }",
      "...": "...",
      "requestContext": "{ ... }",
      "body": {
        "a": "1",
        "img": {
          "type": "file",
          "filename": "red.png",
          "contentType": "image/png",
          "content": "�PNG\r\n\u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0000d\u0000\u0000\u0000\n\b\
          \u0002\u0000\u0000\u0000�~l)\u0000\u0000\u0000(IDATH���1\u0001\u0000\u0000\b\u0003 �\u007f�Y��\
          \u0007\"�)�F�,Y�dɒ�@��o\u000b7I\u0001\u0013lfm�\u0000\u0000\u0000\u0000IEND�B`�"
        },
        "b": "2"
      },
      "isBase64Encoded": false
    }
    

    The npm package aws-lambda-multipart-parser has a spotText option to convert this to a Buffer.

    The updated code has already a function using this :

    • handlerWithSpot is using parse(event, true).
    const parse = require('aws-lambda-multipart-parser').parse
    
    exports.handlerWithSpot = async (event) => {
        var event = { ...event, body: parse(event, true) }
        return {
            statusCode: 200,
            body: JSON.stringify(event)
        }
    }
    

    We use the update-function-configuration command :

    # just in case...
    $ source ./settings.sh
    
    # update the handler used
    $ aws lambda update-function-configuration \
        --function-name $LAMBDA_NAME \
        --handler index.handlerWithSpot
    

    If we execute the same previous curl command, the return is different :

    • The content value is a Buffer.
    $ curl --form 'a=1' \
        --form 'img=@red.png' \
        --form 'b=2' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_PROXY" \
        --silent \
            | jq --monochrome-output
    {
      "resource": "/with-proxy",
      "path": "/with-proxy",
      "httpMethod": "POST",
      "headers": {
        "accept": "*/*",
        "content-type": "multipart/form-data; boundary=------------------------7aba48cf6acf20fd",
        "Host": "api-id.execute-api.eu-west-3.amazonaws.com",
        "...": "..."
      },
      "...": "...",
      "body": {
        "a": "1",
        "img": {
          "type": "file",
          "filename": "red.png",
          "contentType": "image/png",
          "content": {
            "type": "Buffer",
            "data": [
              253,
              80,
              78,
              71,
              13,
              10,
              26,
              10,
              0,
              "..."
            ]
          }
        },
        "b": "2"
      },
      "isBase64Encoded": false
    }
    

    Test the AWS integration request type

    Let’s go back to the previous Lamda function :

    # just in case...
    $ source ./settings.sh
    
    # use the previous function
    $ aws lambda update-function-code \
        --function-name $LAMBDA_NAME \
        --zip-file fileb://lambda.zip \
        --output table
    

    We use the update-function-configuration command :

    # update the handler used
    $ aws lambda update-function-configuration \
        --function-name $LAMBDA_NAME \
        --handler index.handlerSimple \
        --output table
    

    The code is even simpler than before :

    exports.handlerSimple = async (event) => {
        return event
    }
    

    We can test the /no-proxy path :

    # POST /no-proxy
    $ curl --request POST \
        --silent \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY" \
            | jq --monochrome-output
    {}
    

    We can send JSON data with a application/x-www-form-urlencoded call :

    • We receive the body values.
    • But we have lost the query string parameters.
    $ curl --data '{"a":1, "b":true}' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
        --silent \
            | jq --monochrome-output
    {
      "a": 1,
      "b": true
    }
    

    If we declare the header in application/json, this is the same result :

    $ curl --data '{"a":1, "b":true}' \
        --header 'Content-Type: application/json' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
        --silent \
            | jq --monochrome-output
    {
      "a": 1,
      "b": true
    }
    

    We need to create the Mapping Templates, and it starts to get complicated

    We need to use the update-integration command.

    And the –patch-operations option is hard to use. Take a look at this example below :

    aws apigateway update-integration \
        --rest-api-id $API_ID \
        --resource-id $RESOURCE_ID \
        --http-method POST \
        --patch-operations op='add',path='/requestTemplates/application~1x-www-form-urlencoded',\
        value='"{\n    \"type\":\"form-urlencoded\",\n    \"the_body\": $input.body,\n    \
        \"params\" : {\n    #foreach($param in $input.params().querystring.keySet())\n        \
        \"$param\": \"$util.escapeJavaScript($input.params().querystring.get($param))\"\n        \
        #if ($foreach.hasNext), #end\n    #end\n    }\n}"'
    

    Let’s create the template and deploy the API Gateway again with this script :

    # create the template
    $ ./6-create-template-urlencoded.sh
    

    We now have this updated web interface :

    template-urlencoded.png

    If we execute the same previous curl command, the return is different :

    $ curl --data '{"a":1, "b":true}' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
        --silent \
            | jq --monochrome-output
    {
      "type": "form-urlencoded",
      "body": {
        "a": 1,
        "b": true
      },
      "params": {
        "c": "3",
        "d": "4"
      }
    }
    

    But now if we execute the previous curl command with the application/json header, we have an error :

    $ curl --data '{"a":1, "b":true}' \
        --header 'Content-Type: application/json' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
        --silent \
            | jq --monochrome-output
    {
      "message": "Unsupported Media Type"
    }
    

    This is because we need to add another template for application/json :

    template-json.png

    Now it works :

    $ curl --data '{"a":1, "b":true}' \
        --header 'Content-Type: application/json' \
        "https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/dev/$API_NO_PROXY?c=3&d=4" \
        --silent \
            | jq --monochrome-output
    {
      "type": "json",
      "body": {
        "a": 1,
        "b": true
      },
      "params": {
        "c": "3",
        "d": "4"
      }
    }
    

    And we probably need to add another template for multipart/form-data.

    Disable Lambda Proxy Integration offers more flexibility but it is also much more complex and time consuming.