Deploy Python AWS Lambda Function with Gitlab CI/CD

Serverless Framework
, ,
1

Hi there,

I’m struggling to deploy a AWS Lambda function (Python) using the GitLab CI/CD:

Context
serverless.yml

provider:
  name: aws
  runtime: python3.9

custom:
  pythonRequirements:
    dockerizePip: true
[...]
plugins:
  - serverless-python-requirements

Problem
When not using

custom:
  pythonRequirements:
    dockerizePip: true

in serverless.yml I cannot use the packages from requirements.txt (results in No module named … found).
When using the lines above in my serverless.yml file the Gitlab CI/CD pipeline fails with:

Error: `docker run --rm -v /root/.cache/serverless-python-requirements/6bd25b0f69b460c8746719510a2cba57ddec9d3d756fd78c74c0a71c07a75600_x86_64_slspyc:/var/task:z -v /root/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z lambci/lambda:build-python3.8 /bin/sh -c chown -R 0\:0 /var/useDownloadCache && python3.8 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 0\:0 /var/task && chown -R 0\:0 /var/useDownloadCache` Exited with code 125

Question: how does my .gitlab-ci.yml have to look like to make the deploy work?

1 Like
2

One way to get the exact error from the packaging process would be to run the exact command on your local machine to see what is “pip install” complaining about.

The times I’ve had that issue were related to Python packages that are being compiled from source + Linux libraries missing on the container image that is running the build. I resorted to creating a custom building image with the required libraries so that Python could complete the packages installation.

3

I’m facing the same issue. As far as I know, you will need both the docker service and docker-cli available in the image you’re running. I’m working on this issue and will post again if I fix it.

4

Facing a similar issue when deploying via Gitlab CI. I can sls deploy to AWS from the CLI without issue.

I added
services: - docker:dind

but still get the eror msg:

Error: Error: spawn docker ENOENT at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19) at onErrorNT (node:internal/child_process:477:16) at processTicksAndRejections (node:internal/process/task_queues:83:21)

Selection from the POC gitlab-ci.yml

variables: 
  DOCKER_HOST: tcp://docker:2375
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""  # https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03/
 
image: node:16.13.2

tages:
  - dev

dev-build-deploy-job:
  stage: dev
  rules: 
    - if: '$CI_COMMIT_BRANCH == "dev"'
  services:
    - docker:dind
  before_script: 
    - echo "Installing packages..."
    - npm ci --cache .npm --prefer-offline --progress=false
    - npm install -g serverless
    - npm --version
    - npm serverless --version 
  script: 
    - serverless deploy --stage dev --verbose
  environment: 
    name: dev/$CI_COMMIT_REF_NAME
5

@bdb2381
I believe that would be because you don’t have docker installed on the node:16.13.2 image.

@Tdsone
I’ve managed to fix my problem with the following config:

# gitlab-ci.yml
image: node:14-stretch

services:
  - docker:18.09-dind

variables:
  DOCKER_TLS_CERTDIR: ''
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2

stages:
  - build

build:
  artifacts:
    name: '$PACKAGE-$CI_COMMIT_REF_NAME'
    paths:
      - packages/${PACKAGE}/build
  script:
    # Install docker
    - apt-get update
    - apt-get install ca-certificates curl gnupg lsb-release -y
    - curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
    - echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
    - apt-get update
    - apt-get install docker-ce=5:19.03.15~3-0~debian-stretch docker-ce-cli=5:19.03.15~3-0~debian-stretch containerd.io -y
    # Run the build script
    - serverless package
  stage: build

# serverless.yml
service: api
frameworkVersion: '2'

custom:
  pythonRequirements:
    dockerizePip: true
    useDownloadCache: false # Not important as far as I know
    useStaticCache: false # Not important as far as I know

plugins:
  - serverless-python-requirements

provider:
  name: aws
  runtime: python3.8  # Important because there isn't lambda-ci docker image for 3.9 as of yet
  region: us-east-1
  stage: ${opt:stage, 'development'}
  lambdaHashingVersion: 20201221

package:
  include:
    - '!./venv'
    - '!./node_modules'

functions:
  healthCheck:
    handler: handler.health_check
    events:
      - http:
          method: GET
          path: /health

The above would most likely work with node:16 and more up-to-date version of docker too, just make sure the versions of installed engine etc. are compatible with the attached service.