Import local modules Error when in higher folder using serverless-python-requirements plugin

Serverless Framework
1

Summary of Issue:
We are using servereless frame work with runtime python3.7 we are using the plugin servereless-python-requirements. We are creating local python modules to have helper classes. These modules are imported and used by lambda functions and used by other local python module methods that are not lambda functions. We are doing this for testability and cleanliness. These local python modules when called by a lambda function works only when the serverelss.yml is configured with,‘module’ and ‘include’(see .yml below). Our issues (see error message below) arrises when we have a lambda function call a local python module function, that is not a lambda function, and that python module function tries to import a local python module outside of its own folder. The error you will view below occurs when the lambda function ‘marketoOrgCreate’ calls a non-lambda method in the connection.py folder which is trying to import a local class payload.

What I have included in this request for help
Below you will find the error message, a sample of the folder structure of the project, a sample of the serverless.yml, and samples of the folders with local imports. Thank you for your thoughts and direction in advance.

Error:

$ sls invoke -f marketoOrgCreate -l
{
    "errorMessage": "attempted relative import beyond top-level package",
    "errorType": "ValueError", 
   "stackTrace": [         ... <lots of read out> ....
    "  File \"/var/task/marketo/connection.py\", line 11, in <module>\n    from ..resource.payload 
       import Payload\n"
    ]
}

Folder Structure Sample:

publication-service-2
  publication-service
       __init__.py
       serverless.yml
       requirements.txt
       lambdaFunc
          organization.py
          resource
               __init__.py
               payload.py
               requirements.txt
           marketo
                __init__.py
                connection.py
                requirements.txt

servereless.yml (with private stuff removed)

service: publication-service
app: publication-service-app

provider:
  name: aws
  runtime: python3.7
  profile: serverless-admin
  region: us-east-2
  environment:
    KMS: <some long kms>
  iamRoleStatements:
    - Effect: 'Allow'
      Action: 
        - 'ssm:GetParameters'
        - 'ssm:GetParameter'
        - 'ssm:PutParameter'
        - 'ssm:DeleteParameters'
      Resource: <some long arn for our ssm>
    - Effect: 'Allow'
      Action:
        - 'kms:Encrypt'
        - 'kms:Decrypt'
      Resource: <some long arn for our kms>
    - Effect: Allow
      Action: lambda:InvokeFunction
      Resource: "*"
plugins:
  - serverless-python-requirements
  - serverless-offline-python
custom:
  pythonRequirements:
    dockerizePip: true

package:
  individually: true
  exclude:
    - .circleci/**
    - .pytest_cache/**
    - .coverage
functions:
  marketoOrgCreate:
    handler: organization.create
    module: publication-service/lambdaFunc
    include:
      - publication-service/lambdaFunc/organization.py

Sample organization.py import that does work I believe because organization holds the lambda function

from resource.payload import Payload

Sample import statement that is causing error message from
publication-service/marketo/connection.py:

#python AWS SDK library
import boto3

#local
from ..resource.payload import Payload
2

I found the solution:
I was missing a keyword ‘package’ above the ‘include’ statement in servereless.yml and that enables me to include the local modules. Below you will see the updated servereless.yml and the updated import statement in connection.py. These are the only changes needed. Note that the package include statement can be put at the function level or at the package level depending on how many of your lambda functions should have access to the included local python modules. I add my change at the function level.

servereless.yml

service: publication-service
app: publication-service-app

provider:
  name: aws
  runtime: python3.7
  profile: serverless-admin
  region: us-east-2
  environment:
    KMS: <some long kms>
  iamRoleStatements:
    - Effect: 'Allow'
      Action: 
        - 'ssm:GetParameters'
        - 'ssm:GetParameter'
        - 'ssm:PutParameter'
        - 'ssm:DeleteParameters'
      Resource: <some long arn for our ssm>
    - Effect: 'Allow'
      Action:
        - 'kms:Encrypt'
        - 'kms:Decrypt'
      Resource: <some long arn for our kms>
    - Effect: Allow
      Action: lambda:InvokeFunction
      Resource: "*"
plugins:
  - serverless-python-requirements
  - serverless-offline-python
custom:
  pythonRequirements:
    dockerizePip: true

package:
  individually: true
  exclude:
    - .circleci/**
    - .pytest_cache/**
    - .coverage
functions:
  marketoOrgCreate:
    handler: organization.create
    module: publication-service/lambdaFunc
    package:
        include:
            - publication-service/lambdaFunc/organization.py
            - publication-service/lambdaFunc/resource/payload.py

connection.py

from marketo.resource.payload import Payload
1 Like
3

Does it work with local invoke? I designed my packages similarly to yours, however when I do local invoke doesn’t find the packages. Did you test it with the --docker option? If yes how do you pass custom events?
Thanks.

4

This is a known issue, with a stalled PR and a couple of hacky workarounds: https://github.com/UnitedIncome/serverless-python-requirements/issues/520