No module named 'pydantic_core._pydantic_core'

Serverless Framework
,
1

I am trying to deploy a python lambda with the serverless framework which uses the pydantic module (among other things). I am also using the serverless-python-requirements plugin. I am getting this error:

Running “serverless” from node_modules
{
“errorMessage”: “No module named ‘pydantic_core._pydantic_core’”,
“errorType”: “ModuleNotFoundError”,
“requestId”: “44cde922-0d35-406b-97be-2d02a4e1fc8d”,
“stackTrace”: [
" File "/opt/python/serverless_aws_lambda_sdk/instrument/init.py", line 598, in stub\n return self._handler(user_handler, event, context)\n",
" File "/opt/python/serverless_aws_lambda_sdk/instrument/init.py", line 580, in _handler\n result = user_handler(event, context)\n",
" File "/var/task/serverless_sdk/init.py", line 144, in wrapped_handler\n return user_handler(event, context)\n",
" File "/var/task/s_post_rental_app_from_airtable.py", line 25, in error_handler\n raise e\n",
" File "/var/task/s_post_rental_app_from_airtable.py", line 20, in \n user_handler = serverless_sdk.get_user_handler(‘functions.post_rental_app_from_airtable.handler.lambda_handler’)\n",
" File "/var/task/serverless_sdk/init.py", line 56, in get_user_handler\n user_module = import_module(user_module_name)\n",
" File "/var/lang/lib/python3.10/importlib/init.py", line 126, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n",
" File "", line 1050, in _gcd_import\n",
" File "", line 1027, in _find_and_load\n",
" File "", line 1006, in _find_and_load_unlocked\n",
" File "", line 688, in _load_unlocked\n",
" File "", line 883, in exec_module\n",
" File "", line 241, in _call_with_frames_removed\n",
" File "/var/task/functions/post_rental_app_from_airtable/handler.py", line 9, in \n from pydantic import BaseModel, ValidationError\n",
" File "/var/task/pydantic/init.py", line 3, in \n import pydantic_core\n",
" File "/var/task/pydantic_core/init.py", line 6, in \n from ._pydantic_core import (\n"
]
}

This is my serverless.yml:
[skipping header]

provider:
name: aws

latest supported python by lambda + serverless as of 2023-10-23

runtime: python3.10

NOTE: arm64 may offer better price/performance

architecture: ‘arm64’
stackTags:
Project: reffie

functions:
post_rental_app_from_airtable:
handler: functions.post_rental_app_from_airtable.handler.lambda_handler
events:
- http:
path: /manual-trigger
method: post

plugins:

  • serverless-python-requirements

package:
patterns:
# virtualenv
- ‘!env310/’
# unit tests and their data
- '!tests/’
# local node_modules
- ‘!node_modules/’
# python cache
- '!pycache/’
- ‘!**/*.pyc’
# Mac files
- ‘!.DS_Store’

And my requirements.txt:

annotated-types==0.6.0
certifi==2023.7.22
charset-normalizer==3.3.1
greenlet==3.0.0
idna==3.4
inflection==0.5.1
pyairtable==2.1.0.post1
pydantic==2.4.2
pydantic_core==2.10.1
python-dotenv==1.0.0
requests==2.31.0
SQLAlchemy==2.0.22
typing_extensions==4.8.0
urllib3==2.0.7

I tried both with architecture as x86_64 and as arm64. I read online that the error sometimes happens due to a mismatch between the architecture of the lambda and the architecture of the docker container used to package the python modules. Any indication where to start?

2 Likes
2

I tried adding a custom serverless configuration to go with my lambda:

custom:
pythonRequirements:
# this is necessary to avoid cross-platform build issues
dockerizePip: true
# explicitly pass the arm64 platform to the docker build
dockerImage: public.ecr.aws/sam/build-python3.10:latest-arm64
# explicitly tell pip to fetch the arm64 version of the package
dockerRunCmdExtraArgs: [ ‘–platform’, ‘linux/arm64/v8’ ]

and specified the lambda to use the arm64 architecture. Didn’t solve the issue.

3

Did you figure this out? I’m running into the same issue.

4

@boompig @jonrobinson I had the same error using these dependencies in the requirements.txt:

boto3==1.28.1
boto3-stubs==1.28.68
pydantic==2.4.0
pydantic[email]
pydantic-settings==2.1.0
pytest==7.3.1
pytest-mock==3.11.1
mock==5.0.2
pipreqs==0.4.13
black==23.7.0
backoff==2.2.1
mypy==1.5.1

But there aren’t any dependency issues; the errors occur when you build the dependecy package (in pip install step) for your Lambda in a different architecture . In my case, I use a devcontainer with VSCode with a Docker image that I build:

  1. My first mistake was this:
FROM ubuntu:latest

Basically, I didn’t specify the OS architecture in my Dockerfile. So, I replaced that line:

FROM arm64v8/ubuntu:latest
  1. Second mistake in serverless.yml:
    If you don’t specify the type of architecture that you want to use in your Lambdas, AWS sets the default to x86_64 – 64-bit x86 architecture for x86-based processors. So, I added the default architecture type for all my Lambdas:
provider:
  name: aws
  runtime: python3.10
  architecture: arm64

I also specified the architecture type in the default layer definition:

 layer:
      name: ${self:custom.resources.layers.commons.name}
      description: Python requirements lambda layer
      compatibleRuntimes:
        - python3.10
      compatibleArchitectures: # optional, a list of architectures this layer is compatible with
        - arm64

Finally, I deployed my project again and solved my issue.

Conclusion: You must build your dependency package on the same architecture you use to deploy your Lambdas.

PD: About arm64 architecture vs x86 architecture: Lambda instruction set architectures (ARM/x86) - AWS Lambda

2 Likes
5

Sorry for the late reply. Thanks for the update!

6

Downgrading to pydantic_core==2.10.1 (and then unsetting dependent package versions to avoid conflicts – pydantic version becomes 1.9.0) resolved this issue for me. Other relevant configs I am using:

  • Python version: 3.11
  • Architecture: not specified. Defaults to x86_64 as per Lambda config.
1 Like
7

I encountered the same error and solved it by combining the answers from @ernesrocker and @adobelis. Hope to find a better way soon…

1 Like
8

This config worked for me:

provider:
  name: aws
  runtime: python3.11
  architecture: arm64

requirements.txt

pydantic==1.9.0
pydantic_core==2.10.1