Hi folks, I’m trying to trigger a lambda function based on file creation within a S3 bucket.
Once the lambda is triggered, my py3 program needs to process the file which is in parquet
format. So the dependent libraries are fastparquet, numpy, pandas (which exceed 250M) limit.
So I have slimmed, zipped down these dependencies in the yml file but still seeing errors.
My yml file looks like below and since I’m developing on Mac , I need to use docker to deploy lambda
coz of compiled .so for the numpy/pandas.
frameworkVersion: “>=1.34.0”
custom:
pythonRequirements:
layer: true
slim: true
dockerizePip: non-linux
pipCmdExtraArgs:
- “–proxy”
- “http://docker.for.mac.localhost:1942”
But the error I’m getting is this:
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-p173_lgn/fastparquet/ You are using pip version 19.0.2, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
Wonder what can be going on here? Btw, I’m deploying this behind proxy.