# Aws-python function dependencies load

**URL:** <https://forum.serverless.com/t/aws-python-function-dependencies-load/451>\
**Category:** Serverless Framework\
**Created:** [October 4, 2016, 6:23pm UTC](https://forum.serverless.com/t/aws-python-function-dependencies-load/451 "2016-10-04T18:23:24Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![el0ck](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.serverless.com/el0ck/32/1781_2.png) [@el0ck](https://forum.serverless.com/u/el0ck)\
**Post date:** [October 6, 2016, 2:09am UTC](https://forum.serverless.com/t/aws-python-function-dependencies-load/451/4 "2016-10-06T02:09:59Z")

</div>

Dont give up @mpradilla!

This is completely achievable on lambda. It is not necessary to manually zip your package first. Let me describe the way I use lambda to run python code with external python modules that leverage precompiled libraries:

* * *

# External Modules

Firstly, when serverless zips your lambda, it follows symlinks. As such, one can symlink the virtual environment’s `site-packages` into the function directory. That way it doesn’t clutter the `lib/` directory, which can be used solely for your precompiled libraries. In your case numpy’s `multiarray.so`.

### File structure

```bash
▾ lib/
    some_lib.so
  event.json
  handler.py
  serverless.yml
  site-packages -> /Users/jimbo/miniconda2/envs/testenv/lib/python2.7/site-packages

```

For local testing, you run the code **OUTSIDE** your virtual environment to ensure the manual imports (below) are working:

```python
import sys
import os
here = os.path.dirname(os.path.realpath( __file__ ))

# Import installed packages (in site-packages)
site_pkgs = os.path.join(here, "site-packages")
sys.path.append(site_pkgs)

# Now Import your environment packages
import numpy as np
...

```

# Compiled code

## A simple case

As you have found, the [lambda execution environment](http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html) is a linux 64 bit architecture. You are not developing on this so your `multiarray.so` is different, hence your ‘_invalid ELF header_’ error.

You problem is solved by developing on a linux-64 machine. Thus when you install `scikit-learn` and its dependency `numpy`, you will get the shared object files that can be used by lambda.

> It is worth noting for other readers that this error may report in cloudwatch logs as simply an `ImportError`

> `ImportError: No module named numpy`.

So far you know all this, however at some point you may run into the followin case.

## Non site-package Libraries

Sometimes you may require libraries from other directories that are on your local `LD_LIBRARY_PATH` but don’t get zipped up with the lambda function.

### Example

I have lambda code that requires the intel math kernel libraries. Here are my problems:

1. Libraries are not in `site-packages`
2. `LD_LIBRARY_PATH` is read when the python interpreter starts so changing it with `os.environ` in code is too late
3. Some library files are large. AWS lambda zip files have [limits](http://docs.aws.amazon.com/lambda/latest/dg/limits.html) and run faster (and cheaper) when they are small.

### The solution

I need to include only the libraries I need, not entire lib directories. The directory where I put the libraries must already be on the `LD_LIBRARY_PATH` when my lambda starts. This restricts the directory to the `lib/` beside my handler.

I run the function in the lambda execution environment and cherry pick the libraries as they error out and add them to `lib/` thus ensuring I have the bare minimum libraries I need.

Now that this is in the service repository I can continue to develop on my mac because the linux-64 binaries are deployed.

---

_[View the full topic](https://forum.serverless.com/t/aws-python-function-dependencies-load/451)._