I expect the answer is very similar to per stage environment variables in which case it will be something like:
custom:
stage: "${opt:stage, self:provider.stage}"
plugins:
dev:
- serverless-offline
prod:
plugins: ${self.custom.${self.custom.stage}.plugins}
Note: This hasn’t been tested so it may require a little debugging