There are two databases you might use inside RDS: Aurora (which implements mysql 5.6 functionality), and Mysql (which is Mysql 5.7, and hence has things like JSON columns).
I am using both, and both perform very well. Aurora is more self-managing, in that it deals with replication and growing of the database automatically.
Cold starts are not an issue: your application just opens a new connection to the database, just like any other database.
I run RDS inside a VPC, so no direct access from the public Internet is possible, and I also run lambda inside a VPC private subnet. The only tricky bits I had to deal with were:
Having to build a NAT Gateway so that the Lambda function had outbound Internet connectivity (this costs money too, unfortunately, but I wanted all Lambda functions to make outbound connectivity from a fixed IP address)
Configuring an S3 policy to allow access to S3 from the private subnet