Updating README
This commit is contained in:
236
README.md
236
README.md
@@ -1,5 +1,239 @@
|
||||
# Karaoke tracks from normal songs/videos
|
||||
|
||||
# Welcome to your CDK Python project!
|
||||
I'm going to play with [spleeter][https://github.com/deezer/spleeter]
|
||||
to explore ideas around home/portable karaoke station for modern times.
|
||||
|
||||
It works! I can obtain an mp3 file with only the music, ready to sing over.
|
||||
|
||||
## Putting it on the cloud
|
||||
|
||||
To turn this into anything useful it would be nice to have the following:
|
||||
|
||||
- a simple domain (like deepkaraoke.me)
|
||||
- an intuitive interface. You drop a file or a link to a video and the process starts.
|
||||
When it's done it turns into a link, ready to share.
|
||||
- the conversion process should run on a serverless engine (AWS possibly)
|
||||
- the frontend should be distributed via a CDN
|
||||
|
||||
After a failed attempt in packaging spleeter for lambda, I looked for viable alternatives.
|
||||
|
||||
The most promising solution for running a custom docker container on demand (scaling from and to 0 instances), is AWS Batch.
|
||||
|
||||
1. You need to *define a job*
|
||||
- a name
|
||||
- a role
|
||||
- container image
|
||||
- command
|
||||
- vcpus (1 vcpu is equivalent to 1024 CPU shares)
|
||||
- memory (hard limit)
|
||||
- job attempts
|
||||
- parameters (if used in command definition)
|
||||
- environment variables
|
||||
|
||||
2. Configure the *compute environment* and *job queue*
|
||||
|
||||
you submit your jobs to a job queue that stores jobs until the AWS Batch scheduler runs the job on a compute resource within your compute environment.
|
||||
|
||||
- compute environment name, specify a unique name for the compute environment
|
||||
- for Service Role, choose to create a new role that allows the AWS Batch service to make calls to the required AWS APIs on your behalf. (AWSBatchServiceRole is required).
|
||||
- a job queue name
|
||||
|
||||
I've managed to build my simple collaboration of components for this kind of tasks, using AWS Batch.
|
||||
The service allows to run batch jobs with a custom docker image, submitting the job instances with a simple API call.
|
||||
I'm using docker hub, since my project will be open source. A realistic example should include ECR (a docker privare registry) for hosting images.
|
||||
|
||||
Now I need a lambda function to submit a new job for each new audio file being created in the input S3 bucket.
|
||||
|
||||
S3 bucket -> event -> trigger_function -> AWS Batch job submitted
|
||||
|
||||
### What does not work with this approach
|
||||
|
||||
AWS Batch imposes *strict* requirements for memory allocation. If your process exceeds
|
||||
that memory amount, the job gets instantly killed.
|
||||
You can solve this allocating more resources in the computing environment, but I've noticed that doing so, the job will be less likely to be selected for execution in a
|
||||
timely fashion (my experiments easily took more than half an hour to start working).
|
||||
|
||||
While AWS Batch represents a valid solution for pure batch workloads, I was thinking
|
||||
the conversion process something that would start as soon as the user uploads a new
|
||||
file, and that runs in background, like many lengthy tasks that web applications
|
||||
offload out of process to something else.
|
||||
|
||||
This could be an AWS Fargate Service, which again will be running our spleeter based
|
||||
process within a Docker container. The container this time will run continuously based
|
||||
on a Fargate Task definition, which will automatically scale the number of instances
|
||||
when required.
|
||||
|
||||
The complexity of provisioning the required computing resources to properly run a
|
||||
container on Fargate is not trivial.
|
||||
That would certainly be another job for Terraform, but I got caugth by this great
|
||||
live demo, of building the same type of application, using AWS Cloud Development Kit.
|
||||
|
||||
Think about it as a typed *Python* wapper on top of CloudFormation (in reality the Python support is achieved through a language level binding over the Typescript implementation), which is able to generate CloudFormation templates, resolving all references and reusing stacks as you would do with your code libraries.
|
||||
|
||||
### CDK based solution
|
||||
|
||||
#### The worker stack
|
||||
|
||||
A `QueueProcessingFargateService` could be used to implement a separator worker, which will get items
|
||||
(an S3 url to a file) from a job queue.
|
||||
|
||||
Files will be read from an input S3 bucket and output written back to an output bucket.
|
||||
Worker pool will automatically scale when needed.
|
||||
|
||||
#### The HTTP API
|
||||
|
||||
As a frontend, we need an HTTP API, to perform at least the following operations:
|
||||
|
||||
- push a new job and return an id
|
||||
- get the status of a job, given it's id
|
||||
|
||||
In my mind, this is more a job for a lambda function (or a group of lambda functions), especially in 2020, but in
|
||||
order to get a first working version sooner, I would stick to a daemon style container, exposed to the internet
|
||||
using an Application load balancer, which happen to be directly supported by the high level patterns library
|
||||
included in CDK.
|
||||
|
||||
An `ApplicationLoadBalancedFargateService` stack would do the trick then.
|
||||
|
||||
##### Installing CDK
|
||||
|
||||
CDK command line is needed during local development and later will be needed on CircleCI.
|
||||
I installed the CDK command line (which is based on NodeJS) using:
|
||||
|
||||
brew install aws-cdk
|
||||
|
||||
To test it worked, I run:
|
||||
|
||||
cdk --version
|
||||
1.32.2 (build e19e206)
|
||||
|
||||
##### AWS credentials
|
||||
|
||||
I will use the same credentials from my personal AWS account to use CDK.
|
||||
I created a new user `aws-cdk` within my personal account and gave him the following permissions:
|
||||
|
||||
- SystemAdministrator
|
||||
- AWSCloudFormationFullAccess
|
||||
|
||||
As a personal memo, I will use the new `personal-cdk` profile (in `~/.aws/credentials`).
|
||||
|
||||
It's *important* to export the following environment variable:
|
||||
|
||||
export AWS_PROFILE=personal-cdk
|
||||
|
||||
##### Creating the CDK project
|
||||
|
||||
I had to create a new empty directory to successfully run:
|
||||
|
||||
cdk init app --language python
|
||||
|
||||
which conveniently created a blank project which includes a local virtualenv.
|
||||
To activate the virtual environment for the project, issue the following command:
|
||||
|
||||
source .env/bin/activate
|
||||
|
||||
and then, to install the required dependencies:
|
||||
|
||||
pip install -r requirements.txt
|
||||
|
||||
|
||||
##### Bootstraping the CDK toolkit
|
||||
|
||||
CDK toolkit need some resources to run its lifecycle commands.
|
||||
An S3 bucket and a CloudFormation stack will be created performing the following
|
||||
command:
|
||||
|
||||
cdk bootstrap
|
||||
|
||||
Ideally this command should be run only once.
|
||||
This is how it went for me:
|
||||
|
||||
(.env) AWS: personal-cdk domenico@domecbook ~/dev/experiments/karaokeme-cdk master ● cdk bootstrap
|
||||
<aws:personal-cdk>
|
||||
⏳ Bootstrapping environment aws://521097367553/eu-west-1...
|
||||
CDKToolkit: creating CloudFormation changeset...
|
||||
0/3 | 12:28:08 PM | CREATE_IN_PROGRESS | AWS::S3::Bucket | StagingBucket
|
||||
0/3 | 12:28:09 PM | CREATE_IN_PROGRESS | AWS::S3::Bucket | StagingBucket Resource creation Initiated
|
||||
1/3 | 12:28:31 PM | CREATE_COMPLETE | AWS::S3::Bucket | StagingBucket
|
||||
1/3 | 12:28:33 PM | CREATE_IN_PROGRESS | AWS::S3::BucketPolicy | StagingBucketPolicy
|
||||
1/3 | 12:28:34 PM | CREATE_IN_PROGRESS | AWS::S3::BucketPolicy | StagingBucketPolicy Resource creation Initiated
|
||||
2/3 | 12:28:34 PM | CREATE_COMPLETE | AWS::S3::BucketPolicy | StagingBucketPolicy
|
||||
3/3 | 12:28:35 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | CDKToolkit
|
||||
|
||||
After that I've configured my Visual Studio Code with the AWS Toolkit
|
||||
extension, which is really useful and easy to use.
|
||||
|
||||
#### Coding the infrastructure with CDK
|
||||
|
||||
I've started with refactoring the application code as suggested
|
||||
in the demo video.
|
||||
|
||||
There will be shared resorces, and possibly references between the
|
||||
stacks. For this reason the Application object can hold a reference
|
||||
to the other component stacks.
|
||||
|
||||
After some coding (pretty east to be honest), it was time to deploy, which I did with the following command:
|
||||
|
||||
cdk deploy '*'
|
||||
|
||||
Using two simple placeholder containers and after a couple of iterations
|
||||
where I had to destroy the stacks (some missing permissions for the AWS
|
||||
user), it worked as expected.
|
||||
|
||||
A very nice feeling.
|
||||
|
||||
During the next iteration I will need to perform the "actual" work that my
|
||||
projects need to do.
|
||||
|
||||
The real API container must be able to enqueue a new job, both writing to
|
||||
a DynamoDB table the entry and pushing it to the processing queue monitored
|
||||
by the worker pool.
|
||||
|
||||
The worker, is also able to update the DynamoDB table to reflect the job
|
||||
status in the long-term storage.
|
||||
|
||||
#### Refining the architecture
|
||||
|
||||
The *HTTP API* will expose the following operations:
|
||||
|
||||
|
||||
POST /jobs -> {job_id: 4j6h5k6e5h6jke5hjhj5, status: "uploading"}
|
||||
|
||||
it will perform the following things:
|
||||
|
||||
- generate a unique ID and a unique `s3_key`
|
||||
- return a pre-signed URL to upload a file to `s3_key`
|
||||
- add a row to the DynamoDB table including the `job_id`, the `s3_key` and the status `uploading`
|
||||
|
||||
At this point it's possible for the caller to upload a file using the pre-signed URL
|
||||
returned by the first request.
|
||||
|
||||
Once the upload has completed client will call
|
||||
|
||||
POST /jobs/:id/process
|
||||
|
||||
This will trigger the processing, performing the following:
|
||||
- read the object from the DynamoDB table
|
||||
- add an object to the processing queue, including the `s3_key` and the `job_id`
|
||||
- update the DynamoDB row with the status `processing`
|
||||
|
||||
GET /jobs/:id
|
||||
will return the job status or 404
|
||||
|
||||
GET /jobs
|
||||
will return the full list of the jobs
|
||||
|
||||
I could quickly implement it with Flask and boto3.
|
||||
Of course I've omitted a number of things for a production ready
|
||||
setup, like running flask on a WSGI container and didn't write
|
||||
any tests yet (which I'm confident can be arranged with pytest and moto/localstack).
|
||||
|
||||
The *worker cluster* will need to slightly adjust the one
|
||||
I wrote for the terraform version.
|
||||
Some remarks:
|
||||
- I had to add a polling mechanism to peek for messages from
|
||||
the associated processing queue
|
||||
|
||||
### CDK Python instructions
|
||||
|
||||
This is a blank project for Python development with CDK.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user