Apache beam with hop, java 11, load from excel to Pojo to API, read & write zips. With throttle , pause & resume after x rows

need someone who has used and can set up on AWS 2 node cluster, apache beam with hop and create two pipelines with java code.

1. load data, input will be excel (with 3 columns having file names that are relative paths to 1 of 2 zips) for simplicity the zips and excel will be on a shared path that both nodes can access. This job has to be configured so that each row is part independant task and can happen seperately/ resume etc

2. read a csv, call a few apis to get attachments referred in the csv (pdf, images) save and zip those files. This is a cumulative job: all rows of csv have to be processed and then zip (or zips depending on number and size of files) has to be made as last step. Logic to save file, create zip will be provided

Need help with setting up AWS environemnt on ec2 nodes

Programming in Java to set up this cluster

Divide and monitor jobs between the 2 nodes

To sign NDA before work so can share some company private swagger info

Java 11 or 17 with maven or gradle (prefer gradle)

Java Amazon Web Services สถาปัตยกรรมซอฟต์แวร์

หมายเลขโปรเจค: #34002445

เกี่ยวกับโปรเจกต์