Need Spark Advice

Hi, hope you are fine. I have gone through your job description. You have two issues, one is you don't want to hardcode coalesce other is you want to add new attribute with respect to file extracted. I'm a Big Data Engineer with more than 4 years of experience and I can help you in solving above two issues. Let's discuss on chat. Thanks,

I am a data scientist and a big data developer with more than 4 years of experience in developing solutions for the client using advance technologies like Machine Learning, Deep Learning, Apache Spark and Spark Streaming using Python and R. Moreover I have hands on experience in deploying production ready models using cloud services like AWS and Azure. I have knowledge in developing complete data science life cycles which includes Data Collection, Data Wrangling, Feature Engineering, Data Modeling and Data Visualization. Below are the technologies i have experience in - Machine Learning - Supervised and Unsuperived ML Models Big Data - Spark and Hadoop Cloud Tech - AWS Datapipelines, AWS EMR, AWS Aurora DB, Amazon Redshift, AWS Lambda and more, same is with Azure. Orchestration Tools - Apache Airflow, Luigi, Ansible, CronJobs, etc. Reporting Tech - PowerBI and Tableau Programming Languages - Python, R and Scala. I can share with you my previous work in the related fields. I would like to know more about your project/assignment. Thanks.