Pipeline Engineer - Data & Application Pipeline Modernization

Area(s) of responsibility

Job Title: Pipeline Engineer – Rewrite & Deployment

Role Summary

Responsible for redesigning, rewriting, and deploying data/application pipelines to improve scalability, performance, and reliability. The role involves working closely with development, QA, DevOps, and business teams to ensure seamless delivery and deployment of modernized pipelines.

Key Responsibilities

• Analyze existing pipelines (ETL/data/process workflows) and identify gaps, inefficiencies, and improvement areas
• Design and rewrite pipelines using modern frameworks, tools, and best practices
• Optimize pipelines for performance, scalability, and fault tolerance
• Implement automated deployment processes using CI/CD pipelines
• Ensure version control, code quality, and proper documentation of rewritten pipelines
• Perform unit, integration, and system-level validation prior to deployment
• Collaborate with QA teams for validation and testing strategy alignment
• Monitor deployed pipelines and troubleshoot issues post-deployment
• Ensure compliance with security, governance, and data standards
• Work with stakeholders to plan cutover, migration, and rollback strategies

Required Skills & Experience

• Strong experience in pipeline development (ETL/data pipelines or application pipelines)
• Hands-on experience with tools such as Azure Data Factory, AWS Glue, Apache Airflow, Jenkins, or similar
• Proficiency in programming/scripting (Python, SQL, Shell, etc.)
• Experience with CI/CD tools and deployment automation
• Knowledge of cloud platforms (Azure/AWS/GCP)
• Strong debugging and performance optimization skills
• Familiarity with version control systems (Git)
• Understanding of DevOps and Agile methodologies

Preferred Qualifications

• Experience in pipeline migration or legacy system modernization
• Knowledge of containerization (Docker/Kubernetes)
• Exposure to monitoring/logging tools (Splunk, ELK, etc.)
• Experience working in distributed or global teams

Key Deliverables

• Fully rewritten and optimized pipelines
• Deployment-ready artifacts with CI/CD integration
• Reduced execution time / improved efficiency metrics
• Stable and monitored production deployment

Soft Skills

• Strong analytical and problem-solving skills
• Effective stakeholder communication
• Ability to handle multiple priorities and deadlines
• Ownership and accountability for end-to-end delivery