Job Description
Data Factory Services Lead Role Summary
The Data Factory Services Lead is a senior, hands-on onshore technical leader responsible for designing, engineering, deploying, supporting, and continuously improving enterprise data platforms and pipelines. The role owns the complete technical flow across AWS services, workflow orchestration, distributed data processing, query and analytics platforms, CI/CD, observability, and production support. The lead will provide technical direction to distributed teams, conduct design and code reviews, resolve complex production issues, and ensure solutions are secure, scalable, reliable, maintainable, and operationally ready.
Key Responsibilities
- Lead architecture, design, development, enhancement, deployment, and production support of enterprise data factory services and data pipelines.
- Remain actively hands-on in Python and SQL development, code review, debugging, performance tuning, automation, and incident resolution.
- Design and support AWS solutions using Amazon S3, AWS CodeBuild, Amazon Athena, Amazon CloudWatch, and AWS Secrets Manager.
- Build, schedule, monitor, and troubleshoot Apache Airflow workflows, including DAG design, dependencies, retries, alerting, backfills, recovery, and operational controls.
- Use Pepperdata for workload visibility, performance analysis, capacity insights, and optimization of distributed data workloads.
- Design, develop, optimize, and support Databricks workloads for scalable data engineering, transformation, and analytics use cases.
- Use Dremio to enable governed, high-performance access to enterprise data, including semantic-layer design, query optimization, and issue resolution.
- Develop efficient SQL for ingestion, transformation, reconciliation, analytics, validation, and operational reporting across large datasets.
- Create and maintain CI/CD automation through GitHub and AWS CodeBuild, including source control, branching, pull requests, build validation, release controls, and rollback readiness.
- Use Visual Studio Code as a primary engineering environment for Python, SQL, configuration, testing, debugging, and repository-based development.
- Implement monitoring, logging, alerting, and operational dashboards using Amazon CloudWatch and platform-native capabilities.
- Act as the senior escalation point for data, workflow, cloud, query, performance, deployment, and production incidents; lead root-cause analysis and permanent corrective actions.
- Guide offshore engineers through technical design, coding standards, peer reviews, troubleshooting, deployment readiness, documentation, and knowledge transfer.
- Partner with application, data, cloud, DevOps, security, testing, suppor t, architecture, and business stakeholders to deliver dependable services.
- Maintain architecture notes, runbooks, SOPs, dependency maps, production support procedures, release records, and operational knowledge artifacts.
Required Technical Skills - All Mandatory
- AWS Cloud: Amazon S3, AWS CodeBuild, Amazon Athena, Amazon CloudWatch, and AWS Secrets Manager. Candidate must be able to design, configure, troubleshoot, monitor, secure, and support these services in production.
- Workflow Orchestration: Apache Airflow: DAG engineering, scheduling, dependencies, retries, backfills, monitoring, alerting, failure recovery, and production troubleshooting.
- Production Engineering: Observability, secure secrets handling, incident management, root-cause analysis, performance tuning, release validation, reliability, and operational readiness.
- Performance Optimization: Pepperdata: workload monitoring, performance analysis, capacity insights, troubleshooting, and optimization of distributed data workloads.
- Data & Analytics Platforms: Databricks and Dremio: hands-on data engineering, integrations, query performance, semantic access, platform troubleshooting, and production support.
- Programming & Query Engineering: Python and SQL: advanced development, automation, testing, debugging, optimization, data validation, and scalable processing.
- Toolchain: GitHub and Visual Studio Code: repository management, branching, pull requests, reviews, version control, debugging, and collaborative development.
Leadership and Collaboration Skills
- Demonstrated ability to lead through hands-on technical contribution, sound engineering judgment, and end-to-end accountability.
- Experience leading distributed onshore and offshore delivery teams and translating business and technical requirements into executable plans.
- Ability to conduct architecture, code, data pipeline, SQL, security, deployment, and operational-readiness reviews.
- Strong stakeholder communication, analytical problem-solving, decision-making, prioritization, and escalation-management skills.
- Ability to coordinate critical production incidents, planned implementations, post-release validation, and service stabilization.
- Ability to mentor engineers, enforce engineering standards, remove technical blockers, and build sustainable operational knowledge.
Required Qualifications
- Minimum 10 years of relevant technology experience, with recent experience in a senior technical lead or comparable hands-on leadership role.
- Demonstrable production experience across all required technologies : AWS S3, CodeBuild, Athena, CloudWatch, Secrets Manager, Airflow, Pepperdata, Databricks, Dremio, Python, SQL, GitHub, and Visual Studio Code.
- Experience owning enterprise data services through design, build, release, monitoring, support, remediation, and continuous improvement.
- Willingness and ability to work from the San Antonio or Irvine office Tuesday through Thursday under the stated hybrid model.
Preferred Experience
- Experience delivering enterprise data platforms in regulated or audit-sensitive environments.
- Knowledge of data governance, access control, secure engineering, platform reliability, release governance, and operational controls.
- Experience supporting high-volume or business-critical data services with formal incident, problem, and change-management practices.
Salary Range- $110,000-$140,000 a year
Location
Irvine, CA
Function
TECHNOLOGY
Role
Technical Lead
ID
437486
Skills
AWS
Salary Range
$110,000 - $140,000 |
Tata Consultancy Services
$110,000 - $140,000
Oct 28, 2026