Lead the development of data pipelines and transformations in Azure Databricks, converting Scala programs to PySpark while leveraging various Azure technologies.
Job Description :
Location - US (remote) . East coast preferable
Primary Skill :
Technical Lead -Data Engneering
- Azure databricks- Build Data Pipelines using ADB-Leveraging Python and Spark
- Build Data Transformation in ADB
- Convert Scala Pgms to Py Spark Pgms
- Understand the SCALA program in DataBricks
- ADF
- Secondary Skills:
- Azure data lake gen 2
- Synapse analytics (dedicated SQL pool)
- Exp in delta Lake
- Pyspark
Nice to Have : Tidal Scheduler , Exp in Healthcare Domain
Similar Jobs
Insurance
Leads engineering support and architecture for enterprise-scale healthcare data platforms on Azure. Designs Databricks, Synapse, Data Factory, and Data Lake solutions; builds batch and real-time pipelines, analytical data models, BI solutions, and automated data quality frameworks. Implements machine learning and AI-assisted development, integrates low-code systems, ensures HIPAA compliance, and translates business needs into technical solutions. Mentors engineers, resolves technical issues, supports Agile delivery, and partners with healthcare business leaders on Medicare and Medicaid analytics initiatives.
Top Skills:
Arm TemplatesAzure Data FactoryAzure Data Lake StorageAzure DatabricksAzure DevopsAzure Event HubsAzure Machine LearningAzure SynapseGithub CopilotHipaaKafkaAzureMicrosoft O365PegaPower BIPysparkPythonSalesforceScalaSitecoreSpark StreamingSQLTableauTerraform
Information Technology • Consulting
Lead development of enterprise data and analytics systems using Snowflake and Python. Build and optimize data pipelines on AWS (Lambda, Step Functions, ECS), write advanced SQL, and apply domain knowledge of asset management and financial markets.
Top Skills:
Amazon EcsAws LambdaAws Step FunctionsPythonSnowflakeSQL
Information Technology • Consulting
Lead data engineer responsible for designing, building, and operating production data pipelines using Python, Airflow, Snowflake, DBT, and AWS. Automate ETL, implement CI/CD, testing and DBT validations, troubleshoot pipelines, and work client-facing to deliver reliable data solutions.
Top Skills:
Ai IdeApache AirflowAws EcsAws IamAws S3Ci/CdCsvDbtGitJinjaJSONPythonSnowflakeSQLXML
What you need to know about the Charlotte Tech Scene
Ranked among the hottest tech cities in 2024 by CompTIA, Charlotte is quickly cementing its place as a major U.S. tech hub. Home to more than 90,000 tech workers, the city’s ecosystem is primed for continued growth, fueled by billions in annual funding from heavyweights like Microsoft and RevTech Labs, which has created thousands of fintech jobs and made the city a go-to for tech pros looking for their next big opportunity.
Key Facts About Charlotte Tech
- Number of Tech Workers: 90,859; 6.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lowe’s, Bank of America, TIAA, Microsoft, Honeywell
- Key Industries: Fintech, artificial intelligence, cybersecurity, cloud computing, e-commerce
- Funding Landscape: $3.1 billion in venture capital funding in 2024 (CED)
- Notable Investors: Microsoft, Google, Falfurrias Management Partners, RevTech Labs Foundation
- Research Centers and Universities: University of North Carolina at Charlotte, Northeastern University, North Carolina Research Campus
