Job Description
This is a remote position.
About the Job
As a Senior Data Engineer at Aspire, will play a key role in designing, developing, and maintaining data workflows and pipelines, ensuring the efficient and reliable movement of data across various systems.
What you’ll do
- Design, implement, and manage complex DAGs and SubDAGs in Apache Airflow.
- Utilize Operators, macros, connections, variables, and xcom for efficient task execution.
- Conduct unit testing to ensure the robustness of Airflow workflows.
- Perform file operations and data movement between HDFS and S3 (and vice versa).
- Manage Hive external tables, update table DDL, and handle partition operations.
- Execute basic Hadoop commands and navigate/traverse HDFS.
- Work with Hive on Tez for optimized query execution.
- Utilize AWS command line tools for tasks such as listing and copying files from S3.
- Implement and manage secrets using tools like Ansible Vault and AWS Secrets Manager.
- Develop and optimize data processing tasks using PySpark.
- Understand and leverage Livy for efficient Spark job execution.
- Import and export data between Microsoft SQL Server and HDFS using Sqoop.
- Apply dimension modeling principles, define facts and dimensions, and manage surrogate keys.
- Implement effective branching strategies in Git.
- Participate in the process of raising and reviewing Merge Requests in GitLab.
- Utilize SSIS for migrating data workflows, ensuring a smooth transition.
- Model data in Snowflake, differentiating between external and internal tables.
- Understand the use of views and materialized views in Snowflake.
What you’ll need
- Bachelor’s degree in computer science, information technology, or a related field.
- Proficient in Python (both Python 2.7 and 3.6+).
- Experience in unit testing and test-driven development.
- Familiarity with Secrets Management tools and techniques.
- Knowledge of data modeling principles and practices.
- Strong understanding of Git version control and GitLab workflows.
- Experience with SSIS and Snowflake is a plus.
- Excellent problem-solving and communication skills.
Why Aspire
In addition to a competitive long-term total compensation with salary and performance-based bonus, we have a reward philosophy that expands beyond this.
- Be part of a (Remote is here-to stay) organization.
- Work and learn from great minds.
- Explore new opportunities to learn and grow everyday by attending technical and nontechnical training.
- Get market exposure by working with international tech leaders.
- Nursery reimbursement benefit.
- Aspire Wellness Program.
- Attend virtual and onsite international tech conference.