Location: Parker, CO

Job Schedule: Full Time (40 Hours/Week)

Salary: $95,472.00/Annum

Job duties:

• Design and enhance custom ETL frameworks for data migrations, transformations, and integrations using Snowflake, Databricks, and AWS S3
• Develop dynamic tasks to connect with SFTP servers and copy extract files to and from AWS S3.
• Implement dynamic API configurations for REST API integration, supporting incremental data extraction, authentication, and multithreaded processing.
• Build functionalities to process encrypted data using GPG/PGP keys, ensuring secure handling of sensitive information during transfers.
• Design parallel query execution and implement features for error logging and metrics capture during Snowflake curation operations.
• Integrate the Great Expectations library into frameworks for robust data validation, including file size checks, data type enforcement, and custom rule processing.
• Design and implement capabilities to read and write data from and to Kafka, including support for schema transformations and real-time data streams.
• Automate the handling of large files (10GB+), including encryption, custom headers/footers, and multi-sheet output for enhanced usability using Boto3 streams.
• Develop multithreaded processes for archival tasks and data processing to improve performance and scalability.
• Create advanced watermarking features to support incremental and historical data processing, ensuring accurate data tracking.
• Build deployment validation scripts to synchronize configurations, validate cluster setups, and ensure environment compliance.
• Develop user-defined functions (UDFs) in Spark for tasks like geolocation extraction and dynamic parameter handling.
• Implement observability tools to monitor ELT jobs and address issues in regression and production environments.
• Design global variables and reusable configurations to standardize processes across multiple data products and pipelines.

Minimum Education Requirement: This position requires minimum of Bachelor’s degree in computer science, computer information systems, information technology, or a combination of education and experience equating to the U.S. equivalent of a Bachelor’s degree in one of the aforementioned subjects.