We are looking for a hands-on, high-agency Lead Data Engineer to take ownership of the data infrastructure at SnapScan. With a team of 50 people, we operate with a startup mentality, prioritising speed, technical excellence, and innovation over bureaucracy.
We are not looking for a manager to oversee vendors; instead, we need a builder. You will serve as the primary architect of our data stack, responsible for migrating, maintaining, and innovating our data pipelines. You will mentor and champion a data-driven culture, with your primary focus on engineering robust and cost-effective solutions by striking the right balance between open-source and paid-for technologies.
This essential role reports directly to the Head of Engineering and offers an excellent opportunity to engage in various projects, with the potential to grow into more strategic roles.
Key responsibilities:
- Build and Maintain
Design and maintain data pipelines using Dagster, Airbyte, and DBT. You will take ownership of the infrastructure on AWS (including RDS, Redshift, and DMS) and Google Cloud.
- Innovate and Optimise
Foster an open-source-first mindset. You will identify open-source alternatives and integrate various technologies to create a bespoke, cost-efficient technology stack.
- Monitor and Alert
Set up proactive monitoring and alerting for data reliability. You won't wait for a report to break; you will be proactive in identifying data latency or quality issues, ensuring that you are aware of potential problems before they impact the business.
- Mentorship
Provide technical guidance and code reviews to help individuals enhance their engineering capabilities.
- Explore AI and New Technologies
Actively explore how large language models LLMs, AI, and modern data tools can be integrated into our workflow to give us a competitive edge.
This job may be for you if:
- You are a team player. You reach out to peers and cooperate with others to establish collaborative working relationships. You understand that flying solo is a limiting and isolating choice and know that making the team look good works far better than creating a one-man/one-woman show
- You are comfortable working in ambiguity, making decisions with limited information, and iterating rapidly
- You have a bias for action and the ability to push through roadblocks and maintain progress in a fast-paced, evolving environment.
- You are resilient and can navigate setbacks, pivot strategies, and maintain focus on the product vision.
- You’re level-headed and have the ability to adapt to challenges and make decisions under pressure. When things change, you know how to roll with the punches.
- You are a "CS" fundamentalist: You likely hold a degree in Computer Science or Engineering. You understand how databases work under the hood, how to optimise queries, and the importance of clean, maintainable code.
- You are proactive & fast: You don't wait for a ticket. If you see a way to make a query 50% faster or a pipeline more reliable, you take action. You are comfortable working with minimal processes.
- You are a cost-conscious architect: You enjoy the challenge of building robust solutions on a budget, utilising open-source software rather than relying on enterprise contracts only.
- You are an ego-less collaborator: You can communicate complex technical concepts to the business, but you are also happy to jump on a call with DevOps to debug a networking issue.
- You are curious: You stay up to date with the latest developments in the Data and AI landscape and are eager to experiment with new ways to extract value from our data.
You are our ideal candidate if you have:
- A Bachelor's degree in Computer Science, Engineering, or a related technical field.
- 10+ years of experience in Data Engineering, with mastery in SQL and Python being essential.
- Experience managing AWS cloud infrastructure (Redshift/RDS/DMS/S3), GCP infrastructure (BigQuery/Google Analytics), and business intelligence tools (Looker or Power BI)
- Experience with orchestration tools such as Dagster or Airflow.
- A "startup" mindset, demonstrating a willingness to work across the stack, from database administration to troubleshooting BI dashboards.
The benefits of joining our team:
- A competitive salary
- 30 days of annual leave
- A medical aid contribution/fringe benefit of up to R2 000 per month
- An Apple MacBook and the necessary gear
- A trendy office space (when in the office)
- Discretionary annual bonus
- Lots of opportunities to learn
- Flexible office/remote working - you get to choose whether you want to work remotely or from the office, depending on the importance of critical meetings
- Paid parking (when working from the office)
- 3 Mental health days a year
- The opportunity to be part of a great team and culture!
Do you feel like you would be a good fit? If you’re interested in applying for this role and you meet the necessary requirements, please submit your CV, together with your completed application, on the link below.
