Big Data Developer
Description of Role:
- Big Data Developer with Hadoop, Python and (SSIS, SQL) experience. Because we work on the cutting edge of a lot of technologies, we need someone who is a creative problem solver, resourceful in getting things done, and productive working independently or collaboratively.
- Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.).
- Work closely with our engineering team to integrate your amazing innovations and algorithms into our production systems. Process unstructured data into a form suitable for analysis – and then do the analysis.
- Support business decisions with ad hoc analysis as needed.
- Experience with AWS, Azure, EMR and other web services will help you hit the ground running.
- 5-8+ Years of overall experience with at least 2 years with Big Data tools and technologies
- Multiple Hadoop Platform implementations
- Big Data Development using Hadoop Echo system including Pig, Hive, spark, spark streaming and others.
- Analytical and problem solving skills, applied to a Big Data environment
- Experience with large-scale distributed applications
- Proven understanding and related experience with Hadoop, HBase, Hive, Pig, Spark, Flume, Hbase and/or Map/Reduce
- Hands-on experience with related/complementary open source software platforms and languages like Python, Java, scala
- Experience processing large amounts of structured and unstructured data. MapReduce experience is a plus
- Traditional Data Warehouse/ETL experience required (SSIS, SQL)
- Programming experience, ideally in Python, but we are open to other experience if you’re willing to learn the languages we use. Taking lead on design and writing of python scripts for big data implementation.
- Enough programming knowledge to clean and scrub noisy dataset. Creating high level complexity reports and analysis using BI toolset (solutions using SSIS, SSAS, and Tableau).
- Typically requires a Bachelor’s degree in engineering, computer science or a related and 8-12 years of related experience.
- Technical Skills: Hadoop, SPARK, Python, SSIS, SQL.