Big Data Engineer
Participates in the design, build and management of large scale data structures and pipelines and efficient Extract/Load/Transform (ETL) workflows.
Assists in the development of large scale data structures and pipelines to organize, collect and standardize data that helps generate insights and addresses reporting needs. Applies understanding of key business drivers to accomplish own work. Uses expertise, judgment and precedents to contribute to the resolution of moderately complex problems. Leads portions of initiatives of limited scope, with guidance and direction. Writes ETL (Extract / Transform / Load) processes, designs database systems and develops tools for real-time and offline analytic processing. Collaborates with client team to transform data and integrate algorithms and models into automated processes. Uses knowledge in Hadoop architecture, HDFS commands and experience designing & optimizing queries to build data pipelines. Uses programming skills in Python, Java or any of the major languages to build robust data pipelines and dynamic systems. Builds data marts and data models to support clients and other internal customers. Integrates data from a variety of sources, assuring that they adhere to data quality and accessibility standards.
Background Experience
Strong problem solving skills and critical thinking ability.Strong collaboration and communication skills within and across teams.3 or more years of progressively complex related experience.Ability to leverage multiple tools and programming languages to analyze and manipulate data sets from disparate data sources.Ability to understand complex systems and solve challenging analytical problems.Experience with bash shell scripts, UNIX utilities & UNIX Commands.Knowledge in Java, Python, Hive, Cassandra, Pig, MySQL or NoSQL or similar.Knowledge in Hadoop architecture, HDFS commands and experience designing & optimizing queries against data in the HDFS environment.Experience building data transformation and processing solutions.Has strong knowledge of large scale search applications and building high volume data pipelines. Master’s degree or PhD preferred.Bachelor's degree or equivalent work experience in Computer Science, Engineering, Machine Learning, or related discipline