Software Engineer-SE - AWS Big Data
- Full-time
Company Description
NEC Software Solutions (India)
On 1st July 2021, Rave Technologies became NEC Software Solutions India. This change brought us under the global NEC Corporation brand. We are proud to be part of an organisation with 122 years of experience in evolution with technology and innovation.
We have more than 30 years of experience in providing end to end IT services across the globe and have earned a reputation for delighting our customers by consistently surpassing expectations and helping them deliver robust, market-ready software products that meet the highest standards of engineering and user experience. Supported by more than 1300 exceptionally talented manpower, we are a hub for offshore support and technology services.
We work with diverse industry verticals which include publishing, media, financial services, retail, healthcare and technology companies around the world. Our customers range from two-person startups to $bn listed companies.
For more information, visit at www.necsws.com/india.
About NEC Corporation
NEC Corporation is a Japanese multinational information technology and electronics company, headquartered in Tokyo, Japan. It is recognised as a ‘Top 50 Innovative Company’ globally and the NEC Group globally provides “Solutions for Society” that promote the safety, security, fairness and equality of society. Their main goal is to help create a safer society with their innovations in technologies.
NEC Corporation has established itself as a leader in the integration of IT and network technologies while promoting the brand statement of “Orchestrating a brighter world.” NEC enables businesses and communities to adapt to rapid changes taking place in both society and the market as it provides for the social values of safety, security, fairness and efficiency to promote a more sustainable world where everyone has the chance to reach their full potential.
For more information, visit NEC at https://www.nec.com.
Job Description
2-5 years experience as a AWS Big data engineer
Expert in SQL (Postgres / Redshift)
Expertise in AWS Glue, Athena and Spark
Expertise in working with AWS Kinesis
Working with EMR
Expert in SQL (Postgres / Redshift)
- Query factoring (With clause, Common Table Expression)
- Window functions: rank, row_number, lead, lag, first_value, last_value
- Sub-Queries
- Joins: Self, Inner, Outer (Left/Right), Full-Outer
- group by, having clause, orde by etc
- External Tables
- OLAP Cubes and Rollupso
- Bulk load using COPY and Bulk download using UNLOAD
- Stored Procedures
- Transaction Management
- Incremental loading of data from s3 files using upsert/merge
- Using explain plan to improve query performance
- Using incremental refresh Materialized Views
- Effectively Using DISTKEY and SORTKEY to achieve high performance distributed queries
- Excel like Pivot and Unpivot columns in SQL queries
Expertise in AWS Glue, Athena and Spark
- Writing Spark batch and Streaming jobs (Python or Scala)
- Performance tuning of spark jobs
- Stream aggregation using kinesis /kafka and writing to redshift
- Reading and writing partitioned S3 data
- Using glue to join disparate data sources like RDS, Redshift, S3 etc.
- Using Athena to query S3 data.
- Process distributed data sources using Spark
- Writing glue pipelines
Redshift
- Managing clusters
- Creating Cluster
- Scaling: Elastic / Classic
- Taking snapshots
- Restoring clusters from snapshots
- Monitoring CPU/RAM etc
Expertise in working with AWS Kinesis
Working with EMR
Creating transient as well as persistent clusters
Reading and writing S3 files from EMR Spark jobs
Additional Information
Good Communication Skills required.