Free Data Engineer Course (6Months)

Data Engineer:

Brief Job Role Description: Individuals at this job are responsible for working on different aspects of data architecture. S/he will be responsible for developing data integrations.

Personal Attributes: A data engineer needs to be well organized and have good attention to detail and problem solving abilities.

Develop data integrations as per specifications:

Define requirements
To be competent, the user/individual on the job must be able to:
PC1. evaluate whether the application is batch or real-time driven
PC2. evaluate whether the application is using an SQL or NoSQL database
PC3. evaluate feasibility and level of effort to integrate the required data

Develop data warehouses, data lakes and pipelines
To be competent, the user/individual on the job must be able to:
PC4. evaluate different data warehouse, lake or pipeline designs
PC5. evaluate available hardware if an on-premise solution is being designed
PC6. determine the right data format depending on the nature of the application (for ex. batch or real-time driven)
PC7. partition or size the data as appropriate
PC8. transform data into other formats
PC9. develop, test and troubleshoot data pipelines
PC10. combine tools into either macro or micro data pipelines
PC11. design and implement ETL (Extract-Transform-Load) processes
PC12. use data integration processes to populate and refresh the data warehouse
PC13. define data onboarding and transformation Data Engineer Course processes to design the data lake
PC14. create and execute proof-of-concepts for demonstrating the viability of the design approach for the data lake
PC15. select appropriate applications to design and scale the data lake as per requirements
PC16. develop the data lake and define appropriate mechanisms to query the data lake

PC17. load data, metadata and other required categories into the data lake
PC18. develop mechanisms for error and fault handling, and system availability for the data lake
PC19. define performance criteria for the data integrations such as response time
PC20. optimize the data integrations on the basis of the defined performance criteria
PC21. create documentation on developed data integrations for appropriate people
PC22. validate data integrations with appropriate people

Organizational Context
The user/individual on the job needs to know and understand:
KA1. organizational policies, procedures and guidelines which relate to designing data integrations
KA2. organizational policies and procedures for sharing data
KA3. organizational policies and procedures for documenting data integrations
KA4. who to involve when designing data integrations
KA5. the range of standard templates and Data Engineer Course tools available and how to use them

Technical Knowledge
The user/individual on the job needs to know and understand:
KB1. different tools for developing data integrations such as Hive, Storm, Kafka
KB2. different data sources, data integration processes and techniques
KB3. different ETL (Extract-Transform-Load) concepts
KB4. different methodological approaches for testing and troubleshooting data pipelines such as unit testing, component testing, black box testing
KB5. different methodological approaches for populating data warehouses including top-down, bottom-up, hybrid and federated
KB6. different applications used to design data lakes
KB7. how to define mechanisms to query the data lake
KB8. how to load data, metadata and other required categories into the data lake
KB9. different mechanisms for management of Data Engineer Course data lakes including error and fault handling, system availability, duplication control, status updates
KB10. different performance criteria for data integrations such as response time

KB11. different performance criteria for hardware if designing an Data Engineer Course on-premise data integration solution such as data center performance, CPU performance, I/O ratios
KB12. how to query and write scripts to acquire and understand the data
KB13. how to work on various operating systems such as Linux, ubuntu, or windows

Skills (S)

Problem Solving
The user/individual on the job needs to know and understand how to:
SA1. apply problem-solving approaches in different situations
Critical Thinking
The user/individual on the job needs to know and understand how to:
SA2. apply balanced judgments to different situations

Manage your work to meet requirements:

Utilize resources
To be competent, the individual working on the Data Engineer Course job must be able to:
PC1. establish and agree your work requirements with appropriate people
PC2. keep your immediate work area clean and tidy
PC3. utilize your time effectively
PC4. use resources correctly and efficiently
PC5. treat confidential information correctly
Ensure compliance
To be competent, the individual working on the job must be able to:
PC6. work in line with your organization’s policies and procedures
PC7. work within the limits of your job role
PC8. obtain guidance from appropriate people, where necessary
PC9. ensure your work meets the agreed requirements

A. Organizational
Context
You need to know and understand:
KA1. your organization’s policies, procedures and priorities for your area of work and your role and responsibilities in carrying out your work
KA2. limits of your responsibilities and when to involve others
KA3. your specific work requirements and who these must be agreed with
KA4. the importance of having a tidy work area and how to do this
KA5. how to prioritize your workload according to Data Engineer Course urgency and importance and the benefits of this

KA6. your organization’s policies and procedures for dealing with confidential information and the importance of complying with these
KA7. the purpose of keeping others updated with the progress of your work
KA8. who to obtain guidance from and the typical circumstances when this may be required
KA9. the purpose and value of being flexible and adapting work plans to reflect change
Technical
Knowledge
You need to know and understand:
KB1. the importance of completing work accurately and how to do this
KB2. appropriate timescales for completing your work and the implications of not meeting these for you and the organization
KB3. resources needed for your work and how to obtain and use these
KB4. how to store and retrieve information
KB5. how to identify and refer anomalies in data
KB6. how to use information technology effectively to input and/or extract data accurately
KB7. how to keep up to date with changes, procedures and practices in your role

Writing Skills
SA1. complete accurate well written work with attention to detail
Reading Skills
SA2. follow instructions, guidelines, Data Engineer Course procedures, rules and service level agreements
Listening and Speaking Skills
SA3. ask for clarification and advice from appropriate people
SA4. listen effectively and orally communicate information accurately
Decision Making
SA5. make decisions on suitable courses
Plan and Organize
SA6. plan and organize your Data Engineer Course own work to achieve targets and deadlines
SA7. provide accurate reports to line managers in a timely manner as required
Customer Centricity

SA8. check that your own and/or your peers’ work meets customer requirements
SA9. deliver consistent and reliable service to customers
Problem Solving
SA10. refer anomalies to the supervisor
SA11. seek clarification on problems from others
Analytical Thinking
SA12. analyze data and activities
SA13. pass on relevant information to others
Critical Thinking
SA14. apply balanced judgments to different situations
Attention to Detail
SA15. check your work Data Engineer Course is complete and free from errors
SA16. get your work checked by others
Team Working
SA17. work effectively in a team environment

Work effectively with colleagues:

Communicate with colleagues
To be competent, the individual working on the job must be able to:
PC1. communicate with colleagues clearly, concisely and accurately
PC2. work with colleagues to integrate your work effectively with them
PC3. pass on essential information to colleagues in line with organizational requirements
Show respect
To be competent, the individual Data Engineer Course working on the job must be able to:
PC4. work in ways that show respect for colleagues
PC5. carry out commitments you have made to colleagues
PC6. let colleagues know in good time if you cannot carry out your commitments, explaining the reasons
PC7. identify any problems you have working with colleagues and take the initiative to solve these problems
PC8. follow the organization’s policies and procedures for working with colleagues

Organizational
Context
You need to know and understand:
KA1. your organization’s policies and Data Engineer Course procedures for working with colleagues and your role and responsibilities in relation to this
KA2. the importance of effective communication and establishing good working relationships with colleagues
KA3. different methods of communication and the circumstances in which it is appropriate to use these
KA4. benefits of developing productive working relationships with colleagues

KA5. the importance of creating an environment Data Engineer Course of trust and mutual respect in an environment where you have no authority over those you are working with
KA6. where you do not meet your commitments, the implications this will have on individuals and the organization
Technical
Knowledge
You need to know and understand:
KB1. different types of information that colleagues might need and the importance of providing this information when it is required
KB2. the importance of understanding problems from your colleague’s perspective and how to provide support, where necessary, to resolve these
KB3. how to identify and refer anomalies in data
KB4. how to help reach agreements with colleagues
KB5. how to keep up to date with changes, procedures and practices in your role

Writing Skills:
SA1. complete accurate well written work with attention to detail
SA2. communicate effectively with colleagues in writing
Reading Skills:
SA3. follow instructions, guidelines, procedures, Data Engineer Course rules and service level agreements
Listening and Speaking Skills
SA4. ask for clarification and advice from appropriate people
SA5. listen effectively and orally communicate information accurately
Decision Making
SA6. make decisions on suitable courses
Plan and Organize
SA7. plan and organize your own work to achieve targets and deadlines
Customer Centricity
SA8. check that your own and/or your peers’ Data Engineer Course work meets customer requirements
SA9. deliver consistent and reliable service to customers
Problem Solving
SA10. apply problem solving approaches in different situations

Critical Thinking
SA11. apply balanced judgments to different situations
Attention to Detail
SA12. check your work is complete and free from errors
SA13. get your work checked by others
Team Working
SA14. work effectively in a team environment
SA15. work effectively with colleagues and other teams
SA16. treat other cultures with respect

Provide data / information in standard formats:

Obtain information
To be competent, the individual working on the job must be able to:
PC1. establish and agree with appropriate Data Engineer Course people the data/information you need to provide, the formats in which you need to provide it, and when you need to provide it
PC2. obtain the data/information from reliable sources
PC3. check that the data/information is accurate, complete and up-to-date
PC4. obtain advice or guidance from appropriate people where there are problems with the data/information
Analyze and report information
To be competent, the individual working on the job must be able to:
PC5. carry out rule-based analysis of the data/information, if required
PC6. insert the data/information into the agreed formats
PC7. check the accuracy of your work, involving colleagues where required
PC8. report any unresolved anomalies in the data/information to appropriate people
PC9. provide complete, accurate and up-to-date data/information to the appropriate people in the required formats on time

A. Organizational knowledge
You need to know and understand:

KA1. your organization’s procedures and guidelines for providing data/information in standard formats and your role and responsibilities in relation to this
KA2. the knowledge management culture of your organization
KA3. your organization’s policies and procedures for recording and sharing information and the importance of complying with these
KA4. the importance of validating data/information before use and how to do this
KA5. procedures for updating data in appropriate formats and with proper validation
KA6. the purpose of the CRM database
KA7. how to use the CRM database to record and extract information
KA8. the importance of having your data/information reviewed by others
KA9. the scope of any data/information requirements including the level of detail required
KA10. the importance of keeping within the scope of work and adhering to timescales
B. Technical
Knowledge
You need to know and understand:
KB1. data/information you may need to provide including the sources and how to do this
KB2. templates and formats used for data/information including their purpose and how to use these
KB3. different techniques used to obtain data/information and how to apply these
KB4. how to carry out rule-based analysis on the data/information
KB5. typical anomalies that may occur in data/information
KB6. who to go to in the event of inaccurate data/information and how to report this
KB7. how to use information technology effectively to input and/or extract data accurately
KB8. how to validate and update data
KB9. how to identify and refer anomalies in data
KB10. how to store and retrieve information
KB11. how to share information using standard formats and templates
KB12. how to keep up to date with changes, procedures and practices in your role

Writing Skills
SA1. complete accurate well written work with attention to detail
Reading Skills
SA2. follow instructions, guidelines, procedures, rules and service level
agreements
Listening and Speaking Skills
SA3. listen effectively and orally communicate information accurately
Decision Making
SA4. follow rule-based decision making processes
SA5. make decisions on suitable courses
Plan and Organize
SA6. plan and organize your own work to achieve targets and deadlines
Customer Centricity
SA7. check that your own and/or your peers’ work meets customer requirements
SA8. meet and exceed customer expectations
Problem Solving
SA9. apply problem solving approaches in different situations
Analytical Thinking
SA10. configure data and disseminate relevant information to others
Critical Thinking
SA11. apply balanced judgments to different situations
Attention to Detail
SA12. check your work is complete and free from errors
SA13. get your work checked by others
Team Working
SA14. work effectively in a team environment

Develop your knowledge, skills and competence:

Develop knowledge
To be competent, the individual working on the job must be able to:
PC1. obtain advice and guidance from appropriate people to develop your knowledge, skills and competence
PC2. identify accurately the knowledge and skills you need for your job role
PC3. identify accurately your current level of knowledge, skills and competence and any learning and development needs
PC4. agree with appropriate people a plan of learning and development activities to address your learning needs
PC5. undertake learning and development activities in line with your plan
Apply knowledge
To be competent, the individual working on the job must be able to:
PC6. apply your new knowledge and skills in the workplace, under supervision
PC7. obtain feedback from appropriate people on your knowledge and skills and how effectively you apply them

Organizational
Context
You need to know and understand:
KA1. your organization’s procedures and guidelines for developing your knowledge, skills and competence and your role and responsibilities in relation to this
KA2. the importance of developing your knowledge, skills and competence to you and your organization
KA3. different methods used by your organization to review skills and knowledge including:  training need analysis  skills need analysis  performance appraisals
KA4. how to review your knowledge and skills against your job role using different methods and analysis
KA5. different types of learning and development activities available for your job role and how to access these
KA6. how to produce a plan to address your learning and development needs, who to agree it with and the importance of undertaking the planned activities
KA7. different types of support available to help you plan and undertake learning and development activities and how to access these
KA8. why it is important to maintain records of your learning and development
KA9. methods of obtaining and accepting feedback from appropriate people on your knowledge skills and competence
KA10. how to use feedback to develop in your job role
Technical
Knowledge
You need to know and understand:
KB1. the knowledge and skills required in your job role
KB2. your current learning and development needs in relation to your job role
KB3. different types of learning styles and methods including those that help you learn best
KB4. the importance of taking responsibility for your own learning and development
KB5. to the importance of learning and practicing new concepts, theory and how to apply these in the work environment or on samples.
KB6. how to explore sample problems and apply solutions
KB7. how to use information technology effectively to input and/or extract data accurately

Writing Skills
SA1. complete accurate well written work with attention to detail
Reading Skills
SA2. follow instructions, guidelines, procedures, rules and service level agreements
Listening and Speaking Skills
SA3. ask for clarification and advice from line managers
Decision Making
SA4. make decisions on suitable courses
Plan and Organize
SA5. plan and organize your own work to achieve targets and deadlines
Customer Centricity
SA6. check that your own and/or your peers’ work meets customer requirements
Problem Solving
SA7. refer anomalies to the line manager
Analytical Thinking
SA8. analyze data and activities
Critical Thinking
SA9. apply balanced judgments to different situations
Attention to Detail
SA10. check your work is complete and free from errors
SA11. get your work checked by others
Team Working
SA12. work effectively in a team environment

Guidelines for Assessment

1. Criteria for assessment for each Qualification Pack will be created by the Sector Skill Council. Each Performance Criteria (PC) will be assigned marks proportional to its importance in NOS. SSC will also lay down proportion of marks for Theory and Skills Practical for each PC.
2. The assessment for the theory part will be based on knowledge bank of questions created by the SSC.
3. Assessment will be conducted for all compulsory NOS, and where applicable, on the selected elective/option NOS/set of NOS.
4. Individual assessment agencies will create unique question papers for theory part for each candidate at each examination/training center (as per assessment criteria below).
5. Individual assessment agencies will create unique evaluations for skill practical for every student at each examination/training center based on this criterion.
6. To pass a QP, a trainee should score an average of 70% across generic NOS’ and a minimum of 70% for each technical NOS.