Hadoop Developer In Real World

Name: Hadoop Developer In Real World
Rating: 4.5 (3658 reviews)

Free Cluster Access * HDFS * MapReduce * YARN * Pig * Hive * Flume * Sqoop * AWS * EMR * Optimization * Troubleshooting

Highest Rated

Created byBig Data In Real World

Last updated 7/2020

English

What you'll learn

Understand what is Big Data, the challenges with Big Data and how Hadoop propose a solution for the Big Data problem
Work and navigate Hadoop cluster with ease
Install and configure a Hadoop cluster on cloud services like Amazon Web Services (AWS)
Understand the difference phases of MapReduce in detail
Write optimized Pig Latin instruction to perform complex data analysis
Write optimized Hive queries to perform data analysis on simple and nested datasets
Work with file formats like SequenceFile, AVRO etc
Understand Hadoop architecture, Single Point Of Failures (SPOF), Secondary/Checkpoint/Backup nodes, HA configuration and YARN
Tune and optimize slowing running MapReduce jobs, Pig instructions and Hive queries
Understand how Joins work behind the scenes and will be able to write optimized join statements
Wherever possible, students will be introduced to difficult questions that are asked in real Hadoop interviews

Course content

16 sections • 96 lectures • 20h 5m total length

Course Structure11:03

HDFS - Why Another Filesystem?13:20
Split files into fixed-size blocks, maintain a distributed cluster view, and replicate HDFS blocks across nodes to ensure data integrity and scalable big data storage with commodity hardware.
Blocks7:50
Working With HDFS16:09
HDFS - Read & Write9:31
Explore how HDFS handles read and write operations by detailing the roles of the name node and data node, block storage, and replication strategies across the cluster.
HDFS - Read & Write (Program)8:38
Test your understanding of HDFS
HDFS Assignment0:42

Introduction to MapReduce8:51
Dissecting MapReduce Components18:03
Explore MapReduce components in real world workflows, including input splits, mappers, reducers, shuffle and sort, data partitioning, and the role of combiners to optimize performance.
Dissecting MapReduce Program (Part 1)12:00
Dissecting MapReduce Program (Part 2)16:09
Combiner6:20
Counters6:43
Learn how MapReduce counters provide a unified view of a job across mappers and reducers, including file system and framework counters, and how to use user defined counters.
Facebook - Mutual Friends17:38
Explore implementing a Facebook-like mutual friends feature with Hadoop MapReduce, parsing json records, mapping friend pairs, reducing to compute shared friends, and outputting results via sequence files.
New York Times - Time Machine15:43
Test your understanding of MapReduce
MapReduce Assignment0:47

Introduction to Apache Pig12:52
Learn how Apache Pig translates simple Pig Latin instructions into MapReduce jobs to run on a Hadoop cluster, empowering non-programmers to process data with an optimized, flexible data flow language.
Loading & Projecting Datasets13:41
Solving a Problem13:32
Explore loading a dataset, filtering by year, and computing top 10 stock symbols by highest average volume with Pig Latin on Hadoop, including scripts and local testing before cluster execution.
Complex Types21:12
Pig Latin - Joins19:53
Million Song Dataset (Part 1)10:29
Million Song Dataset (Part 2)15:01
Page Ranking (Part 1)8:11
Learn a simplified page ranking model based on backlinks, and prepare a pedia dataset to implement a custom input format and a map function.
Page Ranking (Part 2)19:26
Page Ranking (Part 3)12:17
Test your understanding of Apache Pig
Apache Pig Assignment1:19

Introduction to Hive Window and Analytical functions8:04
Explore how Hive window and analytical functions enable ten-day moving averages by partitioning by symbol, ordering by date, and framing rows, contrasting with simple group by.
Kickstarter campaign duplicates and top campaigns8:02
Explore deduplicating Kickstarter campaign data with Hive using analytic window functions like row_number, lag, and lead to surface top five pledged campaigns per year and their neighboring pledges.
Kickstarter campaign bands and user sessions8:28
Learn how window and analytical functions in hive group Kickstarter campaigns into bands by year, using a 10,000 pledge-difference threshold, and apply the same approach to user sessionization.

HDFS Architechture12:46
Explore hdfs architecture with name node and data nodes, how blocks and locations are tracked, and how replication factor ensures data health. Learn core-site and hdfs-site configurations and block management.
Secondary Namenode11:24
Highly Available Hadoop8:48
Learn how Hadoop achieves high availability through checkpoint and backup nodes, and a primary-standby name node setup with a shared edits log for seamless failover.
MRv1 Architechture10:49
YARN11:22
Test your understanding of Hadoop Architechture

Vendors & Hosting6:35
Cluster Setup (Part 1)23:43
Learn to design and manually deploy a three-node Hadoop cluster on AWS, installing Cloudera CDH 5.4, configuring name node, resource manager, data nodes, and security.
Cluster Setup (Part 2)25:35
Cluster Setup (Part 3)18:01
Amazon EMR15:46
With Amazon EMR we can start a brand new Hadoop cluster and run MapReduce jobs in matter of minutes. This lecture will walk through step by step how to set up a Hadoop cluster and run MapReduce jobs in it.
Test your understanding of Cluster Setup

Requirements

Although you don't have to be an expert in Java, basic knowledge in Java programming is required as we will be looking at programs in Java.
Basic Linux commands

Description

From the creators of the successful Hadoop Starter Kit course hosted in Udemy, comes Hadoop In Real World course. This course is designed for anyone who aspire a career as a Hadoop developer. In this course we have covered all the concepts that every aspiring Hadoop developer must know to SURVIVE in REAL WORLD Hadoop environments.

The course covers all the must know topics like HDFS, MapReduce, YARN, Apache Pig and Hive etc. and we go deep in exploring the concepts. We just don’t stop with the easy concepts, we take it a step further and cover important and complex topics like file formats, custom Writables, input/output formats, troubleshooting, optimizations etc.

All concepts are backed by interesting hands-on projects like analyzing million song dataset to find less familiar artists with hot songs, ranking pages with page dumps from wikipedia, simulating mutual friends functionality in Facebook just to name a few.

Who this course is for:

This course is for anyone who aspire a career as a Hadoop Developer
This course is for anyone who want to learn and understand in depth about Hadoop and Big Data

Hadoop Developer In Real World

What you'll learn

Explore related topics

Course content

Thank You and Let's Get Started1 lecture • 11min

Introduction To Big Data3 lectures • 36min

HDFS6 lectures • 56min

MapReduce9 lectures • 1hr 42min

Apache Pig11 lectures • 2hr 28min

Apache Hive12 lectures • 1hr 50min

Hive Window and Analytical Functions3 lectures • 25min

Architechture5 lectures • 55min

Cluster Setup5 lectures • 1hr 30min

Hadoop Administrator In Real World (Preview)2 lectures • 37min

Requirements

Description

Who this course is for: