What is Hadoop?

Apache Hadoop is an open source framework that’s used to store and process huge datasets, so it’s often used in applications and operations such as big data analytics, data mining, predictive analytics, and machine learning.

Developed by the Apache Software Foundation, the Hadoop framework functions by clustering together computers and servers, known as commodity hardware, so that data can be stored and processed collectively. This model of scalable, distributed data processing means that data can be analysed quickly, more efficiently, and reliably.

Hadoop can also work with a variety of data sizes, formats, and types, including structured data, semi-structured data, and unstructured data, and it supports a wide array of other applications and projects.

What are the benefits of Hadoop?

With its capacity to store, handle, and process petabytes of data, Hadoop has been a game-changer for big data, social media, and the internet of things (IoT) – it even supports the Yahoo search engine. Hadoop is also noteworthy for its:

• ability to run concurrent tasks and jobs.
• scalability, allowing users to easily scale up by adding nodes to the system.
• fault tolerance and high availability, with data protected against hardware loss and failure thanks to Hadoop’s distributed computing and storage.
• flexibility, allowing users to store a variety of data types without having to preprocess them. This differs from traditional, relational databases that require preprocessing.
• cost-effectiveness. Hadoop’s open source framework is free, and its use cases span the financial sector to healthcare and beyond. 

IBM also notes benefits including better data-driven decisions as well as data offload and consolidation through Hadoop.

How does Hadoop work?

Hadoop processes data using a few key steps:

1. Applications collect data submitted by the client. 
2. The data is stored on a Hadoop cluster called a DataNode, and is maintained and managed by the NameNode server – effectively the master node – via an application programming interface (API). This NameNode will oversee multiple DataNodes.
3. The data is then processed.

It is written in Java, but Hadoop programmes can also be coded in Python or C++ programming languages.

Hadoop’s main modules

Hadoop has four core components:

Hadoop Common

Hadoop Common provides the common Java libraries that support Hadoop’s other modules.

Hadoop Distributed File System (HDFS)

HDFS is the distributed file system that provides better data throughput – the amount of data successfully moved from one place to another during a given time period – than traditional systems, as well as higher fault tolerance. Through metadata, HDFS structures directories and files in a tree, and attributes information such as permissions, ownership, and replication factors.

Hadoop YARN (Yet Another Resource Negotiator)

YARN is Hadoop’s framework for job scheduling and cluster resource management. It manages and monitors cluster nodes and resource usage, and schedules jobs and tasks. 

Hadoop MapReduce

MapReduce is a YARN-based framework that enables the parallel processing of large data sets.

The MapReduce programming model takes input data and then converts it into datasets that can be computed in what’s known as key value pairs. MapReduce jobs then aggregate the output.

The Hadoop ecosystem

There are a number of different applications, technologies, and tools that sit within what’s known as the Hadoop ecosystem.

All of these applications use Hadoop technologies to collect, store, process, analyse, and manage data:

• • Spark. Apache Spark is a fast computing engine for Hadoop data. It provides a simple programming model for the distributed processing of big data workloads, and uses in-memory caching and optimised execution for faster performance. It supports a wide range of applications, including machine learning algorithms, stream processing and analytics, graph computation and databases, general batch processing, and ad hoc queries.
  • Ambari. This web-based tool is used for managing and monitoring Apache Hadoop clusters and supporting HDFS, MapReduce, Hive, ZooKeeper, Oozie, Pig, Sqoop, and other applications through visualisation dashboards for cluster health.
  • Avro. Avro is a data serialisation system.
  • Cassandra. This is a scalable, multi-master database with no single points of failure.
  • Chukwa. Chukwa is a data collection system used for managing large, distributed systems.
  • Flume. Flume is distributed and reliable software used for efficiently collecting, aggregating, and moving large amounts of log data.
  • HBase. Apache HBase is a non-relational, scalable, versioned, and distributed database or data store that supports structured data storage and provides real-time access to huge tables – even ones with billions of rows and millions of columns.
  • Hive. Apache Hive is a distributed, fault-tolerant data warehouse that provides data summarization, ad hoc querying, and analytics at a massive scale.
  • Impala. Impala is Apache’s massively parallel processing (MPP) SQL query engine, and is supported by Cloudera.
  • Kafka. Apache Kafka offers distributed event storage and stream-processing.
  • Mahout. Mahout is a scalable machine learning and data mining library.
  • Oozie. Oozie is a workflow scheduling system that’s used to manage Hadoop jobs.
  • Ozone. This is a scalable, distributed object store.
  • Pig. Apache Pig is Hadoop’s high-level data-flow language and execution framework for parallel computation.
  • Presto. Presto is a distributed SQL query engine that’s optimised for low-latency, ad-hoc analysis of data, and can process data from multiple sources, including HDFS and Amazon S3.

• Sqoop. Sqoop is an interface application used to transfer data between relational databases and Hadoop.

• Storm. Storm is a distributed stream processing framework.
• Submarine. This is a unified artificial intelligence (AI) platform that enables engineers and data scientists to run machine learning and deep learning workloads in distributed clusters.
• Tez. Tez is a data-flow programming framework that’s built on YARN and is being adopted by a number of applications as a replacement for MapReduce as an underlying executive engine.
• Zeppelin. Zeppelin is an interactive notebook that enables interactive data exploration.
• ZooKeeper. ZooKeeper is a coordination service for distributed applications.

Explore and apply Hadoop technologies

Deepen your understanding of Apache Hadoop  – and develop other in-demand computer science skills – with the MSc Computer Science at the University of Wolverhampton. This flexible Master’s degree has been developed for ambitious individuals who may not have a background in computer science, and is delivered 100% online.

The degree includes a key module in database systems and security, so you will explore Hadoop technologies in addition to other types of data and database systems to gain an appreciation of where they are applicable. This includes areas of study such as:

• database concepts and techniques
• database design (Oracle Data Modeller)
• relational model concepts working with database technologies 
• SQL and Oracle 
• non-relational systems such as NoSQL, MongoDB, and Oracle NoSQL 
• Spark environments
• database management and administration management of data
• data governance
• data security.

This module will equip you with the skills necessary to design and implement appropriate database systems that support the manipulation of data and keep it secure. You will be able to apply a critical understanding of, and demonstrate ability in, the tools and techniques used in the development of database applications, and understand the fundamental aspects of data governance and data security.