Apache Hive Metastore Explained — What It Is & Why It Matters

Описание: Blog:   / apache-hive-metastore-explained-what-it-is...  

Newsletter:   / subscribe  

Last month, my manager asked me to connect our Springboot backend to our Hive metastore service, and I was like, “Connect Springboot to what?”. So this one’s for anybody who is wondering the same.

Hive Metastore definition
Apache describes it as:

The Hive Metastore (HMS) is a central repository of metadata for Hive tables — https://hive.apache.org/

Honestly, it explains nothing to a dummy like me. But let’s gather what we understood from the above. HMS is a service that stores metadata and information for Hive tables. So the next obvious question that arises is

What is Apache Hive?
Hive is a data warehouse built using Apache Hadoop. To feel you in, Hadoop is a data store framework built specially to handle big data optimally. A data warehouse is a service that lets users analyze big data.

Taking all this into consideration, it is safe to say that Apache Hive is a Hadoop-based service or repository that enables users to analyze big data.

What are Hive Tables?
As described above, Hive is meant to enable users to explore and analyse Big Data. To achieve this, Hive acts as an SQL engine which lets users to query this large amount of Data efficiently. To make this possible, this data is stored in a structured format called Hive Tables, on which the SQL queries are run.

What is Hive Metastore?
Hive Metastore is a service that stores metadata about these Hive Tables, such as the name of DBs and Schemas, Storage location, partitions, etc.

Why do we need Hive Metastore?
If you are like me, you might be wondering why you even need it. For this, it is important to remember that we are dealing with Big Data, which are stored in Data warehouses. Such type of Data is not simply stored in an RDBMS like Postgres, as it’s too expensive to compute and index through petabytes of data. Instead, it is stored as simple files such as CSVs, XLSX, or other file formats in Blob storage solutions such as S3. Now, in order to track these large numbers of files efficiently, we need a system that stores this metadata about the files, which is the Hive Metastore.

I hope this sheds some light on the vast world of Data engineering. See you next time.

#backend #devops #dataengineering