How to Set Up Python Package Dependencies with Git Submodules

Описание: Learn how to handle Python package dependencies when using Git submodules. Follow our detailed guide to streamline your development process without needing a PyPI server.
---
This video is based on the question https://stackoverflow.com/q/69551065/ asked by the user 'hyamanieu' ( https://stackoverflow.com/u/1773344/ ) and on the answer https://stackoverflow.com/a/69552966/ provided by the user 'Dan Nagle' ( https://stackoverflow.com/u/2202018/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: setup with submodules dependencies

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Set Up Python Package Dependencies with Git Submodules: A Simple Guide

When developing Python packages, it's common to rely on external libraries. However, what if those libraries are also your own projects, hosted in separate Git repositories? This situation often arises for developers working on multiple interdependent packages that they prefer to manage internally without using public repositories like PyPI.

In this guide, we’ll take a deep dive into how to effectively manage those dependencies using Git submodules. This will allow you to create a seamless package installation process without needing an external package index.

The Problem

You have a primary Python package that relies on several other packages, which are themselves Git repositories. To keep everything organized, you've made these other packages Git submodules of the main project. The structure of the project looks like this:

[[See Video to Reveal this Text or Code Snippet]]

You’ve attempted to specify your dependencies in setup.py like this:

[[See Video to Reveal this Text or Code Snippet]]

However, it didn’t work as expected. Furthermore, alternatives like placing the submodule in the packages section haven't provided the desired outcome.

So, what’s the solution?

The Solution: Specifying Dependencies Correctly

The key to successfully install packages that are dependencies—especially ones hosted in Git submodules—is to specify exactly how those dependencies should be installed. Here’s how to do that:

Step 1: Modify Your setup.py

Instead of relying on simple paths for your dependencies, you can specify the installation method directly from the repository using the following format:

[[See Video to Reveal this Text or Code Snippet]]

Replace <dependency_name>, <user_name>, <repo_name>, and <branch> with the actual names relevant to your project. For instance:

[[See Video to Reveal this Text or Code Snippet]]

Step 2: Remove Redundant Entries

Ensure your install_requires list no longer includes relative paths to your submodules since you're now specifying how to install them directly. This will allow pip to correctly identify and fetch the package from the provided Git repository path.

Step 3: Install Your Package

With these adjustments, you can now install your primary package without having to upload anything to PyPI or manually install each submodule. Simply run:

[[See Video to Reveal this Text or Code Snippet]]

This command installs your package along with all its dependencies, including those housed in Git submodules.

Conclusion

Using Git submodules to manage dependencies in a Python package can enhance your development workflow, especially in an internal ecosystem. By specifying dependencies correctly in your setup.py, you eliminate the need for cumbersome installation steps or external package indices.

Implementing this method should simplify your interactions with your Python packages and ensure that everything installs neatly with just one command. Happy coding!