How to Generate the Requirements of your Python project based on your Imports

We have provided many tutorials on how to generate the requirements.txt for your python project without environments, how to work with Conda environments, how to work with VS Code and Virtual Environments and so on. Today, we will provide an alternative way to get the requirement.txt file by including only the library that we have used, in other words, only the libraries that we have imported. We will provide two approaches, the first one is when we work with .py files and the second one when we work with Jupyter notebooks.

Working with .py files

Let’s assume that we work on the project, called “pipreqs_example“, where there is our .py file containing the code of the project. In order to make it reproducible, we would like to generate the “requirements.txt” file but ONLY for the used libraries. We can easily achieve that by using the pipreqs library. We can install the library as follows:

pip install pipreqs

Within the project I have a .py file with the following imports:

import numpy as np
import pandas as pd
import re

Let’s see how we can generate the requirements.txt file. We can either specify the whole path of the project, or run the following command within the project path.

pipreqs --force

And the requirements.txt appears in the project directory!

pandas==1.2.5
numpy==1.21.1
 

Usage:

Usage:
    pipreqs [options] [<path>]

Arguments:
    <path>                The path to the directory containing the application files for which a requirements file
                          should be generated (defaults to the current working directory)

Options:
    --use-local           Use ONLY local package info instead of querying PyPI
    --pypi-server <url>   Use custom PyPi server
    --proxy <url>         Use Proxy, parameter will be passed to requests library. You can also just set the
                          environments parameter in your terminal:
                          $ export HTTP_PROXY="http://10.10.1.10:3128"
                          $ export HTTPS_PROXY="https://10.10.1.10:1080"
    --debug               Print debug information
    --ignore <dirs>...    Ignore extra directories, each separated by a comma
    --no-follow-links     Do not follow symbolic links in the project
    --encoding <charset>  Use encoding parameter for file open
    --savepath <file>     Save the list of requirements in the given file
    --print               Output the list of requirements in the standard output
    --force               Overwrite existing requirements.txt
    --diff <file>         Compare modules in requirements.txt to project imports
    --clean <file>        Clean up requirements.txt by removing modules that are not imported in project
    --mode <scheme>       Enables dynamic versioning with <compat>, <gt> or <non-pin> schemes
                          <compat> | e.g. Flask~=1.1.2
                          <gt>     | e.g. Flask>=1.1.2
                          <no-pin> | e.g. Flask

Working with Jupyter notebooks

If you work with Jupyter notebooks, you can use the pipreqsnb library. You can install the library as follows:

pip install pipreqsnb

We work similarly as before, but not the command is:

pipreqsnb --force

Note that pipreqsnb is a very simple fully compatible pipreqs wrapper that supports python files and jupyter notebooks.

Usage:

Usage:
    pipreqsnb [options] <path> 

Options:
    --use-local           Use ONLY local package info instead of querying PyPI
    --pypi-server <url>   Use custom PyPi server
    --proxy <url>         Use Proxy, parameter will be passed to requests library. You can also just set the
                          environments parameter in your terminal:
                          $ export HTTP_PROXY="http://10.10.1.10:3128"
                          $ export HTTPS_PROXY="https://10.10.1.10:1080"
    --debug               Print debug information
    --ignore <dirs>...    Ignore extra directories (sepparated by comma no space)
    --encoding <charset>  Use encoding parameter for file open
    --savepath <file>     Save the list of requirements in the given file
    --print               Output the list of requirements in the standard output
    --force               Overwrite existing requirements.txt
    --diff <file>         Compare modules in requirements.txt to project imports.
    --clean <file>        Clean up requirements.txt by removing modules that are not imported in project.
    --no-pin              Omit version of output packages.

The Takeaway

When you work on projects, using environments of many installed libraries that are not used in that particular project, it is better to share the requirements.txt of the used libraries only. A good application of pipreqs is when you work with Jupyter Notebooks on AWS SageMaker or with Colab and you just want to know what version of libraries you have used.