How to Deal with Configuration and Credential files in Python

Data Scientists and Data Engineers may need to access many systems at once. For example, in ETL/ELT jobs there is a chance to extract and store data in different sources like Hadoop, BigQuery, MySQL, PostgreSQL, Redshift, Snowflake and so on. In addition, there can be different users having different credentials and finally, there can be different environments, like test, stage and production. In this post, we will explain how to deal with different credentials in Python using one configuration file.

The configparser Module

The configparser module is built to handle different credentials in one file. Let’s see how to create a configuration file programmatically. Our goal is to create the credentials of different connections.

Create the Configuration File

import configparser

config = configparser.ConfigParser()


# Connection for the user George
config['George'] = {'username': 'george',
                     'password': 'AStrongPassword01'}


# Connection for the user Billy
config['Billy'] = {'username': 'billy',
                     'password': 'AStrongPassword02'}


# connection to the database
config['DB'] = {'username': 'DataScientist',
                'password': 'AStrongPassword03',
                'db': 'PredictiveHacks',
                'host': 'predictivehacks.com',
                'port':1234}



# write the 
with open('my_config.ini', 'w') as configfile:
    config.write(configfile)

After running the script, the “my_config.ini” file is created, as we can see below:

Alternatively, we can create the configuration file using the text editor. For example, the “my_config.ini“:

[George]
username = george
password = AStrongPassword01

[Billy]
username = billy
password = AStrongPassword02

[DB]
username = DataScientist
password = AStrongPassword03
db = PredictiveHacks
host = predictivehacks.com
port = 1234
 

Use the configuration file

Let’s see how we can get the credentials using Python. First, we will see how we can get all the connections of the configuration file.

import configparser

config = configparser.ConfigParser()

# read the configuration file
config.read('my_config.ini')

# get all the connections
config.sections()

Output:

['George', 'Billy', 'DB']

Now, we will get the username and password of “George”

username = config.get('George', 'username')
password = config.get('George', 'password')

print(f'The username is {username} and the passowrd is {password}')

Output:

The username is george and the passowrd is AStrongPassword01

Alternatively, you can get the credentials as follows:

username = config['George']['username']
password = config['George']['password']

print(f'The username is {username} and the passowrd is {password}')

Output:

The username is george and the passowrd is AStrongPassword01

Clearly, following the same logic as above, we can get the credentials of the other connections like, “Billy” and “DB“.