How to test if DNS resolver and network connections are working from a container?
How to check if Fleio connects successfully to the OpenStack API
fleio backup command gives error: bc command not found, aborting
“504 Gateway Timeout” error shows in web interface / uwsgi keeps restarting
“Verification code is invalid” when enabling second factor authentication
How to test if DNS resolver and network connections are working from a container?¶
Depending on your networking environment and local DNS settings, you may encounter networking errors and DNS resolver issues.
To keep the Fleio Docker container images small and to follow best practices, containers do not include network
debugging utilities, like
telnet. As an exception, the Fleio
utils container (which you can enter
fleio bash command) include these, and many other handy utilities.
celery and other
containers that are actually used for Fleio’s functionality do not include debugging tools.
Use the following commands to confirm that DNS resolver is working and outgoing network connections to the specified host are allowed:
# The following command enters the fleio-celery-1 Docker container, you can replace it with another Fleio container, # e.g. fleio-backend-1. docker exec -it fleio-celery-1 bash # Replace the host ('fleio.com') and TCP sockety (443) with desired values to test. # Connect throws an error if it fails for any reason, otherwise "ok" is printed. python -c "import socket; socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect(('fleio.com', 443)); print('ok')" # exit the container: exit
If a connection is successfully established to the specified host on the specified TCP port, you will get “ok” printed at the console. Otherwise, you will get an error. Some common errors are exemplified below.
If host can not be resolved to IP address, meaning that the DNS resolver is not working correctly in the container (or you used a host that does not have a record in the global DNS system), you will get:
Traceback (most recent call last): File "<string>", line 1, in <module> socket.gaierror: [Errno -2] Name or service not known
If target host refused the connection:
Traceback (most recent call last): File "<string>", line 1, in <module> ConnectionRefusedError: [Errno 111] Connection refused
Connection times out, possibly a firewall is preventing the connection:
Traceback (most recent call last): File "<string>", line 1, in <module> TimeoutError: [Errno 110] Connection timed out
How to check if Fleio connects successfully to the OpenStack API¶
After you fill in the details on the OpenStack settings Credentials tab press Test connection to see if credentials are working. Then by pressing Save & sync or Sync button all the OpenStack objects (projects, instances, volumes, networks etc.) from the Fleio database are reset and the list of objects with states is retrieved from OpenStack.
You can then go to Cloud > Instances and check list of instances (or volumes, or some other resource that you know has elements in OpenStack) and see if there are any objects present.
How to check if OpenStack notifications are working¶
Fleio requires notification messages from the OpenStack’s internal RabbitMQ queue to update the cloud objects’ state in the Fleio database and for most of the pricing rules.
You can confirm that Fleio receives notifications by shutting down an existing instance and see the instance status being updated in Fleio. Follow these steps:
Go to Fleio
Sync OpenStack objects as described in section How to check if Fleio connects successfully to the OpenStack API
Assuming you have a running instance, shut it down
Wait a few seconds for the instance to reach the STOPPED state. If your instance still shows as RUNNING state in Fleio, while you can see the instance being
SHUTOFFin OpenStack (
openstack server list --all-projects), that means that notifications are not working.
These steps apply for other operations as well: create instance, but it never shows up in Fleio, while you can see it created in OpenStack, or instance start for a shutoff instance.
To repeat these steps, make sure you start from a synced database by pressing Sync on the OpenStack Credentials tab.
The checklist to have notifications working is:
Make sure you enabled the OpenStack notifications
you added add the RabbitMQ user for Fleio
you set the notification settings
For debugging purposes, you can also enable notification logging with Notifications logging settings and see in the log file what notifications are received.
See in this video how notifications log should work:
Troubleshoot email messages are not sent¶
To check if your Fleio installation is able to send email messages, try to reset a staff user’s password.
Go to the Fleio staff panel, log out first if you are logged in, and go to the login form (/staff URL). Click on the Forgot password link and fill in an email address of an existing staff user. If email settings are working correctly, you will receive a reset password email message.
If you do not receive a reset password email message, log in to the staff panel and check Utilities > Email message log. You should see Password reset entry marked with a red bar if email sending failed. Click on the entry and check “Error:” line on screen. Fix the email settings according to Email Settings and test reset password again.
If you can’t see a Password reset entry in the Email message log, double check that you used a valid staff user email address.
Troubleshoot database errors on upgrade¶
Fleio database must be created with utf8mb4 character set. You can check the character set by running the following query on your database (usually fleio):
fleio mysql SELECT @@character_set_database, @@collation_database;
If your database contains tables that are using a different character set you will have to update the tables to have the UTF8mb4 set. This can be done with the following steps:
Export current DB
cd /home/fleio/compose db_pass=$(cat /home/fleio/compose/secrets/.db_password) docker compose exec -T db mysqldump fleio -u fleio -p"$db_pass" > fleio`date +%d.%m.%Y`.sql
Adjust the dump so all the
CREATE TABLEstatements will use utf8mb4.
Import the database (replace the path to your FIXED_DB with the proper name)
sudo chown fleio /home/fleio/compose/FIXED_DB.sql sudo -i -u fleio cd /home/fleio/compose # stop all services docker compose stop # and start just the database service docker compose start db # get the database password in a variable db_pass=$(cat /home/fleio/compose/secrets/.db_password) # enter the mysql console docker compose exec db mysql -u fleio -p"$db_pass" # once in console, drop the existing database and create a new one: DROP DATABASE fleio; CREATE DATABASE fleio CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; # exit the console to Bash exit # import the database cat fleio.sql | docker compose exec -T db mysql fleio -u fleio -p"$db_pass"
Apply migrations by running
fleio django migrate
How to enable debugging¶
If you’re in a development environment (not a production server) you can also enable debugging from settings.py file to get more information in browser when you’re accessing an invalid URL and in case an error occurs during development:
DEBUG = True
Now you’ll be able to see the backend API nicely formatted and even perform POST, PUT, and DELETE requests. Note that you need to be authenticated as
Here’s an example of PHP code calling the Fleio API: https://github.com/fleio/fleio-whmcs.
You can also run Django commands from the command line after you activate the Fleio Python virtual environment and move to the Django project directory. Please see Command line interface.
For more information regarding the Django command line see https://docs.djangoproject.com/en/dev/ref/django-admin/
Data points might take some time to be added, so you might want to wait for at least 30 minutes to 1 hour after the instance was deployed before starting to troubleshoot this.
If no metrics are displayed for an instance first step will be to check Fleio backend logs.
If you find no errors in logs or BadRequest errors when retrieving metrics you should ensure gnocchi is installed and both gnocchi and Ceilometer are configured correctly for the instance’s region. See Ceilometer configuration for instructions on how to configure gnocchi and Ceilometer.
Detailed info on the Docker Fleio deployment¶
fleio backup command gives error: bc command not found, aborting¶
You need the
bc system package (bash command line calculator). Here’s how you install it on Debian or Ubuntu:
sudo apt install bc
Other Linux distributions have similar commands to install
“504 Gateway Timeout” error shows in web interface / uwsgi keeps restarting¶
You periodically see “504 Gateway Timeout” toast messages in the web interface and when you check
docker logs fleio-backend-1), you see uWSGI restarting every few seconds. This may be caused by incorrect
system time on the Fleio server. Make sure you keep system time up-to-date.
“Verification code is invalid” when enabling second factor authentication¶
When you try to secure a user account by enabling two-factor authentication you get error “Verification code is invalid” . Try a couple of times and even scan the QR code again and retry. If it still doesn’t work, and you made sure you are using the right code from Google Authenticator (or other application), then this may be caused by incorrect system time on the server where Fleio is installed. Make sure you keep system time up-to-date.