=============== Troubleshooting =============== .. contents:: :local: .. _troubleshoot-openstack-connection: How to check if Fleio connects successfully to the OpenStack API ================================================================ After you fill in the details on the OpenStack settings :ref:`openstack-settings-credentials` tab press **Test connection** to see if credentials are working. Then by pressing **Save & sync** or **Sync** button all the OpenStack objects (projects, instances, volumes, networks etc.) from the Fleio database are reset and the list of objects with states is retrieved from OpenStack. You can then go to **Cloud** > **Instances** and check list of instances (or volumes, or some other resource that you know has elements in OpenStack) and see if there are any objects present. .. _troubleshoot-openstack-notifications: How to check if OpenStack notifications are working =================================================== Fleio requires notification messages from the OpenStack's internal RabbitMQ queue to update the cloud objects' state in the Fleio database and for most of the pricing rules. **You can confirm that Fleio receives notifications** by shutting down an existing instance and see the instance status being updated in Fleio. Follow these steps: #. Go to Fleio ``/staff`` panel #. Sync OpenStack objects as described in section :ref:`troubleshoot-openstack-connection` #. Assuming you have a running instance, shut it down #. Wait a few seconds for the instance to reach the **STOPPED** state. If your instance still shows as **RUNNING** state in Fleio, while you can see the instance being ``SHUTOFF`` in OpenStack ( ``openstack server list --all-projects``, that means that notifications are not working. These steps apply for other operations as well: create instance but it never shows up in Fleio, while you can see it created in OpenStack, or instance start for a shutoff instance. To repeat these steps, make sure you start from a synced database by pressing Sync on the OpenStack :ref:`openstack-settings-credentials` tab. **The checklist to have notifications working is**: * Make sure you :ref:`enabled the OpenStack notifications` * you added :ref:`add the RabbitMQ user` for Fleio * you set the :ref:`notification settings` For debugging purposes, you can also enable notification logging with :ref:`log-notifications` and see in the log file what notifications are received. See in this video how notifications log should work: .. raw:: html .. _troubleshoot-email: Troubleshoot email messages are not sent ======================================== You can check if your email settings are configured correctly in ``settings.py`` by trying to send an email message from the Django shell command line utility. SSH to the server where Fleio is installed and run these commands in bash: .. code-block:: bash fleio shell # you should see the shell prompt now: >>> # let's try to send an email # !!! replace with from and to email addresses accepted by your mail server from django.core.mail import send_mail send_mail('Subject', 'message', 'from@example.com', ['to@example.com'], fail_silently=False) # exit Django shell by typing CTRL_D or exit() exit() If mail sending fails, you'll see the error and a stack trace, try to fix settings in settings.py. See :ref:`edit_settings-py`. All email settings are explained in the Django documentation: https://docs.djangoproject.com/en/dev/topics/email/ If there's no error and "1" is displayed instead, the email was successfully sent and you should analyze the mail server log if you do not receive the test message. .. _troubleshoot-db: Troubleshoot database errors on upgrade ======================================= Fleio database must be created with utf8mb4 character set. You can check the character set by running the following query on your database (usually fleio): .. code-block:: bash cd /home/fleio/compose db_pass=$(cat /home/fleio/compose/secrets/.db_password) docker-compose exec db mysql -u fleio -p"$db_pass" SELECT @@character_set_database, @@collation_database; If your database contains tables that are using a different character set you will have to update the tables so it will have the UTF8mb4 set. This can be done with the following steps: 1. Export current DB .. code-block:: bash cd /home/fleio/compose db_pass=$(cat /home/fleio/compose/secrets/.db_password) docker-compose exec -T db mysqldump fleio -u fleio -p"$db_pass" > fleio`date +%d.%m.%Y`.sql 2. Adjust the dump so all the create table statements will use utf8mb4. 3. Import the database (replace the path to your `FIXED_DB` with the proper name) .. code-block:: bash sudo chown fleio /home/fleio/compose/FIXED_DB.sql sudo -i -u fleio cd /home/fleio/compose # stop all services docker-compose stop # and start just the database service docker-compose start db # get the database password in a variable db_pass=$(cat /home/fleio/compose/secrets/.db_password) # enter the mysql console docker-compose exec db mysql -u fleio -p"$db_pass" # once in console, drop the existing database and create a new one: DROP DATABASE fleio; CREATE DATABASE fleio CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; # exit the console to Bash exit # import the database cat fleio.sql | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass" 4. Run migration using ``docker-compose exec -T backend bash -c "source ../env/bin/activate && django migrate"`` How to enable debugging ======================= If you're in a development environment (not a production server) you can also enable debugging from `settings.py` file to get more information in browser when you're accessing an invalid URL and in case an error occurs during development: .. code-block:: python DEBUG = True Now you'll be able to to see the backend API nicely formatted and even perform POST, PUT, and DELETE requests. Note that you need to be authenticated as Here's an example of PHP code calling the Fleio API: https://github.com/fleio/fleio-whmcs. You can also run Django commands from the command line after you activate the Fleio Python virtual environment and move to the Django project directory. Please see :ref:`djangoCli`. For more information regarding the Django command line see https://docs.djangoproject.com/en/dev/ref/django-admin/ .. _troubleshoot-ceilometer: Ceilometer troubleshooting ========================== Network related metrics ~~~~~~~~~~~~~~~~~~~~~~~ If you properly configured Ceilometer / gnocchi, the metrics should show correctly in the instance's "Metrics" tab. However, if there are no metrics shown even after a few hours after your deployed the instance, you might have some configuration issues. To get the network metrics, please run the following commands: .. code-block:: bash Measures: gnocchi measures show --resource-id=`gnocchi resource list | grep instance_network_interface | tail -n 1 | awk '{print $2}'` --aggregation=max --granularity=3600 network.incoming.bytes Metric: gnocchi metric show --resource-id=`gnocchi resource list | grep instance_network_interface | tail -n 1 | awk '{print $2}'` network.incoming.bytes Archive-policy settings: gnocchi archive-policy show `gnocchi metric show --resource-id=\`gnocchi resource list | grep instance_network_interface | tail -n 1 | awk '{print $2}'\` network.incoming.bytes | grep archive_policy | awk '{print $4}'` If the measures command returns "Aggregation method 'max' at granularity '3600.0' for metric xxxxxx does not exist (HTTP 404)" you must check the archive policy granulation. You should have 1h granulation defined, which was configured with: .. code-block:: bash gnocchi archive-policy create -d granularity:5m,points:290 -d granularity:30m,points:336 -d granularity:1h,points:768 -m max -m mean -m sum your_policy The next step is to run the metrics command and check if your resource is using the correct policy. In the output you should be able to identify the archive_policy/name and you must assure that is the correct one. If all is correct by now, you need to run the third command. This should confirm you that your archive policy does have the correct aggregation methods and it has the correct granularity. If it doesn't have the aggregation_methods max, sum, mean and the correct granularity (shown bellow), please check your configuration. Bellow you will find the correct archive policy: .. code-block:: bash +---------------------+-----------------------------------------------------------------+ | Field | Value | +---------------------+-----------------------------------------------------------------+ | aggregation_methods | max, sum, mean | | back_window | 0 | | definition | - points: 290, granularity: 0:05:00, timespan: 1 day, 0:10:00 | | | - points: 336, granularity: 0:30:00, timespan: 7 days, 0:00:00 | | | - points: 768, granularity: 1:00:00, timespan: 32 days, 0:00:00 | | name | your policy | +---------------------+-----------------------------------------------------------------+ Another thing that needs to be checked is the pipeline.yaml file (see :ref:`pipeline`). In that file, you should check the publishers archive policy (in our example described in the configuring guide is fleio_policy: gnocchi://?filter_project=service&archive_policy=fleio_policy). This does not needs to be persistent across all regions, but it should be the same in both ceilometer's pipeline.yaml file and in gnocchi. If they are different, we recommend to clear all the metrics with ceilometer disabled by running this command: .. code-block:: bash set -e; set -x; for host in $(ssh $(awk '/utility/ {print $NF}' /etc/hosts) "source openrc; openstack hypervisor list -f value -c 'Hypervisor Hostname'"); do ssh $host systemctl stop ceilometer-polling; ssh $host systemctl stop ceilometer-agent-notification; done; ssh $(awk '/utility/ {print $NF}' /etc/hosts) "source openrc; gnocchi archive-policy-rule delete default"; ssh $(awk '/utility/ {print $NF}' /etc/hosts) "source openrc; gnocchi archive-policy create -d granularity:5m,points:290 -d granularity:30m,points:336 -d granularity:1h,points:768 -m max -m mean -m sum fleio; gnocchi archive-policy-rule create -a fleio -m \\* default"; for gnocchi_id in $(ssh $(awk '/utility/ {print $NF}' /etc/hosts) "source openrc; gnocchi resource list -f value -c id"); do ssh $(awk '/utility/ {print $NF}' /etc/hosts) "source openrc; gnocchi resource delete $gnocchi_id"; done; for host in $(ssh $(awk '/utility/ {print $NF}' /etc/hosts) "source openrc; openstack hypervisor list -f value -c 'Hypervisor Hostname'"); do ssh $host systemctl start ceilometer-polling; ssh $host systemctl start ceilometer-agent-notification; done After that, you will have to follow the guide for :ref:`ceilometer-configuration`. CPU usage related metrics ~~~~~~~~~~~~~~~~~~~~~~~~~ .. important:: Data points might take some time to be added, so you might want to wait for at least 30 minutes to 1 hour after the instance was deployed before starting to troubleshoot this. The CPU metrics are required in order to have data in the Instance details -> metrics tab -> CPU metrics graph: .. image:: /_static/images/newstaff/cloud/instances/instance-detailsCPUmetrics.png :scale: 50 If you don't have any data on this graph, you can run the following commands to see if you have any gnocchi metrics for that instance: .. code-block:: bash gnocchi aggregates '(/ (rateofchange (metric cpu max)) 3000000000)' id=$(openstack metric list | grep -w "cpu" | tail -n 1 | awk '{print $2}') --granularity=300 gnocchi aggregates '(/ (rateofchange (metric cpu max)) 18000000000)' id=$(openstack metric list | grep -w "cpu" | tail -n 1 | awk '{print $2}') --granularity=1800 gnocchi aggregates '(/ (rateofchange (metric cpu max)) 36000000000)' id=$(openstack metric list | grep -w "cpu" | tail -n 1 | awk '{print $2}') --granularity=3600 Note that the command from above will return the data points for your last deployed instance. For a specific instance use the following commands: .. code-block:: bash gnocchi aggregates '(/ (rateofchange (metric cpu max)) 3000000000)' id=REPLACE-ME --granularity=300 gnocchi aggregates '(/ (rateofchange (metric cpu max)) 18000000000)' id=REPLACE-ME --granularity=1800 gnocchi aggregates '(/ (rateofchange (metric cpu max)) 36000000000)' id=REPLACE-ME --granularity=3600 If there are datapoints in ceilmoeter, then you should get some values, as in the following example: .. code-block:: none +----------------------------------------------+---------------------------+-------------+---------------------+ | name | timestamp | granularity | value | +----------------------------------------------+---------------------------+-------------+---------------------+ | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T07:40:00+00:00 | 300.0 | 76.76666666666667 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T07:45:00+00:00 | 300.0 | 97.0 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T07:50:00+00:00 | 300.0 | 97.14333333333333 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T07:55:00+00:00 | 300.0 | 97.06 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:00:00+00:00 | 300.0 | 97.15333333333334 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:05:00+00:00 | 300.0 | 96.86333333333333 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:10:00+00:00 | 300.0 | 97.25333333333333 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:15:00+00:00 | 300.0 | 53.93 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:20:00+00:00 | 300.0 | 0.31333333333333335 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:25:00+00:00 | 300.0 | 0.3333333333333333 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:30:00+00:00 | 300.0 | 0.31333333333333335 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:35:00+00:00 | 300.0 | 0.32 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:40:00+00:00 | 300.0 | 0.32 | | ffbe5368-607b-437b-8253-9bc390c8d049/cpu/max | 2021-06-22T08:45:00+00:00 | 300.0 | 0.30666666666666664 | +----------------------------------------------+---------------------------+-------------+---------------------+ If you get the `Metrics not found` error then you will have to re-check the :ref:`ceilometer-configuration` guide. Detailed info on the docker Fleio deployment ============================================ See :ref:`docker_deploy_notes`.