Migrate DEB/RPM to Fleio Docker deployment¶

This document applies if you have Fleio installed with one of the system packages (DEB/RPM) and you want to migrate your data and settings to a new Docker Fleio installation.

If you want to move an existing Fleio Docker installation (not DEB/RPM) to a new machine, see Move Fleio Docker installation.

Fleio version 2021.05.1 was the last release to include DEB and RPM system packages.

Important

Test first

Run these steps in a lab environment first and make sure you don’t get any errors and everything is working correctly before performing the actual production migration to the new Docker install.

Don’t lose RabbitMQ notifications

If your lab installation uses the exact same OpenStack settings as your production, including the Notifications pool name (in the staff panel Settings > OpenStack > NOTIFICATIONS tab), some of the OpenStack notifications will be consumed by the lab installation. This means that your production will lose notifications. This has serious consequence on the OpenStack objects state and usage data. Make sure you set new value for Notifications pool name in your lab installation.

Domain migration strategy

This guide assumes that you have a strategy for pointing your users to the new Fleio installation. As last production migration step you can move IP address to new machine, update DNS settings (showing an “under maintenance” web page on old server while DNS propagates) etc. This guide does not cover this part.

In this guide we use:

“old server”, “existing installation”, “legacy installation” - to refer to the server/machine FROM which you’re migrating Fleio
“new server”, “new installation” - to refer to the server/machine TO which you are migrating Fleio settings and data

The following summary lists the steps needed to migrate Fleio to a new Docker installation.

1. Upgrade existing installation¶

Upgrade your existing (legacy) Fleio installation to the latest Fleio version.

The migrations steps in this guide work with Fleio version 2021.02.0 or newer.

Note

If you’re migrating to Fleio version 2022.11.0 or higher, change “docker-compose” commands to “docker compose”.

2. New installation with Docker¶

On a separate, freshly installed machine, install Fleio using the Docker deploy method. Make sure you install on the new server the exact same Fleio version as on the old server.

Fill in the desired Fleio installation URL. This may be a new test domain if you’re just testing the migration or the exact same URL as production if you’re preparing for final production migration.

If you’re using the production domain you can run your tests by setting the new machine IP in /etc/hosts on a Linux client workstation for tests (not on server!) or in %WINDIR%\System32\drivers\etc\hosts on Windows. This way, when browsing from the test installation, it will connect to the new Fleio installation. This test procedure is also described here https://hosterion.com/client/index.php?rp=/knowledgebase/2/How-to-preview-your-website.html

3. Export, import and convert database to utf8mb4¶

3.1. Export the database from the old server¶

SSH to your old Fleio server and run this as root to export the database:

mysqldump fleio > fleio.sql

3.2. Import the database¶

Copy the fleio.sql file to your new server at /home/fleio/compose/fleio.sql and make sure the file has read permissions for the fleio user.

Warning

Commands below are permanently deleting the fleio database. Make sure you are on the new server, and you do not need the data.

sudo chown fleio /home/fleio/compose/fleio.sql
sudo -i -u fleio
cd /home/fleio/compose
# stop all services
docker-compose stop
# and start just the database service
docker-compose start db
# get the database password in a variable
db_pass=$(cat /home/fleio/compose/secrets/.db_password)
# enter the mysql console
docker-compose exec db mysql -u fleio -p"$db_pass"
# once in console, drop the existing database and create a new one:
DROP DATABASE fleio;
CREATE DATABASE fleio CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
# exit the console to Bash
exit
# import the database
cat fleio.sql | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass"

3.3. Convert database to charset utf8mb4¶

The utf8 charset we’ve used for the Fleio database until now has some limitations. That’s why we’re converting the database charset and collation to utf8mb4.

We’ve created and we’ll use an open source script for this conversion: https://github.com/fleio/utf8mb4-convert

To have access to the database container on the Docker Fleio installation, enter the shell of the utils container and get the database password:

fleio bash
db_pass=$(cat /run/secrets/fleio_db_password)

To confirm the successful database charset conversion we’ll look at charset before and after the conversion script. Here’s how it initially looks:

sql='SELECT DISTINCT column_name,table_name,character_set_name,collation_name FROM information_schema.columns WHERE table_schema = "fleio" AND character_set_name IS NOT NULL;'
mysql -h db -u fleio -p"$db_pass" -e "$sql"

+--------------------------------+------------------------------------------------------------------+--------------------+-----------------+
| column_name                    | table_name                                                       | character_set_name | collation_name  |
+--------------------------------+------------------------------------------------------------------+--------------------+-----------------+
| ip_address                     | billing_serviceassignedip                                        | utf8               | utf8_general_ci |
| name                           | notifications_category                                           | utf8               | utf8_general_ci |
| description                    | notifications_category                                           | utf8               | utf8_general_ci |
...

Notice that the current field charset is utf8 and the collation is utf8_general_ci. That will change after conversion.

Let’s download and run the conversion script:

curl -O https://raw.githubusercontent.com/fleio/utf8mb4-convert/main/convert.sh
chmod +x convert.sh
./convert.sh fleio -h db -u fleio -p"$db_pass"

Details regarding the conversion operation are offered, and you need to confirm the operation twice by pressing y, ENTER.

The conversion itself will take some time.

After conversion, you can check to confirm that all fields have character set utf8mb4 and collation utf8mb4_unicode_ci:

sql='SELECT DISTINCT column_name,table_name,character_set_name,collation_name FROM information_schema.columns WHERE table_schema = "fleio" AND character_set_name IS NOT NULL;'
mysql -h db -u fleio -p"$db_pass" -e "$sql"
+--------------------------------+------------------------------------------------------------------+--------------------+--------------------+
| column_name                    | table_name                                                       | character_set_name | collation_name     |
+--------------------------------+------------------------------------------------------------------+--------------------+--------------------+
| ip_address                     | billing_serviceassignedip                                        | utf8mb4            | utf8mb4_unicode_ci |
| name                           | notifications_category                                           | utf8mb4            | utf8mb4_unicode_ci |
| description                    | notifications_category                                           | utf8mb4            | utf8mb4_unicode_ci |
...

Exit the utils container to the host shell:

exit

4. Update OpenStack settings¶

You have two options here. Depending on if you want to connect the new Fleio installation to another OpenStack cloud (Option A) or you want to connect the new Fleio installation to the same OpenStack cloud as on the old server (Option B).

4.A. Option A: you’ll use another OpenStack cloud (not the production cloud)¶

The imported production database contains connection details to your initial (production) OpenStack cloud.

Let’s overwrite the important OpenStack settings with an empty string. This ensures that your new installation will not connect to the production OpenStack cloud and, more importantly, it will not consume RabbitMQ notification messages:

# we assume you're already under the fleio user, after previously running sudo -i -u fleio
# get the database password in a variable
db_pass=$(cat /home/fleio/compose/secrets/.db_password)
cd /home/fleio/compose
sql="UPDATE conf_option SET value='' WHERE section='OPENSTACK_PLUGIN' AND field='auth_url';"
echo "$sql" | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass"
sql="UPDATE conf_option SET value='' WHERE section='OPENSTACK_PLUGIN' AND field='notifications_url';"
echo "$sql" | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass"

Below is a step to fill in the OpenStack connection details in the staff panel.

or

4.B. Option B: the new installation will be connected to the same cloud as the old installation¶

You want to make sure the production Fleio installation and this installation won’t consume OpenStack RabbitMQ notifications from the same Notifications pool name:

# we assume you're already under the fleio user, after previously running sudo -i -u fleio
# get the database password in a variable
db_pass=$(cat /home/fleio/compose/secrets/.db_password)
cd /home/fleio/compose
# you can change "new-pool" with any value, as it is automatically created
# just DON'T use the same value as in the production installation
sql="UPDATE conf_option SET value='new-pool' WHERE section='OPENSTACK_PLUGIN' AND field='notifications_pool';"
echo "$sql" | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass"

5. Copy backend settings¶

You basically need to merge the old installation’s settings.py with the new Docker install default settings.py.

In the following steps you are using the old settings.py as base file and you only copy the absolutely needed settings from the new installation. This way you make sure that you’re not losing any settings you’ve made on the old installation.

First let’s make a copy of the default docker install settings.py. You’ll need to copy the needed sections. Run this on the new server:

fleio bash cat /var/webapps/fleio/project/fleio/fleiosettings/settings.py > settings.bak

Then edit settings.py on the same new server:

fleio edit settings.py

Replace the full content of the new settings.py (which you should have in the editor after the above command) with the full content of the file from the old server: /var/webapps/fleio/project/fleio/fleiosettings/settings.py.

At this point you’ve made sure you’ve taken all the backend settings you had on the old server. But it also means that the Docker installation is now broken, because some settings are needed for the Docker containers to communicate.

To fix this, you’ll copy and paste to replace the sections below from the settings.bak file you’ve created above. It will be easier if you open two terminals: one running fleio edit settings.py and another one to cat settings.bak content.

So, using two terminals, copy each of the following sections from settings.bak to fleio edit settings.py by REPLACING the same section/variable:

Replace the database connection configuration with the lines from settings.bak (your password will be different, this is just an example)

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'fleio',
        'USER': 'fleio',
        'PASSWORD': '<< YOUR PASSWORD HERE >>',
        'HOST': 'db',
        'PORT': '3306',
        'CONN_MAX_AGE': 500,
        'OPTIONS': {
            'init_command': "SET sql_mode='STRICT_ALL_TABLES'"
        }
    },
}

add these lines at the end of file or replace them, if you find these variables in settings.py:

CELERY_BROKER_URL = 'redis://redis:6379/0'
CELERY_RESULT_BACKEND = 'redis://redis:6379/0/'

Warning

If your new installation is connected to the production cloud, disable periodic tasks. See below how.

If your new installation is connected to the production cloud, you want to avoid processing clients and suspending clients’ resources in OpenStack. Disable periodic tasks in the new installation by adding this at the end of settings.py:

CELERY_BEAT_SCHEDULE = {}

If your new install will have a new URL than the old install, you also need to edit these variables. If you’re keeping the same URL, leave them as are (your domain will differ, this is just an example):

ALLOWED_HOSTS = ['my-domain.com']
FRONTEND_URL = 'http://my-domain.com'

If you have multiple domains configured, and you’re changing domains during this migration, make sure you also update CORS_ORIGIN_WHITELIST.

Save and exit the editor. Answer n (no) when asked if you want to restart the services:

Services restart is required to apply changes. Restart now? [Y/n]: n

Warning

After copying over the updated settings.py, the license will be marked as invalid. You will have to contact fleio support (support at fleio.com) in order to reset your license.

6. Copy frontend settings¶

Copy/paste the content of each of the four frontend configuration files. If you’re also changing the Fleio URL during this migration, update the relevant settings.

Use the same domain you have set during the Docker installation. The example strings below assume you have installed Fleio at https://my-domain.com or http://my-domain.com URL. Note that the new and newstaff prefixes are important, they point to the Angular version of Fleio, which still co-exist with the AngularJS frontend.

Save and exit the fleio edit ... command after each of the four files below.

6.1. Site constants.js¶

The following assumes that you have added custom settings to site’s constants.js. If you don’t have custom settings, you can skip this:

Copy the file content from the old installation: /var/webapps/fleio/frontend/site/constants/constants.js.

Edit new file by removing all existing content and pasting content from the old file above: fleio edit site_constants.js (the command opens an editor).

Even if you don’t have custom settings, this may be needed (depending on your new installation URL):

Optional: update these variables’ values, if you’re changing URL (you’ll have a different domain, this is just an example)
api_url: 'https://my-domain.com/backend/api',
angularPanelUrl: '//my-domain.com/new/',

6.2. Staff constants.js¶

The following assumes that you have added custom settings to staff’s constants.js. If you don’t have custom settings, you can skip this:

Copy the file content from the old installation: /var/webapps/fleio/frontend/site/staff/constants/constants.js.

Edit new file by removing all existing content and pasting content from the old file above: fleio edit staff_constants.js.

Even if you don’t have custom settings, this may be needed (depending on your new installation URL):

Optional: update these variables’ values, if you’re changing URL (you’ll have a different domain, this is just an example)
api_url: 'https://my-domain.com/backend/staffapi',
angularPanelUrl: 'https://my-domain.com/newstaff/',

6.3. enduser.config.json¶

The following assumes that you have added custom settings to enduser.config.json. If you don’t have custom settings, you can skip this:

Copy the file content from the old installation: /var/webapps/fleio/frontend/enduser/assets/config/enduser.config.json.

Edit new file by removing all existing content and pasting content from the old file above: fleio edit enduser.config.json.

Even if you don’t have custom settings, this may be needed (depending on your new installation URL):

Optional: update these variables’ values, if you’re changing URL (you’ll have a different domain, this is just an example)

"backendApiUrl": "//my-domain.com/backend/api/",
"angularJsPanelUrl": "//my-domain.com/",

6.4. staff.config.json¶

The following assumes that you have added custom settings to staff.config.json. If you don’t have custom settings, you can skip this:

Copy the file content from the old installation: /var/webapps/fleio/frontend/staff/assets/config/staff.config.json.

Edit new file by removing all existing content and pasting content from the old file above: fleio edit staff.config.json.

Even if you don’t have custom settings, this may be needed (depending on your new installation URL):

Optional: update these variables’ values, if you’re changing URL (you’ll have a different domain, this is just an example)

"backendApiUrl": "//my-domain.com/client/backend/staffapi/",
"homeName": "staff",
"angularJsPanelUrl": "//my-domain.com/client/staff/"

7. Start and sync¶

Recreate containers to apply settings and start services:

fleio recreate

If you’ve used 4.A. Option A: you’ll use another OpenStack cloud (not the production cloud), you need to fill in connection details of this new OpenStack cloud. Go to URL /newstaff/settings/openstack of the new installation and fill in fields on Credentials and on Notifications tabs.

Now that all the Docker containers are running (including fleio-updated-1, which consumes RabbitMQ message notifications from OpenStack) run one more sync to make sure there aren’t any OpenStack object changes that are not reflected in the Fleio database:

fleio sync

8. Test the new installation¶

Check that the staff panel and the user panel are working correctly.

If you encounter any errors check if all is properly set in your configuration files. Here’s how to edit the configuration files:

fleio edit settings.py
fleio edit site_constants.js
fleio edit staff_constants.js
fleio edit enduser.config.json
fleio edit staff.config.json

You can also contact our support, and we will gladly help you fix your Fleio installation.

9. Perform actual live migration¶

Now that you’ve completed at least one test migration, you will do one more data copy and point your production domain to the new installation.

Important

This step assumes that you have already performed all previous steps with the production URL.

Only perform this step after you have thoroughly tested that everything works on the new installation.

9.1. Put old installation in maintenance¶

To put your old machine on maintenance mode add some HTML file with “in maintenance” message and configure nginx to show this page to any end-user HTTP request.

To make sure the old installation is no longer consuming RabbitMQ messages stop the fleio service and disable it, so that it won’t automatically start on a system, reboot:

systemctl status fleio
systemctl disable fleio

9.2. Copy data one more time¶

Since your production installation probably had some changes during your test migrations it’s now time to get data again. So perform these steps again:

3. Export, import and convert database to utf8mb4.
It’s safer to use a different notification pool, even if the old installation is theoretically deactivated, so: 4.B. Option B: the new installation will be connected to the same cloud as the old installation.
7. Start and sync

9.3. Point production domain to new install¶

The last step is to point your production domain and, hence, your users, to the new Fleio installation. You can move IP to new machine, update new IP in DNS settings or whatever method you choose.

9.4. Enable periodic tasks¶

If you’ve disabled periodic tasks by adding line CELERY_BEAT_SCHEDULE = {} at the end of settings.py, remove it now (use fleio edit settings.py), then run fleio restart to apply changes.

Documentation

Navigation

Related Topics

Migrate DEB/RPM to Fleio Docker deployment¶

1. Upgrade existing installation¶

2. New installation with Docker¶

3. Export, import and convert database to utf8mb4¶

3.1. Export the database from the old server¶

3.2. Import the database¶

3.3. Convert database to charset utf8mb4¶

4. Update OpenStack settings¶

4.A. Option A: you’ll use another OpenStack cloud (not the production cloud)¶

4.B. Option B: the new installation will be connected to the same cloud as the old installation¶

5. Copy backend settings¶

6. Copy frontend settings¶

6.1. Site constants.js¶

6.2. Staff constants.js¶

6.3. enduser.config.json¶

6.4. staff.config.json¶

7. Start and sync¶

8. Test the new installation¶

9. Perform actual live migration¶

9.1. Put old installation in maintenance¶

9.2. Copy data one more time¶

9.3. Point production domain to new install¶

9.4. Enable periodic tasks¶

Migrate DEB/RPM to Fleio Docker deployment¶

1. Upgrade existing installation¶

2. New installation with Docker¶

3. Export, import and convert database to utf8mb4¶

3.1. Export the database from the old server¶

3.2. Import the database¶

3.3. Convert database to charset utf8mb4¶

4. Update OpenStack settings¶

4.A. Option A: you’ll use another OpenStack cloud (not the production cloud)¶

4.B. Option B: the new installation will be connected to the same cloud as the old installation¶

5. Copy backend settings¶

6. Copy frontend settings¶

6.1. Site constants.js¶

6.2. Staff constants.js¶

6.3. enduser.config.json¶

6.4. staff.config.json¶

7. Start and sync¶

8. Test the new installation¶

9. Perform actual live migration¶

9.1. Put old installation in maintenance¶

9.2. Copy data one more time¶

9.3. Point production domain to new install¶

9.4. Enable periodic tasks¶

Join our mailing list to stay up to date on Fleio development, discounts, OpenStack tutorials and OpenStack news.