.. _debrpm-to-docker: ========================================== Migrate DEB/RPM to Fleio Docker deployment ========================================== This document applies if you have Fleio installed with one of the system packages (DEB/RPM) and you want to migrate your data and settings to a new Docker Fleio installation. If you want to move an existing Fleio Docker installation (not DEB/RPM) to a new machine, see :ref:`move-docker`. Fleio version 2021.05.1 was the last release to include DEB and RPM system packages. .. important:: **Test first** Run these steps in a lab environment first and make sure you don't get any errors and everything is working correctly before performing the actual production migration to the new Docker install. **Don't lose RabbitMQ notifications** If your lab installation uses the exact same OpenStack settings as your production, including the ``Notifications pool name`` (in staff panel ``Settings`` > ``OpenStack`` > ``Notifications`` tab), some of the OpenStack notifications will be consumed by the lab installation. This means that your production will lose notifications. This has serious consequence on the OpenStack objects state and usage data. Make sure you set new value for ``Notifications pool name`` in your lab installation. **Domain migration strategy** This guide assumes that you have a strategy for pointing your users to the new Fleio installation. As last production migration step you can move IP address to new machine, update DNS settings (showing an "under maintenance" web page on old server while DNS propagates) etc. This guide does not cover this part. In this guide we use: * "old server", "existing installation", "legacy installation" - to refer to the server/machine FROM which you're migrating Fleio * "new server", "new installation" - to refer to the server/machine TO which you are migrating Fleio settings and data The following summary lists the steps needed to migrate Fleio to a new Docker installation. .. contents:: :local: 1. Upgrade existing installation ================================ :ref:`Upgrade your existing (legacy) Fleio installation` to the latest Fleio version. The migrations steps in this guide work with Fleio version 2021.02.0 or newer. .. note:: If you're migrating to Fleio version 2022.11.0 or higher, change "docker-compose" commands to "docker compose". 2. New installation with Docker =============================== On a separate, freshly installed machine, :ref:`install Fleio using the Docker deploy method`. **Make sure you install on the new server the exact same Fleio version as on the old server.** Fill in the desired Fleio installation URL. This may be a new test domain if you're just testing the migration or the exact same URL as production if you're preparing for final production migration. If you're using the production domain you can run your tests by setting the new machine IP in ``/etc/hosts`` on a Linux client workstation for tests (not on server!) or in ``%WINDIR%\System32\drivers\etc\hosts`` on Windows. This way, when browsing from the test installation, it will connect to the new Fleio installation. This test procedure is also described here https://hosterion.com/client/index.php?rp=/knowledgebase/2/How-to-preview-your-website.html .. _docker-import-convert-db: 3. Export, import and convert database to utf8mb4 ================================================= 3.1. Export the database from the old server -------------------------------------------- SSH to your **old Fleio server** and run this as ``root`` to export the database: .. code-block:: bash mysqldump fleio > fleio.sql 3.2. Import the database ------------------------ Copy the ``fleio.sql`` file to your **new server** at ``/home/fleio/compose/fleio.sql`` and make sure the file has read permissions for the ``fleio`` user. .. warning:: Commands below are permanently deleting the fleio database. Make sure you are on the new server, and you do not need the data. .. code-block:: bash sudo chown fleio /home/fleio/compose/fleio.sql sudo -i -u fleio cd /home/fleio/compose # stop all services docker-compose stop # and start just the database service docker-compose start db # get the database password in a variable db_pass=$(cat /home/fleio/compose/secrets/.db_password) # enter the mysql console docker-compose exec db mysql -u fleio -p"$db_pass" # once in console, drop the existing database and create a new one: DROP DATABASE fleio; CREATE DATABASE fleio CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; # exit the console to Bash exit # import the database cat fleio.sql | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass" 3.3. Convert database to charset utf8mb4 ---------------------------------------- The utf8 charset we've used for the Fleio database until now has some limitations. That's why we're converting the database charset and collation to utf8mb4. We've created and we'll use an open source script for this conversion: https://github.com/fleio/utf8mb4-convert To have access to the database container on the Docker Fleio installation, enter the shell of the ``utils`` container and get the database password: .. code-block:: bash fleio bash db_pass=$(cat /run/secrets/fleio_db_password) To confirm the successful database charset conversion we'll look at charset before and after the conversion script. Here's how it initially looks: .. code-block:: mysql sql='SELECT DISTINCT column_name,table_name,character_set_name,collation_name FROM information_schema.columns WHERE table_schema = "fleio" AND character_set_name IS NOT NULL;' mysql -h db -u fleio -p"$db_pass" -e "$sql" +--------------------------------+------------------------------------------------------------------+--------------------+-----------------+ | column_name | table_name | character_set_name | collation_name | +--------------------------------+------------------------------------------------------------------+--------------------+-----------------+ | ip_address | billing_serviceassignedip | utf8 | utf8_general_ci | | name | notifications_category | utf8 | utf8_general_ci | | description | notifications_category | utf8 | utf8_general_ci | ... Notice that the current field charset is ``utf8`` and the collation is ``utf8_general_ci``. That will change after conversion. Let's download and run the conversion script: .. code-block:: bash curl -O https://raw.githubusercontent.com/fleio/utf8mb4-convert/main/convert.sh chmod +x convert.sh ./convert.sh fleio -h db -u fleio -p"$db_pass" Details regarding the conversion operation are offered, and you need to confirm the operation twice by pressing ``y``, ``ENTER``. The conversion itself will take some time. After conversion, you can check to confirm that all fields have character set ``utf8mb4`` and collation ``utf8mb4_unicode_ci``: .. code-block:: mysql sql='SELECT DISTINCT column_name,table_name,character_set_name,collation_name FROM information_schema.columns WHERE table_schema = "fleio" AND character_set_name IS NOT NULL;' mysql -h db -u fleio -p"$db_pass" -e "$sql" +--------------------------------+------------------------------------------------------------------+--------------------+--------------------+ | column_name | table_name | character_set_name | collation_name | +--------------------------------+------------------------------------------------------------------+--------------------+--------------------+ | ip_address | billing_serviceassignedip | utf8mb4 | utf8mb4_unicode_ci | | name | notifications_category | utf8mb4 | utf8mb4_unicode_ci | | description | notifications_category | utf8mb4 | utf8mb4_unicode_ci | ... Exit the ``utils`` container to the host shell: .. code-block:: bash exit 4. Update OpenStack settings ============================ You have two options here. Depending on if you want to connect the new Fleio installation to another OpenStack cloud (Option A) or you want to connect the new Fleio installation to the same OpenStack cloud as on the old server (Option B). .. _docker-use-prod-cloud: 4.A. Option A: you'll use another OpenStack cloud (not the production cloud) ---------------------------------------------------------------------------- The imported production database contains connection details to your initial (production) OpenStack cloud. Let's overwrite the important OpenStack settings with an empty string. This ensures that your new installation will not connect to the production OpenStack cloud and, more importantly, it will not consume RabbitMQ notification messages: .. code-block:: bash # we assume you're already under the fleio user, after previously running sudo -i -u fleio # get the database password in a variable db_pass=$(cat /home/fleio/compose/secrets/.db_password) cd /home/fleio/compose sql="UPDATE conf_option SET value='' WHERE section='OPENSTACK_PLUGIN' AND field='auth_url';" echo "$sql" | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass" sql="UPDATE conf_option SET value='' WHERE section='OPENSTACK_PLUGIN' AND field='notifications_url';" echo "$sql" | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass" Below is a step to fill in the OpenStack connection details in staff panel. or .. _docker-update-pool: 4.B. Option B: the new installation will be connected to the same cloud as the old installation ----------------------------------------------------------------------------------------------- You want to make sure the production Fleio installation and this installation won't consume OpenStack RabbitMQ notifications from the same ``Notifications pool name``: .. code-block:: bash # we assume you're already under the fleio user, after previously running sudo -i -u fleio # get the database password in a variable db_pass=$(cat /home/fleio/compose/secrets/.db_password) cd /home/fleio/compose # you can change "new-pool" with any value, as it is automatically created # just DON'T use the same value as in the production installation sql="UPDATE conf_option SET value='new-pool' WHERE section='OPENSTACK_PLUGIN' AND field='notifications_pool';" echo "$sql" | docker-compose exec -T db mysql fleio -u fleio -p"$db_pass" 5. Copy backend settings ======================== You basically need to merge the old installation's ``settings.py`` with the new Docker install default ``settings.py``. In the following steps you are using the old ``settings.py`` as base file and copy only the absolutely needed settings from the new installation. This way you make sure that you're not losing any settings you've made on the old installation. First let's make a copy of the default docker install ``settings.py``. You'll need to copy the needed sections. Run this on the new server: .. code-block:: bash fleio bash cat /var/webapps/fleio/project/fleio/fleiosettings/settings.py > settings.bak Then edit ``settings.py`` on the same new server: .. code-block:: bash fleio edit settings.py Replace the full content of the **new** ``settings.py`` (which you should have in the editor after the above command) with the full content of the file **from the old server**: ``/var/webapps/fleio/project/fleio/fleiosettings/settings.py``. At this point you've made sure you've taken all the backend settings you had on the old server. But it also means that the Docker installation is now broken, because some settings are needed for the Docker containers to communicate. To fix this, you'll copy and paste to replace the sections below from the ``settings.bak`` file you've created above. It will be easier if you open two terminals: one running ``fleio edit settings.py`` and another one to ``cat settings.bak`` content. So, using two terminals, copy each of the following sections from ``settings.bak`` to ``fleio edit settings.py`` by **REPLACING** the same section/variable: * Replace the database connection configuration with the lines from ``settings.bak`` (your password will be different, this is just an example) .. code-block:: python DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'fleio', 'USER': 'fleio', 'PASSWORD': '<< YOUR PASSWORD HERE >>', 'HOST': 'db', 'PORT': '3306', 'CONN_MAX_AGE': 500, 'OPTIONS': { 'init_command': "SET sql_mode='STRICT_ALL_TABLES'" } }, } * add these lines at the end of file or replace them, if you find these variables in ``settings.py``: .. code-block:: python CELERY_BROKER_URL = 'redis://redis:6379/0' CELERY_RESULT_BACKEND = 'redis://redis:6379/0/' .. warning:: If your new installation is connected to the production cloud, disable periodic tasks. See below how. If your new installation is connected to the production cloud, you want to avoid processing clients and suspending clients' resources in OpenStack. Disable periodic tasks in the new installation by adding this at the end of ``settings.py``: .. code-block:: python CELERY_BEAT_SCHEDULE = {} If your new install will have a new URL than the old install, you also need to edit these variables. If you're keeping the same URL, leave them as are (your domain will differ, this is just an example): .. code-block:: python ALLOWED_HOSTS = ['my-domain.com'] FRONTEND_URL = 'http://my-domain.com' If you have multiple domains configured, and you're changing domains during this migration, make sure you also update ``CORS_ORIGIN_WHITELIST``. Save and exit the editor. Answer ``n`` (no) when asked if you want to restart the services: .. code-block:: bash Services restart is required to apply changes. Restart now? [Y/n]: n .. warning:: After copying over the updated settings.py, the license will be marked as invalid. You will have to contact fleio support (support at fleio.com) in order to reset your license. 6. Copy frontend settings ========================= Copy/paste the content of each of the four frontend configuration files. If you're also changing the Fleio URL during this migration, update the relevant settings. Use the same domain you have set during the Docker installation. The example strings below assume you have installed Fleio at ``https://my-domain.com`` or ``http://my-domain.com`` URL. Note that the ``new`` and ``newstaff`` prefixes are important, they point to the Angular version of Fleio, which still co-exist with the AngularJS frontend. Save and exit the ``fleio edit ...`` command after each of the four files below. 6.1. Site constants.js ---------------------- The following assumes that you have added custom settings to site's ``constants.js``. If you don't have custom settings, you can skip this: * Copy the file content from the old installation: ``/var/webapps/fleio/frontend/site/constants/constants.js``. * Edit new file by removing all existing content and pasting content from the old file above: ``fleio edit site_constants.js`` (the command opens an editor). Even if you don't have custom settings, this may be needed (depending on your new installation URL): * Optional: update these variables' values, if you're changing URL (you'll have a different domain, this is just an example) .. code-block:: javascript api_url: 'https://my-domain.com/backend/api', angularPanelUrl: '//my-domain.com/new/', 6.2. Staff constants.js ----------------------- The following assumes that you have added custom settings to staff's ``constants.js``. If you don't have custom settings, you can skip this: * Copy the file content from the old installation: ``/var/webapps/fleio/frontend/site/staff/constants/constants.js``. * Edit new file by removing all existing content and pasting content from the old file above: ``fleio edit staff_constants.js``. Even if you don't have custom settings, this may be needed (depending on your new installation URL): * Optional: update these variables' values, if you're changing URL (you'll have a different domain, this is just an example) .. code-block:: javascript api_url: 'https://my-domain.com/backend/staffapi', angularPanelUrl: 'https://my-domain.com/newstaff/', 6.3. enduser.config.json ------------------------ The following assumes that you have added custom settings to ``enduser.config.json``. If you don't have custom settings, you can skip this: * Copy the file content from the old installation: ``/var/webapps/fleio/frontend/enduser/assets/config/enduser.config.json``. * Edit new file by removing all existing content and pasting content from the old file above: ``fleio edit enduser.config.json``. Even if you don't have custom settings, this may be needed (depending on your new installation URL): * Optional: update these variables' values, if you're changing URL (you'll have a different domain, this is just an example) .. code-block:: javascript "backendApiUrl": "//my-domain.com/backend/api/", "angularJsPanelUrl": "//my-domain.com/", 6.4. staff.config.json ---------------------- The following assumes that you have added custom settings to ``staff.config.json``. If you don't have custom settings, you can skip this: * Copy the file content from the old installation: ``/var/webapps/fleio/frontend/staff/assets/config/staff.config.json``. * Edit new file by removing all existing content and pasting content from the old file above: ``fleio edit staff.config.json``. Even if you don't have custom settings, this may be needed (depending on your new installation URL): * Optional: update these variables' values, if you're changing URL (you'll have a different domain, this is just an example) .. code-block:: javascript "backendApiUrl": "//my-domain.com/client/backend/staffapi/", "homeName": "staff", "angularJsPanelUrl": "//my-domain.com/client/staff/" .. _docker-start-sync: 7. Start and sync ================= Recreate containers to apply settings and start services: .. code-block:: bash fleio recreate If you've used :ref:`docker-use-prod-cloud`, you need to fill in connection details of this new OpenStack cloud. Go to URL ``/newstaff/settings/openstack`` of the new installation and fill in fields on ``Credentials`` and on ``Notifications`` tabs. Now that all the Docker containers are running (including ``fleio-updated-1``, which consumes RabbitMQ message notifications from OpenStack) run one more sync to make sure there aren't any OpenStack object changes that are not reflected in the Fleio database: .. code-block:: bash fleio sync 8. Test the new installation ============================= Check that the staff panel and the user panel are working correctly. If you encounter any errors check if all is properly set in your configuration files. Here's how to edit the configuration files: .. code-block:: bash fleio edit settings.py fleio edit site_constants.js fleio edit staff_constants.js fleio edit enduser.config.json fleio edit staff.config.json You can also contact our support, and we will gladly help you fix your Fleio installation. 9. Perform actual live migration ================================= Now that you've completed at least one test migration, you will do one more data copy and point your production domain to the new installation. .. important:: This step assumes that you have already performed all previous steps with the production URL. Only perform this step after you have thoroughly tested that everything works on the new installation. 9.1. Put old installation in maintenance ----------------------------------------- To put your old machine on maintenance mode add some HTML file with "in maintenance" message and configure nginx to show this page to any end-user HTTP request. To make sure the old installation is no longer consuming RabbitMQ messages stop the ``fleio`` service and disable it, so that it won't automatically start on a system, reboot: .. code-block:: bash systemctl status fleio systemctl disable fleio 9.2. Copy data one more time ----------------------------- Since your production installation probably had some changes during your test migrations it's now time to get data again. So perform these steps again: * :ref:`docker-import-convert-db`. * It's safer to use a different notification pool, even if the old installation is theoretically deactivated, so: :ref:`docker-update-pool`. * :ref:`docker-start-sync` 9.3. Point production domain to new install -------------------------------------------- The last step is to point your production domain and, hence, your users, to the new Fleio installation. You can move IP to new machine, update new IP in DNS settings or whatever method you choose. 9.4. Enable periodic tasks -------------------------- If you've disabled periodic tasks by adding line ``CELERY_BEAT_SCHEDULE = {}`` at the end of ``settings.py``, remove it now (use ``fleio edit settings.py``), then run ``fleio restart`` to apply changes.