Upgrade Notes 26.02-26.04 | Webitel Documentation

As part of the new release, Webitel deb package builds have been migrated from Bamboo to GitHub Actions. Along with this, the versioning process has changed: package revision is now tied to the GitHub Actions workflow run, and packages are published to a new private S3 APT repository with a new GPG signing key.

The migration also adds support for Debian 13 alongside Debian 12 and switches part of the package builds to multi-architecture mode:

backend services are now built for amd64 and arm64;
frontend packages are published as architecture-independent (all);
the PostgreSQL extension and FreeSWITCH modules remain amd64-only for now.

For Debian 13, the supported combination is PostgreSQL 18 and the webitel-postgresql-18 package; for Debian 12, PostgreSQL 15 and webitel-postgresql-15 remain. Upgrading from Debian 12 → Debian 13 for installations with the Webitel PostgreSQL extension must be planned together with the major PostgreSQL 15 → 18 upgrade.

There are several practical changes in the packages: some frontend packages have received the -app suffix, backend binaries and some systemd units have received the webitel- prefix, systemd units are no longer shipped via webitel-common, and service backend packages now include maintainer scripts.

Steps

Before upgrading, perform a database backup.

Stopping Webitel services

Bash

systemctl stop webitel-api webitel-app webitel-uac messages-bot messages-srv \
  engine call_center storage flow_manager rabbitmq-server webitel-logger \
  webitel-cases webitel-media-exporter consul freeswitch ngcp-rtpengine-daemon \
  opensips grafana-server nginx

Removing the outdated Webitel repository

Bash

rm -f /etc/apt/sources.list.d/webitel.list
rm -f /etc/apt/auth.conf.d/webitel.conf
rm -f /usr/share/keyrings/webitel-repo.gpg

Dist-upgrade Debian 12 → 13

Performed only when upgrading to Debian 13.

If the installation remains on Debian 12, skip this step.

Temporarily disable custom APT repositories, switch Debian sources to trixie, perform the upgrade, and reboot the server:

Bash

find /etc/apt/sources.list.d -type f \( -name '*.list' -o -name '*.sources' \) \
  -exec mv {} {}.disabled \;

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list

apt update
apt upgrade --without-new-pkgs
apt full-upgrade

reboot

After rebooting, restore the custom repositories and update the codename where the repository supports Debian 13:

Bash

find /etc/apt/sources.list.d -type f -name '*.disabled' \
  -exec sh -c 'for f do mv "$f" "${f%.disabled}"; done' sh {} +

find /etc/apt/sources.list.d -type f \( -name '*.list' -o -name '*.sources' \) \
  -exec sed -i 's/bookworm/trixie/g' {} +

apt update

Connecting the Webitel S3 APT repository

Obtain the current AccessKeyId and SecretAccessKey.

Add Webitel Debian repository script

Bash

apt-get install -y apt-transport-s3

ACCESS_KEY=CLIENT_KEY_ID
ACCESS_SECRET=CLIENT_SECRET_KEY
tee /etc/apt/s3auth.conf <<EOF
AccessKeyId = ${ACCESS_KEY}
SecretAccessKey = ${ACCESS_SECRET}
Region = eu-central-1
EOF

chmod 600 /etc/apt/s3auth.conf

tee /tmp/webitel-apt-repo.gpg.pub <<'EOF'
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBGnOlQoBEACwZi7u6FieiaQ34EHQanCb+5YXJ7ZSJtk7OXjnRT7m3r/PbMnP
Q4PcIqY7coib5OJlan0T+k5ZM2zeU6j1ppUNAW0by9Do7oKWt2cqpa/W0hhN+v6d
XX8RUEBVfzrX0nROQuRhaL05zGc8Su7XF1h/9y5oF4M+ccUSA0eNB667G5F83Tl6
/lURHjyT81G40Jll3DNWPN+MkBZGR/oSH+25XvKuXvqvbXyoDv/42XYANndQT3u+
aZCLX8NMEe+AM6fXP0WRgEvAbC/EVUaULDXxG7m9wAgyKWYKsTFvd8ICWJ2CKtYe
4xkVrVz4+47gWHRbb9zIJ4iMR6f58f5BZIz0U3dFK3F8eOxHAuRl1IHWuNFTm+e1
2mFdBos//6hZ0XJRvwhpY/Ob4PB2AiT8jfTQwiLNKuoMsfdppAoNYnYGq/iNnODA
ZvNKdo5dgfre5vNyQcApt3I46/lu4CvCW7hHAXWCyXPd3byURfGENqhXXfUMRzt2
wHcvhlbc5LEWU5b9PnxtdmJWU1xjvsPkOG4Kecz30OOenulH6EPC6xuWD0EsJEVa
BxRc4rY71FFnR5+CAFmU9RbCijFBOPSIt2LZfq/Zg2eTkoPqpPIb6HMhOZuOHd44
cSx7O/8BR2irc7vkXAaFINnIt6odGNJgO2hZ/N5ihilSQ7MzZFF+kjcoywARAQAB
tCtXZWJpdGVsIEFQVCBSZXBvc2l0b3J5IDxjbG91ZEBAd2ViaXRlbC5jb20+iQJO
BBMBCgA4FiEEWmwW0o67gbGl9rfKN/v3fS6YiiAFAmnOlQoCGwMFCwkIBwIGFQoJ
CAsCBBYCAwECHgECF4AACgkQN/v3fS6YiiD+xg//d3jpGYcYsWhG/Ex730nNeBu9
NvOYR4oWT2cnEa7fdV1raDbviioM0LsbQNghzYE2ctwu8EowkGJIzRxFOYC1ieze
eYLSPevVNeJVP8Ef5/vej+OvvkeBqy6rI/hcKcg8n6EsVOTgJm6P2FpmKRKBJNnM
wNvKsYHZTv98cn76NSwuhFk0bfM1no0KrJOpHRu9EWu6e0wspMLg9wIhcPY7pwqP
4+ByvUt/HX1SZJkHEF3iy2T2JLO5JP0LD/49UFaxMIc4dxkKsB7yTBqFH2s6CX3i
5l0Gm8iYaNW5S6++/+RGHLHIhf6Rc8vCaVvXYxQzqbfjyZxXMRgBQ6LkY0rIE3RP
G/at6AW13oT9kMDxDmB2vNdmM+BIqNcEooeKTcvFyg7vgCQ4CBFi6IAhsoJXkIxc
Czr1wDF48+0htM3p8f4jA8Ig0xTSiKRnNI90hr4wLjWdvf3J95MJqJg7Da2U4/rg
fsaim1ii9yBCVNP+8aOsxLL8LDz2m3UaQg6cBjhdQQNv0jSQIBUns8mK7/IHREoP
0ZtkpOnQmxvcxHZHcuk+CUULaQTOtxvP+hUveukrE7VJ0FU5RZunnBaVnWQJ5Vbl
whirKfrTendGKaB4B8H2QiTzcVH6IzHAy6r/LoVE/0eDXHsFu9cnGlxoBrmPpOLO
6qBN0RZj5dPZwFngiom5Ag0Eac6VCgEQAMYQ3TkDhlZyKPWzBZFsRf4MePtjY/L8
zE47ZT8KruinXhLIvb2KjU/J6ddyYKkC5dmaFCPNTC7WODf4RW9ILG/HqEYfpBPw
wQLP8vSX11Rt9iE38yZwBkKElAXIqs2vOCbdIJQpiZSolIAUHC+etajeCbzvurk6
S3hvRBDDa/UnnWqIbmqSgzyl3sWOVyYON6Tuvslf+oxcBj71fnsuYkgSu+j5AVkM
dahKOCa3lDSxta7oSP4T+sEcWM/EIkyNt2nEVEvoyG32Y4FO+ainnBhrh4ev2gOu
/T7eWbFcty/Rgkt1Ux49NJZjDpnpU5r5biScfiuu58ix5bqcuOIQenvdql6OrpOi
JbclNMNOKt7p8woC2PU/CieSjNrUNpSck0/yhFV63FqUs4iKQrdyKHbigyyhL6+N
hmG6S4t4n9MOtv+rVmQEfcKXRdFz06vk627MrRAtTAEDy6Z40A9s/8j77X9x08Vi
sQbwggRsrCisVlQ65GT8ZtkjX/EYTnMAqXVpWyRwX3KycaGii9vSRt/c+50xBb1V
aCfRnSpgsytuatTMTtfk/3KdAaCBM+afjxPUYalxTUr0i82YMERVGJQCZdWhvReB
n+UZ0LjIX093O94ZeVdWoQD6VNqRkufDLNhUWRmPywm1rBjpiYxI0JQx4o8B0ub+
7G3Tkgqi+I1HABEBAAGJAjYEGAEKACAWIQRabBbSjruBsaX2t8o3+/d9LpiKIAUC
ac6VCgIbDAAKCRA3+/d9LpiKILVsD/9Ew9EdDah3D6UkFVCVPlmMHFwtrFEkO+sd
aN7ASQTiNtGbRLjwhRrUhvV6EiyV4MvSBMuOAhHVGQw3+wY396DNQwdkjv9umUEG
ABz5bi4BPBMYs4hU9yjrhGZmuOAUFzjbCmXOLP64LcBWFcKT7i6oi2rPophG3SW7
24VHCGgSjg/C5rlj9VCaJ2b41B6G2lgQiI58Onbg4Zv3wQS1lURgXA3pQb5me9OG
MNjIOTFdO/tZIorpBrT0BKzLwhPYikwUblxNTDPjqFYOHKEH+PSsmFFSgMZ5EN4X
BPQPjF/jJTe2f44WOemp5HQs2U/9J0zgsXMv0Ntp6Bpj1D3/1Y5MeIKeymkT6bDZ
ODjek1CS7b32Kc1tBEOrfEGzVPZkdBFomZvAwxiDMizBY8I5eE+4S4b9w7n6iW2/
v20ngYIZFTm9vVc/GxFLCc+Xx26C9nQeH7EGNasBQoKLlS9nvljGABdETgTtSjst
KYMLb1LLjpU2IdzLvqlgEgbsJ6A9dnZJMf4cr35DqaV6I6KGmqqhS7sQlRDXngfa
zLsNpC6OgEWoSyBf4OGBHcqa7Y4TA3I39ygzF27yGnkbEO0jendPg1g85XMXxfzj
ltGVzvz7aQd6R4/W/369eEMMMlmKNnEN0m2/5SgGaENibQNCMlGXWwQCbJKK62Zl
WnIqg8eIFg==
=mffO
-----END PGP PUBLIC KEY BLOCK-----
EOF

gpg --dearmor < /tmp/webitel-apt-repo.gpg.pub > /usr/share/keyrings/webitel-apt-repo.gpg

. /etc/os-release
echo "deb [signed-by=/usr/share/keyrings/webitel-apt-repo.gpg] s3://webitel-apt-repo ${VERSION_CODENAME} 26.4-releases" \
  > /etc/apt/sources.list.d/webitel.list

apt update

PostgreSQL 15 → 18

Performed only after migrating to Debian 13.

Prepare the variables and check the disk before installing/running the upgrade.

Bash

export OLD_PG_MAJOR=15
export NEW_PG_MAJOR=18

export OLD_BIN_DIR="/usr/lib/postgresql/${OLD_PG_MAJOR}/bin"
export NEW_BIN_DIR="/usr/lib/postgresql/${NEW_PG_MAJOR}/bin"

export OLD_DATA_DIR="/var/lib/postgresql/${OLD_PG_MAJOR}/main"
export NEW_DATA_DIR="/var/lib/postgresql/${NEW_PG_MAJOR}/main"

du -sh "${OLD_DATA_DIR}"
df -h "${OLD_DATA_DIR}"

By default, to reduce risks, the upgrade requires free disk space approximately equal to the current PostgreSQL data directory + tablespaces, with additional headroom for WAL, auxiliary files, and temporary operations (typically x2 the size of OLD_DATA_DIR).

However, for large databases (or when disk space is insufficient), if the risks and recovery — either from a backup or via a replica — are acceptable, an upgrade mode using hard links is available. This mode does not copy files and therefore requires no additional free disk space. After a successful hard links upgrade, the old PostgreSQL 15 cluster becomes unusable.

See the following steps for details.

Installing packages

The timescaledb version in the new cluster must match the version in the old cluster at the time of the upgrade:

Bash

apt update
apt install postgresql-18 postgresql-client-18

TIMESCALEDB_VERSION=$(dpkg-query -W -f='${Version}' timescaledb-2-postgresql-${OLD_PG_MAJOR})
apt install "timescaledb-2-postgresql-${NEW_PG_MAJOR}=${TIMESCALEDB_VERSION}"

Preparing clusters

Check the existing DB clusters. If 18/main was not created automatically, create it manually.

Both clusters must be stopped before running pg_upgrade:

Bash

pg_lsclusters

OLD_LOCALE=$(su postgres -c "psql -tAc 'SHOW lc_collate' postgres" | tr -d ' ')
OLD_ENCODING=$(su postgres -c "psql -tAc 'SHOW server_encoding' postgres" | tr -d ' ')

if ! pg_lsclusters | awk '{print $1 "/" $2}' | grep -qx "${NEW_PG_MAJOR}/main"; then
  pg_createcluster "${NEW_PG_MAJOR}" main -- \
    --locale="${OLD_LOCALE}" \
    --encoding="${OLD_ENCODING}"
fi

systemctl stop postgresql

Checking `pg_upgrade`

To use hard links during the upgrade, add the --link flag.

Bash

su postgres -c "${NEW_BIN_DIR}/pg_upgrade \
  --old-bindir=${OLD_BIN_DIR} \
  --new-bindir=${NEW_BIN_DIR} \
  --old-datadir=${OLD_DATA_DIR} \
  --new-datadir=${NEW_DATA_DIR} \
  --check"

If --check completed with an error, do not run pg_upgrade until the cause has been resolved.

Running `pg_upgrade`

To use hard links during the upgrade, add the --link flag.

Bash

su postgres -c "${NEW_BIN_DIR}/pg_upgrade \
  --old-bindir=${OLD_BIN_DIR} \
  --new-bindir=${NEW_BIN_DIR} \
  --old-datadir=${OLD_DATA_DIR} \
  --new-datadir=${NEW_DATA_DIR}"

Updating PostgreSQL 18 configuration

pg_upgrade does not migrate configuration files automatically, so this must be done manually for:

/etc/postgresql/18/main/postgresql.conf
/etc/postgresql/18/main/pg_hba.conf

Bash

timescaledb-tune \
  --quiet --yes \
  --pg-version=${NEW_PG_MAJOR} \
  --conf-path=/etc/postgresql/${NEW_PG_MAJOR}/main/postgresql.conf

Starting PostgreSQL 18 and updating TimescaleDB

Bash

systemctl disable postgresql@${OLD_PG_MAJOR}-main
systemctl enable postgresql@${NEW_PG_MAJOR}-main
systemctl start postgresql@${NEW_PG_MAJOR}-main

pg_lsclusters

su postgres -c "psql webitel -c 'ALTER EXTENSION timescaledb UPDATE;'"

Updating PostgreSQL 18 statistics

Before starting the services:

Bash

su postgres -c "vacuumdb --all --analyze-in-stages --missing-stats-only --jobs=$(nproc)"

Updating Webitel packages

Bash

apt update
apt upgrade
apt install webitel-postgresql-18 webitel-postgresql-migrations-18 \
  webitel-supervisor-workspace-app webitel-web-widget-app \
  webitel-flow-diagram-app webitel-agent-workspace-app

apt reinstall webitel-admin-app webitel-audit-app webitel-auth-app \
  webitel-crm-app webitel-history-app webitel-meetings-app

Renaming systemd unit files and binary paths

The script saves the enabled state of the old unit files, creates a backup, renames the unit files, updates the ExecStart binary path, and adds an Alias with the old name:

Migrate systemd units script

Bash

touch /tmp/migrate-webitel-systemd-units.sh
chmod +x /tmp/migrate-webitel-systemd-units.sh
tee /tmp/migrate-webitel-systemd-units.sh <<'EOF'
#!/bin/bash
set -euo pipefail

# old_unit:new_unit[:old_binary:new_binary]
pairs=(
  "engine:webitel-engine"
  "call_center:webitel-call-center"
  "flow_manager:webitel-flow-manager"
  "messages-bot:webitel-messages-bot:messages:webitel-messages"
  "messages-srv:webitel-messages-srv:messages:webitel-messages"
  "storage:webitel-storage"
)

units_to_enable=""
backup_suffix=".pre-26.4.bak"

for pair in "${pairs[@]}"; do
  IFS=: read -r old new old_bin new_bin <<< "$pair"
  old_bin=${old_bin:-$old}
  new_bin=${new_bin:-$new}

  old_file="/etc/systemd/system/${old}.service"
  new_file="/etc/systemd/system/${new}.service"

  [ -f "$old_file" ] || continue

  cp -a "$old_file" "${old_file}${backup_suffix}"

  if systemctl is-enabled --quiet "${old}.service" 2>/dev/null; then
    units_to_enable+=" ${new}.service"
  fi

  systemctl disable "${old}.service" 2>/dev/null || true
  mv "$old_file" "$new_file"

  sed -i "s|/usr/local/bin/${old_bin}\b|/usr/local/bin/${new_bin}|g" "$new_file"

  if ! grep -q '^\[Install\]' "$new_file"; then
    printf '\n[Install]\nWantedBy=multi-user.target\n' >> "$new_file"
  fi

  if ! grep -qE "^Alias=${old}\.service$" "$new_file"; then
    sed -i "/^\[Install\]/a Alias=${old}.service" "$new_file"
  fi

  if [ ! -x "/usr/local/bin/${new_bin}" ]; then
    echo "WARNING: /usr/local/bin/${new_bin} not found or not executable"
  fi
done

systemctl daemon-reload
[ -n "$units_to_enable" ] && systemctl enable $units_to_enable
EOF

/tmp/migrate-webitel-systemd-units.sh

After verifying the services, the backup unit files can be deleted:

find /etc/systemd/system -type f -name '*.service.pre-26.4.bak' -delete

Migrating the DB schema

The migration is taken from webitel-postgresql-migrations. PostgreSQL major is determined automatically. timescaledb.restoring is reset via trap even if the migration fails.

Bash

touch /tmp/migrate-webitel-schema.sh
chmod +x /tmp/migrate-webitel-schema.sh
tee /tmp/migrate-webitel-schema.sh <<'EOF'
#!/bin/bash
set -euo pipefail

DB="webitel"
PG_MAJOR=$(su postgres -c "psql -tAc 'SHOW server_version_num'" | awk '{print int($1/10000)}')
MIGRATION="/usr/share/postgresql/${PG_MAJOR}/webitel/migration/26.02-26.4.sql"

if [ ! -f "$MIGRATION" ]; then
  echo "Migration file not found: $MIGRATION" >&2
  exit 1
fi

cleanup() {
  su postgres -c "psql ${DB} -qxc \"ALTER DATABASE ${DB} RESET timescaledb.restoring;\"" || true
}

trap cleanup EXIT

su postgres -c "psql ${DB} -qxc \"ALTER DATABASE ${DB} SET timescaledb.restoring='on';\""
su postgres -c "psql -v ON_ERROR_STOP=1 -1 -f ${MIGRATION} ${DB}"
EOF

/tmp/migrate-webitel-schema.sh

Reconciling configuration

Review and update according to the current configuration files:

OpenSIPs
FreeSWITCH

Cleanup and restarting services

Restart services in groups in dependency order:

Bash

apt --purge autoremove

systemctl restart consul
systemctl restart postgresql
systemctl restart rabbitmq-server

systemctl restart freeswitch ngcp-rtpengine-daemon opensips

systemctl restart webitel-engine webitel-call-center webitel-flow-manager \
  webitel-storage webitel-messages-srv webitel-messages-bot webitel-api \
  webitel-uac webitel-app webitel-cases webitel-logger webitel-media-exporter \
  grafana-server nginx

Removing old PostgreSQL 15

Only after upgrading PostgreSQL 15 → 18 and verifying the stability of PostgreSQL 18 and Webitel.

Bash

pg_dropcluster --stop 15 main

apt --purge remove postgresql-15 postgresql-client-15 \
    timescaledb-2-postgresql-15 webitel-postgresql-15

apt --purge autoremove

Updating PostgreSQL 18 statistics

After updating the services (can be run in parallel):

Bash

su postgres -c "PGOPTIONS='-c vacuum_cost_delay=0' vacuumdb --all --analyze-only --jobs=$(nproc)"

Steps

Stopping Webitel services

Removing the outdated Webitel repository

Dist-upgrade Debian 12 → 13

Connecting the Webitel S3 APT repository

PostgreSQL 15 → 18

Installing packages

Preparing clusters

Checking pg_upgrade

Running pg_upgrade

Updating PostgreSQL 18 configuration

Starting PostgreSQL 18 and updating TimescaleDB

Updating PostgreSQL 18 statistics

Updating Webitel packages

Renaming systemd unit files and binary paths

Migrating the DB schema

Reconciling configuration

Cleanup and restarting services

Removing old PostgreSQL 15

Updating PostgreSQL 18 statistics

Checking `pg_upgrade`

Running `pg_upgrade`