UFP-UV Upgrade Process

From UFP-UV Sakai Wiki

Jump to: navigation, search



The upgrade process consists on starting from a OOTB Sakai release, patching it to fit UFP-UV needs.
You should be aware of the current Development environment before continue reading.

On the text below - and as example - we will assume we are preparing a 2.7.x build to replace an existing 2.6.x build.

Prepare new UFP-UV build

Checkout Sakai source

Looking at /servicos/sakai-build there some src-* folders:

  • src (link to src-2.6.x)
  • alt-src (link to src-2.5.x)
  • src-2.6.x
  • src-2.5.x
  • (others)

We should remove the alt-src link, checkout the new root folder from the source from Sakai, import into local svn repository, and link alt-src to it. We will be ignoring the external definitions for now:

Prepare main pom.xml files

We should adapt the root pom.xml and .externals files based on the files from the previous build (src-2.6.x/). For now, every UFP or 3rd party module (not part of the core Sakai distribution) should be left commented out (disabled):

  • Edit src-2.7.x/pom.xml
  • Edit src-2.7.x/.externals
  • cp ../src-2.6.x/update-externals.sh .
  • ./update-externals.sh

For the 2.7.x UFP-UV build, here are the relevant already modified files:

Now we can do a regular svn checkout to get the core Sakai modules:

  • svn up

Initial build (Sakai OOTB only)

We should do an initial build so that modules get deployed to local Maven repository:

  • cd /servicos/sakai-build/alt-src
  • mvn install

Add custom tools

Let's get back to the editing of the pom.xml and .externals files, this time to uncomment the previous disabled modules:

  • Uncomment disabled modules from pom.xml
  • Revise URL and uncomment modules from .externals
  • Update the svn properties by executing: update-externals.sh
  • Checkout the UFP and 3rd Party modules by updating the working copy: svn up

Add overlay files

Custom UFP icons

  • cp /servicos/sakai-build/src-overlay/reference/library/src/webapp/icon/*.gif /servicos/sakai-build/src-2.7.x/reference/library/src/webapp/icon
  • cp /servicos/sakai-build/src-overlay/reference/library/src/webapp/icon/*.ico /servicos/sakai-build/src-2.7.x/reference/library/src/webapp/icon

Custom UFP-UV skin

Custom Webservices

  • cp /servicos/sakai-build/src-overlay/webservices/axis/src/webapp/UfpuvWs.jws /servicos/sakai-build/src-2.7.x/webservices/axis/src/webapp/

Mid build (Sakai + custom tools)

Before attempting another full build, we must carefully check that every module is compiling individually. For each custom module (UFP or 3rd Party), navigate into its source folder and attempt to compile it:

  • cd /servicos/sakai-build/src-2.7.x/ufpuv-sis
  • mvn install

These custom modules will probably fail on the first build, because it's common that changes get introduced between Sakai versions. You must check their pom.xml files and attempt to fix them before attempting another build. Usually, looking at some core modules (like content or gradebook) will help to figure out the changes that need to be carried on.
When all build issues gets resolved, a full build should be made again, this time clearing out previous compiled files:

  • cd /servicos/sakai-build/src-2.7.x
  • mvn clean install

Apply local changes

After having the whole building compiling successfully, we should move on and apply the custom UFP changes. The modification list is available for every build at this wiki, so similar pages should be created for this new build. As an example, look at the previous 2.6.x page.

The list of changes for the current release should be shortest than the previous one, as issues get fixed and merged into the Sakai source code. The previous list of changes is available here - a new similar page should be created for the new build, and the issues checked whether they are applied to the core source code or not.

The process for applying custom changes is relatively simple:

  • Checkout a clean, OOTB Sakai source into a separate folder:
  • For each change to be applied:
    • Navigate to the module: cd user
    • Apply the patch from the previous build (may require to manual apply a subset of changes that couldn't be patched): patch -p0 < /servicos/sakai-build/patches/2.6.x/User-SAK-12345.patch
    • Compile the patched module: mvn clean install
    • Create a new patch: svn diff . > /servicos/sakai-build/patches/2.7.x/User-SAK-12345.patch
    • Revert the changes on the separate folder: svn revert -R
    • Apply the new patch to the new UFP-UV build:
      • patch -p0 < /servicos/sakai-build/patches/2.7.x/User-SAK-12345.patch
      • mvn clean install
    • Commit the new patch to the repository: svn commit -m "Added new 2.7.x patch" servicos/sakai-build/patches
    • Update elearning.ufp.pt:/servicos/sakai-build/patches with a svn up

Final build (Sakai + custom tools + custom changes) & Testing

After applying all local changes, we should do a full compilation again, this time using the do-full-package.sh script (using our new alternate source folder (the new custom 2.7.x), and without copying the new build files to production):

  • cd /servicos/sakai-build
  • ./scripts/do-full-package.sh --full -alt-src -no-transfer

A new DB should be created, just for testing, without the need of DB conversion at this stage:

  • Edit the url@javax.sql.BaseDataSource property on /servicos/sakai-home/pre-production-a/sakai.properties to target the new DB: sakai_27x_test
  • mysql -u root -p
  • create database sakai_27x_test default character set utf8;
  • grant all on sakai_27x_test.* to sakaiuser@'localhost' identified by 'sakaipassword';
  • grant all on sakai_27x_test.* to sakaiuser@'' identified by 'sakaipassword';
  • \q

We should go into local QA now. The core Sakai have already went into a QA phase, but we need to make sure that the UFP tools, the 3rd Party tools and, most important of all, our custom changes, are not breaking anything. Some tips for testing:

  • Test each module/tool individually
  • Test each change individually
  • Test overall module/tool functionality to make sure nothing is broken
  • Think as an user, not as a developer

While we are in this process of testing and fixing, for quick compilations, we can build a single module:

  • cd /servicos/sakai-build/user
  • mvn -Pppa clean install sakai-build

This will clean, compile and deploy the code to our Pre-Production A tomcat instance.
The new Pre-Production A Tomcat instance can be started/stopped at

Database conversion

1. We have been testing our new UFP-UV build on an empty database. We should now prepare and adapt the database conversion scripts for migrating DB between 2.6.x and 2.7.x. First, we need to prepare a new folder for our db scripts:

  • cd /servicos/sakai-build/scripts/conversions
  • mkdir 2.6.x_to_2.7.x
  • cd 2.6.x_to_2.7.x

2. We should carefully analyze what scripts need to be run. These must be copied to our new 2.6.x_to_2.7.x folder. Typically, we should be interested in conversion not done between Sakai bugfixes (2.6.0 to 2.6.1), the minor upgrade (2.7.x), and other tools that we may be upgrading:

  • /servicos/sakai-build/src-2.7.x/reference/docs/conversion/sakai_2_6_0-2_6_1_mysql_conversion.sql
  • /servicos/sakai-build/src-2.7.x/reference/docs/conversion/sakai_2_6_2-2_6_3_mysql_conversion.sql
  • /servicos/sakai-build/src-2.7.x/reference/docs/conversion/sakai_2_7_0_mysql_conversion.sql
  • /servicos/sakai-build/src-2.7.x/melete/components/src/sql/mysql/meleteXXX_upgrade.sql
  • /servicos/sakai-build/src-2.7.x/jforum/jforum-tool/src/webapp/WEB-INF/config/database/conversion/jforum_2_X-2_Y


3. Now, we should analyze each sql script file, removing unnecessary code (for example, we don't need OSP stuff since we are not using it, and, we may have introduced major changes on previous releases that required db changes present on this script). We need a production DB dump, create a new db (sakai_pre_production) and load this dump on it. While doing and testing the scripts, we should:

  • execute the scripts one by one
  • execute each line of the script, one by one

After executing each script line individually, we should drop the DB, re-create it, load it with production dump and run the scripts all at once. If everything went OK, we are now ready for the next step!

4. To finish, we want our db conversions folder to be on subversion so we can checkout on our production servers:

  • cd /servicos/sakai-build/scripts/conversions
  • svn add 2.6.x_to_2.7.x
  • svn commit -m "Added 2.7.x db conversion scripts" .

Build & Test

When we have our new UFP-UV build ready for production, we should do our final build, copying it to our production servers:

  • cd /servicos/sakai-build
  • rm src
  • ln -s /servicos/sakai-build/src-2.7.x src
  • ./scripts/do-full-package.sh --full -st -y

We then test once more the production build candidate, before marking it as stable and schedule the production upgrade.

Preparing for the Upgrade

When our new UFP-UV build is ready for production upgrade, we should:

  • Add a new Administration Workspace > MOTD2 message on the UFP-UV gateway page, indicating that UFP-UV will be down for maintenance on a specific day, with an expected duration. This warning message must be added one week prior to the upgrade.
  • Add a new entry on Administration Workspace > Maintenance for warning online users immediately before the upgrade.
  • Add a new entry in /etc/crontab for scheduling the tomcats stop, for every production server. The command should be similar to:
32 05 11 09 * root /etc/init.d/tomcat-a stop


Upgrade is done on the main production server (elearning.ufp.pt):

  • Disable httpd -> tomcats connection:
    • vi /etc/httpd/conf/extra/httpd-modproxy.conf
    • Comment out the BalancerMember entries
    • service httpd restart
  • Convert DB:
    • cd /servicos/sakai-build/
    • svn up
    • cd conversions/2.6.x_to_2.7.x
    • Enter MySQL and run each script individually:
      • mysql -u root -p
      • use sakai
      • source DB_SCRIPT_FILE.SQL (repeat for each script)
  • Make any needed change in Sakai configuration:
    • vi /servicos/sakai/sakai.properties
  • Update & start production tomcats:
    • /servicos/sakai-build/admin-scripts/ufpuv-update update
  • Test that upgrade was successful:
  • If everything is OK, we can re-connect httpd with the tomcats:
    • vi /etc/httpd/conf/extra/httpd-modproxy.conf
    • Enable BalancerMember entries
    • service httpd restart
Personal tools