Matt Borja

Shibboleth Upgrade Notes (3.3.2)

Upgrading Minor Versions

An example of upgrading a minor version in Shibboleth Identity Provider 3 involves following procedures for updating the software from versions 3.3.1 to 3.3.2.

  • Always review Upgrade Procedures
  • Always review the Release Notes for security advisories, Important Notes for Upgraders, and any breaking changes.
  • Be aware of platform-specific upgrade instructions. For example, the Windows installer can be used to upgrade installations to later versions of the IdP, but does not allow you to go backwards with respect to the IdP installation (for instance V3.1.0 to V3.0.1).

Be aware that rolling upgrades are generally guaranteed to work only for patch upgrades (changes in the final digit). Minor upgrades may sometimes include internal changes to storage formats or other implementation details that could prevent certain features from working interoperably between versions. It is best to either plan for a relatively fast rolling upgrade within a maintenance window, or plan for a short period of downtime.

"No step left behind"

For the highest quality production setup, always review security advisories and Important Notes for Upgraders to ensure no additional recommended steps are left out. For example, the October 4, 2017 IdP security advisory not only recommends updating to 3.3.2, but also copying the server's certificate to a file and referencing it with the trustFile attribute.

Once the upgrade has completed, ensure the service is running.

Upgrade Notes (3.3.1.1 to 3.3.2, Windows)

The installer prompted and selected by default to "Install Jetty" but also noted "Requires Java 1.8" in red color.

Fixing path to JVM (reverted)

After upgrading, the service failed to start with the following event log:

    The Shibboleth 3 IdP Daemon service terminated with the following service-specific error: 
    Incorrect function.

This can happen after an upgrade if the path to JVM is specified manually as it will be reverted to "Use default." See Java installed from tarball) for resolution.

Alternatively, you can import the following registry entries to reset your desired JavaHome and Jvm keys (assumes Java 1.8):

    Windows Registry Editor Version 5.00

    [HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2.0\shibd_idp\Parameters\Java]
    "JavaHome"="C:\jre1.8.0_144"
    "Jvm"="C:\jre1.8.0_144in\server\jvm.dll"

Issues with idp.session.StorageService using shibboleth.MemcachedStorageService

Possible root cause: Prior to upgrading, the host's memcached service was removed from the cluster to avoid corrupting cached documents due to possible storage format conflicts, etc.

After upgrading, internal errors using Memcached to store user sessions were encountered:

    ERROR [net.shibboleth.idp.session.impl.StorageBackedSessionManager:528] - Exception while storing new session for principal USERNAME
    java.io.IOException: Memcached operation error
            at org.opensaml.storage.impl.memcached.MemcachedStorageService.handleAsyncResult(MemcachedStorageService.java:665)
    Caused by: java.util.concurrent.ExecutionException: OperationException: SERVER: Internal error
            at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:174)
    Caused by: net.spy.memcached.ops.OperationException: Internal error
            at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:192)

A possible workaround involves reverting to shibboleth.ClientSessionStorageService supporting only a subset of use cases but also much simpler and more robust clustering.

This issue seemed to be resolved by adding fail over nodes to the memcached servers list in global.xml and restarting the service. Attempting to reproduce the original issue by reverting this fix seemed to fail. Reproducing the issue was successful specifying an invalid port indicating a possible underlying communication issue with a faulty memcached node. It may be possible to reproduce the original issue removing the memcached node from the cluster as per the identified Possible root cause.

As of 1/19/18, it is confirmed that the IdP software may be upgraded in-place without removing memcached nodes from the cluster as no changes in client library or storage format or encoding were apparent. The above issue may be avoided by simply skipping this step.