An example of upgrading a minor version in Shibboleth Identity Provider 3 involves following procedures for updating the software from versions
- Always review Upgrade Procedures
- Always review the Release Notes for security advisories, Important Notes for Upgraders, and any breaking changes.
- Be aware of platform-specific upgrade instructions. For example, the Windows installer can be used to upgrade installations to later versions of the IdP, but does not allow you to go backwards with respect to the IdP installation (for instance V3.1.0 to V3.0.1).
Be aware that rolling upgrades are generally guaranteed to work only for patch upgrades (changes in the final digit). Minor upgrades may sometimes include internal changes to storage formats or other implementation details that could prevent certain features from working interoperably between versions. It is best to either plan for a relatively fast rolling upgrade within a maintenance window, or plan for a short period of downtime.
For the highest quality production setup, always review security advisories and Important Notes for Upgraders to ensure no additional recommended steps are left out. For example, the October 4, 2017 IdP security advisory not only recommends updating to
3.3.2, but also copying the server's certificate to a file and referencing it with the
Once the upgrade has completed, ensure the service is running.
The installer prompted and selected by default to "Install Jetty" but also noted "Requires Java 1.8" in red color.
After upgrading, the service failed to start with the following event log:
The Shibboleth 3 IdP Daemon service terminated with the following service-specific error: Incorrect function.
This can happen after an upgrade if the path to JVM is specified manually as it will be reverted to "Use default." See Java installed from tarball) for resolution.
Alternatively, you can import the following registry entries to reset your desired
Jvm keys (assumes Java 1.8):
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2.0\shibd_idp\Parameters\Java] "JavaHome"="C:\jre1.8.0_144" "Jvm"="C:\jre1.8.0_144in\server\jvm.dll"
Possible root cause: Prior to upgrading, the host's memcached service was removed from the cluster to avoid corrupting cached documents due to possible storage format conflicts, etc.
After upgrading, internal errors using Memcached to store user sessions were encountered:
ERROR [net.shibboleth.idp.session.impl.StorageBackedSessionManager:528] - Exception while storing new session for principal USERNAME java.io.IOException: Memcached operation error at org.opensaml.storage.impl.memcached.MemcachedStorageService.handleAsyncResult(MemcachedStorageService.java:665) Caused by: java.util.concurrent.ExecutionException: OperationException: SERVER: Internal error at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:174) Caused by: net.spy.memcached.ops.OperationException: Internal error at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:192)
A possible workaround involves reverting to
shibboleth.ClientSessionStorageService supporting only a subset of use cases but also much simpler and more robust clustering.
This issue seemed to be resolved by adding fail over nodes to the memcached servers list in
global.xml and restarting the service. Attempting to reproduce the original issue by reverting this fix seemed to fail. Reproducing the issue was successful specifying an invalid port indicating a possible underlying communication issue with a faulty memcached node. It may be possible to reproduce the original issue removing the memcached node from the cluster as per the identified Possible root cause.
As of 1/19/18, it is confirmed that the IdP software may be upgraded in-place without removing memcached nodes from the cluster as no changes in client library or storage format or encoding were apparent. The above issue may be avoided by simply skipping this step.