November 12, 2015
Systems Open Call
Systems Open Call
Update on Primo renorm and reindexing discussion with ExL, including Nimble (Al and Wade)
Review of Primo performance slides from Oct 23 site visit by ExLSent on behalf of the Discovery & Delivery Team...
Main discussion points:
1. Nimble was put into production Primo environment on November 7 / 8, 2015
2. Nimble in production environment has thus far had a positive impact on daily renorm / reindex jobs
3. Alliance desire is for full renorm / reindex to complete in under 48 hours
4. Ex Libris has stated that their near-term best result for a full renorm / reindex is around 70 hours
5. The Primo Premium Sandbox is currently undergoing a full renorm and reindexing.
6. As was brought up in the Oct 23 site visit meeting, the Alliance believes that a currently running full renorm / reindex job can have a negative impact on Primo results.
7. During the reindexing that of the Primo Premium Sandbox that will occur on November 13, the Discovery and Delivery team has asked institutions to test for such impacts and to open Sales Force cases to report such issues.
The Discovery and Delivery team communicated this request through the following email (dated Nov 10)
Tomorrow morning, a renorm/reindex process will be started on the Alliance”s Premium sandbox environment in order to implement Primo norm rules changes. The Discovery and Delivery Team would like your help in identifying any erratic behavior in the Primo front end during the reindexing part of this process.
Thursday (11/12) and Friday (11/13), throughout the reindexing/hotswapping process. Email will be sent to these lists indicating when the reindexing/hotswapping has started and again when it completes.
The Alliance Premium sandbox front end at http://alliance-primo-sb.hosted.exlibrisgroup.com/primo_library/libweb/action/search.do?vid=[INST], where the [INST] is replaced with your institution”s view ID.
What to look for:
- Prior problems noticed include, but are not limited to:
- Inability to add items to eShelf
- Error messages when emailing items from the eShelf
- Changes to material types in eShelf lists
- Records that are not discoverable during reindexing, but reappear once the indexing process is complete.
- Records that do not show appropriate request options during reindexing.
- Try to capture the problem ‘live’ using a video application like Camtasia or Jing.
- File a case with Ex Libris about the problem.
- Include any video or screen captures you took.
- Preface the case title with "Reindexing Impact on Primo:".
- Note in your case that a reindexing process was running at the time the issue was observed.
- Add Keith Folsom (email@example.com) as a contact to the case.
- Since we”re trying to catch and report problems that are occurring because of a background process, it”s important to have as many eyes on the front end during this time as possible. This will maximize our chances of finding any problems that may be caused by reindexing. Please share this request for testing with staff at your institution who are frequent users of the Primo user interface and are familiar with its functionality.
The Discovery & Delivery Team
A related issue was discussed, which had do with the e-shelf error message: "The selected record has been deleted from the database"Available updates on recent run of Primo outage events
In a Willamette University case, ExL has responded with the following: ‘I have looked into this issue and it appears to be related to Force Dedup being checked in the pipe in your environment. I am forwarding this case to our Tier 2 team for further analysis. You will continue to be updated via this case. If you have any additional questions or comments, feel free to post them to this case and they will be brought to our attention.’
So, there may be a pipe configuration setting that is affecting this e-shelf case.
Wade briefly summarized the discussion at the October 23 Alliance / Ex Libris meeting as it pertained to poor delivery performance of Root Cause Analyses (RCAs) following system down events. Following the October 23 meeting, ExL did deliver four RCAs related to recent Primo outages, as follows:
October 19 Primo RCA (approx 3 hours)excerpt: The ‘Front Ends’ had experienced a high memory use, larger then usually experienced. The exact reason for the specific large memory use was not identified yet.
Ex Libris is closely monitoring the memory use of the system to prevent situation of happening and identifying the cause of the high use.
October 23 Primo RCA (approx 1 hour)excerpt: The application server experienced occasional disconnection from the database that resulted in service disruption for the environment.
Exlibris R&D found the source of the problem in the code and have created a code correction which will be available in the November release.
October 30 Primo RCA (approx 30 minutes)excerpt: One of the ‘Front Ends’ had experienced an application issue resulting in stalled requests.
Immediate resolution was brought by restart of the problematic front end.
A code correction was added to the environment to allow better performance of the front ends.
November 2 Primo RCA (approx 30 minutes)excerpt: The host on which the database is located had crashed. The crash was identified to be caused by a defect on the host firmware.
Once identified, Ex Libris engineers had immediately worked with third party experts that had provided a work around for the identified problem. The work around was implemented and the system had become available again.
Ex Libris engineers are working with third party experts to test a correction that had been provided to the defect identified. One the fix will be thoroughly tested in the lab it will be implemented on the production environment.
Work procedures had been updated to be able to identify a failure caused by the specific defect in order to allow immediate workaround implementation if the situations is repeated.
Primo ‘no-PDS’ authentication testing / update
The SLA Working Group will continue to review these RCAs and push for faster, more accurate delivery of RCAs when necessary.
Wade briefly described the discussion from the October 23 meeting surrounding the new Primo authentication architecture available in the coming Primo November 2015 release.General check-in with membership about transition to V2 user schema
This new architecture will allow direct Primo authentication against SAML and LDAP, without the need for the PDS.
Following the installation of Primo November 2015 release in the Primo Premium Sandbox, the new authentication mechanism will tested by the University of Puget Sound and Portland State University, after which testing results and recommended next steps will be communicated to the systems community.
Wade provided information about new information offered by ExL through SF case 185208.Proposed working group. Alliance Sandbox Testing Working Group
Namely, the user V1 schema is considered deprecated, and will no longer be updated or the target of bug fixes. However, the user V1 schema will continue to be available for use for user imports into Alma.
So, there is no hard deadline for institutions to migrate to the new user V2 schema.
From the Ex Libris Case #185208
Hello Al:On a related note, in case 170445, Ex Libris has indicated that the issue of variable expiration times when using the user V2 schema ‘ has been given a tentative release for this quarter.’ Wade will seek additional info from ExL on the timing and implementation of this fix.
Version 1 of SIS is considered deprecated, meaning that no improvement or bug fixes will be applied to it.
However, institutions that already use version 1 (v.1) can continue. We do not plan drop it from Alma.
If you have any further questions, please let me know.
NA Alma Tier 2 Support
Wade introduced and Al discussed an idea for an Alliance Sandbox Testing Working Group. This group would provide more formalized management of the Alma and Primo sandbox environments than has previously been put into place.
Membership: Al as chair, will solicit volunteers for members (particularly from Systems representatives who haven't served on a team or group recently); as release testing groups are formed (for example, a group planned by the Technical Services Working Group), the Alliance Sandbox Working Group will work to establish a liaison relationship with these groups
Prior to putting such a working group in place, a proposal will need to be made to the Policy & Coordination Team for review at an upcoming P & C Team meeting.
High-level working group functions:
- Provide support for use the Alliance Alma and Primo sandbox environments
- Local documentation
- Support for an active Alma sandbox environment refresh cycle
- Support for Alma sandbox anonymization process
- Improve roles management in the environments
- Determine if additional sandbox resources are necessary to meet testing needs (and document any recommendations in writing for Systems Team review)
Ping Fu, of Central Washington University, as a member of the DC JWG (described below) provided an overview of the work of the working group.
Digital Content Joint Working Group (DC JWG) of the Content Creation & Dissemination (CCD) Team has put forth a proposed digital content metadata standards, available at https://www.orbiscascade.org/file_viewer.php?id=3171. These draft standards are open for review until close of business on Wednesday, November 24.
The DC JWG will offer two webinar opportunities for Alliance membership to offer comment and discussion on the proposal:
Friday, Nov 13, 10 AM PST/11 AM MST
Monday, Nov 16, 1 PM PST/2 PM MST
- You can also dial in using your phone.United States +1 (571) 317-3122
- Access Code: 626-485-845
- You can also dial in using your phone.United States +1 (408) 650-3123
- Access Code: 940-053-733