Data Transfer between Discovery Services and Library Catalogs: A Case Study of EBSCO Discovery Service and ILS (SirsiDynix Symphony) at the University of the Fraser Valley Hongfei Li1, and Diane Cruickshank 2 1 University of the Fraser Valley, Abbotsford, Canada University of the Fraser Valley, Abbotsford, Canada 2 Abstract. This case study explores the challenges and successes that the University of the Fraser Valley had with Bibliographic/holding data transfer and other customizations between their ILS (SirsiDynix Symphony) and EBSCO Discovery Service. Two sets of data transfer will be considered: the identification and transfer of deleted ILS MARC records to the EDS, and the EDS inclusion of a dynamic location map, from UFV’s library catalogue, in the EDS layer. Technical details are included. Keyword: Web-Scale Discovery Interface, EBSCO Discovery Service, Integrated Library Systems (ILS), library catalog, MARC record, Bibliographic data, holding data. 1 Introduction The Web-Scale Discovery interface is becoming commonplace within academic libraries. Diverse library resources with various platforms, and the need to improve ease of use for library users, is leading this adoption. Previously resource management and discovery were in the realm of the Integrated Library systems (ILS) vendors. If data transfer problems arose, the ILS vendor could come up with a solution as the discovery tool was their own. However, there has been a technological direction change where resource management has been separated from resource discovery, and online catalog modules of Integrated Library systems (ILS) are no longer being updated. Instead, the focus is on developing independent discovery tools. [1, 2]. Currently, the three dominant discovery service vendors are Serials Solutions’ Summon, EBSCO Discovery Service (EDS), and OCLC WorldCat Discovery. With these Discovery Service providers, libraries are finding themselves once again struggling with issues of synchronizing their ILS holding data with the discovery service. Discovery services require the Bibliographic and holding data of each individual library to be loaded remotely onto the discovery service’s server. Furthermore, the library must update these records as often as changes are made in the local library systems. In August of 2012, the University of the Fraser Valley (UFV) implemented the Web-Scale discovery system with EBSCO’s EDS. The initial data loading of MARC records in the EDS went smoothly, but very soon the ILS data updates became an issue. It was difficult to keep the EDS records accurate as deleted and discarded items, continued to be found, by the students using the discovery layer. Grand Valley State University and Eastern Michigan University Library had similar issues with their discovery layer. [3, 4] EBSCO Customer Support advised us to do a refresh loading weekly. However, doing so took 4 hours manually, with a net result of the changes only being reflected in the EDS interface a week later. With many items (physical and e-books) being added and removed weekly, the discrepancy between real time searches of the catalogue and the EDS were significant. *This is a post refereed version of the article published as Li, H., & Cruickshank, D. (2015). Data Transfer between Discovery Services and Library Catalogs: A Case Study at the University of the Fraser Valley. 15th International Conference on Asian Digital Libraries, ICADL 2015, Seoul, Korea, December 9-12, 2015. Proceeding. Published in R.B. Allen et al. (Eds.): ICADL 2015, LNCS (Lecture Notes in Computer Science) 9469, pp. 153–157, 2015. DOI: 10.1007/978-3-319-27974-9_15 *This is a post refereed version of the article : DOI: 10.1007/978-3-319-27974-9_15 A solution was found when UFV library developed a method, which captured all changes in the library catalog by creating MARC records with all additions, and deletions in one single file. The data, using FTP, was sent to the EDS daily. To explore the other end of the data transfer, we also will cover how the EDS helped us to integrate one of the features we had in our library catalog, a dynamic location map, into the EDS search layer. 2 LITERATURE REVIEW There are not many case studies focused on data transfer between discovery services and library catalogs. A consortium of colleges in North Carolina (NC-PALS) writes about updating the records daily in Ex Libris Primo Discovery Service. They used three scripts, run once a day. The first daily script extracts added or updated authority records, the second extracts catalog keys for any items that have been deleted, and the final script extracts any added or modified MARC records [5]. However, the paper does not provide the technical details of how the discovery service knew which records should be deleted or changed. No detailed case studies were found specifically for libraries using SirsiDynix ILS and the EBSCO Discovery Service. In the book by Popp and Dallis, Indiana University reports that their data was sent to the EDS and updated weekly [6], the same situation that UFV found itself in, initially. 3 METHODS AND PROCESS Daily Holdings updates: Shortly after we loaded our catalog data with the EBSCO Discovery Service, we decided to give up the weekly complete loading update. It was too time consuming and was proving to be ineffective. Instead we focused on resolving the asynchronous Bibliographic data match between library catalog and EDS. This could be done with a daily update. SirsiDynix ILS contains a delivered API script, which can extract all records added and changed daily. For newly added book titles or copies, this script can successfully catch and create the MARC records. Difficulties arise however, because the API script cannot identify the deleted or shadowed records. When the EDS receives the data file, these records are not highlighted for removal; the result, students are finding items in the EDS that no longer existed. Manually editing our MARC records with a “delete” was, again, too time consuming. Instead UFV needed to find a way to identify the record to be deleted, and mark it automatically with a “d.” We created a data flow task of four separate parts: extracting the deletion records; changing the character in Leader 05 to “d” (to indicate a deleted file); merging addition/deletion changes all into one file, and; using a cron job to FTP the data to the EDS. All the items, that needed to be deleted in the EDS, were selected using a modified SirsiDynix API script. The original API script needed MARC records. Unfortunately, some of the items we needed to be deleted no longer had MARC records available in the ILS. It was therefore not possible to create a proper record for the EDS to read. UFV found, though, that if we searched the ILS transaction logs and captured the catalog keys for deleted records, we could then create “fake” MARC records which the EDS processing program would cross-check and delete. *This is a post refereed version of the article : DOI: 10.1007/978-3-319-27974-9_15 The deletion files were extracted from the ILS transaction logs, and converted into a text based file. The text file was read, using a PERL script, to identify the start of each record. Then, the position of the leader 05 was located, and replaced with a ‘d’. Fortunately, it didn’t matter whether the rest of the MARC fields were completed or not. The next step was to merge all updated records together into one file, and then FTP to the EDS. All processes were run automatically, daily, without using staff time. The strategy was implemented under Oracle Solaris 10 with SirsiDynix Symphony 3.3.1. We used the SirsiDynix API script to catch the records changed, and PERL script to find catalog keys for deleted records in transaction logs. We also used PERL script to locate and replace the leader 05 for all EDS deletion records. Korn Shell commands were used to merge the “fake” MARC records and other holdings data into one ready-to-send file. Figure 1 shows the data flow. Quarterly full MARC records ILS Yes Daily changes, deletions…… Deletions? No Discard, missing, lost, shadowed… Deleted ID from transaction log EDS 1. Create MARC records 2. Merge deletion files 3. Add a `d’ in leader 05 Create MARC Records Merge files into one, cron the job, ftp to EDS for processing Fig. 1. Data processing and transfer from UFV’s ILS to EBSCO’s Discovery service After introducing this method for EDS daily update, the mismatch between the library catalog and the EDS interface disappeared. We should mention that it is not truly synchronous as there is a one-day delay for the EDS processing program to update the changes in the EDS. Customizing the EDS layer with UFV’s dynamic map UFV has a dynamic map in the library catalog which shows users a book’s position on the shelf. It is a script based on item location and call number [7]. The dynamic map display has received positive feedback so we wanted to add this feature into both the results list and detail page in the EDS. EBSCO technical support created a hypertext link “Floor Map” for the searched item, which redirects the request to a layer program housed on a Linux server at UFV. This program is designed to receive the catalog key from EDS interface, query the ILS for the call number/location, and generate dynamic maps for the selected book title. A pop-up window opens, displaying the dynamic map. The program uses PHP 5.3.3 script and a pre-created base floor map database. Figure. 2 shows the data flow. *This is a post refereed version of the article : DOI: 10.1007/978-3-319-27974-9_15 EDS No Catalog Key E-Book? Yes ILS Base Maps Processing layer for querying data from ILS Process data and generate dynamic maps Display Dynamic Maps Display no map Fig. 2. Data transfer from EBSCO Discovery Service to ILS 4. Conclusion UFV will continue to work with EBSCO to integrate library catalog additions into the discovery layer, and we will continue to use our scripts to send Bibliographic and holding data updates. We use similar methods with our other discovery interfaces at UFV, such as OCLC WorldCat Discovery, and Outlook Online (Canada British Columbia Library Union Catalog). The methodology and scripts have been requested and used by many SirsiDynix libraries, including those in Asia. It can also be partially used for other ILSs and Discovery Services, such as III Millennium with Serials Solutions Summons. The next step is to work with EBSCO technical support to remove e-Books directly purchased from vendors. These eBook deletion records are directly from the book vendors, not from the ILS, therefore have no catalog keys on which the whole automation process is based. Possibly, a second record ID, such as OCLC number, could be used for identifying the existing records in EDS. With the emergence and popularity of Web-Scale discovery services in the library world, the issue of synchronicity between library catalogs and Discovery interfaces will continue to pose challenges. Fortunately, scripts and technical solutions are being developed and documented for future improvements. References 1. Breeding, M.: The State of the Art in Library Discovery 2010. Computers in Libraries 30(1), 31-34 (2010) 2. Breeding, M.: Looking Forward to the Next Generation of Discovery Services. Computers In Libraries 32(2), 28-31 (2012) 3. Daniels J, Roth P.: Incorporating Millennium Catalog Records into Serials Solutions' Summon. Technical Services Quarterly 29(3), 193-199 (2012) 4. Wrosch, J., Rogers-Collins, K., Barnes, M., Marino, W. Search Me: Eastern Michigan University's Journey through the Highs and Lows of Implementing the Summon Discovery Tool. College & Undergraduate Libraries 19(2-4), 367-386 (2012) 5. Mortimore, J., Dunn, L., McNaughton, D., Novicki, E., Wade, E., Whittington, C.: Smoke Gets in Your Eyes: Trials and Triumphs of Implementing the Ex Libris Primo Discovery Service in a Small Regional Consortium. College & Undergraduate Libraries 19(2-4), 344-366 (2012) 6. Popp, M. P., Dallis, D.: Planning and Implementing Resource Discovery Tools in Academic Libraries. IGI Global, Hershey, PA (2012) 7. Li, H., Deng, S.: Linking Location and Shelf Mapping from OPAC Search Results: With Reference to Wichita State University. New Library World 109(3/4), 107 – 116 (2008)