GCMD/IDN Software Release Notes
- Removed stack trace messages for applications (DocBUILDER and Search 3).
- Security improvements for Search 2.
- Keywords updated with user requests in Keyword Management System (KMS).
- Security updates for Search 2.
- Keyword Management System migrated from war to standalone jar. Also optimized concept schemes, trimmed space around keywords, and added test for finding Concepts by Scheme.
- Security updates for Search 3.
- Removed daily check to CMR for Search 3 interface.
- Updated QA Viewer and Keyword Management Service (KMS) applications. Optimized concepts in scheme retrieving in KMS.
- Created new Search 3 portals for Southern Ocean Observing System (SOOS), Socioeconomic Data and Applications Center (SEDAC) and Physical Oceanography DAAC (PODAAC).
- Updated sitemap and portal listing pages.
- Optimized static export in the Keyword Management Service (KMS) application.
- Added log configuration for Jetty within Keyword Manager and DocBUILDER.
- Updated GCMD mission statement page.
- Updated Search 2 interface titles view and updated the URL redirects within Data Service records.
- Keyword viewer application updated to only display Keyword Management Service (KMS) keywords.
- Collaborations page updated to include NOAA OneStop and modified the ESIP description.
- IDN home page "Show all GCMD Keywords" and IDN sitemap updated to link to GCMD Keyword Access page.
- Added the recent staff presentations to the GCMD publications page.
- Removed deprecated data service portals from GCMD sitemap.
- Older web pages were removed from website and archived offline for reference.
- Updated how URLs are built in Keyword Management Service (KMS) application.
- Removed CMR Health check pop up window within Search 3 interface.
- Expired encrypted URLs removed from Metadata record display.
- All Data Service deprecated portals were redirected to the GCMD Data Service URL.
- Removed CEOS member page from International Directory Network (IDN) web site; users can view existing CEOS member data sets from the IDN home page.
- Updated privacy web page to reflect content within the NASA's privacy statement.
- Updated encoding and improved logging within the Change List tool application.
- Enhanced Keyword Management System(KMS) application by loading keywords and populating the record counts from CMR data within the keyword viewer.
- Added Google Analytics dimension tags to docBUILDER application.
- Struts updated to latest version.
- Released GCMD Keywords version 8.6.
- Change list tool was modified to enable update and delete of the XML element "DIF/Related_URL/URL_Content_Type" in DIF 9 and DIF 10.
- Updated Google Analytics dimension tags within Keyword Management Service.
- Added status notification feature to all public facing GCMD applications. The new feature will allow GCMD to post notifications about maintenance or alerts to users.
- Added Google Analytic tags and dimensions to Keyword Manager and Search 2 applications.
- Reduced the frequency CMR is checked within Search 3 application.
- Updated 'What's New' section on the GCMD and IDN home pages.
- Struts updated to latest version.
- Updated Services portal to include Google Analytics tags. Also added google analytic dimension tags to Search 3 and Search 2 records, and Keyword Management System (KMS).
- Mime type was added to DIF10 writer's guide.
- Staff updated within DocumentBuilder application for Antarctic Master Directory record submission notifications.
- Conference and presentation citations updated on the GCMD staff publication page.
- Removed four data service related customized portals from the web site.
- Removed reference to reformat record display to FGDC transformation option from the Search 2 interface.
- Updated the spacing between titles view layout within the International Directory Network Search 3 portal.
- Mimetype support in csv output was added in Keyword Management System (KMS) application.
- Three new Search 3 portals were released to public as beta view (OBIS, NSIDC, and LANCE).
- Updated Search 2 portals by adding goolge analytics tags.
- Staff names updated to reflect the new NASA responsible official on GCMD and IDN web site.
- Removed older content from web pages and the web site.
- Six new Search 3 portals were released to public as beta view (OpeNDAP. Sun Earth, GOSIC, CLIVAR, GLOBEC and Goddard DAAC). Updated the query for the Langley beta portal.
- Multiple data centers now appear properly in Search 3 record display.
- Added a docker build to support testing phase in Bamboo.
- Personnel contact interface updated in docBUILDER and KMS.
- Legacy Services now support SERF publishing onto internal server.
- Staff completed the curation of the NASA Non-EOSDIS SERFs.
- Staff migrated 339 JAXA records from DIF-9 to DIF-10.All records were ingested into the CMR.
- QAOps software released functionality that captures delta changes to all incoming metadata to CMR. Implemented as a plug-in to GCMD's SyncTool. A Graphical User Interface provides access to automatic report generation.
- Staff updated web site and all online applications to include the new ESDIS tophat menu bar.
- GCMD Staff participated in the ESIP Summer Meeting, July 24-28 in Bloomington, IN and conducted the "Strengthening Ties Between the GCMD Keywords and Their User Communities" workshop, presented on "Lowering Barriers to Metadata Curation", presented a poster on "Discover Keywords and Associated Data Through the GCMD Keyword Viewer", and attended various sessions on metadata quality, keywords/ontologies and data search interfaces.
- Staff coordinated and provided ingest support for the transfer of 203 DIF-9 records from NSIDC to USAP-DC ownership. This set of records, formerly under NSIDC jurisdiction, were removed from the NSIDCV0 data provider and re-ingested into the SCIOPS data provider.
- Staff successfully released the QA Operations Tool to production. This light-weight application provides the science team with a simple interface to easily insert, update, remove, query and retrieve metadata directly from the CMR. It provides the ability to both batch load, edit and validate metadata as well as translate records between metadata formats. The retrieval interface allows users to query for a specific collection record and formulate more advanced queries using CMR's JSON query language and download sets of records directly to the user's desktop.
- The Search3 caching subsystem was updated to better address CMR concerns with the "ALL" query being over called. This new system will not repeat any query within the same session, making all calls for a user more efficient.
- Keyword Manager includes support for keyword abbreviations. This added functionality allows the coordinators to author abbreviations and other alternate labels in KMS.
- Eight portals were converted to Search 3 (AMD, NCDC, NCAR, EOSDIS, JAXA, ESIP, Water and Earthquakes).
- All EOSDIS DAAC, ESDIS and SIPS data center keyword URLs were reviewed to determine if they needed to be upgraded from http to https. No changes were made to the Data Center short name or long name. The changes only apply to the Data Center Keywor URL, a total of 19 URLs were updated.
- QAOps alpha was released to UAT. This light-weight application provides the science team with a simple interface to easily insert, update, remove, query and retrieve metadata directly from the CMR. It provides the ability to both batch load and validate metadata as well as translate records between metadata formats. The retrieval interface allows users to query for a specific collection record and formulate more advanced queries using CMR's JSON query language and download sets of records directly to the user's desktop
- docBUILDER was set up to use maven instead of ant. This allows for easier management of third party software libraries.
- Search3 was migrated to the latest version of GWT and GXT. This was done to ensure that the search3 software takes advantage of the latest security releases of these packages and can support the largest set of browsers supported.
- Build-a-portal MVP is completed and available on production.
- MySQL software was migrated to current version and included the following upgrades: security patches, SSL support for secure communications (required for EBNet migration). Connection settings were monitored post-upgrade and optimizations were made to reduce connection timeouts and total connection counts.
- Implemented script to test the submission of records into CMR. This script runs once a day on all systems (SIT, UAT and PROD) and reports if and why a record failed to ingest. This CMR Ingest test will help identify and mitigate possible ingest issues in advance.
- GCMD legacy software repository was migrated from SVN into GIT for three reasons: (1) to consolidate software into one system (all CMR work has been done under GIT), (2) to support future collaborations and open-source initiatives, and (3) to simplify auto deployment of GCMD software.
- The in-house GCMD User Support Comment Form has been fully replaced with Kayako.
- Staff implemented new capability of the Change List Tool to allow bulk changes to Personnel and Organization sub-fields (name, address, email, etc.).
- All GCMD software configurations and static web pages were successfully transitioned to work with newly implemented HTTPS protocol.
- A Data Center column, for display and sorting, was added to Search3 results page.
- Significant work has been accomplished towards the implementation of a document neutral metadata presentation. The latest version of Search3 in development can now pull the UMM version of a metadata record rather than a specific format (DIF, ECHO, etc.) and augment the UMM with any critical missing fields. For example, both DIF Data Resolution and Multimedia Sample are not included in UMM, so if a DIF record includes these fields, they are attached to the augmented UMM and available to the presentation layer.
- An IDN demo site incorporating the new "Search For Data Sets by CEOS Member" feature was released for review by IDN stakeholders. This new refinement search allows users to search for collections by CEOS Member organizations (pull down list).
- MySQL configuration settings were updated and tested on DEV systems in preparation for an update to the production environment. Actions included: updating the MySQL configuration file, importing content, and siege testing. Deploying and testing on DEV systems first will help to decrease production down time as much as possible.
- Metadata JSP files were updated to call RedirectAction.do with flags in order to better track user website interactions.
- The in-house GCMD User Support Comment Form was fully replaced with Kayako form.
- Completed CentOS Operating System upgrades on GCMD and IDN systems in order to meet security directives by Mar 31, 2016. In total, 20 systems were upgraded from CentOS 5 to CentOS 6; nine systems were taken off-line for decommissioning; and two systems were scheduled for waiver due to incompatible hardware. This action 1) extends our Operating System Support by three years ensuring timely security updates, and 2) contributes to the down-sizing efforts for GCMD and IDN servers requested by ESDIS and GCDC. Also completed upgrade of security monitoring software from Modsec 2.5 to 2.7 and wrote new rules to support the new version
- Keyword facets now correspond with the number of keywords within that hierarchy, not the total number of records.
- Concepts in the keyword viewer now display the long name and not the short name acronym in Search 3 tree-view hierarchy.
- Search 3 load times were reduced from 5-10 seconds to 2 seconds. Additional work will be done to achieve sub-second load times.
- A prototype desktop client for Change List tool was developed as proof of concept for future development. The desktop client allows anyone to pull records from the CMR or their local file system and run change lists on those files without the need for server side access. No internet connection is needed if using a local store of metadata records.
- Keyword Viewer now supports .csv output. Included in .csv are: short name. long name, definition, uuid, keyword path, and related keywords.
- The backend code for Metadata Pretty View can now display fields from multiple metadata formats (UMM-JSON is converted to GCMD-UMM). This accomplishes three design requirements: 91) Future-proof the Keyword Viewer to use expected long term data source data, (2) Ensure no fields of interest are missing if they are not mapped by UMM, and (3) Prevent display layer from needing to know the metadata format.
- Japan Aerospace Exploration Agency (JAXA) requested, GCMD/IDN staff update 309 JAXA metadata records (DIF-9) with the new User Services email address. Using the Change List tool, a total of 335 fields were updated on the same day the request was received.
- Tyler Stevens attended the 2017 ESIP Winter Meeting and (1) Gave a presentation during the GeoSemantics Symposium on "GCMD Keywords: A Resource for Building an Earth Science Ontology", (2) Gave a presentation on "GCMD Keywords: A Community Reviewed Vocabulary", (3) Conducted a workshop on "Build a Portal: A Simple Solution to Support A Customized Self-Service Search Interface/Query", (4) Gave a poster presentation on "Community Involvement in Enhancing the GCMD Controlled Vocabularies", (5) Attended relevant sessions on Earth science informatics including data discovery, ontologies, search relevancy, and Earth science data.
- At the request of ESDIS ESO, staff re-assessed the impact of Science Keyword Release 8.5 on DAAC metadata. The assessment included information on the number of records affected by the proposed keyword changes, organized by data center. The revised report will be used by ESO to help determine which DAACs should participate in the official keyword review.
- Staff delivered a self-service UNIX shell script for obtaining broken link reports from the GCMD/IDN QATool API. Initially requested by the Goddard DAAC, it provides an opportunity for external partners to test the QATool and provide feedback.
- docBUILDER is now able to automatically switch document type (DIF-9, DIF-10, ECHO-10, SERF) based on a users selection via the web interface. Previously, users were required to edit the URL to switch document types. This improves the usability of the authoring tool by making it easier for users to author and edit multiple metadata document types.
- Created detailed metrics for the QA Triage Metrics tool, including: timestamp, concept_id, rev_id, qa_score and grade_result.These triage details should help us answer these questions: new records since last reprocessing, records and list of Concept Ids where QA score did not change, records and list of Concept Ids where score did change, records that do no change and now pass, and records that do not change and now fail.
- DocBUILDER users now see distinct alert messages for the tool (separate from the general sever announcements).
- Created the IDN Search 3 dynamic home (http://idn.ceos.org/search/Home.do?search=&subset=idn) layout to match the existing static http://idn.ceos.org/index.html page.
Software Release Notes for 2016
- Staff released GCMD web site. New site includes an updated home page. To see list of features view: https://gcmd.nasa.gov/learn/release_announcement_10.html
- GCMD Data set search fully integrated with NASA Common Metadata Repository (CMR).
- Provider option to submit metadata using the docBUILDER authoring tool in DIF-9 and DIF-10 format. Users can also import and export metadata to and from the CMR using the docBUILDER tool.
- Staff added a portal configuration to allow portals to define the portal specific rule file that is to be used when calling the QATool for inline validation and when launching the QAViewer.
- Staff generated a report showing how the database connections have decreased on the legacy database and gone up on the heritage database.
- Staff attended the WGISS-41 CEOS meeting.
- Staff wrote a sync tool plugin to publish a static html file for every new record that comes in the CMR using the search 3 Metadata.do interface. These records are included in the record directory.
- Staff implemented a user interface element which can assist users in confirming granule to collection linkages. Also implemented a design for showing granule counts in docBUILDER tool.
- Staff included a new announcement on the IDN site, to notify users that the IDN portal now uses the same content from the GCMD portal.
- Staff updated Metadata Web Services (MWS) tool to include variable level 2 and 3
- Staff incorporated improvements to the Search 3 titles view.
- Staff removed the struts dependency from search 3.
- Staff Released Keyword Version 8.2 in Keyword Management System (KMS).
- Staff modified and created script to add CWIC tags (CWIC Status, Native_ID, and Provider Name) to back populated tags into IDN Heritage Tagging Table. Created scripts that generated reports of CWIC tagged records in CMR tagging and IDN Heritage system.
- Staff analyzed CSW usage on a month by month basis, identifying the most prevalent user per month from June 2015 through June 2016. The number of hits per user were counted, and the cumulative number of hits was used for sorting.
- Staff added Twenty Science Keywords and associated definitions to Keyword Management System (KMS) for Keyword Version 8.4.1.
- Staff redesigned and published the International Directory Network site (http://idn.ceos.org). Staff incorporated request from NASA representative Andy Mitchell. The portal’s search interface was migrated to GCMD Search 3 functionality to achieve full integration with the CMR and GCMD Search options.
- Staff made two presentations at the Earth Science Information Partners (ESIP) Summer Meeting.
- Staff completed the tagging of all CEOS WGISS Integrated Catalog (CWIC) records (5,108) and all GEOSSDataCore compliant records (15,779) in the CMR.
- Developer staff optimized two metadata curation tools (QA Scoring Tool and DocBUILDER) making the automated validation of metadata 10 times faster.
- Staff attended the Fall 2016 CEOS-WGISS meeting and presented the GCMD/IDN accomplishments over the last year, including the redesigned IDN web site.
- GCMD staff increased the number of records curated from 1575 in September 2016 to 9902 in October 2016, the largest once month increase since April 2013.
- GCMD staff published the GCMD Keyword Landing page on Earthdata and redirected all associated keyword pages from the GCMD website to the new page (https://earthdata.nasa.gov/about/gcmd/global-change-master-directory-gcmd-keywords).
- GCMD's Keyword Community Forum had the most new and active topics since going public in August 2016. Six new topics were posted and 47 unique visitors visited the forum.
- GCMD applied a continuous integration software tool (Jenkins) to automate the management and deployment of development code.
- Staff attended and presented at the American Geophysical Union (AGU) Annual Meeting, December 12-16, 2016.
- Staff implemented the capability to track metadata changes in the CMR over time. A "snapshot" of triage data at a specific point in time is captured, which includes: # of records, # of records that have failed QA checks, if a record passed or failed during that time, etc. An automated report will be produced every Monday providing metrics of how things have changed from week to week.
Please contact firstname.lastname@example.org with questions or feedback.