User Tools

Site Tools


start

Evergreen Developer Documentation Project

GOAL: To combine developer knowledge, old wiki pages, and old conference presentations into coherent documentation for Evergreen developers.

00. Getting Started

    • (text source also available in same directory)
  • Helpful explanation of “gather()” vs. “recv()”
    • dbwells: “I think there are common misconceptions about what 'gather' does, and part of that I think comes from the name. AFAIK, it doesn't actually do anything to bundle responses. It calls a wait stage, handles some error checking, and (if passed a true value) will handily declare a request finished. It is often used with 'atomic' requests (which does the bundling), but as long as you don't finish, you could in theory use gather to loop over the responses just like recv. (Though it makes less sense with the added wait stage, I think.)”

Overview

Dealing With Data

  • Cstore
    • MAGIC PROPERTIES: isnew, ischanged, isdeleted
    • MAGIC AUTOGENERATED FUNCTIONS, based on IDL
      • open-ils.cstore.direct.actor.workstation.create {}
      • open-ils.cstore.direct.actor.workstation.retrieve ID {}
      • open-ils.cstore.direct.actor.workstation.update {}
      • open-ils.cstore.direct.actor.workstation.delete ID
      • open-ils.cstore.direct.biblio.record_entry.search {}
      • open-ils.cstore.direct.biblio.record_entry.batch_retrieve {}
      • open-ils.cstore.direct.biblio.record_entry.retrieve_all {}
      • open-ils.cstore.direct.biblio.record_entry.id_list[.atomic]
        • same as .search, but returns a stream (or an array, for .atomic) of object pkey values
      • Show syntax from different contexts
        • Perl: my $vols = OpenILS::Utils::CStoreEditor->new->batch_retrieve_asset_call_number($vol_ids);
  • Calling an Opensrf/Open-ils Service
    • Testing logging in with SRFSH
      • srfsh# request open-ils.auth open-ils.auth.login {"identifier":"admin","password":"fakepassword", "org":4}
    • Shortcut for login with SRFSH:
      • srfsh# login USERNAME PASSWORD
  • Pcrud
    • MAGIC PROPERTIES: isnew, ischanged, isdeleted (NOTE: The original “Easing gently” article incorrectly says “isupdated” instead of “ischanged”)
    • MAGIC AUTOGENERATED FUNCTIONS, based on IDL
      • open-ils.pcrud.create.clm
      • open-ils.pcrud.retrieve.clm
      • open-ils.pcrud.update.clm
      • open-ils.pcrud.delete.clm
        • NOTE: In Javascript, “delete” is a reserved word, so this function was renamed “eliminate” in PermaCrud.js.
      • open-ils.pcrud.search.ahtc[.atomic]
      • open-ils.pcrud.id_list.sitem
      • Show syntax from different contexts
        • Perl
          • #taken from lib/OpenILS/WWW/EGCatLoader/Account.pm
            my $pcrud = OpenSRF::AppSession->create('open-ils.pcrud');
            $pcrud->connect();
            my $messages = $pcrud->request(
                'open-ils.pcrud.search.auml.atomic',
                $self->editor->authtoken, {
                    usr     => $self->editor->requestor->id,
                    deleted => 'f'
                },{
                    flesh => 1,
                    flesh_fields => { auml => ['sending_lib'] },
                    order_by => { auml => 'create_date DESC' },
                    %paging
                }
            )->gather(1);
            $pcrud->kill_me;
          • Transaction example:
            # BEGIN
            $pcrud->request('open-ils.pcrud.transaction.begin', $self->editor->authtoken)->gather(1);
            # RETRIEVE
            my $aum = $pcrud->request( 'open-ils.pcrud.retrieve.auml', $self->editor->authtoken, $id )->gather(1);
            # UPDATE
            $pcrud->request('open-ils.pcrud.update.auml', $self->editor->authtoken, $aum)->gather(1);
            # ROLLBACK
            $pcrud->request('open-ils.pcrud.transaction.rollback', $self->editor->authtoken)->gather(1);
            # OR COMMIT
            $pcrud->request('open-ils.pcrud.transaction.commit', $self->editor->authtoken)->gather(1);
        • Dojo
          • //taken from templates/conify/global/config/coded_value_map.tt2
            var pcrud = new openils.PermaCrud();
            pcrud.search('ccraed', {coded_value : oldObj.id}, {
                oncomplete : function(r) {}
            });
        • Web Client
    • “Unlike cstore, pcrud access has to be explicitly granted in the IDL” (Quoting Bill Erickson) FIXME (Does this simply mean that all (or nearly all) IDL entries have cstore listed OOTB?)
    • “I thought pcrud could do count(*) calls, but I'm not finding any evidence of that now.” (Quoting Bill Erickson)
  • PermaCRUD
    • “Effectively deprecated” (quoting Bill Erickson)
    • The name lives on in the IDL (<permacrud> permissions block for pcrud services) and PermaCrud.js (pcrud interface for Dojo JS)
  • Storage service
    • Still used as ultimate back-stop, “straight” SQL (Perl CDBI classes)
    • From Easing gently into OpenSRF: “open-ils.storage: A deprecated method of providing access to Evergreen fieldmapper objects. Implemented in Perl, this service has largely been replaced by the much faster C-based open-ils.cstore service.”
  • Qstore (revived in bug LP#1689608)
  • Fielder vs. Flat Fielder services
    • Long-lost doc: docs/TechRef/Flattener/design.adoc (2012?)
    • Read-only
    • open-ils.fielder.<class> sits atop cstore — public data only
    • Flat fielder uses pcrud — can access all pcrud-permissable data
    • Implemented in Perl
      • Calls traverse Perl to cstore/pcrud
  • Supercat / UnAPI
  • XML-RPC Gateway
    • Allows public HTTP access (POST-ing to /eg/xml-rpc using the standard XML-RPC protocol) to services listed in opensrf.xml under opensrf/default/xml-rpc
  • Adding a new cstore/pcrud API
    • For autogenerated functions: Edit the IDL
      • add: controller=“open-ils.cstore open-ils.pcrud”
      • If pcrud, add permissions
    • For custom functions: Register a perl function with an API name
      • The discussion on bug #1768022 explains some OpenSRF API options, such as max_bundle_count, max_bundle_size, and max_chunk_size.

IDL

  • Raw SQL in the IDL (look for <oils_persist:source_definition>)
    • Example: “metabib::field_entry” (mfe)
  • Permissions:
  • <links>
    • “has_many” means the foreign table has the ID from my table
      • “might_have” means the foreign table has the ID from my table, but with either zero or one objects (not multiple). Thus, it has the same linkage as “has_many” and the same usage as “has_a”.
    • “has_a” means my table has the ID of the foreign table
    • These have nothing to do with nullability or guarantee of data existing.
    • This is modeled after CDBI (see perl Class DBI for original intended meanings)
    • FIXME ? Evergreen currently appears to handle this incorrectly (actually relying on the “virtual”-ness of a field instead of the “has_a” vs. “might_have” distinction ?)

Database Development

Functions In the Database

Evergreen makes extensive use of PostgreSQL features to manage business logic within the database itself. This includes using custom functions or stored procedures (using pl/pgSQL and pl/perl), rules, table views, and database triggers. This may lead to confusion when troubleshooting or following the logic flow in the code, especially on a live installation, since the database functions do not reside with the installed code. Also, database triggers add an extra layer of unexpected behavior. For example, Perl code may be updating a database table, but it is not clear what other triggers may be attached to that table. Often the quickest way to answer such questions is to search the database schema files in the source code. All of Evergreen's custom database functions can be found in Open-ILS/src/sql/Pg/ (in various files).

Some examples of custom functions include:

  • Important search functions (e.g. search.query_parser_fts())
  • User management (purging, deleting, merging)
  • Authority linking
  • Vandelay ingest
  • Data normalization
  • Internationalization (i18n) for translatable strings residing within the database
  • MARC extracting/ingesting
  • XPATH-to-table functions
  • Password generation
  • Permissions checking
  • Calculating holds
  • Acquisitions fund management (propagation, rollover)

Config in the Database

  • config.internal_flag
  • config.global_flag (child table of internal_flag; has “server admin” UI in staff client)

Notes about Database Structure

  • Database Views
    • The data in a view only exists in other tables.
    • In some cases, Evergreen uses multiple layers of database views.
    • Examples:
      • metabib.full_rec is a view of metabib.real_full_rec
    • Naming conventions for views:
      • acq.SCHEMA_TABLE_lifecycle
      • auditor.SCHEMA_TABLE_lifecycle
      • acq.*_total
      • acq.*_balance
      • money.*_summary
      • money.*_view
  • Database Table Inheritance (child tables)
    • Child tables inherit and add to the structure of their parent table.
    • Some of the data visible in a child table only exists in the parent table. Child tables are like a live wrapper of the parent table with some added fields.
  • Materialized Views
    • These tables are derived from data in other tables and are maintained by triggers.
    • If you depend on data in a materialized table (e.g. for testing), you may need to perform the triggers yourself or maintain the test data manually.
    • Examples:
      • money.materialized_billable_xact_summary
      • reporter.hold_request_record

Database Table Inheritance (child tables)

  • acq.lineitem_attr_definition
    • acq.lineitem_marc_attr_definition
    • acq.lineitem_provider_attr_definition
    • acq.lineitem_generated_attr_definition
    • acq.lineitem_usr_attr_definition
    • acq.lineitem_local_attr_definition
  • action.transit_copy
    • action.hold_transit_copy
    • action.reservation_transit_copy
  • asset.copy
    • serial.unit
  • config.internal_flag
    • config.global_flag (has a “server admin” UI in staff client)
  • config.remote_account
    • acq.edi_account
  • money.billable_xact
    • action.circulation
    • money.grocery
    • booking.reservation
  • money.payment (rationale)
    • money.bnm_payment
      • money.forgive_payment
      • money.account_adjustment
      • money.work_payment
      • money.credit_payment
      • money.goods_payment
      • money.bnm_desk_payment
        • money.cash_payment
        • money.check_payment
        • money.credit_card_payment
  • vandelay.queue
    • vandelay.authority_queue
    • vandelay.bib_queue
  • vandelay.queued_record
    • vandelay.queued_authority_record
    • vandelay.queued_bib_record

Further Reading

Web Staff Client

The Web Staff Client introduces some new technologies into the Evergreen ecosystem. First and most prominent is AngularJS, which is used to construct many of the user interfaces, connect the display with the data and keep both updated, and communicate with our various OpenSRF services. In addition, the Web Client introduces Hatch, a browser plugin that enables unmediated printing, extended local storage of preferences, and the potential for an offline version of the client (not yet developed).

AngularJS basics

Quick overview: https://docs.angularjs.org/guide/concepts

config.tt2

  • EXPAND_WEB_IMPORTS = 1 #use source script/css files instead of compressed build files

Modules/Apps

  • Usually contained in an “dir/app.js” file, but not always (e.g. cat/item/missing_pieces.js)
  • created or extended via “myApp = angular.module('appName', [services], …)”
  • connects to template via <html ng-app='appName'>
  • In the Evergreen Web Client, each page is a separate app (“egItemStatus”, “egItemReplaceBarcode”, “egCatCopyBuckets”, “egCatRecordBuckets”)

Routes ($routeProvider)

  • setup using myApp.config()
    $routeProvider.when('/circ/checkin/checkin', {
        templateUrl: './circ/checkin/t_checkin',
        controller: 'CheckinCtrl',
        resolve : resolver
    });
  • “A resolve contains one or more promises that must resolve successfully before the route will change.” (odetocode.com)

Views

  • created in the template using angular “ng-view”: <div ng-view>
  • This element gets populated with the template file “templateUrl” specified in the app.js routes ($routeProvider…templateUrl inside the .config() section)

Services

  • created using myApp.factory('serviceName', …)
    • Services can also be created using “.service(…)” or “.provider(…)”, which have distinct uses. The Evergreen Web Staff Client mostly uses the “.factory(…)” approach.
  • injected into apps via myApp.controller('controllerName', [serviceName], …)

Controllers

  • created using myApp.controller('controllerName', …)
  • declares which services should be injected/available
  • connects to a template via <html ng-controller='controllerName'>

Directives

Filters

  • created using myApp.filter('filterName', …)
  • used in templates as {{value | filterName}}

Read the documentation in the source code, especially in web/js/ui/default/staff/services/

Show which files provide services, directives, filters, etc.

  • cd js/ui/default/staff/
  • grep -re "^\.\(directive\|filter\|factory\|config\|controller\)" *

Show which files create or add to angular modules

  • grep -re "^angular\.module" *

Find block comments near modules, services, etc.

  • grep --exclude-dir=build --line-number --recursive --regexp="^\( \*\|angular.module\|\.\(directive\|filter\|factory\|config\|controller\)\)"

For Further Reading

Authentication

  • Retrieving an authtoken resets the timeout (re: c-apps/oils_auth.c: oilsAuthRetrieve or oilsAuthResetTimeout)
    • e.g. open-ils.auth.session.retrieve
  • Testing logging in with SRFSH
    • srfsh# request open-ils.auth open-ils.auth.login {"identifier":"admin","password":"fakepassword", "org":4}
  • Shortcut for login with SRFSH:
    • srfsh# login USERNAME PASSWORD

Caching with Memcached

How to use OpenSRF::Utils::Cache

From dbs on IRC:

  • To remove the cached images right away, you can do something like
    • memcrm --servers=localhost ac.jacket.small.0471828726
  • To show you all of the keys for cached large jackets:
    • memcdump --servers=localhost | grep ac.jacket.large

Similar commands on EG magic spells page:

Short blog post:

Error Handling

OpenSRF

Other Material

Localization (L10N)

Development Tips

  • You need to restart an OpenSRF service before changes to it will be used. Example:
    • osrf_control --localhost --restart --service open-ils.acq
  • To check the state of all OpenSRF services:
    • osrf_control --localhost --diagnostic
  • Ideas for safely testing an Action Trigger (without sending a million emails)
    • Hard-code the “TO:” email header to your own email address in the action_trigger.event_definition
    • Create the event without running it:
      • action_trigger_runner.pl --process-hooks --hooks=hook1[,hook2,...] --verbose
    • Examine the event in the database:
      • SELECT * FROM action_trigger.event WHERE event_def=EVENT_DEF_ID
  • XUL Client startup flags:
    • -console: Opens a debug console
    • -purgecaches: Clears all caches before starting the client
    • Example: “C:\Program Files (x86)\Evergreen Staff Client 2.10\evergreen.exe” -console -purgecaches

Other Useful Repositories, Scripts, Etc.

Highly Technical Features

While not developer documentation per se, these features are both highly technical and allow for major functionality changes with zero to minimal programming.

Action Triggers

  • From official docs:
  • From IRC discussions:
  • Admin creates a passive A/T event definition
    • Click the name link
    • “Trigger Event Environment” (action_trigger.environment) objects connect to the IDL via “Field Path”
    • “Trigger Event Parameters” (action_trigger.event_params) are passed to the validator and reactor as the env->{params} object, and to the event definition template as the “params” object.
      • These params are expected to be Perl code (re: Bug 1745025)
    • “Event Definition Tests” is currently only setup for events with a hook core type of “circ”
      • conify/global/action_trigger/event_definition.js
      • OpenILS/Application/Circ.pm::test_batch_circ_events()
  • Admin creates A/T events (only necessary for passive hooks):
    • action_trigger_runner.pl --process-hooks --hooks=hook1[,hook2,...] --custom-filters=[defaults to action_trigger_filters.json]
      • Script loops through each event def using this hook:
        • looks up the core_type column (a Fieldmapper class) in action_trigger.hook table
        • fetches all core_type objects using options from --custom-filters file
        • creates events
          • sets “run_time” to now() + processing_delay
  • Admin runs A/T events:
    • action_trigger_runner.pl --run-pending --granularity=...
      • Script loops through each event with matching granularity and with now() > run_time (meaning, this needs to be run now):
        • sends to Validator, skips event if false
        • sends to Reactor
      • NOTE: To run events with no granularity, you must use --run-pending with no granularity. Thus, a blank granularity is treated as a separate granularity value, rather than being included every time events are run.

Search / Normalization / Indexing

Appendix

Useful Developer Oriented Wiki Pages

Useful Presentations

  • EG 2010 (Contents)
    • Bib Template (deprecated) (Dan Scott)
    • Lifecycle of an OpenSRF Request (Scott McKellar)
  • EG 2011 (Contents)
    • Xen virtualization and multi-brick environment experiences
  • EG 2014 (Contents)
    • Git Tutorial (Galen Charlton)
    • Exploring a Browser-Based Staff Client (Bill Erickson)
  • EG 2015 (Contents)
    • Oh CRUD, my API has a Flesh Wound (Bill Erickson)
    • Evergreen Search Tune-up: Tips and tricks to taking the default search indexes and making them more awesome
  • EG 2016 (Contents)
    • Not Your High School Geometry Class: How to Develop for the Browser Client with AngularJS (contains helpful diagram of “Browser Client Architecture”)
    • Metadata Abattoir: Prime Cuts of MARC
  • EG 2017 (Contents)
    • We Aim to Misbehave: Making the Catalogue a Progressive Web App (Dan Scott)
    • There’s Gold in Them Thar Nooks and Crannies: Unveiling Hidden Things in Evergreen (Galen Charlton)
    • Take Action! Or, Improving Your Notification Action Triggers (Terran McCanna, Chris Sharp)
start.txt · Last modified: 2018/08/08 16:23 by rjs7