4 minutes

Technical choices for DMAOnline

Published Nov 13, 2015 in blog
Scroll

Technical choices for DMAOnline

Here at Lancaster we have 2 developers working on DMAOnline, Adrian Albin-Clark and John Krug. We work on DMAOnline part time. John is working on the back-end server infrastructure and Adrian on the front-end dashboard technology.

We decided on a clear separation of front and back-end technology and development to enable effective separate working in the limited time available. We wanted to use existing skills where possible, the phase 1 prototype was very time constrained and we wanted to reduce the learning of new technology to an acceptable level.

The clear separation between front and back-ends with all communication via API calls means it will be easy for a subscribing institution to integrate data from DMAOnline into an institutional dashboard or data warehouse.

John, working on the back-end server went with Linux for the operating system, Postgres for data storage, Openresty/Nginx for the RESTful API and dashboard application code server. Openresty/Nginx has the Lua scripting language integrated and this is used to develop the database integration and query construction code. Python code is being used for data ingest code, it has great XML processing libraries. This has proved to be suitable so far though a review will be held at the start of before phase 3 to ensure this remains the case.

Adrian started with Dashboard theming technology from Metronic with datatables.js for data display and d3.js for graphs. Using datatables.js worked fine for the phase 1 prototype but it became apparent that all the data to support the use cases we wanted in the data model would not be directly available from source data systems. For example, whether a Data Management Plan (DMP) has been reviewed or not needs to be directly recorded in DMAOnline, as it’s not available from a CRIS (Current Research Information System) such as Pure. Firstly, we gave consideration to developing an administrator’s interface to do these data updates, probably in Rails. However in order to keep integration between display and administration functionality we looked for enhancements to datatables.js or a replacement that would allow in-table editing.

Adrian found ui-grid.js, a component in AngularUI, the companion suite to Angular.js. Like datatables.js, all communication with the back-end including data updates is done using the standard DMAOnline API calls maintaining the de-coupling of front and back-ends.

Adrian developed an API service in JavaScript which abstracts all the necessary API calls into a single location for general use within the dashboard application.

Some useful features in ui-grid are: easy selection of columns to display, column sorting and searching, in place editing of columns, selectable rows to create sets and export of those sets in CSV or PDF for simple reporting and analysis elsewhere.

We also wanted an easy mechanism for filtering displayed results by faculty, department and date ranges. The filters use some features of Angular to provide a dynamic update of displayed data when a filter is selected.

Some interesting bits

Back-end

John has developed a mechanism in Lua for templating SQL queries used for query and update. A query is defined using a Lua data structure called a table and processed by Lua code which fills in parameters in the template and passes the resulting SQL query to Postgres, returning the results as JSON.

As a simple example, the following code fragment shows the template for the API call to retrieve an API key to be used in subsequent API requests.

o_get_api_key = {
    query = [[
        select
            api_key
        from
            institution
        where inst_id in
            (
                select
                    inst_id
                from
                    users
                where
                    inst_id = #inst_id#
                and
                    username = #username#
                and
                    passwd = #passwd#
            )
    ]]
},

This template is activated when an API call to o_get_api_key is made. A piece of Lua code iterates through the provided inst_id, username and passwd parameters substituting the #<parameter_name># template placeholders with the appropriate values. The result is a valid SQL query which is sent to the database.

Note the use of the institution table and inst_id identifier. DMAOnline has been designed from the start to be multi-institution capable. The aim is to develop a national shared service to support RDM efforts. It’s not just for use at Lancaster.

Front-end

The ui-grid component provides an interface and editor that is much like a spreadsheet and is easy to navigate with mouse and keyboard.

Some ui-grid features are best seen with screen images:

Row Selection

Row Selector

Editable columns

Editable Column

Note that the editable column heading is shown in a different colour and that when selected an editable cell has a brighter blue background with bold white text.

Data export and PDF print options

Export

Search

Cheers, John & Adrian

2 minutes

Developing and Integrating DMAOnline with Archivematica

As part of DMAOnline stakeholder meeting, we went and spoke to York about further developments of both DMAOnline and Archivematica, and how these products will support each other.

Published Oct 10, 2015 in blog