Skip to end of metadata
Go to start of metadata

The One Degree Imager (ODI), an optical imager on the WIYN 3.5m telescope at Kitt Peak, Arizona, is designed to deliver atmosphere-limited image quality (<0.4”) over a one degree field of view. The ODI Pipeline, Portal, and Archive (ODI-PPA) is a web science gateway that provides astronomers a single point of access to ODI data, and rich computational and visualization capabilities. Its goal is to support scientists in handling complex data sets, and to enhance the WIYN observatory's scientific productivity beyond data acquisition.

ODI-PPA lets astronomers do their science -- while the gateway seamlessly manages the data!

The ODI-PPA portal offers a modern web interface to search and download data (like traditional archives). It is designed to be a compute archive that has several built-in frameworks:

  1. Collections that allow an astronomer to create logical collations of data products intended for publication, further research, instructional purposes, or to execute data processing tasks 
  2. Image Explorer and Source Explorer, which together enable real-time interactive visual analysis of massive astronomical data products in a DS9-style interface within an HTML5 capable web browser, and the visualization of standard catalog and SExtractor-generated source markers and overlays 
  3. Workflow framework which makes it possible for the PPA administrators to rapidly integrate data processing pipelines on an associated compute cluster, and for users to request such pipelines to be executed on their data via custom user interfaces.
    1. Currently integrated pipelines/libraries include a python based QuickReduce pipeline, SExtractor, an IRAF based calibration pipeline, SWarp, and basic IRAF modules.
  4. Association information between data products to help astronomers understand where one or more data products came from – be they from the instrument directly or the output from a data reduction pipeline. These association information are displayed in Collections and while inspecting details about a particular image.

Brief Technical Description

The ODI-PPA service stack, deployed on Linux servers at Indiana University (IU), is made up of a set of light-weight services tied together by an Advanced Message Queuing Protocol (AMQP) message bus and the web portal. The web front-end is built using the Twitter/Bootstrap and jQuery Javascript libraries while the backend services are written in PHP (using the Zend framework) and Python. A data subsystem handles the archival and file-staging tasks while a core service called Spider is responsible for appropriate handling and redirection of various data products. Several other services invoke and handle registration of new data products, and user-requested pipeline executions.  A set of compute nodes – some dedicated and others acquired via request on the petaflop-scale Big Red II supercomputer – handle all data processing requests including operator or user requested pipeline executions, tile generation for use on Image Explorer and Source Explorer, and staging of data in response to download requests.

  

 

ODI-PPA leverages CyberInfrastructure Indiana University including the geographically replicated 42 PB Scholarly Data Archive (SDA) tape system for safe archival of data, the 1 Petaflop Cray Big Red II and Karst supercomputers, and the 5 PB Data Capacitor shared file system for temporary staging of data. A set of download servers running Apache HTTP handle all download requests.  A replicated MySQL cluster and a RabbitMQ event cluster handle all PPA database and message bus needs respectively.