SVOCat Documentation

Version 0.5, June 2016, author: Carlos Rodrigo

HomeDownloadDocumentationExamplesCreditsHelp-Desk
1. Introduction
2. Download
    2.1. Extract
    2.2. Permissions
3. The files
4. Example
5. Configure
    5.1. First
    5.2. Project
    5.3. Mysql
    5.4. Web
    5.5. VO Curation
    5.6. ConeSearch
    5.7. Fields
    5.8. Photometry
    5.9. Search Opts.
    5.10. File Paths
    5.11. Scripts
6. Web Design
    6.1. style.css
    6.2. Colors
    6.3. header.php
    6.4. footer.php
7. Extra tips
    7.1. MOC files
    7.2. VO registry

Configuration

In order to configure the application with the properties of your catalogue please point your web navigator to:

http://www.mylab.org/catalogues/mycat/admin.php

(where, remember, http://www.mylab.org/catalogues/mycat/ should be the main URL that corresponds to your installation)

Before starting

It is now when we are starting to work with your catalogue.

In a first installation you will get all the configuration sections filled wih the properties of the example catalogue. You must change those properties to the ones corresponding to your catalogue.

You should also delete the file data-csv/cmc15n2-part1.csv and put there the CSV file with your catalogue data.

If you want to keep both, the example catalogue and your own catalogue, for being able to make comparations during installation, we recommend you to install SVOCAT twice. Once in "mycat" directory (where you will have the example catalogue) and once in "mycat2" directory, where you will make your own configuration with your own data. That means repeating the steps in the Download section but using a different directory.

The first time: admin password

The first time that enter the web admin page, you will be asked to choose an administrative password. Please, remember this password because you will need it in the future to log in into admin.php.

Once you give some password here you will be asked to log in using it

Write the same password that you wrote before and you can start configuring things.

If you ever forget your admin password, you can delete the file config/config_admuser.php. And the admin.php application will prompt you for a new password as explained above.

Project

In this section we ask you just for a label to identify the project. It must be a single word with no blanks or special symbols. It is not used in very obvious places, but we will need it to create files and things like that.


(You can click in this image to see a larger version. Actually, if some image is not very clear in this documentation, you will usually have the option to see a larger version clicking on it)

MySQL database configuration

This application uses a MySQL relational database to store the catalogue data.

We assume that you have MySQL installed in your server, you are allowed to create a database (or you can use an existing one) to store the data and you know how to do this. Otherwise, please contact your system administrator.

There are useful applications that help to work with MySQL databases, like phpMyAdmin. You can take a look at it to see if it makes your life easier. Here we will give examples assuming the simple command line interface coming with MySQL itself.

If you are used to work with MySQL most of the explanations that follow will probably be obvious. If you have never worked with it, they will probably be confusing and not enough and you will need help from somebody. We just try to give some tips in the hope that they are useful.

Just to give a naive description of concepts:

  • A database is mostly a collection of tables.
  • A table is a collection of data structured as rows and columns.
  • There are users (each one with its password) that are allowed to interact with the database. At least there will exist a user named "root" (not the same one as the linux root) but other ones can be created.
  • There are different levels of permissions specifying in detail what a user can do with a given database or with a particular table in a database or even a column of a table. Some users can be allowed to do whatever they want (like the "root" user) while some others can be restricted to do only some specific things.

For SVOCat we need:

  • A database already existing. SVOCat will not attempt to create it.
  • The name of a table to store the catalogue data.
    • SVOCat will not create the table. It will generate a small "MySQL" script that you can use to create the table accordingly with the information given by you in the configuration (Fields section).
    • SVOCat will not insert the data in the table. If you provide a CSV file with the data (and the structure in this CSV file corresponds with the configuration given by you in the Fields configuration section) SVOCat will generate a "MySQL" script that you can use to insert the data in the database.
    • You can use these scripts or you can create the database table by your own means and just provide here the name of the table.
    • For any of these missions you will need a mysql user who is allowed to create a database, to create a table and to insert data in the table. But SVOCat does not need any information about that user.
  • SVOCat only will use the database to make search (SELECT) queries. And we need that you provide, in the configuration, a mysql username and password who is allowed to do, at least, that. SVOCat will not create the user in the database, it must be an existing one and with the adequate permissions.

In our example:

  • We have a MySQL server running in "localhost" as hostname at the default port. This in fact is the usual situation. If your server has a special configuration (running in a remote machine, or in a different port) the system administrator will tell you what to write here.
  • We have created a database named "vocats". You can choose another name and create it or use a previously existing database. In any case, give here the name of an existing database.
    Just in case this is useful, a typical command to create a database is:

    > mysqladmin -u root -p create vocats

  • We have chosen to use a table named "mycat". A database can contain many different tables. We only need one to store the data from your catalogue. Choose some name for your table. There are many restrictions about table names in MyQSL. Choose a name with no blanks and no special characters.
  • We have created a user for our database named "catuser" with password "catp@ss" who is allowed to make SELECT queries in the "mycat" table (in practice, in all tables in the "vocats" database). You can (and probably should) use a different username/password in your final configuration.

Web options

WEB page texts

In order to generate the web page for your catalogue some information is needed about it.

In these images we show you where these texts will be used in the web page so that you have a better idea of what you want to put there.

  • 'Title' will be used in the top header of all pages, but also as a title in the homepage.
  • 'Subtitle' will be used in the top header of all pages, but also as part of the title in the homepage.
  • 'Email' must be a valid e-mail address and will be used as the Help-Desk email contact.
  • 'Description' will be used as the first part of the home page, describing the project.
  • 'Acknowledge' will be used as the last part of the home page, below the list of resources. It is typically used to include acknowledgements, and hence its name, but you can write there whatever you want.

You can include html in these fields.

WEB functionalities/menu

In fact, as you see in the image above, SVOCat will generate not only a search form so that people can see your data but a simple web portal to offer more information about it. That's why a menu is included and you can configure which items you want to show in the menu and, thus, which functionalities you want to offer.

The currently available options are:

  • 'Data Retrieval': it is the main functionality here. It creates a web form to search for data in the catalogue and displays the data corresponding to the search criteria allowing to visualize them, download them or send them to other VO applications. This is implemented in the search.php, that you should not edit.
  • 'News': this should be a page showing news that you want to communicate to your users. We haven't done anything smart here. It is just a HTML/PHP file (news.php) that you must edit to include the news.
  • 'Documentation': should be a web page showing the documentation about your catalogue. It is also a HTML/PHP file (documentation.php) that you must edit to include the documentation, as html, as links to pdf documents or in the way you wish.
  • 'Help-desk': this is a link to send an email to the email address that you have specified in the previous form.
  • The "Other (1)" option is a placeholder just in case you want to program some extra functionality yourself and add it easily to the menu. For instance, you could just set "Team pictures" as the Menu Title here and include the team pictures editing the other1.php file. If you want more extra functionalities read the Advanced section of this documentation.

What you must do here is:

  • You must check each item that you want to offer (on uncheck it if you don't want it). I would say that this application does not make sense if you don't ckeck the "Data Retrieval" functionality, but you can uncheck it also if you want.
  • In the 'Menu Title' field you can write an alternative text to name the link that will be offered in the menu. For instance, you could rename "Data Retrieval" as "Catalogue Search" and this will be the text shown in the menu. If you don't write an alternative text, the one shown in the "Functionality" column will be used for the menu. Be careful about changing this because long titles could be very ugly in the menu. And please, don't use html here or you could get terrible results.

Coverage Map

One of the available options above is including a "Coverage Map" page.

A coverage map gives a fast idea of what area of the sky is covered by your catalogue, both to VO applications and to final users (if you make a simple image displaying the coverage).

SVOCat does not help to generate the coverage map, you must do it by yourself. It only provides a simple way to display these files once you have generated them.

In order to do this you must first generate a MOC fits file. You must generate it by yourself (using, for instance, STILTS) and just write here where the location of your MOC file.

You can also generate some images displaying the coverage in a user friendly way so that we can show these images in the web page.

SVOCat expects that you provide:

  • 'MOC file': Relative path to the fits file containing the MOC generated by stilts. A typical value could be "coverage/moc.fits"
  • 'MOC resolution': A free text to display what resolution you have used to generate the MOC file. For instance, it could be something like "3.665° (max order=4)"
  • 'Image (AITOFF)': Relative path to the image with a AITOFF projection of your coverage map. A typical value could be "coverage/aitoff.png"
  • 'Image (xy)': Relative path to the image with a cartesian projection of your coverage map. A typical value could be "coverage/xy.png"

A SVOCat installation includes a folder named "coverage" as a covenient place to store these files. But you can decide to put them weherever you want.

The information provided in the configuration will be automatically used to generate the "Coverage" web page, as follows:

WEB logos

VO curation

Virtual Observatory services are usually services that respond to special http queries with a document in a particular XML format named VOTable.

In these VOTables it is important to include information about the origin of the data, authors, etc. In the configuration application a short explanation of the meaning of these fields has been given. For a better explanation you can take a look to the SSAP VO Standard (sections 4.2.5.5 and 4.2.5.6 mainly).

These fields will be used in the metadata header of all the VO query responses for this service. If, for instance, you take a look to:

http://www.mylab.org/catalogues/mycat/cs.php?format=metadata

(where, remember, http://www.mylab.org/catalogues/mycat/ should be the main URL that corresponds to your installation)

in the first lines you will see a series of PARAMS giving the information that we have writen above.

Given that these values must be included in XML responses like the ones above and that XML has some restrictions about it, please, try to use plain text here (no html tags, no & symbols...).

ConeSearch options

ConeSearch is a very simple protocol that is used in the Virtual Observatory for giving information about astronomical catalogues. You could want to take a look to the Simple Cone Search IVOA specification for more details (it is a very simple protocol, so it is a good point to start if this is the first time that you read a VO 'standard' document).

As a summary, the main idea of this protocol is that it allows to search for objects in a catalogue around a given point (specified by its RA and DEC coordinates in decimal degrees) and within a Search Radius (SR) also in decimal degrees.

Thus, a typical ConeSearch query would be something like:

http://.../cs.php?RA=1.01&DEC=10.04&SR=0.1

meaning that we want objects in the catalogue around RA=1.01, DEC=10.04 (degrees) and within a radius of 0.1 degrees.

The answer will be a VOTable (special VO document in XML format) containing the info in the catalogue for objects that match the conditions.

In the version of ConeSearch implemented by SVOCat we have included aditional options that allow to restrict the queries using other catalogue columns if you want to allow it. We will explain that later.

In this configuration section we only ask for very simple information:

  • 'MaxSR': the maximum Search Radius (in decimal degrees) that is allowed for queries. If your catalogue is small you don't need to specify this (leave it blank or write 180 here). But if your catalogue is big you will probably want to restrict queries to small radius so that queries are faster and the list of results is not too large.
  • 'Description': A simple description of the catalogue that will be included in all the VOTables generated by the service (See the example above). You can not use html here. Use plain text.

Catalogue fields

This one is probably the most important section of the configuration. And, given that you must write a lot of information, it can be the most confusing one. So we will try to explain the meaning of everything.

The main idea is that your catalogue can be seen as a series of rows (one for each observation) and columns (a column for each property of that observation).

We want that information stored in a database table with the same structure: rows and columns.

And you probably have the information in a CSV file with the information separated by commas (meaning different columns).

Here we ask you several things about the columns.

In order to explain this more clearly let's look at the example included with the distribution.

The structure of the data is:

and you have it as a CSV file like:

The idea is that there are 11 columns with different meanings. We want to know the meaning of each column (and some extra information).

IMPORTANT!. SVOCat needs that there exist two columns named RAdeg and DECdeg with the RA (Right Ascension) and DEC (Declination) in decimal degrees. In this example case they are the fourth and fifth columns and we just need to name them correctly in the configuration form.

Now let's go to the configuration form:

(Click in the image for a larger version. It will be opened in a different window)

Here you must fill as many rows as columns you have in your CSV file. That is, you have to give information about the 11 properties in your catalogue.

  • First, you must give a name to each property. This name will be the name of this property in the MySQL database when we create it (or it must be the name of the column in the database if you have already created it by yourself). It can only contain alphanumeric characters and the '_' symbol. And none of them can be a reserved word in MySQL.
  • Then you say the datatype of this field. It can be: string, double, integer or the special type 'img+coor' (this strange type will be explained later, ignore it by now). This will be used to create the table properly. String fields allow searches with wildcards characters whereas numerical fields allow searches by range (see the Search Options section below).
  • The 'Show' column says if you want to show this field in the search results or not deppending on the level of verbosity required. The ConeSearch protocol defines three different possible levels of verbosity: 1 for minimal verbosity (only show the most important properties), 2 for medium verbosity (this is the default) and 3 for maximum (usually: show all fields). You can specify 'never' if you don't want that a property is shown in any case. In our example catalogue we have RA and DEC expressed in hexadecimal format and we want to keep that information in our database but we don't want to use it (in fact, SVOCat will show RA and DEC in the results both in decimal and hexadecimal format so adding these fields would be unnecessary and even confusing). Then we mark these two fields as 'never'. We have marked all the other fields as 'VERB=1' so that all of them will be always shown in the search results.
  • 'CSV Order': here you must say in what position is this field in your CSV file. In our case, we have written the fields here in the same order than in the CSV file and the numbers run from 1 to 11.
  • 'Res. Order': here you must say how you want to order the fields in search results. For instance, in our example, the 'epoch' property is in the eight position in the CSV file (before J,H,Ks magnitudes) but we want to show it in the last position in the search results. And as we don't want to show ra_hms and dec_hms we don't give an order for them.
  • 'Title': this is the title that you want for this field when we show it it the search results table in the web page. For instance, we decide that the DECdeg column must be shown as 'DEC (ICRS)' although its name in the database will continue to be 'DECdeg'.

  • 'UCD': the UCD's, or Unified Content Descriptors, are a special vocabulary defined for the Virtual Observatory to give more information, in VO services, about the physical meaning of each field. The ConeSearch protocol requires:
    • UCD=POS_EQ_RA_MAIN for RAdeg.
    • UCD=POS_EQ_DEC_MAIN for DECdeg.
    • UCD=ID_MAIN for the main object identifier (it will probably there exist such a field in your catalogue, in our example, it is the objID field).
    For the other fields you can use the UCD builder at the CDS to find the adequate UCD for each case. It is not a big problem if you decide to leave this blank if you don't find a good UCD for a given catalogue property.
  • 'Unit': write a string for the unit corresponding to this field. Leave it empty if units don't apply.
  • 'Description': a text description of this property. It will be used both in the web page (a window is opened with the field description if the users moved the mouse over the "?" symbol) and the ConeSearch VO responses. Please,use plain text here.
  • 'Phot. Group': if some fields correspond to photometric data give a name here to label the group to which it belongs. Then you will have to describe the group properties in the 'Photom. Groups' configuration section. Take into account that several fields can belong to the same group (for instance, a given magnitude and its error). In our example we have five fields corresponding to photometric data: magr, sig_magr, magJ, magH and magKs. But two of them (magr and sig_mar) should be grouped together as they correspond to the same observation (magnitude and error). Then we write the same label (sdss_r) for these two fields and different labels for the other three 2mass magnitudes. More about this will be explained in the next section.

Photometry groups

Astronomical catalogues often contain information about photometric measures of the catalogued objects (usually as magnitudes). When this happens it is very useful to have as much information about this magnitudes as possible.

The IVOA Photometry Data Model specifies how to describe a photometric value (and/or the associated photometric filter) in the VO context and you could want to read it for further information.

In the configuration you must do two different things:

  • In the "Fields" tab (see figure above), group together those columns that correspond to the same "photometric filter" (for instance, a magnitude and its error will be in the same group). That means:
    • For each magnitude, give an arbitrary name to the "Phot.Group" column (usually, a different one for each magnitude).
    • It there are other columns that can be associated to a given magnitude, repeat the same "Phot.Group" name that you chose for the magnitude.
    • In our example catalogue we have four different magnitudes and we have chosen:
      • magr field ⇒ sdss_r group.
      • sig_magr field ⇒ sdss_r group (this corresponds to the same "filter" than magr so we asign the same group to it and, thus, they will share the same filter information.
      • magJ field ⇒ 2mass_j group.
      • magH field ⇒ 2mass_h group.
      • magKs field ⇒ 2mass_ks group.
  • In the "Photom. Groups" tab:
    • give the main characteristics of each photometric filter.

For each photometry group you just need to specify:

  • 'Phot.Group': the name for the photometry group.
  • 'Value type': if it is a magnitude or a flux (flux, flux density.. whatever).
  • 'Filter ID': The filter name in the SVO FPS.
  • 'Mag type': Is this a pogson magnitude (the usual case) or asinh o linear?
  • 'Zero Point': if the value is given as a magnitude and you do not find the right filter in the Filter Profile Service, you can give a zero point here (in erg/cm2/s/A, please).
  • 'ZP type': Is the zero point in Vega, AB or ST system?

We assume here that each photometric value can be described as corresponding to one of the photometry filters available in the SVO Filter Profile Service. Ideally, this means that the observation has been made using that filter, but it is enough if you think that the observing filter is close enough to one in the Filter Porfile Service.

Giving a Filter ID for the observation implies including a lot of information about the filter and, thus, about the observation (and how to understand a given magnitude/flux).

All this information about filters will be used in the web page so that when a user moves the mouse over the table header a small windows appears giving the description of each field.

And, what's more important, it will be used in the VO ConeSearch response to include metadata groups with the fields corresponding to the same filter and the appropiate links to the Filter Profile Service's to get extra information about the filter and the adequate calibration. This allows a VO application to automatically retrieve all the data that it needs about this photometric measure. It could be a zero point to transform it to flux, a wavelength to plot the value or even the actual filter transmission curve to compute synthetic photometry to compare with the observed one.

You don't need to find that information very intuitive. It's there more for being understood by applications than for human reading. Just as a short summary, in each GROUP there are several PARAM's giving information about the group (in this case, links to the Filter Profile Service filter and calibration) and one or several FIELDref's giving the fields that share those properties.

Search Options

By default the 'Data Retrieval' section of the web page will include a simple Cone Search form where users can search for catalogue entries around a given point in the sky (RA and DEC) and within a search radius (SR).

But you have the option to configure your service so that users can restrict searches using other catalogue fields.

For each field (except RA and DEC, that are always required) you can set the 'Search' column to "Yes" and it will be optionally allowed for searches.

If you mark a field as "Yes", you must also set the "Search type" and "Form Type" to specify how this field shoud be used.

  • 'Search Type' can have three different values:
    • 'Value': the user will be allowed to specify only a value for this field. In this case, only results matching the given value will be displayed. For numeric type fields this means that the value is exactly the same. For string fields, the search will be done in a SQL "...like 'value'..." way, what means that "%" can be used as wildcard.
    • 'Range': the user can specify a range of values. Queries will be restricted to results so that the field value is in the range. A open range (for instance, leaving the maximum blank) will be understood as no boundary.
    • 'List': the user can specify a list of desired values separated by commas.
  • 'Form type' can have two different values:
    • 'input': a open field will be shown to the user so that he/she can write a free value.
    • 'select': a select box will be shown to the user with all the available values in the database for this field.

You must set this configuration values wisely. For instance:

  • It does not make sense to set 'Search type'='range' for string fields.
  • It is usually better to set 'Search type'='range' for double fields (they often contain an unspecified number of decimals and it's difficult for the user to match an exact value).
  • It only makes sense to set 'Form type'='Select' if the list of different values for that field (in the whole catalogue) is short. For instance, if you have a catalogue column specifying the object type as "star, galaxy, probably star, probably galaxy, unknown" it is ok. But it does not make much sense offering a select box with hundreds or thousands of values in it.

In our case we have allowed to restrict searches using 'objID' and 'magJ' and we have allowed ranges for 'magJ'. Now the search form will show the corresponding entries for these two fields:

of course, if the user does not set any restriction on them, the search results won't be affected. In this case, there are 664 catalogue entries in a radius of one degree around RA=0, DEC=10.

But users can restrict the results to those with a 2MASS.J magnitude between 14 and 14.5 (just an example):

you can see that the list of results has changed. Now there are 78 objects in a radius of one degree around RA=0, DEC=10 and with magJ between 14 and 14.5.

But users can also restrict the query asking for results which objID contains the string '050':

and now only three catalogue entries match all the restrictions.

Just as an example, if we had set 'Form type'='select' for the magJ field the web form would have been different, showing a select box for minimum and maximum magJ.

where a user could have seen a list of all values for magJ present in the catalogue and choose among them (both for the minimum and maximum of the range). I don't think that this makes sense when the list of posible values is very long, but of course it's your decision.

Search options in ConeSearch

Although the VO ConeSearch protocol only specifies that RA/DEC/SR must be the search criteria, it does not forbid to restrict searches using other catalogue fields.

In our particular implementation of ConeSearch all the search restrictions that are allowed for the web page are also allowed for the ConeSearch queries.

In order to make these options public, we have implemented a 'format=metadata' option in the same way as it is specified in the SSA protocol:

http://www.mylab.org/catalogues/mycat/cs.php?format=metadata

(where, remember, http://www.mylab.org/catalogues/mycat/ should be the main URL that corresponds to your installation)

There, among other things, the list of parameters allowed for queries are specified in the usual fashion (and the available range or list of posible values when appropiate).

so that valid queries to the catalogue ConeSearch service could be, for instance:

  • http://www.mylab.org/catalogues/mycat/cs.php?RA=0&DEC=10&SR=1
  • http://www.mylab.org/catalogues/mycat/cs.php?RA=0&DEC=10&SR=1&magJ_MIN=14&magJ_MAX=14.5
  • http://www.mylab.org/catalogues/mycat/cs.php?RA=0&DEC=10&SR=1&magJ_MIN=14&magJ_MAX=14.5&objID=%050%

depending on how the user wants to restrict the query.

File Paths

In principle, this section is only important if you are going to populate the database starting with a CSV file and you want SVOCat to generate scripts to make it easier for you.

If this is the case you only have to write here the computer path to the folder containing the CSV file or files (if there are several .csv files in that folder, all of them will be considered).

(in this case, the value for the path is somewhere in my computer, you should change it to somewhere in YOUR computer).

Remember that this path (and the .csv files in the directory) must be readable by the Apache web server. We recommend you to put the csv files in the data-csv directory that comes with the SVOCat distribution.

If you have created the database by your own means, you can ignore this configuration section in most cases. That is, unless you have set in the 'Fields' section that one of your fields corresponds to a file type ('img+coor').

In the very special case that you have set, in the 'Fields' section, that one (or several) of your fields corresponds to a file type ('img+coor') you must say here where those files are located.

This will be explained better in the 'advanced' section. But, just for giving an example, if we had set the magH and magKs fields as 'img+coor' we would have to specify here where the corresponding files are located:

(of course, the mag fields do not correspond to any file and this particular example does not make much sense, but we show the image just as an example of how this configuration section can change depending on the fields types.

Scripts

In practice you don't need to configure anything here.

This configuration section is there so that you can click on the "Create scripts" button whenever you want and the scripts in the "work" folder will be generated again taking into account any changes that you have made in other configuration sections.

These scripts intend to be useful to help you to create and populate the database with your catalogue data starting with your csv file/s. If you want to create the database and populate the database table by your own, you can ignore this configuration section.

If you want to use the scripts you can do as follows:

  1. First, you need to create a table in the database with the columns that correspond to your catalogue. In the "work/create_table.my" file you have the commands in SQL to create the table (SQL is the language for interacting with relational databases).

    If you understand a little SQL you can read the file. If not, in summary, it deletes the table named "mycat" (or the name given by you in the "Mysql database" configuration section) if it already exists and then creates the table again using the information about the fields given by you in the "Catalogue fields" section.

    Typically you can use this file as follows:

    > mysql -u catuser -p vocats < work/create_table.my

    2.  

  2. The second step is reading the csv file/s and loading the actual data in the mysql table. For this, SVOCat generates a simple csh script named work/load_data2.csh. This script will read your csv file/s and generate files in SQL so that you can insert the data in the MySQL database easily.

    Typically you can use this file as follows:

    • First you move to the directory where you have installed your catalogue:

      > csh /var/www/catalogues/mycat

    • Then you execute the script:

      > csh work/load_data2.csh

      This will generate a XXX.csv.my file for each of your XXX.csv files.

    • Then you load each of those files in the database as:

      > mysql -u catuser -p vocats < XXX.csv.my