Data submission - General Information

Login

In order to access the PI Area and the Upload Service you have to provide your username and password; an email containing the personal login details has been sent to the site PI and collaborators indicated by the PI.

If you don't have an account valid to submit data please ask the site PI to add you to the Collaborators list using the tools available in the PI Area. If you do not remember the username or password use the “Forgotten password?” tool in the login page, enter your email address and a message containing your login details will be sent to your email address.

Upload process

After login you will be asked to provide a number of information in addition to the file to upload. Use the “data info” field to send information about the additional variables you sent or comment about a new version of your data. You can send the data as excel file or ASCII file, see the Data format section for details.

IMPORTANT: Do not write non standard ASCII characters in either the file and its information (data info) posted in the page otherwise an error message will appear during the upload process.

Click on “Submit” button, a dialog box will display the upload progress. Please note that it is not possible to submit compressed files.

The received files are evaluated and if they are formally corrected, they are imported and processed.

New versions of the data

When a variable already present in the database is resubmitted, will be imported and overwritten in the system, creating a new version of the products. The new version will be described through the information reported by the PI in the “data info” field during the submission step ("Upload data" page).

General rules

The first two columns of the file must be always the timestamps. Timestamps must indicate the start and end of the averaging period and midnight must be "0:00” of the following day. The format must be the ISO standard YYYYMMDDHHMM. Timestamp variables must be named TIMESTAMP_START and TIMESTAMP_END.

Example:

TIMESTAMP_START, TIMESTAMP_END

200910082300,200910082330    averaging period: 8/10/2009 23:00 - 23:30

200910082330,200910090000    averaging period: 8/10/2009 23:30 - 24:00 (0:00)

200910090000,200910090030     averaging period: 9/10/2009 0:00 - 0:30

Time convention: report the local time without “Daylight Saving Time”.

Missing data must be indicated as -9999.

Text format

Text ASCII files must contain the variables codes in their first line. Variable names have to be chosen according with the list rules reported in the section below. First column two columns must report the timestamps according with the general instructions above. The comma separator must be used. No units can be reported, use the units defined for the variable.

Example:

TIMESTAMP_START,TIMESTAMP_END,CO2,H2O,ZL,FC

200210070530,200210070600,375.0026343,13.81902137,0.075133,2.225711711

200210070600,200210070630,375.6178651,13.81904135,0.067499,1.611090395

200210070630,200210070700,375.1484745,13.77998531,0.032477,1.11762877

200210070700,200210070730,374.0334503,13.73454349,0.024741,0.236125726

An example of the ASCII template is provided at the end of this page.

Excel format

Data must be placed in the first sheet of the file (you can check it on the bottom left corner of excel) and the first row must be filled with the variables codes according with the naming conventions explained below.

Be sure that only numeric data are in the cells (no characters, no DIV/0…), and fill the empty cells with -9999 because data series are read until the first empty cell in the date column is found.

An example of the Excel template is provided at the end of this page.

Rules for variables names

To submit the data it is fundamental to use the correct variables names and units as reported in the Variables codes page. However, in addition to the variable name it is requested to use a 3 figures code which identifies univocally the location within the site where measurements are taken. The importance of this code is related to the possibility to associate to each variable a set of metadata like the method or instrument used, the measurement depth/height, the last calibration of the sensor etc.

Generally, the name of the variable has to be indicated as:

VAR_x₁_x₂_x₃

VAR= variable name (refer to the list of input variables)

The first figure stands for the position of the sensors on a 2D space. For instance, if soil temperature is measured at 3 different locations, these can be numbered as 1, 2, 3.

 

TS_1_x_x		TS_2_x_x		TS_3_x_x
location 1		location 2		location 3

The second figure represents the vertical level (height above the ground/depth of the soil) of the sensors. Supposing to measure soil temperature at 3 different depths along a vertical profile at each location, then each variable can be identified as follows:

 

TS_1_1_x		TS_2_1_x		TS_3_1_x		depth 1
TS_1_2_x		TS_2_2_x		TS_3_2_x		depth 2
TS_1_3_x		TS_2_3_x		TS_3_3_x		depth 3
location 1		location 2		location 3

Numeration increases moving downwards along a vertical profile but same number at two different locations can indicate different height/depth, that is specified in the metadata. So for example the deeper soil temperature measurement point in two different profiles of 4 sensors each will be indicated as TS_1_4_1 and TS_2_4_1 even if the first is at 80cm and the second 120cm below ground.

The third figure identifies the presence of replicated measurements in the same location. For instance, given 2 replicates of soil temperature measurements at position 1 and depth 1 (i.e. for sensors comparison), these would be indicated as:

 

TS_1_1_1

TS_1_1_2

Another example where the third figure should be used is for measurements at tree level like sap flow sensors. In this case the first figure indicate the tree (again specified in the metadata), the second the measurement height if you have for example sap flow measurements on the same tree at different height and the third can be used to identify different sensors at the same height, commonly used in sap flow measurements.

Remember: All variables to be submitted must be characterized by the 3 figure code and in case of a variable being measured with a single sensor in a given point in space, the three figures of the code clearly equals 1.

New variables

We tried to consider all the common variables collected in the network also on the basis of what has been collected in previous projects. However new measurements are continuously added and it is possible that a variable to be submitted is not present in the “Variables codes”. In this case please contact the Databse Staff at the address database@unitus.it and we will think together to the best solution. In any case do not send variables with names not consistent with those listed because these would be rejected by the system.

IMPORTANT: Additional information regarding the variables (height, depth) must be included in the metadata and not in the data file neither in the variable name. All additional information included in the data file are lost during the upload process.

General information

The template to submit ancillary data is the BADM (Biological, Ancillary, Disturbances and Metadata template) available at the end of this page. The BADM is an international standard used also in Ameriflux and FLUXNET and the format is directly importable in the database structure

For this reason the ancillary variables and the metadata must be submitted using the BADM. There are variables that can be submitted using both the continuous data format or the BADM, the amount of data and the time resolution should be considered when the format is decided (e.g, automatic chambers can be submitted as continuous data but the variable code must be the same used in the BADM).

IMPORTANT: the final BADM versions are still under finalization and are released as separated templates as soon as finally agreed internationally. Currently the final BADM are available to send data about the general characteristics of the site and the metadata about the instruments and link between instruments and variables submitted. The final approved templates are available here:

1. BADM for the general site information and example

2. BADM for soil description

3. BADM for the instruments description and instructions

4. BADM for the instruments operation and variables mapping and instructions

5. BADM for general reporting of disturbances and management

6. BADM for vegetation cover characteristics (LAI, Species etc.)

Instructions for completing BADM templates

Use column D and higher in the first worksheet to enter your information. Only use the columns past D when you have more than one entry that you need to put in a row. If you have nothing to enter in a row, leave it blank.

Do not enter information in shaded columns (columns A, B and C, first raw).

Variables with units of LIST(xxx)—see second worksheet in each template—should use only the choices listed. For the Temporary Template the options are provided in the pick-up list

Enter all values in the units requested. For numeric entries, use only numbers (no letters or symbols).

Fill in the submitter’s name—whoever is filling in the template (likely you).

See Explanations, the third worksheet, for variable groups that must be reported together:

1. Column A lists the variable groups.

2. Column C has a “p” if a variable is the primary variable for a group of variables.

3. Column D lists an “r” if a variable is a required qualifier for a primary variable. For example, if H2O is entered for FLUX_MEASUREMENTS_VARIABLE on the Site-General-Info Template, the FLUX_MEASUREMENTS_DATE_START is required.

Don’t forget that…

1. All the files must be submitted using the data upload service web page exclusively: files sent as mail attachment will not be considered.

2. Do not write non standard ASCII characters in the file information field (data info) during the upload process.

3. When continuous data are submitted, the first two fields must always be the start and end of the averaging period (TIMESTAMP_START and TIMESTAMP_END); in the timestamp midnight must be reported as 0:00 of the following day.

4. Continuous data Excel format: data must be placed in the first sheet of the file, other sheets are not considered in the data import process.

5. Continuous data text format: text files must contain the variable codes in the first line.

6. Additional information regarding the variables (height, deepness, instruments,…) must be included in the BADM template and not in the data file neither in the variable name. All additional information included in the data file are lost during the automatic upload process.

7. If a variable needs to be cancelled because errors have been found, it is possible to send again only the variable corrected that will overwrite the one imported (attention: the -9999 value also will overwrite the old value).

8. Remember that for data processing, these variables are essential: FC, SC (only if measured with a profile) or CO2, SW_IN or PPFD, TA, TS, USTAR or TAU, RH. Very important are also H, LE, SWC and Precipitation.

9. Ancillary information, metadata and not continuous measurements must be submitted using the BADM template. Also maps and photos can be uploaded using the upload file tool in the PI Area