Upload data

Contents:

Import Image data

This section shows how to import data either using the Java Desktop client or the Command Line Interface (CLI).

Contents:

Import data using the Desktop Client

Description:

In the first part, we first show how to import data by yourself and for yourself into OMERO using various import strategies. This will be mainly done using the OMERO.insight desktop client.

Second part of this import section will show how to import data for another user, using OMERO.insight. The user importing the data needs to have some admin or restricted-admin privileges. More information about restricted privileges can be found at https://docs.openmicroscopy.org/latest/omero/sysadmins/restricted-admins.html

The import for another user requires that the user doing the import has specific privileges. We will use the user importer1, this could be, for example, a facility manager.

We will show:

• How to install the OMERO.insight desktop client on Windows, Mac and Linux.
• How to import data for the user logged in using OMERO.insight.
• How to select a target Project and Dataset or create a new ones in OMERO for the imports.
• How to add Tags to imported images at the import stage, to facilitate the management of these images later in OMERO.server.
• How to import data for other users in OMERO.insight.

Setup:

OMERO.insight desktop client installation instructions:.

Download the OMERO.insight client corresponding to your operating system at: https://www.openmicroscopy.org/omero/downloads

Windows

• From version 5.5.0, OMERO.insight comes with two installers: .msi and .exe.

• Click onto the downloaded .exe or .msi file.

• This will run the installer. The .exe file installs by default in the userspace apps folder. This can be changed during the installation process.

  

• The .msi installer will deploy the application in the Program Files folder of Windows. OMERO.insight is then available to all the users of the target machine. OMERO.insight installed in the userspace is only available to the user who did the installation. A Desktop icon and a new OMERO.insight Start menu item will be created.

Mac

• From version 5.5.0, OMERO.insight can be installed using an Apple DMG (.dmg) file.

• Mac OS X: Click onto the downloaded .dmg installer to start the installation.

  

• This will mount the DMG it to your Mac. The DMG mounts in two places: on your Desktop and in the Finder sidebar under your hard drive.

  

• Clicking either one of these opens the DMG file. When you open a DMG file, you will usually see two things:

  • the app itself.
  • a link to your applications folder.

• Depending on your settings, the Applications folder icon might not appear. In such case, drop the app icon into the Applications folder.

  

Linux

• Unzip the downloaded .zip file.

  

• Click on the omero-insight file to start the application.

  

Step-by-step:

1. Open OMERO.insight and in the login dialog, click on the wrench icon

2. This will open a list of servers to which you can connect. By default, only “localhost” is listed. Click on the plus icon to add a new line to the list and type into the line the server address.

   

   Note

   If your server is running on a default port (4064), as is usually the case, then you can simply just type in the server name such as my.server.name. For servers running on a non-default port, e.g. 54064, type the address of the server as my.server.name:54064 into the above dialog. Alternatively, you can also type in the IP address of your server, or connect using websockets.

3. When done, click Apply.

4. Log in using the username and password provided.

5. In OMERO.insight, click on the Importer Icon in the toolbar.

6. Browse your local hierarchy in the left-hand pane of the importer, select single images or whole folders and add these to the Queue by clicking on the arrow icon.

7. In the Import Location window, select the target Project and Dataset (existing or create a new one) to import to.

8. Note: If no Dataset is selected or created, a new Dataset will be automatically created and named after the folder containing the images to be imported.

9. Optional: Go to the Options tab

   • Click on to bring the Tag selection dialog.
   • Select the tag(s) on the left-hand side or create a new one.
   • Click to move the tag(s) to the right-hand side.
   • Click Save.

10. Click on the Import button in the bottom-right corner of the Importer window. You should see two progress bars for every image imported, Upload and Processing.

11. Note: The import of the next image in the queue starts immediately after the Upload of the previous one is finished. The Processing phase of the import is done on the OMERO.server only, and can be finished while the next image(s) is/are being uploaded to the server.

12. Once imports are finished, go back to the OMERO.insight main window and click the Refresh button above the right-hand pane. This will display the imported images inside the Dataset and/or Project you specified previously in the Import Location window.

13. (demo only step) Now, the demonstrator will log out from OMERO.insight and log in again, this time as some other user and will show the import process again, this time importing a different set of images. After this import, the two sets of images (belonging to two different users) will be shown in the webclient.

Import for another user

In this example, we show how to import data for another user. A facility manager importer1 with restricted admin privileges imports the data for user-1. The facility manager has been given the ability to import for others.

The steps are the same as before for the normal import, but as importer1 has the permission to import for another user there are two drop down menus for selecting the user and group to import for:

• Select Group: ‘Lab1’
• Select User: ‘Francis Crick’
• Continue the import workflow as usual.

A restriction of OMERO.insight is that the user importer1 needs to be a member of the groups he wants to import for. This restriction does not hold when importing the the Command Line Interface (CLI) (link to CLI import g.doc).

Import data using the Command Line Interface (CLI)

Description:

This chapter will show how to import data for another user, using Command Line Interface (CLI).

The user importing the data needs to have some admin (or restricted-admin) privileges. More information about restricted privileges can be found at https://docs.openmicroscopy.org/latest/omero/sysadmins/restricted-admins.html

The import for another user will be done only as demo since the user is required to have specific privileges. We will use a user with login name importer1, who could be in real life e.g. a facility manager.

We will show:

• How to import data using the CLI for myself and for others
• How to import data using the CLI “in-place”, which means not copying the imported data into OMERO. Instead, OMERO will point to the original location of “in-place” imported files, thus preventing data duplication.
• How to deal with imports of large amounts of data in CLI, using the –bulk option and helper csv and yml files which define what is to be imported and how.

Resources:

• Documentation:
  • https://docs.openmicroscopy.org/latest/omero/users/cli/installation.html
  • https://docs.openmicroscopy.org/omero/latest/users/cli/index.html
  • https://docs.openmicroscopy.org/omero/latest/users/cli/import-target.html
  • https://docs.openmicroscopy.org/omero/latest/sysadmins/in-place-import.html
  • https://docs.openmicroscopy.org/omero/latest/users/cli/import-bulk.html
• Data: example images from
  • https://downloads.openmicroscopy.org/images/DV/will/FRAP/
• Bash script for performing in-place imports:
  • https://github.com/ome/training-scripts/blob/master/maintenance/scripts/in_place_import_as.sh
• Example files for bulk import
  • https://github.com/ome/training-scripts/tree/latest/practical/other

Setup:

CLI Importer installation

Client libraries from the OMERO.server have to be installed on the client to import images using CLI. The installation instructions can be found at https://docs.openmicroscopy.org/latest/omero/users/cli/installation.html.

Note: When importing for another user using the CLI, the importer1 does not have to be a member of the target group.

Step-by-step:

1. Open a terminal and connect to the server as importer1 using ssh.

2. The aim is to import an image from /OMERO/in-place-import/FRAP

   $ ls /OMERO/in-place-import/FRAP

3. Activate the virtual environment where omero-py is installed or add it to PATH e.g.

   $ export PATH=/opt/omero/server/venv3/bin:$PATH

4. Point OMERODIR to the location where the OMERO server is installed e.g.

   $ ls export OMERODIR=/opt/omero/server/OMERO.server

5. The importer1 user logs in as themselves

   $ omero -u importer1 login

6. Creates a Dataset import_for_myself

   $ DID=$(omero obj new Dataset name=import_for_myself)

7. Import the data in the newly created Dataset:

   $ omero import -d $DID /OMERO/in-place-import/FRAP/U20S-RCC1.10_R3D_FRAP.dv

8. The importer1 user logs in as user-1:

   $ omero --sudo importer1 -u user-1 login

9. Create a Dataset import_for_user_one as user-1:

   $ DID=$(omero obj new Dataset name=import_for_user_one)

10. Import the data in the newly created Dataset:

    $ omero import -d $DID /OMERO/in-place-import/FRAP/U20S-RCC1.10_R3D_FRAP.dv

11. Check that the images are successfully imported.

In-place Import CLI

Instead of being copied into OMERO’s managed repository, the image files stay at their original place and are just linked into the repository.

It is only available for the CLI importer, using the argument --transfer=ln_s.

Advantages:

• All in-place import scenarios provide non-copying benefit. Data that is too large toxist in multiple places, or which is accessed too frequently in its original form to be renamed,remains where it was originally acquired.

Limitations:

• Only available on the OMERO server system itself.
• Do not edit or move the files after an in-place import. OMERO may no longer be able to access them if you do.

Important:

Someone wanting to perform an in-place import MUST have:

• a regular OMERO account

• an OS-account with access to bin/omero

• read access to the location of the data

• write access to the ManagedRepository or one of its subdirectories. More information about the ManagedRepository can be found at https://docs.openmicroscopy.org/latest/omero/developers/Server/FS.html

  

Step-by-step:

• We are still in the omero server directory: /opt/omero/server/OMERO.server/bin
• Log in again: $ ./omero --sudo importer1 -u user-1 login

• ‘In place’ import a large SVS file into the same dataset as before:

  $ ./omero import -d $DID --transfer=ln_s /OMERO/in-place-import/svs/77917.svs

• Check that the image is successfully imported.
• Click on the paths icon to show the difference between the normal and in-place (ln_s) imported images. Validate that In-place import is indicated .
• Note: The script https://github.com/ome/training-scripts/blob/master/maintenance/scripts/in_place_import_as.sh shows how to perform the in-place import steps described above in one single command.

Bulk Import CLI

In this example, we show how to combine several import strategies using a configuration file. This is a strategy heavily used to import data to https://idr.openmicroscopy.org/.

We import two folders named siRNA-HeLa and condensation. For this training, the path to the OMERO.server is /opt/omero/server.

1. Open a terminal and connect to the server as importer1 over SSH.

2. Note: Connecting over SSH is necessary only if you intend to import in-place. If a classic import is being performed, you can connect to the server remotely using OMERO.cli and still use the bulk import as described below.

3. Description of the files used to set up the import, the files are in the directory /OMERO/in-place-import. See https://github.com/ome/training-scripts/tree/master/practical/other and https://docs.openmicroscopy.org/latest/omero/users/cli/import-bulk.html#bulk-importsfor further details.

   • import-paths.csv: (.csv, comma-separated values) this file has at least two columns. In this case the columns are separated by commas. The first column is the name of the target Dataset and the second one is the path to the folder to import. We will import two folders (the import-paths.csv has two rows).

     Example csv (note the comma between the “HeLa” and “/OMERO…”):

     *Dataset:name:Experiment1-HeLa,/OMERO/in-place-import/siRNAi-HeLa*

     *Dataset:name:Experiment2-condensation,/OMERO/in-place-import/condensation*

   • bulk.yml: this file defines the various import options: transfer option, checksum algorithm, format of the .csv file, etc. Note that setting the dry_run option to true allows to first run an import in dry_run mode and copy the output to an external file. This is useful when running an import in parallel.

     Example bulk.yml:

*continue: "true"*

*transfer: "ln_s"*

*# exclude: "clientpath"*

*checksum_algorithm: "File-Size-64"*

*logprefix: "logs"*

*output: "yaml"*

*path: "import-paths.csv"*

*columns:*

    -  *target*

    -  *path*

1. Go to /OMERO/in-place-import i.e. cd /OMERO/in-place-import

2. The importer1 (Facility Manager with ability to import for others) user logs in as user-1:

   $ bin/omero --sudo importer1 -u user-1 login

3. Import the data using the –bulk command:

   $ bin/omero import --bulk bulk.yml

4. Go to the webclient during the import process to show the newly created dataset. The new datasets in OMERO are named Experiment1-HeLa and Experiment2-condensation. This was specified in the first column of the import-paths.csv file.

5. Select an image.

6. In the right-hand panel, select the General tab to validate:

   • Click on to show the import details.
   • Validate that In-place import is indicated .

Advantages:

• Large amount of data imported using one import command.
• Reproducible import.

Limitations:

• Preparation of the .csv or .tsv file.

For more information about CLI import options, go to https://docs.openmicroscopy.org/latest/omero/users/cli/import.html.

Import data using OMERO.dropbox

Description:

OMERO.dropbox allows to import files into OMERO automatically, by means of offline import from a watched directory. Typically, each user has a folder into which they “drop” their images as these are being acquired. The folder can be directly on the machine where OMERO is installed, or, more commonly, OMERO can watch directories mounted on the machine where OMERO is installed.

Alternatively, the files to import can be copied into the folders watched by OMERO.dropbox by cron jobs https://en.wikipedia.org/wiki/Cron or systems developed by the community users such as https://github.com/imcf/auto-tx, which automatically harvest the files from acquisition computers and transfer them onto shared network drives.

We will show

• How to set up OMERO.dropbox on your OMERO.server.
• How to set up the directories/folders into which the users or the automatic systems described above will copy the files to import.
• How to import several files for two different users using OMERO.dropbox. The image files are manually copied into the prepared folders watched by OMERO.dropbox.

Resources:

• Documentation:
  • https://docs.openmicroscopy.org/latest/omero/sysadmins/dropbox.html
  • https://docs.openmicroscopy.org/omero/latest/users/cli/import-target.html
• Data: example images from https://downloads.openmicroscopy.org/images/DV/alexia/cajal-bodies/

Setup:

This example setup will help you to understand OMERO.dropbox functionality. Later, you can choose the setup you like for your OMERO.server studying detailed instructions at https://docs.openmicroscopy.org/latest/omero/sysadmins/dropbox.html#advanced-use

Below are the installation instructions.

• First connect to the OMERO.server over SSH using your administrator account.
• In the OMERO.server directory set the following lines to enable and configure Dropbox:
  • $ bin/omero config set omero.fs.watchDir "/home/DropBox"
  • $ bin/omero config set omero.fs.importArgs "-T \\"regex:^.*/(?<Container1>.*?)\""

Note: The last line makes sure the created Dataset into which the images will be imported will have the same name as the deepest folder in the DropBox folder for that particular user. Every image in any Experiment-1 directory will end up in the same Dataset for that user, however the superstructure of folders have been.

To set up different import options, go to https://docs.openmicroscopy.org/omero/latest/users/cli/import-target.html.

For example, it is possible to create a Dataset with a fixed name for every user or to create a Dataset whose name is picked from the first subfolder under images folder.

• On the OMERO.server machine, create a folder called DropBox under /home.

• $ cd /home

• $ mkdir DropBox

• Create under the /home/DropBox directory two subdirectories for two users e.g. user-1 and user-2. The names of these directories should match the login names of those users in OMERO.

  Note: the omero system user must be able to read from those directories. Also, the users or the system which will drop files into those directories must have write permissions there.

• $ cd /home/DropBox

• $ mkdir user-1

• $ mkdir user-2

Step-by-step:

1. Open a browser window.
2. Enter the URL provided.
3. Login as user-1 or any user with the right to see user-1’s data.
4. Leave the browser window open.
5. On your computer, open a new terminal window.
6. Connect over SSH to the machine where the OMERO.server is running.
7. Open the logfile DropBox.log under var/log/ e.g.
   • $ tail -f /path/to/OMERO.server/var/log/DropBox.log
8. Open a new terminal window.
9. Connect over SSH to the machine where the OMERO.server is running.
10. In any folder on that machine, create a directory Experiment-1 into which you copy an image. Drop the Experiment-1 directory into the user-1 folder you created during the setup above:
    • $ cd /path/to/other/dir
    • $ mkdir Experiment-1
    • $ scp 090829_5_HeLa_siCTL_coilin_ATUB01_05_R3D_D3D.dv ./Experiment-1
    • $ cd /home/DropBox
    • $ scp -r /path/to/other/dir/Experiment-1 ./user-1 #copy the whole directory “Experiment-1” into the directory watched by OMERO
11. The OMERO.dropbox will intentionally wait for 60 seconds between registration of the new drop into the folder and the actual import, in anticipation of further file drops of files into the DropBox folder.
12. After you have copied the directory into the user-1 folder, you should see in the DropBox.log lines like as follows.

2019-08-12 17:09:23,644 INFO [ fsclient.fsDropBoxMonitorClient]
(Thread-3 ) New entry
/home/DropBox/user-1/Experiment-1/090829_5_HeLa_siCTL_coilin_ATUB01_05_R3D_D3D.dv
contains 1 file(s). Files=1 Timers=1

2019-08-12 17:10:23,644 INFO [ fsclient.fsDropBoxMonitorClient]
(Thread-17 ) Removed key
/home/DropBox/user-1/Experiment-1/090829_5_HeLa_siCTL_coilin_ATUB01_05_R3D_D3D.dv

2019-08-12 17:10:23,666 INFO [ fsclient.fsDropBoxMonitorClient]
(Thread-17 ) Importing
/home/DropBox/user-1/Experiment-1/090829_5_HeLa_siCTL_coilin_ATUB01_05_R3D_D3D.dv
(session=2155e6d0-445a-496b-a6bc-8afeb93ac58d)

2019-08-12 17:10:23,955 INFO [ omero.util.Resources] (Thread-18 )
Starting

2019-08-12 17:10:23,961 WARNI [ stderr] (Thread-17 ) Joined session
for user-2@localhost:4064. Idle timeout: 10 min. Current group: Lab1

2019-08-12 17:10:31,989 INFO [ fsclient.fsDropBoxMonitorClient]
(Thread-17 ) Import of
/home/DropBox/user-1/Experiment-1/090829_5_HeLa_siCTL_coilin_ATUB01_05_R3D_D3D.dv
completed (session=2155e6d0-445a-496b-a6bc-8afeb93ac58d)

2019-08-12 17:10:32,001 INFO [ omero.util.Resources] (Thread-18 )
Halted

1. Go back to OMERO.web. Refresh the tree.
2. Observe that a new Dataset was created, with the name Experiment-1. The image is imported into that Dataset.
3. The image is always imported into the default group of the user.
4. Repeat the workflow for user-2. First, go to your browser, logout and login again as user-2.
5. Connect over SSH to the machine where the OMERO.server is running if required.
6. Create again a folder, Experiment-2.
7. Copy an image into it
8. Copy the whole Experiment-2 folder under /home/DropBox/user-2.
9. Go back to the browser, refresh and verify that you can see a Dataset Experiment-2 under user-2’s data with the image inside.
10. Note: Even if user-2’s folder in the previous workflow uses the same name for their dataset as user-1 (Experiment-1), the data of user-2 would not be imported into user-1’s Dataset. Instead, a new Dataset Experiment-1 would be created under user-2’s data, belonging to user-2, into which the image would be imported.