Tag Archives: command line

Notes on processing large files with GDAL

Context

I have recently had a project where I was provided with a reasonable number of large aerial image files. These were 600-800MB each and there were 10-15 of them. I needed to cut out the area of interest (which covered some part of all of the images), mosaic the images into a single representation of the site and find a way to share the data with non-specialists.

The workflow in my head was Clip – Mosaic – Webmap

Things to note

The images were too slow to render for me to mess about trying to do this using a GUI system, and the GDAL functions in QGIS3 (for me at least) don’t seem to be up and running properly (milage may vary).

Certain things that I have found when doing this in GDAL are that it’s best to define a data type for the bands, and to set the data structure as cloud optimised. The easiest way to create cloud optimised geotiffs is to use the following:

gdal_translate in.tif out.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=DEFLATE

Initially I was going to use gdal_merge.py to mosaic the clipped images as that is what is used in QGIS. However, gdal_merge.py reads everything into memory and I quickly found out that 16GB of RAM wasn’t enough as my swap partition started to be used and the whole thing ground to a halt.

The trick is to use the CPU if you can. First make a virtual raster using gdalbuildvrt, and then use gdal_translate (which utilises the CPU) to change the vrt file to whatever format you want (e.g. cloud optimised geotiff). This was actually really fast, and used the structure found in the following commands:

gdalbuildvrt output.vrt /path/to/folder/of/*.tif
gdal_translate -of GTiff output.vrt mosaic.tif

This ends up with an image that will be relatively large (1.2 GB in my case) which is still tricky to share. But fear not! We can use gdal2tiles.py to create a whole load of tiles and automatically generate some leaflet code to allow you to see the data. Then it’s just a question of moving that folder onto a web server and sending your contacts the link.

A quick thing to note/remember is that the webmap zoom factors relate to the following:

  • 0 represents the whole world (1:500,000,000)
  • 19 is very close up (1:1000)

Workflow

The workflow that I used in the end was as follows:

First crop each aerial image to the site shapefile

maindir="/path/to/folder/of/incoming/aerial/files"
cd ${maindir}

for f in *.tif

do
gdalwarp -ot Byte -of GTiff -cutline siteBoundary.shp 
-crop_to_cutline -dstnodata 0.0 -overwrite ${f} 
outputfolder/${f}_cropped.tif-co TILED=YES -co COPY_SRC_OVERVIEWS=YES
 -co COMPRESS=DEFLATE
done

Then build the virtual image

gdalbuildvrt output.vrt /path/to/folder/of/*.tif

Output the virtual image to a cloud optimised geotiff

gdal_translate -ot Byte -of GTiff -co TILED=YES 
-co COPY_SRC_OVERVIEWS=YES -co COMPRESS=DEFLATE output.vrt mosaic.tif

Create the web tiles and map file using the following command

gdal2tiles.py -s EPSG:27700 -z 11-19 mosaic.tif

Sources

https://lostingeospace.blogspot.com/2011/04/rapid-mosaicking-with-gdal.html

http://www.cogeo.org/

Advertisements
Tagged , , ,

VirtualBox shared folders

This helped when i didn’t have access rights to the host folder (host OSX, guest Ubuntu Mate):

https://darrenma.wordpress.com/2012/07/18/you-do-not-have-the-permissions-necessary-to-view-the-contents-of-shared_folder/

Basically, type this in Linux (changing username to that of your user):

sudo usermod -a -G vboxsf username

Tagged ,

Convert 16bit to 8bit GeoTiff

Note to self: use the following command to scale 16 bit spatial raster data down to 8 bit.

gdal_translate -a_nodata 0 -of GTiff -ot Byte -scale 0 65535 0 255 input.tif 8bit_output.tif

Tagged , , , ,

RDP from Ubuntu into AWS Ubuntu

Simple way to get RDP into Ubuntu AWS box:

  • Create the instance in the EC2 Management console and enable incoming RDP in Security Groups
  • Create a new user with a password
  • Add the new user to the sudoers file and copy over the AWS server’s public key
  • Log in as the new user and install xrdp
  • Load Remmina on your local machine and open a new RDP connection

Notes

In EC2 Management Console, click on the instance you want to connect to and scroll the page to the right. There will be a URL link to the relevant Security Group configuration page. Click on the Inbound tab and then the Edit button. Add a new rule, look for RDP as the type and set anywhere as the source (or make it specific if you want).

Move to the Terminal on your local machine and login to your AWS server with a command similar to:

LocalMachine:$ ssh -i ~/AWS-keypair.pem ubuntu@ec2-some-id.amazonaws.com

Create new user e.g. george

AWSMachine:$ sudo adduser george

Follow the prompts to enter the password and other optional user information. Add the new user to the sudoers file using the visudo command:

AWSMachine:$ sudo visudo

Add the following line after the comment line, “User privilege specification” and save the file.

george   ALL=(ALL:ALL) ALL

Ensure that the user george is in the sudo groupusing:

AWSMachine:$ sudo adduser george sudo

The new user account cannot be accessed (e.g. via SSH) until the public key from the AWS instance’s key pair (the .pem file) is installed for the new user. To do this, copy the public key installed for the ubuntu user and paste it into the correct file in the new user account. The public key can be found in the file ~/.ssh/authorized_keys under the ubuntu account.

AWSMachine:$ cat ~/.ssh/authorized_keys

SThe command above will print the public key. Select everything apart from the name of the key pair at the end of the key and copy it.

Next, switch to the new user account, george, and make sure you are in the home directory.

AWSMachine:$ sudo su george

Create the SSH directory and authorised users file, with the correct permissions in the home directory. Then edit the authorized_keys file with a text editor.

AWSMachine@george:$ mkdir .ssh
AWSMachine@george:$ chmod 700 .ssh
AWSMachine@george:$ touch .ssh/authorized_keys
AWSMachine@george:$ chmod 600 .ssh/authorized_keys
AWSMachine@george:$ nano .ssh/authorized_keys

Paste in the public key you previously copied and save the file, before exiting the system.

AWSMachine@george:$ exit
AWSMachine:$ exit

You should now be able to log into the new user george on the remote Amazon AWS server using the same keypair. On your local machine, use the SSH command:

LocalMachine:$ ssh -i ~/AWS-keypair.pem george@ec2-some-id.amazonaws.com

We now are logged in as user george.

Install the lxde lightweight desktop manager and start it.

 sudo apt-get update 
 sudo apt-get install lxde
 sudo start lxdm

Also install xrdp to establish a remote desktop connection.

 sudo apt-get install xrdp

Remote desktop connections require a username and password. The username will be george and the password will be the one set when the user was created. If you forget this the password can be reset using:

 sudo passwd ubuntu
In Remmina, create a new connection and paste in the public DNS of the AWS box (ec2-some-id.amazonaws.com) into the Server box. Set the user as george, make sure the protocol is RDP and set the Colour Depth as True Colour 32-bit. Save and Connect.


None of this would be known without the notes and guidance from others on the web, namely:

http://www.brianlinkletter.com/how-to-set-up-a-new-userid-on-your-amazon-aws-server-instance/

http://comtechies.com/how-to-set-up-gui-on-amazon-ec2-ubuntu-server.html

https://serverfault.com/questions/776391/unable-to-remote-connect-from-windows-to-ubuntu-desktop-using-ec2

Tagged , , , ,

Ubuntu Unity

A couple of months ago I ran the usual

sudo apt update && sudo apt upgrade

on my Ubuntu box and when I logged back in, there was no graphical user interface. Nothing. Not even a Ctrl-Alt-T into a terminal.

An internet search suggested it was a Compiz issue but nothing seemed to work. Until now! I thank Jonayad Rahman on the Ask Ubuntu forums for this fix:

Press Ctrl-Alt-F2 to call up a command-line. Log in with your usual credentials. You can then fix the compiz problem by removing the local config files:

sudo rm -fr ~/.cache/compizconfig-1
sudo rm -fr ~/.compiz

Then try this if your session still does not load:

sudo rm -fr ~/.Xauthority
sudo rm -fr ~/.config/autostart

Reinstall compiz:

sudo apt-get install --reinstall ubuntu-desktop unity compizconfig-settings-manager upstart

Reboot the machine and you should be able to log into Unity. Finally clear and restart the Unity Desktop by opening a terminal and typing:

sudo dconf reset -f /org/compiz/
setsid unity

So I have Unity back. More or less. I don’t seem to be able to right-click on the desktop and get a menu, and my wallpaper choices don’t seem to get saved, but it’s good to have some windows visible.

One thing I’ve discovered in the meantime is that I actually prefer the Gnome desktop!

Tagged , , ,
Advertisements