Category Archives: Computing

Rasterio conflict in Anaconda

Based on a comment from ocefpaf at https://github.com/conda-forge/gdal-feedstock/issues/69

I recently had an issue where my Anaconda environment was having issues when I wanted to import gdal or rasterio. I was getting the following error when I tried to import rasterio:

ImportError: libmfhdf.so.0: cannot open shared object file: 
No such file or directory

It seems to be a reasonably common issue based on online searches. The following fix worked for me and I was able to install other packages that I needed into the new env and it (so far) seems to be working OK.

conda create -n rasterio_test_env python=3.5 rasterio --yes -c conda-forge 
source activate rasterio_test_env
python -c "import rasterio; print(rasterio.__version__)"

 

Advertisements
Tagged

RDP from Ubuntu into AWS Ubuntu

Simple way to get RDP into Ubuntu AWS box:

  • Create the instance in the EC2 Management console and enable incoming RDP in Security Groups
  • Create a new user with a password
  • Add the new user to the sudoers file and copy over the AWS server’s public key
  • Log in as the new user and install xrdp
  • Load Remmina on your local machine and open a new RDP connection

Notes

In EC2 Management Console, click on the instance you want to connect to and scroll the page to the right. There will be a URL link to the relevant Security Group configuration page. Click on the Inbound tab and then the Edit button. Add a new rule, look for RDP as the type and set anywhere as the source (or make it specific if you want).

Move to the Terminal on your local machine and login to your AWS server with a command similar to:

LocalMachine:$ ssh -i ~/AWS-keypair.pem ubuntu@ec2-some-id.amazonaws.com

Create new user e.g. george

AWSMachine:$ sudo adduser george

Follow the prompts to enter the password and other optional user information. Add the new user to the sudoers file using the visudo command:

AWSMachine:$ sudo visudo

Add the following line after the comment line, “User privilege specification” and save the file.

george   ALL=(ALL:ALL) ALL

Ensure that the user george is in the sudo groupusing:

AWSMachine:$ sudo adduser george sudo

The new user account cannot be accessed (e.g. via SSH) until the public key from the AWS instance’s key pair (the .pem file) is installed for the new user. To do this, copy the public key installed for the ubuntu user and paste it into the correct file in the new user account. The public key can be found in the file ~/.ssh/authorized_keys under the ubuntu account.

AWSMachine:$ cat ~/.ssh/authorized_keys

SThe command above will print the public key. Select everything apart from the name of the key pair at the end of the key and copy it.

Next, switch to the new user account, george, and make sure you are in the home directory.

AWSMachine:$ sudo su george

Create the SSH directory and authorised users file, with the correct permissions in the home directory. Then edit the authorized_keys file with a text editor.

AWSMachine@george:$ mkdir .ssh
AWSMachine@george:$ chmod 700 .ssh
AWSMachine@george:$ touch .ssh/authorized_keys
AWSMachine@george:$ chmod 600 .ssh/authorized_keys
AWSMachine@george:$ nano .ssh/authorized_keys

Paste in the public key you previously copied and save the file, before exiting the system.

AWSMachine@george:$ exit
AWSMachine:$ exit

You should now be able to log into the new user george on the remote Amazon AWS server using the same keypair. On your local machine, use the SSH command:

LocalMachine:$ ssh -i ~/AWS-keypair.pem george@ec2-some-id.amazonaws.com

We now are logged in as user george.

Install the lxde lightweight desktop manager and start it.

 sudo apt-get update 
 sudo apt-get install lxde
 sudo start lxdm

Also install xrdp to establish a remote desktop connection.

 sudo apt-get install xrdp

Remote desktop connections require a username and password. The username will be george and the password will be the one set when the user was created. If you forget this the password can be reset using:

 sudo passwd ubuntu
In Remmina, create a new connection and paste in the public DNS of the AWS box (ec2-some-id.amazonaws.com) into the Server box. Set the user as george, make sure the protocol is RDP and set the Colour Depth as True Colour 32-bit. Save and Connect.


None of this would be known without the notes and guidance from others on the web, namely:

http://www.brianlinkletter.com/how-to-set-up-a-new-userid-on-your-amazon-aws-server-instance/

http://comtechies.com/how-to-set-up-gui-on-amazon-ec2-ubuntu-server.html

https://serverfault.com/questions/776391/unable-to-remote-connect-from-windows-to-ubuntu-desktop-using-ec2

Tagged , , , ,

Ubuntu Unity

A couple of months ago I ran the usual

sudo apt update && sudo apt upgrade

on my Ubuntu box and when I logged back in, there was no graphical user interface. Nothing. Not even a Ctrl-Alt-T into a terminal.

An internet search suggested it was a Compiz issue but nothing seemed to work. Until now! I thank Jonayad Rahman on the Ask Ubuntu forums for this fix:

Press Ctrl-Alt-F2 to call up a command-line. Log in with your usual credentials. You can then fix the compiz problem by removing the local config files:

sudo rm -fr ~/.cache/compizconfig-1
sudo rm -fr ~/.compiz

Then try this if your session still does not load:

sudo rm -fr ~/.Xauthority
sudo rm -fr ~/.config/autostart

Reinstall compiz:

sudo apt-get install --reinstall ubuntu-desktop unity compizconfig-settings-manager upstart

Reboot the machine and you should be able to log into Unity. Finally clear and restart the Unity Desktop by opening a terminal and typing:

sudo dconf reset -f /org/compiz/
setsid unity

So I have Unity back. More or less. I don’t seem to be able to right-click on the desktop and get a menu, and my wallpaper choices don’t seem to get saved, but it’s good to have some windows visible.

One thing I’ve discovered in the meantime is that I actually prefer the Gnome desktop!

Tagged , , ,

AWS & satellite data: a primer

Massive thanks to Annekatrien for her original post on this topic which should be available using this link: http://www.digital-geography.com/accessing-landsat-and-sentinel-2-on-amazon-web-services/

This is mainly a post to remind me of the steps I undertook (following the blog post above) to access Sentinel 2 data on AWS (developed and managed by Sinergise).

I use an Ubuntu 16.04 Mate VM with shared folders to the host machine as my processing/dev box. I use the Anaconda Python 3.5 distribution and the following has been tested and works on that system.

Python setup

I installed the AWS commandline packages using the conda-forge repository
conda install -c conda-forge awscli=1.10.44

and tested the installation using
aws help

which returned the man pages.

AWS Setup

I am currently on the free tier (https://aws.amazon.com/free/) which is fine for what is listed in this post.

Sign in and go to the Amazon Console.

Go to the Services tab at the top of the console then select:
Security&Identity > IAM > Users > Create a new user

Ensure that the check box to generate a new key is ticked, add your user name and click Create. Download your user credentials BEFORE clicking Close.

To access the data on Amazon S3, change the permissions using the Console by clicking on
Services > Security&Identity > IAM > Users

again and then clicking on the name of your new user. This will open a summary page where you can manage a user. Click on
Permissions > Attach policy

and choose AmazonEC2FullAccess and AmazonS3FullAccess (and any others you want) before clicking Attach. The IAM user should now be set up.

AWSCLI Config

In the bash terminal, type:
aws configure

and type in your access key ID and secret access key from the file downloaded earlier, when prompted. For region use the appropriate value from the table below based on the data you want access to. For output format, use json.

Landsat Sentinel-2
us-west-2 eu-central-1

Accessing data

To list the data available for Landsat use the following Terminal commands:
aws s3 ls landsat-pds

and for Sentinel use
aws s3 ls sentinel-s2-l1c

As Annakatrien says in her blog post, ‘to go deeper into the storage, and see separate images, you have to know what you’re looking for’.

In general you will use the following structure for landsat:
aws s3 ls landsat-pds/L8/<path>/<row>/<image name>/

and this for Sentinel-2:
aws s3 ls sentinel-s2-l1c/tiles/<UTM zone number>/<grid number>/<subgrid number>/<year>/<month>/<day>/

To download an image use one of the following commands:
aws s3 cp s3://landsat-pds/L8/201/024/LC82015242016111LGN00/ ~/Downloads/ --recursive
aws s3 cp s3://sentinel-s2-l1c/tiles/30/U/YC/2016/4/13/0/ ~/Downloads/ --recursive

More effort is then required to sort the downloads (if more than one image at a time) into a file structure on the local computer, as all images are download to the same directory.

Processing set up

This post is here as a reminder of what I did to setup my coding environment. If it is of any use or interest to anyone else then that’s great 🙂

First up I set up a VirtualBox VM on my desktop (from Entroware!). It is running an Ubuntu Mate 16.04 guest, but I guess that is inconsequential, as it could be running any version. It has the VirtualBox Additions installed, NAT and Bridge Adaptor networking are enabled, and a shared folder on the host has been setup.

On the guest, the shared folder is mounted using the command:

sudo mount -t vboxsf -o uid=1000 sharedVMfolder path/to/mountpoint

Anaconda Python has been installed on the guest and a new environment called spatialP3 has been created using:

conda create --name spatialP3

This is activated using:

source activate spatialP3

In that environment, a number of spatial libraries have been installed using:

conda install --name spatialP3 packagename

There were a number of version conflicts with some of the default packages, so I searched the Anaconda Cloud to find the most recent packages and installed them from the Anaconda Cloud repository using:

conda install -c conda-forge shapely=1.5.16

and similar.

To launch a Jupyter Notebook server with no local display (i.e. on the guest) use:

jupyter notebook --ip=0.0.0.0 --no-browser

Port forwarding has been set up in Virtualbox using the TCP protocol and with the host port being 8899 and the guest port being 8888 (the default for Jupyter Notebooks).

This means that you can run the notebooks from the host (as long as the VM and Jupyter server are both running) by typing the following into your browser:

localhost:8899

or from any machine on the network by typing in the local IP address of the host machine and the port 8899. If you start them in the correct location i.e. the shared folder that was mounted in the VM, then the notebooks will be accessible irrespective of whether the VM is running, and can be part of a backup strategy for the host machine.

Tagged , ,