In addition to these instructions, you can find a video tutorial here:

Step 1: Install vscode

The first thing you need to do is install vscode on your local device (person computer). We won’t go into the details of installing it here as it varies depending on your device (Linux, Windows, iOS). The install is usually straight-forward.
The vscode download for various platforms can be found here:

Step 2: Install Remote-SSH Extension

vscode cannot make remote connections initially. The Remote-SSH extension is required

Open vscode and click on the Extensions panel button (see Red Box in the image below)

Click on the extension search bar (see Red Box in the image below) and start typing remote ssh.

The Remote – SSH extension should appear (see Green Box in the image below).

Click on the Install button (see Purple Box in the image below).

Step 3: Create Remote-SSH Connection

A new panel button called Remote Explorer should appear – you may have to close and reopen vscode (see Red Box in the image below). Click on the Remote Explorer button.

Expand the tree down until you see the SSH branch (see Green Box in the image below).

Click on the plus (+) button to begin adding a new SSH connection (see Purple Box in the image below).

In the SSH Connection box that opens (see Red Box in the image below)
Enter ssh student@134.117.130.199 – replacing the IP with you own 134.117.#.# IP address

It will ask you to save the connection information to an ssh config file. The default one will vary depending on what operating system you are running. In general you can select the first option (see Red Box in the image below)

A pop-up will appear saying the host was added. If you want, you can manually modify the ssh config file by clicking Open Config. However, this is not required, and in this case we will just click Connect (see Red Box in the image below)

You will then be prompted for the password for the student account (see Red Box in the image below).

It will then show a pop-up on the bottom right showing the vscode-server being added to the remote account. This make take a few moments to complete.

And that should complete adding the connection. A new empty vscode workspace should open that is connected to that remote server.

Step 3 OPTIONAL: SSH with a Jump Host (No VPN) and Passkeys

As an alternative to VPN, you can ssh through the SCS access.scs.carleton.ca jump host. You can see this article for more information on using the access as a jump host.

We recommend you confirm with a regular ssh shell (terminal) application that you are able to use the access jump host to jump to your target server (eg: your openstack instance).

Once confirmed you can edit the vscode ssh config file manually and add the below config:
NOTE: If you are only connecting with a password, do NOT include the OPTIONAL IdentityFile and AddKeysToAgent options below, as they do not apply. If you wish to use the public/private key IdentityFile so that you do not need to types your password every time, then you should follow this article that explains how to set those up.

Host access-jump-host
  HostName access.scs.carleton.ca
  User cmail_username
  # OPTIONAL use of identify file (so you do not have to use a password)
  IdentityFile /Path/To/Private/Key/privatekey.ed25519
  # OPTIONAL add key to your local computer's ssh agent if you use a key with a passphrase.  You will need to research how to setup the ssh agent on your particular operating system (beyond the scope of this article)
  AddKeysToAgent yes

Host my-openstack-instance
  # The below IP must be your openstack IP or other destination server
  HostName 134.117.128.32
  User student
  ProxyJump access-jump-host
  # OPTIONAL use of identify file (so you do not have to use a password)
  IdentityFile /Path/To/Private/Key/
  # OPTIONAL add key to your local computer's ssh agent if you use a key with a passphrase.  You will need to research how to setup the ssh agent on your particular operating system (beyond the scope of this article)
  AddKeysToAgent yes

Once you have saved the above ssh config in vscode, replacing usernames, IPs, etc as required for your particular situation, you will now see my-openstack-instance as a connection option when you open a new remote ssh connection in vscode.

Step 4: Open Folder on Remote Server

Now that the vscode workspace is connected to the remote server, we need to open a folder. Click the Explorer navigation button (see Red Box in the image below) and click Open Folder (see Green Box in the image below)

The open folder search box will automatically start in the home directory (/home/student) of the student account, so simply click OK (see Red Box in the image below) unless you want to open a specific sub-folder.

You will again be asked to enter the password for the student account (see Red Box in the image below)

A window will pop up asking if you trust the authors of files in this folder. If a check box is shown to trust all files in the folder ‘student’, check it (see Red Box in the image below), although this may not appear in all cases. Click Yes, I trust the authors (see Green Box in the image below)

The Explorer view will now show all of the files and folders in the student account home directory (see Red Box in the image below).
NOTE: The exact content of the directory will vary depending on which course image you are connecting

Step 5: Open a terminal, edit some code, run some code!

It can be convenient to open a terminal window so we can run commands just like we would if we had ssh’d into the system. You can open a terminal window in many ways. Here are three easy ones:

• With the Ctrl + ` (Ctrl + backtick) hot key
• By right-clicking on any file or folder in the student home directory and clicking Open in Integrated Terminal
• By clicking on any file or folder in the student home directory, and going to the View menu and click Terminal

Either of these actions will open a terminal, as seen in the Red Box in the image below

Next, we can open a file to edit with vscode. In the example below, we have a folder called assignment-code, with a python file called hello-world.py (see Red Box in the image below). If you click on the hello-world.py file, it will open the file in the workspace so you can edit it (see Green Box in the image below).
NOTE: If this is the first time you have edited a python file, vscode may helpfully offer to install a python extension to help with you python coding (see Purple Box in the image below). It will often do this for any programming language that you open for the first time. It is up to you to decide if you want this extension. In some cases, your instructor may ask that you install specific extensions with certainly capabilities. So make sure to check your course notes.

We can use the terminal window to conveniently run commands. In this example, we manually run the python program hello-world.py.

• First we cd assignment-code to get into the assignment directory (see Red Box in the image below)
• Then we enter ls to list the files and make sure we see the hello-world.py file (see Green Box in the image below)
• Then we run the python file with python3 hello-world.py and we see the output (it prints Hello, world!â€�) (see Purple Box in the image below)
  NOTE: The commands you run to execute code will vary, this is just one example of a possible way of running a python file

Troubleshooting: Failed Login

vscode failed logins are most often caused by two things:

1. New Openstack Instance with Expired Password: New openstack instances have a default username and password, and the password is expired and must be changed. This cannot be done through the vscode remote connection interface, so you must use a different Connection Tool to connect to your instance for the first time. Once you have set a new password, you can use vscode normally.
2. Instance Disk Full: If an instance’s disk drive fills up, it will not be possible to login with vscode as the tool needs to setup temporary files to initialize. The indicator of this problem is (usually): when you enter the password, vscode will hang, sometimes showing an installing vscode server message that never goes away. If you see that behavior, it is likely that the disk is full. You must login with another Connection Tool, free up some disk space, and then return to using vscode.

Troubleshooting: Disk Full

vscode installs a lot of libraries and support tools to make you coding easier. It also stores a cache of all the original downloaded installers, which it keeps even after the tool has been successfully added to vscode. It is quite common for all of this to quickly fill an openstack instance’s disk drive. Check with your course instructor for instructions on how to clear the vscode cache as it can vary.

The post vscode: Remote Access and Code Editing appeared first on School of Computer Science.

IMPORTANT: VPN Changes Beginning Fall 2023

°ä²¹°ù±ô±ð³Ù´Ç²Ô’s&²Ô²ú²õ±è;Information Technology Services has decided that, beginning in Fall 2023, ÐÓ°ÉÔ­´´ University students that need to use VPN will have to setup the Microsoft Azure MFA (Multi Factor Authentication) for their student cmail account. The procedures to do this take a bit of time, so you should do this well in advance of needing VPN.
There are two ways to setup your Azure MFA access to use with VPN:

1. Permanent Azure MFA VPN Setup:
   Using this method, you can permanently setup your Azure MFA and VPN access for your cmail account. This method is more time consuming as it requires you to speak to an ITS help desk representative to complete the Multi Factor setup. The instructions can be found here: Student Requiring VPN Access
2. Course Linked Azure MFA VPN Setup:
   This method can be completed more quickly, as it is based on your membership in a course that requires VPN access. However, once that course ends, if you have no other courses that require VPN access, your VPN access will be revoked. The instructions can be found here: Students in Courses Requiring VPN

This most directly impacts students who are using the SCS Openstack, as the SCS Openstack can only be accessed via a ÐÓ°ÉÔ­´´ network. So from home you are required to use VPN to access the SCS Openstack.

The post VPN Changes Beginning Fall 2023 appeared first on School of Computer Science.

Openstack technical support is provided by the SCS Technical Staff. In order to provide effective support, please include the following items in your email so that we have as much information as possible.
WARNING: To be provided technical support, you must first:

1. Have thoroughly reviewed the technical support articles found on our Openstack Technical Support page.
2. Followed these instructions for getting in-person support, which includes first engaging with your TAs and Lab Coordinators, who are trained to help with many issues.

Openstack Support Request Email

When you email the SCS Technical Staff to request Openstack technical support, you must include the following items in your email:

1. Course Code for your course, including the section number
   
   eg: 1) COMP1405A-W22
2. Openstack Project Name for your course
   
   eg: 2) COMP1405-W22
3. Confirmation that you can Login to ÐÓ°ÉÔ­´´’s VPN as per these VPN connection instructions
   
   eg: 3) YES - I can login to VPN
4. Openstack Instance Name for the instance that you are experiencing problems
   
   eg: 4) readthedocs-03
5. Openstack Floating IP Address for the instance that you are experiencing problems
   
   eg: 5) 134.117.26.78
6. Description of the Problem including:
   • General explanation of the problem – an overview of what you are trying to do and what is not working
   • Exact steps to reproducing the problem so we can attempt to reproduce it as well
   • List any error messages or error codes
   • Provide screen captures or photos of any errors or other problems, where possible
7. Names of the people (TAs, Lab Coordinators, and/or Instructors) that you worked with so far on this technical issue
   
   eg: TA 1: John Doe; TA 2: Jane Doe; Lab Coordinator: Jon Smith

The post Prepare an Openstack Instance Support Request appeared first on School of Computer Science.

Computing Devices not Supported by SCS

Many courses offered by the School of Computer Science, especially undergraduate courses rely heavily on Virtual Machines and other forms of virtualization. Given this, it is required that you have access to a personal computing device which supports virtualization, while completing your program in the SCS.

Devices that do not support virtualization and Apple devices using the M-series chipset

There are many devices that do NOT support virtualization, including:

• Some Intel and AMD chipsets for lower-end laptops and netbooks (You can check the BIOS to see if virtualization is available)
• Most hand-held devices such as phones and tablets based on Android or iOS
• Chromebooks and other similar devices

Also, recent Apple computers, (those manufactured after November 2020) using the new ARM-based M-chipset (M1/M2/M3) use virtualization that is not yet supported by most hypervisors (such as VirtualBox, VMware, KVM, etc). The M-processor Apple computers offer excellent performance and battery life, but because of this limitation, you will not be able to run the virtualization software required by some of your SCS courses. Due to this, as indicated in the CPU section of the Laptop Requirements Specification, the School of Computer Science recommends that students do not purchase new Apple computers employing the M-chipset.

In line with this, the SCS Technical Support Team does NOT provide any specific technical support for M-processor Mac devices or other devices without virtualization capabilities.

I have already purchased a device that does not support virtualization, What can I do?

If your personal computing device does not support virtualization, or if your devices is experiencing issues with Virtual Machines, there are several options for completing course work that requires virtualization:

OPTION 1: SCS Windows Labs

You can use the SCS Windows PC Labs, either visiting an SCS lab in person or remotely using a web based remote access tool. VirtualBox is installed and many of the SCS course Virtual Machines are made available there. Please review important information about the SCS Lab Virtual Machines as they do have limitations compared to Virtual Machines running on your own device.

OPTION 2: SCS Openstack

Many SCS courses that use Virtual Machines also make Virtual Machine resources available on the SCS Openstack. Your course instructor can tell you if such resources are available for their course.
General (non-course specific) instructions can be found on our SCS Openstack Step-By-Step Guide and these instructions are completely sufficient for most of our openstack courses.
On openstack, you will be able to launch the Virtual Machine as an “instance” and it will run on our servers for the duration of the term. Importantly: any changes you make will persist until the instance is destroyed at the end of the term (unlike using Virtual Machines in our SCS Windows Labs).
Further details will typically be provided by your instructor, lab coordinators and TAs as the academic term gets underway.

I already purchased an Apple computer with the M-chipset. Do I have any other options?

Yes! There is an another virtualization option for the Apple M-procssor Mac computers in addition to the two options above!

OPTION 3: Parallels (for Mac M-processor users)

For users with the M-processor Mac chipset, the 3rd party software provides an additional virtualization option. It is a hypervisor similar to VirtualBox, VMware, KVM, etc. More recent versions of the software is able to provide virtualization support on M-processor Macs. We have heard anecdotal evidence that it works really well, really poorly or not at all – so you will need to try it to see if it will work for you!

A yearly subscription fee applies to the software. We recommend that you test with their Free Trial first before you commit to purchasing it.

The SCS Technical Support Team does NOT provide any support for this software.

The post M-processor Mac Devices & Other Non-Virtualization Capable Devices Not Supported by SCS appeared first on School of Computer Science.

DEPRECATED – x2go is not used on newer images created since the Fall of 2024 – Instead use TurboVNC with our newer images

Download x2go:

You can find a video demo of these instructions here:

Install x2go, following the application’s instructions

Open x2go

• On Windows you may get a warning to allow x2go to access the network (firewall rule change); approve the request. This may also pop-up each time you create a new session (described below), so you may have to approve the request more than once.

Create a new session in x2go by clicking Session -> New Session
The key settings that must be entered in the x2go session settings window that pops up are:

• Host: 134.117.216.10
• Login: student
• Session Type: XFCE – This is used for many of our course images, but your course may have a different preferred desktop. Another common one we use is LXDE

Once the settings are entered, click OK

In x2go you will see a new session connection:

Click on the session to initiate the connection:

• You will be asked to enter the instance’s password
• If it is the first time you are connecting to the instance with x2go, you will be asked if you trust and approve connecting to the host. Again, approve the connection

Finally, a window should pop-up connected to the desktop:

The post SSH Connection with x2go Remote Desktop Client appeared first on School of Computer Science.

This article explains how to connect to an openstack web console using a web browser.

Openstack Console – Security Warning

SECURITY WARNING: It is important to remember than ANY user logged into an openstack project can view the console for any openstack instance, so make sure you set a strong password, and only use the console when the other options below do not work:

• Connect remotely (ssh terminal – putty) – | text
  Connect remotely to the instance using an ssh terminal (examples uses the putty ssh tool)
• Connect remotely (desktop – x2go) – | text
  Connect remotely to the instance using a remote desktop tool (examples uses the tool)

Other openstack project users can observe activities taking place on the console. It is no different if they were watching over your shoulder on a real computer screen connected to your instance. Once you have logged into the console, other users can use your account, opening applications, running commands, etc.

Connect to an OpenStack Instance Using the OpenStack Console

See a demo video of these instructions here:

Step 1 – Go into your instance details page

1. Make sure you are in your course project (green box in the image below)
2. Click on the instances’ tab (Yellow Box in the image below)
3. Click on instances’ link (red box in the image below)

Step 2 – Open the console

1. In the instances details page, click on the Console tab (green box in the image below)
2. Click on the link Click here to show only console (red box in the image below)

Step 3 – Login to the Instance

• You will now be on the console screen and you can login using your account

The post Connecting to an Instance Using the Openstack Console appeared first on School of Computer Science.

SSH Connection Using an SSH Terminal Client

We recommend using PowerShell on Windows as the quickest way to open an SSH terminal connection, but there are many SSH terminal tools out there you use.

NOTE: This example uses putty, but the settings are the same for any terminal tool that supports SSH.
Putty can be installed through the Windows Store

You can find a video demo of these instructions here:

Enter settings into putty:

Enter username / password:

IMPORTANT: – Remember that the first time you login, you will be prompted to change your password. See the Login Credentials section at the top of this page for details.

You should see a similar prompt when you successfully connect:

The post SSH Connection Using the putty SSH Terminal Client appeared first on School of Computer Science.

Step 1: Set up Multiple Compute Nodes
Step 2: Update hostname of the compute nodes
Step 3: Set up the SSH Keys and Environment
Step 4: Replicate the SSH Environment
Step 5: Replicate /etc/hosts file
Step 6: Test Your Passwordless SSH Connections!
Graphic: Set up of SSH Enabled Multi-node Compute Environment
Copy Files Between Remote Nodes

This guide will take you through the steps of setting up a multi-node compute environment that uses ssh keys to support scripts and computing tools that communicate and move data between the compute nodes.

You can learn more general information about passwordless ssh login here:
Passwordless SSH With Key Pairs

Step 1: Set up Multiple Compute Nodes

For this example, we will set up the following nodes (instances) on:

• primary-node – main control node for your multi-node environment; this node will typically have an OpenStack floating IP address assigned so that you can access the environment from the ÐÓ°ÉÔ­´´ network (either an on-campus computer or your home device using VPN).
• node-1, node-2, node-3, …, node-n – compute nodes (as many as required)

Instructions for setting up nodes (instances), assigning floating IPs, and connecting to instances can be found here: OpenStack Step-by-Step Guide

Step 2: Update hostnames of the compute nodes

In this step, you will need to edit two system files: /etc/hostname and /etc/hosts. To do so, you will need to open them with root privileges by prefixing your command with the sudo command. For example, you can use the simple nano editor to edit one of the files like so:
sudo nano /etc/hostname – sudo checks that your user is authorized to run commands as root and will prompt for your password

• On the primary-node and node-1 -> node-n, edit the /etc/hostname files to have a suitable name for each host. After a reboot, this will be reflected in the system name that appears on the console, the shell and the desktop. This is helpful for making sure you do not make changes to the wrong system!
• On the primary-node, edit the /etc/hosts file to include IP HOSTNAME records.
  
  For example:
  
  192.168.5.123 primary-node

192.168.5.179 node-1

192.168.5.65 node-2

192.168.5.221 node-3
  
  NOTE 1: You will need to find the IP address of your nodes from either the OpenStack interface or running the commands ipconfig or ip addr on the node itself
  
  NOTE 2: On the SCS OpenStack, the IP address will ALWAYS start with 192.168.x.x

Step 3: Set up the SSH Keys and Environment

Setting up a passwordless ssh key environment will allow your scripts or compute tools to communicate and move data between all of your compute nodes. This tutorial will not go into all the background about ssh key pairs, but you can learn more about that in this guide: Passwordless SSH With Key Pairs.

Steps for setting up the ssh key pair environment:

1. Login to the primary-node as the user that will be running the scripts or compute tools and run a terminal or shell (student user works fine on most of our images)
2. Create an SSH public/private key pair:
   
   ssh-keygen
   
   Do NOT enter the a filename (leave the default /home/student/.ssh/id_rsa)
   
   Do NOT enter a passphrase (leave it blank)
   
   This will create the public and private keys:
   
   ~/.ssh/id_rsa – private key
   
   ~/.ssh/id_rsa.pub – public key
3. Copy the user’s public to that user’s account on every node (including the primary-node itself):
   
   This is done using the ssh-copy-id command. You will need to say yes when prompted to verify the host-key (if this is the first time you are connecting to that node). You will also be prompted to enter the password for the user’s account
   
   ssh-copy-id node-1

ssh-copy-id node-2

ssh-copy-id node-3

...

ssh-copy-id node-n

ssh-copy-id primary-node – This will ensure that the ~/.ssh/known_hosts file has the host key for EVERY node AND the primary-node. It will also add the user’s public key to the ~/.ssh/authorized_keys file on the primary-node, meaning the private key can ALSO unlock the primary-node, in addition to node-1 -> node-n

SSH public and private key (key pair) and environment files are now set up
After completing the above steps, all four ~/.ssh/ files should be configured:
~/.ssh/id_rsa – private key
~/.ssh/id_rsa.pub – public key
~/.ssh/authorized_keys – includes the public key (It will be identical to the id_rsa.pub file if no other keys were set up previously)
~/.ssh/known_hosts – includes the host keys for EVERY host (primary-node and node-1 -> node-n)

Step 4: Replicate the SSH Environment

The SSH public and private keys and the environment files can now be COPIED from primary-node to each of node-1 -> node-n
scp ~/.ssh/* node-1:~/.ssh/
scp ~/.ssh/* node-2:~/.ssh/
scp ~/.ssh/* node-3:~/.ssh/
...
scp ~/.ssh/* node-n:~/.ssh/

Step 5: Replicate /etc/hosts file

To make it easier to move between the primary-node and node-1 -> node-n, COPY the /etc/hosts file to each node.
NOTE: Because /etc/hosts is a system file, you will need to first copy the file to each node, then ssh onto the node and use sudo to copy the file to that system’s /etc/hosts

Copy the /etc/hosts file to each node (repeat the below steps for each of node-1 -> node-n:)

1. Copy the /etc/hosts file from the primary-node to node-1:
   
   scp /etc/hosts node-1:~/
2. SSH to node-1:
   
   ssh node-1
3. Move the hosts file you just copied to overwrite the system /etc/hosts file:
   
   sudo mv ~/hosts /etc/hosts
   
   As was the case earlier in this guide, when you edit (or in this case, overwrite) a system file such as /etc/hosts, you need to prepend your command with the sudo command

Step 6: Test Your Passwordless SSH Connections!

You should now be able to ssh between all of your nodes without using a password.

• Test from the primary-node:
  
  ssh node-1

ssh node-2

ssh node-3

...

ssh node-n

ssh primary-node
• Test from at least one of the nodes, such as node-1:
  
  ssh primary-node

ssh node-1

ssh node-2

...

ssh node-n
  
  NOTE: Ideally test all of the systems, but if you have a LOT of them, a few random tests should be sufficient
  
  NOTE: Some tools and scripts only require that you can ssh from the primary-node to node-1 -> node-n, so you would not need to do this step.

Graphic: Set up of SSH Enabled Multi-node Compute Environment

Graphical overview of setting up a multi-node compute environment, using ssh keys for inter-node connections:

Copy Files Between Remote Nodes

Once you have the ability to ssh between computers, either using passwords or the ssh keys as setup above, it is easy to copy files the computers. If you do have ssh keys setup allowing you to ssh with no password, then these tools will not prompt for a password. Here are a few example tools:

• Copy files with rsync: rsync options source_path destination_path

rsync -r -v ~/Downloads student@node1:~/Downloads
  
  
  
  • -r – tells rsync to copy files recursively (all sub-files and folders)
  
  
  • -v – tells rsync to show the files as they are copied
  
  
  • ~/Downloads – tells rsync to copy the files in the user’s Downloads folder
  
  
  • student@node1:~/Downloads – tells rsync to copy the files to the remote server node1 using the student user account, and copy the files into the student user’s Downloads directory
  
  
  
  

  rsync is a powerful tool as you can specify various options that control what is copied. For example, it will not “sync” files that are already up-to-date on the destination node.
  

• Copy files with scp: scp options source_path destination_path

scp -r -v ~/Downloads student@node1:~/Downloads
  
  
  
  • -r – tells rsync to copy files recursively (all sub-files and folders)
  
  
  • -v – tells rsync to show the files as they are copied
  
  
  • ~/Downloads – tells scp to copy the files in the user’s Downloads folder
  
  
  • student@node1:~/Downloads – tells rsync to copy the files to the remote server node1 using the student user account, and copy the files into the student user’s Downloads directory
  
  
  
  

  scp is not “tunable” to precisely control what files are copied. It also does not check if a file is already up-to-date as the destination.
  

The post Using SSH Keys to Support Multi-node Compute Tools & Scripts appeared first on School of Computer Science.

Uploading Custom Images to Openstack
Snapshots: Customizing Existing Openstack Images
Openstack Image Considerations
Openstack Image Tips
Minimize Image Size

Uploading Custom Images to Openstack

Due to space constraints, only the SCS Technical Staff can upload new custom images to Openstack. Please contact them to arrange for your image to be uploaded.

Snapshots: Customizing Existing Openstack Images

Snapshots are an easier alternative to building your own image before waiting for the SCS Technical Staff to vet and upload the image. To make a snapshot, you simply launch an instance using one of our existing images and then customize the instance as required. The instance can then be captured as a snapshot which can then be used as an image to launch other instances.

Benefits of snapshots based on an existing Openstack image:

• The SCS Technical Staff can permit you to create snapshots on Openstack
  
  As opposed to custom images which can only be uploaded by the SCS Technical Staff
• Our images are already tested on our Openstack environment. Customizing one allows you to avoid the risk of your image not being compatible with our Openstack or not working as expected.
• Our images are quite small, so building off them may help you keep your own image compact so that it launches as quickly as possible.
• Our images have a consistent environment and user interface, which also matches that of our course virtual machines (virtualbox/vmware). Thus, a large number of our Students, TAs, Instructors and Researchers will already be familiar with the basic environment. This may be helpful if other users will be using your image.
• If you plan to use a Multi-node compute cluster, you can create a customized instance with the environment required for your nodes. You can then snapshot it and use that snapshot to spawn all of the nodes. This way you don’t have to repeat the steps for setting up your environment on each of your nodes

Openstack Image Considerations

If you wish to put your own image onto the SCS Openstack, there are a few things to consider:

• Disk Size Considerations:
  
  Ideally, your initial image’s disk space should be no larger than 8-16 GB. You can get additional space once it is operating in the SCS Openstack, but initially, you should try to keep the operating system and installed applications under that size so that they can be rapidly staged onto the compute nodes.
  
  NOTE: The above size is the size of the physical disk, not the amount of actual data (which will be less). When creating an image with a virtualization tool, start with a disk size that is as small as possible and only increase it by a small amount as required.
• Networking Considerations:
  
  The operating system may differ in how it names virtual networking devices, so you should NOT hardcode network settings into your operating system and instead rely on a generic config that uses DHCP, as required by Openstack. This way, whether you create the image in Openstack, virtualbox, vmware, kvm, etc., the instances should still properly get an IP address when launched on Openstack

Openstack Image Tips

• Snapshots: Instead of building an image on your own computer, launch an instance on the SCS Openstack using one of our existing – optimized – images. You can then customize this image as you require, and we can then capture a snapshot of it for you to use as an image.
• No Data!: Do not save computational data on your image, as this will increase its size and cause delays when you want to launch an image. It is always possible to add the data to the instances once they are running
• Minimum Physical Disk Size: Keep the size of your physical disk as small as possible (8-16 GB – see above). Ideally, you should only have 30-40% or less of free space. It is easy to add space to an image later, but removing it to make the image smaller requires additional effort.

Minimize Image Size

Here are some tips that can help you reduce the amount of data in your image, and reduce the size of the image/snapshot file:

• Remove swap partitions/files – swap partitions are not needed in an image. They can easily be added later if needed
  
  swapon -s – should not show any files or partitions
  
  /etc/fstab – make sure no swap is defined.
  
• Clean-up distro cache – make sure there are no unnecessary packages and that the cache is clear (eg: ubuntu):
  
  apt autoremove – removes dependent packages that are no longer required
  
  apt clean – cleans the cache
  
• Clear excess log files – check /var/log for large log files that are not needed
  
  Clear ubuntu system log: journalctl --vacuum-size=32M – This clears all but the last 32 MB of log files
  
• IMPORTANT Final step: zero the disk – “zero” all of the empty disk space on the server. This is vital as deleting files in linux usually just de-links the file, but does not “zero” the data. When openstack captures the image, even though the file is deleted, the data (bits) is still there and gets copied into the image/snapshot. Zeroing the disk ensures this space appears empty and is not captured in the image/snapshot.
  
  cat /dev/zero > zero.file ; sync ; sleep 1 ; sync ; rm zero.file – This sequence of commands uses the /dev/zero device to create a file of just zero (0) bits that fills the entire disk, and is then deleted.

The post Using a Custom Image on the SCS OpenStack appeared first on School of Computer Science.

Why extend a disk?
What is LVM Partition
Check Existing Space
Extend Partition using the Script
Extend Partition Manually

Why extend a disk?

Many of our images (primarily on openstack) do not grow their partition/file system size automatically if you choose a flavor (openstack resource definition) that allocates more disk space to your instance than was available in the original image. It is possible to use tools such as cloud-init to do this automatically. However, in our public images, which we use on multiple environments (openstack, virtualbox, vmware, etc), we find cloud-init can sometimes cause issues. Instead, we have scripts designed to resize the partitions for the user.

What is LVM Partition

Historically, many of our standard images for Openstack and VirtualBox virtual machines use LVM (Linux Logical Volume Manager) to provide the disk volume for the operating system, as well as additional storage volumes (if applicable). LVM makes it easy to resize and re-purpose space within Linux, usually without even requiring a reboot.

You can learn more about LVM here: – LVM is available for most Linux distros.

Check Existing Space

Before you can enlarge the disk, first you have to check to see if it is possible:

Check physical disk size versus the partition or LVM size (using fdisk):

• Run fdisk, to see if there is more space available that could be added to the LVM:
  
  sudo fdisk -l
• In most of our basic images, this should show just a single physical device, such as:
  
  Disk /dev/vda: 32 GiB, 34359738368 bytes, 67108864 sectors – 32 GB physical disk
• It will also show the main partition, typically the LAST partition (which could be used by LVM):
  
  Regular partition example:
  
  Device Start End Sectors Size Type

/dev/vda1 2048 4095 2048 1M BIOS boot

/dev/vda2 4096 16777182 16773087 8G Linux filesystem
  
  In this case it is an 8 GB partition (/dev/vda2)
  
  LVM Example:
  
  Device Boot Start End Sectors Size Id Type

/dev/vda1 2048 16777215 16775168 8G 8e Linux LVM
  
  In this case it is an 8 GB partition (/dev/vda1)
• And finally it will also show the LVM itself if LVM is being used:
  
  Disk /dev/mapper/COMPbase--vg-root: 8 GiB, 8585740288 bytes, 16769024 sectors

So, in this case, the Physical disk is 32 GB, but only 8 GB are currently allocated to the LVM, and no other partitions are using the rest of the space for anything else. So we can extend it!

Extend Partition using the Script

If you are using one of our standard SCS images, either for OpenStack or for VirtualBox virtual machines, it usually comes with a small script called extend-disk.sh. The script is found in the home directory of the student user under the extend-disk folder. NOTE:Older images may have the script called extend-lvm.sh, found in the extend-lvm folder in the student home directory

To run the script:

• Open a terminal
• Go to the directory with the script, usually: cd ~student/extend-disk/
• Enter: sudo ./extend-disk.sh /dev/vda – NOTE: the /dev/vda may be different on your system. It is the device you found in Check Existing Space above

That’s it! Now when you run fdisk, you should see the last partition is the full size of the physical disk: sudo fdisk -l.

To update or download the script:

• If you do not have the script, you can get it using git:
  
  git clone https://git.scs.carleton.ca/git/extend-disk.git
• If you have the script and it was pulled from the git repository, then you can pull updates:
  
  cd extend-disk go into the extend-disk git folder
  
  git pull to pull the updates
  
  NOTE: If you are unsure if it was pulled from the git repository, then check if there is a .git hidden folder in the extend-disk folder (check for hidden folders with ls -la); If not, then you need to clone the repository using the git clone command above.

Extend Partition Manually

Here are the steps to manually extend the partition and the filesystem, including for the LVM case.

NOTE: We assume the same setup as the above Check Existing Space

• Extend the physical drive partition:
  • sudo fdisk /dev/vda – Enter the fdisk tool to modify /dev/vda
    
    NOTE: End each one letter command by pressing [Enter]; if the instructions do not specify a specific response to a question, just press [Enter] to accept the default
  • p – p command prints the disk info, same as running fdisk -l /dev/vda
  • d – d command delete the last partition (usually /dev/vda1 or /dev/vda2 on our images)
  • n e n command creates a new partition; e makes that an extended partition
  • Optional: t – t changes the type of partition. NOTE: This should not be required as fdisk should use the same type as was removed. You can also enter L when prompted to find the list of codes for the types. Make sure to use the same type as was listed by fdisk for the partition you removed in the earlier d step.
    
  • w – w writes the changes to disk and exits fdisk
• IF using LVM: Modify (extend) the LVM:
  • Tell LVM the physical partition size has changed: sudo pvresize /dev/vda1
  • Find the actual path of the LVM logical volume: sudo lvdisplay – The LV Path is the value needed
  • Tell LVM to extend the logical volume to use all of the new partition size: sudo lvextend -l +100%FREE /dev/COMPbase-vg/root – Using the LV Path from above
• Resize the file system:
  
  sudo resize2fs /dev/vda2 or for lvm: sudo resize2fs /dev/COMPbase-vg/root

That’s it! Now, when you run sudo fdisk -l, it will show that the LVM is using all of the space on the physical disk!

The post Extend Linux Disk (disk space) appeared first on School of Computer Science.