StuBS Development Environment

Table of Contents

• tl;dr:
• Introduction
• Using the Linux pool remotely
  • SSH
    • Using SSH-Keys and Gitlab
  • Shared tmux session
• Required Software
• *StuBS-Template
• Testen und Debuggen von OOStuBS (Make-Targets)
• Quellcodeverwaltung in einem gemeinsamen Git für die ganze Gruppe

tl;dr:

• SSH and X2Go are available on our Linux computersauf die Pool-Rechner sind verfügbar. Zu einer Remote-Abgabe treffen wir uns auf l00p34.

    $ ssh ccd5802@l00p34.rz.tu-harburg.de # Computer 00, Pool 3c
    $ source /opt/OO-MPStuBS/env.sh       # Setup PATH for qemu/gcc
  

• Shared tmux session

    $ tmux -S /tmp/my-shared-tmux new                       # tmux session erstellen
    $ setfacl -m "u:xxd1234:rw" /tmp/my-shared-tmux         # xxd1234 hat zugriff
    $ tmux -S /tmp/my-shared-tmux attach                    # auf die Session verbinden (von beiden auszuführen)
  

• Setup development environment on your on Linux computer (or WSL2)

    $ apt install nasm qemu qemu-system-x86 qemu-kvm build-essential binutils gcc-multilib g++ g++-multilib
  

  • QEMU (qemu, qemu-system-x86, qemu-kvm)
  • GCC (build-essentials, binutils, gcc-multilibs, g++)
  • nasm

Introduction

For working on the practical exercises (= development of OOStuBS/MPStuBS/StuBSmI), all components are installed on the Linux pools of TUHH. The machines are reachable via SSH and X2Go and further setup instructions can be found in this document. Of course you can also work on the exercises on your own computer, but you will need a modern Linux installation or setup your Windows with Windows Subsystem for Linux. If we talk about your solution remotely, you have to setup a shared TMUX session on the l00p35 machine.

Attention

Those who wish to install the software at home are, of course, fully responsible for any problems that may occur. Failed installations are also not accepted an excuse for late solution deliveries.

As OS development is quite error prone, it is essential that you test your solution thouroughly, before you ask us to accept the solution. Our tool of choice for testing is the full-system emulator qemu (with and without kvm acceleration). The exercise acceptance will always be tested with qemu and on the physical machine in the pool room. Therfore, you have to test you solution with both. The pool is the reference plattform.

Using the Linux pool remotely

You can remotely log into the Linux pool with SSH. Alternatively, you can use X2Go to get a whole graphical desktop enviroment.

SSH

Important: The SSH login only forms from the TUHH network. Therefore, you have to use the TUHH VPN or the SSH Jumphost to log in from home.

As we have to do some load balancing when working remotely, you have to do some suffling for selecting a computer. For this, you take your group number modulo 12 and log into the machine:

$ ssh <your_login_name>@l<computer number>p34.rz.tu-harburg.de

So, for the user xxd1234 and group 17, the login would be on computer 5:

$ ssh xxd1234@l05p34.rz.tu-harburg.de

At home, you can define a shortcut for this in your SSH configuration (~/.ssh/config), which allows you to do ssh pool:

Host pool
     User xxd1234
     Hostname l05p34.rz.tu-harburg.de
     GSSAPIAuthentication yes
     GSSAPIDelegateCredentials yes
     ForwardAgent yes

As the computer pools have installed an antique Redhat Linux, we had to compile a few tools (GCC 10, Nasm 2.15, etc.) by our own. Therefore, you have to include those tools in your PATH by sourcing our enviroment file:

$ source /opt/OO-MPStuBS/env.sh

You can, and you should include this in your ~/.bashrc. Thereby, this happens automatically when you login.

Using SSH-Keys and Gitlab

For collaborating, we use the TUHH Gitlab, where we create group repositories for you. To access this repository, you have to create an SSH keypair and register the public key with your login. With the following command you can create a private-public keypair with ssh-keygen

$ ssh-keygen -t rsa -b 4096

By default the key is stored as ~/.ssh/id_rsa in your home in the hidden directory .ssh and the key is automatically offered to the remote server when you try to login into the remote machine. If you give it another name, you have to configure it in your ssh configuration (see ssh_config(1)).

If you have created the SSH key on you local machine, you can use an ssh-agent to forward the key over SSH so that you can use git commands on the pool machine. For this you have to start an ssh-agent if your desktop enviroment does not already start it and add the key:

$ eval $(ssh-agent)   # Start ssh-agent
$ ssh-add             # Add .ssh/id_rsa to the ssh-agent
$ ssh-add .ssh/gitlab # Add .ssh/gitlab to the ssh-agent
$ ssh-add -l          # List all keys the agent knows about

The created key has to be registered with our Gitlab: User Settings > SSH Keys. Afterwards, you can create all git commands without entering a passworts (with ssh-agent also on the remote machine).

Shared tmux session

For working together you can use tmux to share a TTY session with another user on the same local machine. The two users can then first see and edit the same terminal. Furthermore, tmux works session-based, i.e. one shares not only one terminal, but a whole session, specifically any number of terminals and their state and layout.

A new tmux session is created and attached to a Unix doman socket with (-S / -L):

$ tmux -S /tmp/my-shared-tmux new

If no other tmux session listens on that file, a new session is created. Please note, that you should change the socket name my-shared-tmux to something else, like group8.

Other uses can now attach on this socket with the attach command:

$ tmux -S /tmp/my-shared-tmux attach

When tmux is connected, you can detach from the session with Ctrl-b d.

In order to give access to another user, he must have the access to the socket file, which can be done in a fine-grained manner with ACLs (abc9876 is the username of your partner):

$ setfacl -m "u:abc9876:rw" /tmp/my-shared-tmux

Please be aware, that giving someone access to your tmux session allows him to execute commands on your behalf. So do not use the usual Linux permissions and 777 or something like that.

For more details on tmux, please consult the official tmux documentation or tmux' excellent manpage.

Required Software

For compiling, our Makefiles use g++, for assembling the startup code and the hardware-related subroutines we use the Netwide Assembler (nasm). The x86 emulator QEMU is suitable for preliminary testing and, by a built-in GDB stub, it is also very useful in combination with the GNU Debugger. In the Linux pools have already set up the appropriate environment; if you want to do the whole thing at home, you have to set up the mentioned software accordingly. If you have any problems, feel free to ask us.

*StuBS-Template

• For every exercise, we will provide yoy with an template for OOStuBS (https://collaborating.tuhh.de/e-exk4/teaching/ss23/bsb-oostubs) respective MPStuBS (https://collaborating.tuhh.de/e-exk4/teaching/ss23/bsb-mpstubs). The templates published in the TUHH Gitlab, where we will push incremental new updates for the other exercises. We will create repositories for your groups in Gitlab and you can get assigned to these groups by the tutor.
• The description of the group repository describes how you can integrate the template (either OOStuBS or MPStuBS) into your repository.
• For the other tasks we will extend the template repositories so that you only have to drag the changes into your repository. These modified defaults will contain new files added for each task.
• All the defaults you receive from us should compile correctly, but will contain incomplete code. So, for each task, you need to complete the code in the functions and classes described below.
• The compilation of *StuBS is done by calling make in the solution directory. All .cc and .asm files in the solution directory are then compiled with the appropriate tools (compiler and assembler respectively) and linked together as a bootable system image. Afterwards the commands make {kvm,qemu}{,-gdb,-ddd}{,-noopt} are available for testing and debugging (more about this in the next section).
• You can get an overview of the possible make targets with make help.

Testen und Debuggen von OOStuBS (Make-Targets)

• The simplest and fastest way to test your implmenentation can be done with make qemu-curses, which builds your system image and starts it in QEMU:

  ccd5802@l00p34:~/oostubs$ make qemu-curses
  

  By default, we start QEMU to emulate a system with four processors. For the development of OOStuBS/StuBSmI this should not bother you much, because the additional CPUs are simply "left aside" without any further action. For the MPStuBS tinkerers: through the KVM mode your system is really run in parallel on several cores. Therefore, we are relatively close to the test on the real hardware in terms of race conditions and faulty synchronization.

Attention

To quit a QEMU that was started in Curses mode, you have to switch to the monitor view (Alt-2) and type quit. To return from the monitor view to the CGA view, press Alt-1. By the way, on another virtual QEMU window (Alt-3) the serial port is displayed.

• To ease debugging, you can qemu in the full emulation mode (qemu-* targets) where all cores are emulated pseudo-paralell. With the kvm-* targets you get full parallel execution.

• If you are stuck with simple printf debugging, you can use QEMU's built-in GDB stub to connect to the emulation with a debugger (gdb). This way you can easily execute your operating system code step by step to find out the reason of any crashes or unwanted behavior. To make it easier for you we provide several targets to start qemu and the gdb. With this command, you start a QEMU (in windowed) mode and gdb in the same console.

    dietrich@obelix:~/oostubs$ make gdb
  

  If you work remotely, or if you want to see the output of QEMU, you have to open two terminals (one for QEMU, one for GDB) and use the -gdb targets:

  dietrich@obelix:~/oostubs$ make kvm-gdb     # Terminal 1: Start QEMU in KVM mode and connect via gdb
  dietrich@obelix:~/oostubs$ make connect-gdb # Terminal 2: Start gdb and connect to QEMU.
  

  A short GDB reference can be found here here. For detailed information, how to a specific GDB command works, you can use the built-in help function

  (gdb) help <command>
  

  Hint: As QEMU pauses the operating system when the gdb connects, you have to resume executeion with continue instead of starting the "program" with run.

  For a faster overview about registers and stacks you can use the excellent gdbinit configuration by storing it in your home under the name .gdbinit

• The easiest way to work from home is via an SSH connection. However, you cannot start a window on this connection by itself. But QEMU is able to map the CGA screen in text mode completely with NCurses to a terminal. We have therefore set up targets for you, which start QEMU/KVM with curses and work from SSH.

    dietrich@obelix:~/oostubs$ make qemu-curses
    dietrich@obelix:~/oostubs$ make kvm-curses
  

• By default, the build system compiles everything with -O3. If you suspect that you have some problem with undefined behavior, you can append an -noopt to the make target to build everything without optimizations. For example, the following command starts an unoptimized StBS in QEMU in the curses mode:

  dietrich@obelix:~/oostubs$ make qemu-curses-noopt
  

• An overview over all make targets is shown with:

   dietrich@obelix:~/oostubs$ make help
  

  Here is a collection of very useful gdb commands for OS development:

command	Help
bt	Show a function call back trace
info threads	Show an overview about all four CPUs (CPUs are modeled as "threads"
thread 3	Switch to CPU 3
disassemble	Disassemble the current function
info registers	Show all CPU registers (gdb style)
display /i *$pc	Display the next executed instruction on each prompt
x/10a $esp	Display the 10 top-most words on the stack an interpret them as **a**dresses
monitor info registers	Instruct QEMU to prin all registers (more detailed version)
monitor info mem	Show the current memory mapping!
monitor system_reset	Warm-Reboot of the emulated system

Quellcodeverwaltung in einem gemeinsamen Git für die ganze Gruppe

Wie bereits erwähnt, ist die Vorlage zu OOStuBS bzw. MPStuBS im Gitlab zu finden. Wir haben dort jeder Übungsgruppe ein Repository angelegt, welches ihr zur gemeinsamen Arbeit nutzen sollt. Um das Vorlagenrepo als Remote zu eurem Repo hinzuzufügen, benutzt folgende Befehle:

Für die Single-Core Variante:

    git remote add vorlage git@collaborating.tuhh.de:e-exk4/teaching/ss23/bsb/oostubs.git

Oder für die Multi-Core Variante:

    git remote add vorlage git@collaborating.tuhh.de:e-exk4/teaching/ss23/bsb/mpstubs.git

Vorlage pullen mit:

    git pull vorlage

Eine kurze Übersicht der Git-Befehle findet ihr hier. Als tieferen Einstieg in die verteilte Quellcodeverwaltung empfehlen wir das Pro Git Buch, welches als Creative Commons verfügbar ist.