NoOS - No Operating System, Just Rust

A minimal 64-bit operating system kernel written in Rust, designed to explore the fascinating world of bare-metal systems programming. NoOS boots via GRUB2 with Multiboot2 compliance and demonstrates how modern memory-safe languages can handle the most demanding low-level tasks.

The Philosophy Behind NoOS

Operating system development has traditionally been the domain of C and assembly language, where memory bugs and undefined behavior lurk around every corner. NoOS challenges this paradigm by leveraging Rust’s ownership model and type system to create a kernel that’s both safe and performant.

The name “NoOS” reflects the project’s educational purpose: this isn’t a production operating system, but rather a clear window into how operating systems work at the most fundamental level, stripped of the complexity that obscures understanding.

Boot Process Architecture

NoOS follows a carefully orchestrated boot sequence that transitions the CPU from its initial state to a fully functional 64-bit environment.

Stage 1: GRUB2 and Multiboot2

The boot process begins with GRUB2 loading the kernel binary:

BIOS → GRUB2 → Multiboot2 Header → Kernel Entry

The kernel includes a Multiboot2 header that tells GRUB2 how to load it:

; boot/src/multiboot2.asm
section .note.GNU-stack note
section .multiboot_header

header_start:
    ; Multiboot2 header
    dd 0xe85250d6                 ; Magic number (Multiboot2)
    dd 0                          ; Architecture 0 (protected mode i386)
    dd header_end - header_start  ; Header length

    ; Checksum
    dd 0x100000000 - (0xe85250d6 + 0 + (header_end - header_start))

    ; Required end tag
    dw 0  ; Type
    dw 0  ; Flags
    dd 8  ; Size

header_end:

Stage 2: Protected Mode Entry

GRUB2 hands control to the kernel in 32-bit protected mode. The assembly bootstrap code then:

1. Sets up the stack: Establishes a 16KB stack for the boot process
2. Stores boot information: Preserves the Multiboot2 info pointer for later use
3. Checks CPU features: Verifies CPUID support and long mode availability
4. Sets up paging: Creates identity-mapped page tables for the transition

Stage 3: Long Mode Transition

The transition to 64-bit mode requires precise steps. First, the CPU’s long mode support is verified:

; boot/src/long_mode.asm
section .text
bits 32

global check_long_mode
global enable_paging

check_long_mode:
    ; Check for CPUID
    pushfd
    pop eax
    mov ecx, eax
    xor eax, 1 << 21
    push eax
    popfd
    pushfd
    pop eax
    push ecx
    popfd
    cmp eax, ecx
    je .no_long_mode

    ; Check for extended CPUID
    mov eax, 0x80000000
    cpuid
    cmp eax, 0x80000001
    jb .no_long_mode

    ; Check for long mode support
    mov eax, 0x80000001
    cpuid
    test edx, 1 << 29
    jz .no_long_mode
    ret

.no_long_mode:
    mov esi, error_message
    call print_error

Then paging is enabled with the transition to long mode:

enable_paging:
    ; Load P4 to cr3
    mov eax, p4_table
    mov cr3, eax

    ; Enable PAE
    mov eax, cr4
    or eax, 1 << 5
    mov cr4, eax

    ; Enable long mode
    mov ecx, 0xC0000080
    rdmsr
    or eax, 1 << 8
    wrmsr

    ; Enable paging
    mov eax, cr0
    or eax, 1 << 31
    mov cr0, eax

    ret

Stage 4: Rust Kernel Entry

Once in 64-bit mode, the assembly transfers to the Rust kernel:

; boot/src/boot.asm (continued)
bits 64

long_mode_start:
    ; Load null segment selectors
    mov ax, 0
    mov ss, ax
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax

    call remap_pic    ; Remap PIC
    sti               ; Enable interrupts

    pop rdi           ; Get multiboot info
    call kernel_main  ; Call the Rust entry point
    cli               ; If we return, hang

.hang:
    hlt
    jmp .hang

The Rust kernel_main function takes over:

#[unsafe(no_mangle)]
pub extern "C" fn kernel_main(multiboot_info: u64) -> ! {
    crate::clearscr!();

    multiboot::load_boot_info(multiboot_info);
    idt::init();

    if multiboot::debug_mode_enabled() { crate::println!("Debug mode enabled"); }
    if multiboot::release_mode_enabled() { crate::println!("Release mode enabled"); }

    crate::println!("Kernel initialized successfully!");
    crate::println!("Welcome to NoOS!");

    let mut shell = shell::Shell::new();
    shell.show_prompt();

    loop {
        process_scancodes(&mut shell);
        unsafe { hlt() }
    }
}

Memory Management

Page Table Structure

NoOS implements x86_64 4-level paging with identity mapping:

PML4 (Page Map Level 4)
  └── PDPT (Page Directory Pointer Table)
        └── PDT (Page Directory Table)
              └── 2MB Huge Pages (identity mapped)

The page tables are set up in assembly before entering long mode:

; boot/src/paging.asm
section .text
bits 32

global setup_page_tables

setup_page_tables:
    ; Map P4[0] -> P3
    mov eax, p3_table
    or eax, 0b11         ; Present + writable
    mov [p4_table], eax

    ; Map P3[0] -> P2
    mov eax, p2_table
    or eax, 0b11         ; Present + writable
    mov [p3_table], eax

    ; Map P2 entries (identity map first 1GB with 2MB pages)
    mov ecx, 0

.map_p2_table:
    mov eax, 0x200000   ; 2MB
    mul ecx
    or eax, 0b10000011  ; Present + writable + huge page
    mov [p2_table + ecx * 8], eax
    inc ecx
    cmp ecx, 512
    jne .map_p2_table

    ret

Identity Mapping

NoOS uses identity mapping where virtual addresses equal physical addresses. The first 1GB of memory is mapped using 512 entries of 2MB huge pages (0b10000011 flags = Present + Writable + Huge). This eliminates address translation complexity during boot while enabling the paging hardware required for long mode.

VGA Text Mode Driver

NoOS includes a VGA text mode driver that writes directly to the video memory at address 0xB8000. Each character cell consists of two bytes:

Byte 0: ASCII character code
Byte 1: Attribute byte (foreground/background colors)

The color attribute byte format:

Bits 0-3: Foreground color (0-15)
Bits 4-6: Background color (0-7)
Bit 7:    Blink enable

Color Palette

pub enum Color {
    Black = 0,
    Blue = 1,
    Green = 2,
    Cyan = 3,
    Red = 4,
    Magenta = 5,
    Brown = 6,
    LightGray = 7,
    DarkGray = 8,
    LightBlue = 9,
    LightGreen = 10,
    LightCyan = 11,
    LightRed = 12,
    Pink = 13,
    Yellow = 14,
    White = 15,
}

Rust’s Role in Kernel Development

The no_std Environment

Operating system kernels can’t rely on the standard library since there’s no operating system underneath to provide services. NoOS uses the x86_64 crate for hardware abstractions:

// kernel/src/idt.rs - Interrupt Descriptor Table setup
use x86_64::structures::idt::InterruptDescriptorTable;
use lazy_static::lazy_static;

use crate::drivers::keyboard::keyboard_interrupt_handler;

lazy_static! {
    static ref IDT: InterruptDescriptorTable = {
        let mut idt = InterruptDescriptorTable::new();
        idt[0x21].set_handler_fn(keyboard_interrupt_handler);
        idt
    };
}

pub fn init() {
    IDT.load();
}

The PIC (Programmable Interrupt Controller) is managed in Rust:

// kernel/src/pic.rs
use x86_64::instructions::port::Port;

pub fn send_eoi(irq: u8) {
    unsafe {
        if irq >= 8 { Port::new(0xA0).write(0x20u8); }  // slave EOI
        Port::new(0x20).write(0x20u8);                  // master EOI
    }
}

Memory Safety Without Runtime

Rust’s ownership model provides memory safety guarantees at compile time, eliminating the need for a garbage collector or runtime checks. This is perfect for kernel development where:

• No allocator: The kernel manages memory directly
• No runtime overhead: Zero-cost abstractions compile away
• Compile-time verification: Buffer overflows and use-after-free bugs are caught before the kernel runs

Project Structure

noos/
├── kernel/
│   ├── src/
│   │   ├── lib.rs           # Kernel entry point and panic handler
│   │   ├── multiboot.rs     # Multiboot2 info parsing
│   │   ├── idt.rs           # Interrupt Descriptor Table
│   │   ├── pic.rs           # Programmable Interrupt Controller
│   │   ├── shell.rs         # Interactive shell
│   │   └── drivers/
│   │       ├── mod.rs       # Driver module exports
│   │       ├── vga.rs       # VGA text mode driver
│   │       ├── keyboard.rs  # PS/2 keyboard driver
│   │       └── serial.rs    # Serial port driver
│   ├── .cargo/
│   │   └── config.toml      # Build configuration for no_std target
│   └── Cargo.toml           # Dependencies: spin, lazy_static, x86_64
├── boot/
│   ├── src/
│   │   ├── boot.asm         # Main bootstrap assembly
│   │   ├── multiboot2.asm   # Multiboot2 header
│   │   ├── long_mode.asm    # CPU mode transition
│   │   ├── paging.asm       # Page table setup
│   │   ├── gdt.asm          # Global Descriptor Table
│   │   ├── extern.asm       # External symbols (hlt)
│   │   ├── utils/
│   │   │   └── print.asm    # Error printing
│   │   └── pic/
│   │       ├── remap.asm    # PIC remapping
│   │       └── keyboard.asm # Keyboard IRQ unmasking
│   └── grub.cfg             # GRUB menu configuration
├── config/
│   └── linker.ld            # Custom linker script (kernel @ 1MB)
└── Makefile                 # Complete build automation

Linker Script

The linker script controls the kernel’s memory layout with proper segment permissions:

/* config/linker.ld */
PHDRS {
    rodata PT_LOAD FLAGS(4);    /* read-only */
    text PT_LOAD FLAGS(5);      /* read + execute */
    data PT_LOAD FLAGS(6);      /* read + write */
}

ENTRY(_start)

SECTIONS {
    . = 1M;

    .boot : AT(0x100000) {
        KEEP(*(.multiboot_header))
    } :text

    .text : AT(ADDR(.text)) {
        *(.text .text.*)
    } :text

    .rodata : AT(ADDR(.rodata)) {
        *(.rodata .rodata.*)
    } :rodata

    .data : AT(ADDR(.data)) {
        *(.data .data.*)
    } :data

    .bss : AT(ADDR(.bss)) {
        *(.bss .bss.*)
    } :data

    /DISCARD/ : {
        *(.eh_frame)
        *(.comment)
    }
}

Building and Running

Prerequisites

NoOS requires several tools:

• Rust nightly toolchain with rust-src component
• NASM assembler for bootstrap code
• GRUB2 utilities for bootable image creation
• QEMU for testing

Build Process

# Install dependencies and Rust toolchain
make setup

# Build kernel binary
make kernel

# Create bootable ISO
make iso

# Build and launch in QEMU
make run

# Clean build artifacts
make clean

Challenges and Solutions

Challenge: Long Mode Transition

The CPU starts in 32-bit protected mode, but Rust code expects a 64-bit environment. The solution is assembly bootstrap code that:

1. Sets up identity-mapped page tables
2. Enables PAE and long mode
3. Loads a 64-bit GDT
4. Far jumps to 64-bit code

Challenge: GDT for 64-bit Mode

A Global Descriptor Table is required for the 64-bit far jump:

; boot/src/gdt.asm
section .rodata

gdt64:
    dq 0  ; Zero entry

.code: equ $ - gdt64
    dq (1<<43) | (1<<44) | (1<<47) | (1<<53)  ; Code segment

.pointer:
    dw $ - gdt64 - 1
    dq gdt64

The code segment descriptor sets:

• Bit 43: Executable
• Bit 44: Code/data segment
• Bit 47: Present
• Bit 53: 64-bit mode

Challenge: Keyboard Input

Handling keyboard interrupts requires careful synchronization. NoOS uses a scancode queue with spinlocks:

// kernel/src/drivers/keyboard.rs
use pc_keyboard::{Keyboard, layouts, ScancodeSet1, DecodedKey, HandleControl};
use x86_64::instructions::{port::Port, interrupts::without_interrupts};
use x86_64::structures::idt::InterruptStackFrame;
use heapless::spsc::Queue;
use spin::Mutex;

use crate::pic::send_eoi;

lazy_static::lazy_static! {
    pub static ref SCANCODE_QUEUE: Mutex<Queue<u8, 1024>> = Mutex::new(Queue::new());
}

pub struct KeyboardDriver {
    keyboard: Mutex<Keyboard<layouts::Us104Key, ScancodeSet1>>,
}

impl KeyboardDriver {
    pub const fn new() -> Self {
        KeyboardDriver { 
            keyboard: Mutex::new(Keyboard::new(
                ScancodeSet1::new(), 
                layouts::Us104Key, 
                HandleControl::MapLettersToUnicode
            )) 
        }
    }

    fn read_scancode(&self) -> u8 {
        let mut port = Port::new(0x60);
        unsafe { port.read() }
    }
}

pub extern "x86-interrupt" fn keyboard_interrupt_handler(_stack_frame: InterruptStackFrame) {
    let scancode = KEYBOARD.read_scancode();
    
    if let Some(mut queue) = SCANCODE_QUEUE.try_lock() {
        if queue.enqueue(scancode).is_err() {
            queue.dequeue();
            queue.enqueue(scancode).ok();
        }
    }
    
    send_eoi(1);
}

Current Features & Future Directions

NoOS already includes:

• Interrupt Handling: Full IDT setup with keyboard IRQ
• Keyboard Driver: PS/2 keyboard with scancode queue
• Interactive Shell: Command parsing and execution framework
• Serial Driver: COM1 output support

Planned extensions:

• Memory Allocator: Heap allocation for dynamic data structures
• Filesystem: Simple ramdisk or ext2 support
• Process Management: Process scheduler, context switching
• System Calls: User-space/kernel-space boundary

Educational Value

NoOS demonstrates several important concepts:

1. Boot Process: How computers transition from power-on to running code
2. CPU Modes: Real mode → Protected mode → Long mode transitions
3. Memory Layout: How kernels organize themselves in memory
4. Hardware Abstraction: Direct hardware access vs. OS-provided abstractions
5. Language Safety: How Rust’s guarantees apply to systems programming

Conclusion

NoOS proves that operating system development doesn’t have to be mysterious or unsafe. By using Rust’s modern tooling and safety features, the project creates a clear educational path into the world of systems programming.

The kernel demonstrates that memory safety and low-level control aren’t mutually exclusive. Rust’s ownership model catches bugs at compile time that would be runtime crashes in C, while still allowing the precise hardware manipulation that OS development requires.

Whether you’re curious about how computers boot, interested in Rust’s capabilities for systems programming, or want to understand the foundations that all software builds upon, NoOS provides a clear and safe starting point.

Explore the source code, study the boot sequence, and discover how a few hundred lines of Rust and assembly can bring a computer to life.