SCST-Usermode-Adaptation

SCST iSCSI Storage Server Usermode Adaptation
An adaptation of the iSCSI-SCST storage server software to run entirely in usermode on an unmodified Linux kernel
David A. Butterfield

This project adapts the SCST iSCSI storage server software, which normally resides in the Linux kernel, to run entirely in usermode on an unmodified kernel. The resulting executable can run as a regular (non-super) user, as long as it has permission to access the backing storage. A paper describing the work and performance study is here.

The master branch contains the original work described in the paper.

Branches:

master - the original work described by the paper, using local files or block devices
usermode - updated work supporting tcmu-runner backstore handlers in addition to local files and block devices

The adaptation uses about 80,000 lines of the SCST source code, a subset supporting the iSCSI transport type (via socket calls), and SCSI Block Commands (vdisk_fileio) backed by either a file or a block device.

The SCST iSCSI Usermode Adaptation depends on

Usermode Compatibility (UMC) — a shim for running some Linux kernel code in usermode
Multithreaded Engine (MTE) — a high-performance multi-threaded event dispatching engine for usermode
SCST files in the master branch began as a snapshot of https://sourceforge.net/projects/scst /scst and /iscsi-scst svn -r7105.
To support usermode, about 300 lines of changes were made to the SCST snapshot files (mostly under #ifdef)
Additional changes: AIO, optimizations for usermode (under #ifdef), and some fixes and XXX comments for SCST
A little more work would be required to run on architectures other than x86
Possibly less work would be needed to run on non-Linux POSIX systems having gcc and the libraries
It shouldn’t matter much, but I have only tested with these:
- Linux 3.13.0-101-generic #148-Ubuntu SMP x86_64
- Linux 4.4.0-70-generic #91-Ubuntu SMP x86_64 GNU/Linux
- Linux 4.8.0-46-generic #49-Ubuntu SMP x86_64 GNU/Linux

Hints to help get started running iSCSI-SCST in usermode (This is rather terse; some existing familiarity with SCST will be helpful)

  # apt install libaio-dev              # required
  # apt install libfuse-dev             # required
  # apt install subversion              # or github accessor of your choice
  # apt install cscope                  # (optional with makefile edit)
  # apt install exuberant-ctags         # (optional with makefile edit)
  # apt install valgrind                # (optional)

  $ mkdir Usermode_SCST ; cd Usermode_SCST   # or use whatever name you want for this one
  $ svn co https://github.com/DavidButterfield/MTE.git MTE   # Makefile expects these names
  $ svn co https://github.com/DavidButterfield/usermode_compat.git UMC
  $ svn co https://github.com/DavidButterfield/SCST-Usermode-Adaptation.git SCST

  $ more SCST/trunk/usermode/BUGS.txt   # ** MORE HINTS **

  $ pushd MTE/trunk/src                 # build the Multithreaded Engine library
  $ make
  $ sudo make install                   # needs permission for /lib, /usr/include
  $ popd

  $ cd SCST/trunk                       # if you use git, ignore the "/trunk" part
  $ sudo make scstadm_install           # patched to know where /fuse/scst/proc is
  $ make enable_proc                    # configure so it can compile for usermode

  $ cd usermode
  $ make                                # build the SCST iSCSI server binary
  $ ls -l scst.out                      # in SCST/trunk/usermode/

  ### Manually create /etc/{iscsi-scst,scst_usermode}.conf in SCST /proc format (See *.sample)

  # mkdir -p  /var/lib/scst/vdev_mode_pages /var/lib/scst/pr
  # chmod 777 /var/lib/scst/vdev_mode_pages /var/lib/scst/pr   # or writable by SCST's UID

  # mkdir -p /fuse/scst/proc ; chmod 777 /fuse/scst/proc       # mount point for SCST's /proc
  ### Edit /etc/fuse.conf and uncomment the line with "user_allow_other"

  $ [ gdb | valgrind ] ./scst.out -f    # run as normal user, with or without accessories In another terminal window

  # scstadmin -config /etc/scst_usermode.conf
  # ls -l `find /fuse -type f` #### Diagrams showing the relationship between UMC, MTE, and Usermode SCST * * * ![SCST usermode service map](https://davidbutterfield.github.io/SCST-Usermode-Adaptation/docs/SCST_usermode_service_map.png  "SCST Usermode Service Map") * * * ![SCST usermode header and library inclusions](https://davidbutterfield.github.io/SCST-Usermode-Adaptation/docs/SCST_usermode_includes.png "SCST Usermode Header and Library Inclusions") * * * **Diagram showing the datapath of SCST configured with iSCSI and vdisk_fileio** (either usermode or kernel-resident) * * * ![SCST datapath](https://davidbutterfield.github.io/SCST-Usermode-Adaptation/docs/SCST_iSCSI_datapath.png  "SCST Usermode Service Map") * * *