iSCSI-SCST Storage Server Usermode Adaptation
David A. Butterfield — February 2017
| Abstract | This paper describes an adaptation of the iSCSI-SCST storage server software to run entirely in usermode on an unmodified Linux kernel; performance measurements and model; and an experimental algorithm to improve performance for small Read operations. The Appendix lists a few issues discovered in the SCST source code. | |
|---|---|---|
| In a standard installation of SCST the iscsi-scstd daemon runs as a single-threaded Linux usermode process that cooperates with the kernel-resident SCST implementation using ioctl(2) and netlink(7) for communication. | ||
| In the iSCSI-SCST Usermode Adaptation the iscsi-scstd daemon runs on the main thread in a multi-threaded process in which other usermode threads are concurrently providing the services and executing the SCST code that would be running inside the kernel in a standard installation of SCST. | ||
| The subset of SCST used includes the SCST Core, the iSCSI daemon and kernel logic, the vdisk device, and the /proc interface; comprising about 80,000 lines of code. To support running in usermode, around 55 (fifty-five) lines of executable C code have been added or changed under #ifdef in SCST source files. | ||
| For a single session over 1 Gb Ethernet being serviced by a single 2.4 GHz CPU: the described Adaptive Nagle optimization improves peak throughput performance for 512-Byte Random Read of /dev/zero from around 63,000 IOPS to more than 100,000 IOPS, with no adverse impact below Queue Depth 17. |
| Contents | 1 | Introduction | |
|---|---|---|---|
| 2 | Motivation | ||
| 2 | Implemented Functionality | ||
| 2 | Implementation Overview | ||
| 4 | Threading | ||
| 5 | CPU Assignments | ||
| 6 | Performance Measurements | ||
| 8 | Experimental Performance Improvement for Small Read Sizes | ||
| 17 | Performance Model | ||
| 35 | SCST Issues Discovered (Summary) | ||
| 35 | Sample top(1) Display | ||
| 36 | Sample Content from /fuse/scst/proc/scsi_tgt | ||
| 37 | Sample Logging (Startup, before scstadmin) | ||
| 37 | Sample Logging (Startup, during scstadmin) | ||
| 38 | Sample Logging (Startup, after scstadmin) | ||
| 38 | Sample Logging (Shutdown, SIGINT-->exit) | ||
| 39 | Initiator and Server Machine Settings | ||
| 40 | Appendix: SCST Issues Discovered |
| Paper (PDF) | 1 | Part 1: Adaptation and Performance Measurements | |
|---|---|---|---|
| 17 | Part 2: Performance Model | ||
| 35 | Part 3: Displays, Logs, Listings | ||
| CODE |
| About the Author |
|---|
| David Butterfield [david.butterfield@acm.org] began programming for usermode in 2008 after a prior gigasecond or so working on software in various versions of the Unix and Solaris kernels (or without any kernel). He holds an MSc in Computer Science from UCLA, where his undergraduate degree was in Mathematics and Computer Science. |
| One of the founders of Locus Computing Corporation, he designed the first Virtual Machine Monitor for x86, to "Merge" MS-DOS and its applications under Unix SVR2; and led an engineering team in its implementation. The OS-Merge product was first marketed by AT&T under the name "Simultask" on their 6300+ (IBM AT clone) model. [Sometime after his involvement that product evolved into two descendant products known as NeTraverse Merge and Win4Lin] |
| He joined Sun Microsystems to establish and lead the first Solaris x86 device driver development team, later accepting an international assignment to Dublin, Ireland to start another driver engineering team there. |
| Back in the U.S., at LeftHand Networks he contributed many performance improvements to the SAN/iQ event-driven distributed storage application, introducing application-transparent multi-threading into the existing single-threaded event framework and devising other optimizations amounting in total to a 2.5x increase in throughput (IOPS) capability. Some of his diagrams were said to inspire awe. |
| His most recent (unpaid) project is described in this paper. He is presently looking for an interesting (well-paid) project. |