{"@context":"http://iiif.io/api/presentation/2/context.json","@id":"https://repo.library.stonybrook.edu/cantaloupe/iiif/2/manifest.json","@type":"sc:Manifest","label":"Scalable End-to-End Data I/O over Enterprise and Data-Center Networks","metadata":[{"label":"dc.description.sponsorship","value":"This work is sponsored by the Stony Brook University Graduate School in compliance with the requirements for completion of degree."},{"label":"dc.format","value":"Monograph"},{"label":"dc.format.medium","value":"Electronic Resource"},{"label":"dc.identifier.uri","value":"http://hdl.handle.net/11401/77230"},{"label":"dc.language.iso","value":"en_US"},{"label":"dc.publisher","value":"The Graduate School, Stony Brook University: Stony Brook, NY."},{"label":"dcterms.abstract","value":"Data-intensive applications in commercial cloud and scientific computing demand ultra-high-speed end-to-end data access capability between data storage and computing locations. Meanwhile, advancements in hardware systems continuously change the landscape of data centers\u00e2\u20ac\u2122 core capabilities, i.e., computing, networking, and storage.  The two trends expose new research and development challenges and opportunities to bring the bare-metal capacity and performance of state-of-the-art hardware to the rising needs for high performance by applications. Simply deploying and tuning existing software and services on state-of-the-art platforms does not necessarily ensure expected performance due to the overhead on data-copy in OS kernel functions and conservative network protocol. We adopt holistic approach, from the ground up, to reconsider network protocol, storage management, and software architecture, and align them with the new hardware characteristics to better orchestrate system resources.  This is extremely challenging in several aspects. First, we can not rely on existing data-copy based OS libraries and network protocols. Secondly, simple synchronous sequential programming paradigm and network protocol becomes barrier to system performance. Therefore, the new design should follow more complex asynchronous parallel model and carefully investigate the tradeoff between the programming overheads and performance improvement. Two major objectives are the focus of this work: designing user-level end-to-end protocol and software to coordinate data movements and to bypass OS kernels, and scaling storage caching system performance in multi-core environments with large scale asymmetric memory layout (NUMA).  We design and build real systems to achieve these objectives. First, to fully utilize hardware capabilities, we propose an asynchronous memory-centric software framework for high throughput data-intensive applications. We implement a zero-copy data movement protocol using asynchronous Remote Direct Memory Access (RDMA) to maximize the parallelism of data transmission. The design achieves 97% network hardware bandwidth. Our software achieves more than 2X speedup over the existing solutions (for example, GridFTP) in replicating data across the entire storage-to-storage path. Secondly, we design and implement a NUMA-aware caching system for storage area networks that optimizes in-memory data locality on serving raw storage blocks. We further improve its performance with our decentralized and parallel event processing framework. The data locality awareness provides up to 80% throughput improvement on large scale memory caching system; The decentralized event processing shows its linear scalability with the number of threads on multi-core systems. The unprecedented data volume and the continuing trend of adopting cloud computing and storage by industry and consumers give rise to the pressing need for efficient software design and network protocol to distribute and replicate data over high performance networks.  Our research focuses on the need and proposes a scalable framework to manage and coordinate multi-/many-core computing, deep hierarchy storage and high speed network in a coherent way.  Therefore, this framework enables in-situ data retrieval, checksum calculation, encryption, and transmission to address the growing concerns of data privacy, security, integrity and on-demand delivery in the cloud era. It paves the path to harness multi-core/many-core architecture, i.e., GPGPU and Intel Coprocessor, by accelerating data I/O, to replace outdated software that was designed for traditional rotary magnetic disks, and to enhance the IOPS throughput of storage system to incorporate newly-emerging Solid State Drives (SSD) and Non-volatile random-access memory (NVRAM)."},{"label":"dcterms.available","value":"2017-09-20T16:52:14Z"},{"label":"dcterms.contributor","value":"Yu, Dantong"},{"label":"dcterms.creator","value":"Ren, Yufei"},{"label":"dcterms.dateAccepted","value":"2017-09-20T16:52:14Z"},{"label":"dcterms.dateSubmitted","value":"2017-09-20T16:52:14Z"},{"label":"dcterms.description","value":"Department of Computer Engineering."},{"label":"dcterms.extent","value":"167 pg."},{"label":"dcterms.format","value":"Monograph"},{"label":"dcterms.identifier","value":"http://hdl.handle.net/11401/77230"},{"label":"dcterms.issued","value":"2015-12-01"},{"label":"dcterms.language","value":"en_US"},{"label":"dcterms.provenance","value":"Made available in DSpace on 2017-09-20T16:52:14Z (GMT). No. of bitstreams: 1\nRen_grad.sunysb_0771E_12543.pdf: 1982962 bytes, checksum: 16bf6a1b2d99809e1be7c7c4a7bd5b1e (MD5)\n  Previous issue date:    1"},{"label":"dcterms.publisher","value":"The Graduate School, Stony Brook University: Stony Brook, NY."},{"label":"dcterms.subject","value":"cache, data transfer, I/O systems, NUMA, protocol design, RDMA"},{"label":"dcterms.title","value":"Scalable End-to-End Data I/O over Enterprise and Data-Center Networks"},{"label":"dcterms.type","value":"Dissertation"},{"label":"dc.type","value":"Dissertation"}],"description":"This manifest was generated dynamically","viewingDirection":"left-to-right","sequences":[{"@type":"sc:Sequence","canvases":[{"@id":"https://repo.library.stonybrook.edu/cantaloupe/iiif/2/canvas/page-1.json","@type":"sc:Canvas","label":"Page 1","height":1650,"width":1275,"images":[{"@type":"oa:Annotation","motivation":"sc:painting","resource":{"@id":"https://repo.library.stonybrook.edu/cantaloupe/iiif/2/35%2F68%2F81%2F35688181937395506115123695459700166409/full/full/0/default.jpg","@type":"dctypes:Image","format":"image/jpeg","height":1650,"width":1275,"service":{"@context":"http://iiif.io/api/image/2/context.json","@id":"https://repo.library.stonybrook.edu/cantaloupe/iiif/2/35%2F68%2F81%2F35688181937395506115123695459700166409","profile":"http://iiif.io/api/image/2/level2.json"}},"on":"https://repo.library.stonybrook.edu/cantaloupe/iiif/2/canvas/page-1.json"}]}]}]}