對于MongoDB的存儲基本單元BSON文檔對象,字段值可以是二進制類型,基于此特點,我們可以直接在MongoDB中存儲文件,但是有一個限制,由于MongoDB中單個BSON對象不能大于16MB,故而如果需要存儲更大的文件,就需要GridFS了。
我們先看下MongoDB存儲小文件系統的例子:
先采用MongoDB的mongofiles執行文件上傳:
D:/MongoDB/Server/3.2/bin>mongofiles.exe list2017-03-06T13:41:03.283+0800 connected to: localhostD:/MongoDB/Server/3.2/bin>mongofiles.exe put E:/deliveryTask.doc2017-03-06T13:41:23.535+0800 connected to: localhostadded file: E:/deliveryTask.docD:/MongoDB/Server/3.2/bin>mongofiles.exe list2017-03-06T13:41:30.114+0800 connected to: localhostE:/deliveryTask.doc 2971通過mongos命令查看文件存儲情況:
> use testswitched to db test> show collectionsfs.chunksfs.filesrestaurantsuser> db.fs.files.find(){ "_id" : ObjectId("58bcf683afa0fa20bc854a2b"), "chunkSize" : 261120, "uploadDate" : ISODate("2017-03-06T05:41:23.604Z"), "length" : 2971, "md5" : "5434b803306299fff57c8a54d3adf78b", "filename" : "E://deliveryTask.doc" }可以看到文件上傳成功了
由于本章節主要涉及到的是部分理論以及運維實踐,暫不涉及具體代碼的開發實現(具體代碼將以java為例在后面章節中介紹)。
上傳一個大于16MB的文件試一試:
D:/MongoDB/Server/3.2/bin>mongofiles.exe put E:/synch.rar2017-03-06T14:33:11.028+0800 connected to: localhostadded file: E:/synch.rarD:/MongoDB/Server/3.2/bin>mongofiles.exe list2017-03-06T14:33:15.265+0800 connected to: localhostE:/deliveryTask.doc 2971E:/synch.rar 24183487通過mongos命令查看文件存儲情況:
> db.fs.files.find(){ "_id" : ObjectId("58bcf683afa0fa20bc854a2b"), "chunkSize" : 261120, "uploadDate" : ISODate("2017-03-06T05:41:23.604Z"), "length" : 2971, "md5" : "5434b803306299fff57c8a54d3adf78b", "filename" : "E://deliveryTask.doc" }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "chunkSize" : 261120, "uploadDate" : ISODate("2017-03-06T06:33:12.013Z"), "length" : 24183487, "md5" : "bbfe4d8579372aa0729726185997e908", "filename" : "E://synch.rar" }也成功了,
查看chunks:
> db.fs.chunks.find({},{data:0}){ "_id" : ObjectId("58bcf683afa0fa20bc854a2c"), "files_id" : ObjectId("58bcf683afa0fa20bc854a2b"), "n" : 0 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b2d"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 0 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b2e"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 1 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b2f"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 2 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b30"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 3 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b31"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 4 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b32"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 5 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b33"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 6 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b34"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 7 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b35"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 8 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b36"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 9 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b37"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 10 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b38"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 11 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b39"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 12 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b3a"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 13 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b3c"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 15 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b3b"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 14 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b3e"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 17 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b3d"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 16 }{ "_id" : ObjectId("58bd02a7afa0fa21d4a14b3f"), "files_id" : ObjectId("58bd02a7afa0fa21d4a14b2c"), "n" : 18 }Type "it" for more可以看到大文件被分成了好多個chunk,那么為什么上傳16MB以上的文件也會成功呢,因為我們采用的就是GridFS系統存儲的文件,因為我們采用的是mongfiles方式上傳的文件。
下面分別是查詢、下載、刪除操作:
D:/MongoDB/Server/3.2/bin>mongofiles.exe search rar2017-03-06T14:45:31.974+0800 connected to: localhostE:/synch.rar 24183487D:/MongoDB/Server/3.2/bin>mongofiles.exe --local D:/mongodb_download.rar get E:/synch.rar2017-03-06T14:47:17.841+0800 connected to: localhostfinished writing to D:/mongodb_download.rarD:/MongoDB/Server/3.2/bin>mongofiles.exe delete E:/synch.rar2017-03-06T14:47:56.649+0800 connected to: localhostsuccessfully deleted all instances of 'E:/synch.rar' from GridFSD:/MongoDB/Server/3.2/bin>mongofiles.exe list2017-03-06T14:48:03.886+0800 connected to: localhostE:/deliveryTask.doc 2971實際上,我們還可以自定義集合的前綴,默認是fs,或者設置chunk的大小,默認是256KB。
那么在實際場景的分布式文件存儲系統中如何確定改用哪種存儲方案呢,可以采用如下方式: 1. 對于用戶上傳的任何文件,在客戶端進行大小判斷; 2. 當文件大小小于16MB時,則直接存儲到MOngoDB普通集合中 3. 當文件大小大于16MB時,上傳到GridFS中,利用集合fs.files以及fs.chunks來保存文件 4. 當用戶下載文件時,再根據不同文件的大小屬性不同到不同的集合中查找
另外,對于fs.chunks文件我們可以分片存儲,片鍵可以選擇索引字段{“files_id”},該字段盡量保證了此文件在所有被分割的chunk都在同一個片上,fs.files不需要分片,此集合只保存文件的元數據信息,數據量不大,同時還可以設置默認塊大?。?56KB)
需要注意的是:GridFS并不適合小文件存儲,因為從GridFS中讀取文件涉及到兩次查詢操作,先查詢fs.files集合,再查詢fs.chunks集合,chunks合并后獲取整個文件。
需要注意的另一點是:文件分塊大小為256KB,而分片的塊大小默認64MB,不要搞混了。
新聞熱點
疑難解答