前言 已经实施过 2 次 Hadoop 大数据平台的基础环境搭建,对于硬盘的 RAID 模式都不约而同选择了 JBOD,毕竟大数据对于文件系统的管理本身就是基于分布式,由至少 3 份副本来保障数据可靠性,只是不做 RAID 后的 SAS 硬盘读写性能就有点惨不忍睹。这次 Hadoop 大数据平台工具选择的是 Cloudera,大家感兴趣的化也可以浏览下它的官网。本篇文章主要介绍如何实现在不停机状态下在线更换 JBOD 硬盘。
利用 StorCLI 命令不停机在线更换 JBOD 硬盘
更新历史 2016 年 12 月 31 日 - 初稿
阅读原文 - https://liaojiaxin158.github.io/post/jbod/
扩展阅读
Cloudera - http://www.cloudera.com/
JBOD 原理 JBOD ( Just a Bunch Of Disks )不是标准的 RAID 等级,它通常用来表示一个没有控制软件提供协调控制的磁盘集合。 JBOD 将多个物理磁盘串联起来,提供一个巨大的逻辑磁盘。 JBOD 的数据存放机制是由第一块磁盘开始按顺序往后存储,当前磁盘存储空间用完后,再依次往后面的磁盘存储数据。 JBOD 存储性能完全等同于单块磁盘,而且也不提供数据安全保护。它只是简单提供一种扩展存储空间的机制, JBOD 可用存储容量等于所有成员磁盘的存储空间之和。目前 JBOD 常指磁盘柜,而不论其是否提供 RAID 功能。
RAID 磁盘阵列配置和调优小结 - https://liaojiaxin158.github.io/post/raid/
参考文档和工具 StorCLI_RefMan_revfhttps://www.broadcom.com/support/download-search/?dk=StorCLI
StorCLI (Command Line) Utility for Storage Management v1.07.07 on Linuxhttps://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5092950
使用方式 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 [root@localhost Linux] Attached devices: Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: IBM Model: ServeRAID M5210 Rev: 4.62 Type: Direct-Access ANSI SCSI revision: 05 unzip ibm_utl_sraidmr_storcli-1.07.07_linux_32-64.zip rpm -Uvh storcli-1.07.07-1.noarch.rpm ln -s /opt/MegaRAID/storcli/storcli64 /bin/storcli ln -s /opt/MegaRAID/storcli/storcli64 /sbin/storcli [root@localhost Linux] Status Code = 0 Status = Success Description = None Controller Count = 2 [root@localhost Linux] Controller = 0 Status = Success Description = None Virtual Drives : ============== ----------------------------------------------------------- DG/VD TYPE State Access Consist Cache sCC Size Name ----------------------------------------------------------- 0/0 RAID1 Optl RW Yes RWTD - 278.465 GB ----------------------------------------------------------- Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read Write|B=Blocked|Consist=Consistent| R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled Check Consistency [root@localhost Linux] Controller = 0 Status = Success Description = Show Drive Information Succeeded. Drive Information : ================= ------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp ------------------------------------------------------------------------- 252:16 8 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U 252:17 9 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U ------------------------------------------------------------------------- EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded CFShld-Configured shielded [root@localhost Linux] Controller = 1 Status = Success Description = Show Drive Information Succeeded. Drive Information : ================= ------------------------------------------------------------------------ EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp ------------------------------------------------------------------------ 8:0 23 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:1 24 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:2 22 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:3 19 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:4 13 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:5 14 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:6 18 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:7 12 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:8 10 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:9 16 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:10 11 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:11 21 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:12 15 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U 8:13 9 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U ------------------------------------------------------------------------ EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded CFShld-Configured shielded storcli /c1/e8/s0 show all storcli /c1/e8/s0 start locate storcli /c1/e8/s0 set offline storcli /c1/e8/s0 set online storcli /c1/e8/s0 set jbod
StorCLI Help 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 [root@localhost Linux] Storage Command Line Tool Ver 1.07.07 Nov 15, 2013 (c)Copyright 2013, LSI Corporation, All Rights Reserved. storcli -v storcli -h| -help | ? storcli -h| -help | ? legacy storcli show storcli show all storcli show ctrlcount storcli /cx add vd r[0|1|5|6|00|10|50|60] [Size=<VD1_Sz>,<VD2_Sz>,..|all] [name=<VDNAME1>,..] drives=e:s|e:s-x|e:s-x,y,e:s-x,y,z [PDperArray=x][SED] [pdcache=on|off|default][pi][DimmerSwitch(ds)=default|automatic(auto)| none|maximum(max)|MaximumWithoutCaching(maxnocache)] [wt|wb|awb][nora|ra] [direct|cached] [cachevd] [Strip=<8|16|32|64|128|256|512|1024>] [AfterVd=X] [Spares = [e:]s|[e:]s-x|[e:]s-x,y] [force][ExclusiveAccess] storcli /cx add vd each r0 [name=<VDNAME1>,..] [drives=e:s|e:s-x|e:s-x,y] [SED] [pdcache=on|off|default][pi] [DimmerSwitch(ds)=default|automatic(auto)| none|maximum(max)|MaximumWithoutCaching(maxnocache)] [wt|wb|awb] [nora|ra] [direct|cached] [Strip=<8|16|32|64|128|256|512|1024>] [ExclusiveAccess] storcli /cx add VD cachecade|nytrocache r[aid][0,1,10, 1EC] drives = [e:]s|[e:]s-x|[e:]s-x,y [WT| WB] [assignvds = 0,1,2] [BOOTVOLSIZE=x] storcli /cx/ex show all storcli /cx/ex show status storcli /cx/vx del [cachecade] [force] storcli /cx delete events storcli /cx show events [[type = <sincereboot| sinceshutdown| includedeleted| latest=x| ccincon vd=<0,1,...>] filter=<info|warning|critical|fatal>] file=<filepath> storcli /cx show eventloginfo storcli /cx show health [all] storcli /cx delete securitykey storcli /cx set securitykey=xxxxxxxx {passphrase=xxxx} {keyid=xxx} storcli /cx start Diag Duration=<Val> storcli /cx set securitykey keyid=xxx storcli /cx compare securitykey=xxxxxxxxxx storcli /cx set termlog[=on|off|offthisboot] storcli /cx show termlog [type =config|contents] storcli /cx delete termlog storcli /cx Compare factorysettings file=<filepath> storcli /cx Compare factorysettings raw file=<filepath> storcli /cx set securitykey=xxxxxxxx oldsecuritykey=xxxxxxxx {passphrase=xxxx} {keyid=xxx} storcli /cx/dx show storcli /cx/dall show cachecade storcli /cx/dx show all storcli /cx show freespace storcli /cx/fall show [all] [securityKey = xxx] storcli /cx/fall del|delete [securityKey = xxx] storcli /cx/fall import [preview] [securityKey = xxx] storcli /cx/vx set ssdcaching=on|off storcli /cx/dx set security=on storcli /cx/vx show expansion storcli /cx show fshinting storcli /cx/vx set fshinting=<value> storcli /cx/vx expand Size=<xx> [expandarray] storcli /cx[/ex]/sx show storcli /cx[/ex]/sx show all storcli /cx[/ex]/sx start rebuild storcli /cx[/ex]/sx stop rebuild storcli /cx[/ex]/sx pause rebuild storcli /cx[/ex]/sx resume rebuild storcli /cx[/ex]/sx show rebuild storcli /cx[/ex]/sx show health storcli /cx[/ex]/sx start copyback target=e:s storcli /cx[/ex]/sx stop copyback storcli /cx[/ex]/sx pause copyback storcli /cx[/ex]/sx resume copyback storcli /cx[/ex]/sx show copyback storcli /cx[/ex]/sx start initialization storcli /cx[/ex]/sx stop initialization storcli /cx[/ex]/sx show initialization storcli /cx[/ex]/sx start locate storcli /cx[/ex]/sx stop locate storcli /cx[/ex]/sx show securitykey keyid storcli /cx[/ex]/sx add hotsparedrive [DGs=<N|0,1,2...>] [enclaffinity] [nonrevertible] storcli /cx[/ex]/sx delete hotsparedrive storcli /cx[/ex]/sx spinup storcli /cx[/ex]/sx spindown storcli /cx[/ex]/sx set online storcli /cx[/ex]/sx set offline storcli /cx[/ex]/sx set missing storcli /cx[/ex]/sx set jbod storcli /cx[/ex]/sx set good [force] storcli /cx[/ex]/sx insert dg=A array=B row=C storcli /cx/vx set emulationType=0|1 storcli /cx/vx set wrcache=WT|WB|AWB storcli /cx/vx set rdcache=RA|NoRA storcli /cx/vx set iopolicy=Cached|Direct storcli /cx/vx set accesspolicy=RW|RO|Blocked|RmvBlkd storcli /cx/vx set pdcache=On|Off|Default storcli /cx/vx set name=<NameString> storcli /cx/vx set HostAccess=ExclusiveAccess|SharedAccess storcli /cx/vx set ds=Default|Auto|None|Max|MaxNoCache storcli /cx/vx set autobgi=On|Off storcli /cx/vx set pi=Off storcli /cx/vx show storcli /cx/vx show all storcli /cx/vx show init storcli /cx/vx show cc storcli /cx/vx show erase storcli /cx/vx show migrate storcli /cx/vx show bgi storcli /cx/vx show autobgi storcli /cx/vx show trim storcli /cx set consistencycheck|cc=[off|seq|conc] [delay=value] starttime=yyyy/mm/dd hh] [excludevd=x-y,z] storcli /cx show cc|consistencycheck storcli /cx show ocr storcli /cx set ocr=<on|off> storcli /cx/vx start init[Full][Force] storcli /cx/vx start erase [simple|normal|thorough|standard][patternA=<val>] [patternB=<val>] storcli /cx/vx start cc[Force] storcli /cx/vx start migrate type =raidx [option=add|remove drives=[e:]s|[e:]s-x|[e:]s-x,y] [Force] storcli /cx/vx stop init storcli /cx/vx stop erase storcli /cx/vx stop cc storcli /cx/vx stop bgi storcli /cx/vx pause cc storcli /cx/vx pause bgi storcli /cx/vx resume cc storcli /cx/vx resume bgi storcli /cx show storcli /cx show all storcli /cx show preservedcache storcli /cx/vx delete preservedcache[force] storcli /cx[/ex]/sx download src=<filepath> [satabridge] [mode= 5|7] storcli /cx/ex download src=<filepath> [forceActivate] storcli /cx[/ex]/sx secureerase [force] storcli /cx[/ex]/sx start erase [simple| normal| thorough | standard| threepass] [patternA=<val>] [patternB=<val>] storcli /cx[/ex]/sx stop erase storcli /cx[/ex]/sx show erase storcli /cx[/ex]/sx show rawdata pageaddr=<pageaddress in hex> file=<filename> storcli /cx[/ex]/sx set bootdrive=<on|off> storcli /cx/vx set bootdrive=<on|off> storcli /cx show bootdrive storcli /cx show bootwithpinnedcache storcli /cx set bootwithpinnedcache=<on|off> storcli /cx show activityforlocate storcli /cx set activityforlocate=<on|off> storcli /cx show copyback storcli /cx set copyback=<on|off> type =smartssd|smarthdd|all storcli /cx show jbod storcli /cx set jbod=<on|off> storcli /cx set autorebuild=<on|off> storcli /cx show autorebuild storcli /cx show cachebypass storcli /cx set cachebypass=<on|off> storcli /cx show usefdeonlyencrypt storcli /cx set usefdeonlyencrypt=<on|off> storcli /cx show prcorrectunconfiguredareas storcli /cx set prcorrectunconfiguredareas=<on|off> storcli /cx show batterywarning storcli /cx set batterywarning=<on|off> storcli /cx show abortcconerror storcli /cx set abortcconerror=<on|off> storcli /cx show ncq storcli /cx set ncq=<on|off> storcli /cx show maintainpdfailhistory storcli /cx set maintainpdfailhistory=<on|off> storcli /cx show restorehotspare storcli /cx set restorehotspare=<on|off> storcli /cx set bios [state=<on|off>] [Mode=<SOE|BE|HCOE|HSM>] [abs=<on|off>] storcli /cx show bios storcli /cx show alarm storcli /cx set alarm=<on|off|silence> storcli /cx show foreignautoimport storcli /cx set foreignautoimport=<on|off> storcli /cx show directpdmapping storcli /cx set directpdmapping=<on|off> storcli /cx show rebuildrate storcli /cx set rebuildrate=<value> storcli /cx show loadbalancemode storcli /cx set loadbalancemode=<on|off> storcli /cx show eghs storcli /cx set eghs [state=<on|off>] [eug=<on|off>] [smarter=<on|off>] storcli /cx show cacheflushint storcli /cx set cacheflushint=<value> storcli /cx show prrate storcli /cx set prrate=<value> storcli /cx show ccrate storcli /cx set ccrate=<value> storcli /cx show bgirate storcli /cx set bgirate =<value> storcli /cx show dpm storcli /cx set dpm =<on|off> storcli /cx show sgpioforce storcli /cx set sgpioforce =<on|off> storcli /cx set supportssdpatrolread =<on|off> storcli /cx show migraterate storcli /cx set migraterate=<value> storcli /cx show spinupdrivecount storcli /cx show wbsupport storcli /cx set spinupdrivecount=<value> storcli /cx show spinupdelay storcli /cx set spinupdelay=<value> storcli /cx show coercion storcli /cx set coercion=<value> storcli /cx show smartpollinterval storcli /cx set smartpollinterval=<value> storcli /cx show eccbucketsize storcli /cx set eccbucketsize=<value> storcli /cx show eccbucketleakrate storcli /cx set eccbucketleakrate=<value> storcli /cx show backplane storcli /cx set backplane mode=<value> expose=<on/off> storcli /cx show perfmode storcli /cx set perfmode=<value> [maxflushlines=<value> numiostoorder=<value>] storcli /cx show perfmodevalues storcli /cx show pi storcli /cx set pi [state=<on|off>] [import=<on|off>] storcli /cx show time storcli /cx set time=<yyyymmdd hh:mm:ss | systemtime> storcli /cx show ds storcli /cx set ds=OFF type =1|2|3|4 storcli /cx set ds=ON type =1|2 [properties] storcli /cx set ds=ON type =3|4 DefaultLdType=<val> [properties] storcli /cx set ds [properties] storcli /cx show safeid storcli /cx show rehostinfo storcli /cx show pci storcli /cx show ASO storcli /cx set aso key=<key value> preview storcli /cx set aso key=<key value> storcli /cx set aso transfertovault storcli /cx set aso rehostcomplete storcli /cx set aso deactivatetrialkey storcli /cx set factory defaults storcli /cx download file=<filepath> [fwtype=<val>] [ResetNow] [nosigchk] [noverchk] storcli /cx flush|flushcache storcli /cx/px show storcli /cx/px show all storcli /cx/px set linkspeed=0|1.5|3|6|12 storcli /cx/bbu show storcli /cx/bbu set SCAPVPD file=<input file path> VPDPage=<SCapVPDFixed> storcli /cx/bbu show all storcli /cx/bbu show status storcli /cx/bbu show properties storcli /cx/bbu show learn storcli /cx/bbu show gasgauge Offset=xxxx Numbytes=n storcli /cx/bbu start learn storcli /cx/bbu show modes storcli /cx/bbu set [ learnDelayInterval=<val> | bbuMode=<val> |learnStartTime=[DDD HH | off] | autolearnmode=<val> | powermode=sleep | writeaccess=sealed ] storcli /cx/cv show storcli /cx/cv show all storcli /cx/cv show status storcli /cx/cv show learn storcli /cx/cv start learn storcli /cx show securitykey keyid storcli /cx start patrolread storcli /cx stop patrolread storcli /cx pause patrolread storcli /cx resume patrolread storcli /cx show patrolRead storcli /cx show powermonitoringinfo storcli /cx set patrolread = {{on mode=<auto|manual> }|{off}} storcli /cx set patrolread [starttime=< yyyy/mm/dd hh>] [maxconcurrentpd =<value>] [includessds=<on|onlymixed|off>] [uncfgareas=on|off] storcli /cx set patrolread delay = <value> storcli /cx[/ex]/sx show diag paniclog [Query] | [ExtractSlot=x] | [EraseSlot=x] [file=filepath] storcli /cx[/ex]/sx show diag smartlog [file=filepath] storcli /cx[/ex]/sx show diag errorlog [file=filepath] storcli /cx del Nytrocache [force] storcli /cx show badblocks storcli /cx flasherase storcli /cx shutdown storcli /cx/mx set mode=<Internal | External | Auto> storcli /cx/mx show storcli /cx transform iMR storcli /cx restart storcli /cx/vx show BBMT storcli /cx/vx delete BBMT storcli /xd add type =CacheDev <devices=comma separated cache devices | ALL> [force] storcli /xd add type =VirtualDrive <devices=comma separated Virtual drives | ALL> storcli /xd delete type =<CacheDev | VirtualDrive> <devices=comma separated cache devices or Virtual drives as applicable | ALL> storcli /xd show type =<CacheDev | VirtualDrive> [state=<assigned|unassigned>] storcli /xd show storcli /xd show perfmon storcli /xd/wd<X|ALL> show storcli /xd/wd<X|ALL> show safeid storcli /xd set online [Vds=comma separated VD ids] storcli /xd/wdX apply key=value storcli /cx[/ex]/sx start format [thorough] storcli /cx show dequeuelog file=<filepath> Note:Use 'page=[x]' as the last option in all the commands to set the page break . X=lines per page. E.g. 'storcli help page=10'