HowTo [SOLVED] mdadm: Cannot open /dev/sdb2: Device or resource busy

[root@1-srv-h ~]# mdadm /dev/md0 —manage —add /dev/sdb2
mdadm: Cannot open /dev/sdb2: Device or resource busy

[root@1-srv-h ~]# dmsetup status
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS: 0 976773168 multipath 2 0 0 0 1 1 A 0 1 0 8:16 A 0
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp3: 0 959174656 linear
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp2: 0 819200 linear
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp1: 0 16777216 linear

[root@1-srv-h ~]# dmsetup —help
Usage:

help [-c|-C|—columns]
create [-j|—major -m|—minor ]
[-U|—uid ] [-G|—gid ] [-M|—mode ]
[-u|uuid ] [{—addnodeonresume|—addnodeoncreate}]
[—notable | —table

| ]
remove [-f|—force]
remove_all [-f|—force]
suspend [—noflush]
resume [{—addnodeonresume|—addnodeoncreate}]
load []
clear
reload []
wipe_table
rename [—setuuid]
message
ls [—target ] [—exec ] [-o options] [—tree]
info []
deps [-o options] []
status [] [—noflush] [—target ]
table [] [—target ] [—showkeys]
wait [] [—noflush]
mknodes []
mangle []
udevcreatecookie
udevreleasecookie []
udevflags
udevcomplete
udevcomplete_all
udevcookies
targets
version
setgeometry
splitname []

may be device name or -u or -j -m
is one of ‘none’, ‘auto’ and ‘hex’.
are comma-separated. Use ‘help -c’ for list.
Table_file contents may be supplied on stdin.
Options are: devno, devname, blkdevname.
Tree specific options are: ascii, utf, vt100; compact, inverted, notrunc;
blkdevname, [no]device, active, open, rw and uuid.

[root@1-srv-h ~]# dmsetup info
Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS
State: ACTIVE
Read Ahead: 1024
Tables present: LIVE
Open count: 3
Event number: 1
Major, minor: 253, 0
Number of targets: 1
UUID: mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp3
State: ACTIVE
Read Ahead: 1024
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 3
Number of targets: 1
UUID: part3-mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp2
State: ACTIVE
Read Ahead: 1024
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 2
Number of targets: 1
UUID: part2-mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp1
State: ACTIVE
Read Ahead: 1024
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 1
Number of targets: 1
UUID: part1-mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

[root@1-srv-h ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00093535

Device Boot Start End Blocks Id System
/dev/sdb1 1 1045 8388608 fd Linux raid autodetect
/dev/sdb2 * 1045 1096 409600 fd Linux raid autodetect
/dev/sdb3 1096 60802 479587328 fd Linux raid autodetect
[root@1-srv-h ~]# smartctl /dev/sdb
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.29.2.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

ATA device successfully opened

Use ‘smartctl -a’ (or ‘-x’) to print SMART (and more) information

[root@1-srv-h ~]# smartctl -a /dev/sdb
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.29.2.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model: TOSHIBA DT01ACA050
Serial Number: Y3H0YMMAS
LU WWN Device Id: 5 000039 ff8ce86be
Firmware Version: MS1OA750
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Fri Oct 17 15:36:14 2014 ART
SMART support is: Available — device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 3668) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 62) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always — 0
2 Throughput_Performance 0x0005 141 141 054 Pre-fail Offline — 73
3 Spin_Up_Time 0x0007 134 134 024 Pre-fail Always — 160 (Average 182)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always — 39
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always — 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always — 0
8 Seek_Time_Performance 0x0005 103 103 020 Pre-fail Offline — 39
9 Power_On_Hours 0x0012 100 100 000 Old_age Always — 3838
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always — 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always — 39
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always — 79
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always — 79
194 Temperature_Celsius 0x0002 206 206 000 Old_age Always — 29 (Min/Max 19/41)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always — 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always — 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline — 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always — 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@1-srv-h ~]# man dmsetup

[root@1-srv-h ~]# dmsetup ls
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS (253:0)
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp3 (253:3)
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp2 (253:2)
1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp1 (253:1)

[root@1-srv-h ~]# dmsetup remove 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS
device-mapper: remove ioctl on 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS failed: Device or resource busy
Command failed

[root@1-srv-h ~]# dmsetup remove -f 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS
device-mapper: reload ioctl on 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS failed: Invalid argument
device-mapper: remove ioctl on 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS failed: Device or resource busy
Command failed

[root@1-srv-h ~]# dmsetup remove -f /dev/sdb
Device sdb not found
Device sdb not found
Device sdb not found
Command failed

[root@1-srv-h ~]# dmsetup suspend 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

[root@1-srv-h ~]# dmsetup suspend 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp3

[root@1-srv-h ~]# dmsetup suspend 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp2

[root@1-srv-h ~]# dmsetup suspend 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp1

[root@1-srv-h ~]# dmsetup info
Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS
State: SUSPENDED
Read Ahead: 1024
Tables present: LIVE
Open count: 3
Event number: 1
Major, minor: 253, 0
Number of targets: 1
UUID: mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp3
State: SUSPENDED
Read Ahead: 1024
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 3
Number of targets: 1
UUID: part3-mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp2
State: SUSPENDED
Read Ahead: 1024
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 2
Number of targets: 1
UUID: part2-mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

Name: 1ATA_TOSHIBA_DT01ACA050_Y3H0YMMASp1
State: SUSPENDED
Read Ahead: 1024
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 1
Number of targets: 1
UUID: part1-mpath-1ATA_TOSHIBA_DT01ACA050_Y3H0YMMAS

[root@1-srv-h ~]# mdadm /dev/md0 —manage —add /dev/sdb2
mdadm: Cannot open /dev/sdb2: Device or resource busy

[root@1-srv-h ~]# dmsetup remove_all -f

[root@1-srv-h ~]# dmsetup info
No devices found

[root@1-srv-h ~]# mdadm /dev/md0 —manage —add /dev/sdb2
mdadm: added /dev/sdb2

[root@1-srv-h ~]# cat /proc/mdstat
Personalities : [raid10] [raid1]
md1 : active raid10 sda1[0] sdd1[3] sdc1[2]
16768000 blocks super 1.1 512K chunks 2 near-copies [4/3] [U_UU]

md0 : active raid1 sdb2[5] sdd2[3] sdc2[2] sda2[4]
409536 blocks super 1.0 [4/4] [UUUU]

md2 : active raid10 sdd3[3] sdc3[2] sda3[0]
958911488 blocks super 1.1 512K chunks 2 near-copies [4/3] [U_UU]
bitmap: 7/8 pages [28KB], 65536KB chunk

unused devices:

[root@1-srv-h ~]# mdadm /dev/md2 —manage —add /dev/sdb3
mdadm: added /dev/sdb3

[root@1-srv-h ~]# cat /proc/mdstat
Personalities : [raid10] [raid1]
md1 : active raid10 sda1[0] sdd1[3] sdc1[2]
16768000 blocks super 1.1 512K chunks 2 near-copies [4/3] [U_UU]

md0 : active raid1 sdb2[5] sdd2[3] sdc2[2] sda2[4]
409536 blocks super 1.0 [4/4] [UUUU]

md2 : active raid10 sdb3[4] sdd3[3] sdc3[2] sda3[0]
958911488 blocks super 1.1 512K chunks 2 near-copies [4/3] [U_UU]
[>………………..] recovery = 0.0% (128640/479455744) finish=124.1min speed=64320K/sec
bitmap: 7/8 pages [28KB], 65536KB chunk

unused devices:

[root@1-srv-h ~]# mdadm /dev/md1 —manage —add /dev/sdb1
mdadm: added /dev/sdb1

[root@1-srv-h ~]# cat /proc/mdstat
Personalities : [raid10] [raid1]
md1 : active raid10 sdb1[4] sda1[0] sdd1[3] sdc1[2]
16768000 blocks super 1.1 512K chunks 2 near-copies [4/3] [U_UU]
resync=DELAYED

md0 : active raid1 sdb2[5] sdd2[3] sdc2[2] sda2[4]
409536 blocks super 1.0 [4/4] [UUUU]

md2 : active raid10 sdb3[4] sdd3[3] sdc3[2] sda3[0]
958911488 blocks super 1.1 512K chunks 2 near-copies [4/3] [U_UU]
[>………………..] recovery = 0.5% (2506368/479455744) finish=143.3min speed=55450K/sec
bitmap: 7/8 pages [28KB], 65536KB chunk

unused devices:

[root@1-srv-h ~]# watch -n 1 cat /proc/mdstat
Every 1.0s: cat /proc/mdstat Fri Oct 17 20:01:47 2014

Personalities : [raid10] [raid1]
md0 : active raid1 sdd2[3] sdc2[2] sda2[4] sdb2[5]
409536 blocks super 1.0 [4/4] [UUUU]

md1 : active raid10 sdd1[3] sdc1[2] sda1[0] sdb1[4]
16768000 blocks super 1.1 512K chunks 2 near-copies [4/4] [UUUU]

md2 : active raid10 sdd3[3] sdc3[2] sdb3[4] sda3[0]
958911488 blocks super 1.1 512K chunks 2 near-copies [4/3] [U_UU]
[==============>……] recovery = 74.2% (355966848/479455744) finish=70.2min speed=29290K/sec
bitmap: 7/8 pages [28KB], 65536KB chunk

unused devices: