Hi,

We are running an C++ application in AWS EC2 instance (CentOS 5.4) mounted with an EBS Volume (say /mymountpoint). We do more simultaneous writes to the EBS Volume from our application. But at some point we get 'ERROR: Input/output error'. After this, 'ls -l /mymountpoint' command itself fails with the i/o error. The filesystem which we use for the EBS Volume is xfs.

I unmounted the drive and done xfs_check and again mounted the drive. Now, everything seems to be working fine. But the issue still persists everytime when we do simultaneous writes.

I believe the following details will be useful,

[root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
CentOS release 5.4 (Final)

[root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
Filesystem Type Inodes IUsed IFree IUse% Mounted on
/dev/sda1 ext3 1310720 107566 1203154 9% /
/dev/sdb ext3 19546112 11 19546101 1% /mnt
none tmpfs 186059 1 186058 1% /dev/shm
/dev/sdh xfs 1934272 495857 1438415 26% /mymountpoint

[root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 i686 i386 GNU/Linux

Output of dmesg :

SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
SGI XFS Quota Management subsystem
Filesystem "sdh": Disabling barriers, not supported by the underlying device
XFS mounting filesystem sdh
Ending clean XFS mount for filesystem: sdh
Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c. Caller 0xee201944
[<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
[<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
[<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
[<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
[<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
[<c1078d88>] vfs_rename+0x2c3/0x441
[<c107a77f>] sys_renameat+0x15a/0x1b4
[<c1074b7f>] sys_stat64+0xf/0x23
[<c1072d3b>] __fput+0x140/0x16a
[<c10841ee>] mntput_no_expire+0x11/0x6a
[<c107a800>] sys_rename+0x27/0x2b
[<c1005688>] syscall_call+0x7/0xb
=======================
xfs_force_shutdown(sdh,0x8) called from line 1139 of file fs/xfs/xfs_trans.c. Return address = 0xee217778
Filesystem "sdh": Corruption of in-memory data detected. Shutting down filesystem: sdh
Please umount the filesystem, and rectify the problem(s)
I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001 ("xfs_trans_read_buf") error 5 buf count 512
I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001 ("xfs_trans_read_buf") error 5 buf count 512
xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c. Return address = 0xee217778
xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c. Return address = 0xee217778
Filesystem "sdh": Disabling barriers, not supported by the underlying device
XFS mounting filesystem sdh
Starting XFS recovery on filesystem: sdh (logdev: internal)
Ending XFS recovery on filesystem: sdh (logdev: internal)

The XFS utilities are in v2.9.4

Any help would be appreciated.

Regards,
Srinivasan