Приветствую!Сервер HP ProLiant DL140 G3, процессор Xeon 5140
SATA/SAS Hot-Swap PCI-X RAID-контроллер LSI SAS3080X-HP (на базе LSI1068)
Диски Seagate Barracuda 7.2K SATA 250GB в RAID-1
Прошивки BIOS мат.платы и RAID-контроллера -- последние с сайта HP
Система FreeBSD/amd64 -- 6.2-RELEASE
Ядро -- "монолитное", не пересобиравшееся
Установка протекала довольно медленно, да и после установки система явно задумывалась на операциях с дисками.
# uname -a
FreeBSD bee.departament.com 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007 root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP amd64
bee# dmesg
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007
root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP
ACPI APIC Table: <PTLTD APIC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2327.51-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>,<b18>>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
real memory = 5100273664 (4864 MB)
avail memory = 4121800704 (3930 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <PTLTD RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib4
mpt0: <LSILogic SAS/SATA Adapter> port 0x2000-0x20ff mem 0xdc210000-0xdc213fff,0xdc200000-0xdc20ffff irq 24 at device 1.0 on pci7
mpt0: [GIANT-LOCKED]
mpt0: MPI Version=1.5.14.0
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
mpt0: mpt_cam_event: 0x12
mpt0: Unhandled Event Notify Frame. Event 0x12 (ACK not required).
mpt0: mpt_cam_event: 0x12
mpt0: Unhandled Event Notify Frame. Event 0x12 (ACK not required).
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
mpt0: mpt_cam_event: 0xb
mpt0: Unhandled Event Notify Frame. Event 0xb (ACK not required).
pcib5: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci8: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci12: <ACPI PCI bus> on pcib6
pcib7: <PCI-PCI bridge> at device 5.0 on pci0
pci13: <PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci14: <ACPI PCI bus> on pcib8
pcib9: <PCI-PCI bridge> at device 7.0 on pci0
pci15: <PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci19: <ACPI PCI bus> on pcib10
bge0: <Broadcom BCM5750 B1, ASIC rev. 0x4101> mem 0xdc300000-0xdc30ffff irq 16 at device 0.0 on pci19
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
pcib11: <ACPI PCI-PCI bridge> at device 28.1 on pci0
pci20: <ACPI PCI bus> on pcib11
bge1: <Broadcom BCM5750 B1, ASIC rev. 0x4101> mem 0xdc400000-0xdc40ffff irq 17 at device 0.0 on pci20
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
uhci0: <UHCI (generic) USB controller> port 0x1800-0x181f irq 23 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x1820-0x183f irq 23 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x1840-0x185f irq 23 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xdc000000-0xdc0003ff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <EHCI (generic) USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci21: <ACPI PCI bus> on pcib12
pci21: <display, VGA> at device 2.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xdc000-0xdffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ukbd0: ServerEngines SE USB Device, rev 1.10/0.01, addr 2, iclass 3/1
kbd2 at ukbd0
ums0: ServerEngines SE USB Device, rev 1.10/0.01, addr 2, iclass 3/1
ums0: 8 buttons and Z dir.
Timecounters tick every 1.000 msec
acd0: DVDROM <DV-28E-V/C.AB> at ata0-master UDMA33
SMP: AP CPU #1 Launched!
da0 at mpt0 bus 0 target 1 lun 0
da0: <LSILOGIC Logical Volume 3000> Fixed Direct Access SCSI-2 device
da0: 300.000MB/s transfers, Tagged Queueing Enabled
da0: 237464MB (486326272 512 byte sectors: 255H 63S/T 30272C)
Trying to mount root from ufs:/dev/da0s1a
bge0: link state changed to UP
Различные программы оценки производительности дали примерно похожие результаты:
# /usr/local/sbin/bonnie++
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
300M 439 97 6290 1 4735 1 708 99 +++++ +++ 1909 28
Latency 78878us 442ms 399ms 30444us 854us 17635us
Version 1.93c ------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2820 8 +++++ +++ +++++ +++ 2211 6 +++++ +++ +++++ +++
Latency 1368ms 31us 54us 1352ms 34us 62821us
# ./tiotest -f 100
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 400 MBs | 58.0 s | 6.901 MB/s | 0.5 % | 8.4 % |
| Random Write 16 MBs | 26.2 s | 0.596 MB/s | 0.1 % | 1.8 % |
| Read 400 MBs | 0.5 s | 788.714 MB/s | 52.3 % | 530.3 % |
| Random Read 16 MBs | 0.0 s | 347.099 MB/s | 16.1 % | 605.7 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 2.121 ms | 477.872 ms | 0.00000 | 0.00000 |
| Random Write | 1.952 ms | 6976.487 ms | 0.02500 | 0.00000 |
| Read | 0.013 ms | 46.806 ms | 0.00000 | 0.00000 |
| Random Read | 0.032 ms | 13.569 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 1.064 ms | 6976.487 ms | 0.00047 | 0.00000 |
`--------------+-----------------+-----------------+----------+-----------'
# ./tiotest
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 40 MBs | 5.5 s | 7.337 MB/s | 0.2 % | 9.9 % |
| Random Write 16 MBs | 16.2 s | 0.964 MB/s | 0.0 % | 1.0 % |
| Read 40 MBs | 0.0 s | 1249.102 MB/s| 12.6 % | 560.5 % |
| Random Read 16 MBs | 0.0 s | 1074.326 MB/s| 0.0 % | 394.8 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 1.980 ms | 147.902 ms | 0.00000 | 0.00000 |
| Random Write | 0.148 ms | 553.000 ms | 0.00000 | 0.00000 |
| Read | 0.007 ms | 14.814 ms | 0.00000 | 0.00000 |
| Random Read | 0.005 ms | 0.130 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.736 ms | 553.000 ms | 0.00000 | 0.00000 |
`--------------+-----------------+-----------------+----------+-----------'
# /usr/local/bin/dbench 1000
dbench version 3.04 - Copyright Andrew Tridgell 1999-2004
Running for 600 seconds with load '/usr/local/share/dbench/client.txt' and minimum warmup 120 secs
1000 clients started
1000 58 3.79 MB/sec execute 462 sec
1000 58 3.78 MB/sec execute 463 sec
1000 58 3.77 MB/sec execute 464 sec
1000 58 3.76 MB/sec execute 465 sec
1000 58 3.76 MB/sec execute 466 sec
1000 58 3.75 MB/sec execute 467 sec
1000 58 3.75 MB/sec execute 468 sec
1000 58 3.76 MB/sec execute 469 sec
1000 58 3.76 MB/sec execute 470 sec
1000 58 3.76 MB/sec execute 471 sec
1000 58 3.76 MB/sec execute 472 sec
1000 59 3.76 MB/sec execute 473 sec
iostat под нагрузкой bonnie++
# iostat 1
tty da0 pass0 cpu
tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 115 64.00 98 6.13 0.00 0 0.00 2 0 1 0 97
0 346 62.57 97 5.93 0.00 0 0.00 2 0 1 0 97
0 115 63.39 97 6.01 0.00 0 0.00 2 0 2 0 96
0 115 64.00 98 6.13 0.00 0 0.00 2 0 1 0 97
0 115 64.00 98 6.13 0.00 0 0.00 2 0 1 0 97
0 115 64.00 98 6.13 0.00 0 0.00 0 0 0 0 99
0 115 64.00 100 6.25 0.00 0 0.00 1 0 0 0 99
0 115 64.00 97 6.07 0.00 0 0.00 3 0 0 0 97
0 115 59.25 97 5.62 0.00 0 0.00 2 0 1 0 98
0 115 64.00 98 6.13 0.00 0 0.00 3 0 1 0 96
0 115 64.00 101 6.31 0.00 0 0.00 1 0 0 1 98
0 115 64.00 99 6.19 0.00 0 0.00 0 0 0 0 100
0 117 62.30 97 5.91 0.00 0 0.00 1 0 0 0 99
0 115 64.00 98 6.13 0.00 0 0.00 0 0 0 0 100
diskinfo показывает, что все вроде бы ок
# diskinfo -ctv da0
da0
512 # sectorsize
248999051264 # mediasize in bytes (232G)
486326272 # mediasize in sectors
30272 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
I/O command overhead:
time to read 10MB block 0.128195 sec = 0.006 msec/sector
time to read 20480 sectors 2.330130 sec = 0.114 msec/sector
calculated command overhead = 0.108 msec/sector
Seek times:
Full stroke: 250 iter in 1.425764 sec = 5.703 msec
Half stroke: 250 iter in 1.442128 sec = 5.769 msec
Quarter stroke: 500 iter in 4.539487 sec = 9.079 msec
Short forward: 400 iter in 2.336504 sec = 5.841 msec
Short backward: 400 iter in 2.220089 sec = 5.550 msec
Seq outer: 2048 iter in 0.248946 sec = 0.122 msec
Seq inner: 2048 iter in 0.247340 sec = 0.121 msec
Transfer rates:
outside: 102400 kbytes in 1.289482 sec = 79412 kbytes/sec
middle: 102400 kbytes in 1.463749 sec = 69957 kbytes/sec
inside: 102400 kbytes in 2.551353 sec = 40136 kbytes/sec
Самые странные результаты у iozone (значения в KB/s)
# iozone -Ra -i 0 -i 1
Writer report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 349613 481234 577521 727850 801764
128 326403 475609 649331 595328 649331 656477
256 372593 509895 562815 608074 606700 557844 612584
512 440256 553502 612376 632952 682649 671129 709019 769761
1024 490192 590876 665390 688096 751287 755118 739898 772225 756848
2048 11254 11159 12089 11105 11519 11986 11016 11191 12012 11377
4096 8736 8872 8873 8767 9042 8883 8813 8815 8823 8897 9023
8192 7407 7390 7407 7468 7407 7412 7386 7471 7173 7396 7419 7449
16384 7403 7409 7405 7427 7295 7411 7405 7439 7406 7203 7398 7437 7412
32768 0 0 0 0 7110 7116 6903 7122 7116 7089 7130 7120 7112
65536 0 0 0 0 7021 7031 7014 6943 7052 7047 7050 7051 7032
131072 0 0 0 0 6925 6950 6963 6970 6938 6988 6964 6913 6973
262144 0 0 0 0 6925 6916 6924 6901 6905 6931 6930 6935 6931
524288 0 0 0 0 6893 6885 6890 6895 6876 6894 6880 6885 6891
Reader report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 2006158 3057153 3203069 3588436 3791156
128 1802755 2784517 3359523 3657016 3380677 4135958
256 1839210 2842047 3368013 3879045 4054829 4350555 3879045
512 1848401 2860238 2141475 3905895 4228947 4305250 4195896 5177083
1024 2043465 2746483 3594699 3521025 4300102 4033569 4762630 4550690 4474829
2048 1782585 2000075 3108009 3459769 3261415 3690142 3709263 3671217 2797290 2863496
4096 1593117 2131284 2490267 2360908 2878275 2948925 3003050 2981164 2680666 2154808 1686151
8192 1447895 1931537 2207527 2400293 2509877 2528346 2522962 2421438 2157488 1792481 1465996 1272827
16384 1421239 1855938 2114913 2304336 2378593 2411141 2390092 2307586 2048282 1699386 1421239 1239331 1224775
32768 0 0 0 0 2381552 2411175 2396544 2303247 2050446 1689958 1389945 1218097 1200824
65536 0 0 0 0 2394953 2433198 2403833 2319198 2049220 1684210 1374322 1207326 1191718
131072 0 0 0 0 2411317 2444915 2423360 2334025 2073894 1706776 1398233 1224892 1204187
262144 0 0 0 0 1117989 1126933 1120483 1099925 1030051 932950 837432 785389 782262
524288 0 0 0 0 1122732 1131054 1128274 1106011 1037329 939671 844884 788923 787122
Итого -- "безумные" цифры на чтениут и (следствие кэширования?), и что-то неприлиличное на записи. 6MB/s -- как такое может быть?
Здесь (и в разных рассылках тоже) уже встречались жалобы на связку HP-DL+LSI-RAID+FreeBSD-6.x (например, http://www.opennet.ru/openforum/vsluhforumID1/74296.html)
Советы, в итоге, сводились к пересборке на CURRENT
На пробу решил это сделать -- обновить все исходники до CURRENT и установить:
## ---- sup-file for CURRENT ----
*default tag=.
*default host=cvsup.ru.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs delete use-rel-suffix compress
## Main Source Tree.
#
# The easiest way to get the main source tree is to use the "src-all"
# mega-collection. It includes all of the individual "src-*" collections,
# except the export-restricted collections.
src-all
## ---- end of sup-file ----
Ядро собирал из GENERIC, с выключенными debug-options и устройствами типа PC-CARD, WLAN, Firewire.
саму систему -- все по дефолту.
В итоге, на получившейся 7.0-CURRENT amd64 имел ровно те же самые результаты, что и на 6.2-RELEASE
Строчка
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
в dmesg также присутствовала.
Пробовал поставить 7.0-CURRENT с нуля с июньского snapshot -- не вышло по причине того, что примерно на середине загрузки ядра клавиатура перестает откликаться.
В итоге откатился обратно на 6.2-RELEASE
Да, еще пару раз на 6.2-RELEASE глухо зависало при загрузке сразу после строчки
acd0: DVDROM <DV-28E-V/C.AB> at ata0-master UDMA33
В какую сторону рыть?
Или забить и ставить что-то из официально поддерживаемого (тут лишь два варианта -- RHEL 4/5 или SLES10) ? :)
Задачи сервера: боевой хостинг небольшого (до 100) количества сайтов, POP3, VoIP(CommuniGate). Хотелось бы все же остаться с FreeBSD.
Спасибо!
Александр
