zfs의 화일 시스템 에러(permanent error)와 패키지 수정 (pkg fix)

최신 버젼의 zfs를 쓰면서 우연히 알게 된 것은 화일 시스템의 오류가 왕왕 생긴 다는 점입니다.
아마도, 랩탑이기 때문에 파워가 없어서 그냥 꺼졌거나 하면서 생긴 것으로 보이는데.. 생각보다
이런 오류가 생각보더 훨씬 자주 나타난다는 게 약간 맘에 걸립니다. 이게 정상인지..

간혹 fmstat나 zpool을 보면 아래와 같이 오류가 발생하는 경우가 있습니다.

$ pfexec zpool status -xv
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: scrub completed after 4h46m with 6 errors on Wed Jul 15 20:19:19 2009
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     7
      c5d0s0    ONLINE       0     0    55  291K repaired

errors: Permanent errors have been detected in the following files:

        rpool/export/home/bhkim:<0x2d071>
        rpool/ROOT/opensolaris-2@install:/usr/lib/amd64/libcspi.so.0.10.11
        rpool/ROOT/opensolaris-2@install:/usr/bin/w3m
        rpool/ROOT/opensolaris-2@install:/usr/bin/what
        rpool/ROOT/opensolaris-2@install:/usr/bin/wracct
        rpool/ROOT/opensolaris-2@install:/usr/bin/wtpt

오류라고 나온 화일을 실행시도 해봤더니, 정말 I/O 에러라고 나옵니다. ㅡㅡ;

bhkim@vaio-bhkim:~/MyVideos$ wracct
bash: /usr/bin/wracct: I/O error

패키지 수정을 시도해야 겠습니다. 일단 해당 화일이 어떤 패키지에 속하는 지 알아야 하는데 다음과 같이해서 알아낼 수 있습니다.
#pkg search /usr/bin/wracct
SUNWcs라는 것을 얘기해주는데, 결과가 길어서 여기서는 생략합니다.

일단, 오류로 나온 다른 화일의 패키지로 검사를 해봤습니다.
#pkg fix SUNWgnome-a11y-libs

#
엥... 아무런 무언가가 없습니다. 오류가 없이 verify 했다는 얘기입니다.
여기서 오픈 솔라리스 에서 2% 안타까운 것은 zfs에서 해당 화일이 오류가 있다고 나왔다면
pkg 에서도 오류임을 자동으로 감지해서 fix할때 fix해주었으면 하는데, pkg는 화일의 목록만 검사하는 듯 하군요.
오류 없다고 지나가는 것을 보니까 말이죠.

그렇다면, 해당 화일을 삭제하고 fix를 시도하면 감지할 것으로 예상하고 이번에는 실제 작업 대상 패키지에서 작업해봤습니다.

$ pfexec bash
# rm /usr/bin/wracct
일단 삭제했습니다.

# pkg fix SUNWcs
Verifying: pkg:/SUNWcs                                          ERROR  <--- 오.. 역시 에러가 나오네요.
    file: usr/bin/cat
        Warning: package may contain bobcat!  (http://xkcd.com/325/)   ;; 근데 뭔 에러가... 이리 많지 ㅡㅡ;
        file: var/spool/cron/crontabs/root        ..... --
        Group: 'root' should be 'sys'
        file: etc/nsswitch.conf                   ..... ||
        Group: 'root' should be 'sys'
        file: usr/bin/wracct                      ..... //             ;; 아하.. 화일이 없음을 감지했습니다.
        File does not exist
        hardlink: etc/rc2.d/S82mkdtab             ..... //
        No such path etc/rc2.d/S82mkdtab
Created ZFS snapshot: 2009-07-16-00:42:02         ..... //             ;; 스냅샷을 찍어주고 수리들어가네요.
Repairing: pkg:/SUNWcs                                      
PHASE                                          ITEMS
Indexing Packages                            481/670

죽어버린 X-window (X window crash)

오픈 솔라리스 2009.06 버젼으로 업데이트해서 잘 사용하고 있었습니다.
너무 문제가 없어서 그랬는지, 어제는 악마의 속삭임에 빠져서, 이것 저것 업데이트를 했습니다.
pkg로 업데이트를 하면 괜찮았을 텐데, 별개로 컴파일해서 인스톨하고 했습니다.

제가 설치한 화일들은 잘 돌은 듯 했습니다.

문제는 오늘 발생. 모니터를 붙여서 화면 클로닝으로 설정하려고 했습니다. nVidia configuration으로 평소에 하던 대로 아주 편하게 했습니다. 그런데 이게 웬걸.. 화면 구성을 하고 apply를 누르는 순간 X-window가 죄다 없어지고, 텍스트 콘솔 화면이 잠깐 보였다고 다시 재로그인하는 화면으로 돌아옵니다.

여러 창으로 작업을 하던 상황에서 이렇게 되니 당황스럽습니다.  일단, X server가 시그널 받아서 죽는 것 같다는 느낌이 듭니다. 그런데,  명령어 창을 띄워야 뭘 live로 추적해볼 텐데.. 윈도우즈가 죽어버리니.. 어쩌나 하는 생각을 하게 됐습니다. 급기야, 'dtrace를 쓸 수 있나? ' 하는 생각에 이르렀습니다.

마침 /opt/DTT 디렉토리에 Dtrace Toolkit이 설치되어 있었습니다.

창을 하나 열었습니다. 그리고 다음처럼 명령을 실행했습니다.
$pfexec dtrace -s /opt/DTT/Proc/crash.d > /var/tmp/crash.report &

마침 Dtrace Toolkit에는 프로세스가 죽을때를 캡쳐해주는 스크립트가 있더군요.
다시 화면 구성을 시도했습니다. 예상대로 또 죽었습니다.  이럴 때 안죽으면 사실 좀 황당한데. 아주 착하게 잘 죽었습니다. ㅡㅡ;

재 로그인을 한 후 /var/tmp/crash.report 화일을 확인해봤습니다.

아~~주 훌륭합니다.  프로세스 죽을 때 이렇게 분석해주는 보고서는 첨 봤습니다.
단지 놀라울 따름입니다.


:/var/tmp$ cat crash.report
Waiting for crashing applications...

-----------------------------------------------------
CRASH DETECTED at 2009 Jul 15 15:47:06
-----------------------------------------------------
Type:    SIGSEGV
Program: Xorg
Args:    /usr/X11/bin/Xorg :0 -depth 24 -nolisten tcp -audit 0 -br -auth /var/lib/gdm/:0\0
PID:     1640
TID:     1
LWPs:    1
PPID:    1639
UID:     0
GID:     0
TaskID:  115
ProjID:  0
PoolID:  0
ZoneID:  0
zone:    global
CWD:     /var/lib/gdm
errno:   0

User Stack Backtrace,
              libglx.so.1`0xfffffd7ffd1c45ae

Kernel Stack Backtrace,
              genunix`sigaddqa+0x5f
              genunix`trapsig+0x119
              unix`trap+0x12a1
              unix`0xfffffffffb8001d9

Ansestors,
    1640 /usr/X11/bin/Xorg :0 -depth 24 -nolisten tcp -audit 0 -br -auth /var/lib/gdm/:0\0
      1639 /usr/sbin/gdm-binary\0
        1351 /usr/sbin/gdm-binary\0
          1 /sbin/init\0
            0 sched\0

Times,
    User:    9182 ticks
    Sys:     8066 ticks
    Elapsed: 209133 ms

Sizes,
    Heap:   43286536 bytes
    Stack:  577536 bytes

보고서에 나온대로 Xorg 서버의 현재 작업 디렉토리로 가봤습니다.  예상대로 core 알을 까놨군요.
보고서 상으로 보면 libglx.so에서 죽었네요. ( libglx.so.1`0xfffffd7ffd1c45ae )

core를 좀 더 확인해봤습니다.
$echo "\$C" | mdb core ; pstack과 같습니다.
/var/lib/gdm# echo "\$C" | mdb core
fffffd7fffdfe450 libc.so.1`_lwp_kill+0xa()
fffffd7fffdfe470 libc.so.1`raise+0x19()
fffffd7fffdfe4b0 libc.so.1`abort+0xd9()
fffffd7fffdfe4d0 0x4c32ba()
fffffd7fffdfe4e0 AbortServer+0x13()
fffffd7fffdfe5d0 FatalError+0xe4()
fffffd7fffdfe700 xf86SigHandler+0xb5()
fffffd7fffdfe710 libc.so.1`__sighndlr+6()
fffffd7fffdfe7a0 libc.so.1`call_user_handler+0x2a7()
fffffd7fffdfe7d0 libc.so.1`sigacthandler+0xc6(b, 0, fffffd7fffdfe7f0)
000000000100c640 0xfffffd7ffd4c43de()
000000000217b910 0()
0000000000001000 0()

고개가 갸우뚱 하네요... 어디서 죽은 거지 ? dtrace 말이 맞는 건가 ?
어드레스 맵을 한번 더 확인해보기로 했습니다.
# echo "\$m" | mdb core
            BASE            LIMIT             SIZE NAME
          400000           5fc000           1fc000 /usr/X11/bin/amd64/Xorg
          60c000           620000            14000 /usr/X11/bin/amd64/Xorg
          800000          6c3b000          643b000 /usr/X11/bin/amd64/Xorg
fffffd7ff52d6000 fffffd7ff54d6000           200000
fffffd7ff6b88000 fffffd7ff6d88000           200000
fffffd7ff6ef8000 fffffd7ff70f8000           200000
fffffd7ff7d46000 fffffd7ff7f46000           200000
fffffd7ff80e8000 fffffd7ff82e8000           200000
fffffd7ff84ec000 fffffd7ff86ec000           200000
fffffd7ff873a000 fffffd7ff893a000           200000
fffffd7ff8ad6000 fffffd7ff8cd6000           200000
fffffd7ff8dc6000 fffffd7ff8fc6000           200000
fffffd7ff9400000 fffffd7ff9442000            42000 /usr/X11/lib/modules/amd64/libfb.so
fffffd7ff9452000 fffffd7ff9454000             2000 /usr/X11/lib/modules/amd64/libfb.so
fffffd7ff9800000 fffffd7ff9801000             1000 /usr/X11/lib/modules/amd64/libfb.so
fffffd7ff9a00000 fffffd7ff9d85000           385000 /usr/X11/lib/modules/drivers/amd64/nvidia_drv.so
fffffd7ff9e85000 fffffd7ff9f1f000            9a000 /usr/X11/lib/modules/drivers/amd64/nvidia_drv.so
fffffd7ff9f1f000 fffffd7ff9f3e000            1f000 /usr/X11/lib/modules/drivers/amd64/nvidia_drv.so
fffffd7ff9ff0000 fffffd7ff9ff1000             1000
fffffd7ffa000000 fffffd7ffa004000             4000 /usr/X11/lib/modules/extensions/amd64/libIA.so
fffffd7ffa014000 fffffd7ffa015000             1000 /usr/X11/lib/modules/extensions/amd64/libIA.so
fffffd7ffa400000 fffffd7ffa401000             1000 /usr/X11/lib/modules/extensions/amd64/libIA.so
fffffd7ffa7f0000 fffffd7ffa7f1000             1000
fffffd7ffa800000 fffffd7ffa809000             9000 /usr/X11/lib/modules/extensions/amd64/libdri.so
fffffd7ffa819000 fffffd7ffa81a000             1000 /usr/X11/lib/modules/extensions/amd64/libdri.so
fffffd7ffac00000 fffffd7ffac01000             1000 /usr/X11/lib/modules/extensions/amd64/libdri.so
fffffd7ffaff0000 fffffd7ffaff1000             1000
fffffd7ffb000000 fffffd7ffb009000             9000 /usr/X11/lib/modules/extensions/amd64/librecord.so
fffffd7ffb019000 fffffd7ffb01a000             1000 /usr/X11/lib/modules/extensions/amd64/librecord.so
fffffd7ffb400000 fffffd7ffb401000             1000 /usr/X11/lib/modules/extensions/amd64/librecord.so
fffffd7ffb7f0000 fffffd7ffb7f1000             1000
fffffd7ffb800000 fffffd7ffb801000             1000 /usr/X11/lib/modules/fonts/amd64/libfreetype.so
fffffd7ffb811000 fffffd7ffb812000             1000 /usr/X11/lib/modules/fonts/amd64/libfreetype.so
fffffd7ffbab7000 fffffd7ffbbff000           148000
fffffd7ffbc00000 fffffd7ffbc01000             1000 /usr/X11/lib/modules/fonts/amd64/libfreetype.so
fffffd7ffbc8b000 fffffd7ffbceb000            60000
fffffd7ffc000000 fffffd7ffccf4000           cf4000 /usr/X11/lib/NVIDIA/amd64/libGLcore.so.1
fffffd7ffcdf4000 fffffd7ffd245000           451000 /usr/X11/lib/NVIDIA/amd64/libGLcore.so.1
fffffd7ffd245000 fffffd7ffd257000            12000 /usr/X11/lib/NVIDIA/amd64/libGLcore.so.1
fffffd7ffd270000 fffffd7ffd2d0000            60000
fffffd7ffd400000 fffffd7ffd540000           140000 /usr/X11/lib/modules/extensions/NVIDIA/amd64/libglx.so.1
fffffd7ffd57d000 fffffd7ffd5dd000            60000
fffffd7ffd5de000 fffffd7ffd63e000            60000
fffffd7ffd63f000 fffffd7ffd6a5000            66000 /usr/X11/lib/modules/extensions/NVIDIA/amd64/libglx.so.1
fffffd7ffd6a5000 fffffd7ffd6a7000             2000 /usr/X11/lib/modules/extensions/NVIDIA/amd64/libglx.so.1
fffffd7ffd6cd000 fffffd7ffd72d000            60000
fffffd7ffd7f0000 fffffd7ffd7f1000             1000
fffffd7ffd800000 fffffd7ffd805000             5000 /usr/X11/lib/modules/extensions/amd64/libdbe.so
fffffd7ffd815000 fffffd7ffd816000             1000 /usr/X11/lib/modules/extensions/amd64/libdbe.so
fffffd7ffd826000 fffffd7ffd86f000            49000
fffffd7ffd8af000 fffffd7ffd90f000            60000
fffffd7ffd910000 fffffd7ffd970000            60000
fffffd7ffd971000 fffffd7ffd9d1000            60000
fffffd7ffda70000 fffffd7ffdad0000            60000
fffffd7ffdaff000 fffffd7ffdb5f000            60000
fffffd7ffdb9f000 fffffd7ffdbff000            60000
fffffd7ffdc00000 fffffd7ffdc01000             1000 /usr/X11/lib/modules/extensions/amd64/libdbe.so
fffffd7ffdcef000 fffffd7ffdd4f000            60000
fffffd7ffdd6c000 fffffd7ffddcc000            60000
fffffd7ffddcd000 fffffd7ffde2d000            60000
fffffd7ffde2e000 fffffd7ffde8e000            60000
fffffd7ffde8f000 fffffd7ffdeef000            60000
fffffd7ffdf0c000 fffffd7ffdf10000             4000
fffffd7ffdf1a000 fffffd7ffdf7a000            60000
fffffd7ffdf90000 fffffd7ffdf98000             8000
fffffd7ffdf9f000 fffffd7ffdfff000            60000
fffffd7ffe000000 fffffd7ffe027000            27000 /usr/X11/lib/modules/extensions/amd64/libextmod.so
fffffd7ffe037000 fffffd7ffe039000             2000 /usr/X11/lib/modules/extensions/amd64/libextmod.so
fffffd7ffe04a000 fffffd7ffe0aa000            60000
fffffd7ffe0b5000 fffffd7ffe1b5000           100000
fffffd7ffe1b6000 fffffd7ffe1b7000             1000
fffffd7ffe1ba000 fffffd7ffe21a000            60000
fffffd7ffe21b000 fffffd7ffe27b000            60000
fffffd7ffe28f000 fffffd7ffe2cf000            40000
fffffd7ffe2d0000 fffffd7ffe2dc000             c000
fffffd7ffe2dd000 fffffd7ffe33d000            60000
fffffd7ffe33e000 fffffd7ffe39e000            60000
fffffd7ffe39f000 fffffd7ffe3ff000            60000
fffffd7ffe400000 fffffd7ffe401000             1000 /usr/X11/lib/modules/extensions/amd64/libextmod.so
fffffd7ffe413000 fffffd7ffe414000             1000
fffffd7ffe424000 fffffd7ffe444000            20000
fffffd7ffe445000 fffffd7ffe4a5000            60000
fffffd7ffe4b7000 fffffd7ffe506000            4f000
fffffd7ffe507000 fffffd7ffe567000            60000
fffffd7ffe568000 fffffd7ffe5c8000            60000
fffffd7ffe5c9000 fffffd7ffe5cd000             4000
fffffd7ffe5cf000 fffffd7ffe6cf000           100000
fffffd7ffe6d0000 fffffd7ffe6d1000             1000
fffffd7ffe6d9000 fffffd7ffe719000            40000
fffffd7ffe730000 fffffd7ffe733000             3000
fffffd7ffe73b000 fffffd7ffe79e000            63000
fffffd7ffe79f000 fffffd7ffe7ff000            60000
fffffd7ffe800000 fffffd7ffe8a3000            a3000 /usr/X11/lib/modules/amd64/libpcidata.so
fffffd7ffe8b3000 fffffd7ffe902000            4f000 /usr/X11/lib/modules/amd64/libpcidata.so
fffffd7ffe912000 fffffd7ffe913000             1000
fffffd7ffe91d000 fffffd7ffe97d000            60000
fffffd7ffe97e000 fffffd7ffe9de000            60000
fffffd7ffe9f0000 fffffd7ffe9f8000             8000
fffffd7ffea12000 fffffd7ffea13000             1000
fffffd7ffea23000 fffffd7ffea24000             1000
fffffd7ffea7f000 fffffd7ffeadf000            60000
fffffd7ffeae0000 fffffd7ffeae3000             3000
/usr/X11/lib/amd64/libXau.so.6
fffffd7ffeaf3000 fffffd7ffeaf4000             1000
/usr/X11/lib/amd64/libXau.so.6
fffffd7ffeb06000 fffffd7ffeb0a000             4000
fffffd7ffeb10000 fffffd7ffeb12000             2000
fffffd7ffeb20000 fffffd7ffeb36000            16000 /lib/amd64/libz.so.1
fffffd7ffeb45000 fffffd7ffeb47000             2000 /lib/amd64/libz.so.1
fffffd7ffeb60000 fffffd7ffeb61000             1000
fffffd7ffeb70000 fffffd7ffebdf000            6f000 /lib/amd64/libm.so.2
fffffd7ffebee000 fffffd7ffebf1000             3000 /lib/amd64/libm.so.2
fffffd7ffec00000 fffffd7ffec01000             1000
fffffd7ffec5e000 fffffd7ffec7f000            21000
fffffd7ffec80000 fffffd7ffec81000             1000
fffffd7ffec90000 fffffd7ffecd6000            46000 /usr/X11/lib/modules/NVIDIA/amd64/libnvidia-wfb.so.1
fffffd7ffece5000 fffffd7ffece7000             2000 /usr/X11/lib/modules/NVIDIA/amd64/libnvidia-wfb.so.1
fffffd7ffed00000 fffffd7ffed01000             1000
fffffd7ffed20000 fffffd7ffed21000             1000
fffffd7ffed30000 fffffd7ffed34000             4000 /usr/X11/lib/modules/input/amd64/kbd_drv.so
fffffd7ffed44000 fffffd7ffed45000             1000 /usr/X11/lib/modules/input/amd64/kbd_drv.so
fffffd7ffed55000 fffffd7ffed56000             1000
fffffd7ffed60000 fffffd7ffed6c000             c000 /usr/X11/lib/modules/input/amd64/mouse_drv.so
fffffd7ffed7c000 fffffd7ffed7e000             2000 /usr/X11/lib/modules/input/amd64/mouse_drv.so
fffffd7ffed80000 fffffd7ffed81000             1000
fffffd7ffedd0000 fffffd7ffedd1000             1000
fffffd7ffee00000 fffffd7ffee01000             1000 /usr/X11/lib/NVIDIA/amd64/libnvidia-tls.so.1
fffffd7ffee11000 fffffd7ffee12000             1000
fffffd7ffee20000 fffffd7ffee21000             1000
fffffd7ffee32000 fffffd7ffee33000             1000
fffffd7ffee54000 fffffd7ffee64000            10000
fffffd7ffee70000 fffffd7ffee71000             1000
fffffd7ffee72000 fffffd7ffee73000             1000
fffffd7ffee82000 fffffd7ffee83000             1000
fffffd7ffee8e000 fffffd7ffee9f000            11000
fffffd7ffeea0000 fffffd7ffeea1000             1000
fffffd7ffeeb0000 fffffd7ffeebd000             d000 /usr/X11/lib/X11/xserver/amd64/libdrm.so.2
fffffd7ffeecd000 fffffd7ffeece000             1000 /usr/X11/lib/X11/xserver/amd64/libdrm.so.2
fffffd7ffeee0000 fffffd7ffeee1000             1000
fffffd7ffeef1000 fffffd7ffeef2000             1000
fffffd7ffeef6000 fffffd7ffeef7000             1000
fffffd7ffef00000 fffffd7ffef01000             1000 /usr/X11/lib/NVIDIA/amd64/libnvidia-tls.so.1
fffffd7ffef12000 fffffd7ffef13000             1000
fffffd7ffef16000 fffffd7ffef17000             1000
fffffd7ffef20000 fffffd7ffef21000             1000
fffffd7ffef30000 fffffd7ffef32000             2000 /lib/amd64/librt.so.1
fffffd7ffef40000 fffffd7ffef41000             1000
fffffd7ffef52000 fffffd7ffef53000             1000
fffffd7ffef60000 fffffd7ffef61000             1000
fffffd7ffef72000 fffffd7ffef73000             1000
fffffd7ffef80000 fffffd7ffef81000             1000
fffffd7ffef90000 fffffd7ffef9c000             c000 /usr/sfw/lib/amd64/libgcc_s.so.1
fffffd7ffefab000 fffffd7ffefac000             1000 /usr/sfw/lib/amd64/libgcc_s.so.1
fffffd7ffefc0000 fffffd7ffefc1000             1000
fffffd7ffefca000 fffffd7ffefcb000             1000
fffffd7ffefd0000 fffffd7ffefe0000            10000
fffffd7ffeff0000 fffffd7ffeff6000             6000
fffffd7fff000000 fffffd7fff001000             1000
fffffd7fff010000 fffffd7fff0d0000            c0000 /lib/amd64/libnsl.so.1
fffffd7fff0d0000 fffffd7fff0d7000             7000 /lib/amd64/libnsl.so.1
fffffd7fff0d7000 fffffd7fff0e0000             9000 /lib/amd64/libnsl.so.1
fffffd7fff0f0000 fffffd7fff101000            11000 /lib/amd64/libsocket.so.1
fffffd7fff111000 fffffd7fff112000             1000 /lib/amd64/libsocket.so.1
fffffd7fff112000 fffffd7fff113000             1000 /lib/amd64/libsocket.so.1
fffffd7fff120000 fffffd7fff121000             1000
fffffd7fff130000 fffffd7fff131000             1000
fffffd7fff140000 fffffd7fff1a5000            65000 /usr/X11/lib/X11/xserver/amd64/libXfont.so.1
fffffd7fff1b5000 fffffd7fff1b8000             3000 /usr/X11/lib/X11/xserver/amd64/libXfont.so.1
fffffd7fff1b8000 fffffd7fff1bb000             3000 /usr/X11/lib/X11/xserver/amd64/libXfont.so.1
fffffd7fff1c0000 fffffd7fff355000           195000 /lib/amd64/libc.so.1
fffffd7fff365000 fffffd7fff371000             c000 /lib/amd64/libc.so.1
fffffd7fff371000 fffffd7fff373000             2000 /lib/amd64/libc.so.1
fffffd7fff375000 fffffd7fff387000            12000
fffffd7fff390000 fffffd7fff391000             1000
fffffd7fff3a0000 fffffd7fff3a1000             1000
fffffd7fff3a3000 fffffd7fff3eb000            48000 /lib/amd64/ld.so.1
fffffd7fff3fb000 fffffd7fff3fe000             3000 /lib/amd64/ld.so.1
fffffd7fff3fe000 fffffd7fff3ff000             1000 /lib/amd64/ld.so.1
fffffd7fffd72000 fffffd7fffe00000            8e000 [ stack ]

core에서 나오는 메모리 맵에서는 base map 400000에서 시작하니, d5c4...  영역을 봤더니
libglx.so 가 맞군요.

NVidia driver 화일이군요... Nvidia를 다시 설치해야 겠군요.

pkg search libglx.so 로 어떤 패키지인가 검색해봤더니... 예상대로
pkg:/SUNWxorg-mesa@7.2-0.111

어떻게 이 패키지만 업데이트 하나 고민하고 있습니다.  차라리 앞으로 롤백을 해야 할까요? 흠...

opensolaris 2009년 6월 에디션의 부팅 직후 메모리 맵입니다.

2G 메모리를 가지고 있는 소니 랩탑에서 오픈 솔라리스 2009년 6월 에디션을 부팅한 직후의 메모리 상태입니다.
ZFS file data 영역과 Anon 영역이 670MB 나 되는 군요... 메모리를 무지 쓰고 있군요... 흠...

/etc/system에서 maxusers=64로 낮춘 상태에서의 로딩입니다. default는 2038인가 정도 했던 것 같습니다.

~# echo "::memstat" | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     132109               516   25%
ZFS File Data               82974               324   16%
Anon                        89869               351   17%
Exec and libs                3657                14    1%
Page cache                  20857                81    4%
Free (cachelist)            10609                41    2%
Free (freelist)            181731               709   35%

Total                      521806              2038
Physical                   521805              2038

~# echo "::kmastat" | mdb -k
cache                        buf    buf    buf    memory     alloc alloc
name                        size in use  total    in use   succeed  fail
------------------------- ------ ------ ------ ---------- --------- -----
kmem_magazine_1               16   3356   3514     57344B      3356     0
kmem_magazine_3               32   3550   3625    118784B      3573     0
kmem_magazine_7               64   6246   6262    413696B      6246     0
kmem_magazine_15             128   3707   3720    491520B      3879     0
kmem_magazine_31             256      0    180     49152B       166     0
kmem_magazine_47             384      0    140     57344B       131     0
kmem_magazine_63             512      0    161     94208B       155     0
kmem_magazine_95             768     54     55     45056B        54     0
kmem_magazine_143           1152      0      0         0B         0     0
kmem_slab_cache               72  78113  78155   5820416B     78121     0
kmem_bufctl_cache             24 273145 273212   6701056B    273195     0
kmem_bufctl_audit_cache      192      0      0         0B         0     0
kmem_va_4096                4096  46354  46368 189923328B     46368     0
kmem_va_8192                8192   1407   1408  11534336B      1407     0
kmem_va_12288              12288    275    280   3670016B       275     0
kmem_va_16384              16384  10924  10928 179044352B     10924     0
kmem_va_20480              20480    360    360   7864320B       360     0
kmem_va_24576              24576     66     70   1835008B        66     0
kmem_va_28672              28672    500    500  16384000B       500     0
kmem_va_32768              32768     45     48   1572864B        45     0
kmem_alloc_8                   8  32208  32695    266240B    369157     0
kmem_alloc_16                 16  20271  22088    360448B    133928     0
kmem_alloc_24                 24  10139  11690    286720B     70094     0
kmem_alloc_32                 32  19903  20875    684032B   2369647     0
kmem_alloc_40                 40  19053  22600    925696B    477081     0
kmem_alloc_48                 48   6254   8466    417792B    137913     0
kmem_alloc_56                 56  10852  13348    770048B   2372840     0
kmem_alloc_64                 64  10946  13020    860160B   1266893     0
kmem_alloc_80                 80  15931  16150   1323008B    121931     0
kmem_alloc_96                 96  18972  20418   2039808B     96164     0
kmem_alloc_112               112 112438 113435  13275136B    130292     0
kmem_alloc_128               128   6882  11439   1511424B    262361     0
kmem_alloc_160               160   3587   3875    634880B     16150     0
kmem_alloc_192               192    533    580    118784B   2287251     0
kmem_alloc_224               224   1191   7089   1708032B     74335     0
kmem_alloc_256               256   5266   6330   1728512B    153107     0
kmem_alloc_320               320   4226   7956   2715648B    214767     0
kmem_alloc_384               384    562    570    233472B     12381     0
kmem_alloc_448               448    143    152     77824B      6150     0
kmem_alloc_512               512    386    392    200704B      5727     0
kmem_alloc_640               640    364    378    258048B      5083     0
kmem_alloc_768               768     86     95     77824B     37586     0
kmem_alloc_896               896     74     81     73728B      2172     0
kmem_alloc_1152             1152    261    280    327680B    247575     0
kmem_alloc_1344             1344     80     90    122880B     82802     0
kmem_alloc_1600             1600     26     30     49152B       921     0
kmem_alloc_2048             2048    456    478    978944B     10930     0
kmem_alloc_2688             2688    695    711   1941504B      4488     0
kmem_alloc_4096             4096    217    865   3543040B     15630     0
kmem_alloc_8192             8192    649    653   5349376B     14609     0
kmem_alloc_12288           12288     64     69    847872B      1259     0
kmem_alloc_16384           16384     39     41    671744B       360     0
kmem_alloc_24576           24576     46     51   1253376B       211     0
kmem_alloc_32768           32768     33     36   1179648B       128     0
streams_mblk                  64   2770   2790    184320B    266835     0
streams_dblk_16              128    105    186     24576B    181534     0
streams_dblk_80              192    765   1040    212992B    573591     0
streams_dblk_144             256      1     90     24576B     13159     0
streams_dblk_208             320    211    480    163840B     12046     0
streams_dblk_272             384      0    280    114688B     40931     0
streams_dblk_336             448      1     16      8192B      1503     0
streams_dblk_528             640      0     18     12288B      3222     0
streams_dblk_1040           1152      0     49     57344B      3243     0
streams_dblk_1488           1600      1     10     16384B       594     0
streams_dblk_1936           2048      2     12     24576B       622     0
streams_dblk_2576           2688      0     12     32768B      2884     0
streams_dblk_3856           3968      0      4     16384B       508     0
streams_dblk_8192            112    511    558     73728B     23371     0
streams_dblk_12048         12160      0      4     49152B       396     0
streams_dblk_16384           112      0     31      4096B       147     0
streams_dblk_20240         20352      0      4     81920B       218     0
streams_dblk_24576           112      0     31      4096B        33     0
streams_dblk_28432         28544      0      3     86016B       263     0
streams_dblk_32768           112      0     31      4096B       186     0
streams_dblk_36624         36736      0      3    110592B        39     0
streams_dblk_40960           112      0     31      4096B        36     0
streams_dblk_44816         44928      0      2     90112B         2     0
streams_dblk_49152           112      0     31      4096B         7     0
streams_dblk_53008         53120      0      2    106496B         2     0
streams_dblk_57344           112      0     31      4096B         3     0
streams_dblk_61200         61312      0      2    122880B         4     0
streams_dblk_65536           112      0     31      4096B       803     0
streams_dblk_69392         69504      0      0         0B         0     0
streams_dblk_73728           112      0      0         0B         0     0
streams_dblk_esb             112      0     31      4096B       614     0
streams_fthdr                408      0      0         0B         0     0
streams_ftblk                376      0      0         0B         0     0
multidata                    248      0      0         0B         0     0
multidata_pdslab            7112      0      0         0B         0     0
multidata_pattbl              32      0      0         0B         0     0
log_cons_cache                48     17     83      4096B        22     0
taskq_ent_cache               56   1847   2698    155648B      8463     0
taskq_cache                  216    103    126     28672B       140     0
kmem_io_2G_128               128      2     31      4096B       489     0
kmem_io_2G_256               256      4     15      4096B        23     0
kmem_io_2G_512               512      4      8      4096B         4     0
kmem_io_2G_1024             1024      0      4      4096B        12     0
kmem_io_2G_2048             2048      2      2      4096B         3     0
kmem_io_2G_4096             4096      3      4     16384B        44     0
kmem_io_16M_128              128      0      0         0B         0     0
kmem_io_16M_256              256      0      0         0B         0     0
kmem_io_16M_512              512      0      0         0B         0     0
kmem_io_16M_1024            1024      0      0         0B         0     0
kmem_io_16M_2048            2048      0      0         0B         0     0
kmem_io_16M_4096            4096      0      0         0B         0     0
id32_cache                    32      0    125      4096B      3543     0
bp_map_4096                 4096      0      0         0B         0     0
bp_map_8192                 8192      0      0         0B         0     0
bp_map_12288               12288      0      0         0B         0     0
bp_map_16384               16384      0      0         0B         0     0
bp_map_20480               20480      0      0         0B         0     0
bp_map_24576               24576      0      0         0B         0     0
bp_map_28672               28672      0      0         0B         0     0
bp_map_32768               32768      0      0         0B         0     0
htable_t                      72   2767   2805    208896B     17748     0
hment_t                       64 138049 142600   9420800B    980866     0
hat_t                        176    105    110     20480B      1791     0
HatHash                     4096    105    108    442368B      1975     0
segkp_4096                  4096     97    112    458752B      1180     0
segkp_8192                  8192      0      0         0B         0     0
segkp_12288                12288      0      0         0B         0     0
segkp_16384                16384      0      0         0B         0     0
segkp_20480                20480      0      0         0B         0     0
umem_np_4096                4096      0     32    131072B       577     0
umem_np_8192                8192      0     16    131072B       494     0
umem_np_12288              12288      0     10    131072B       421     0
umem_np_16384              16384      0      8    131072B        73     0
umem_np_20480              20480      0      0         0B         0     0
umem_np_24576              24576      0      5    131072B        31     0
umem_np_28672              28672      0      0         0B         0     0
umem_np_32768              32768      0      4    131072B        39     0
mod_hash_entries              24   1300   1336     32768B     11890     0
ipp_mod                      304      0      0         0B         0     0
ipp_action                   368      0      0         0B         0     0
ipp_packet                    64      0      0         0B         0     0
seg_cache                     96  15615  16031   1601536B    130812     0
seg_pcache                   104      0     38      4096B         3     0
fnode_cache                  176      6     20      4096B        27     0
pipe_cache                   320    134    144     49152B       787     0
snode_cache                  152    438    468     73728B     24448     0
dv_node_cache                176    406    506     94208B       884     0
mac_impl_cache              1776      3      9     16384B         3     0
mac_ring_cache               136      0      0         0B         0     0
flow_entry_cache            3032     13     16     49152B        17     0
flow_tab_cache               216      3     18      4096B         3     0
mac_soft_ring_cache          344      9     10      4096B         9     0
mac_srs_cache               1528      6      8     12288B         6     0
mac_bcast_grp_cache           80     10     50      4096B        14     0
mac_client_impl_cache        728      3     11      8192B         3     0
mac_promisc_impl_cache       112      0      0         0B         0     0
dls_link_cache               336      3     11      4096B         3     0
dls_devnet_cache             320      3     12      4096B         3     0
sdev_node_cache              256   1510   1515    413696B      1903     0
dev_info_node_cache          640    152    162    110592B      1427     0
ndi_fm_entry_cache            32      6    125      4096B        72     0
thread_cache                 872    296    306    278528B      2149     0
lwp_cache                   1424    285    288    442368B       718     0
turnstile_cache               64    619    992     65536B      4074     0
tslabel_cache                 48      2     83      4096B         2     0
cred_cache                   188    147    160     32768B      1465     0
rctl_cache                    48   1434   1494     73728B     16397     0
rctl_val_cache                64   2923   2976    196608B     34315     0
task_cache                   128     41     62      8192B       121     0
kmem_defrag_cache            192      1     20      4096B         1     0
kmem_move_cache               56      0      0         0B         0     0
rootnex_dmahdl              2536   2200   2216   5672960B     64335     0
timeout_request              128      1     31      4096B         1     0
cyclic_id_cache               72      9     55      4096B         9     0
callout_cachebabecafe         80    374    403     53248B       374     0
callout_lcachebabecafe        48   9848   9858    651264B      9848     0
dnlc_space_cache              24      0      0         0B         0     0
vfs_cache                    208     70     76     16384B        76     0
vn_cache                     200  26478  26490   7233536B     36324     0
vsk_anchor_cache              40     40    100      4096B        44     0
file_cache                    56   1366   1420     81920B     71295     0
stream_head_cache            384    956    970    397312B      4600     0
queue_cache                  656   1718   1746   1191936B      7850     0
syncq_cache                  160     12     25      4096B        33     0
qband_cache                   64      2     62      4096B         2     0
linkinfo_cache                48     11     83      4096B        12     0
ciputctrl_cache              128      0      0         0B         0     0
serializer_cache              64    298    310     20480B      1959     0
as_cache                     224    104    119     28672B      1790     0
marker_cache                 120      0     33      4096B       556     0
anon_cache                    48  96891  96944   4784128B    271601     0
anonmap_cache                112  11179  11445   1339392B     69975     0
segvn_cache                  168  15565  15962   2842624B    111959     0
segvn_szc_cache1            4096      0      0         0B         0     0
flk_edges                     48      0     83      4096B         1     0
fdb_cache                    104      0      0         0B         0     0
timer_cache                  136      0      0         0B         0     0
vmu_bound_cache               32      0      0         0B         0     0
vmu_object_cache              32      0      0         0B         0     0
physio_buf_cache             248      0      0         0B         0     0
kcf_sreq_cache                48      0      0         0B         0     0
kcf_areq_cache               288      0      0         0B         0     0
kcf_context_cache            112      0      0         0B         0     0
clnt_clts_endpnt_cache        88      0      0         0B         0     0
zio_cache                    720      0   3553   2646016B    485479     0
zio_link_cache                48      0   3652    180224B    204521     0
zio_buf_512                  512  73559  73928  37851136B     77161     0
zio_data_buf_512             512   1212   1232    630784B      2843     0
zio_buf_1024                1024   1484   1496   1531904B      5263     0
zio_data_buf_1024           1024    105    112    114688B       881     0
zio_buf_1536                1536    309    328    503808B      1168     0
zio_data_buf_1536           1536    124    128    196608B       202     0
zio_buf_2048                2048    178    186    380928B      1084     0
zio_data_buf_2048           2048    105    110    225280B       856     0
zio_buf_2560                2560    133    144    368640B       946     0
zio_data_buf_2560           2560     77     88    225280B       152     0
zio_buf_3072                3072    103    112    344064B       881     0
zio_data_buf_3072           3072     71     80    245760B       726     0
zio_buf_3584                3584     77    128    458752B      2146     0
zio_data_buf_3584           3584     57     64    229376B       119     0
zio_buf_4096                4096    141    145    593920B      4036     0
zio_data_buf_4096           4096     45     48    196608B       235     0
zio_buf_5120                5120     62     80    409600B      4101     0
zio_data_buf_5120           5120     65     68    348160B       320     0
zio_buf_6144                6144     58    156    958464B      1681     0
zio_data_buf_6144           6144     84     88    540672B       297     0
zio_buf_7168                7168     50     60    430080B       590     0
zio_data_buf_7168           7168     54     56    401408B       163     0
zio_buf_8192                8192     20     25    204800B      1293     0
zio_data_buf_8192           8192  35665  36578 299646976B     37990     0
zio_buf_10240              10240     27     36    368640B       430     0
zio_data_buf_10240         10240     30     34    348160B        49     0
zio_buf_12288              12288      7     13    159744B       463     0
zio_data_buf_12288         12288     21     23    282624B        33     0
zio_buf_14336              14336     15    876  12558336B     10537     0
zio_data_buf_14336         14336     43     46    659456B        67     0
zio_buf_16384              16384  10869  10882 178290688B     12124     0
zio_data_buf_16384         16384     25     27    442368B        30     0
zio_buf_20480              20480     12     22    450560B       422     0
zio_data_buf_20480         20480     46     49   1003520B       149     0
zio_buf_24576              24576      7     15    368640B       752     0
zio_data_buf_24576         24576     37     40    983040B        54     0
zio_buf_28672              28672      5     15    430080B       281     0
zio_data_buf_28672         28672     26     26    745472B        26     0
zio_buf_32768              32768      1      9    294912B       349     0
zio_data_buf_32768         32768     31     31   1015808B        31     0
zio_buf_36864              36864      0      6    221184B       118     0
zio_data_buf_36864         36864     23     24    884736B        25     0
zio_buf_40960              40960      3     25   1024000B       325     0
zio_data_buf_40960         40960     15     15    614400B        16     0
zio_buf_45056              45056      6     10    450560B        91     0
zio_data_buf_45056         45056      7      7    315392B         7     0
zio_buf_49152              49152      1      6    294912B       199     0
zio_data_buf_49152         49152     10     10    491520B        10     0
zio_buf_53248              53248      0      3    159744B        28     0
zio_data_buf_53248         53248      7      9    479232B        13     0
zio_buf_57344              57344      1      7    401408B       135     0
zio_data_buf_57344         57344     10     10    573440B        10     0
zio_buf_61440              61440      0      4    245760B        22     0
zio_data_buf_61440         61440     14     14    860160B        28     0
zio_buf_65536              65536    161    167  10944512B      9529     0
zio_data_buf_65536         65536      6      6    393216B         6     0
zio_buf_69632              69632      0      3    208896B        10     0
zio_data_buf_69632         69632      3      3    208896B         3     0
zio_buf_73728              73728      1      6    442368B       113     0
zio_data_buf_73728         73728      8      8    589824B         8     0
zio_buf_77824              77824      0      5    389120B        32     0
zio_data_buf_77824         77824      4      4    311296B         4     0
zio_buf_81920              81920      0      4    327680B       114     0
zio_data_buf_81920         81920      4      5    409600B        24     0
zio_buf_86016              86016      0      3    258048B        12     0
zio_data_buf_86016         86016      3      3    258048B         3     0
zio_buf_90112              90112      0      4    360448B       121     0
zio_data_buf_90112         90112      5      5    450560B         5     0
zio_buf_94208              94208      0      2    188416B         4     0
zio_data_buf_94208         94208      1      4    376832B        41     0
zio_buf_98304              98304      1      5    491520B        77     0
zio_data_buf_98304         98304      1      1     98304B         1     0
zio_buf_102400            102400      0      2    204800B         7     0
zio_data_buf_102400       102400      2      2    204800B         2     0
zio_buf_106496            106496      1      4    425984B        49     0
zio_data_buf_106496       106496      0      0         0B         0     0
zio_buf_110592            110592      0      1    110592B         2     0
zio_data_buf_110592       110592      6      6    663552B         6     0
zio_buf_114688            114688      0      5    573440B        86     0
zio_data_buf_114688       114688      2      2    229376B         2     0
zio_buf_118784            118784      0      3    356352B         8     0
zio_data_buf_118784       118784      3      3    356352B         3     0
zio_buf_122880            122880      1      5    614400B        64     0
zio_data_buf_122880       122880      1      2    245760B         9     0
zio_buf_126976            126976      1      4    507904B         9     0
zio_data_buf_126976       126976      0      1    126976B         4     0
zio_buf_131072            131072      1     37   4849664B      3145     0
zio_data_buf_131072       131072    231    289  37879808B      2137     0
dmu_buf_impl_t               192 116077 117380  24039424B    124235     0
dnode_t                      664  70505  70638  48222208B     71003     0
arc_buf_hdr_t                168  57125  57132  10174464B     62433     0
arc_buf_t                     48  55775  56689   2797568B     65108     0
zil_lwb_cache                200      2   1020    208896B      1927     0
zfs_znode_cache              192  20087  22720   4653056B     81883     0
ip_minor_arena_sa_1            1     26     64        64B       309     0
ip_minor_arena_la_1            1     69    128       128B        71     0
ip_conn_cache                504      1      8      4096B        27     0
tcp_conn_cache              1864     55     62    126976B       113     0
udp_conn_cache              1248     29     36     49152B       712     0
rawip_conn_cache            1168      2     10     12288B         2     0
rts_conn_cache               616      7     12      8192B        11     0
ip_helper_stream_cache        32     67    124      8192B       547     0
ire_cache                    336     38     44     16384B        65     0
rt_entry                     168      2     23      4096B         2     0
radix_mask                    32      1    125      4096B         1     0
radix_node                   120      1     33      4096B         1     0
ipsec_actions                 72      0      0         0B         0     0
ipsec_selectors               80      0      0         0B         0     0
ipsec_policy                  72      0      0         0B         0     0
ipsec_info                   352      0     11      4096B        78     0
tcp_timercache                88     65     90      8192B        66     0
tcp_sack_info_cache           80      2     50      4096B        42     0
tcp_iphc_cache               120     55     66      8192B       111     0
squeue_cache                 168      3     20      4096B         3     0
sctp_conn_cache             2264      1      9     20480B         1     0
sctp_faddr_cache             176      0      0         0B         0     0
sctp_set_cache                24      0      0         0B         0     0
sctp_ftsn_set_cache           16      0      0         0B         0     0
ire_gw_secattr_cache          32      0      0         0B         0     0
socket_cache                 528     67     75     40960B       553     0
socktpi_cache                872      0      9      8192B         1     0
socktpi_unix_cache           872    525    540    491520B      1942     0
sock_sod_cache               680      0      0         0B         0     0
drv_secobj_cache             296      5     13      4096B         5     0
dld_str_cache                288      7     13      4096B       457     0
process_cache               3168    106    117    372736B      1099     0
exacct_object_cache           40      0      0         0B         0     0
hal0_cache                   464      0      8      4096B         1     0
fctl_cache                   112      0      0         0B         0     0
kssl_cache                  1608      0      0         0B         0     0
tl_cache                     432    547    567    258048B      2088     0
keysock_1                      1      0      0         0B         0     0
spdsock_1                      1      0     64        64B         1     0
namefs_inodes_1                1     26     64        64B        28     0
port_cache                    80   2515   6800    557056B      7013     0
lnode_cache                   32      1    125      4096B         1     0
pty_map                       64      3     62      4096B         3     0
ufs_inode_cache              368      0      0         0B         0     0
directio_buf_cache           272      0      0         0B         0     0
lufs_save                     24      0      0         0B         0     0
lufs_bufs                    256      0      0         0B         0     0
lufs_mapentry_cache          112      0      0         0B         0     0
Hex0xffffff0148631658_minor_1      1      0      0         0B         0     0
Hex0xffffff0149d75548_minor_1      1      0      0         0B         0     0
dtrace_state_cache         16384      0      0         0B         0     0
authkern_cache                72      0     55      4096B         7     0
authnone_cache                72      0      0         0B         0     0
authloopback_cache            72      0      0         0B         0     0
authdes_cache_handle          80      0      0         0B         0     0
rnode_cache                  656      2      6      4096B         2     0
nfs_access_cache              56      1     71      4096B         1     0
client_handle_cache           32      1    125      4096B         1     0
rnode4_cache                 968      0      0         0B         0     0
svnode_cache                  40      0      0         0B         0     0
nfs4_access_cache             56      0      0         0B         0     0
client_handle4_cache          32      0      0         0B         0     0
nfs4_ace4vals_cache           48      0      0         0B         0     0
nfs4_ace4_list_cache         264      0      0         0B         0     0
NFS_idmap_cache               48      0      0         0B         0     0
lm_vnode                     184      0      0         0B         0     0
lm_xprt                       32      0      0         0B         0     0
lm_sysid                     160      0      0         0B         0     0
lm_client                    128      0      0         0B         0     0
lm_async                      32      0      0         0B         0     0
lm_sleep                      96      0      0         0B         0     0
lm_config                     80      2     50      4096B         2     0
vnic_cache                   112      0      0         0B         0     0
aggr_port_cache              936      0      0         0B         0     0
aggr_grp_cache              2152      0      0         0B         0     0
crypto_session_cache         104      0      0         0B         0     0
fcsm_job_cache               104      0      0         0B         0     0
sppptun_map                  432      0      0         0B         0     0
------------------------- ------ ------ ------ ---------- --------- -----
Total [hat_memload]                              9629696B    998614     0
Total [kmem_msb]                                13852672B    368877     0
Total [kmem_va]                                411828224B     59945     0
Total [kmem_default]                           433041408B  14276864     0
Total [kmem_io_2G]                                 36864B       575     0
Total [umem_np]                                   786432B      1635     0
Total [id32]                                        4096B      3543     0
Total [zfs_file_data_buf]                      355504128B     47590     0
Total [segkp]                                     458752B      1180     0
Total [ip_minor_arena_sa]                             64B       309     0
Total [ip_minor_arena_la]                            128B        71     0
Total [spdsock]                                       64B         1     0
Total [namefs_inodes]                                 64B        28     0
------------------------- ------ ------ ------ ---------- --------- -----

vmem                         memory     memory    memory     alloc alloc
name                         in use      total    import   succeed  fail
------------------------- ---------- ----------- ---------- --------- -----
heap                      574271488B 1092953833472B         0B     36057     0
    vmem_metadata          17424384B   17432576B  17432576B      4032     0
        vmem_seg           16166912B   16166912B  16166912B      3932     0
        vmem_hash           1006080B    1007616B   1007616B        52     0
        vmem_vmem            258440B     286256B    249856B        79     0
    static                        0B          0B         0B         0     0
        static_alloc              0B          0B         0B         0     0
    hat_memload             9629696B    9629696B   9629696B      2357     0
    kstat                    309608B     315392B    249856B      1006     0
    kmem_metadata          16400384B   16515072B  16515072B      3521     0
        kmem_msb           13852672B   13852672B  13852672B      3387     0
        kmem_cache           249984B     307200B    307200B       424     0
        kmem_hash           2231808B    2240512B   2240512B       263     0
    kmem_log                  65952B      69632B     69632B         6     0
    kmem_firewall_va       34492416B   34492416B  34492416B       270     0
        kmem_firewall             0B          0B         0B         0     0
        kmem_oversize      34413115B   34492416B  34492416B       270     0
    mod_sysfile                 744B       4096B      4096B        18     0
    kmem_va               436310016B  436310016B 436310016B      3472     0
        kmem_default      433053696B  433053696B 433053696B     60274     0
    kmem_io_2G                36864B      36864B     36864B      3414     0
    kmem_io_16M                   0B          0B         0B         0     0
    bp_map                        0B          0B         0B         0     0
    umem_np                  786432B     786432B    786432B        86     0
    ksyms                   2871496B    2899968B   2899968B       317     0
    ctf                     1395410B    1462272B   1462272B       312     0
heap_core                   3092480B  931115008B         0B        50     0
    heaptext               15360000B   67108864B         0B       220     0
        module_text        15583192B   15872000B  15355904B       317     0
    id32                       4096B       4096B      4096B         1     0
    module_data             3302669B    3395584B   3088384B       427     0
logminor_space                   33B     262137B         0B        38     0
taskq_id_arena                   56B 2147483647B         0B        81     0
zfs_file_data             355504128B 2139095040B         0B     37662     0
    zfs_file_data_buf     355504128B  355504128B 355504128B     37662     0
device                      1597440B 1073741824B         0B     33135     0
segkp                      15523840B 2139095040B         0B      2514     0
mac_minor_ids                    15B     261142B         0B        16     0
rctl_ids                         35B      32767B         0B        35     0
zoneid_space                      0B       9998B         0B         0     0
taskid_space                     41B     999999B         0B       120     0
pool_ids                          0B     999998B         0B         0     0
contracts                        44B 2147483646B         0B       135     0
ip_minor_arena_sa                64B     262140B         0B         1     0
ip_minor_arena_la               128B 4294705152B         0B         2     0
lib_va_32                  10252288B 2138816512B         0B        20     0
tl_minor_space                  547B     262138B         0B      2071     0
keysock                           0B 4294967295B         0B         0     0
spdsock                          64B 4294967295B         0B         1     0
namefs_inodes                    64B      65536B         0B         1     0
lib_va_64                 119177216B 137988696055808B         0B       131     0
 
devfsadm_event_channel            1B        101B         0B         1     0
devfsadm_event_channel            1B          2B         0B         1     0
ptms_minor                        3B         16B         0B         3     0
Hex0xffffff0148631658_minor         0B 4294967294B         0B         0     0
Hex0xffffff0149d75548_minor         0B 4294967294B         0B         0     0
syseventd_channel                 3B        101B         0B         3     0
syseventd_channel                 1B          2B         0B         1     0
syseventconfd_door                0B        101B         0B         0     0
syseventconfd_door                1B          2B         0B         1     0
dtrace                         1710B 4294967295B         0B      5295     0
dtrace_minor                      0B 4294967293B         0B         0     0
lmsysid_space                     0B      16383B         0B         0     0
aggr_portids                      0B      65534B         0B         0     0
aggr_key_ids                      0B      64535B         0B         0     0
crypto                            0B         16B         0B         1     0
ipnet_minor_space                 4B     262141B         0B         4     0
ds_minors                         0B     262140B         0B         0     0
logdmux_minor                     0B        256B         0B         0     0
sppptun_minor                     0B         16B         0B         0     0
shmids                            1B         64B         0B        31     0