IMM boot problem
I purchased a x3650 m3 2u IBM server, which on boot would run the fans constantly at maximum RPM and report a failure to connect to IMM. After several minutes (~15min) of reporting a failure to connect to imm the system would boot into UEFI, but no bios settings could be saved except for boot volumes.
I tried to use the BOMC (Bootable Media Creator) to flash a new version of IMM, or at least recover it. However this mechanisim relies on linux detecting a usb ethernet driver to talk to the IMM module. It appeared that the IMM was dead.
After looking around the internals of the server I determined that the IMM was a VSC452 - but there is only a product information datasheet avialable - not useful at all. However from the datasheet it appears that the VSC452 has a mips processor - one that possibly runs linux.
So I binwalked the IMM firmware binary on the BoMC and found uboot, uImage and 2 jffs2 partitions confirming there was mips processor that runs linux.
Next job was to gain access to serial to inspect the boot loader - there were some pads on the motherboarded labeled iBMC_UART_CONN. I probbed these with a mutlimeter and finally a logic analyser to find out which pins are VCC, GND and TX. RX was found by trial and error. The logic analyser also determined the baud rate and the multimeter confirmed it was using 5V. The image below shows some wires I attached to connect to a USB to serial converter.
Firmware
TFTP Server Setup
Extracting firmware
The firmware I used was an older version - ibm_fw_imm_yuooe3c-1.33_linux_32-64.bin
Extract the firmware with binwalk:
binwalk -Me ibm_fw_imm_yuoog6c-1.44_linux_32-64.bin
-e extract
-M recursively extract
Copy files from the extracted bootfs:
cd _ibm_fw_imm_yuoog6c-1.44_linux_32-64.bin.extracted/_ibmc.tgz.extracted/_0.extracted/_bootfs.jffs2.extracted/jffs2-root/fs_1
cp lib/ld-uClibc-0.9.29.so /tftpboot/ubiimages/ld-uClibc.so.0
cp lib/libuClibc-0.9.29.so /tftpboot/ubiimages/libc.so.0
cp bin/ubimkvol /tftpboot/ubiimages/ubimkvol
cp bin/ubirmvol /tftpboot/ubiimages/ubirmvol
cp bin/ubiupdatevol /tftpboot/ubiimages/ubiupdatevol
Copy the bootfs, rootfs and uImage
cd _ibm_fw_imm_yuoog6c-1.44_linux_32-64.bin.extracted/_ibmc.tgz.extracted/_0.extracted/
cp bootfs.jffs2 /tftpboot/ubiimages/
cp rootfs.jffs2 /tftpboot/ubiimages/
cp uImage /tftpboot/ubiimages/
cp uImage /tftpboot/
Create dummy flash_eraseall:
echo "#!/bin/sh" > /tftpboot/ubiimages/flash_eraseall
Serial Port
Serial Port on Motherboard
Between the two usb ports and VGA connector there is an unpopulated connector labeled iBMC_UART_CONN, show below
The pins are as following, in the order from the USB port to the VGA connector:
--- 123456 0.---.0
| | ,'''''', | | | |
| | |_-----_|
USB VGA
2 - RX
4 - TX
6 - GND
U-Boot
Enable IBMC_FORCE_UPDATE_N (sw3,4 on) then power the unit and "FORCE UPDATE ENABLED" should appear as shown in the output below. This enables tftp recovery by booting a uImage with ramfs.
IMM UBoot 1.1.4 (Dec 1 2011 - 07:36:30)
-> IMM revision 7 detected
-> Clock set to 300MHz
-> Caches enabled
-> Super Loader
-> Initializing BUS 5..DONE
-> Attempt #1
-> reading i2c..OK
-> validating..OK
-> Default ver wins [04] >= [04]
-> MAGIC# = 4D
-> VERSION = 04
-> waiting for DDR init..OK
-> Available RAM set to 112M
Top of RAM usable for U-Boot at: 87000000
Reserving 169k for U-Boot at: 86fd4000
Reserving 1028k for malloc() at: 86ed3000
Reserving 44 Bytes for Board Info at: 86ed2fd4
Reserving 36 Bytes for Global Data at: 86ed2fb0
Reserving 8k for boot params() at: 86ed0fb0
Stack Pointer at: 86ed0f98
Now running in RAM - U-Boot at: 86fd4000
Flash: 512 kB
-> Detecting Platform ... PLATFORM_ID = 0x00500002
-> gpio[ 0x06 ] = ( 1, 8 ) - POWER_STATUS
-> gpio[ 0x07 ] = ( 2, 11 ) - POWER_CONTROL
-> gpio[ 0x08 ] = ( 2, 1 ) - POWER_LED
-> gpio[ 0x13 ] = ( 4, 0 ) - WYVERN_ALERT
-> gpio[ 0x02 ] = ( 1, 1 ) - HEARTBEAT_LED
-> gpio[ 0x01 ] = ( 1, 4 ) - HALFROM
-> gpio[ 0x05 ] = ( 4, 6 ) - FORCEROM_UPDATE
Nand: Manuf=Hynix ID=0xdc 512 MiB
In: serial
Out: serial
Err: serial
-> Chip Type=452, Revision=7
-> IOR start 0x00000000 00008000
-> SIF5-SIBS was 0x43. Clearing...
-> IOR end 0x00000000 00008000
-> Configuring eth1 for MDIO
-> Configuring eth0 MAC address
-> Configuring eth1 MAC address
Net: eth0, eth1 [PRIME]
Hit any key to stop autoboot: 0
Disabling watchdog
!-> FORCE UPDATE ENABLED
->Configuring eth0 for NCSI
[eth0] NCSI: Package found 0
[eth0] NCSI: Channel found 0
[eth0] NCSI: Initialization complete
-> Attempting to TFTP a kernel uImage
-> Configuring eth1 MAC address
-> Configuring eth0 MAC address
Using eth1 device
TFTP from server 192.168.70.200; our IP address is 192.168.70.125
Filename 'uImage'.
Load address: 0xa1800000
Loading: T
Break into u-boot by either pressing ctrl-c when u-boot is trying to tftp the image or at the "Hit any key to stop autoboot". At the u-boot command line configure the following:
setenv ipaddr
setenv serverip
setenv bootargs AUTOMTD=1 ETH1=
saveenv
resetimm chip
The unit will reset and be prepared to press S to interrupt the kernel boot.
...
!-> FORCE UPDATE ENABLED
-> Attempting to TFTP a kernel uImage
-> Configuring eth1 MAC address
-> Configuring eth0 MAC address
Using eth1 device
TFTP from server 192.168.1.148; our IP address is 192.168.1.33
Filename 'uImage'.
Load address: 0xa1800000
Loading: TT#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################
done
Bytes transferred = 1828603 (1be6fb hex)
-> Attempting to boot...## Booting image at a1800000 ...
Image Name: IMM Linux-2.6.16.46-377
Image Type: MIPS Linux Kernel Image (gzip compressed)
Data Size: 1828539 Bytes = 1.7 MB
Load Address: a1000000
Entry Point: 812fb000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
Starting kernel ...
IMM LINUX started...
Linux version 2.6.16.46-377 (geeko@buildhost) (gcc version 3.4.4) #2 PREEMPT Thu Dec 21 18:00:53 UTC 2017
CPU revision is: 0001906c
...
Freeing unused kernel memory: 536k freed
Algorithmics/MIPS FPU Emulator v1.5
==> Running /init (stage 0)
-> Autobooting MTD
-> Boot to [M]TD, [S]hell or [B]urn NAND?
-> Drop Shell selected
-> Dropping to mini-shell. Type 'exit' to restart init
(IMM-minishell) [~] >
At the shell run the following commands:
/etc/init.d/ifs.burn
Then press Y to proceed - this process should fail with error shown below, but the libraries and utilities have been tftped.
UBI Library Error at ubi_mkvol: ioctl returned -1 errno=17
Cannot create volume: File exists
err=-1
Jan 01 00:01:36Unable to create volumes. Perhaps the device is not large enough?
Run the following to obtain the ubirmvol command and remove all the ubi volumes:
tftp -b 2048 -g -l /bin/ubirmvol -r ubiimages/ubirmvol 192.168.1.148
chmod +x /bin/ubirmvol
ubirmvol -d0 -n1
ubirmvol -d0 -n2
ubirmvol -d0 -n3
ubirmvol -d0 -n4
ubirmvol -d0 -n5
ubirmvol -d0 -n6
ubirmvol -d0 -n7
ubirmvol -d0 -n8
Then reboot and break into u-boot again and run:
setenv bootargs AUTOMTD=1 ETH1= TFTP= BOOTVOL=1
setenv bootvolume=1
savenv
bootbmc
Interrupt the kernel with B this time and press Y to initiate flashing, for example:
> setenv bootvolume=1
Disabling watchdog
> setenv bootargs AUTOMTD=1 ETH1=192.168.1.33 TFTP=192.168.1.148 BOOTVOL1=1
Disabling watchdog
> saveenv
Disabling watchdog
Saving Environment to Flash...
Un-Protected 1 sectors
Erasing Flash...[s_first=7 s_last=7]
Erased 1 sectors
Writing to Flash... [src=0x86ec0b18 addr=0xbe070000 cnt=65536]
done
Protected 1 sectors
> bootbmc
Disabling watchdog
!-> FORCE UPDATE ENABLED
-> Attempting to TFTP a kernel uImage
-> Configuring eth1 MAC address
-> Configuring eth0 MAC address
Using eth1 device
TFTP from server 192.168.1.148; our IP address is 192.168.1.3
Filename 'uImage'.
Load address: 0xa1800000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################
done
Bytes transferred = 1828603 (1be6fb hex)
-> Attempting to boot...## Booting image at a1800000 ...
Image Name: IMM Linux-2.6.16.46-377
Image Type: MIPS Linux Kernel Image (gzip compressed)
Data Size: 1828539 Bytes = 1.7 MB
Load Address: a1000000
Entry Point: 812fb000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
Starting kernel ...
IMM LINUX started...
Linux version 2.6.16.46-377 (geeko@buildhost) (gcc version 3.4.4) #2 PREEMPT Thu Dec 21 18:00:53 UTC 2017
CPU revision is: 0001906c
Determined physical RAM map:
memory: 003a8000 @ 01000000 (reserved)
memory: 06458000 @ 013a8000 (usable)
memory: 00800000 @ 07800000 (reserved)
Built 1 zonelists
Kernel command line: AUTOMTD=1 ETH1=192.168.1.33 TFTP=192.168.1.148 BOOTVOL1=1 FORCEUP=1 MANMODE=1
2 MIPSR2 register sets available
Primary instruction cache 16kB, physically tagged, 2-way, linesize 16 bytes.
Primary data cache 16kB, 2-way, linesize 16 bytes.
Synthesized TLB refill handler (20 instructions).
Synthesized TLB load handler fastpath (32 instructions).
Synthesized TLB store handler fastpath (32 instructions).
Synthesized TLB modify handler fastpath (31 instructions).
PID hash table entries: 512 (order: 9, 8192 bytes)
CPU frequency 300.00 MHz
Using 150.000 MHz high precision timer.
VSC452 general purpose serial console setup!
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 101632k/102752k available (2563k kernel code, 1092k reserved, 484k data, 536k init, 0k highmem)
Security Framework v1.0.0 initialized
Mount-cache hash table entries: 512
Checking for 'wait' instruction... available.
NET: Registered protocol family 16
JFFS2 version 2.2. (NAND) (SUMMARY) (C) 2001-2003 Red Hat, Inc.
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
Generic RTC Driver v1.07
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
Kernel Build: 2:ibmc51_2:mcp:2017:12:21:175504:5:1:0:
NAND device: Manufacturer ID: 0xad, Chip ID: 0xdc (Hynix NAND 512MiB 3,3V 8-bit)
busw=0, options=0xc0000001
Scanning device for bad blocks
Bad eraseblock 263 at 0x020e0000
Bad eraseblock 1194 at 0x09540000
Bad eraseblock 1608 at 0x0c900000
Bad eraseblock 2167 at 0x10ee0000
Bad eraseblock 2638 at 0x149c0000
Bad eraseblock 3624 at 0x1c500000
Bad eraseblock 3625 at 0x1c520000
Bad eraseblock 3730 at 0x1d240000
cmdlinepart partition parsing not available
Setting MTD NAND partition from 536870912 to 536870912
Using static partition definition
Creating 1 MTD partitions on "vsc452-nand":
0x00000000-0x20000000 : "UBI"
VSC GPUART driver initialized
ttySC0 at MMIO map 0x0 mem 0xb0200300 (irq = 6) is a vsc452 gpuart
Netfilter messages via NETLINK v0.30.
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 2, 16384 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
ip_tables: (C) 2000-2006 Netfilter Core Team
ipt_recent v0.3.1: Stephen Frost . http://snowman.net/projects/ipt_recent/
arp_tables: (C) 2002 David S. Miller
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
NET: Registered protocol family 15
802.1Q VLAN Support v1.8 Ben Greear
All bugs added by David S. Miller
NET: Registered protocol family 27
Freeing unused kernel memory: 536k freed
Algorithmics/MIPS FPU Emulator v1.5
==> Running /init (stage 0)
-> Autobooting MTD
-> Boot to [M]TD, [S]hell or [B]urn NAND?
-> Burn NAND selected
NAND Preparation Script
-----------------------
This script will erase your NAND flash device and populate it
with UBI images stored on a TFTP server.
This script will format the volumes as follows (customer layout):
Volume 1: Kernel 2.0MiB - 17 blocks
Volume 2: Kernel Backup 2.0MiB - 17 blocks
Volume 3: Root Filesystem 50.0MiB - 413 blocks
Volume 4: Root Filesystem Backup 50.0MiB - 413 blocks
Volume 5: R/W FileSystem 40.1MiB - 331 blocks
Volume 6: Staging Area 280.1MiB - 2313 blocks
Volume 7: Boot Filesystem 2.0MiB - 17 blocks
Volume 8: Boot Filesystem Backup 2.0MiB - 17 blocks
PreLayout Block Pool 19.1MiB - 158 blocks
PostLayout Block Pool 48.5MiB - 400 blocks
3.125% UBI Overhead 16.0MiB - n/a
------- -----------
(536870912 bytes) 512.0MiB 4096 blocks
-> Proceed?(Y/N) y
Jan 01 00:00:02 -> Loading ethernet kernel module
Jan 01 00:00:08 -> Looking for TFTP server (192.168.1.148)
Jan 01 00:00:10 -> Found TFTP server on host 192.168.1.148
Jan 01 00:00:10 -> Looking for remote file (ld-uClibc.so.0)
Jan 01 00:00:10 -> File found at TFTP location 192.168.1.148:ubiimages/ld-uClibc.so.0)
Jan 01 00:00:10 -> Looking for remote file (libc.so.0)
Jan 01 00:00:11 -> File found at TFTP location 192.168.1.148:ubiimages/libc.so.0)
Jan 01 00:00:11 -> Looking for remote file (ubimkvol)
Jan 01 00:00:11 -> File found at TFTP location 192.168.1.148:ubiimages/ubimkvol)
Jan 01 00:00:11 -> Looking for remote file (ubiupdatevol)
Jan 01 00:00:11 -> File found at TFTP location 192.168.1.148:ubiimages/ubiupdatevol)
Jan 01 00:00:11 -> Looking for remote file (flash_eraseall)
Jan 01 00:00:11 -> File found at TFTP location 192.168.1.148:ubiimages/flash_eraseall)
Jan 01 00:00:11 -> All common files downloaded
Jan 01 00:00:11 -> Looking for remote file (uImage)
Jan 01 00:00:13 -> File found at TFTP location 192.168.1.148:ubiimages/uImage)
Jan 01 00:00:13 -> Looking for remote file (bootfs.jffs2)
Jan 01 00:00:15 -> File found at TFTP location 192.168.1.148:ubiimages/bootfs.jffs2)
Jan 01 00:00:15 -> Looking for remote file (rootfs.jffs2)
Jan 01 00:01:13 -> File found at TFTP location 192.168.1.148:ubiimages/rootfs.jffs2)
-> All burn files downloaded
Jan 01 00:01:13 -> All common prechecks passed
-> All burn prechecks passed
-> Preparing MTD for images
Jan 01 00:01:13 -> Erasing NAND flash device
dummy!
Jan 01 00:01:13 -> Loading ubi kernel module
Jan 01 00:01:18 -> Checking primary ubi device nodes
-> Creating volumes
Jan 01 00:01:18 -> Creating all volume device nodes
-> Populated all volumes
Operation complete.
We will reboot shortly. On next reboot, use the 'bootbmc'command inside UBoot
When prompted, select [M]TD mount.
Rebooting in 5 seconds...
Restarting system.
Wait for the system to restart and then break into uboot and power down the unit and turn off IBMC_FORCE_UPDATE_N (sw3,4 off).
