ERROR Invalid kernel: elf_xen_note_check: ERROR: Will only load images
built for the generic loader or Linux images
xc_dom_parse_image returned -1
Root Cause:-
• Supplied kernel is not supported by GRUB
• Fallback kernel does not exist
Solution:-
A. If server is Amazon EBS-backed:-
Use the following procedure:
1. Stop the instance.
2. Replace with working kernel.
3. Install a fallback kernel.
4. Modify the AMI by correcting the kernel.
B. If server is Instance store-backed:-
Use the following procedure:
1. Terminate the instance and launch a new
instance with the correct kernel.
request_module: runaway loop modprobe binfmt-464c
Root Cause:-
Using an unstable or old
Linux kernel (for example, 2.6.16-xenU) can cause an interminable loop condition at startup.
Solution:-
A. If server is Amazon EBS-backed:-
Use a newer kernel, either GRUB-based or static,
using one of the following options:
Option 1:
Terminate the instance and launch
a new instance, specifying the –kernel and –
ramdisk parameters.
Option 2:
1. Stop the instance.
2. Modify the kernel and ramdisk attributes to use
a newer kernel.
3. Start the instance.
B. If server is Instance store-backed:-
Terminate the instance and launch a new
instance, specifying the –kernel and –ramdisk
parameters.
fsck.ext3: No such file or directory while trying to open /dev/sdh
Root caused:-
• A bug exists in ramdisk filesystem definitions /etc/fstab
• Misconfigured filesystem definitions in /etc/fstab
• Missing/failed drive
Solution:-
A. If server is Amazon EBS-backed:-
Use the following procedure:
1. Stop the instance, detach the root volume,
repair/modify /etc/fstab the volume, attach
the volume to the instance, and start the
instance.
2. Fix ramdisk to include modified /etc/fstab (if
applicable).
3. Modify the AMI to use a newer ramdisk.
The sixth field in the fstab defines availability
requirements of the mount – a nonzero value
implies that an fsck will be done on that volume
and must succeed. Using this field can be
problematic in Amazon EC2 because a failure
typically results in an interactive console prompt
that is not currently available in Amazon EC2.
Use care with this feature and read the Linux man
page for fstab.
B. If server is Instance store-backed:-
Use the following procedure:
1. Terminate the instance and launch a new
instance.
2. Detach any errant Amazon EBS volumes and
the reboot instance.
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)
Root caused:-
Amazon EBS-backed
• Device not attached correctly.
• Root device not attached at correct device
point.
• Filesystem not in expected format.
• Use of legacy kernel (such as 2.6.16-XenU).
• A recent kernel update on your instance (faulty
update, or an update bug)
Instance store-backed
• Hardware device failure.
Solution:-
A. If server is Amazon EBS-backed:-
• Stop and then restart the instance.
• Modify root volume to attach at the correct
device point, possible /dev/sda1 instead of /
dev/sda.
• Stop and modify to use modern kernel.
• Refer to the documentation for your Linux
distribution to check for known update bugs.
Change or reinstall the kernel.
B. If server is Instance store-backed:-
Terminate the instance and launch a new instance
using a modern kernel.
Waiting 10 seconds for device /dev/xvda1 ...
Root device '/dev/xvda1' doesn't exist. Attempting to create it. ERROR: Unable to determine major/minor number of root device '/dev/xvda1'.
You are being dropped to a recovery shell
Type 'exit' to try and continue booting
sh: can't access tty; job control turned off
[ramfs /]#
Root caused :-
• Missing or incorrectly configured virtual block device driver
• Device enumeration clash (sda versus xvda)
• Incorrect choice of instance kernel
Solution:-
A. If server is Amazon EBS-backed:-
Use the following procedure:
1. Stop the instance.
2. Detach the volume.
3. Fix the device mapping problem.
4. Start the instance.
5. Modify the AMI to address device mapping
issues.
B. If server is Instance store-backed:-
Use the following procedure:
1. Create an AMI with the appropriate fix (map
block device correctly).
2. Terminate the instance and launch a new
instance using the AMI you created.
[root@ip-172-31-84-176 ~]# sudo chmod +x /usr/local/bin/docker-compose Note: If the command docker-compose fails after installation, check your path. You can also create a symbolic link to /usr/bin or any other directory in your path. [root@ip-172-31-84-176 ~]# sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
...
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
/dev/sda1 has gone 361 days without being checked, check forced
Root Caused:-
Filesystem check time passed; a filesystem check is being forced.
Solution:-
• Wait until the filesystem check completes. A filesystem check can take a long time depending on the
size of the root filesystem.
• Modify your filesystems to remove the filesystem check (fsck) enforcement using tune2fs or tools
appropriate for your filesystem.
Cleaning up ifupdown....
Loading kernel modules...done.
...
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.16.2
/sbin/fsck.xfs: /dev/sdh does not exist fsck died with exit status 8
[31mfailed (code 8).[39;49m
Root caused:-
Ramdisk looking for missing drive
Filesystem consistency check forced
Drive failed or detached
Solution:-
A. If server is Amazon EBS-backed:-
Try one or more of the following to resolve the
issue:
• Stop the instance, attach the volume to an
existing running instance.
GNU GRUB version 0.97 (629760K lower / 0K upper memory)
[ Minimal BASH-like line editing is supported. For
the first word, TAB lists possible command
completions. Anywhere else TAB lists the possible
completions of a device/filename. ] grubdom> Root Caused:-
If server is Amazon EBS-backed
• Missing GRUB configuration file.
• Incorrect GRUB image used, expecting GRUB
configuration file at a different location.
• Unsupported filesystem used to store
your GRUB configuration file (for example,
converting your root file system to a type that is
not supported by an earlier version of GRUB).
If server is Instance store-backed
• Missing GRUB configuration file.
• Incorrect GRUB image used, expecting GRUB
configuration file at a different location.
• Unsupported filesystem used to store
your GRUB configuration file (for example,
converting your root file system to a type that is
not supported by an earlier version of GRUB).
Solution:-
A. If server is Amazon EBS-backed:-
Option 1: Modify the AMI and relaunch the
instance:
1. Modify the source AMI to create a GRUB
configuration file at the standard location (/
boot/grub/menu.lst).
2. Verify that your version of GRUB supports the
underlying file system type and upgrade GRUB
if necessary.
3. Pick the appropriate GRUB image, (hd0-1st
drive or hd00 – 1st drive, 1st partition).
4. Terminate the instance and launch a new one
using the AMI that you created.
Option 2: Fix the existing instance:
1. Stop the instance.
2. Detach the root filesystem.
3. Attach the root filesystem to a known working
instance.
4. Mount filesystem.
5. Create a GRUB configuration file.
6. Verify that your version of GRUB supports the
underlying file system type and upgrade GRUB
if necessary.
7. Detach filesystem.
8. Attach to the original instance.
9. Modify kernel attribute to use the appropriate
GRUB image (1st disk or 1st partition on 1st
disk).
10.Start the instance
B. If server is Instance store-backed:-
Option 1: Modify the AMI and relaunch the
instance:
1. Create the new AMI with a GRUB configuration
file at the standard location (/boot/grub/
menu.lst).
2. Pick the appropriate GRUB image, (hd0-1st
drive or hd00 – 1st drive, 1st partition).
3. Verify that your version of GRUB supports the
underlying file system type and upgrade GRUB
if necessary.
4. Terminate the instance and launch a new
instance using the AMI you created.
Option 2: Terminate the instance and launch a
new instance, specifying the correct kernel.
Note :- To recover data from the existing
instance, contact AWS Support.
...
Bringing up loopback interface: [ OK ] Bringing up interface eth0: Device eth0 has different MAC address than expected, ignoring.
[FAILED]
Starting auditd: [ OK ]
Root caused:-
There is a hardcoded interface MAC in the AMI configuration
Solution:-
A. If server is Amazon EBS-backed:-
Do one of the following:
• Modify the AMI to remove the hardcoding and
relaunch the instance.
• Modify the instance to remove the hardcoded
MAC address.
OR use below
1. Stop the instance.
2. Detach the root volume.
3. Attach the volume to another instance and
modify the volume to remove the hardcoded
MAC address.
4. Attach the volume to the original instance.
5. Start the instance.
B. If server is Instance store-backed:-
Do one of the following:
:
• Modify the instance to remove the hardcoded
MAC address.
• Terminate the instance and launch a new
instance.
audit(1313445102.626:2): enforcing=1 old_enforcing=0 auid=4294967295 Unable to load SELinux Policy. Machine is in enforcing mode. Halting now.
Kernel panic - not syncing: Attempted to kill init!
Root Caused:-
SELinux has been enabled in error:
• Supplied kernel is not supported by GRUB (GRand Unified Bootloader)
• Fallback kernel does not exist
Solution:-
A. If server is Amazon EBS-backed:-
1. Stop the failed instance.
2. Detach the failed instance's root volume.
3. Attach the root volume to another running
Linux instance (later referred to as a recovery
instance).
4. Connect to the recovery instance and mount
the failed instance's root volume.
5. Disable SELinux on the mounted root volume
On some systems, you disable SELinux
by setting SELINUX=disabled in the
/mount_point/etc/sysconfig/
selinux file, where mount_point
is the location that you mounted the
volume on your recovery instance.
6. Unmount and detach the root volume from the
recovery instance and reattach it to the original
instance.
Linux version 2.6.16-xenU (builder@xenbat.amazonsa) (gcc version 4.0.1
20050727 (Red Hat 4.0.1-5)) ✔1 SMP Mon May 28 03:41:49 SAST 2007
...
XENBUS: Timeout connecting to devices!
...
Kernel panic - not syncing: No init found. Try passing init= option to kernel.
Root Caused:-
The block device not is connected to the instance.
This instance is using an old instance kernel.
Solution:-
1. If server is Amazon EBS-backed:-
Do one of the following:
• Modify the AMI and instance to use a modern
kernel and relaunch the instance.
• Reboot the instance.
2. If server is Instance store-backed:-
Do one of the following:
• Terminate the instance.
• Modify the AMI to use a modern kernel, and
launch a new instance using this AMI.
When we are using an unstable or old
Linux kernel (for example, 2.6.16-xenU) can cause an interminable(endless) loop condition at startup and we will get error like below :-
If input/output error is indicated by a system log of EC2 server as show below:-
...
block drbd1: Local IO failed in request_timer_fn. Detaching...
Aborting journal on device drbd1-8.
block drbd1: IO ERROR: neither local nor remote disk
Buffer I/O error on device drbd1, logical block 557056
lost page write due to I/O error on drbd1
JBD2: I/O error detected when updating journal superblock for drbd1-8.
Root Caused:-
If our server is Amazon EBS-backed :- A failed Amazon EBS volume
If our server is Instance store-backed :- A failed physical drive Solution:-
Terminate the instance and launch a new instance.
For an Amazon EBS-backed instance you can recover data from a recent snapshot by creating an image(AMI) from it. Any data added after the snapshot cannot be recovered.
If input/output error is indicated by a system log of EC2 server as show below:-
[9943662.053217] end_request: I/O error, dev sde, sector 52428288
[9943664.191262] end_request: I/O error, dev sde, sector 52428168
[9943664.191285] Buffer I/O error on device md0, logical block 209713024
[9943664.191297] Buffer I/O error on device md0, logical block 209713025
[9943664.191304] Buffer I/O error on device md0, logical block 209713026
[9943664.191310] Buffer I/O error on device md0, logical block 209713027
[9943664.191317] Buffer I/O error on device md0, logical block 209713028
[9943664.191324] Buffer I/O error on device md0, logical block 209713029
[9943664.191332] Buffer I/O error on device md0, logical block 209713030
[9943664.191339] Buffer I/O error on device md0, logical block 209713031
[9943664.191581] end_request: I/O error, dev sde, sector 52428280
[9943664.191590] Buffer I/O error on device md0, logical block 209713136
[9943664.191597] Buffer I/O error on device md0, logical block 209713137
[9943664.191767] end_request: I/O error, dev sde, sector 52428288
[9943664.191970] end_request: I/O error, dev sde, sector 52428288
[9943664.192143] end_request: I/O error, dev sde, sector 52428288
[9943664.192949] end_request: I/O error, dev sde, sector 52428288
[9943664.193112] end_request: I/O error, dev sde, sector 52428288
[9943664.193266] end_request: I/O error, dev sde, sector 52428288
...
Root Caused :-
If our server is Amazon EBS-backed :- A failed Amazon EBS volume
If our server is Instance store-backed :- A failed physical drive
Solution:-
Amazon EBS-backed server:-
1. Stop the instance.
2. Detach the volume.
3. Attempt to recover the volume.
Take snapshot of volume ----> Then create new volume from snapshot in same AZ.
Note
It's good practice to snapshot your
Amazon EBS volumes often. This
dramatically decreases the risk of data
loss as a result of failure.
4. Re-attach the volume
Instance store-backed Server:-
Terminate the instance and launch a new instance.
Note:- Data cannot be recovered. if you kept any backups then only you can Recover from
backups.
It's a good practice to use either Amazon
S3 or Amazon EBS for backups.
If we are getting out-of-memory error in the system log similar to the one shown below.
[115879.769795] Out of memory: kill process 20273 (httpd) score 1285879
or a child
[115879.769795] Killed process 1917 (php-cgi) vsz:467184kB, anonrss:101196kB, file-rss:204kB
Root Cause:- Exhausted memory:- (extremely tired) not able to take further load
Solution:-
1. If our instance type is Amazon EBS-backed:-
Do one of the following:
Change to larger or a memory-optimized instance type.
Stop the instance --> Modify the instance
to use a different instance type --> Start
the instance again.
Reboot the instance to return it to a nonimpaired status. The problem will probably
occur again unless you change the instance
type.
2. If our instance type is Instance store-backed :-
Do one of the following:
Terminate the instance after creating an AMI and launch a new
instance from that, specifying a larger or a memory-optimized instance type.
Reboot the instance to return it to an
unimpaired status. The problem will probably
occur again unless you change the instance
type.
If we get error in system log like Memory management update failures
...
Press `ESC' to enter the menu... 0 [H[J Booting 'Amazon Linux 2011.09
(2.6.35.14-95.38.amzn1.i686)'
root (hd0)
Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.35.14-95.38.amzn1.i686 root=LABEL=/ console=hvc0 LANG=
en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-2.6.35.14-95.38.amzn1.i686.img
ERROR: mmu_update failed with rc=-22 Root caused:- Issue with Amazon Linux
1. Delayed Instance Termination:- If your instance remains in the shutting-down state longer than a few minutes, it might be delayed
due to shutdown scripts being run by the instance.
Another possible cause is a problem with the underlying host computer. If your instance remains in the
shutting-down state for several hours, Amazon EC2 treats it as a stuck instance and forcibly terminates
it.
If it appears that your instance is stuck terminating and it has been longer than several hours, Then create a ticket with AWS support team.
2.Terminated Instance Still Displayed :- After you terminate an instance, it remains visible for a short while before being deleted. The state
shows as terminated. If the entry is not deleted after several hours, contact Support team by creating a ticket with AWS support team.
3. Automatically Launch or Terminate Instances:- If you terminate all your instances, you may see that AWS launch a new instance for you. If you launch an
instance, you may see that AWS terminate one of your instances. If you stop an instance, you may see that AWS terminate the instance and launch a new instance.Generally, these behaviors mean that you've used
Amazon EC2 Auto Scaling or Elastic Beanstalk to scale your computing resources automatically based on
criteria that you've defined.
If you are connecting to your instance with Putty and you receive the error "Server unexpectedly closed
network connection,"
Then to Resolved this:-
Go to PuTTy options --> Connection
Change the default value for "Seconds between keepalives(0 to turn off)" : from 0 to 600 (10 minutes) -- in my case taking 30 secound That means it sends a "ping" every 30 seconds to prevent the connection from timing out
Check the "Enable TCP_keepalives (SO_KEEPALIVE option)" check box.
The ping command is a type of ICMP traffic — if you are unable to ping your instance, ensure that your
inbound security group rules allow ICMP traffic for the Echo Request message from all sources, or from
the computer or instance from which you are issuing the command.
If you are unable to issue a ping
command from your instance, ensure that your outbound security group rules allow ICMP traffic for the
Echo Request message to all destinations, or to the host that you are attempting to ping.
C:\Users\raj>ping 52.207.212.227
Pinging 52.207.212.227 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 52.207.212.227:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
In the above case ping fail from our local PC to our EC2 server because ICMP port are not open
If you use PuTTY to connect to your instance and get either of the following errors, Error: Server
refused our key or Error: No supported authentication methods available, verify that
you are connecting with the appropriate user name
The appropriate user names are as follows:
• For Amazon Linux 2 or the Amazon Linux AMI, the user name is ec2-user.
• For a CentOS AMI, the user name is centos.
• For a Debian AMI, the user name is admin or root.
• For a Fedora AMI, the user name is ec2-user or fedora.
• For a RHEL AMI, the user name is ec2-user or root.
• For a SUSE AMI, the user name is ec2-user or root.
• For an Ubuntu AMI, the user name is ubuntu.
• Otherwise, if ec2-user and root don't work, check with the AMI provider.
You should also verify that your private key (.pem) file has been correctly converted to the format
recognized by PuTTY (.ppk).
If we get Error: Private key must begin with "-----BEGIN RSA PRIVATE KEY-----" and end with "-----END RSA PRIVATE KEY-----"
This is due to
If you use a third-party tool, such as ssh-keygen, to create an RSA key pair, it generates the private key in
the OpenSSH key format. When you connect to your instance, if you use the private key in the OpenSSH
format to decrypt the password,
you'll get the error Private key must begin with "-----BEGIN
RSA PRIVATE KEY-----" and end with "-----END RSA PRIVATE KEY-----".
To resolve the error, the private key must be in the PEM format.
Use the following command to create
the private key in the PEM format:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0774 for 'rajkp.pem' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "rajkp.pem": bad permissions
Permission denied (publickey).
1. Verify that your private key (.pem) file has been converted to the format recognized by PuTTY (.ppk).In PuTTYgen, load your private key file and select Save Private Key rather than Generate.
2. Verify that you are connecting with the appropriate user name for your AMI
• For Amazon Linux 2 or the Amazon Linux AMI, the user name is ec2-user.
• For a CentOS AMI, the user name is centos.
• For a Debian AMI, the user name is admin or root.
• For a Fedora AMI, the user name is ec2-user or fedora.
• For a RHEL AMI, the user name is ec2-user or root.
• For a SUSE AMI, the user name is ec2-user or root.
• For an Ubuntu AMI, the user name is ubuntu.
• Otherwise, if ec2-user and root don't work, check with the AMI provider.
3. Verify that you have an inbound security group rule to allow inbound traffic to the appropriate port.
If you use SSH to connect to your instance
• Use ssh -vvv to get triple verbose debugging information while connecting:
ssh -vvv -i [your key name].pem ec2-user@[public DNS address of your
instance].compute-1.amazonaws.com
then we will get output like below which help us in debugging
open/ANT/myusername/.ssh/known_hosts).
debug2: bits set: 504/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: boguspem.pem ((nil))
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: boguspem.pem
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA 9c:4c:bc:0c:d0:5c:c7:92:6c:8e:9b:16:e4:43:d8:b2
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
If you use SSH (MindTerm) to connect to your instance
• If Java is not enabled, the server does not recognize the user key. To enable Java, go to
Internet Explorer
Click Tools and then Internet Options
Select the Security tab, and select the Custom Level button
If you try to connect from your laptop or PC to your EC2 instance by using IPv6
• Your subnet must be associated with a route table that has a route for IPv6 traffic (::/0) to an
internet gateway.
• Your security group rules must allow inbound traffic from your local IPv6 address on the proper port
(22 for Linux and 3389 for Windows).
• Your network ACL rules must allow inbound and outbound IPv6 traffic.
• If you launched your instance from an older AMI, it may not be configured for DHCPv6 (IPv6 addresses
are not automatically recognized on the network interface).
• Your local computer must have an IPv6 address, and must be configured to use IPv6.
If you try to connect from your laptop or PC to your EC2 instance and get an error message Network error: Connection timed
out or Error connecting to [instance], reason: -> Connection timed out: connect,
Then do the below:-
• Check your security group rule
Verify that there is a rule that allows traffic from your computer
to port 22 (SSH) for Linux and port 3389 (RDP) for window.
• Check the route table for the subnet. You need a route that sends all traffic destined outside the VPC
to the internet gateway for the VPC.
• Check the network access control list (ACL) for the subnet. The network ACLs must allow inbound and
outbound traffic from your local IP address on the proper port. The default network ACL allows all
inbound and outbound traffic.
• If your computer is on a corporate network, ask your network administrator whether the internal
firewall allows inbound and outbound traffic from your computer on port 22 (for Linux instances) or
port 3389 (for Windows instances).
If you have a firewall on your computer, verify that it allows inbound and outbound traffic from your
computer on port 22 (for Linux instances) or port 3389 (for Windows instances).
• Check that your instance has a public IPv4 address. If not, you can associate an Elastic IP address with
your instance.
