Rocky ENA, EXPRESS, PTP, RT
https://docs.rockylinux.org/guides/custom-linux-kernel/
# ENA (PHC and express)
Recommandation from the ena documentation:
use boot param net.ifnames=0sudo grubby --update-kernel=ALL --args="net.ifnames=0"
When predictable network naming is enabled, Linux might change the device name and affect the network configuration. This can lead to a loss of network on boot. To disable this feature add net.ifnames=0 to the kernel boot params.
sudo dnf update -y
better to restart after this (kernel could be updated)
sudo dnf -y groupinstall 'Development Tools'
sudo dnf -y install ncurses-devel openssl-devel elfutils-libelf-devel python3
sudo dnf -y install dwarves
Depending on kernel (most likely you just need a restart)
sudo dnf -y install kernel-rt-devel
sudo dnf -y install kernel-devel
sudo dnf -y install kernel-devel-matched
BUT, you might just require a restart, because the kernel was updated, and it installs the package for the newer kernel, but uname -r points to the older kernel
check first:
/usr/lib/modules/`uname -r`/build/
sudo cp /sys/kernel/btf/vmlinux /usr/lib/modules/`uname -r`/build/
Check the current installed ena
[rocky@ip-10-200-143-99 ena]$ modinfo ena
filename: /lib/modules/5.14.0-427.35.1.el9_4.aarch64/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz
license: GPL
description: Elastic Network Adapter (ENA)
author: Amazon.com, Inc. or its affiliates
rhelversion: 9.4
srcversion: BB1142CD4A2B0039E7071E4
alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:
intree: Y
name: ena
vermagic: 5.14.0-427.35.1.el9_4.aarch64 SMP preempt mod_unload modversions aarch64
sig_id: PKCS#7
signer: Rocky kernel signing key
sig_key: 2C:74:65:25:85:21:15:6E:10:25:AC:DA:76:82:FD:B3:AF:25:AB:F4
sig_hashalgo: sha256
signature: 1D:90:78:69:64:93:D7:14:A3:4F:04:FD:0E:47:92:F8:07:EA:DD:2A:
C6:F9:F4:9C:72:FA:00:3D:6A:19:30:3E:D7:7C:67:44:86:B1:04:03:
D7:9A:A9:B6:00:B1:79:33:F3:F2:83:E0:A0:AE:A8:B9:11:44:BE:52:
64:DA:7C:0F:36:F3:B8:4D:96:0F:12:CD:44:EA:E3:32:0B:63:43:DC:
D7:06:F5:32:14:CF:94:B8:D3:8C:04:2C:9A:AE:D9:A9:2D:04:76:FB:
95:8A:DA:FA:AA:7C:9D:0A:92:47:12:1B:34:DB:89:C3:D5:DB:B2:AF:
27:33:4C:61:A0:48:4F:08:56:A2:ED:90:CA:EC:FB:96:2B:2F:53:B6:
4A:55:F9:8C:C0:34:EF:09:CE:50:1B:5F:A5:60:71:D8:AD:39:CB:56:
A3:8B:10:48:D8:BC:56:19:57:A3:A5:F5:58:95:67:AD:53:6B:6D:EA:
26:A3:9C:74:D2:02:FA:48:69:D2:2C:34:5A:BA:8C:3E:D5:8A:DE:F3:
60:27:88:06:16:78:5B:E7:78:C4:66:F5:BC:23:B2:9F:4F:FB:5A:9B:
CE:CF:2A:DA:4B:40:43:78:DB:9B:E3:9F:3F:E9:C4:24:55:BD:C3:16:
61:F6:34:77:28:E2:03:CF:90:2A:A1:55:5F:CD:17:91:1D:29:59:25:
EA:09:4C:F2:C8:F3:78:C7:EC:C3:C1:60:42:09:DB:8F:77:ED:1E:CE:
E8:12:BF:77:9B:A7:8C:EC:8E:8B:FE:E9:4A:4C:E6:57:14:DE:FE:A0:
44:19:A2:38:84:28:0B:A9:F6:AF:58:99:C8:36:CA:3C:37:E5:DC:34:
91:0A:7A:11:B5:0A:C8:78:03:1A:50:88:17:B3:C7:8B:90:F7:1E:63:
71:41:6F:67:C1:4D:E1:4D:9B:EE:0D:10:7F:F5:8A:FC:C8:0D:DA:86:
73:E4:B1:C3:69:9A:17:FE:09:7C:18:70:E9:53:ED:AE:68:73:16:7E:
C9:5E:8C:0C
You can see where the current ena module file is located:
/lib/modules/5.14.0-427.35.1.el9_4.aarch64/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz
It should be present in this location
/lib/modules/$(uname -r)/kernel/drivers/net/ethernet/amazon/ena/
But the extension might differ ena.ko.xz vs ena.ko.
On Ubuntu it might be ena.ko but on Rocky it appears to be ena.ko.xz.
So make a backup of the current module:
Ubuntu
sudo cp /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/amazon/ena/ena.ko ~/ena.ko.backup
Rocky
sudo cp /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz ~/ena.ko.xz.backup
git clone https://github.com/amzn/amzn-drivers.git
cd amzn-drivers/kernel/linux/ena
ENA_PHC_INCLUDE=1 make
Copy the newer one to that location
Ubuntusudo cp ena.ko /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/amazon/ena/
Rocky
sudo cp ena.ko /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz
Load the module
sudo depmod -a
sudo modprobe -r ena && sudo modprobe ena phc_enable=1
Check that is loaded
[rocky@ip-10-200-143-99 ena]$ modinfo ena
filename: /lib/modules/5.14.0-427.35.1.el9_4.aarch64/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz
version: 2.13.0g
license: GPL
description: Elastic Network Adapter (ENA)
author: Amazon.com, Inc. or its affiliates
rhelversion: 9.4
srcversion: 2166FB2E66C072B0B42093C
alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:
name: ena
vermagic: 5.14.0-427.35.1.el9_4.aarch64 SMP preempt mod_unload modversions aarch64
parm: debug:Debug level (-1=default,0=none,...,16=all) (int)
parm: rx_queue_size:Rx queue size. The size should be a power of 2. Depending on instance type, max value can be up to 16K
(int)
parm: force_large_llq_header:Increases maximum supported header size in LLQ mode to 224 bytes, while reducing the maximum TX queue size by half.
(int)
parm: num_io_queues:Sets number of RX/TX queues to allocate to device. The maximum value depends on the device and number of online CPUs.
(int)
parm: enable_bql:Enable BQL.
(int)
parm: lpc_size:Each local page cache (lpc) holds N * 1024 pages. This parameter sets N which is rounded up to a multiplier of 2. If zero, the page cache is disabled. Max: 32
(int)
parm: phc_enable:Enable PHC.
(uint)
[rocky@ip-10-200-143-99 ena]$
Also check
ls /sys/class/ptp/
Now we need to update chrony config to use ptp.
Locate the chrony.conf, on Rocky its /etc/chrony.conf, on Ubuntu it is /etc/chrony/chrony.conf, so:
Ubuntu
sudo sh -c 'echo "refclock PHC /dev/ptp0 poll 0 delay 0.000010 prefer" >> /etc/chrony/chrony.conf'
Rocky
sudo sh -c 'echo "refclock PHC /dev/ptp0 poll 0 delay 0.000010 prefer" >> /etc/chrony.conf'
restart chrony:
sudo systemctl restart chronyd
Check chrony sources:
[rocky@ip-10-200-143-99 ~]$ chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
#* PHC0 0 0 37 0 +6ns[ +36ns] +/- 5031ns
^? time.cloudflare.com 3 6 3 2 -385us[ -385us] +/- 60ms
^- sv1.localdomain1.com 2 6 7 0 +248us[ +248us] +/- 30ms
^- gotoky.hojmark.net 2 6 7 1 +175us[ +175us] +/- 21ms
^? v160-16-113-133.ntp.tky2> 3 6 3 2 +359us[ +359us] +/- 3303us
^? 169.254.169.123 1 4 3 2 -581us[ -581us] +/- 671us
[rocky@ip-10-200-143-99 ~]$
PERSITENCE
Persistence in such a fucking pain in the ass (specially on Rocky), so just make a systemd script:
sudo vi /usr/local/bin/ena-phc-support.sh
#!/bin/bash
sudo modprobe -r ena && sudo modprobe ena phc_enable=1
sudo chmod +x /usr/local/bin/ena-phc-support.sh
sudo vi /etc/systemd/system/ena-phc.service
[Unit]
Description=Load ENA module with PHC support and restart chronyd
After=network.target
Before=chronyd.service
[Service]
Type=oneshot
ExecStart=/bin/bash /usr/local/bin/ena-phc-support.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable ena-phc.service
sudo systemctl start ena-phc.service
Reboot to test (check chronyc sources!!)
If instance supports ena express and it is enabled on the ENI, then you should have ena express:
[rocky@ip-10-200-128-98 ~]$ ethtool -S ens5 | grep ena_srd
ena_srd_mode: 3
ena_srd_tx_pkts: 0
ena_srd_eligible_tx_pkts: 0
ena_srd_rx_pkts: 0
ena_srd_resource_utilization: 0
RT
sudo dnf config-manager --set-enabled devel sudo dnf repolist dnf install kernel-rt.aarch64
sudo dnf -y install kernel-rt-devel
IF YOU HAVE THE ENA EXPRESS SYSTEMD SERVICE ENABLED BETTER DISABLED ITsudo systemctl disable ena-phc.service
Check the available kernelsls /boot | grep vml | grep rt
Set the default kernel
sudo grubby --set-default="/boot/vmlinuz-5.14.0-427.35.1.el9_4.aarch64+rt"
Check that it might be appliedsudo grubby --default-kernel
REBOOT
Check kerneluname -r
uname -a
You must have chrony running properly, so comment out the phc line and restart chrony
Ena for RT (or any custom kernel)
git clone https://github.com/amzn/amzn-drivers.git
cd amzn-drivers/kernel/linux/ena
make clean
make -C /lib/modules/5.14.0-427.35.1.el9_4.aarch64+rt/build M=/home/rocky/amzn-drivers/kernel/linux/ena clean
make[1]: *** /lib/modules/5.14.0-427.35.1.el9_4.aarch64+rt/build: No such file or directory. Stop.
make: *** [Makefile:104: clean] Error 2
Here we need the devel package
Also dont forget
sudo cp /sys/kernel/btf/vmlinux /usr/lib/modules/`uname -r`/build/
ENA_PHC_INCLUDE=1 make
You will have error, for example:
CC [M] /home/rocky/amzn-drivers/kernel/linux/ena/ena_netdev.o
In file included from /home/rocky/amzn-drivers/kernel/linux/ena/ena_netdev.h:9,
from /home/rocky/amzn-drivers/kernel/linux/ena/ena_netdev.c:25:
/home/rocky/amzn-drivers/kernel/linux/ena/kcompat.h:344:20: error: redefinition of ‘skb_mark_napi_id’
344 | static inline void skb_mark_napi_id(struct sk_buff *skb,
| ^~~~~~~~~~~~~~~~
#ifndef CONFIG_NET_RX_BUSY_POLL
static inline void skb_mark_napi_id(struct sk_buff *skb,
struct napi_struct *napi)
{
}
static inline void napi_hash_del(struct napi_struct *napi)
{
}
static inline void napi_hash_add(struct napi_struct *napi)
{
}
#endif /* CONFIG_NET_RX_BUSY_POLL */
turned into this:
//******************************************************************************
// #ifndef CONFIG_NET_RX_BUSY_POLL
// static inline void skb_mark_napi_id(struct sk_buff *skb,
// struct napi_struct *napi)
// {
//
// }
//
// static inline void napi_hash_del(struct napi_struct *napi)
// {
//
// }
//
// static inline void napi_hash_add(struct napi_struct *napi)
// {
//
// }
// #endif /* CONFIG_NET_RX_BUSY_POLL */
//*****************************************************************************/
Follow the instructions from above and dont forget the chrony sources phc change and systemd service!!!!!!!