IPSec always felt very painful to me. Even though I read through the RFCs, it always felt so disconnected from the software implementing it. It also didn't help to understand it that many people mix together IKE, IPSec, and NAT traversal. Also, some operating systems have multiple distinct implementations for IPSec (looking at your Windows, with one in group policies and another within the Windows Firewall as "Connection Security Rules").
I also would like to mention that tools like OpenSwan, StrongSwan, ... are technically NOT for IPSec but IKE. Especially when you're RFC focused, this may be the reason for a lot of your confusion. The actual IPSec client is already on your system, it is provided by your operating system.
So what is this IKE and what does it do?
The role of IKE is to set up and rotate the crypto keys used by IPSec. So it basically it isn't involved in the traffic flow at all. A somewhat good comparison is IKE is to IPSec what ICMP is to IP. Also, almost always if you're having any IPSec issues, it is actually an issue with IKE. Like a firewall dropping it or having incorrect configuration at either one or both ends. Without the crypto keys being properly exchanged, IPSec can't work, as it literally never got the correct keys to begin with. (Oh, also a common issue is to NOT exclude IKE traffic from the IPSec tunnel, thereby creating a chicken-and-egg problem of the tunnel having to be already up in order to be initialized).
So do we need IKE after all?
Well, no, but you almost certainly want to have it anyway once you know what problems it solves for you. I'll include an example IKE configuration script below. You can use that on two or more newly installed Linux servers to establish a simple IPSec VPN. Basically IKE sets the ID, KEY1, and KEY2 variables. For security reasons, you want to have these values never repeat, change continuously, be unpredictable for 3rd parties (and attackers). Also, they need to be the same on all the involved nodes. Technically you could easily replace IKE by another means of exchanging the keys, but in practice the main issue with this is ensuring you change the keys for one flow (that is, a unidirectional connection between two nodes) on both involved nodes simultaneously. Simultaneously, meaning you'll have to change the key on the recipient side and sending side so that no package is sent with the old key after the new key is configured on the recipient and no package with the new key is sent while the old key is still configured at the recipient. Just getting this to work without issues is already a tough challenge. Doing so securely is another.
#
# Interface IPs: 2001:db8:11:1::1 2001:db8:11::1 2001:db8:10::1 2001:db8:10:1::1
# >------------------|R1|---------------------------------|R2|------------------<
# Network Prefix: 2001:db8:11:1::/80 ::/0 2001:db8:10:1::/80
#
#
# Generate reqid and AES key (do not use, this is what StrongSwan and OpenSwan would be tasked with generating and rotating.
ID=0x`dd if=/dev/urandom count=4 bs=1 2> /dev/null| xxd -p -c 8`
KEY1=0x`dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 40`
KEY2=0x`dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 40`
# example generated values (do not use)
#ID="0x93cbac56"
#KEY1="0x1957a32b898cad749a689a697b5b415354457e0b"
#KEY2="0x1930ad8de188a9781b9cc883447e3c80dfa7bdba"
#
# Endpoints
SiteAPublic="2001:db8:11::1"
SiteBPublic="2001:db8:10::1"
SiteAPrivate="2001:db8:11:1::/80"
SiteBPrivate="2001:db8:10:1::/80"
SiteAPrivate_IP="2001:db8:11:1::1"
SiteBPrivate_IP="2001:db8:10:1::1"
servers="server001 server002 server003 ..."
for server in $servers; do
ssh $server /bin/sh <<EOF
# This will remove all existing IPSec configuration.
ip -6 xfrm state flush && ip -6 xfrm policy flush
# One xfrm state and three (in, fwd, out) xfrm policies per direction, if you have two nodes you need one for each direction (that's 2 in total). For three it is 9 in total.
# You can provide less, but then you'll need to forward traffic through one of the other nodes (more complex, requires a dynamic routing protocol like BGP).
# If you want fault tolerance in case a direct connection from A to C fails but A to B to C is still possible you'll also need the more complex setup.
ip -6 xfrm state add src $SiteAPublic dst $SiteBPublic proto esp spi $ID reqid $ID mode tunnel aead 'rfc4106(gcm(aes))' $KEY1 128
for dir in `echo "in fwd out"`; do
ip -6 xfrm policy add src $SiteAPrivate dst $SiteBPrivate dir $dir tmpl src $SiteAPublic dst $SiteBPublic proto esp reqid $ID mode tunnel
done
ip -6 xfrm state add src $SiteBPublic dst $SiteAPublic proto esp spi $ID reqid $ID mode tunnel aead 'rfc4106(gcm(aes))' $KEY2 128
for dir in `echo "in fwd out"`; do
ip -6 xfrm policy add src $SiteBPrivate dst $SiteAPrivate dir $dir tmpl src $SiteBPublic dst $SiteAPublic proto esp reqid $ID mode tunnel
done
EOF
done