How to compile and configure Sendmail 8.19.0.2 snapshot on OmniOS stable

I installed Cyrus IMAPD 3.12.2 first so I got SASLv2 libraries installed as dependencies. You have to install OpenLDAP libraries too for LDAPMAP to work. OpenSSL is installed as default just like Oracle DB (former Berkeley DB).

USE_EAI means SMTPUTF8 support. The required libraries were installed as default. I found out that Sendmail 8.19.0.2 SMTPUTF8 works, but it is useless to me as Cyrus IMAPD lmtpd does not support it.

root@omnios:~# uname -a
SunOS omnios 5.11 omnios-r151058-c1eded413b i86pc i386 i86pc

root@omnios:~# /opt/site/sbin/sendmail -bt -d0.1 < /dev/null
Version 8.19.0.2
Compiled with: ALLOW_255 DANE HAVE_SSL_CTX_dane_enable MAX_TLSA_RR=64
DNSMAP IPV6_FULL LDAPMAP LDAP_NETWORK_TIMEOUT
SM_CONF_LDAP_INITIALIZE SM_CONF_LDAP_MEMFREE LOG MAP_REGEX
MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6
NETUNIX NEWDB=5.3 PIPELINING SASLv2 SCANF SOCKETMAP STARTTLS
MTA_HAVE_TLSv1_3 TLS_EC= 1 TLS_VRFY_PER_CTX USERDB USE_EAI
USE_LDAP_INIT XDEBUG

============ SYSTEM IDENTITY (after readcf) ============
(short domain name) $w = omnios
(canonical domain name) $j = omnios.local
(subdomain name) $m = local
(node name) $k = omnios
========================================================

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>

~@omnios $ cat /home/kalevi/src/sendmail-8.19.0.2/cf/cf/sendmail.mc
divert(-1)
divert(0)dnl
VERSIONID(`$Id: generic-solaris.mc,v 8.14 2013-11-22 20:51:08 ca Exp $')
OSTYPE(solaris2)dnl
DOMAIN(generic)dnl
define(`confAUTH_MECHANISMS', `LOGIN PLAIN')dnl
TRUST_AUTH_MECH(`PLAIN LOGIN')dnl
define(`confAUTH_OPTIONS', `A p y')dnl
define(`confLOG_LEVEL', `14')dnl
define(`confSERVER_CERT', `/etc/mail/certs/omnios.local.crt')dnl
define(`confSERVER_KEY', `/etc/mail/certs/omnios.local.key')dnl
define(`confCLIENT_CERT', `/etc/mail/certs/omnios.local.crt')dnl
define(`confCLIENT_KEY', `/etc/mail/certs/omnios.local.key')dnl
define(`confCACERT', `/etc/mail/certs/ca.crt')dnl
define(`confCACERT_PATH', `/etc/mail/certs')dnl
VIRTUSER_DOMAIN_FILE(`/etc/mail/virtuser-domains')dnl
FEATURE(`tls_session_features')dnl
FEATURE(access_db, `hash -o -T<TMPF> /etc/mail/access')dnl
FEATURE(blocklist_recipients)dnl
FEATURE(local_lmtp)dnl
FEATURE(mailertable, `hash -o /etc/mail/mailertable')dnl
FEATURE(`virtusertable', `hash -o /etc/mail/virtusertable')dnl
MAILER_DEFINITIONS
define(`confLOCAL_MAILER', `cyrusv2')dnl
define(`CYRUSV2_MAILER_ARGS', `FILE /var/opt/ooce/cyrus/socket/lmtp')dnl
MAILER(cyrusv2)dnl
MAILER(smtp)dnl
LOCAL_CONFIG
O SMTPUTF8=true

~@omnios $ cat /etc/opt/ooce/sasl2/Sendmail.conf
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

root@omnios:~# ps -ef|grep saslauthd|grep -v grep
root 4787 1 0 Jul 01 ? 0:00 /opt/ooce/sbin/saslauthd -a pam -c -m /var/run/saslauthd

~@omnios $ cat /home/kalevi/src/sendmail-8.19.0.2/devtools/Site/site.config.m4
define(`confMAPDEF', `-DNEWDB -DMAP_REGEX -DSOCKETMAP -DNAMED_BIND=1 -I/opt/ooce/include/ -L/opt/ooce/lib -L/opt/ooce/lib/sasl2 -L/opt/ooce/lib/amd64')dnl
define(`confINCDIRS', `-I/opt/ooce/include/')dnl
APPENDDEF(`conf_sendmail_ENVDEF', `-DSTARTTLS -D_FFR_TLS_1 -D_FFR_TLS_USE_CERTIFICATE_CHAIN_FILE -DDANE -D_FFR_EKU_NOCLIENTAUTH -D_FFR_KEX -DHASUNSETENV')dnl
APPENDDEF(`confLIBDIRS', `-L/opt/ooce/lib/amd64 -R/opt/ooce/lib/amd64')dnl
APPENDDEF(`confINCDIRS', `-I/opt/ooce/include')dnl
APPENDDEF(`conf_sendmail_LIBS', `-lssl -lcrypto -ldb -lsasl2')dnl
APPENDDEF(`confENVDEF', `-DSASL=2 -I/opt/ooce/include/')dnl
APPENDDEF(`conf_sendmail_ENVDEF', `-DMILTER')dnl
APPENDDEF(`confMAPDEF', `-DLDAPMAP -DLDAP_DEPRECATED')dnl
APPENDDEF(`confENVDEF', `-DSM_CONF_LDAP_MEMFREE=1')dnl
APPENDDEF(`confENVDEF',`-DUSE_EAI')dnl
APPENDDEF(`confLIBS', `-licuuc -lldap -llber')dnl

~@omnios $ cat /lib/svc/manifest/site/sendmail-8190snap.xml
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<!--
CDDL HEADER START

The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.

You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.

When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]

CDDL HEADER END

Copyright 2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Copyright 2020 OmniOS Community Edition (OmniOSce) Association.
-->
<service_bundle type="manifest"
name="sendmail8190snap">

<service name="kalevi/mta/sendmail8190snap"
type="service"
version="1">

<dependency name="filesystem-local"
grouping="require_all"
restart_on="error"
type="service">
<service_fmri value="svc:/system/filesystem/local:default" />
</dependency>
<method_context security_flags="aslr">
<method_credential user="root"
group="smmsp" />
</method_context>

<exec_method type="method"
name="start"
exec="/opt/site/sbin/sendmail -bd -q1h"
timeout_seconds="60" />

<exec_method type="method"
name="stop"
exec="/usr/bin/pkill -TERM -x sendmail"
timeout_seconds="60" />

<property_group name="startd"
type="framework">
<propval name="duration"
type="astring"
value="contract" />
</property_group>

<instance name="default"
enabled="false">

</instance>

<stability value="External" />

</service>

</service_bundle>

Copy the Sendmail-related binaries under /opt/site/sbin and add /opt/site/sbin to root's PATH so that makemap and other commands can be found. You also need to remove some symlinks and recreate them to point to /opt/site/sbin. Remove mailq and newaliases symlinks and create these. It appears to me that OmniOS cron uses /usr/lib/sendmail.

root@omnios:~# ls -l /usr/sbin/sendmail /usr/lib/sendmail
lrwxrwxrwx 1 root root 23 Jul 1 23:22 /usr/lib/sendmail -> /opt/site/sbin/sendmai
lrwxrwxrwx 1 root root 23 Jul 1 23:10 /usr/sbin/sendmail -> /opt/site/sbin/sendmai

Sendmail manifest can be brought under SMF control like this:

root@omnios:~# svcadm import /lib/svc/manifest/site/sendmail-8190snap.xml
root@omnios:~# svcs -a|grep sendmail
disabled 18:44:19 svc:/kalevi/mta/sendmail8190snap:default
root@omnios:~# svcadm enable svc:/kalevi/mta/sendmail8190snap:default && echo OK
OK
root@omnios:~# ps -ef|grep sendmail
root 6410 1 0 18:44:51 ? 0:00 /opt/site/sbin/sendmail -bd -q1h
root 6413 28900 0 18:44:57 pts/4 0:00 grep sendmail

If you want to test EKU exceptions, first create a self-signed certificate on a client machine. Remember to include EKU extensions for Server Authentication. Put the created Server Authentication certificate in /etc/mail/certs/ca.crt on OmniOS server.

Then use something like the following in OmniOS /etc/mail/access to enable EKU exception for the client:

root@omnios:~# cat /etc/mail/access
TLS_Srv_Features:192.168.1.154 flags=O
TLS_Srv_Features:fedora.local flags=O
Srv_Features:192.168.1.154 v
CertIssuer:/C=FI/ST=Uusimaa/L=Helsinki/O=Home/CN=fedora.local RELAY

On a client machine, use swaks to send a test message to OmniOS:

fedora$ swaks --tls-cert=fedora-serverauth.crt --tls-key=fedora-serverauth.key -tls --to=kalevi@kolttonen.fi --server omnios.local

You should see something like this on OmniOS /var/log/syslog:

Jul 4 04:06:38 localhost sendmail[24503]: [ID 702911 mail.info] tls_srv_features=flags=O, relay=fedora.local [192.168.1.154]
Jul 4 04:06:38 localhost sendmail[24503]: [ID 702911 mail.debug] tls_srv_features=parsed, flags=O, relay=fedora.local [192.168.1.154]
Jul 4 04:06:38 localhost sendmail[24503]: [ID 702911 mail.info] STARTTLS=server, relay=fedora.local [192.168.1.154], version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256, key_exchange=ECDHE, eku=overrode_no_client_auth
Jul 4 04:06:38 localhost sendmail[24503]: [ID 702911 mail.info] STARTTLS=server, cert-subject=/C=FI/ST=Uusimaa/L=Helsinki/O=Home/CN=fedora.local, cert-issuer=/C=FI/ST=Uusimaa/L=Helsinki/O=Home/CN=fedora.local, verifymsg=ok

So relaying should be allowed despite the fact that the fedora.local SMTP client presented a certificate with Server Authentication EKU.