====== Systemd admin howto ======

https://wiki.debian.org/systemd

===== basics commands =====

  ''systemctl'' : List all running services
  ''systemctl start example1'' : Activates the service "example1" immediately
  ''systemctl stop example1'' : Deactivates the service "example1" immediately
  ''systemctl restart example1'' : Restarts the service "example1" immediately: 
  ''systemctl status example1'' : Shows status of the service "example1"
  ''systemctl enable example1'' : Enables "example1" to be started on bootup
  ''systemctl disable example1'' : Disables "example1" to neot start during bootup
  ''systemctl list-units --type=service''  : list all service units

===== add a service =====

<code bash>
vi /etc/systemd/system/<service_name>.service
chmod 644 /etc/systemd/system/<service_name>.service
systemctl --system daemon-reload
systemctl start mon-service.service
systemctl enable mon-service.service
</code>

===== debugging =====

See all systemctl since last boot : ''journalctl -xa''\\
Remove "quiet" from Kernel command line (so called "cmdline" or "grub line") : ''GRUB_CMDLINE_LINUX="systemd.log_target=kmsg systemd.log_level=debug"'' => update-grub => check : ''cat /proc/cmdline'' after reboot\\
Increase verbosity via ///etc/systemd/system.conf// : uncomment : "LogLevel=debug" and "LogTarget=syslog-or-kmsg".\\
Boot an emergency shell : Add ''systemd.unit=rescue.target'' to Kernel command line.\\
Enable the debug shell : ''systemctl enable debug-shell.service'' (You can do this in a chroot environment after booting a rescue system.) This start a root shell on TTY 9.\\
See : http://freedesktop.org/wiki/Software/systemd/Debugging
  
===== Warning =====

bind mount : 'umount $CHROOT/dev' will unmount /dev ! 

<code bash>
mount --bind --make-rslave / $CHROOT
mount --bind --make-rslave /dev/ $CHROOT/dev
</code>

===== Optimize =====

<code bash>
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain [UNIT...]
</code>

Graphs, dump, etc : http://www.freedesktop.org/software/systemd/man/systemd-analyze.html

===== Understand =====

Systemd-bootchart : http://www.freedesktop.org/software/systemd/man/systemd-bootchart.html

<code bash>
apt-get install graphviz
systemd-analyze plot > plot.svg
systemd-analyze dot --order | dot -Tsvg > systemd.svg
</code>

See bootchart2 ?
    
===== Unit files definition precedence =====
  - 1st, running : /run/systemd/system
  - 2nd, customs : /etc/systemd/system
  - 3rd, originals : /lib/systemd/system

===== Systemd delta =====

<code bash>
systemd-delta : To see all local configuration:
systemd-delta /run : To see all runtime configuration:
systemd-delta systemd/system : To see all system unit configuration changes:
systemd-delta --type=extended /run/systemd/system : To see all runtime "drop-in" changes for system units:
</code>

===== Systemd service files =====

from : https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

==== Some features ====

  * socket-based activation: monitor daemon startup by 1st access
  * bus-based activation: state with bus publishing
  * path-based activation: inotify
  * device-based activation: udev events
  * implicit dependency mapping:
  * instances and templates: 
  * easy security hardening: 

==== Unit files definition precedence ====

  * 1st, running : /run/systemd/system
  * 2nd, customs : /etc/systemd/system
  * 3rd, originals : /lib/systemd/system

==== types of units : <servicename>.<typeofunit>, ex : sshd.service ====

  * .service: how to manage a service (how to start/stop, dependancy, ordering)
  * .socket: network or IPC socket or FIFO buffer, which starts a .service on activity (socket based activation)
  * .device: udev/sysfs, used for ordering, mounting, and accessing the devices
  * .mount: defines a mountpoint managed by systemd, can generate entry in fstab
  * .automount: automount, refers to a .mount
  * .swap: swap space
  * .target: sync points (boot/state change/shutdown)
  * .path: inotify based
  * .timer: timer or schudeling
  * .snapshot: created by 'systemctl snapshot ', store the system state (temporary)
  * .slice: restriction with Linux Control Group nodes
  * .scope: automatically generated by systemd from bus interfaces, used to manage sets of system processes that are created externally

units files : case sensitive sections

==== [Unit] section directives (mandatory, describes the unit, the dependancies, the conditions) ====

  * Description=: short, specific, and informative description
  * Documentation=: URIs to docs
  * Requires=: unit dependancy HARD WAY (parallel start) - fail
  * Wants=: unit dependancy SOFT WAY (parallel start) - continue
  * BindsTo=: = Requires but stop current unit if 'BindsTo listed unit' stops
  * Before=: current unit must start before 'Before listed units'
  * After=: 'After listed units' must start before current unit
  * Conflicts=: mutual exclusion start (last started, last stopped)
  * Condition...=: if the CONDITION is not met, the unit is gracefully SKIPPED
  * Assert...=: if the CONDITION is not met, FAILURE

==== Unit-Specific Section Directives ( directives that only apply to a specific type) ====

not available for device, target, snapshot, and scope.

=== [Service] Section ===

  * Type=: categorizes services by their process and daemonizing behavior :
  * simple: main process main process, communication handled outside (ex : via .socket) - (default if ExecStart= is set and Type= and Busname= are not set)
  * forking: when the service forks a child process, exiting the parent process almost immediately
  * oneshot: will be short-lived, systemd waits for the process exit (default when Type= and ExecStart= are not set)
  * notify: issue a notification when startup finished, systemd waits for this notification
  * NotifyAccess=: specifies access to the socket for 'notify' (none, main, all)
  * idle: service will not be run until all jobs are dispatched
  * RemainAfterExit=: (used with the oneshot), service active after exit
  * PIDFile=: if forking, gives the path of the file containing PID of the forked child to monitor
  * BusName=: set to the D-Bus bus name the service will attempt to acquire
  * Environment=: gives environment variables (basic substitution available : ${} (ex : Environment="ONE=one" 'TWO=two two')
  * ExecStart=: full path and the arguments of the command to be executed (if "-<cmd>", non zero exit != fail) (multiples times if oneshot service)
  * ExecStartPre=: provide additional commands that should be executed before the main process is started (multiples times) - idem '-'
  * ExecStartPost=: idem but run after service start
  * ExecReload=: command necessary to reload the configuration of the service
  * ExecStop=: command needed to stop the service (if none, process is killed)
  * ExecStopPost=: commands to execute after the stop command
  * RestartSec=: if auto restart enabled, specify interval of restart attempts
  * Restart=: the circumstances under which systemd will attempt to automatically restart the service (always, on-success, on-failure, on-abnormal, on-abort, on-watchdog) - trigger a restart according to the way that the service was stopped
  * TimeoutSec= / TimeoutStartSec= / TimeoutStopSec= : max amount of time to stop or start a service

=== [Socket] Section (socket-based activation, Each socket unit must have a matching service unit that will be activated when the socket receives activity) ===

  * ListenStream=: TCP socket address
  * ListenDatagram=: UDP socket address
  * ListenSequentialPacket=: UNIX socket address
  * ListenFIFO: fifo buffer address
  * Accept=: whether an additional instance of the service will be started for each connection (default : false)
  * SocketUser= / SocketGroup= : Unix socket, specifies the owner/group of the socket (default : root/root)
  * SocketMode= : Unix sockets or FIFO buffers, this sets the permissions on the created entity
  * Service=: if the service name does not match the .socket name, the service can be specified with this directive.

=== [Mount] Section (leading slash is removed, all other slashes are translated into dashes "-", and all dashes and unprintable characters are replaced with C-style escape codes ;  implicit dependency on other mounts above it in the hierarchy) ===

  * What=: The absolute path to the resource that needs to be mounted.
  * Where=: The absolute path of the mount point where the resource should be mounted (conventional notation)
  * Type=: The filesystem type of the mount.
  * Options=: Any mount options that need to be applied. This is a comma-separated list.
  * SloppyOptions=: A boolean that determines whether the mount will fail if there is an unrecognized mount option.
  * DirectoryMode=: If parent directories need to be created for the mount point, this determines the permission mode of these directories.
  * TimeoutSec=: Configures the amount of time the system will wait until the mount operation is marked as failed.

=== [Automount] Section (allows an associated .mount unit to be automatically mounted at boot, Mount notation) ===

  * Where=: The absolute path of the automount point on the filesystem. This will match the filename except that it uses conventional path notation instead of the translation.
  * DirectoryMode=: If the automount point or any parent directories need to be created, this will determine the permissions settings of those path components.

=== [Swap] Section (swap file or device, Mount notation) ===

  * What=: The absolute path to the location of the swap space, whether this is a file or a device.
  * Priority=: This takes an integer that indicates the priority of the swap being configured.
  * Options=: Any options that are typically set in the /etc/fstab file can be set with this directive instead. A comma-separated list is used.
  * TimeoutSec=: The amount of time that systemd waits for the swap to be activated before marking the operation as a failure.

=== [Path] Section (defines a filesystem path that systmed can monitor for changes) ===

Another unit must exist that will be be activated when certain activity is detected at the path location. Path activity is determined thorugh inotify events.
  * PathExists=: This directive is used to check whether the path in question exists. If it does, the associated unit is activated.
  * PathExistsGlob=: This is the same as the above, but supports file glob expressions for determining path existence.
  * PathChanged=: This watches the path location for changes. The associated unit is activated if a change is detected when the watched file is closed.
  * PathModified=: This watches for changes like the above directive, but it activates on file writes as well as when the file is closed.
  * DirectoryNotEmpty=: This directive allows systemd to activate the associated unit when the directory is no longer empty.
  * Unit=: This specifies the unit to activate when the path conditions specified above are met. If this is omitted, systemd will look for a .service file that shares the same base unit name as this unit.
  * MakeDirectory=: This determines if systemd will create the directory structure of the path in question prior to watching.
  * DirectoryMode=: If the above is enabled, this will set the permission mode of any path components that must be created.

=== [Timer] Section (used to schedule tasks to operate at a specific time or after a certain delay.) ===

  * OnActiveSec=: This directive allows the associated unit to be activated relative to the .timer unit's activation.
  * OnBootSec=: This directive is used to specify the amount of time after the system is booted when the associated unit should be activated.
  * OnStartupSec=: This directive is similar to the above timer, but in relation to when the systemd process itself was started.
  * OnUnitActiveSec=: This sets a timer according to when the associated unit was last activated.
  * OnUnitInactiveSec=: This sets the timer in relation to when the associated unit was last marked as inactive.
  * OnCalendar=: This allows you to activate the associated unit by specifying an absolute instead of relative to an event.
  * AccuracySec=: This unit is used to set the level of accuracy with which the timer should be adhered to. By default, the associated unit will be activated within one minute of the timer being reached. The value of this directive will determine the upper bounds on the window in which systemd schedules the activation to occur.
  * Unit=: This directive is used to specify the unit that should be activated when the timer elapses. If unset, systemd will look for a .service unit with a name that matches this unit.
  * Persistent=: If this is set, systemd will trigger the associated unit when the timer becomes active if it would have been triggered during the period in which the timer was inactive.
  * WakeSystem=: Setting this directive allows you to wake a system from suspend if the timer is reached when in that state.

=== [Slice] Section : man systemd.resource-control ===

=== [Install] Section Directives (optional) ===

  * WantedBy=: Want injection to target via symlinks in /etc/systemd/system/<target>.wants
  * RequiredBy=: idem, /etc/systemd/system/<target>.requires
  * Alias=: unit can be enabled under another name, can serve as function provider service
  * Also=: enabling/disabling as a set
  * DefaultInstance=: used for template units (fallback value for name)

==== Creating Instance Units from Template Unit Files ====

Provide flexibility in configuring units by allowing certain parts of the file to utilize dynamic information that will be available at runtime.
identification : 
  * templace : example@.service
  * instance : example@instance1.service (symbolic link to the template file)

==== Templating (within the usual unit definition) ====
  * %n : Anywhere where this appears in a template file, the full resulting unit name will be inserted.
  * %N : This is the same as the above, but any escaping, such as those present in file path patterns, will be reversed.
  * %p : This references the unit name prefix. This is the portion of the unit name that comes before the @ symbol.
  * %P : This is the same as above, but with any escaping reversed.
  * %i : This references the instance name, which is the identifier following the @ in the instance unit. This is one of the most commonly used specifiers because it will be guaranteed to be dynamic. The use of this identifier encourages the use of configuration significant identifiers. For example, the port that the service will be run at can be used as the instance identifier and the template can use this specifier to set up the port specification.
  * %I : This specifier is the same as the above, but with any escaping reversed.
  * %f : This will be replaced with the unescaped instance name or the prefix name, prepended with a /.
  * %c : This will indicate the control group of the unit, with the standard parent hierarchy of /sys/fs/cgroup/ssytemd/ removed.
  * %u : The name of the user configured to run the unit.
  * %U : The same as above, but as a numeric UID instead of name.
  * %H : The host name of the system that is running the unit.
  * %% : This is used to insert a literal percentage sign.